Firewall stops working with UDP nat reflection

adam65535

I learned that UDP nat reflection has never really worked but assumed it did so I created a rule with NAT reflection (which is the default but not sure if I changed something a long time ago to make it the default or not).

I hosted the UDP service on an internal LAN client desktop ip (192.168.0.70 for instance). I then launched the client from the same internal desktop IP to connect to the external IP. I then expected the firewall to create a source port NAT connection back to the client. I noticed that the server did not like the connection from the client and didn't respond properly. I could not connect properly to the service.

A tcpdump trace on the client system showed that packets were being NATed with the LAN gateway IP address going back to the client which looked ok. I also noticed that every UDP packet that was sent was creating a new NAT connection going back to the client from the gateway (The second part of the reflection). The odd part is that every packet from client to external IP started to get amplified so that it would create a new connection going back to the client each time with a different/new source port. 1 packet sent to external IP would equal 1 packet back to client. The second packet to the external IP created 2 packets going back to the client. The third packet that was sent created 3 more packets going back to the client all with different NAT source ports.

A minute or so later the firewall stopped passing traffic. This is repeatable every time I do it. I have to pull the plug on the firewall to get it to work again.

21:53:59.251376 IP l.a.n.70.40363 > w.a.n.54.23099: UDP, length 36
21:53:59.251845 IP l.a.n.1.35620 > l.a.n.70.23098: UDP, length 36
21:53:59.252082 IP l.a.n.70.23098 > l.a.n.1.35620: UDP, length 33
21:53:59.757407 IP l.a.n.70.40363 > w.a.n.54.23099: UDP, length 36
21:53:59.759661 IP l.a.n.70.23098 > l.a.n.1.35620: UDP, length 33
21:53:59.770203 IP l.a.n.1.18585 > l.a.n.70.23098: UDP, length 36
21:53:59.776343 IP l.a.n.70.23098 > l.a.n.1.18585: UDP, length 34
21:54:00.278434 IP l.a.n.70.23098 > l.a.n.1.18585: UDP, length 34
21:54:00.770314 IP l.a.n.70.23098 > l.a.n.1.35620: UDP, length 33
21:54:00.773479 IP l.a.n.70.40363 > w.a.n.54.23099: UDP, length 36
21:54:00.786511 IP l.a.n.1.27079 > l.a.n.70.23098: UDP, length 36
21:54:00.786930 IP l.a.n.70.23098 > l.a.n.1.27079: UDP, length 34
21:54:01.289332 IP l.a.n.70.23098 > l.a.n.1.18585: UDP, length 34
21:54:01.289402 IP l.a.n.70.23098 > l.a.n.1.27079: UDP, length 34
21:54:02.294863 IP l.a.n.70.23098 > l.a.n.1.27079: UDP, length 34
21:54:02.772178 IP l.a.n.70.23098 > l.a.n.1.35620: UDP, length 33
21:54:02.788878 IP l.a.n.70.40363 > w.a.n.54.23099: UDP, length 36
21:54:02.801449 IP l.a.n.1.20942 > l.a.n.70.23098: UDP, length 36
21:54:02.802052 IP l.a.n.70.23098 > l.a.n.1.20942: UDP, length 34
21:54:03.294112 IP l.a.n.70.23098 > l.a.n.1.18585: UDP, length 34
21:54:03.302494 IP l.a.n.70.23098 > l.a.n.1.20942: UDP, length 34
21:54:04.295313 IP l.a.n.70.23098 > l.a.n.1.27079: UDP, length 34
21:54:04.311932 IP l.a.n.70.23098 > l.a.n.1.20942: UDP, length 34
21:54:06.317826 IP l.a.n.70.23098 > l.a.n.1.20942: UDP, length 34
21:54:06.777090 IP l.a.n.70.23098 > l.a.n.1.35620: UDP, length 33
21:54:06.788337 IP l.a.n.70.40363 > w.a.n.54.23099: UDP, length 36
21:54:06.800923 IP l.a.n.1.6351 > l.a.n.70.23098: UDP, length 36
21:54:06.804213 IP l.a.n.70.23098 > l.a.n.1.6351: UDP, length 34
21:54:06.903131 IP l.a.n.70.40363 > w.a.n.54.23099: UDP, length 10
21:54:06.915716 IP l.a.n.1.50623 > l.a.n.70.23098: UDP, length 10
21:54:07.297365 IP l.a.n.70.23098 > l.a.n.1.18585: UDP, length 34
21:54:07.305698 IP l.a.n.70.23098 > l.a.n.1.6351: UDP, length 34
21:54:08.300712 IP l.a.n.70.23098 > l.a.n.1.27079: UDP, length 34
21:54:08.316432 IP l.a.n.70.23098 > l.a.n.1.6351: UDP, length 34
21:54:10.319425 IP l.a.n.70.23098 > l.a.n.1.20942: UDP, length 34
21:54:10.319497 IP l.a.n.70.23098 > l.a.n.1.6351: UDP, length 34

This happened with an earlier september build so I updated to the latest version and it still locks up the firewall when I do it. I do notice that the firewall responds with an arp but any packets sent to the firewall do not get forwarded it seems when the firewall locks up. Unplugging the firewall and plugging it back it gets it to work again after it reboots unless I connect to the UDP external IP again with the client.

2.2-BETA (i386)
built on Wed Oct 08 15:28:14 CDT 2014
FreeBSD 10.1-RC1

cmb

UDP NAT reflection works with pure NAT mode, it's just the old mode where UDP wasn't supported.

I don't see any indication of amplification of traffic there by the firewall. Whatever you're reflecting traffic to, though, seems to start blasting connections back to the source IP it sees in the request it received - which is the firewall's IP, it has to be for reflection to function.

There isn't enough traffic there to tell you what the problem is definitively, but based on that clearly broken behavior, it seems highly likely you're confusing that application into pummeling your firewall to death (almost certainly maxing out the state table, which the NIC cycling link would help bring back down quickly).

jimp

If you have it set for NAT + Proxy it can blow up in that way. I haven't tried it on 2.2 but on 2.1.x it would accept the UDP connection and attempt to reflect it but fail and then it would hold the "connection" open and keep piling up.

Pure NAT mode should work.

adam65535

Thanks for the info guys. I did try the pure NAT reflection mode with UDP and the client still didn't work (i think it still locked up too) but maybe I need some 'Advanced' setting for reflection or netcat stayed running or something. I will do more careful testing of all the different advanced options with pure NAT mode when I get home tonight and verify that netcat (nat + proxy) processes are not running.

adam65535

It turns out UDP only seems to work in Pure NAT mode with the 'Enable automatic outbound NAT for Reflection' option in the advanced firewall settings is checked. If I use NAT + Proxy then the firewall stops accepting all LAN outgoing (or to the firewall connections including ping) when connections are initated from the LAN to the external IP on that UDP port. The firewall is able to connect to LAN systems during that time though which is odd. As soon as I reboot the firewall the LAN works again. The firewall console works just fine during all this time.

Didn't work
Pure NAT reflection (Enable automatic outbound NAT for Reflection = UNCHECKED):
LAN udp l.a.n.70:23098 (w.a.n.54:23098) <- l.a.n.70:44428 NO_TRAFFIC:SINGLE
LAN udp l.a.n.70:44428 -> l.a.n.70:23098 SINGLE:NO_TRAFFIC

–------------------------

Worked:
Pure NAT reflection (Enable automatic outbound NAT for Reflection = CHECKED):
LAN udp l.a.n.70:23098 (w.a.n.54:23098) <- l.a.n.70:59853 MULTIPLE:MULTIPLE
LAN udp l.a.n:33208 (l.a.n.70:59853) -> l.a.n.70:23098 MULTIPLE:MULTIPLE

Firewall stops passing incoming connections from the LAN. Can't ping firewall LAN IP, DNS fails, etc. Connections from the firewall itself to the lan (SSH to a lan server worked):
NAT + Proxy (Enable automatic outbound NAT for Reflection = CHECKED):

Didn't test this time but this is the scenario that caused the same issues above the last time I tested.
NAT + Proxy (Enable automatic outbound NAT for Reflection = UNCHECKED):

I am using rtl drivers so maybe this is just triggering some other hardware or rtl driver issue. With these results it seems the firewall should only allow Pure NAT with UDP because if something connects to the external IP on the UDP port it can cause traffic incoming from LAN clients to stop working completely. The text description in the Advanced firewal settings mentions that NAT + proxy works for UDP. At a minimum maybe that should be changed.

NAT Reflection mode for port forwards	 :
When enabled, this automatically creates additional NAT redirect rules for access to port forwards on your external IP addresses from within your internal networks. 

The NAT + proxy mode uses a helper program to send packets to the target of the port forward. It is useful in setups where the interface and/or gateway IP used for communication with the target cannot be accurately determined at the time the rules are loaded. Reflection rules are not created for ranges larger than 500 ports and will not be used for more than 1000 ports total between all port forwards. Only TCP and UDP protocols are supported.	

The pure NAT mode uses a set of NAT rules to direct packets to the target of the port forward. It has better scalability, but it must be possible to accurately determine the interface and gateway IP used for communication with the target at the time the rules are loaded. There are no inherent limits to the number of ports other than the limits of the protocols. All protocols available for port forwards are supported.	

Individual rules may be configured to override this system setting on a per-rule basis.

I am running the latest build during these tests…
2.2-BETA (i386)
built on Mon Oct 13 18:40:19 CDT 2014
FreeBSD 10.1-RC2

Hardware is an Axiomtek NA-0043A which I have been using for about 2 years on pfsense using the Realtek 8100C chipset with 1GB of ram.