[Partial Fix] NAT Reflection problem in 2.0-RC1
-
Problem confirmed.
Appears to be a glitch in NAT Reflection.
Reverting back to PfSense 1.2.3-RELEASE resolved the NAT reflection problem where there were thousands of processes. The same rules are setup, the same forwards, etc. The problem appears to be that on pfSense 2.0-RC1 the inetd/netcat processes are not terminating properly in all cases so they continue to build. The problem does not happen with little traffic but for some reason when I push traffic through it it crashes eventually. The second I push traffic I have about 300 processes immediately of inetd then they continue to build until the system runs out of memory. Running on 1.2.3-RELEASE is no problem and reflection works properly.
-
Found the problem on 2.0-RC1 that does not exist on 1.2.3-RELEASE:
NAT Reflection on 2.0-RC1 works normal for all hosts, however if you run DNS servers on the internal network with NAT that are also accessed externally (i.e. name servers for a domain name) then if a request is made inside the network to the external IP address for port 53/UDP then it triggers a problem inside the NAT Reflection scripts/system.
I confirmed this is only for port 53/UDP – no other ports are affected as far as I can tell.
In 1.2.3-RELEASE the NAT reflection for DNS does not work which is probably why the the issue is not on this version.
On 2.0-RC1 NAT reflection for DNS does appear to work but there is a bug that causes these processes to not terminate or keep building. Again, this is only on port 53/UDP DNS traffic. To test try to access your DNS servers with the external IP.
Hope this helps and I will continue to try and locate the source of the problem.
If there is a reason why DNS does not work with NAT reflection on 1.2.3-RELEASE? Knowing this I can narrow in on the problem in 2.0-RC1.
ADDITIONAL INFO: No DNS forwarding service, DNS server or other package is installed that would also run on 53/UDP.
NOTE: You really will not come across this problem unless you have multiple servers inside your network, DNS servers that need to be accessible inside and out, and web servers that lookup domains on other servers also inside your network that also run DNS.
TEMPORARY WORK AROUND IN 2.0-RC1:
- Go to NAT and edit the rules for your DNS server(s) - disable NAT Reflection
- Reboot to clear all existing inetd processes
This does not provide a permanent fix but it will stop the inetd spawning problem and it will work as it does in 1.2.3-RELEASE with no NAT reflection for DNS. As soon as I can track the problem down and create a permanent fix I will post it here.
-
Try to use DNS forwarder for an internal domain….and see if it blocks all NAT reflection.
-
Try to use DNS forwarder for an internal domain….and see if it blocks all NAT reflection.
Hi supermule: The only concern is that we have multiple web servers and some of them run DNS so they need to refference there internal DNS as opposed to a single network wide DNS. So what happens is they work fine and resolve all domains outside of the network but if there is a domain inside the network that is not on the DNS within that server but on another DNS server within the network they fail because they cannot make it back in via a public/external IP.
-
Yes…..i couldnt get in touch with any of my sites that used pfsense as DNS forwarder...so I got the PFSense login page in 2.0. Did exactly the same as in 1.2.3 which worked no issues.
-
Yes…..i couldnt get in touch with any of my sites that used pfsense as DNS forwarder...so I got the PFSense login page in 2.0. Did exactly the same as in 1.2.3 which worked no issues.
Yes, we actually have no problems with any ports under NAT reflection except DNS. I found that NAT reflection just is not reflecting DNS traffic. In 2.0-RC1 it works (sort of), it tries to forward DNS traffic but instead the inetd processes hang up and keep building with each new DNS request. They never close out so eventually the system will run out of memory. Now if you go to the port forwards for all of your DNS servers on 2.0-RC1 and disable reflections the problem stops with processes building but you still can't reflect DNS requests.
I wonder if it's just DNS or all UDP traffic?
-
I have seen this as well, but I haven't figured out what causes it. I think it is indeed UDP in general. First thing to check would probably be the lines in /var/etc/inetd.conf to see if anything has changed between those versions. I'll get a VM of 1.2.3 up and running to check it out myself.
-
Thanks :)
@Efonne:
I have seen this as well, but I haven't figured out what causes it. I think it is indeed UDP in general. First thing to check would probably be the lines in /var/etc/inetd.conf to see if anything has changed between those versions. I'll get a VM of 1.2.3 up and running to check it out myself.
-
The only difference I see in inetd.conf is that there is a tab instead of a space between the program path and arg list in 2.0. I'll also check out the parameters used to run the program to see if anything changed there.
-
@Efonne:
The only difference I see in inetd.conf is that there is a tab instead of a space between the program path and arg list in 2.0. I'll also check out the parameters used to run the program to see if anything changed there.
I spent hours looking through it the other night. I cannot tell whether the problem is DNS specific or all UDP traffic. I did a script that hit the server with several DNS requests in a row and they never closed out they actually just 'hung'. Now, NAT reflection is working for everything else it's only DNS that has been triggering it although I don't have any other UDP services I can test against.
The reason it doesn't happen in 1.2.3-RELEASE is because DNS reflection does not work at all so there is no way it could have the inetd/spawning issue. Every other NAT reflection rule works perfectly, just the DNS (or UDP?) Perhaps DNS NAT Reflection is not working in 1.2.3-RELEASE for a reason? Maybe there was a conflict somewhere. Maybe something running on pfSense is triggering itself? – I do have all DNS services disabled on pfSense too. Don't know too much about netcat.
I will continue to test with DNS/UDP traffic and post the results.
-
FYI Running on 2.0-RC1 again. Since I disabled NAT reflection for all internal DNS servers the issue is completely gone, I have been running in production/under load for awhile now and the issue is gone.
Now, I will setup another system and allow NAT reflection for DNS to reproduce and pinpoint the issue. I will also test with UDP traffic, not just DNS.
-
For what it's worth, I was also having runaway process totals with 2.0-RC1. Disabling reflection on DNS rules fixed it for me.