300+ inetd processes burning up RAM, crashing services

dniesen

We are running nanobsd 2.0.1-RELEASE (i386) and recently have been having an inordinate amount of inetd processes spawning that chews up the available RAM then causing services to crash. OpenVPN always seems to be the first to go but IPSEC follows shortly after.

Rebooting gets rid of the processes but within a day or two the problem starts up again.

Our number of NAT rules is not insignificant, here is the contents of our /var/etc/inetd.conf file:


tftp-proxy      dgram   udp     wait            root    /usr/libexec/tftp-proxy tftp-proxy -v
19000   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.50 1194
19001   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.50 22
19002   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.103 51413
19002   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.103 51413
19003   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.114 443
19004   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.222 8040
19005   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.222 8041
19006   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.102 5500
19007   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.107 80
19008   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.152 5500
19009   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.109 5500
19010   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.222 445
19011   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.222 135
19012   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.222 139
19013   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.106 80
19014   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.79 33333
19015   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.79 5666
19016   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.119 22
19017   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.39.200.100 22
19018   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.120 8080
19019   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.120 6443
19020   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.120 6080
19021   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.157 8080
19021   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.157 8080
19022   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.157 8443
19022   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.157 8443
19023   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.157 8880
19023   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.157 8880
19024   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.157 8843
19024   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.157 8843
19025   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.157 22
19025   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.157 22
19026   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.157 3478
19026   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.157 3478
19027   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.160 3389
19028   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.154 6080
19028   dgram   udp     nowait/0        nobody  /usr/bin/nc     nc -u -w 2000 10.29.0.154 6080
19029   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.119 80
19030   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.223 443
19031   stream  tcp     nowait/0        nobody  /usr/bin/nc     nc -w 2000 10.29.0.223 80

Is this normal and I just need a beefier firewall for the amount of traffic we're pushing? We have an ALIX 2D3 board with 256MB of RAM. I haven't looked at upgrading the RAM yet but I assume that might be possible.

dhatz

Actually these are nc (netcat) processes, most probably due to you having enabled "NAT reflection".
The quickest/easiest way would be to just disable NAT reflection (System -> Advanced -> Firewall/NAT) and use "split-brain DNS".

Another reason to disable NAT reflection is that the current netcat-based NAT reflection has problems with UDP, see
http://redmine.pfsense.org/issues/2281

dniesen

Dang, split-brain DNS is not ideal. I suppose I need to set our internal DNS server to check pfSense first before external DNS then to make that work?

Is it the UDP NAT reflection that causes the high usage?

That ticket doesn't seem to be getting much traction.

cmb

2.1 can do pure NAT mode reflection which gets rid of netcat. That ticket probably just needs to be closed since the pure NAT mode should work fine in every case including UDP. The original mode is retained just in case someone prefers it.