[Partial Fix] NAT Reflection problem in 2.0-RC1
-
Does anyone know of a bug that would allow NAT reflection to spawn thousands of inetd processes? This is only on pfSense 2.0-RC1. Running the same system with the same traffic on pfSense 1.2.3-RELEASE works ok. We do not have a lot of internal reflection requests.
I should also point out if I change the reflection timeout to 5 seconds (for testing) and I telnet to a public ip within the network (SMTP) I see a new process appear but after the 5 seconds it disappears and disconnects the telnet connection. This shows me it's acting normal and disconnecting/terminating the process.
Something is doing on pfSense 2.0 that is doing this, any ideas?
Thanks!
ADDITIONAL INFO: Not using 1:1, port forwards only. Somehow something is triggering inetd over and over.
Runs fine on pfSense 1.2.3-RELEASE.
-
top:
last pid: 55541; load averages: 0.08, 0.16, 0.12 up 0+12:57:12 13:02:50
4968 processes:1 running, 4967 sleeping
CPU: 7.6% user, 4.7% nice, 11.9% system, 0.0% interrupt, 75.8% idle
Mem: 400M Active, 52M Inact, 221M Wired, 184K Cache, 110M Buf, 315M Free
Swap:PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
62300 root 1 45 0 12928K 9732K RUN 0:11 0.98% top
61681 root 1 55 0 60852K 5908K nanslp 1:44 0.00% php
2419 root 1 76 20 3656K 644K wait 0:22 0.00% sh
32137 root 1 44 0 3316K 676K select 0:08 0.00% apinger
28873 root 1 44 0 3436K 484K select 0:07 0.00% inetd
48712 dhcpd 1 44 0 6752K 1984K select 0:03 0.00% dhcpd
37469 root 1 76 0 54708K 3484K accept 0:02 0.00% php
37586 root 1 55 0 54708K 12716K accept 0:01 0.00% php
2539 root 1 76 0 3436K 1052K so_rcv 0:01 0.00% inetd
35358 root 1 44 0 6588K 1664K kqread 0:01 0.00% lighttpd
24077 root 1 44 0 5912K 176K bpf 0:01 0.00% tcpdump
36627 root 1 47 0 54708K 3256K accept 0:00 0.00% php
6709 root 1 71 0 3316K 240K nanslp 0:00 0.00% minicron
62632 _ntp 1 44 0 3316K 580K select 0:00 0.00% ntpd
23679 root 1 44 0 7992K 3496K select 0:00 0.00% sshd
23782 root 1 44 0 3380K 596K select 0:00 0.00% syslogd
37199 root 1 76 0 52660K 3248K wait 0:00 0.00% phpps aux:
root 61407 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 61409 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 61422 0.0 0.0 3436 36 ?? I 12:11AM 0:00.00 inetd: wrapping (
root 61444 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 61453 0.0 0.0 3436 36 ?? I 1:03AM 0:00.00 inetd: wrapping (
root 61454 0.0 0.1 3436 1132 ?? I 11:03AM 0:00.00 inetd: wrapping (
root 61462 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 61472 0.0 0.1 3436 1132 ?? I 11:13AM 0:00.00 inetd: wrapping (
root 61473 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 61474 0.0 0.1 3436 1132 ?? I 12:27PM 0:00.00 inetd: wrapping (
root 61489 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 61491 0.0 0.0 3436 36 ?? I 12:25AM 0:00.00 inetd: wrapping (
root 61495 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 61548 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 61583 0.0 0.1 3436 1132 ?? I 10:58AM 0:00.00 inetd: wrapping (
root 61597 0.0 0.1 3436 1104 ?? I 8:02AM 0:00.00 inetd: wrapping (
root 61609 0.0 0.1 3436 1132 ?? I 11:13AM 0:00.00 inetd: wrapping (
root 61614 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 61632 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 61665 0.0 0.0 3436 36 ?? I 3:59AM 0:00.00 inetd: wrapping (
root 61679 0.0 0.1 3436 1132 ?? I 10:00AM 0:00.00 inetd: wrapping (
root 61683 0.0 0.0 3436 36 ?? I 4:11AM 0:00.00 inetd: wrapping (
root 61693 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 61714 0.0 0.1 3436 1104 ?? I 8:02AM 0:00.00 inetd: wrapping (
root 61719 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 61782 0.0 0.0 3436 36 ?? I 1:03AM 0:00.00 inetd: wrapping (
root 61798 0.0 0.1 3436 1132 ?? I 10:41AM 0:00.00 inetd: wrapping (
root 61799 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 61832 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 61844 0.0 0.0 3436 36 ?? I 1:03AM 0:00.00 inetd: wrapping (
root 61866 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 61867 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 61919 0.0 0.0 3436 36 ?? I 3:59AM 0:00.00 inetd: wrapping (
root 61933 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 61940 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 61975 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 61985 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 61994 0.0 0.1 3436 1132 ?? I 10:41AM 0:00.00 inetd: wrapping (
root 62000 0.0 0.1 3436 1132 ?? I 10:00AM 0:00.00 inetd: wrapping (
root 62045 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 62051 0.0 0.1 3436 1132 ?? I 10:41AM 0:00.00 inetd: wrapping (
root 62056 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 62061 0.0 0.0 3436 36 ?? I 1:21AM 0:00.00 inetd: wrapping (
root 62077 0.0 0.1 3436 1132 ?? I 12:00PM 0:00.00 inetd: wrapping (
root 62083 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62084 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 62087 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62097 0.0 0.1 3436 1132 ?? I 10:51AM 0:00.00 inetd: wrapping (
root 62129 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 62144 0.0 0.0 3436 36 ?? I 1:03AM 0:00.00 inetd: wrapping (
root 62146 0.0 0.1 3436 1132 ?? I 12:00PM 0:00.00 inetd: wrapping (
root 62153 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 62159 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 62189 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 62191 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62197 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 62211 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 62280 0.0 0.0 3436 36 ?? I 1:21AM 0:00.00 inetd: wrapping (
root 62282 0.0 0.1 3436 1132 ?? I 10:00AM 0:00.00 inetd: wrapping (
root 62294 0.0 0.0 3436 36 ?? I 3:59AM 0:00.00 inetd: wrapping (
root 62317 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 62322 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 62326 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62348 0.0 0.0 3436 36 ?? I 7:22AM 0:00.00 inetd: wrapping (
root 62350 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 62356 0.0 0.1 3436 1132 ?? I 10:51AM 0:00.00 inetd: wrapping (
root 62359 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 62361 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 62368 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 62371 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 62378 0.0 0.0 3436 36 ?? I 3:59AM 0:00.00 inetd: wrapping (
root 62392 0.0 0.1 3436 1132 ?? I 10:51AM 0:00.00 inetd: wrapping (
root 62396 0.0 0.1 3436 1132 ?? I 10:58AM 0:00.00 inetd: wrapping (
root 62418 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62425 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 62448 0.0 0.0 3436 36 ?? I 1:03AM 0:00.00 inetd: wrapping (
root 62475 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 62479 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 62480 0.0 0.1 3436 1132 ?? I 10:58AM 0:00.00 inetd: wrapping (
root 62485 0.0 0.1 3436 1132 ?? I 10:51AM 0:00.00 inetd: wrapping (
root 62488 0.0 0.1 3436 1132 ?? I 12:47PM 0:00.00 inetd: wrapping (
root 62489 0.0 0.1 3436 1132 ?? I 12:47PM 0:00.00 inetd: wrapping (
root 62510 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 62518 0.0 0.1 3436 1132 ?? I 11:13AM 0:00.00 inetd: wrapping (
root 62519 0.0 0.0 3436 36 ?? I 1:03AM 0:00.00 inetd: wrapping (
root 62522 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 62530 0.0 0.0 3436 36 ?? I 3:59AM 0:00.00 inetd: wrapping (
root 62547 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 62554 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 62557 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62582 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 62586 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 62601 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 62604 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62605 0.0 0.0 3436 36 ?? I 1:03AM 0:00.00 inetd: wrapping (
root 62613 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 62627 0.0 0.1 3436 1132 ?? I 10:00AM 0:00.00 inetd: wrapping (
root 62633 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 62653 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 62660 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 62661 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62667 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 62682 0.0 0.1 3436 1132 ?? I 10:51AM 0:00.00 inetd: wrapping (
root 62704 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 62721 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 62739 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 62745 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 62749 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 62750 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 62757 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 62762 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 62781 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 62794 0.0 0.0 3436 36 ?? I 3:59AM 0:00.00 inetd: wrapping (
root 62820 0.0 0.1 3436 1132 ?? I 10:41AM 0:00.00 inetd: wrapping (
root 62824 0.0 0.0 3436 36 ?? I 2:59AM 0:00.00 inetd: wrapping (
root 62835 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 62869 0.0 0.1 3436 1132 ?? I 10:00AM 0:00.00 inetd: wrapping (
root 62876 0.0 0.1 3436 1132 ?? I 12:47PM 0:00.00 inetd: wrapping (
root 62885 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 62892 0.0 0.1 3436 1132 ?? I 10:44AM 0:00.00 inetd: wrapping (
root 62896 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 62916 0.0 0.1 3436 1132 ?? I 10:51AM 0:00.00 inetd: wrapping (
root 62929 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 62930 0.0 0.0 3436 36 ?? I 2:50AM 0:00.00 inetd: wrapping (
root 62931 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 62946 0.0 0.1 3436 992 ?? I 7:22AM 0:00.00 inetd: wrapping (
root 62956 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 62957 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 62958 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 62970 0.0 0.0 3436 36 ?? I 3:59AM 0:00.00 inetd: wrapping (
root 62972 0.0 0.0 3316 416 ?? Ss 12:06AM 0:00.03 ntpd: [priv] (ntp
root 62988 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 63003 0.0 0.1 3436 1132 ?? I 10:51AM 0:00.00 inetd: wrapping (
root 63015 0.0 0.0 3436 36 ?? I 3:38AM 0:00.00 inetd: wrapping (
root 63024 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 63036 0.0 0.1 3436 1132 ?? I 10:51AM 0:00.00 inetd: wrapping (
root 63049 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 63050 0.0 0.1 3436 1132 ?? I 11:03AM 0:00.00 inetd: wrapping (
root 63060 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 63061 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 63062 0.0 0.1 3436 1132 ?? I 10:00AM 0:00.00 inetd: wrapping (
root 63087 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 63091 0.0 0.1 3436 1132 ?? I 11:03AM 0:00.00 inetd: wrapping (
root 63096 0.0 0.1 3436 1132 ?? I 10:41AM 0:00.00 inetd: wrapping (
root 63097 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 63116 0.0 0.1 3436 1132 ?? I 10:54AM 0:00.00 inetd: wrapping (
root 63152 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 63160 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 63172 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 63173 0.0 0.0 3436 36 ?? I 2:50AM 0:00.00 inetd: wrapping (
root 63180 0.0 0.1 3436 1132 ?? I 10:41AM 0:00.00 inetd: wrapping (
root 63183 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 63184 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 63185 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 63187 0.0 0.1 3436 1132 ?? I 11:03AM 0:00.00 inetd: wrapping (
root 63200 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 63206 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 63216 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 63248 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 63343 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 63344 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 63382 0.0 0.0 3436 36 ?? I 3:49AM 0:00.00 inetd: wrapping (
root 63388 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 63395 0.0 0.0 3436 36 ?? I 2:26AM 0:00.00 inetd: wrapping (
root 63416 0.0 0.0 3436 36 ?? I 2:59AM 0:00.00 inetd: wrapping (
root 63419 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 63433 0.0 0.0 3436 36 ?? I 2:50AM 0:00.00 inetd: wrapping (
root 63440 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 63457 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 63459 0.0 0.1 3436 1132 ?? I 11:03AM 0:00.00 inetd: wrapping (
root 63486 0.0 0.0 3436 36 ?? I 1:23AM 0:00.00 inetd: wrapping (
root 63505 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 63511 0.0 0.1 3436 1132 ?? I 12:47PM 0:00.00 inetd: wrapping (
root 63512 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 63524 0.0 0.1 3436 1132 ?? I 10:15AM 0:00.00 inetd: wrapping (
root 63554 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 63576 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 63585 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 63592 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 63609 0.0 0.1 3436 1132 ?? I 11:03AM 0:00.00 inetd: wrapping (
root 63610 0.0 0.1 3436 1132 ?? I 11:03AM 0:00.00 inetd: wrapping (
root 63631 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 63653 0.0 0.0 3436 36 ?? I 2:59AM 0:00.00 inetd: wrapping (
root 63658 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 63659 0.0 0.1 3436 1132 ?? I 12:47PM 0:00.00 inetd: wrapping (
root 63661 0.0 0.0 3436 36 ?? I 12:39AM 0:00.00 inetd: wrapping (
root 63709 0.0 0.0 3436 36 ?? I 2:50AM 0:00.00 inetd: wrapping (
root 63719 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 63724 0.0 0.0 3436 36 ?? I 2:50AM 0:00.00 inetd: wrapping (
root 63725 0.0 0.1 3436 1132 ?? I 11:03AM 0:00.00 inetd: wrapping (
root 63731 0.0 0.0 3436 36 ?? I 2:59AM 0:00.00 inetd: wrapping (
root 63733 0.0 0.1 3436 1132 ?? I 10:54AM 0:00.00 inetd: wrapping (
root 63793 0.0 0.1 3436 1104 ?? I 8:02AM 0:00.00 inetd: wrapping (
root 63821 0.0 0.0 3436 36 ?? I 2:50AM 0:00.00 inetd: wrapping (
root 63829 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 63834 0.0 0.1 3436 1132 ?? I 10:48AM 0:00.00 inetd: wrapping (
root 63840 0.0 0.0 3436 36 ?? I 2:59AM 0:00.00 inetd: wrapping (
root 63856 0.0 0.1 3436 1132 ?? I 10:44AM 0:00.00 inetd: wrapping (
root 63864 0.0 0.1 3436 1132 ?? I 10:44AM 0:00.00 inetd: wrapping (
root 63883 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 63889 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 63896 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 63897 0.0 0.1 3436 1132 ?? I 10:44AM 0:00.00 inetd: wrapping (
root 63911 0.0 0.1 3436 1132 ?? I 12:47PM 0:00.00 inetd: wrapping (
root 63922 0.0 0.0 3436 36 ?? I 12:12AM 0:00.00 inetd: wrapping (
root 63930 0.0 0.0 3436 36 ?? I 12:36AM 0:00.00 inetd: wrapping (
root 63954 0.0 0.0 3436 36 ?? I 2:59AM 0:00.00 inetd: wrapping (
root 63959 0.0 0.0 3436 36 ?? I 3:57AM 0:00.00 inetd: wrapping (
root 63965 0.0 0.0 3436 36 ?? I 12:13AM 0:00.00 inetd: wrapping (
root 63982 0.0 0.1 3436 1132 ?? I 12:54PM 0:00.00 inetd: wrapping (
root 63992 0.0 0.0 3436 36 ?? I 12:50AM 0:00.00 inetd: wrapping (
root 2419 0.0 0.1 3656 644 u0- IN 12:06AM 0:21.87 /bin/sh /var/db/r
root 24077 0.0 0.0 5912 176 u0- S 12:06AM 0:00.51 /usr/sbin/tcpdump
root 24118 0.0 0.0 3316 28 u0- I 12:06AM 0:00.00 logger -t pf -p l
root 29819 0.0 0.0 3684 72 u0 Is 12:06AM 0:00.01 login [pam] (logi
root 29907 0.0 0.0 3656 44 u0 I 12:06AM 0:00.00 -sh (sh)
root 31900 0.0 0.0 3656 44 u0 I 12:06AM 0:00.00 /bin/sh /etc/rc.i
root 45512 0.0 0.0 4696 40 u0 I+ 12:06AM 0:00.01 /bin/tcsh
root 61681 0.0 0.6 60852 5908 u0- S 12:06AM 1:44.01 /usr/local/bin/ph
_ntp 62632 0.0 0.1 3316 580 u0- I 12:06AM 0:00.07 ntpd: ntp engine
root 26104 0.0 0.1 3656 1540 0 Is 12:53PM 0:00.01 /bin/sh /etc/rc.i
root 34185 0.0 0.2 4696 2352 0 S 12:54PM 0:00.01 /bin/tcsh
root 61555 0.0 0.5 8576 5300 0 R+ 1:03PM 0:00.02 ps auxand so on…
Has to be NAT reflection, but we do not have that many reflection requests. Any reason why it would be triggered like this over and over? System crashes everyday now with 1GB of ram.
-
Just confirmed this problem was on pfSense 2.0-RC1 PC Install and embedded. I was not waiting long enough before for the processes to build. Before I did the change over about a week ago I was running pfSense 1.2.3.-Release with no issues. I will go back to that and see if the problem is solved, if not maybe something is doing inside our network. I will let you all know.
-
Problem confirmed.
Appears to be a glitch in NAT Reflection.
Reverting back to PfSense 1.2.3-RELEASE resolved the NAT reflection problem where there were thousands of processes. The same rules are setup, the same forwards, etc. The problem appears to be that on pfSense 2.0-RC1 the inetd/netcat processes are not terminating properly in all cases so they continue to build. The problem does not happen with little traffic but for some reason when I push traffic through it it crashes eventually. The second I push traffic I have about 300 processes immediately of inetd then they continue to build until the system runs out of memory. Running on 1.2.3-RELEASE is no problem and reflection works properly.
-
Found the problem on 2.0-RC1 that does not exist on 1.2.3-RELEASE:
NAT Reflection on 2.0-RC1 works normal for all hosts, however if you run DNS servers on the internal network with NAT that are also accessed externally (i.e. name servers for a domain name) then if a request is made inside the network to the external IP address for port 53/UDP then it triggers a problem inside the NAT Reflection scripts/system.
I confirmed this is only for port 53/UDP – no other ports are affected as far as I can tell.
In 1.2.3-RELEASE the NAT reflection for DNS does not work which is probably why the the issue is not on this version.
On 2.0-RC1 NAT reflection for DNS does appear to work but there is a bug that causes these processes to not terminate or keep building. Again, this is only on port 53/UDP DNS traffic. To test try to access your DNS servers with the external IP.
Hope this helps and I will continue to try and locate the source of the problem.
If there is a reason why DNS does not work with NAT reflection on 1.2.3-RELEASE? Knowing this I can narrow in on the problem in 2.0-RC1.
ADDITIONAL INFO: No DNS forwarding service, DNS server or other package is installed that would also run on 53/UDP.
NOTE: You really will not come across this problem unless you have multiple servers inside your network, DNS servers that need to be accessible inside and out, and web servers that lookup domains on other servers also inside your network that also run DNS.
TEMPORARY WORK AROUND IN 2.0-RC1:
- Go to NAT and edit the rules for your DNS server(s) - disable NAT Reflection
- Reboot to clear all existing inetd processes
This does not provide a permanent fix but it will stop the inetd spawning problem and it will work as it does in 1.2.3-RELEASE with no NAT reflection for DNS. As soon as I can track the problem down and create a permanent fix I will post it here.
-
Try to use DNS forwarder for an internal domain….and see if it blocks all NAT reflection.
-
Try to use DNS forwarder for an internal domain….and see if it blocks all NAT reflection.
Hi supermule: The only concern is that we have multiple web servers and some of them run DNS so they need to refference there internal DNS as opposed to a single network wide DNS. So what happens is they work fine and resolve all domains outside of the network but if there is a domain inside the network that is not on the DNS within that server but on another DNS server within the network they fail because they cannot make it back in via a public/external IP.
-
Yes…..i couldnt get in touch with any of my sites that used pfsense as DNS forwarder...so I got the PFSense login page in 2.0. Did exactly the same as in 1.2.3 which worked no issues.
-
Yes…..i couldnt get in touch with any of my sites that used pfsense as DNS forwarder...so I got the PFSense login page in 2.0. Did exactly the same as in 1.2.3 which worked no issues.
Yes, we actually have no problems with any ports under NAT reflection except DNS. I found that NAT reflection just is not reflecting DNS traffic. In 2.0-RC1 it works (sort of), it tries to forward DNS traffic but instead the inetd processes hang up and keep building with each new DNS request. They never close out so eventually the system will run out of memory. Now if you go to the port forwards for all of your DNS servers on 2.0-RC1 and disable reflections the problem stops with processes building but you still can't reflect DNS requests.
I wonder if it's just DNS or all UDP traffic?
-
I have seen this as well, but I haven't figured out what causes it. I think it is indeed UDP in general. First thing to check would probably be the lines in /var/etc/inetd.conf to see if anything has changed between those versions. I'll get a VM of 1.2.3 up and running to check it out myself.
-
Thanks :)
@Efonne:
I have seen this as well, but I haven't figured out what causes it. I think it is indeed UDP in general. First thing to check would probably be the lines in /var/etc/inetd.conf to see if anything has changed between those versions. I'll get a VM of 1.2.3 up and running to check it out myself.
-
The only difference I see in inetd.conf is that there is a tab instead of a space between the program path and arg list in 2.0. I'll also check out the parameters used to run the program to see if anything changed there.
-
@Efonne:
The only difference I see in inetd.conf is that there is a tab instead of a space between the program path and arg list in 2.0. I'll also check out the parameters used to run the program to see if anything changed there.
I spent hours looking through it the other night. I cannot tell whether the problem is DNS specific or all UDP traffic. I did a script that hit the server with several DNS requests in a row and they never closed out they actually just 'hung'. Now, NAT reflection is working for everything else it's only DNS that has been triggering it although I don't have any other UDP services I can test against.
The reason it doesn't happen in 1.2.3-RELEASE is because DNS reflection does not work at all so there is no way it could have the inetd/spawning issue. Every other NAT reflection rule works perfectly, just the DNS (or UDP?) Perhaps DNS NAT Reflection is not working in 1.2.3-RELEASE for a reason? Maybe there was a conflict somewhere. Maybe something running on pfSense is triggering itself? – I do have all DNS services disabled on pfSense too. Don't know too much about netcat.
I will continue to test with DNS/UDP traffic and post the results.
-
FYI Running on 2.0-RC1 again. Since I disabled NAT reflection for all internal DNS servers the issue is completely gone, I have been running in production/under load for awhile now and the issue is gone.
Now, I will setup another system and allow NAT reflection for DNS to reproduce and pinpoint the issue. I will also test with UDP traffic, not just DNS.
-
For what it's worth, I was also having runaway process totals with 2.0-RC1. Disabling reflection on DNS rules fixed it for me.