Periodic traffic interruption on non standard ports (8081, 8082)
-
Hi, I've been running around trying to identify an issue that has appeared in my environment. I'm not sure if it's a bug in 2.5.2-RELEASE (CE Edition) or something just weird happening.
I have a PFSense box as my gateway. One of my Static IP's I have set up as a 1 to 1 Nat to another internal firewall. The Gateway box firewall config for that ip is basic. It allows through a handful of ports TCP, (22,80,220,443,1080,8081,8082) all for basic use (SSH, HTTP, HTTPS that sort of thing). The internal firewall does any inspection on the subset of ports if needed.
Everything seems to be working fine, but periodically 8081 and 8082 stop passing traffic to the servers behind everything. These happen to be HTTPS ports for what it's worth. I have narrowed it down to the gateway pfsense box.
I have ran packet captures on the WAN and LAN side and and clearly the traffic isn't passing the box.
I enabled logging on those specific rules and when I Try to pass traffic across those ports nothing shows up in the logs.
I tried removing the firewall entries and re-entering them , that made no difference
I suspected that something that was running on the box (additional package) was stealing the ports, so I've uninstalled (and rebooted just about everything). The only remaining packages installed are arpwatch, frr, mailreport, mrt-nox11, openvpn-client-export, siproxd.
nothing shows up in an netstat on those ports.
Usually , if I reboot the pfsense box the issue clears for a period of time and then it comes back. In this last case the box was last rebooted Jan 10th at 11:30pm. and at 3:11am on the 12th port 8081 stopped responding and at 3:15am port 8082 stopped (according to my NMS).
and lastly, I tried trying forwarding 8081 on one of the other NAT's that I have to another server, and it's not working either, so it seems to be port related not specific IP/Nat related for sure.
I'm really confused what could be causing this.
I'm going to try and reboot the box again tonight again to try and get it to clear again, but it's certainly not a fix lol.
I appreciate any suggestions/comments.
Thanks !
-
@dlewis_nepean
Just to updateRebooted the box the evening that I posted this (Jan 12th) @ 11:54pm,
Issue cleared, ports were open again .@2:25am on the 13th (about 2 and half hours later) they both started failing again.
the only thing I saw in the logs was a reference to
kernel: sonewconn: pcb 0xfffff802cef9f3d0: Listen queue overflow: 8 already in queue awaiting acceptance (2 occurrences)
after some reading I found a reference related to setting
kern.ipc.somaxconn to a larger value. So I picked 4096
I rebooted the box again last night (the 13th) around 10:30pm.
Issue cleared this time until 7:25am this morning.Still scratching my head as to a solution to this.
-
It looks like I found the issue.
Thanks to another post elsewhere that pointed me to the status.php page, which gave me some information that I wasn't seeing elsewhere.
I discovered that what was messing things up was a combination of my NVR and UPNP. I had UPNP enabled for some other devices as a recommendation for other issues, which was fine, but I didn't know my NVR was using those ports, and as well I didn't realize that with UPNP that if you have multiple WAN IP's that the UPNP would grab those ports globally, which is exactly what it's doing. So even though my device is using the global many-to-one nat'd ip to the web, UPNP is grabbing the ports across the entire bank of IP's that I have.
So Fixed, which is good, but it would be nice if UPNP could be tied to one ip or another from a WAN perspective. I'm completely unsure if that is even something that could be coded, but it would be nice :)
Hopefully this will help someone else in the future.
-
@dlewis_nepean so your saying even with specific wan interface selected, or specific IP given in the external.
it still starts listening on all your external IPs?
-
Interesting, I missed that, let me try that and see what happens. That might be what I was looking for.
Thanks !
-
So tested the recommendation and it doesn't seem to work.
I have a PPPOE interface plus a /29 so 6 ip's total.the PPPOE interface is the one I use as my general nat. when I put that IP in the override wan address ( I even rebooted). The same situation happens. the 8081/8082 ports which are on one of the other ip's fail. The Device that is using the uPNP reports the correct external address. but when I go to the status page in pfsense, the entries that show those ports and internal IP show "any" under Ext IP.