Connectivity Issues and Listen Queue Overflow

emerge30

2 days ago, I started seeing some intermittent connectivity issues to and from the outside world. Nothing has materially changed with the setup in several months, the only recent change being an upgrade to 2.2.5 about 2 or 3 weeks ago.

I am seeing several notices for Listen Queue Overflow in the system logs, without much in the way of a reason why. The hex code doesn't match any active connections in netstat.

Dec 18 09:06:44	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (807 occurrences)
Dec 18 09:07:44	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (291 occurrences)
Dec 18 09:08:44	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (610 occurrences)
Dec 18 09:09:45	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (747 occurrences)
Dec 18 09:10:45	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (510 occurrences)
Dec 18 09:11:45	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (755 occurrences)
Dec 18 09:12:45	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (655 occurrences)
Dec 18 09:13:45	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (760 occurrences)
Dec 18 09:14:48	kernel: sonewconn: pcb 0xfffff8025838a000: Listen queue overflow: 8 already in queue awaiting acceptance (707 occurrences)
Dec 18 09:16:09	kernel: sonewconn: pcb 0xfffff8025838a620: Listen queue overflow: 8 already in queue awaiting acceptance (685 occurrences)
Dec 18 09:17:18	kernel: sonewconn: pcb 0xfffff8025838a000: Listen queue overflow: 8 already in queue awaiting acceptance (411 occurrences)
Dec 18 09:18:33	kernel: sonewconn: pcb 0xfffff8025838a000: Listen queue overflow: 8 already in queue awaiting acceptance (742 occurrences)

When doing a netstat, I find nothing matching either hex:


[2.2.5-RELEASE][root@fw01]/root: netstat -an | grep 0xfffff8025838a620
[2.2.5-RELEASE][root@fw01]/root: netstat -an | grep 0xfffff8025838a000

I increased kern.ipc.somaxconn to 4096, and this reflects in sysctl:


[2.2.5-RELEASE][root@fw01]/root: sysctl kern.ipc.somaxconn
kern.ipc.somaxconn: 4096

But has not relieved this or other connectivity issues. The other connectivity issues relate to API calls some of the servers behind the firewall do. 95% of the time, they return a valid result, but occassionally, they won't receive data back in from the API. Additionally, some external services that connect via SSH to servers behind the firewall report not being able to connect.

The server is running a few services, HAProxy, OpenVPN, but not Squid or anything like that where solutions have already been proposed.

Any ideas? I'm stumped.

Thanks.

mer

after bumping up the sysctl did you reboot? It's possible that whatever process is reporting the issue needs to be restarted.

A link to something that looks relevant, even though it's a couple years old.

https://forums.freebsd.org/threads/listen-queue-overflow.43712/

emerge30

Just an update here - this looked to be related to the TCP offload engine being 'enabled' after upgrading to PFsense 2.2.4 a few months ago. I didn't start noticing issues right away, but when I did they were connectivity limiting. For some reason only my master firewall had this enabled, the backup firewall did not get the TOE option enabled after update.

Disabling TOE fixed this issue.

We've since upgraded to 2.2.5 and the issue did not repeat.