kern.ipc.maxsockets limit reached

[OpIT GmbH]

Hi,

we are using a Netgate 7100.

Extra Services:
Snort (no blocking Mode)
PfBlockerNG
Captive Portal

Now sometimes the Router hangs, only console is working an the error what you see there is:
kern.ipc.maxsockets limit reached
Unable to start pfSense module in Unknown on line 0

[OpIT GmbH]

[stephenw10]

Are you seeing that immediately at boot?

Anything in the system logs?

[OpIT GmbH]

@stephenw10
no this happens not at boot. i happens randomly after time, sometimes multiple times per day. sometimes after some days

[stephenw10]

It just happens to be shown above the console menu there then?

Is there anything further shown in the system logs when this happens?

Check the System > Monitoring graphs for any resource exhaustion that might be happening.

[OpIT GmbH]

@stephenw10
Yeah u can just see the Error in the Console Menu, iam not able to connect with a Webbrowser, iam also not able to Ping the Netgate. Its seams all Network traffic is brocken, just Serial Connection is Working.

Could this be a Problem with the Hardware (RAM)? or with pfBlocker?

[stephenw10]

If it was a hardware problem I'd expect to see it fail entirely, even at the console.

How do you restore connectivity? Reboot from the console?

Check the firewall logs and monitoring graphs after regaining access.

Steve

[OpIT GmbH]

@stephenw10

just now the Netgate have the same Problem. I can connect with Serial Connection. But with i hit 5 to reboot the System i get this Error: "Unable to start pfSense module in Unknown on line 0"

The only Menu was working is 8 (Shell). So i hit 8 and then with "reboot" the Netgate reboot and its working again.

[OpIT GmbH]

@stephenw10

System Logs says just this..

Dec 30 02:38:00 NPC-Chalet kernel: [zone: udp_inpcb] kern.ipc.maxsockets limit reached

[stephenw10]

Try running: vmstat -z | egrep 'USED|inpcb'

See if that output changes over time like something is leaking or perhaps is very low initially.

Steve

[OpIT GmbH]

@stephenw10

after some test, i think its the Captive Portal Function. I just have enable it with Bandwidth restriction. The System has been working for about 3 Hours, now its hangs. I can connect with Serial and i can normally use the Reboot Function, but i cant ping to WAN or LAN....

We are using Captive Portal with multiple VLAN Interfaces (about 60 VLAN's)

I already Patched the Router with:
https://github.com/pfsense/pfsense/commit/b37f3f5d497493256f092619f94a266573dd6f04.patch
and
https://github.com/pfsense/pfsense/commit/c0f216b9b1b6455afc96cb37e6319a23bf28a98d.patch

[stephenw10]

Hmm, neither of those have been tested extensively against 22.05 though I'd expect them to work there.
When this happens if you disable pf at the cli with pfctl -d does that allow you to regain access?
If you then restart the captive portal does that clear to blocks for some time?

Steve

[OpIT GmbH]

@stephenw10

i need to test this.

But again, i think it has something to do with the multiple Interface (VLAN) selection in the Captive Portal. I have a other Netgate (1537) with just one VLAN selected in CP, and also installed both Patched > Here i don't see this Problem.

As the Netgate get monitored with PRTG, i can see the exakt time when it happens, so maybe some logs are helpful?

[stephenw10]

I could definitely believe that. Using a single zone for multiple interfaces is far more unusual and one of that patches addresses that situation specifically.
If you looks in Diag > Tables for the Cpzoneid table. Do you correctly see all the interface IPs listed?

[OpIT GmbH]

@stephenw10

yes...

[stephenw10]

In which case the kern.ipc.maxsockets limit reached error could just be a symptom of the captive portal blocking traffic. Let me know if disabling pf allows it to pass again.

[OpIT GmbH]

@stephenw10
when i enter pfctl -d in the shell, nothing happens. i need to hard reboot the Netgate...

[OpIT GmbH]

@stephenw10
no idea what else can i do? i thinks its definitively the Captive Portal function with multiple interfaces selected. At the Moment CP is disabled and the Netgate is running now for some days...

[stephenw10]

How much traffic do you have through that captive portal?

Do you think the traffic passing it might trigger this? I.e. does it seem to stay up longer with fewer clients connected for example?

Are you able to test a 23.01 snapshot? There are numerous CP fixes there.

Steve

[OpIT GmbH]

@stephenw10

i think it can have something todo with traffic. when i enable CP, its might run 15 Min but it also can be Days or Week before the Router crash.

Iam 100% sure the Problem is with multiple selected VLAN's in one CP Interface. Also i have bandwidth limitation set there