LAN loses WAN egress; no other problems

bldr

I can repeatably cause the client on my LAN to lose WAN access, without causing other disruptions by running a webserver load test.

I run a pretty basic pfSense installation:
Single WAN (Cable Modem)
Single LAN (DHCP w/some static leases)
OpenVPN Server
OpenVPN Client (to remote server)
Handful of Port-Forwards (80/443/22)

One of my port-forwards is to a webserver (80/443). When I run a load test (Loader.io ~1500 requests in 60 seconds) against a page from that server my setup serves the requests, and there are zero failed requests.
After executing such a test (or two) my internal LAN clients (phones/laptops/Roku) cannot access the internet, nor can my outgoing OpenVPN client remain connected.

From outside my network I can still access my web server (through the port-forward).
I can connect LAN-to-LAN.
I can connect from WAN to the OpenVPN server and access LAN clients.
I can connect to the admin interface from LAN or WAN.

Waiting a couple of hours does not magically correct the problem.
After reboot of my pfSense box everything is back to normal.

This behavior does not seem right and it has occurred over at least one (minor) version update. Not that I have a need to handle such regular web requests but I don't understand why I would experience this.

I have no idea where to begin to chase down why this happens.

Any direction on this would be greatly appreciated.
Happy to elaborate on config/hardware/etc. if that is helpful.

edit: Consistent persistent spelling problems.

stephenw10

Hmm, that's curious behaviour.

1500 connections is not that many it would be unlikely to exhaust your state table for example. Unless you happened to already have a large number of states. Even if it did existing states, like the the OpenVPN client, would remain.

Do the system logs show anything?

Do you see anything spike in The system Monitoring Graphs?

Are you running any packages? Something like Snort/Suricata could be triggering.

Steve

bldr

I do not have snort, or really any other packages installed/running.

Exhausting the state table is interesting as I generally have pretty darn good uptime on my pfSense box and I assume states just collect and build up overtime. Though looking at it just now my main status page shows:
State table size
0% (408/403000)

Which seems like an awful lot of headroom for basically two people.
As far as in-coming traffic I guess I would have to check apache, but there is no real public site (yes its public in that there is a way to get to it, but no real "published" service to use).

The machine is an:
Intel(R) Atom(TM) CPU N2800 @ 1.86GHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: No - so sad for my future :(

4gb Ram and another 4gb of swap configured

2.4.4-RELEASE (amd64)
built on Thu Sep 20 09:03:12 EDT 2018
FreeBSD 11.2-RELEASE-p3

Seems like it be best to:

reboot the pfSense box
confirm things are running normally
run the external load test, recreate the issue
jump into the GUI and take a look at the State table and the system logs

Is there a particular set of the System Logs to look at, or would it best to look in all of the options to see what got logged in the window of time where we go from working ok to less than ok?

Is there any information I should look at or grab from the gui before my test?

Thanks.

Grimson

@bldr said in LAN loses WAN egress; no other problems:

2.4.4-RELEASE (amd64)

That build has had problems with the new gateway settings, sometimes loosing the default gateway. Update to 2.4.4p2 and make sure the right gateway is set as the default.

In the future you should make sure you keep your firewall updated and read the release notes.

stephenw10

@bldr said in LAN loses WAN egress; no other problems:

AES-NI CPU Crypto: No - so sad for my future :(

But not for a while: https://forum.netgate.com/post/823904

Yes, update to 2.4.4p2 and confirm it still happens there before going further.

Steve