Pfsense 2.2 - Overwhelmed by large package load

itsme01 · Dec 8, 2014, 1:58 PM

I am using pfsense as a firewall for a network of systems which I use to run masscan against my company's internet facing infrastructure (really!). I run masscan with a rate of 1500 packages/second. With pfsense 2.1 I had no problems at all and everything worked fine. Since I updated to 2.2 (even with today's release Dec-8,14) everything freezes up after about 5 seconds, the GUI is non responsive and all other clients loose their connection to the internet when I run the masscan. Also the results I get are not consistent. As soon as I stop the scan everything goes back to normal.
I don't see anything in the logs…

Anybody else seeing such behaviour?

eri-- · Dec 8, 2014, 5:24 PM

Probably you are reaching state table limits.
Can you see the load on the system and increase the state table limit?

itsme01 · Dec 9, 2014, 12:57 PM

Hi ermal

The state table size is set to 1'000'000. In 2.1 we reached about 40-45% doing the same scans. I cannot check the state table while the scans are running as the system is not reachable. But as soon as I stop the scan everything works again and the state table is not full (which it should be if it is a state table problem as the state table would not empty immediately when I stop the scan).

eri-- · Dec 9, 2014, 6:22 PM

Probably you need to tune the interfaces.
Either add interrupt moderation or other recommandations for FreeBSD.
You did not notice this in previous versions because you could not even forward that much traffic concurrently.

itsme01 · Dec 10, 2014, 1:57 PM

Thanks ermal. Interrupt moderation is enabled by default. I played around with the settings and nothing changes. I also tried a lot of other interface tuning parameters, but nothing really changes the problem.

But back to the history of this problem: I have pfsense 2.1 and can produce reproducable results with masscan running at 1500 packets per second. During the scan, I can access the web GUI and make an SSH connection to my pfsense. All works fine. On the same hardware I update to pfsense 2.2. If I leave the masscan settings as they are, then my pfsense becomes unresponsive (WebGUI and SSH). I have played with the masscan settings and can only have a stable system with 150 packets per second.

There has to be a strange setting in 2.2 which is making the system react so differently…

cmb · Dec 11, 2014, 1:38 AM

What hardware are you running?

It's not a general problem, I run nmap scans racking up way more than 1500 connections/sec routinely for testing purposes. Just tried massscan and things do degrade a little if you really hammer a system (of course), but web interface still works, SSH still fine.

itsme01 · Dec 11, 2014, 7:34 AM

Hi cmb

I am running pfsense on an APU board (http://www.pcengines.ch/apu.htm) with 4GB RAM and an mSATA SSD. Could it be a bug in the Realtek Interface driver?

cmb · Dec 11, 2014, 11:39 PM

It might be, I'll try to replicate on an APU. I was testing with a more powerful system than that and one with much better NICs.

itsme01 · Dec 17, 2014, 7:30 AM

Hi cmb

Any update on your tests on an APU board?

cmb · Dec 17, 2014, 6:28 PM

Yes I was able to replicate the issue, there and elsewhere afterwards. It should have been fixed yesterday, if you can try today's snapshot or newer I don't think you'll see that anymore.

itsme01 · Dec 19, 2014, 7:47 AM

SOLVED!!! Thanks a lot. Works perfectly again with 1500pps. Just out of curiosity: what was the problem resp. what did you fix?

cmb · Dec 19, 2014, 6:36 PM

Thanks for the confirmation. Some work (funded by Netgate) was done on the hash alg in part of pf which got merged into FreeBSD (newer than 10.1), and the patch set we included was wrong, only hashing a quarter of the bytes. It's a nice little performance improvement (when it's included correctly). In circumstances like the one you described where you didn't have significantly more CPU than necessary for the job at hand, that slowed things down dramatically.