Poor SG-5100 Performance?

Grunt0307

After talking with NetGate sales a few months back, I was steered towards a SG-5100. The use case I presented at the time was running at 1GB line speed with full IPS enabled.

I've had the device up and running for about a month now and this thing struggles to saturate a 400Mb connection. If I enable pfBlockerNG and run a speed test, it locks up and I have to perform a hard reset to get internet back. I've upgraded the RAM to 16GB and added a m2 SSD in hopes of removing any potential bottlenecks but it still happens. Checking CPU stats using top during speed tests, I don't see high usage at all, hits about 60-70%. Thermals are also well within operating limits, I've never seen it over about 42c

Is this expected performance from this device? Where's a good place to start to identify the bottleneck or buggy configuration/application?

I'm running the latest release of pfSense, 2.4.4-RELEASE-p3, pfBlockerNG-devel 2.2.5_23, and Suricata 4.1.4.

One other thing of note, if I go to System>Netgate Coreboot Upgrade, it throws up an error saying This function is only available for Netgate Inc. hardware

bmeeks

How are you defining "locks up"? By that I mean can you still initiate say an SSH connection to the firewall? Enable SSH connectivity (if you have not already) and make sure you can connect using something like PuTTY or another SSH client. Then enable pfBlockerNG and try the speed test again. Does the firewall actually lock up to the extent that not even the SSH connection still works? pfBlockerNG simply puts a lot of firewall rules in place. Can't really see how that would lock up the box outside of exhausting memory, but with 16 GB that is not likely.

Suricata can be a resource hog if you use lots of enabled rules. Many folks tend to overdo it on the number of rules they enable by turning on far more rules than their network might actually need in order to be secure. Tuning is key to Suricata performance. Tuning involves careful selection of enabled rules and also some settings such as memory caps may need to be adjusted. There are also some suggested system tuneables for the NIC drivers that are detailed here: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces.

Rico

The SG-5100 is not using Coreboot BIOS.
You can just remove this package.

-Rico

Grunt0307

@bmeeks, I have SSH enabled and when it locks up; internet connectivity, the WebGUI, and SSH all stop functioning. The interface LEDs are all still active as if it's working properly, but it's non-responsive otherwise. I've let it sit for about 30 minutes thinking it was overloaded or something was stuck but it doesn't come back until I reboot the system.

I have a VM infrastructure with some publicly available services, game server and web server, so I have a couple extra IPS rule sets configured than would be necessary for a typical home user. I did just notice the section Services>Suricata>Interfaces>LAN Categories and I have all the ET rules plus Snort GPLv2 community rules enabled, nto sure how I missed this. I've disabled about a dozen of the rulesets for services I don't need to be checking on...sql, smtp, scada, etc... Beyond that, below are the rules sets I have configured in SID management to drop.

emerging-attack_response
emerging-botcc
emerging-compromised
emerging-ciarmy
emerging-drop
emerging-dshield
emerging-exploit
emerging-malware
emerging-scan
emerging-tor
emerging-trojan
emerging-web_server

I've doubled just about every memory config metric I've found for Suricata/pfBlockerNG in the past so they have plenty of room to play, but I'll take a look at that link and make sure I haven't missed anything.

bmeeks

A hard lockup like you describe makes me suspect hardware someplace. I would get with the Netgate support team and work with them. Slow performance I could see, but a hard lockup that needs a power off/on type reset to recover from is strange.

Do you get any kind of messages in the system log just prior to the lockup? You could take a look immediately after rebooting. Might also set the system log max entries number pretty high to make sure the log does not rollover since pfSense uses clog, the circular logging engine.

Is it only pfBlockerNG that causes the lockup? And only when running a speed test? Could be something related to NIC driver interrupts when under heavy load maybe ???

gfeiner

@Grunt0307 said in Poor SG-5100 Performance?:

Is this expected performance from this device?

No it is not. My SG-5100 can saturate my ATT 1Gb service (around 920-940 up/down). I'm also running 2.4.4-p3, pfblocker, and snort (on 2 LAN interfaces). I also added a m.2 SSD but I kept the RAM at the 4GB it comes with. My memory usage is only around 20-30%. Are you sure you need that 16GB? Maybe your memory is bad (because of your lockup problem). Try removing the extra memory you added.

stephenw10

Yes I would expect far better performance and certainly it should not 'lock up'.

Do you see a crash report when it reboots?

Try running the test with the serial console connected, does that lockup? Try entering ctl+t at the console, that can sometimes respond when nothing else will.

Steve

Derelict

@Grunt0307

@gfeiner said in Poor SG-5100 Performance?:

Are you sure you need that 16GB? Maybe your memory is bad (because of your lockup problem). Try removing the extra memory you added.

That is the first thing I would do. You have two problems it sounds like. The lockups and the throughput. This might solve the lockups issue then work on the other.

Grunt0307

@bmeeks said in Poor SG-5100 Performance?:

How are you defining "locks up"? By that I mean can you still initiate say an SSH connection to the firewall? Enable SSH connectivity (if you have not already) and make sure you can connect using something like PuTTY or another SSH client. Then enable pfBlockerNG and try the speed test again. Does the firewall actually lock up to the extent that not even the SSH connection still works? pfBlockerNG simply puts a lot of firewall rules in place. Can't really see how that would lock up the box outside of exhausting memory, but with 16 GB that is not likely.

Suricata can be a resource hog if you use lots of enabled rules. Many folks tend to overdo it on the number of rules they enable by turning on far more rules than their network might actually need in order to be secure. Tuning is key to Suricata performance. Tuning involves careful selection of enabled rules and also some settings such as memory caps may need to be adjusted. There are also some suggested system tuneables for the NIC drivers that are detailed here: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces.

Sorry for the really late response gents. So I had been through that link before, but appears I missed the netmap buffer setting. Initially I set it to 2048 and was able to make it through half a speed test before it would lock up. Bumped it to 3072 and I could get through about 1 1/2 speed tests before it would start throttling the connection but it would stay up. Bumped a further GB to 4096 and it stays up through three consecutive speed tests. I felt like it was pretty solid at that point and didn't test it any further. More importantly, I don't see any degraded connectivity during real world usage.

I need to take some time to run a memtest and see if that's the culprit next.

Grunt0307

To answer questions that I neglected...

@stephenw10 said in Poor SG-5100 Performance?:

Yes I would expect far better performance and certainly it should not 'lock up'.

Do you see a crash report when it reboots?

Try running the test with the serial console connected, does that lockup? Try entering ctl+t at the console, that can sometimes respond when nothing else will.

Steve

No, I don't see any crash reports when it comes back online in the GUI, is there some place I can look to see them?

@gfeiner said in Poor SG-5100 Performance?:

@Grunt0307 said in Poor SG-5100 Performance?:

Is this expected performance from this device?

Are you sure you need that 16GB? Maybe your memory is bad (because of your lockup problem). Try removing the extra memory you added.

With everything configured the way I currently have it, RAM usage seems to hover around 26-32% at "idle".

Derelict

I would still remove the aftermarket memory you added and see if the problems cease.

pfSense (FreeBSD) will often allocate more memory than is actually necessary if it is available.

stephenw10

@Grunt0307 said in Poor SG-5100 Performance?:

No, I don't see any crash reports when it comes back online in the GUI, is there some place I can look to see them?

If crash reports are present you will see an alert on the dashboard reporting that. They are stored in /var/crash though.

Steve