2.3.b.20160324.0516 + Suricata IPS eventually kills internet traffic issue…

Tantamount

I've tried both legacy and inline mode, but the symptom os the same.

It starts out filtering traffic fine for a few minutes, and then this happens:

Mar 28 22:12:15 pfSense kernel: igb0: link state changed to DOWN
Mar 28 22:12:15 pfSense check_reload_status: Linkup starting igb0
Mar 28 22:12:16 pfSense php-fpm[41930]: /rc.linkup: DEVD Ethernet detached event for wan
Mar 28 22:12:17 pfSense kernel: arpresolve: can't allocate llinfo for 108.214.100.1 on igb0
Mar 28 22:12:17 pfSense kernel: arpresolve: can't allocate llinfo for 108.214.100.1 on igb0
Mar 28 22:12:17 pfSense kernel: arpresolve: can't allocate llinfo for 108.214.100.1 on igb0
Mar 28 22:12:17 pfSense kernel: arpresolve: can't allocate llinfo for 108.214.100.1 on igb0

eventually a bunch of network services restart and igb0 shows as up, but no traffic flows through the internet.

If I use "legacy mode", after igb0 goes down and the networking services restart, I start to see this if I attempt to access the internet:

Mar 28 22:01:30 pfSense kernel: 690.488886 [1162] netmap_grab_packets bad pkt at 452 len 2425
Mar 28 22:01:30 pfSense kernel: 690.549520 [1162] netmap_grab_packets bad pkt at 463 len 2425
Mar 28 22:01:30 pfSense kernel: 690.626332 [1162] netmap_grab_packets bad pkt at 477 len 2437
Mar 28 22:01:31 pfSense kernel: 691.245050 [1162] netmap_grab_packets bad pkt at 499 len 2425
Mar 28 22:01:31 pfSense kernel: 691.251754 [1162] netmap_grab_packets bad pkt at 503 len 2239
Mar 28 22:01:34 pfSense kernel: 694.128259 [1162] netmap_grab_packets bad pkt at 567 len 2417

The nic is this:
igb0@pci0:3:0:0: class=0x020000 card=0xe0001458 chip=0x15398086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I211 Gigabit Network Connection'
class = network
subclass = ethernet

Any ideas? Suricata is mostly at defaults – I adjusted some memory settings due to the 8 cores of the processor:

Flow Memory Cap: 67108864
Stream Memory Cap: 134217728

Tantamount

Okay, discovered that igb0 goes down the moment this happens in Suricata:
28/3/2016 – 22:43:35 - <info>-- building signature grouping structure, stage 3: building destination address lists... complete

So it's not the driver crashing -- it's just Suricata finishing loading and then restarting the network stuff.

In troubleshooting, I updated pfsense to the latest version, disabled the hardware acceleration options for the nic, and rebooted.

Everything seems stable now, although I do very occasionally see these still:
Mar 28 22:44:52 pfSense kernel: 292.482947 [1162] netmap_grab_packets bad pkt at 59 len 2159</info>

Raul Ramos

Suricata have some big changes. Read, at least, the first topic on https://forum.pfsense.org/index.php?topic=107847.0.

gsiemon

You may want to review this post from Bill as he says you need to disable all hardware acceleration on nics running with Suricata IPS.

https://forum.pfsense.org/index.php?topic=108068.msg601891#msg601891

bmeeks

This –

@gsiemon:

You may want to review this post from Bill as he says you need to disable all hardware acceleration on nics running with Suricata IPS.

https://forum.pfsense.org/index.php?topic=108068.msg601891#msg601891

Netmap (used for inline IPS mode) requires both TCP segmentation offloading and hardware checksum calcs be disabled as illustrated in the linked post.

Bill

Tantamount

Thanks for the confirmation.

Were the warnings that already exist for hardware acceleration supposed to give fair warning that dragons be here?

I'm just trying to figure out if I should feature request some type of settings conflict check for the user or not. Does pfsense do any type of sanity check elsewhere for the user?

bmeeks

@Tantamount:

Thanks for the confirmation.

Were the warnings that already exist for hardware acceleration supposed to give fair warning that dragons be here?

I'm just trying to figure out if I should feature request some type of settings conflict check for the user or not. Does pfsense do any type of sanity check elsewhere for the user?

No, the package does not currently check those settings. But it is a valid suggestion that it should. I will add that in a coming update. Working on a few other things, too, so give me a little while to get everything done. A small update to the package was posted today, but that only fixes a Barnyard2 run dependency that was missing and causing the Barnyard2 binary to not get installed along with Suricata.

Bill

Tantamount

I added Suricata to my LAN interface, and am running into the same problem as above:

It works for about half a minute, and then all communication on my LAN interface goes dead.

/var/log/system.log is mostly silent when this happens:

Apr 1 16:03:17 pfSense kernel: em0: link state changed to DOWN
Apr 1 16:03:17 pfSense check_reload_status: Linkup starting em0

The hardware acceleration settings under System->Advanced->Networking are supposed to be global, correct? All the hardware choices are disabled.

The nic hardware is fairly recently supported I think? Perhaps there are still some bugs to work out?

em0@pci0:0:31:6: class=0x020000 card=0xe0001458 chip=0x15b88086 rev=0x31 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Connection (2) I219-V'
class = network
subclass = ethernet

Oddly I had enabled Suricata on the LAN a number of days back, but forgot to enable the ET Pro classifications. Once I did that, about a few hours later I started running into this trouble. Hopefully I just need to allocate more memory somewhere?

Edit:

Am running the latest version of 2.3 beta:

2.3-RC (amd64)
built on Thu Mar 31 23:48:37 CDT 2016

Tantamount

Apparently Suricata 3.0.1 was released today. There's mention of lots of stability and memory fixes… perhaps this will help my situation?

http://suricata-ids.org/news/

bmeeks

@Tantamount:

I added Suricata to my LAN interface, and am running into the same problem as above:

It works for about half a minute, and then all communication on my LAN interface goes dead.

/var/log/system.log is mostly silent when this happens:

Apr 1 16:03:17 pfSense kernel: em0: link state changed to DOWN
Apr 1 16:03:17 pfSense check_reload_status: Linkup starting em0

The hardware acceleration settings under System->Advanced->Networking are supposed to be global, correct? All the hardware choices are disabled.

The nic hardware is fairly recently supported I think? Perhaps there are still some bugs to work out?

em0@pci0:0:31:6: class=0x020000 card=0xe0001458 chip=0x15b88086 rev=0x31 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Connection (2) I219-V'
class = network
subclass = ethernet

Oddly I had enabled Suricata on the LAN a number of days back, but forgot to enable the ET Pro classifications. Once I did that, about a few hours later I started running into this trouble. Hopefully I just need to allocate more memory somewhere?

Edit:

Am running the latest version of 2.3 beta:

2.3-RC (amd64)
built on Thu Mar 31 23:48:37 CDT 2016

Were you running Suricata on just the LAN interface or both LAN and WAN at the same time?

Bill

Tantamount

Both.

bmeeks

@Tantamount:

Both.

Thanks for the info. Just looking for some baseline data to start with troubleshooting/reproducing the problem. I admit to not having run Suricata for an extended period with inline IPS mode enabled.

Bill

Tantamount

Well, I'm not certain if it was due to prolonged use, or the more recent enabling of the et pro rules.

Also, bad things happen even when I disable the "block" setting (Which I assume disables the inline stuff?).

I've just unchecked all of the LAN categories and attempted to re-enable, but the same problem happened.

If I use the console and kill the suricata process attached to the lan nic, the lan traffic immediately starts to work again.