Intel NIC I-226V
-
You have SpeedShift enabled?
Be good to see more of that top output so we know what's generating that CPU load.
-
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 187 ki31 0B 64K CPU2 2 122:26 58.07% [idle{idle: cpu2}] 11 root 187 ki31 0B 64K CPU1 1 122:25 57.91% [idle{idle: cpu1}] 11 root 187 ki31 0B 64K RUN 3 122:21 57.13% [idle{idle: cpu3}] 11 root 187 ki31 0B 64K CPU0 0 124:38 56.37% [idle{idle: cpu0}] 69815 root 24 0 855M 469M select 3 0:57 17.04% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 69815 root 24 0 855M 469M select 3 0:58 16.84% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 69815 root 23 0 855M 469M select 3 0:50 15.64% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 69815 root 23 0 855M 469M select 2 1:05 14.86% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 69815 root 24 0 855M 469M select 2 1:05 14.11% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 12 root -60 - 0B 320K WAIT 3 0:22 10.66% [intr{swi1: netisr 1}] 69815 root 23 0 855M 469M select 0 1:24 9.82% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 69815 root 21 0 855M 469M select 0 1:32 7.52% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 56778 root 21 0 550M 390M bpf 1 0:43 7.41% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i igc1 -i igc0 --dns-mode 0 --local-ne 56778 root 21 0 550M 390M bpf 0 0:52 7.24% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i igc1 -i igc0 --dns-mode 0 --local-ne 0 root -60 - 0B 1808K - 0 0:39 6.15% [kernel{if_io_tqg_0}] 12 root -60 - 0B 320K WAIT 0 0:28 6.06% [intr{swi1: netisr 0}] 0 root -60 - 0B 1808K RUN 1 0:34 6.03% [kernel{if_io_tqg_1}] 0 root -60 - 0B 1808K - 3 0:39 5.93% [kernel{if_io_tqg_3}] 12 root -60 - 0B 320K WAIT 2 0:42 5.76% [intr{swi1: netisr 2}] 0 root -64 - 0B 1808K - 0 0:22 5.58% [kernel{dummynet}] 12 root -60 - 0B 320K WAIT 1 0:33 5.44% [intr{swi1: netisr 3}] 0 root -60 - 0B 1808K - 2 0:49 4.18% [kernel{if_io_tqg_2}] 69815 root 20 0 855M 469M select 0 1:23 2.37% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 69815 root 20 0 855M 469M uwait 3 0:45 0.24% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 56778 root 20 0 550M 390M nanslp 2 0:30 0.21% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i igc1 -i igc0 --dns-mode 0 --local-ne 6263 root 20 0 1411M 136M uwait 1 0:07 0.17% /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml{crowdsec} 6263 root 20 0 1411M 136M kqread 2 0:07 0.12% /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml{crowdsec} 56778 root 20 0 550M 390M uwait 3 0:00 0.12% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i igc1 -i igc0 --dns-mode 0 --local-ne 30798 unbound 20 0 355M 305M kqread 0 0:13 0.08% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 2 root -60 - 0B 64K WAIT 2 0:02 0.08% [clock{clock (2)}] 8 root -16 - 0B 16K - 3 0:02 0.08% [rand_harvestq] 2 root -60 - 0B 64K WAIT 3 0:01 0.07% [clock{clock (3)}] 56712 root 20 0 36M 12M kqread 1 0:05 0.07% redis-server: /usr/local/bin/redis-server 127.0.0.1:6379 (redis-server){redis-server} 2 root -60 - 0B 64K WAIT 0 0:02 0.07% [clock{clock (0)}] 37116 root 20 0 14M 4468K CPU1 1 0:01 0.07% top -HaSP 2 root -60 - 0B 64K WAIT 1 0:01 0.06% [clock{clock (1)}] 69815 root 20 0 855M 469M nanslp 1 0:49 0.06% /usr/local/bin/suricata --netmap -D -c /usr/local/etc/suricata/suricata_12484_igc1/suricata.yaml --pidfile /var/run/suric 56778 root 20 0 550M 390M uwait 0 0:00 0.05% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i igc1 -i igc0 --dns-mode 0 --local-ne -
@stephenw10 Yes Speed Shift enabled
-
Ok so all Suricata load basically. If you disable it as a test without any other changes do you get full bandwidth?
-
@stephenw10 Yes, exactly, if suricata disabled speed test show as should be about 95% of max capacity as set in traffic shaping limiters!
-
Hmm. Do you see the same in legacy and in-line mode?
-
@stephenw10 Use in-line mode, don't use legacy mode. Do you want to test in legacy mode?
-
Yes, test legacy mode. See if there is any change.
-
@stephenw10 In legacy mode looks like speed as should be
-
Hmm, so possibly some netmap issue. If you run
sysctl dev.netmap.admode=1it should prevent netmap running in emulated mode which is very slow. However that does mean it can only run in native mode so may break if that's not possible for some reason. -
@stephenw10 I will try to set in native mode, but also have net.bpf.zerocopy_enable=1 is it not a problem?
-
I wouldn't expect that to make any difference there if netmap is working. Why do you have that set?
-
@stephenw10 During suricata installation at the end was this tip and I did)))
-
Oh, in the install log? Ok well I would definitely try disabling that then. Almost all users would not set that.
-
@Antibiotic Unfortunately, sysctl dev.netmap.admode=1 doesn't help at all. So the option only to use in legacy mode?
-
But it didn't break traffic?
That implies igc is running netmap in native mode but for some reason the performance isn't as expected. That's not something I've ever poked at in any depth. There may be some tuning option there though.
-
@stephenw10 I'm left netmap in native mode and remove from sysctl net.bpf.zerocopy_enable=1. The speed the same as not to be normal. Its not a break traffic. Can you please give me some links for tuning netmap in native mode? BTW after reboot all this services ( suricata, crowdsec, ntopng and pfblockerng need to start manually something wrong)
-
Well nothing specific since, as I say, I've not really poked at this. But I'd start here: https://man.freebsd.org/cgi/man.cgi?query=netmap
Bill can probably tell us if this is worth investigating.
-
@stephenw10 FreeBSD man page too hard for me)))) I'm not IT pro)) Could some read our conversation and give some tips! I will wait
-
How many Suricata instances are you running and how many enabled rules on each instance?
Remove this setting:
sysctl net.bpf.zerocopy_enable=1Ignore those notes at the end of the package install. They are coming from the default notes packaged from upstream and have no usefulness in pfSense.
Running Suricata, ntopng, and Crowdsec is asking a lot from this firewall.
Post back the output of this command:
ps -ax | grep suricataI'm guessing you have multiple Suricata instances running on the same interface and thus are severely wasting CPU cycles doing duplicate work.
