PC Engines apu2 experiences
-
@fireodo Maybe report it? Just checked and can confirm 2.4.5 and bios 4.11.0.4 gives
[2.4.5-RELEASE][root@pfSense.localdomain]/root: sysctl dev.cpu.0.freq dev.cpu.0.freq: 1000 -
@Qinn said in PC Engines apu2 experiences:
@fireodo Maybe report it? Just checked and can confirm 2.4.5 and bios 4.11.0.4 gives
[2.4.5-RELEASE][root@pfSense.localdomain]/root: sysctl dev.cpu.0.freq dev.cpu.0.freq: 1000Done: https://github.com/pcengines/coreboot/issues/389
-
@fireodo said in PC Engines apu2 experiences:
@Qinn said in PC Engines apu2 experiences:
@fireodo Maybe report it? Just checked and can confirm 2.4.5 and bios 4.11.0.4 gives
[2.4.5-RELEASE][root@pfSense.localdomain]/root: sysctl dev.cpu.0.freq dev.cpu.0.freq: 1000Sorry I'm in Home Office - I have not so much time ...
Before there can be a report, we need to be sure if this i bug is for all apu2 or specific to certain versions, as I see you use a C0 and I a C4.
At the moment I cannot upgrade to 4.11.0.5 as I cannot "Halt" this router, maybe later in the day...
-
@fireodo said in PC Engines apu2 experiences:
@Qinn said in PC Engines apu2 experiences:
@fireodo Maybe report it? Just checked and can confirm 2.4.5 and bios 4.11.0.4 gives
[2.4.5-RELEASE][root@pfSense.localdomain]/root: sysctl dev.cpu.0.freq dev.cpu.0.freq: 1000Done: https://github.com/pcengines/coreboot/issues/389
Kudos!
-
@Qinn It seems to be a good strategy to NOT do software/BIOS/firmware updates immediately after release. Lets first wait for others to burn themselves, then when the fire is put out, do the update.
-
Did anyone experience similar issues with their APUs?
https://forum.netgate.com/topic/151963/throughput-issues-on-pfsense-2-4-5-with-apu2e4I myself am using 4.11.0.4.
-
@soder Yes always sound to deploy to test systems first and/or wait for others.
In this case this is not a serious bug unless you are modifying the dev.cpu.* sysctl to reduce clock speed from default 1.0/1.4GHz default. Benchmarking shows that CPU performance is unchanged between releases.
The likely 3mdeb APU coreboot change which caused this is https://github.com/pcengines/coreboot/commit/9550e97304ecc1c1b6271d50ea089c82b9a82946
Anyway @fireodo has raised issue with 3mdeb ... have to wait for feedback.
-
From https://github.com/pcengines/coreboot/issues/389
"it looks like I rushed things... FreeBSD 12.x detects CPU on ACPI correctly with v4.11.0.5. If pfSense 2.5.0 based on FreeBSD 12.0 is out it will work I guess.
Sorry about that. Will revert it on next release."
-
@bigsy Thanks for informing us
-
That 4.11.05 bios! Back on 4.11.04 now.
Another issue I have on .05 is that my throughput decreased drastically.
I have a 500/500 line and could easily saturate it on .04:
iperf3 -c x.x.x. -P 4
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 158 MBytes 132 Mbits/sec 130 sender [ 5] 0.00-10.00 sec 157 MBytes 131 Mbits/sec receiver [ 7] 0.00-10.00 sec 118 MBytes 98.7 Mbits/sec 152 sender [ 7] 0.00-10.00 sec 116 MBytes 97.5 Mbits/sec receiver [ 9] 0.00-10.00 sec 139 MBytes 117 Mbits/sec 102 sender [ 9] 0.00-10.00 sec 138 MBytes 116 Mbits/sec receiver [ 11] 0.00-10.00 sec 190 MBytes 159 Mbits/sec 158 sender [ 11] 0.00-10.00 sec 188 MBytes 158 Mbits/sec receiver [SUM] 0.00-10.00 sec 604 MBytes 507 Mbits/sec 542 sender [SUM] 0.00-10.00 sec 599 MBytes 503 Mbits/sec receiverOn .05:
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 49.8 MBytes 41.8 Mbits/sec 230 sender [ 5] 0.00-10.00 sec 49.3 MBytes 41.4 Mbits/sec receiver [ 7] 0.00-10.00 sec 50.4 MBytes 42.3 Mbits/sec 240 sender [ 7] 0.00-10.00 sec 49.8 MBytes 41.8 Mbits/sec receiver [ 9] 0.00-10.00 sec 42.3 MBytes 35.5 Mbits/sec 222 sender [ 9] 0.00-10.00 sec 42.0 MBytes 35.3 Mbits/sec receiver [ 11] 0.00-10.00 sec 40.0 MBytes 33.6 Mbits/sec 259 sender [ 11] 0.00-10.00 sec 39.7 MBytes 33.3 Mbits/sec receiver [SUM] 0.00-10.00 sec 183 MBytes 153 Mbits/sec 951 sender [SUM] 0.00-10.00 sec 181 MBytes 152 Mbits/sec receiverAPU2C4 pfSense 2.4.5 with this configuration: https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/
-
@stefanl Thanks for informing, maybe also report it at https://github.com/pcengines/coreboot/issues/
-
@fireodo said in PC Engines apu2 experiences:
sysctl dev.cpu.0.freq
I have also noticed that the telegraf CPU temp script reports CPU temp much lower than expected.
https://www.reddit.com/r/PFSENSE/comments/b496kt/get_cpu_temp_using_telegraf/
This line now only reports total CPU temperate, not the temperature of each core.
VAR1=$(sysctl -a | awk '/temperature/ {print $2;}' | sed s/C// | tr '\n' '+' | sed 's/\(.*\)+/\1/') -
@stefanl I've struggling with low througput
Made some tweaks from https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/ and no matter the settings no performance change was gained.
Tested on APU3C4 on pfsense and opnsense.BIOS factory installed was 4.11.0.2. Upgraded to 4.11.0.5
Accepted connection from 192.168.10.188, port 31427 [ 5] local 192.168.10.21 port 5201 connected to 192.168.10.188 port 21880 [ 7] local 192.168.10.21 port 5201 connected to 192.168.10.188 port 60804 [ 9] local 192.168.10.21 port 5201 connected to 192.168.10.188 port 18408 [ 11] local 192.168.10.21 port 5201 connected to 192.168.10.188 port 37996 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 10.9 MBytes 91.1 Mbits/sec [ 7] 0.00-1.00 sec 9.88 MBytes 82.8 Mbits/sec [ 9] 0.00-1.00 sec 12.0 MBytes 101 Mbits/sec [ 11] 0.00-1.00 sec 12.9 MBytes 108 Mbits/sec [SUM] 0.00-1.00 sec 45.6 MBytes 383 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 10.8 MBytes 90.8 Mbits/sec [ 7] 1.00-2.00 sec 11.7 MBytes 98.3 Mbits/sec [ 9] 1.00-2.00 sec 11.9 MBytes 100 Mbits/sec [ 11] 1.00-2.00 sec 12.0 MBytes 101 Mbits/sec [SUM] 1.00-2.00 sec 46.5 MBytes 390 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 12.4 MBytes 104 Mbits/sec [ 7] 2.00-3.00 sec 11.9 MBytes 100 Mbits/sec [ 9] 2.00-3.00 sec 11.7 MBytes 98.4 Mbits/sec [ 11] 2.00-3.00 sec 10.1 MBytes 84.9 Mbits/sec [SUM] 2.00-3.00 sec 46.2 MBytes 387 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 11.5 MBytes 96.5 Mbits/sec [ 7] 3.00-4.00 sec 11.8 MBytes 99.3 Mbits/sec [ 9] 3.00-4.00 sec 10.7 MBytes 89.8 Mbits/sec [ 11] 3.00-4.00 sec 12.0 MBytes 100 Mbits/sec [SUM] 3.00-4.00 sec 46.0 MBytes 386 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 7.71 MBytes 64.7 Mbits/sec [ 7] 4.00-5.00 sec 13.2 MBytes 111 Mbits/sec [ 9] 4.00-5.00 sec 9.66 MBytes 81.0 Mbits/sec [ 11] 4.00-5.00 sec 14.8 MBytes 124 Mbits/sec [SUM] 4.00-5.00 sec 45.4 MBytes 380 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 11.7 MBytes 98.1 Mbits/sec [ 7] 5.00-6.00 sec 12.1 MBytes 101 Mbits/sec [ 9] 5.00-6.00 sec 11.4 MBytes 95.6 Mbits/sec [ 11] 5.00-6.00 sec 11.2 MBytes 94.0 Mbits/sec [SUM] 5.00-6.00 sec 46.3 MBytes 389 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 11.8 MBytes 99.2 Mbits/sec [ 7] 6.00-7.00 sec 11.3 MBytes 94.9 Mbits/sec [ 9] 6.00-7.00 sec 10.5 MBytes 87.9 Mbits/sec [ 11] 6.00-7.00 sec 12.7 MBytes 107 Mbits/sec [SUM] 6.00-7.00 sec 46.3 MBytes 389 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 11.2 MBytes 93.6 Mbits/sec [ 7] 7.00-8.00 sec 7.00 MBytes 58.7 Mbits/sec [ 9] 7.00-8.00 sec 13.2 MBytes 111 Mbits/sec [ 11] 7.00-8.00 sec 14.2 MBytes 119 Mbits/sec [SUM] 7.00-8.00 sec 45.5 MBytes 382 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 13.3 MBytes 112 Mbits/sec [ 7] 8.00-9.00 sec 7.52 MBytes 63.0 Mbits/sec [ 9] 8.00-9.00 sec 12.1 MBytes 102 Mbits/sec [ 11] 8.00-9.00 sec 13.1 MBytes 110 Mbits/sec [SUM] 8.00-9.00 sec 46.1 MBytes 387 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 10.6 MBytes 89.1 Mbits/sec [ 7] 9.00-10.00 sec 11.0 MBytes 92.2 Mbits/sec [ 9] 9.00-10.00 sec 12.9 MBytes 108 Mbits/sec [ 11] 9.00-10.00 sec 12.2 MBytes 102 Mbits/sec [SUM] 9.00-10.00 sec 46.7 MBytes 391 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 10.00-10.01 sec 95.5 KBytes 121 Mbits/sec [ 7] 10.00-10.01 sec 87.0 KBytes 110 Mbits/sec [ 9] 10.00-10.01 sec 89.8 KBytes 113 Mbits/sec [ 11] 10.00-10.01 sec 22.8 KBytes 28.8 Mbits/sec [SUM] 10.00-10.01 sec 295 KBytes 372 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.01 sec 112 MBytes 93.9 Mbits/sec receiver [ 7] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender [ 7] 0.00-10.01 sec 107 MBytes 90.1 Mbits/sec receiver [ 9] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender [ 9] 0.00-10.01 sec 116 MBytes 97.4 Mbits/sec receiver [ 11] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender [ 11] 0.00-10.01 sec 125 MBytes 105 Mbits/sec receiver [SUM] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender [SUM] 0.00-10.01 sec 461 MBytes 386 Mbits/sec receiverAfter flashing BIOS Legacy v4.0.30
Accepted connection from 192.168.10.188, port 62725 [ 5] local 192.168.10.21 port 5201 connected to 192.168.10.188 port 61859 [ 7] local 192.168.10.21 port 5201 connected to 192.168.10.188 port 12000 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 69.4 MBytes 582 Mbits/sec [ 7] 0.00-1.00 sec 39.8 MBytes 333 Mbits/sec [SUM] 0.00-1.00 sec 109 MBytes 915 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 67.2 MBytes 564 Mbits/sec [ 7] 1.00-2.00 sec 45.3 MBytes 380 Mbits/sec [SUM] 1.00-2.00 sec 113 MBytes 944 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 74.8 MBytes 628 Mbits/sec [ 7] 2.00-3.00 sec 35.4 MBytes 297 Mbits/sec [SUM] 2.00-3.00 sec 110 MBytes 925 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 74.8 MBytes 628 Mbits/sec [ 7] 3.00-4.00 sec 35.4 MBytes 297 Mbits/sec [SUM] 3.00-4.00 sec 110 MBytes 925 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 75.0 MBytes 629 Mbits/sec [ 7] 4.00-5.00 sec 35.0 MBytes 294 Mbits/sec [SUM] 4.00-5.00 sec 110 MBytes 923 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 74.7 MBytes 626 Mbits/sec [ 7] 5.00-6.00 sec 37.8 MBytes 317 Mbits/sec [SUM] 5.00-6.00 sec 112 MBytes 943 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 74.3 MBytes 623 Mbits/sec [ 7] 6.00-7.00 sec 38.4 MBytes 322 Mbits/sec [SUM] 6.00-7.00 sec 113 MBytes 945 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 68.9 MBytes 578 Mbits/sec [ 7] 7.00-8.00 sec 43.8 MBytes 368 Mbits/sec [SUM] 7.00-8.00 sec 113 MBytes 946 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 75.1 MBytes 630 Mbits/sec [ 7] 8.00-9.00 sec 37.6 MBytes 315 Mbits/sec [SUM] 8.00-9.00 sec 113 MBytes 945 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 76.8 MBytes 644 Mbits/sec [ 7] 9.00-10.00 sec 35.7 MBytes 300 Mbits/sec [SUM] 9.00-10.00 sec 113 MBytes 944 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 10.00-10.03 sec 1.89 MBytes 616 Mbits/sec [ 7] 10.00-10.03 sec 971 KBytes 309 Mbits/sec [SUM] 10.00-10.03 sec 2.84 MBytes 925 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.03 sec 733 MBytes 613 Mbits/sec receiver [ 7] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender [ 7] 0.00-10.03 sec 385 MBytes 322 Mbits/sec receiver [SUM] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender [SUM] 0.00-10.03 sec 1.09 GBytes 935 Mbits/sec receiverI'm making more tests and will post results here.
-
@cysiacom said in PC Engines apu2 experiences:
@stefanl I've struggling with low througput
Made some tweaks from https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/ and no matter the settings no performance change was gained.
Tested on APU3C4 on pfsense and opnsense.BIOS factory installed was 4.11.0.2. Upgraded to 4.11.0.5
Accepted connection from 192.168.10.188, port 31427 [SUM] 10.00-10.01 sec 295 KBytes 372 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - After flashing BIOS Legacy v4.0.30Accepted connection from 192.168.10.188, port 62725
[ 5] local 192.168.10.21 port 5201 connected to 192.168.10.188 port 61859
[ 7] local 192.168.10.21 port 5201 connected to 192.168.10.188 port 12000
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 69.4 MBytes 582 Mbits/sec
[ 7] 0.00-1.00 sec 39.8 MBytes 333 Mbits/sec
[SUM] 0.00-1.00 sec 109 MBytes 915 Mbits/sec
I'm making more tests and will post results here.The differences are huge! :-( I have read that with 4.11.0.4 the speed should be OK. Maybe you give it a try!
Regards,
fireodo -
@fireodo Glad to say you are completely right.
Same hardware.
Same config.
BIOS v4.11.0.5- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.00 sec 115 MBytes 96.1 Mbits/sec receiver [ 7] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 7] 0.00-10.00 sec 112 MBytes 94.2 Mbits/sec receiver [ 9] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 9] 0.00-10.00 sec 115 MBytes 96.8 Mbits/sec receiver [ 11] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 11] 0.00-10.00 sec 121 MBytes 101 Mbits/sec receiver [SUM] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [SUM] 0.00-10.00 sec 463 MBytes 388 Mbits/sec receiverBIOS v4.11.0.4
- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.02 sec 492 MBytes 412 Mbits/sec receiver [ 7] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [ 7] 0.00-10.02 sec 405 MBytes 339 Mbits/sec receiver [ 9] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [ 9] 0.00-10.02 sec 105 MBytes 87.8 Mbits/sec receiver [ 11] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [ 11] 0.00-10.02 sec 118 MBytes 99.2 Mbits/sec receiver [SUM] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [SUM] 0.00-10.02 sec 1.09 GBytes 939 Mbits/sec receiver|When setting up the first firewall I did an inmediate upgrade from factory BIOS v4.11.0.2 to BIOS v4.11.0.5.
The rollback to BIOS v4.11.0.4 as you suggested did the trick. -
This post is deleted! -
@cysiacom said in PC Engines apu2 experiences:
@fireodo Glad to say you are completely right.
Same hardware.
Same config.
BIOS v4.11.0.5- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.00 sec 115 MBytes 96.1 Mbits/sec receiver [ 7] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 7] 0.00-10.00 sec 112 MBytes 94.2 Mbits/sec receiver [ 9] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 9] 0.00-10.00 sec 115 MBytes 96.8 Mbits/sec receiver [ 11] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 11] 0.00-10.00 sec 121 MBytes 101 Mbits/sec receiver [SUM] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [SUM] 0.00-10.00 sec 463 MBytes 388 Mbits/sec receiverBIOS v4.11.0.4
- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.02 sec 492 MBytes 412 Mbits/sec receiver [ 7] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [ 7] 0.00-10.02 sec 405 MBytes 339 Mbits/sec receiver [ 9] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [ 9] 0.00-10.02 sec 105 MBytes 87.8 Mbits/sec receiver [ 11] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [ 11] 0.00-10.02 sec 118 MBytes 99.2 Mbits/sec receiver [SUM] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender [SUM] 0.00-10.02 sec 1.09 GBytes 939 Mbits/sec receiver|When setting up the first firewall I did an inmediate upgrade from factory BIOS v4.11.0.2 to BIOS v4.11.0.5.
The rollback to BIOS v4.11.0.4 as you suggested did the trick.This issue has been explained in https://github.com/pcengines/coreboot/issues/387
They enabled PCIe ASPM (Active State Power Management) which is responsible for the drop in performance.
There is a kernel param which can be set on Linux to disable ASPM but I couldn't find an equivalent for FreeBSD.PC Engines is going to make this configurable in a later firmware. So for now I would stick with 4.11.0.4.
-
@sToRmInG good that the cause is found
-
I'm wondering what kind of performance and CPU load you guys have with an APU2C4. I want to know if I should lower my expectations or tweak my settings.
I have a 500/500 fiber line.
The ~100mbit load is mostly from my Tor relay in a docker container. It is hard capped to 100 mbit. The cpu load is already quite high with this throughput. If I set it to 300mbit the network becomes unusable, dns timeouts, etc. Most cpu time goes to igb que interrupts.
Load with 100mbit throughput:
last pid: 79066; load averages: 1.38, 2.08, 2.50 up 12+20:03:08 08:58:38 230 processes: 6 running, 190 sleeping, 34 waiting CPU: 6.4% user, 0.0% nice, 4.5% system, 20.5% interrupt, 68.6% idle Mem: 401M Active, 209M Inact, 798M Wired, 296M Buf, 2498M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 64K CPU0 0 127.4H 73.22% [idle{idle: cpu0}] 11 root 155 ki31 0K 64K RUN 3 129.5H 71.20% [idle{idle: cpu3}] 11 root 155 ki31 0K 64K CPU1 1 133.2H 66.98% [idle{idle: cpu1}] 11 root 155 ki31 0K 64K RUN 2 139.1H 57.81% [idle{idle: cpu2}] 12 root -92 - 0K 544K WAIT 2 48.8H 19.57% [intr{irq258: igb0:que 2}] 12 root -92 - 0K 544K WAIT 2 58.3H 11.25% [intr{irq268: igb2:que 2}] 12 root -92 - 0K 544K WAIT 0 69.2H 10.17% [intr{irq266: igb2:que 0}] 12 root -92 - 0K 544K WAIT 1 52.1H 9.09% [intr{irq267: igb2:que 1}] 12 root -92 - 0K 544K WAIT 3 61.5H 8.20% [intr{irq269: igb2:que 3}] 12 root -92 - 0K 544K WAIT 0 69.2H 8.14% [intr{irq256: igb0:que 0}] 58124 root 20 0 97404K 39156K accept 0 0:56 6.76% php-fpm: pool nginx (php-fpm) 12 root -92 - 0K 544K WAIT 1 67.4H 6.37% [intr{irq257: igb0:que 1}] 45047 root 41 0 97468K 39804K piperd 0 1:25 6.28% php-fpm: pool nginx (php-fpm) 12 root -92 - 0K 544K WAIT 3 63.1H 5.00% [intr{irq259: igb0:que 3}] 15231 stfn 20 0 9868K 4624K CPU2 2 0:00 1.56% top -aSH 20 root -16 - 0K 16K pftm 3 149:55 0.69% [pf purge] 59099 root 20 0 23680K 9460K kqread 0 1:16 0.47% nginx: worker process (nginx) 12 root -60 - 0K 544K WAIT 0 24:23 0.16% [intr{swi4: clock (0)}] 50468 root 20 0 6404K 2464K select 1 13:01 0.12% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf -b 10.0.0.2 72979 root 20 0 13912K 11596K select 3 0:01 0.12% /usr/local/sbin/clog_pfb -f /var/log/filter.log 86683 root 20 0 10304K 6168K select 0 0:03 0.09% /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf 39603 dhcpd 20 0 16460K 11008K select 1 0:01 0.06% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid igb2 25 root 20 - 0K 32K sdflus 1 3:06 0.05% [bufdaemon{/ worker}] 1722 root 20 0 6292K 1988K select 0 7:22 0.04% /usr/sbin/powerd -b hadp -a hadp -n hadpLoad with 300mbit throughput:
last pid: 82661; load averages: 7.41, 5.28, 4.66 up 14+05:05:29 18:00:59 229 processes: 12 running, 188 sleeping, 29 waiting CPU: 8.1% user, 0.0% nice, 17.0% system, 59.9% interrupt, 14.9% idle Mem: 272M Active, 353M Inact, 796M Wired, 292M Buf, 2486M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root -92 - 0K 544K CPU2 2 53.9H 47.76% [intr{irq258: igb0:que 2}] 12 root -92 - 0K 544K CPU0 0 75.1H 32.51% [intr{irq266: igb2:que 0}] 12 root -92 - 0K 544K WAIT 3 66.2H 30.62% [intr{irq269: igb2:que 3}] 12 root -92 - 0K 544K WAIT 3 67.6H 27.98% [intr{irq259: igb0:que 3}] 12 root -92 - 0K 544K CPU0 0 74.4H 25.48% [intr{irq256: igb0:que 0}] 12 root -92 - 0K 544K WAIT 1 57.4H 22.66% [intr{irq267: igb2:que 1}] 12 root -92 - 0K 544K RUN 2 62.7H 21.08% [intr{irq268: igb2:que 2}] 11 root 155 ki31 0K 64K RUN 0 144.0H 18.86% [idle{idle: cpu0}] 11 root 155 ki31 0K 64K RUN 1 148.7H 18.59% [idle{idle: cpu1}] 12 root -92 - 0K 544K CPU1 1 73.9H 15.59% [intr{irq257: igb0:que 1}] 92651 root 52 0 95420K 37880K piperd 1 1:26 14.63% php-fpm: pool nginx (php-fpm) 11 root 155 ki31 0K 64K RUN 3 146.5H 14.17% [idle{idle: cpu3}] 11 root 155 ki31 0K 64K RUN 2 155.8H 11.19% [idle{idle: cpu2}] 73008 root 52 0 56396K 42476K piperd 2 13:31 1.61% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog 20 root -16 - 0K 16K pftm 3 165:40 0.97% [pf purge] 38833 root 52 0 95356K 38180K accept 2 3:22 0.85% php-fpm: pool nginx (php-fpm) 93443 stfn 20 0 9868K 4516K CPU3 3 0:00 0.69% top -aSH 95491 root 20 0 55856K 19000K select 1 0:38 0.25% /usr/local/libexec/ipsec/charon --use-syslog{charon} 86683 root 20 0 10304K 6216K select 0 14:08 0.24% /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf -
A couple of suggestions:
- Intel NIC tuning. Remove igb RX processing limit by adding the following line to /boot/loader.conf.local (and rebooting):
hw.igb.rx_process_limit=-1- Unfortunately there also appears to be a routing performance regression with pfSense 2.4.5 - likely resulting from the OS update to FreeBSD 11.3-STABLE. FreeBSD 11+ offers a routing path optimisation known as ip_tryforward(), which will be used to route IP packets (excluding IPSEC) as long as ICMP redirects are disabled. To disable ICMP redirects, under System / Advanced / System Tunables set sysctls net.inet.ip.redirect & net.inet6.ip6.redirect to 0.
This should result in 5-20% reduction of system CPU for routing (depending on workload).
