Throughput issues on pfSense 2.4.5 with APU2E4
-
Hi
Since upgrading to pfSense 2.4.5 I have a massive issue with throughput on my APU2E4. These were and are not present on 2.4.4p3 and 2.5.0.a.20200330.2017.
I already tried the following:
- Clean install pfSense 2.4.5 RELEASE: issues present
- Clean install pfSense 2.4.4p3 RELEASE: no issues
- Upgrade clean installed pfSense 2.4.4p3 RELEASE to 2.4.5 RELEASE: issues present
- Upgrade clean installed pfSense 2.4.4p3 RELEASE to 2.4.5 RELEASE and then to 2.5.0 SNAPSHOT: issues present on 2.4.5 RELEASE are no longer present (back to normal)
Please find below the tests I did with iperf3 (the APU is currently being set up and therefore the WAN IP is internal as well).
pfSense 2.4.4:----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.77.77.183, port 39487 [ 5] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 23449 [ 7] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 6267 [ 9] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 49437 [ 11] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 28998 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 31.3 MBytes 263 Mbits/sec [ 7] 0.00-1.00 sec 14.0 MBytes 118 Mbits/sec [ 9] 0.00-1.00 sec 32.5 MBytes 272 Mbits/sec [ 11] 0.00-1.00 sec 31.8 MBytes 267 Mbits/sec [SUM] 0.00-1.00 sec 110 MBytes 919 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 28.5 MBytes 239 Mbits/sec [ 7] 1.00-2.00 sec 26.7 MBytes 224 Mbits/sec [ 9] 1.00-2.00 sec 28.4 MBytes 238 Mbits/sec [ 11] 1.00-2.00 sec 28.2 MBytes 236 Mbits/sec [SUM] 1.00-2.00 sec 112 MBytes 938 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 30.1 MBytes 253 Mbits/sec [ 7] 2.00-3.00 sec 17.3 MBytes 145 Mbits/sec [ 9] 2.00-3.00 sec 32.6 MBytes 274 Mbits/sec [ 11] 2.00-3.00 sec 30.5 MBytes 255 Mbits/sec [SUM] 2.00-3.00 sec 111 MBytes 927 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 28.3 MBytes 238 Mbits/sec [ 7] 3.00-4.00 sec 14.4 MBytes 121 Mbits/sec [ 9] 3.00-4.00 sec 36.5 MBytes 306 Mbits/sec [ 11] 3.00-4.00 sec 32.5 MBytes 273 Mbits/sec [SUM] 3.00-4.00 sec 112 MBytes 938 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 27.0 MBytes 226 Mbits/sec [ 7] 4.00-5.00 sec 24.8 MBytes 208 Mbits/sec [ 9] 4.00-5.00 sec 30.7 MBytes 258 Mbits/sec [ 11] 4.00-5.00 sec 28.6 MBytes 240 Mbits/sec [SUM] 4.00-5.00 sec 111 MBytes 932 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 31.2 MBytes 262 Mbits/sec [ 7] 5.00-6.00 sec 22.6 MBytes 190 Mbits/sec [ 9] 5.00-6.00 sec 28.4 MBytes 238 Mbits/sec [ 11] 5.00-6.00 sec 29.2 MBytes 245 Mbits/sec [SUM] 5.00-6.00 sec 111 MBytes 935 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 27.8 MBytes 233 Mbits/sec [ 7] 6.00-7.00 sec 28.5 MBytes 239 Mbits/sec [ 9] 6.00-7.00 sec 27.9 MBytes 234 Mbits/sec [ 11] 6.00-7.00 sec 28.2 MBytes 236 Mbits/sec [SUM] 6.00-7.00 sec 112 MBytes 942 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 27.9 MBytes 234 Mbits/sec [ 7] 7.00-8.00 sec 22.4 MBytes 188 Mbits/sec [ 9] 7.00-8.00 sec 29.4 MBytes 247 Mbits/sec [ 11] 7.00-8.00 sec 29.8 MBytes 250 Mbits/sec [SUM] 7.00-8.00 sec 110 MBytes 919 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 29.8 MBytes 250 Mbits/sec [ 7] 8.00-9.00 sec 12.8 MBytes 107 Mbits/sec [ 9] 8.00-9.00 sec 36.5 MBytes 306 Mbits/sec [ 11] 8.00-9.00 sec 32.8 MBytes 275 Mbits/sec [SUM] 8.00-9.00 sec 112 MBytes 938 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 28.7 MBytes 241 Mbits/sec [ 7] 9.00-10.00 sec 25.3 MBytes 212 Mbits/sec [ 9] 9.00-10.00 sec 29.2 MBytes 245 Mbits/sec [ 11] 9.00-10.00 sec 29.1 MBytes 244 Mbits/sec [SUM] 9.00-10.00 sec 112 MBytes 943 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 291 MBytes 244 Mbits/sec receiver [ 7] 0.00-10.00 sec 209 MBytes 175 Mbits/sec receiver [ 9] 0.00-10.00 sec 312 MBytes 262 Mbits/sec receiver [ 11] 0.00-10.00 sec 301 MBytes 252 Mbits/sec receiver [SUM] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec receiver
pfSense 2.4.5
----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.77.77.183, port 59825 [ 5] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 16642 [ 7] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 46320 [ 9] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 49564 [ 11] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 12839 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 292 KBytes 2.39 Mbits/sec [ 7] 0.00-1.00 sec 43.5 MBytes 365 Mbits/sec [ 9] 0.00-1.00 sec 198 KBytes 1.62 Mbits/sec [ 11] 0.00-1.00 sec 44.2 MBytes 371 Mbits/sec [SUM] 0.00-1.00 sec 88.2 MBytes 740 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec [ 7] 1.00-2.00 sec 45.1 MBytes 379 Mbits/sec [ 9] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec [ 11] 1.00-2.00 sec 46.0 MBytes 386 Mbits/sec [SUM] 1.00-2.00 sec 91.2 MBytes 765 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec [ 7] 2.00-3.00 sec 45.3 MBytes 380 Mbits/sec [ 9] 2.00-3.00 sec 37.1 KBytes 304 Kbits/sec [ 11] 2.00-3.00 sec 45.9 MBytes 385 Mbits/sec [SUM] 2.00-3.00 sec 91.2 MBytes 765 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec [ 7] 3.00-4.00 sec 45.4 MBytes 380 Mbits/sec [ 9] 3.00-4.00 sec 38.5 KBytes 315 Kbits/sec [ 11] 3.00-4.00 sec 46.1 MBytes 386 Mbits/sec [SUM] 3.00-4.00 sec 91.5 MBytes 767 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 39.9 KBytes 327 Kbits/sec [ 7] 4.00-5.00 sec 44.3 MBytes 372 Mbits/sec [ 9] 4.00-5.00 sec 567 KBytes 4.65 Mbits/sec [ 11] 4.00-5.00 sec 45.6 MBytes 383 Mbits/sec [SUM] 4.00-5.00 sec 90.6 MBytes 760 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec [ 7] 5.00-6.00 sec 45.2 MBytes 379 Mbits/sec [ 9] 5.00-6.00 sec 1.43 KBytes 11.7 Kbits/sec [ 11] 5.00-6.00 sec 45.8 MBytes 384 Mbits/sec [SUM] 5.00-6.00 sec 91.0 MBytes 763 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec [ 7] 6.00-7.00 sec 45.3 MBytes 380 Mbits/sec [ 9] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec [ 11] 6.00-7.00 sec 45.9 MBytes 385 Mbits/sec [SUM] 6.00-7.00 sec 91.1 MBytes 764 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec [ 7] 7.00-8.00 sec 45.1 MBytes 378 Mbits/sec [ 9] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec [ 11] 7.00-8.00 sec 45.9 MBytes 385 Mbits/sec [SUM] 7.00-8.00 sec 91.0 MBytes 763 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec [ 7] 8.00-9.00 sec 45.0 MBytes 378 Mbits/sec [ 9] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec [ 11] 8.00-9.00 sec 45.7 MBytes 383 Mbits/sec [SUM] 8.00-9.00 sec 90.8 MBytes 761 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec [ 7] 9.00-10.00 sec 45.2 MBytes 379 Mbits/sec [ 9] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec [ 11] 9.00-10.00 sec 45.6 MBytes 383 Mbits/sec [SUM] 9.00-10.00 sec 90.9 MBytes 762 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 332 KBytes 272 Kbits/sec receiver [ 7] 0.00-10.00 sec 449 MBytes 377 Mbits/sec receiver [ 9] 0.00-10.00 sec 843 KBytes 690 Kbits/sec receiver [ 11] 0.00-10.00 sec 457 MBytes 383 Mbits/sec receiver [SUM] 0.00-10.00 sec 907 MBytes 761 Mbits/sec receiver
pfSense 2.5.0.a.20200330.2017:
----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.77.77.183, port 36901 [ 5] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 64405 [ 7] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 9249 [ 9] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 24535 [ 11] local 10.77.77.165 port 5201 connected to 10.77.77.183 port 21955 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 32.1 MBytes 269 Mbits/sec [ 7] 0.00-1.00 sec 27.1 MBytes 228 Mbits/sec [ 9] 0.00-1.00 sec 20.1 MBytes 169 Mbits/sec [ 11] 0.00-1.00 sec 31.8 MBytes 267 Mbits/sec [SUM] 0.00-1.00 sec 111 MBytes 933 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 32.3 MBytes 271 Mbits/sec [ 7] 1.00-2.00 sec 27.2 MBytes 228 Mbits/sec [ 9] 1.00-2.00 sec 19.9 MBytes 167 Mbits/sec [ 11] 1.00-2.00 sec 31.4 MBytes 263 Mbits/sec [SUM] 1.00-2.00 sec 111 MBytes 929 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 32.2 MBytes 270 Mbits/sec [ 7] 2.00-3.00 sec 27.2 MBytes 228 Mbits/sec [ 9] 2.00-3.00 sec 20.1 MBytes 168 Mbits/sec [ 11] 2.00-3.00 sec 32.1 MBytes 269 Mbits/sec [SUM] 2.00-3.00 sec 112 MBytes 936 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 32.1 MBytes 270 Mbits/sec [ 7] 3.00-4.00 sec 27.2 MBytes 228 Mbits/sec [ 9] 3.00-4.00 sec 19.8 MBytes 166 Mbits/sec [ 11] 3.00-4.00 sec 31.8 MBytes 267 Mbits/sec [SUM] 3.00-4.00 sec 111 MBytes 931 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 32.0 MBytes 269 Mbits/sec [ 7] 4.00-5.00 sec 27.1 MBytes 227 Mbits/sec [ 9] 4.00-5.00 sec 19.8 MBytes 166 Mbits/sec [ 11] 4.00-5.00 sec 32.1 MBytes 269 Mbits/sec [SUM] 4.00-5.00 sec 111 MBytes 931 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 32.2 MBytes 270 Mbits/sec [ 7] 5.00-6.00 sec 27.2 MBytes 228 Mbits/sec [ 9] 5.00-6.00 sec 19.9 MBytes 167 Mbits/sec [ 11] 5.00-6.00 sec 32.1 MBytes 269 Mbits/sec [SUM] 5.00-6.00 sec 111 MBytes 935 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 32.0 MBytes 268 Mbits/sec [ 7] 6.00-7.00 sec 27.2 MBytes 229 Mbits/sec [ 9] 6.00-7.00 sec 20.0 MBytes 168 Mbits/sec [ 11] 6.00-7.00 sec 32.5 MBytes 272 Mbits/sec [SUM] 6.00-7.00 sec 112 MBytes 937 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 32.0 MBytes 268 Mbits/sec [ 7] 7.00-8.00 sec 27.1 MBytes 227 Mbits/sec [ 9] 7.00-8.00 sec 20.1 MBytes 168 Mbits/sec [ 11] 7.00-8.00 sec 32.3 MBytes 271 Mbits/sec [SUM] 7.00-8.00 sec 111 MBytes 935 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 32.1 MBytes 269 Mbits/sec [ 7] 8.00-9.00 sec 27.1 MBytes 227 Mbits/sec [ 9] 8.00-9.00 sec 19.9 MBytes 167 Mbits/sec [ 11] 8.00-9.00 sec 32.3 MBytes 271 Mbits/sec [SUM] 8.00-9.00 sec 111 MBytes 934 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 32.0 MBytes 268 Mbits/sec [ 7] 9.00-10.00 sec 27.0 MBytes 226 Mbits/sec [ 9] 9.00-10.00 sec 19.9 MBytes 167 Mbits/sec [ 11] 9.00-10.00 sec 31.9 MBytes 268 Mbits/sec [SUM] 9.00-10.00 sec 111 MBytes 929 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 10.00-10.01 sec 385 KBytes 282 Mbits/sec [ 7] 10.00-10.01 sec 311 KBytes 227 Mbits/sec [ 9] 10.00-10.01 sec 230 KBytes 168 Mbits/sec [ 11] 10.00-10.01 sec 361 KBytes 264 Mbits/sec [SUM] 10.00-10.01 sec 1.26 MBytes 941 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.01 sec 321 MBytes 269 Mbits/sec receiver [ 7] 0.00-10.01 sec 272 MBytes 228 Mbits/sec receiver [ 9] 0.00-10.01 sec 200 MBytes 167 Mbits/sec receiver [ 11] 0.00-10.01 sec 321 MBytes 269 Mbits/sec receiver [SUM] 0.00-10.01 sec 1.09 GBytes 933 Mbits/sec receiver
Note the 0.00 bits/sec on 2.4.5.
Do you have any idea why this is happening?
The following custom loader.conf.local was used and NIC hardware offloading was activated as well on all tested versions to enable gigabit throughput on PC Engines' APU2s:
# agree with Intel license terms legal.intel_igb.license_ack="1" legal.intel_iwi.license_ack="1" legal.intel_ipw.license_ack="1" # this is the magic. If you don't set this, queues won't be utilized properly # allow multiple processes for receive/transmit processing hw.igb.rx_process_limit="-1" hw.igb.tx_process_limit="-1"
I used firmware 4.11.0.4 on the APU for all tests so it is not related to this reported issue: https://github.com/pcengines/coreboot/issues/387
-
You're not alone, please see this thread.
-
@muppet You're right. In my case it seems to be CPU related. The utilization is much higher than before and therefore the individual cores cannot handle the load for the queue they're assigned to.