Reply to PC Engines apu2 optimisation needed (pfSense 2.4.5 vs 2.5.0 performance comparison) on Mon, 01 Mar 2021 23:19:07 GMT

picov — Mon, 01 Mar 2021 23:19:07 GMT

Please note that on post #1 the iperf server and client was running into the APU2 devices thus the traffic isn't the best achieveable.

I' ve repeated tha same tests in a better scenario that leaves the APU2 devices to only handle the traffic generated elsewhere.

                        1                  2
    ┌────────┐    ┌───────────┐      ┌───────────┐    ┌────────┐
    │        │    │           │      │           │    │        │
    │        ├─┐  │    APU2   │      │    APU2   │  ┌─┤        │
    ├────────┤ │  │           │      │           │  │ ├────────┤
    │┼┼┼┼┼┼┼┼│ │  └─┬───────┬─┘      └─┬───────┬─┘  │ │┼┼┼┼┼┼┼┼│
    │┼┼┼┼┼┼┼┼│ │    │       │          │       │    │ │┼┼┼┼┼┼┼┼│
    └────────┘ │    │       │    WAN   │       │    │ └────────┘
  iperf server └────┘       └──────────┘       └────┘ iperf client
                LAN1                             LAN2

APU2 (AMD GX-412TC 1 GHz quad core - 4GB RAM - 3 Intel I210 NIC)

BIOS v4.13.0.3 mainline

AES-NI CPU Crypto: Yes (inactive) (same results if activated so this option has no effect)

System tested:

pfSense 2.4.5 (default)
pfSense 2.4.5 (modified)
/boot/loader.conf.local
legal.intel_igb.license_ack="1"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
Advanced -> tunables
net.inet.ip.redirect: 0
net.inet6.ip6.redirect: 0
pfSense 2.5.0 (default)
pfSense 2.5.0 (modified)
Advanced -> tunables
net.inet.ip.redirect: 0
net.inet6.ip6.redirect: 0

LAN1 to LAN2 (1 connection)

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 592 MBytes 497 Mbits/sec sender
[ 5] 0.00-10.00 sec 590 MBytes 495 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 589 MBytes 494 Mbits/sec sender
[ 5] 0.00-10.00 sec 587 MBytes 492 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 466 MBytes 391 Mbits/sec sender
[ 5] 0.00-10.00 sec 464 MBytes 389 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 466 MBytes 391 Mbits/sec sender
[ 5] 0.00-10.00 sec 465 MBytes 390 Mbits/sec receiver

LAN1 to LAN2 (10 connections)

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 13.3 MBytes 11.1 Mbits/sec [ 5] 0.00-10.03 sec 12.0 MBytes 10.1 Mbits/sec [ 7] 0.00-10.00 sec 51.0 MBytes 42.8 Mbits/sec [ 7] 0.00-10.03 sec 50.8 MBytes 42.5 Mbits/sec [ 9] 0.00-10.00 sec 326 MBytes 273 Mbits/sec [ 9] 0.00-10.03 sec 325 MBytes 272 Mbits/sec [ 11] 0.00-10.00 sec 32.4 MBytes 27.2 Mbits/sec [ 11] 0.00-10.03 sec 32.0 MBytes 26.8 Mbits/sec [ 13] 0.00-10.00 sec 3.24 MBytes 2.72 Mbits/sec [ 13] 0.00-10.03 sec 3.06 MBytes 2.56 Mbits/sec [ 15] 0.00-10.00 sec 358 MBytes 300 Mbits/sec [ 15] 0.00-10.03 sec 357 MBytes 299 Mbits/sec [ 17] 0.00-10.00 sec 121 MBytes 102 Mbits/sec [ 17] 0.00-10.03 sec 121 MBytes 101 Mbits/sec [ 19] 0.00-10.00 sec 36.2 MBytes 30.4 Mbits/sec [ 19] 0.00-10.03 sec 36.0 MBytes 30.1 Mbits/sec [ 21] 0.00-10.00 sec 41.4 MBytes 34.7 Mbits/sec [ 21] 0.00-10.03 sec 40.8 MBytes 34.1 Mbits/sec [ 23] 0.00-10.00 sec 51.7 MBytes 43.3 Mbits/sec [ 23] 0.00-10.03 sec 49.8 MBytes 41.6 Mbits/sec [SUM] 0.00-10.00 sec 1.01 GBytes 867 Mbits/sec [SUM] 0.00-10.03 sec 1.00 GBytes 860 Mbits/sec sender
receiver
sender
receiver
sender
receiver
sender
receiver
sender
receiver
sender
receiver
sender
receiver
sender
receiver
sender
receiver
sender
receiver
sender
receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 112 MBytes 93.7 Mbits/sec sender
[ 5] 0.00-10.01 sec 111 MBytes 93.0 Mbits/sec receiver
[ 7] 0.00-10.00 sec 2.25 MBytes 1.89 Mbits/sec sender
[ 7] 0.00-10.01 sec 382 KBytes 313 Kbits/sec receiver
[ 9] 0.00-10.00 sec 129 MBytes 108 Mbits/sec sender
[ 9] 0.00-10.01 sec 129 MBytes 108 Mbits/sec receiver
[ 11] 0.00-10.00 sec 2.82 MBytes 2.37 Mbits/sec sender
[ 11] 0.00-10.01 sec 1.24 MBytes 1.04 Mbits/sec receiver
[ 13] 0.00-10.00 sec 160 MBytes 134 Mbits/sec sender
[ 13] 0.00-10.01 sec 158 MBytes 133 Mbits/sec receiver
[ 15] 0.00-10.00 sec 3.25 MBytes 2.73 Mbits/sec sender
[ 15] 0.00-10.01 sec 1.25 MBytes 1.05 Mbits/sec receiver
[ 17] 0.00-10.00 sec 236 MBytes 198 Mbits/sec sender
[ 17] 0.00-10.01 sec 235 MBytes 197 Mbits/sec receiver
[ 19] 0.00-10.00 sec 240 MBytes 202 Mbits/sec sender
[ 19] 0.00-10.01 sec 239 MBytes 201 Mbits/sec receiver
[ 21] 0.00-10.00 sec 231 MBytes 194 Mbits/sec sender
[ 21] 0.00-10.01 sec 230 MBytes 192 Mbits/sec receiver
[ 23] 0.00-10.00 sec 2.22 MBytes 1.86 Mbits/sec sender
[ 23] 0.00-10.01 sec 349 KBytes 286 Kbits/sec receiver
[SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec sender
[SUM] 0.00-10.01 sec 1.08 GBytes 926 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 110 MBytes 92.3 Mbits/sec sender
[ 5] 0.00-10.01 sec 109 MBytes 91.6 Mbits/sec receiver
[ 7] 0.00-10.00 sec 111 MBytes 93.4 Mbits/sec sender
[ 7] 0.00-10.01 sec 111 MBytes 93.1 Mbits/sec receiver
[ 9] 0.00-10.00 sec 113 MBytes 94.8 Mbits/sec sender
[ 9] 0.00-10.01 sec 113 MBytes 94.5 Mbits/sec receiver
[ 11] 0.00-10.00 sec 109 MBytes 91.7 Mbits/sec sender
[ 11] 0.00-10.01 sec 109 MBytes 91.1 Mbits/sec receiver
[ 13] 0.00-10.00 sec 113 MBytes 95.0 Mbits/sec sender
[ 13] 0.00-10.01 sec 113 MBytes 94.6 Mbits/sec receiver
[ 15] 0.00-10.00 sec 113 MBytes 95.0 Mbits/sec sender
[ 15] 0.00-10.01 sec 113 MBytes 94.7 Mbits/sec receiver
[ 17] 0.00-10.00 sec 113 MBytes 94.6 Mbits/sec sender
[ 17] 0.00-10.01 sec 112 MBytes 94.3 Mbits/sec receiver
[ 19] 0.00-10.00 sec 113 MBytes 95.1 Mbits/sec sender
[ 19] 0.00-10.01 sec 113 MBytes 94.8 Mbits/sec receiver
[ 21] 0.00-10.00 sec 112 MBytes 93.8 Mbits/sec sender
[ 21] 0.00-10.01 sec 111 MBytes 93.1 Mbits/sec receiver
[ 23] 0.00-10.00 sec 112 MBytes 93.7 Mbits/sec sender
[ 23] 0.00-10.01 sec 111 MBytes 93.4 Mbits/sec receiver
[SUM] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec sender
[SUM] 0.00-10.01 sec 1.09 GBytes 935 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 113 MBytes 94.6 Mbits/sec sender
[ 5] 0.00-10.01 sec 112 MBytes 94.1 Mbits/sec receiver
[ 7] 0.00-10.00 sec 96.3 MBytes 80.8 Mbits/sec sender
[ 7] 0.00-10.01 sec 95.5 MBytes 80.0 Mbits/sec receiver
[ 9] 0.00-10.00 sec 115 MBytes 96.5 Mbits/sec sender
[ 9] 0.00-10.01 sec 115 MBytes 96.2 Mbits/sec receiver
[ 11] 0.00-10.00 sec 126 MBytes 106 Mbits/sec sender
[ 11] 0.00-10.01 sec 126 MBytes 105 Mbits/sec receiver
[ 13] 0.00-10.00 sec 116 MBytes 97.0 Mbits/sec sender
[ 13] 0.00-10.01 sec 115 MBytes 96.4 Mbits/sec receiver
[ 15] 0.00-10.00 sec 93.7 MBytes 78.6 Mbits/sec sender
[ 15] 0.00-10.01 sec 93.0 MBytes 77.9 Mbits/sec receiver
[ 17] 0.00-10.00 sec 110 MBytes 92.5 Mbits/sec sender
[ 17] 0.00-10.01 sec 109 MBytes 91.8 Mbits/sec receiver
[ 19] 0.00-10.00 sec 105 MBytes 87.8 Mbits/sec sender
[ 19] 0.00-10.01 sec 104 MBytes 87.2 Mbits/sec receiver
[ 21] 0.00-10.00 sec 127 MBytes 107 Mbits/sec sender
[ 21] 0.00-10.01 sec 127 MBytes 106 Mbits/sec receiver
[ 23] 0.00-10.00 sec 116 MBytes 97.6 Mbits/sec sender
[ 23] 0.00-10.01 sec 116 MBytes 97.1 Mbits/sec receiver
[SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec sender
[SUM] 0.00-10.01 sec 1.09 GBytes 932 Mbits/sec receiver

OpenVPN (P2P PSK L2L) AES128-CBC SHA1

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 108 MBytes 90.4 Mbits/sec sender
[ 5] 0.00-10.01 sec 108 MBytes 90.2 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 102 MBytes 85.9 Mbits/sec sender
[ 5] 0.00-10.01 sec 102 MBytes 85.6 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 94.1 MBytes 78.9 Mbits/sec sender
[ 5] 0.00-10.02 sec 93.9 MBytes 78.6 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 88.6 MBytes 74.4 Mbits/sec sender
[ 5] 0.00-10.02 sec 88.4 MBytes 74.0 Mbits/sec receiver

OpenVPN (P2P PSK L2L) AES256-CBC SHA256

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 89.2 MBytes 74.8 Mbits/sec sender
[ 5] 0.00-10.02 sec 89.0 MBytes 74.6 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 85.6 MBytes 71.8 Mbits/sec sender
[ 5] 0.00-10.02 sec 85.4 MBytes 71.5 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 78.1 MBytes 65.5 Mbits/sec sender
[ 5] 0.00-10.02 sec 78.0 MBytes 65.3 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 71.5 MBytes 60.0 Mbits/sec sender
[ 5] 0.00-10.02 sec 71.4 MBytes 59.7 Mbits/sec receiver

OpenVPN (P2P TLS L2L) AES128-GCM

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 133 MBytes 112 Mbits/sec sender
[ 5] 0.00-10.00 sec 133 MBytes 112 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 139 MBytes 116 Mbits/sec sender
[ 5] 0.00-10.00 sec 139 MBytes 116 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 115 MBytes 96.6 Mbits/sec sender
[ 5] 0.00-10.01 sec 115 MBytes 96.2 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 122 MBytes 102 Mbits/sec sender
[ 5] 0.00-10.01 sec 122 MBytes 102 Mbits/sec receiver

OpenVPN (P2P TLS L2L) AES256-GCM

pfSense 2.4.5 (default)
[ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec sender
[ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 120 MBytes 101 Mbits/sec sender
[ 5] 0.00-10.00 sec 120 MBytes 100 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 101 MBytes 84.5 Mbits/sec sender
[ 5] 0.00-10.00 sec 101 MBytes 84.3 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 107 MBytes 89.5 Mbits/sec sender
[ 5] 0.00-10.01 sec 107 MBytes 89.4 Mbits/sec receiver

IPSEC VPN L2L AES128-CBC SHA1

Use asynchronous mode to parallelize multiple cryptography jobs = OFF

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 85.0 MBytes 71.3 Mbits/sec sender
[ 5] 0.00-10.14 sec 84.2 MBytes 69.6 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 80.6 MBytes 67.6 Mbits/sec sender
[ 5] 0.00-10.00 sec 78.6 MBytes 65.9 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 77.0 MBytes 64.6 Mbits/sec sender
[ 5] 0.00-10.00 sec 75.3 MBytes 63.2 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 79.8 MBytes 67.0 Mbits/sec sender
[ 5] 0.00-10.00 sec 77.9 MBytes 65.3 Mbits/sec receiver

Use asynchronous mode to parallelize multiple cryptography jobs = ON

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 198 MBytes 166 Mbits/sec sender
[ 5] 0.00-10.05 sec 198 MBytes 166 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 193 MBytes 162 Mbits/sec sender
[ 5] 0.00-10.03 sec 193 MBytes 161 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 65.2 MBytes 54.7 Mbits/sec sender
[ 5] 0.00-10.15 sec 65.2 MBytes 53.9 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 62.5 MBytes 52.4 Mbits/sec sender
[ 5] 0.00-10.14 sec 62.5 MBytes 51.7 Mbits/sec receiver

IPSEC VPN L2L AES256-CBC SHA256

Use asynchronous mode to parallelize multiple cryptography jobs = OFF

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 77.9 MBytes 65.3 Mbits/sec sender
[ 5] 0.00-10.08 sec 76.5 MBytes 63.7 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 73.8 MBytes 61.9 Mbits/sec sender
[ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 70.8 MBytes 59.4 Mbits/sec sender
[ 5] 0.00-10.00 sec 68.8 MBytes 57.7 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 74.6 MBytes 62.6 Mbits/sec sender
[ 5] 0.00-10.00 sec 73.4 MBytes 61.5 Mbits/sec receiver

Use asynchronous mode to parallelize multiple cryptography jobs = ON

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 181 MBytes 151 Mbits/sec sender
[ 5] 0.00-10.05 sec 181 MBytes 151 Mbits/sec receiver
pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 182 MBytes 152 Mbits/sec sender
[ 5] 0.00-10.09 sec 182 MBytes 151 Mbits/sec receiver
pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 55.9 MBytes 46.9 Mbits/sec sender
[ 5] 0.00-10.24 sec 55.9 MBytes 45.8 Mbits/sec receiver
pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 55.7 MBytes 46.7 Mbits/sec sender
[ 5] 0.00-10.33 sec 55.7 MBytes 45.3 Mbits/sec receiver

Conclusions

Switching from 2.4.5 to 2.5.0 (on APU2 platform) leads to performances loss about 10%-25% in almost all tests done.
The heavy loss on IPSEC VPN with the option "Use asynchronous mode to parallelize multiple cryptography jobs"=enabled definitely needs some investigations.

I would be curious to know if the same situation occurs on other hardware thus I hope that someone will post some results.

Reply to PC Engines apu2 optimisation needed (pfSense 2.4.5 vs 2.5.0 performance comparison) on Sat, 27 Feb 2021 19:33:36 GMT

tman222 — Sat, 27 Feb 2021 19:33:36 GMT

@picov - great find! Now I'm quite curious if tuning the default settings in iflib will help at all or whether the only other option is to downgrade / buy faster hardware.

Reply to PC Engines apu2 optimisation needed (pfSense 2.4.5 vs 2.5.0 performance comparison) on Sat, 27 Feb 2021 18:42:35 GMT

picov — Sat, 27 Feb 2021 18:42:35 GMT

It seems that the performance loss on this hardware was already predicted here:

https://github.com/ocochard/netbenches/blob/master/AMD_GX-412TC_4Cores-Intel_i210AT/forwarding-pf-ipfw/results/fbsd12-head-vs-11-stable.r318516/README.md

Reply to PC Engines apu2 optimisation needed (pfSense 2.4.5 vs 2.5.0 performance comparison) on Sat, 27 Feb 2021 17:52:18 GMT

tman222 — Sat, 27 Feb 2021 17:52:18 GMT

Unfortunately I don't have a an APU2 system to test with, but I would check out the available em and iflib tunables. It may end up being a bit trial an error to determine a configuration that works well:

https://www.freebsd.org/cgi/man.cgi?query=iflib&apropos=0&sektion=4&manpath=FreeBSD+12.2-stable&arch=default&format=html

https://www.freebsd.org/cgi/man.cgi?query=em&apropos=0&sektion=4&manpath=FreeBSD+12.2-stable&arch=default&format=html