Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PC Engines apu2 optimisation needed (pfSense 2.4.5 vs 2.5.0 performance comparison)

    Scheduled Pinned Locked Moved Hardware
    5 Posts 2 Posters 1.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      picov
      last edited by picov

      SOME RAW DATA COMPARISION ON APU2 (3 Intel I210 NIC) BIOS v4.11.0.3 mainline

      Two devices connected directly on WANs (same hardware, same software, same cables, same firmware)

      AES-NI CPU Crypto: Yes (inactive) (same results if activated so this option has no effect)

      • psSense 2.4.5
        (tuned only with with hw.igb.rx_process_limit="-1" and hw.igb.tx_process_limit="-1" in loader.conf.local)

      • psSense 2.5.0
        (same tuning of 2.4.5 not works with 2.5.0)

      WAN to WAN (1 connection)

      • pfSense 2.4.5
      [2.4.5-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.1.1
      Connecting to host 192.168.1.1, port 5201
      [  5] local 192.168.1.2 port 9389 connected to 192.168.1.1 port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec   381 MBytes   320 Mbits/sec    0             sender
      [  5]   0.00-10.02  sec   381 MBytes   319 Mbits/sec                  receiver
      
      • pfSense 2.5.0
      [2.5.0-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.1.1
      Connecting to host 192.168.1.1, port 5201
      [  5] local 192.168.1.2 port 28220 connected to 192.168.1.1 port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec   336 MBytes   282 Mbits/sec    0             sender
      [  5]   0.00-10.03  sec   336 MBytes   281 Mbits/sec                  receiver
      

      WAN to WAN (10 connections)

      • pfSense 2.4.5
      [2.4.5-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.1.1 -P 10
      Connecting to host 192.168.1.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  79.2 MBytes  66.4 Mbits/sec    0             sender
      [  5]   0.00-10.03  sec  79.1 MBytes  66.2 Mbits/sec                  receiver
      [  7]   0.00-10.00  sec  78.2 MBytes  65.6 Mbits/sec   60             sender
      [  7]   0.00-10.03  sec  78.2 MBytes  65.4 Mbits/sec                  receiver
      [  9]   0.00-10.00  sec   184 MBytes   154 Mbits/sec   61             sender
      [  9]   0.00-10.03  sec   184 MBytes   154 Mbits/sec                  receiver
      [ 11]   0.00-10.00  sec  73.8 MBytes  61.9 Mbits/sec    0             sender
      [ 11]   0.00-10.03  sec  73.8 MBytes  61.7 Mbits/sec                  receiver
      [ 13]   0.00-10.00  sec  78.2 MBytes  65.6 Mbits/sec    0             sender
      [ 13]   0.00-10.03  sec  78.2 MBytes  65.4 Mbits/sec                  receiver
      [ 15]   0.00-10.00  sec  77.8 MBytes  65.3 Mbits/sec    0             sender
      [ 15]   0.00-10.03  sec  77.8 MBytes  65.1 Mbits/sec                  receiver
      [ 17]   0.00-10.00  sec   125 MBytes   105 Mbits/sec    0             sender
      [ 17]   0.00-10.03  sec   125 MBytes   105 Mbits/sec                  receiver
      [ 19]   0.00-10.00  sec   168 MBytes   141 Mbits/sec   68             sender
      [ 19]   0.00-10.03  sec   168 MBytes   140 Mbits/sec                  receiver
      [ 21]   0.00-10.00  sec   130 MBytes   109 Mbits/sec   66             sender
      [ 21]   0.00-10.03  sec   130 MBytes   109 Mbits/sec                  receiver
      [ 23]   0.00-10.00  sec   123 MBytes   103 Mbits/sec    0             sender
      [ 23]   0.00-10.03  sec   123 MBytes   103 Mbits/sec                  receiver
      [SUM]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec  255             sender
      [SUM]   0.00-10.03  sec  1.09 GBytes   934 Mbits/sec                  receiver
      
      • pfSense 2.5.0
      [2.5.0-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.1.1 -P 10
      Connecting to host 192.168.1.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  91.6 MBytes  76.8 Mbits/sec    0             sender
      [  5]   0.00-10.04  sec  91.6 MBytes  76.6 Mbits/sec                  receiver
      [  7]   0.00-10.00  sec  72.6 MBytes  60.9 Mbits/sec    0             sender
      [  7]   0.00-10.04  sec  72.5 MBytes  60.6 Mbits/sec                  receiver
      [  9]   0.00-10.00  sec   238 MBytes   200 Mbits/sec    0             sender
      [  9]   0.00-10.04  sec   238 MBytes   199 Mbits/sec                  receiver
      [ 11]   0.00-10.00  sec  71.3 MBytes  59.8 Mbits/sec    0             sender
      [ 11]   0.00-10.04  sec  71.3 MBytes  59.6 Mbits/sec                  receiver
      [ 13]   0.00-10.00  sec  73.8 MBytes  61.9 Mbits/sec    0             sender
      [ 13]   0.00-10.04  sec  73.7 MBytes  61.6 Mbits/sec                  receiver
      [ 15]   0.00-10.00  sec  42.6 MBytes  35.7 Mbits/sec    0             sender
      [ 15]   0.00-10.04  sec  42.5 MBytes  35.5 Mbits/sec                  receiver
      [ 17]   0.00-10.00  sec  73.8 MBytes  61.9 Mbits/sec    0             sender
      [ 17]   0.00-10.04  sec  73.8 MBytes  61.7 Mbits/sec                  receiver
      [ 19]   0.00-10.00  sec  71.0 MBytes  59.6 Mbits/sec    0             sender
      [ 19]   0.00-10.04  sec  71.0 MBytes  59.3 Mbits/sec                  receiver
      [ 21]   0.00-10.00  sec  73.2 MBytes  61.4 Mbits/sec    0             sender
      [ 21]   0.00-10.04  sec  73.2 MBytes  61.1 Mbits/sec                  receiver
      [ 23]   0.00-10.00  sec  42.4 MBytes  35.6 Mbits/sec    0             sender
      [ 23]   0.00-10.04  sec  42.4 MBytes  35.4 Mbits/sec                  receiver
      [SUM]   0.00-10.00  sec   851 MBytes   713 Mbits/sec    0             sender
      [SUM]   0.00-10.04  sec   850 MBytes   711 Mbits/sec                  receiver
      

      OPENVPN L2L AES128-CBC SHA1

      • pfSense 2.4.5
      [  5] local 10.10.11.2 port 45726 connected to 192.168.101.1 port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  62.9 MBytes  52.8 Mbits/sec    0             sender
      [  5]   0.00-10.17  sec  62.9 MBytes  51.8 Mbits/sec                  receiver
      
      • pfSense 2.5.0
      Connecting to host 192.168.101.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  58.8 MBytes  49.3 Mbits/sec    0             sender
      [  5]   0.00-10.03  sec  58.7 MBytes  49.1 Mbits/sec                  receiver
      

      OPENVPN L2L AES256-CBC SHA256

      • pfSense 2.4.5
      [2.4.5-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.102.1
      Connecting to host 192.168.102.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  54.9 MBytes  46.1 Mbits/sec    0             sender
      [  5]   0.00-10.01  sec  54.9 MBytes  46.0 Mbits/sec                  receiver
      
      • pfSense 2.5.0
      Connecting to host 192.168.102.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  51.0 MBytes  42.8 Mbits/sec    0             sender
      [  5]   0.00-10.04  sec  51.0 MBytes  42.6 Mbits/sec                  receiver
      

      OPENVPN L2L AES128-GCM

      • pfSense 2.4.5
      Connecting to host 192.168.103.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  64.8 MBytes  54.3 Mbits/sec    0             sender
      [  5]   0.00-10.37  sec  64.8 MBytes  52.4 Mbits/sec                  receiver
      
      • pfSense 2.5.0
      [2.5.0-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.103.1
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  62.8 MBytes  52.7 Mbits/sec    0             sender
      [  5]   0.00-10.05  sec  62.8 MBytes  52.4 Mbits/sec                  receiver
      

      VPN L2L IPSEC AES128 SHA1

      Use asynchronous mode to parallelize multiple cryptography jobs DISABLED on both APU2

      • pfSense 2.4.5
      Connecting to host 192.168.104.1, port 5201
      [  5] local 192.168.204.1 port 46658 connected to 192.168.104.1 port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  60.5 MBytes  50.7 Mbits/sec    0             sender
      [  5]   0.00-10.01  sec  60.5 MBytes  50.7 Mbits/sec                  receiver
      
      • pfSense 2.5.0
      [2.5.0-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.104.1
      Connecting to host 192.168.104.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  34.0 MBytes  28.5 Mbits/sec    0             sender
      [  5]   0.00-10.28  sec  34.0 MBytes  27.8 Mbits/sec                  receiver
      

      VPN L2L IPSEC AES128 SHA1

      Use asynchronous mode to parallelize multiple cryptography jobs ENABLED on both APU2

      • pfSense 2.4.5
      [2.4.5-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.104.1
      Connecting to host 192.168.104.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec   143 MBytes   120 Mbits/sec    0             sender
      [  5]   0.00-10.05  sec   143 MBytes   119 Mbits/sec                  receiver
      
      • pfSense 2.5.0
      [2.5.0-RELEASE][root@pfSense-site2.localdomain]/root: iperf3 -c 192.168.104.1
      Connecting to host 192.168.104.1, port 5201
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bitrate         Retr
      [  5]   0.00-10.00  sec  40.0 MBytes  33.5 Mbits/sec    0             sender
      [  5]   0.00-10.01  sec  40.0 MBytes  33.5 Mbits/sec                  receiver
      

      A FreeBSD guru able to tune 2.5.0 in order to regain te performance loss is probably needed.

      1 Reply Last reply Reply Quote 1
      • T
        tman222
        last edited by

        Unfortunately I don't have a an APU2 system to test with, but I would check out the available em and iflib tunables. It may end up being a bit trial an error to determine a configuration that works well:

        https://www.freebsd.org/cgi/man.cgi?query=iflib&apropos=0&sektion=4&manpath=FreeBSD+12.2-stable&arch=default&format=html

        https://www.freebsd.org/cgi/man.cgi?query=em&apropos=0&sektion=4&manpath=FreeBSD+12.2-stable&arch=default&format=html

        1 Reply Last reply Reply Quote 0
        • P
          picov
          last edited by picov

          It seems that the performance loss on this hardware was already predicted here:

          https://github.com/ocochard/netbenches/blob/master/AMD_GX-412TC_4Cores-Intel_i210AT/forwarding-pf-ipfw/results/fbsd12-head-vs-11-stable.r318516/README.md

          T 1 Reply Last reply Reply Quote 1
          • T
            tman222 @picov
            last edited by

            @picov - great find! Now I'm quite curious if tuning the default settings in iflib will help at all or whether the only other option is to downgrade / buy faster hardware.

            1 Reply Last reply Reply Quote 0
            • P
              picov
              last edited by picov

              Please note that on post #1 the iperf server and client was running into the APU2 devices thus the traffic isn't the best achieveable.

              I' ve repeated tha same tests in a better scenario that leaves the APU2 devices to only handle the traffic generated elsewhere.

                                      1                  2
                  ┌────────┐    ┌───────────┐      ┌───────────┐    ┌────────┐
                  │        │    │           │      │           │    │        │
                  │        ├─┐  │    APU2   │      │    APU2   │  ┌─┤        │
                  ├────────┤ │  │           │      │           │  │ ├────────┤
                  │┼┼┼┼┼┼┼┼│ │  └─┬───────┬─┘      └─┬───────┬─┘  │ │┼┼┼┼┼┼┼┼│
                  │┼┼┼┼┼┼┼┼│ │    │       │          │       │    │ │┼┼┼┼┼┼┼┼│
                  └────────┘ │    │       │    WAN   │       │    │ └────────┘
                iperf server └────┘       └──────────┘       └────┘ iperf client
                              LAN1                             LAN2
                        
              

              APU2 (AMD GX-412TC 1 GHz quad core - 4GB RAM - 3 Intel I210 NIC)

              BIOS v4.13.0.3 mainline

              AES-NI CPU Crypto: Yes (inactive) (same results if activated so this option has no effect)

              System tested:

              • pfSense 2.4.5 (default)

              • pfSense 2.4.5 (modified)
                /boot/loader.conf.local
                legal.intel_igb.license_ack="1"
                hw.igb.rx_process_limit="-1"
                hw.igb.tx_process_limit="-1"
                Advanced -> tunables
                net.inet.ip.redirect: 0
                net.inet6.ip6.redirect: 0

              • pfSense 2.5.0 (default)

              • pfSense 2.5.0 (modified)
                Advanced -> tunables
                net.inet.ip.redirect: 0
                net.inet6.ip6.redirect: 0

              LAN1 to LAN2 (1 connection)

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 592 MBytes 497 Mbits/sec sender
                [ 5] 0.00-10.00 sec 590 MBytes 495 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 589 MBytes 494 Mbits/sec sender
                [ 5] 0.00-10.00 sec 587 MBytes 492 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 466 MBytes 391 Mbits/sec sender
                [ 5] 0.00-10.00 sec 464 MBytes 389 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 466 MBytes 391 Mbits/sec sender
                [ 5] 0.00-10.00 sec 465 MBytes 390 Mbits/sec receiver

              LAN1 to LAN2 (10 connections)

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 13.3 MBytes 11.1 Mbits/sec sender
                [ 5] 0.00-10.03 sec 12.0 MBytes 10.1 Mbits/sec receiver
                [ 7] 0.00-10.00 sec 51.0 MBytes 42.8 Mbits/sec sender
                [ 7] 0.00-10.03 sec 50.8 MBytes 42.5 Mbits/sec receiver
                [ 9] 0.00-10.00 sec 326 MBytes 273 Mbits/sec sender
                [ 9] 0.00-10.03 sec 325 MBytes 272 Mbits/sec receiver
                [ 11] 0.00-10.00 sec 32.4 MBytes 27.2 Mbits/sec sender
                [ 11] 0.00-10.03 sec 32.0 MBytes 26.8 Mbits/sec receiver
                [ 13] 0.00-10.00 sec 3.24 MBytes 2.72 Mbits/sec sender
                [ 13] 0.00-10.03 sec 3.06 MBytes 2.56 Mbits/sec receiver
                [ 15] 0.00-10.00 sec 358 MBytes 300 Mbits/sec sender
                [ 15] 0.00-10.03 sec 357 MBytes 299 Mbits/sec receiver
                [ 17] 0.00-10.00 sec 121 MBytes 102 Mbits/sec sender
                [ 17] 0.00-10.03 sec 121 MBytes 101 Mbits/sec receiver
                [ 19] 0.00-10.00 sec 36.2 MBytes 30.4 Mbits/sec sender
                [ 19] 0.00-10.03 sec 36.0 MBytes 30.1 Mbits/sec receiver
                [ 21] 0.00-10.00 sec 41.4 MBytes 34.7 Mbits/sec sender
                [ 21] 0.00-10.03 sec 40.8 MBytes 34.1 Mbits/sec receiver
                [ 23] 0.00-10.00 sec 51.7 MBytes 43.3 Mbits/sec sender
                [ 23] 0.00-10.03 sec 49.8 MBytes 41.6 Mbits/sec receiver
                [SUM] 0.00-10.00 sec 1.01 GBytes 867 Mbits/sec sender
                [SUM] 0.00-10.03 sec 1.00 GBytes 860 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 112 MBytes 93.7 Mbits/sec sender
                [ 5] 0.00-10.01 sec 111 MBytes 93.0 Mbits/sec receiver
                [ 7] 0.00-10.00 sec 2.25 MBytes 1.89 Mbits/sec sender
                [ 7] 0.00-10.01 sec 382 KBytes 313 Kbits/sec receiver
                [ 9] 0.00-10.00 sec 129 MBytes 108 Mbits/sec sender
                [ 9] 0.00-10.01 sec 129 MBytes 108 Mbits/sec receiver
                [ 11] 0.00-10.00 sec 2.82 MBytes 2.37 Mbits/sec sender
                [ 11] 0.00-10.01 sec 1.24 MBytes 1.04 Mbits/sec receiver
                [ 13] 0.00-10.00 sec 160 MBytes 134 Mbits/sec sender
                [ 13] 0.00-10.01 sec 158 MBytes 133 Mbits/sec receiver
                [ 15] 0.00-10.00 sec 3.25 MBytes 2.73 Mbits/sec sender
                [ 15] 0.00-10.01 sec 1.25 MBytes 1.05 Mbits/sec receiver
                [ 17] 0.00-10.00 sec 236 MBytes 198 Mbits/sec sender
                [ 17] 0.00-10.01 sec 235 MBytes 197 Mbits/sec receiver
                [ 19] 0.00-10.00 sec 240 MBytes 202 Mbits/sec sender
                [ 19] 0.00-10.01 sec 239 MBytes 201 Mbits/sec receiver
                [ 21] 0.00-10.00 sec 231 MBytes 194 Mbits/sec sender
                [ 21] 0.00-10.01 sec 230 MBytes 192 Mbits/sec receiver
                [ 23] 0.00-10.00 sec 2.22 MBytes 1.86 Mbits/sec sender
                [ 23] 0.00-10.01 sec 349 KBytes 286 Kbits/sec receiver
                [SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec sender
                [SUM] 0.00-10.01 sec 1.08 GBytes 926 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 110 MBytes 92.3 Mbits/sec sender
                [ 5] 0.00-10.01 sec 109 MBytes 91.6 Mbits/sec receiver
                [ 7] 0.00-10.00 sec 111 MBytes 93.4 Mbits/sec sender
                [ 7] 0.00-10.01 sec 111 MBytes 93.1 Mbits/sec receiver
                [ 9] 0.00-10.00 sec 113 MBytes 94.8 Mbits/sec sender
                [ 9] 0.00-10.01 sec 113 MBytes 94.5 Mbits/sec receiver
                [ 11] 0.00-10.00 sec 109 MBytes 91.7 Mbits/sec sender
                [ 11] 0.00-10.01 sec 109 MBytes 91.1 Mbits/sec receiver
                [ 13] 0.00-10.00 sec 113 MBytes 95.0 Mbits/sec sender
                [ 13] 0.00-10.01 sec 113 MBytes 94.6 Mbits/sec receiver
                [ 15] 0.00-10.00 sec 113 MBytes 95.0 Mbits/sec sender
                [ 15] 0.00-10.01 sec 113 MBytes 94.7 Mbits/sec receiver
                [ 17] 0.00-10.00 sec 113 MBytes 94.6 Mbits/sec sender
                [ 17] 0.00-10.01 sec 112 MBytes 94.3 Mbits/sec receiver
                [ 19] 0.00-10.00 sec 113 MBytes 95.1 Mbits/sec sender
                [ 19] 0.00-10.01 sec 113 MBytes 94.8 Mbits/sec receiver
                [ 21] 0.00-10.00 sec 112 MBytes 93.8 Mbits/sec sender
                [ 21] 0.00-10.01 sec 111 MBytes 93.1 Mbits/sec receiver
                [ 23] 0.00-10.00 sec 112 MBytes 93.7 Mbits/sec sender
                [ 23] 0.00-10.01 sec 111 MBytes 93.4 Mbits/sec receiver
                [SUM] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec sender
                [SUM] 0.00-10.01 sec 1.09 GBytes 935 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 113 MBytes 94.6 Mbits/sec sender
                [ 5] 0.00-10.01 sec 112 MBytes 94.1 Mbits/sec receiver
                [ 7] 0.00-10.00 sec 96.3 MBytes 80.8 Mbits/sec sender
                [ 7] 0.00-10.01 sec 95.5 MBytes 80.0 Mbits/sec receiver
                [ 9] 0.00-10.00 sec 115 MBytes 96.5 Mbits/sec sender
                [ 9] 0.00-10.01 sec 115 MBytes 96.2 Mbits/sec receiver
                [ 11] 0.00-10.00 sec 126 MBytes 106 Mbits/sec sender
                [ 11] 0.00-10.01 sec 126 MBytes 105 Mbits/sec receiver
                [ 13] 0.00-10.00 sec 116 MBytes 97.0 Mbits/sec sender
                [ 13] 0.00-10.01 sec 115 MBytes 96.4 Mbits/sec receiver
                [ 15] 0.00-10.00 sec 93.7 MBytes 78.6 Mbits/sec sender
                [ 15] 0.00-10.01 sec 93.0 MBytes 77.9 Mbits/sec receiver
                [ 17] 0.00-10.00 sec 110 MBytes 92.5 Mbits/sec sender
                [ 17] 0.00-10.01 sec 109 MBytes 91.8 Mbits/sec receiver
                [ 19] 0.00-10.00 sec 105 MBytes 87.8 Mbits/sec sender
                [ 19] 0.00-10.01 sec 104 MBytes 87.2 Mbits/sec receiver
                [ 21] 0.00-10.00 sec 127 MBytes 107 Mbits/sec sender
                [ 21] 0.00-10.01 sec 127 MBytes 106 Mbits/sec receiver
                [ 23] 0.00-10.00 sec 116 MBytes 97.6 Mbits/sec sender
                [ 23] 0.00-10.01 sec 116 MBytes 97.1 Mbits/sec receiver
                [SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec sender
                [SUM] 0.00-10.01 sec 1.09 GBytes 932 Mbits/sec receiver

              OpenVPN (P2P PSK L2L) AES128-CBC SHA1

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 108 MBytes 90.4 Mbits/sec sender
                [ 5] 0.00-10.01 sec 108 MBytes 90.2 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 102 MBytes 85.9 Mbits/sec sender
                [ 5] 0.00-10.01 sec 102 MBytes 85.6 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 94.1 MBytes 78.9 Mbits/sec sender
                [ 5] 0.00-10.02 sec 93.9 MBytes 78.6 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 88.6 MBytes 74.4 Mbits/sec sender
                [ 5] 0.00-10.02 sec 88.4 MBytes 74.0 Mbits/sec receiver

              OpenVPN (P2P PSK L2L) AES256-CBC SHA256

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 89.2 MBytes 74.8 Mbits/sec sender
                [ 5] 0.00-10.02 sec 89.0 MBytes 74.6 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 85.6 MBytes 71.8 Mbits/sec sender
                [ 5] 0.00-10.02 sec 85.4 MBytes 71.5 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 78.1 MBytes 65.5 Mbits/sec sender
                [ 5] 0.00-10.02 sec 78.0 MBytes 65.3 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 71.5 MBytes 60.0 Mbits/sec sender
                [ 5] 0.00-10.02 sec 71.4 MBytes 59.7 Mbits/sec receiver

              OpenVPN (P2P TLS L2L) AES128-GCM

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 133 MBytes 112 Mbits/sec sender
                [ 5] 0.00-10.00 sec 133 MBytes 112 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 139 MBytes 116 Mbits/sec sender
                [ 5] 0.00-10.00 sec 139 MBytes 116 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 115 MBytes 96.6 Mbits/sec sender
                [ 5] 0.00-10.01 sec 115 MBytes 96.2 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.01 sec 122 MBytes 102 Mbits/sec sender
                [ 5] 0.00-10.01 sec 122 MBytes 102 Mbits/sec receiver

              OpenVPN (P2P TLS L2L) AES256-GCM

              • pfSense 2.4.5 (default)
                [ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec sender
                [ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 120 MBytes 101 Mbits/sec sender
                [ 5] 0.00-10.00 sec 120 MBytes 100 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 101 MBytes 84.5 Mbits/sec sender
                [ 5] 0.00-10.00 sec 101 MBytes 84.3 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 107 MBytes 89.5 Mbits/sec sender
                [ 5] 0.00-10.01 sec 107 MBytes 89.4 Mbits/sec receiver

              IPSEC VPN L2L AES128-CBC SHA1

              Use asynchronous mode to parallelize multiple cryptography jobs = OFF

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 85.0 MBytes 71.3 Mbits/sec sender
                [ 5] 0.00-10.14 sec 84.2 MBytes 69.6 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 80.6 MBytes 67.6 Mbits/sec sender
                [ 5] 0.00-10.00 sec 78.6 MBytes 65.9 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 77.0 MBytes 64.6 Mbits/sec sender
                [ 5] 0.00-10.00 sec 75.3 MBytes 63.2 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 79.8 MBytes 67.0 Mbits/sec sender
                [ 5] 0.00-10.00 sec 77.9 MBytes 65.3 Mbits/sec receiver

              Use asynchronous mode to parallelize multiple cryptography jobs = ON

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 198 MBytes 166 Mbits/sec sender
                [ 5] 0.00-10.05 sec 198 MBytes 166 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.01 sec 193 MBytes 162 Mbits/sec sender
                [ 5] 0.00-10.03 sec 193 MBytes 161 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 65.2 MBytes 54.7 Mbits/sec sender
                [ 5] 0.00-10.15 sec 65.2 MBytes 53.9 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 62.5 MBytes 52.4 Mbits/sec sender
                [ 5] 0.00-10.14 sec 62.5 MBytes 51.7 Mbits/sec receiver

              IPSEC VPN L2L AES256-CBC SHA256

              Use asynchronous mode to parallelize multiple cryptography jobs = OFF

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 77.9 MBytes 65.3 Mbits/sec sender
                [ 5] 0.00-10.08 sec 76.5 MBytes 63.7 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 73.8 MBytes 61.9 Mbits/sec sender
                [ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 70.8 MBytes 59.4 Mbits/sec sender
                [ 5] 0.00-10.00 sec 68.8 MBytes 57.7 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 74.6 MBytes 62.6 Mbits/sec sender
                [ 5] 0.00-10.00 sec 73.4 MBytes 61.5 Mbits/sec receiver

              Use asynchronous mode to parallelize multiple cryptography jobs = ON

              • pfSense 2.4.5 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.01 sec 181 MBytes 151 Mbits/sec sender
                [ 5] 0.00-10.05 sec 181 MBytes 151 Mbits/sec receiver

              • pfSense 2.4.5 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 182 MBytes 152 Mbits/sec sender
                [ 5] 0.00-10.09 sec 182 MBytes 151 Mbits/sec receiver

              • pfSense 2.5.0 (default)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.00 sec 55.9 MBytes 46.9 Mbits/sec sender
                [ 5] 0.00-10.24 sec 55.9 MBytes 45.8 Mbits/sec receiver

              • pfSense 2.5.0 (modified)
                [ ID] Interval Transfer Bitrate
                [ 5] 0.00-10.01 sec 55.7 MBytes 46.7 Mbits/sec sender
                [ 5] 0.00-10.33 sec 55.7 MBytes 45.3 Mbits/sec receiver

              Conclusions

              • Switching from 2.4.5 to 2.5.0 (on APU2 platform) leads to performances loss about 10%-25% in almost all tests done.

              • The heavy loss on IPSEC VPN with the option "Use asynchronous mode to parallelize multiple cryptography jobs"=enabled definitely needs some investigations.

              I would be curious to know if the same situation occurs on other hardware thus I hope that someone will post some results.

              1 Reply Last reply Reply Quote 3
              • First post
                Last post
              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.