Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Virtualbox pfSense VM - download/upload speed

    Scheduled Pinned Locked Moved CE 2.7.0 Development Snapshots (Retired)
    8 Posts 3 Posters 911 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      pfaf
      last edited by

      Dear all,

      first of all I would like to send many thanks to the pfSense team and NetGate for this wonderful software firewall.
      We have been using it and supporting it for more than 5 years
      and we still believe it has the potential to advance even further.

      We use it mainly as a VM in Virtualbox and it has proved stable for our small customer installations.

      Now, in Virtualbox v6.1.16 on Debian v10.7 we are testing

      • pfSense v2.5.2
      • pfSense v2.6.0
      • pfSense v2.7.0-DEVELOPMENT-amd64-20220610-0600
        as a VM firewall.

      On the virtual firewall device we have attached two network cards, one for the outside interface and one for the inside interface.
      Outside interface has two ISP providers, one running at download/upload rate 100Mbps/100Mbps and the other running at 100Mbps/10Mbps.

      The two ISP routers are on different VLANs which we use via a L2 managed switch (eno4, vlans are tagged 31 and 32).
      The inside interface uses a different ethernet port on the host (eno1).

      We have seen that using VirtIO emulation and netdrivers in versions v2.5.2 and v2.6.0 does not work well,
      it must be that the netdrivers in FreeBSD (vtnet) may not be right.

      Best option for Virtualbox netcard drivers is to use "Intel Pro/1000 MT Server (82545EM)" which the FreeBSD sees via the (em) driver.

      Using em drivers in versions v2.5.2 and v2.6.0 we managed to get download speeds of about 60Mbps, and to achieve this we also had to:

      A) on the host

      • disabling the tso, gso, gro on the host with commands like below:
        ethtool -K eno1 tso off gso off gro off (host inside interface)
        ethtool -K eno4 tso off gso off gro off (host outside interface)

      • increase host netcard ring parameters
        ethtool -G eno1 rx 2047
        ethtool -G eno4 rx 2047

      B) inside the VM pfSense

      • Via the web interface: System -> Advanced -> Networking
        Section: Network Interfaces
        Enable (Tick) Disable hardware checksum offload
        Leave enabled (ticked) Disable hardware TCP segmentation offload
        Leave enabled (ticked) Disable hardware large receive offload
        Disable (Untick) Enable the ALTQ support for hn NICs.

      • disable tso inside the pfSense
        (System -> Advanced -> Tunables - net.inet.tcp.tso) changed from 1 to 0.

      • disable flow control for the two network cards
        (System -> Advanced -> Tunables ) created the two tunables below:
        Tunable Name Description Value
        dev.em.0.fc disable flow control 0
        dev.em.1.fc disable flow control 0

      • Disable tso also during boot, set some more recommender boot parameters:
        vi /boot/loader.conf.local
        net.inet.tcp.tso="0"
        kern.ipc.nmbclusters="1000000"
        kern.hz=100

      BUT on pfSense v2.7.0-DEVELOPMENT-amd64-20220610-0600 we did not have to do any of the above (B) configurations.
      Actually, we setup a new VM, and tested the download/upload speeds. We got 85Mbps/84Mbps on the one provider, and 95Mbps/10Mbps on the second.

      Then we reverted all the above (A) configurations on the host, and the download/upload speeds remained the same.

      The questions are:

      • Does the development version has any disabled features and thus works faster?
      • Why the recommended (A) and (B) configurations do not improve the download/upload speed in pfSense v2.7.0-DEVELOPMENT-amd64-20220610-0600 version?
        Can we improve speed even more using a VM?
      • Another version of pfSense, pfSense 2.4.4-p1 we have used in Virtualbox v5.0.40 on Debian 7.11 with "Intel Pro/1000 MT Desktop (82540EM)", manages to achive download speeds of 97Mbps/10Mbps on the second ISP line. What has happened to v2.5.2, v2.6.0 and v2.7.0 versions of pfSense and can achieve lower download speeds?

      Thank you in advance for any help you can provide.

      Kind regards,

      Panos Fafakos.

      NollipfSenseN 1 Reply Last reply Reply Quote 0
      • NollipfSenseN
        NollipfSense @pfaf
        last edited by

        @pfaf How are you conducting the speed test, pfSense VM as client?

        pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
        pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

        P 1 Reply Last reply Reply Quote 0
        • P
          pfaf @NollipfSense
          last edited by

          @nollipfsense
          The pfSense VM is running on a HP DL 380 G10 server, eno1 on the server is connected to LAN, eno4 is connected to the ISPs VLANs using a managed switch.
          pfSense VM sees eno4 as outside WAN interface, on which we have created em0.31 and em0.32 802.1q VLANs connected using a managed switch.
          pfSense VM sees eno1 as inside LAN interface em1.
          Speedtests are run using another PC on the LAN using
          https://www.speedtest.net/

          NollipfSenseN 2 Replies Last reply Reply Quote 0
          • NollipfSenseN
            NollipfSense @pfaf
            last edited by

            @pfaf Okay, good...just for fun could you try http://openspeedtest.com/

            pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
            pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

            1 Reply Last reply Reply Quote 0
            • NollipfSenseN
              NollipfSense @pfaf
              last edited by NollipfSense

              @pfaf Okay, good...just for fun could you try
              http://openspeedtest.com/
              or
              https://openspeedtest.com/?ref=SSL-OST-Results

              pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
              pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

              P 1 Reply Last reply Reply Quote 0
              • P
                pfaf @NollipfSense
                last edited by pfaf

                @nollipfsense ok, so here it is:

                pfSense 2.7.0-DEVELOPMENT (amd64): built on Fri Jun 10 06:13:51 UTC 2022, FreeBSD 12.3-STABLE
                speedtest.net: ping 2ms, download 85.27Mbps, upload 89.07Mbps
                openspeedtest.com: ping 50ms, download 85.15Mbps, upload 66.67Mbps

                pfSense 2.6.0-RELEASE (amd64): built on Mon Jan 31 19:57:53 UTC 2022 FreeBSD 12.3-STABLE
                speedtest.net: ping 15ms, download 4.16Mbps, upload 9.92Mbps
                openspeedtest.com: ping 56ms, download 0.94Mbps, upload 10.31Mbps

                pfSense 2.5.2-RELEASE (amd64): built on Fri Jul 02 15:33:00 EDT 2021, FreeBSD 12.2-STABLE
                speedtest.net: ping 2ms, download 59.47Mbps, upload 61.73Mbps
                openspeedtest.com: ping 51ms, download 57.84Mbps, upload 62.87Mbps

                Notes:

                • All virtualbox card emulations are set to "Intel Pro/1000 MT Server (82545EM)".

                • For pfSense 2.7.0-DEVELOPMENT (amd64) settings for:
                  a) Disable hardware checksum offload = unticked
                  b) Disable hardware TCP segmentation offload = ticked
                  c) Disable hardware large receive offload = ticked
                  d) Enable the ALTQ support for hn NICs = ticked
                  e) net.inet.tcp.tso = 1 (Enabled)
                  where left to their defaults!

                • For pfSense 2.6.0 & 2.5.2 settings where changed from the defaults like noted below:
                  a) Disable hardware checksum offload = ticked
                  b) Disable hardware TCP segmentation offload = ticked
                  c) Disable hardware large receive offload = ticked
                  d) Enable the ALTQ support for hn NICs = unticked
                  e) net.inet.tcp.tso = 0 (Disabled)

                Host Network card settings where not changed during the tests,
                eno1 is the inside lan card of the server,
                eno4 is the outside lan card of the server, where the vlans are set:

                # ethtool -g eno1
Ring parameters for eno1:
Pre-set maximums:
RX:             2047
RX Mini:        0
RX Jumbo:       0
TX:             511
Current hardware settings:
RX:             200
RX Mini:        0
RX Jumbo:       0
TX:             511

# ethtool -k eno1
Features for eno1:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]


# ethtool -g eno4
Ring parameters for eno4:
Pre-set maximums:
RX:             2047
RX Mini:        0
RX Jumbo:       0
TX:             511
Current hardware settings:
RX:             200
RX Mini:        0
RX Jumbo:       0
TX:             511

# ethtool -k eno4
Features for eno4:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

                Why is pfSense v2.6.0 so crippled ?
                Why settings that should impove v2.7.0 dev do not do anything?

                NollipfSenseN 1 Reply Last reply Reply Quote 0
                • NollipfSenseN
                  NollipfSense @pfaf
                  last edited by

                  @pfaf Interesting, indeed.

                  pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                  pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    There aren't any features disabled in 2.7.0, but there are numerous improvements throughout that may have contributed to it working better.

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.