Virtualbox pfSense VM - download/upload speed

pfaf

Dear all,

first of all I would like to send many thanks to the pfSense team and NetGate for this wonderful software firewall.
We have been using it and supporting it for more than 5 years
and we still believe it has the potential to advance even further.

We use it mainly as a VM in Virtualbox and it has proved stable for our small customer installations.

Now, in Virtualbox v6.1.16 on Debian v10.7 we are testing

pfSense v2.5.2
pfSense v2.6.0
pfSense v2.7.0-DEVELOPMENT-amd64-20220610-0600
as a VM firewall.

On the virtual firewall device we have attached two network cards, one for the outside interface and one for the inside interface.
Outside interface has two ISP providers, one running at download/upload rate 100Mbps/100Mbps and the other running at 100Mbps/10Mbps.

The two ISP routers are on different VLANs which we use via a L2 managed switch (eno4, vlans are tagged 31 and 32).
The inside interface uses a different ethernet port on the host (eno1).

We have seen that using VirtIO emulation and netdrivers in versions v2.5.2 and v2.6.0 does not work well,
it must be that the netdrivers in FreeBSD (vtnet) may not be right.

Best option for Virtualbox netcard drivers is to use "Intel Pro/1000 MT Server (82545EM)" which the FreeBSD sees via the (em) driver.

Using em drivers in versions v2.5.2 and v2.6.0 we managed to get download speeds of about 60Mbps, and to achieve this we also had to:

A) on the host

disabling the tso, gso, gro on the host with commands like below:
ethtool -K eno1 tso off gso off gro off (host inside interface)
ethtool -K eno4 tso off gso off gro off (host outside interface)
increase host netcard ring parameters
ethtool -G eno1 rx 2047
ethtool -G eno4 rx 2047

B) inside the VM pfSense

Via the web interface: System -> Advanced -> Networking
Section: Network Interfaces
Enable (Tick) Disable hardware checksum offload
Leave enabled (ticked) Disable hardware TCP segmentation offload
Leave enabled (ticked) Disable hardware large receive offload
Disable (Untick) Enable the ALTQ support for hn NICs.
disable tso inside the pfSense
(System -> Advanced -> Tunables - net.inet.tcp.tso) changed from 1 to 0.
disable flow control for the two network cards
(System -> Advanced -> Tunables ) created the two tunables below:
Tunable Name Description Value
dev.em.0.fc disable flow control 0
dev.em.1.fc disable flow control 0
Disable tso also during boot, set some more recommender boot parameters:
vi /boot/loader.conf.local
net.inet.tcp.tso="0"
kern.ipc.nmbclusters="1000000"
kern.hz=100

BUT on pfSense v2.7.0-DEVELOPMENT-amd64-20220610-0600 we did not have to do any of the above (B) configurations.
Actually, we setup a new VM, and tested the download/upload speeds. We got 85Mbps/84Mbps on the one provider, and 95Mbps/10Mbps on the second.

Then we reverted all the above (A) configurations on the host, and the download/upload speeds remained the same.

The questions are:

Does the development version has any disabled features and thus works faster?
Why the recommended (A) and (B) configurations do not improve the download/upload speed in pfSense v2.7.0-DEVELOPMENT-amd64-20220610-0600 version?
Can we improve speed even more using a VM?
Another version of pfSense, pfSense 2.4.4-p1 we have used in Virtualbox v5.0.40 on Debian 7.11 with "Intel Pro/1000 MT Desktop (82540EM)", manages to achive download speeds of 97Mbps/10Mbps on the second ISP line. What has happened to v2.5.2, v2.6.0 and v2.7.0 versions of pfSense and can achieve lower download speeds?

Thank you in advance for any help you can provide.

Kind regards,

Panos Fafakos.

NollipfSense

@pfaf How are you conducting the speed test, pfSense VM as client?

pfaf

@nollipfsense
The pfSense VM is running on a HP DL 380 G10 server, eno1 on the server is connected to LAN, eno4 is connected to the ISPs VLANs using a managed switch.
pfSense VM sees eno4 as outside WAN interface, on which we have created em0.31 and em0.32 802.1q VLANs connected using a managed switch.
pfSense VM sees eno1 as inside LAN interface em1.
Speedtests are run using another PC on the LAN using
https://www.speedtest.net/

NollipfSense

@pfaf Okay, good...just for fun could you try http://openspeedtest.com/

NollipfSense

@pfaf Okay, good...just for fun could you try
http://openspeedtest.com/
or
https://openspeedtest.com/?ref=SSL-OST-Results

pfaf

@nollipfsense ok, so here it is:

pfSense 2.7.0-DEVELOPMENT (amd64): built on Fri Jun 10 06:13:51 UTC 2022, FreeBSD 12.3-STABLE
speedtest.net: ping 2ms, download 85.27Mbps, upload 89.07Mbps
openspeedtest.com: ping 50ms, download 85.15Mbps, upload 66.67Mbps

pfSense 2.6.0-RELEASE (amd64): built on Mon Jan 31 19:57:53 UTC 2022 FreeBSD 12.3-STABLE
speedtest.net: ping 15ms, download 4.16Mbps, upload 9.92Mbps
openspeedtest.com: ping 56ms, download 0.94Mbps, upload 10.31Mbps

pfSense 2.5.2-RELEASE (amd64): built on Fri Jul 02 15:33:00 EDT 2021, FreeBSD 12.2-STABLE
speedtest.net: ping 2ms, download 59.47Mbps, upload 61.73Mbps
openspeedtest.com: ping 51ms, download 57.84Mbps, upload 62.87Mbps

Notes:

All virtualbox card emulations are set to "Intel Pro/1000 MT Server (82545EM)".
For pfSense 2.7.0-DEVELOPMENT (amd64) settings for:
a) Disable hardware checksum offload = unticked
b) Disable hardware TCP segmentation offload = ticked
c) Disable hardware large receive offload = ticked
d) Enable the ALTQ support for hn NICs = ticked
e) net.inet.tcp.tso = 1 (Enabled)
where left to their defaults!
For pfSense 2.6.0 & 2.5.2 settings where changed from the defaults like noted below:
a) Disable hardware checksum offload = ticked
b) Disable hardware TCP segmentation offload = ticked
c) Disable hardware large receive offload = ticked
d) Enable the ALTQ support for hn NICs = unticked
e) net.inet.tcp.tso = 0 (Disabled)

Host Network card settings where not changed during the tests,
eno1 is the inside lan card of the server,
eno4 is the outside lan card of the server, where the vlans are set:

# ethtool -g eno1
Ring parameters for eno1:
Pre-set maximums:
RX:             2047
RX Mini:        0
RX Jumbo:       0
TX:             511
Current hardware settings:
RX:             200
RX Mini:        0
RX Jumbo:       0
TX:             511

# ethtool -k eno1
Features for eno1:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]


# ethtool -g eno4
Ring parameters for eno4:
Pre-set maximums:
RX:             2047
RX Mini:        0
RX Jumbo:       0
TX:             511
Current hardware settings:
RX:             200
RX Mini:        0
RX Jumbo:       0
TX:             511

# ethtool -k eno4
Features for eno4:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

Why is pfSense v2.6.0 so crippled ?
Why settings that should impove v2.7.0 dev do not do anything?

NollipfSense

@pfaf Interesting, indeed.

jimp

There aren't any features disabled in 2.7.0, but there are numerous improvements throughout that may have contributed to it working better.