Performance Issue - Virtualized 2.3 under KVM

[randyruiz]

Summary
I am getting 85Mbits max from my VM of pfsense 2.3. When i iperf from the host DOM 0 console I get 970Mbits so the issue has to be between the host and the VM and not the host and network. I have both interfaces (WAN/LAN) on the guest configured as VIRTIO/MACVTAP devices.  Any specific tuning advice around this configuration?

Hardware Setup
Intel C2578 SOC chip
SUPERMICRO MBD-A1SRi-2758F-O
16 GB RAM (for host)

Software Setup
Centos 7
KVM
Latest cut of pfsense 2.3

VM config
4 cores
4 GB of RAM
VIRTIO devices for network and storage
MACVTAP out of primary interface on host

pfsense config
MBUFF = 164mb
Disabled all off loading (all of them) in System/Advanced/Networking.

I have attached a systems activity screenshot of while I am iperfing and also a services screenshot to show you what I am running.


[fohdeesha]

This is most likely the issue people run into with non-checksummed traffic generated from other vm's/hosts in Xen (and KVM), as PF/freebsd drops this traffic and it slows things way down - https://forum.pfsense.org/index.php?topic=109751.msg611201#msg611201

EDIT: after poking around in the virtualization subforum, this problem is identical on KVM and is most likely what you're seeing. Worth noting the offload settings within pfsense don't affect this and can actually make it worse once you have the proper workaround set on the hypervisor, so I would suggest putting those back to default and rebooting the box.

Then you're going to have to find out the equivalent command to disable checksum offload for the pfsense virtio interfaces on the hypervisor side as we do in xen (eg "ethtool -K eth0 tx off" ) for each pfsense virtual interface on the hypervisor.

Looks like for KVM, on the host system (run for each pfsense interface):

ethtool -K vif123.3 tx off

(where virt123.3 is the virtual interface used by the pfsense VM. you'll need to stick this in a startup script eventually as ethtool settings don't pertain across reboots. centos/kvm might have vif scripts already for this that you need to edit)

Some googling shows a few articles stating that checksum needs to be turned off on the hosts real interface system wide to overcome this issue, so if all else fails and the above still doesn't work, on the host system run "ethtool -K eth0 tx off" (replacing eth0 with the hosts real network adapters) and see if you get your speed back. Hopefully it only needs to be done for the virtual interfaces on the host though.

Please report back your findings as there are a lot of older threads in the virtualization forum with this issue on KVM that never came back with a clean solution and instructions like we have for Xen and I'm sure others would be insterested to know if the above works  :)

[randyruiz]

Ok so I made the changes on the interfaces at the host level here is the output I got from the ehttool command

[root@vm01 ~]# ethtool -K enp0s20f2 tx off
Actual changes:
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ipv6: off
tx-checksum-sctp: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp6-segmentation: off [requested on]

Transfer rate is up to 150mb from 85mb before the change. Still no where near the 980Mb I get from the host adapter off of the DOM0 console.

[randyruiz]

To add a further data point. I have a centos 7 vm spun up on the same host using virtio drivers going out of the same MACVTAP interface. This centos vm is giving me 970Mb as measured by IPERF in throughput. So this points further to freebsd and pfsense as where the problem lies.

[fohdeesha]

@randyruiz:

To add a further data point. I have a centos 7 vm spun up on the same host using virtio drivers going out of the same MACVTAP interface. This centos vm is giving me 970Mb as measured by IPERF in throughput. So this points further to freebsd and pfsense as where the problem lies.

you might try changing rx offloading off as well (so rx off) and see if that gets the rest back, although usually only tx off is required. But yeah Freebsd is one of the outliers right now that drops these unchecksummed packets, most other OS'ses handle them without issue

Of course make sure you're doing the tx/rx off for both the pfsense wan and lan virtio adapters not just one. If that's still not doing it I'd monitor cpu usage inside and outside of the VM and see if you're not hitting a bottleneck elsewhere

Sorry I can't be of more help as I've never used KVM before. You might get more KVM specific help if you ask in the virtualization subforum