Poor Network performance on 2.3 with KVM

[randyruiz]

Summary
I am getting 85Mbits max from my VM of pfsense 2.3. When i iperf from the host DOM 0 console I get 970Mbits so the issue has to be between the host and the VM and not the host and network. I have both interfaces (WAN/LAN) on the guest configured as VIRTIO/MACVTAP devices.  Any specific tuning advice around this configuration?

Hardware Setup
Intel C2578 SOC chip
SUPERMICRO MBD-A1SRi-2758F-O
16 GB RAM (for host)

Software Setup
Centos 7
KVM
Latest cut of pfsense 2.3

VM config
4 cores
4 GB of RAM
VIRTIO devices for network and storage
MACVTAP out of primary interface on host

pfsense config
MBUFF = 164mb
Disabled all off loading (all of them) in System/Advanced/Networking.

I have attached a systems activity screenshot of while I am iperfing and also a services screenshot to show you what I am running.

[fohdeesha]

I helped as much as I could in his original topic here in the general section but the general tx offload of the known xen/kvm problem didn't fuilly solve it ( it did increase bandwidth, but still not nearly enough) https://forum.pfsense.org/index.php?topic=110201.0

hopefully someone experienced with KVM can help

[webdawg]

Heh That's Great,

I am about to install pfSense on a proxmox system as soon as I get done testing the hardware.  One of the reason's I am moving to KVM and virtio is because with the Xen HVM drivers (pfsense disable pv xen stuff in loader.conf) I can only pull 100 mbits max, and about 60 mbits in the other direction.

I hope I do not have to move to ESX.  I will report results here in a bit.

How are you testing may I ask?  SCP?

[webdawg]

https://forum.pfsense.org/index.php?topic=98213.msg548124#msg548124

https://devcentral.f5.com/articles/configuring-the-kvm-hypervisor-to-run-a-linerate-guest

[randyruiz]

webdawg
To test I have been using a combo of IPERF3 and SCP.

[randyruiz]

webdawg,
I checked your links. given that I only have 4 vcpus on the vm there is no change to the default settings.

[webdawg]

Are you sure the default is 4, I am reading some other docs that say otherwise.

[randyruiz]

The link you sent me defines the optimal number of queues to be 1 for up to 4 vcpu's.  I have 4 cores allocated to the VM. Do you have other docs that say otherwise?

[webdawg]

Who cares what that guy says.  I would set that stuff manually.

I just installed Proxmox, set 4 queues to the VM network interface.

I then put 1 cpu and 4 cores to it.

client to freebsd 10.3
[  3]  0.0-10.0 sec  943 MBytes  791 Mbits/sec

freebsd 10.3 to client
[  3]  0.0-10.0 sec  769 MBytes  644 Mbits/sec

This is on an i3 laptop with 4gb memory.  Also who knows how good the network equipment is between here and there.

I do not know much about proxmox at this point but the only way I could get to to boot virtio with 4 queues was to put 4 cores into the VM itself.

randyrulz it really seems like you should be able to make it run faster.

I am about to install pfsense next.  This was just a freebsd 10.3 test which pfsense 2.3 runs on at this point.