PFSense 2.1 & KVM VirtIO

Are

Hi, thanks for the info i have been looking for this and it worked right away.

However, it still is very cpu hungry compared to the ubuntu/shorewall
firewall running in the same environment. ( proxmox 2.1)

instead of eating 100% (2.01) cpu nat'ing 100/100 internet pipe at full speed it use about 70%.

Idle is at 6% instead of 7-8% in pfsense 2.01

So its a step in the right direction but still to high.
Linux idle at 0.1 % and max at 10% doing the same nat.

Thanks

dhatz

Thanks for the info JR.

I'd like to add a quick note reminding everybody that you should use /boot/loader.conf.local instead of /boot/loader.conf for tuning, since /boot/loader.conf is over-written by pfsense's startup scripts.

jimp

Should probably turn this into a doc wiki article since it's a somewhat popular topic/faq. If there are no objections I can just copy the text into the wiki.

eri--

@Are:

Hi, thanks for the info i have been looking for this and it worked right away.

However, it still is very cpu hungry compared to the ubuntu/shorewall
firewall running in the same environment. ( proxmox 2.1)

instead of eating 100% (2.01) cpu nat'ing 100/100 internet pipe at full speed it use about 70%.

Idle is at 6% instead of 7-8% in pfsense 2.01

So its a step in the right direction but still to high.
Linux idle at 0.1 % and max at 10% doing the same nat.

Thanks

Not sure on comparison side/fairness.

Helpful would be to show a top -SH and what is eating your CPU.

JR

@Are:

…

However, it still is very cpu hungry compared to the ubuntu/shorewall
firewall running in the same environment. ( proxmox 2.1)

...

Nice to hear it worked for you!

Regarding performance, yeah, that's still an issue but for now I believe the "best" you can get. From what I hear/read, didn't tested it myself, FreeBSD 9 and VirtIO should perform much better. For now PFSense is using 8.3, so I guess it will take some more time before you can benefit from that upgrade.

@dhatz: Thanks for mentioning, I'll edit the original post.

jimp

Added to the wiki (with some edits) here: http://doc.pfsense.org/index.php/VirtIO_Driver_Support

Thanks!

toomeek

What about perfromance test results?
I hit ~250Mbit/s limit with standard Intel E1000 emulation, what about virtio?

thulle

I'm not able to get it running; I have to run at least one interface as non-virtio, forwarding packets between two virtio-interfaces works for icmp, but any tcp-session just freezes.

Seems to be this bug; http://www.freebsd.org/cgi/query-pr.cgi?pr=165059

Edit; oh, and I've tried 5 different snapshots between PFSense 2.1-BETA0 SnapShot-20120701 to 0713

host is;

ubuntu 12.04, with kernel 3.2.0-26-generic
qemu-kvm 1.0
libvirt 0.9.12 - (tried with 0.9.11 and 0.9.8 too, had to upgrade for it to work nicely with openvswitch)
openvswitch 1.6.90 - tried with regular bridges too

tomcos

Hey everyone, been virtualizing PfSense for a while now and havn't run into any problem I couldn't figure out until now. I followed the directions and I believe everything works fine except for when I input the

<model type="virtio">from <model type="e1000">into my xml file. After, when PfSense beings to boot, no interfaces are found, so it cannot start.
I'm running
2.0.1-RELEASE (amd64)
FreeBSD 8.1-RELEASE-p6
on
Ubuntu 12.04 x64
latest versions of all virtualization packages

here is a snapshot of my xml file if it helps

<domain type="kvm"><name>Router</name>
<uuid>e2968164-4c0c-bc9e-3406-4043ea694ff4</uuid>
<description>PFSENSE</description>
<memory>3145728</memory>
<currentmemory>3145728</currentmemory>
<vcpu>2</vcpu>
<os><type arch="x86_64" machine="pc-1.0">hvm</type></os>
<features><acpi><apic><pae></pae></apic></acpi></features>
<clock offset="utc"><on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices><emulator>/usr/bin/kvm</emulator>
<disk type="file" device="disk"><driver name="qemu" type="raw"><source file="/pfSense/pfsense">
<target dev="hda" bus="ide"><address type="drive" controller="0" bus="0" unit="0">

and my loader.conf.local

virtio_load="YES"
virtio_pci_load="YES"
if_vtnet_load="YES"
virtio_balloon_load="YES"

any suggestions would be lovely :)</address></memballoon> </address></model></video></graphics> </address></model></mac></interface> </address></model></mac></interface> </address></controller> </address></readonly></target></driver></disk> </address></target></driver></disk></devices></clock></domain></model></model>

athurdent

@tomcos:

2.0.1-RELEASE (amd64)
FreeBSD 8.1-RELEASE-p6

I might be wrong here but wasn't VIRTIO support first added in 2.1-beta?
They work fine on Ubuntu 12.04 x64 for me. Throughput is also very nice, around 800 Mbit with iperf if I remember correctly.

jimp

Correct, the virtio drivers are only in 2.1, they aren't in 2.0.x

tomcos

ah! now it all makes sense. Thank you for the reply :)

athurdent

Don't know if it's related to VirtIO, but when I use
-smp cores=2
pfSense crashes after a few minutes. Without the smp Parameter everything's OK. Anyone else noticing this?
I'm running Ubuntu 12.04 LTS x64.

Startup Parameters:

/usr/bin/kvm \
-no-fd-bootchk \
-k de \
-cpu host \
-smp cores=2 \
-m 512 \
-machine type=pc,accel=kvm \
-drive file=/data/vms/wan-dmz-pfsense-qcow2.img,if=virtio,cache=writeback \
-net nic,model=virtio,macaddr=DE:AD:BE:EF:A0:88,vlan=88 \
-net tap,vlan=88,ifname=tap88,script=/etc/qemu-ifup-br1,downscript=/etc/qemu-ifdown-br1 \
-net nic,model=virtio,macaddr=DE:AD:BE:EF:AF:FE,vlan=89 \
-net tap,vlan=89,ifname=tap89,script=/etc/qemu-ifup-br2,downscript=/etc/qemu-ifdown-br2 \
-daemonize

wb-munzinger

Works great here, even with 2 cores.
I had to enable "Disable hardware checksum offload" again. Without I had no connection at all.

lothan

After activated "VirtIO Memory Ballooning" still does not automatically granted more memory from the available range, to be increased manually. Options are: CURRENT(256m) \ MAX(2048m)

snap2.png_thumb

snap3.png_thumb

athurdent

@wb-munzinger:

Works great here, even with 2 cores.

I think I have found the reason for the crashes. As soon as I turn on the SNMP CPU check in Cacti (running on my monitor host), pfSense crashes after some time. It's the same on VMWare, I have submitted the crash info for the VMWare host yesterday. Using only one virtual core or turning off SNMP CPU checks fixes the problem for me.

Mattz

This worked for me, as the vtnet was shown but not on my new installs with the latest snapshots anymore.

I can load virtio @ the commandline but it won't load everyting during boot. My lines are in /boot/loader.conf.local

Nachtfalke

Thank you JR for this great tutorial.

I just installed for testing a pfsense 2.1BETA1 snapshot from 21.03.2013 on a ProxmoxVE 2.3 installation.
Loading HDD VirtIO drivers and NIC VirtIO drivers is working.

Not sure about ballooning. On proxmox I can type "info balloon" into the "monitor" and it always shows me the MAX mem of 1024MB.
The same on another VM with Windows 7 x64 and the ballooning drivers shows me other values which are below the maximum.

Are there any other way/methods to test if ballooning is working or not ? I am no linus/freebsd expert so perhaps someone can explain it more in detail if possible ;)

pfsense 2.1 "System activity" shows me this line:

    7 root     -16    -     0K    16K vtbslp   0:01  0.00% virtio_balloon

Nevertheless thank you very much for that tutorial!

invitu

I also have a big problem with incorrect cksum packets

configuration :
host : Centos 6.4 64 bit (2.6.32-358.6.2.el6.x86_64) + qemu-kvm-0.12.1.2
nic hardware : e1000e
lan : eth0
vlan : eth0.254
br0 : bridge to eth0
br254 : bridge to eth0.254

guest : pfsense 2.1RC0 64 bits snapshot (31/05/2013) with virtio or 2.1BETA0
LAN : br0 (virtio)
WAN : br254 (virtio)

another guest : for example fedora12
LAN : br0

from the host or the other guest, I can icmp or udp:53 to Internet but tcp:80 or udp:123 are stuck…

here is tcpdump extract for "ntpdate us.pool.ntp.org" :

226.76.50.123.dsl.dyn.mana.pf.44299 > 192.168.254.253.rockwell-csp2: Flags [.], cksum 0xe925 (correct), seq 97, ack 432, win 320, options [nop,nop,TS val 3629676 ecr 27931684], length 0
10:00:13.398297 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 76)
192.168.254.253.35791 > kapu.skafari.com.ntp: [bad udp cksum b920!] NTPv4, length 48
Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 3s, precision -6
Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
Reference Timestamp: 0.000000000
Originator Timestamp: 0.000000000
Receive Timestamp: 0.000000000
Transmit Timestamp: 3579105613.398010432 (2013/06/01 10:00:13)
Originator - Receive Timestamp: 0.000000000
Originator - Transmit Timestamp: 3579105613.398010432 (2013/06/01 10:00:13)
192.168.254.253.59909 > housetree.sugarlabs.org.ntp: [bad udp cksum 290e!] NTPv4, length 48
Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 3s, precision -6
Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
Reference Timestamp: 0.000000000
Originator Timestamp: 0.000000000
Receive Timestamp: 0.000000000
Transmit Timestamp: 3579105613.998007059 (2013/06/01 10:00:13)
Originator - Receive Timestamp: 0.000000000
Originator - Transmit Timestamp: 3579105613.998007059 (2013/06/01 10:00:13)

with (net2k_pci driver for br0 and virtio driver for br254) or (net2k_pci for both), everything is fine

I also had no problem with fedora14 64 bits + qemu-kvm-0.13.0 (exactly same hardware) as the host and pfsense 2.1BETA0 as the guest (I just upgraded my host from fedora 14 to centos 6.4…)

I installed a linux-based firewall distrib (ipfire.org) as a guest with same settings and I have no problem...

do you think the problem is with qemu-kvm-0.12.1.2 + freebsd virtio drivers ?

wallabybob

Suppose pfSense has hardware checksums enabled. A tcpdump on pfSense will show checksum errors because software doesn't compute the checksums because the hardware does it. But in a virtualised environment pfSense doesn't have access to real hardware. Perhaps the hypervisor doesn't correctly get the hardware to compute the checksums.

What is the pfSense setting related to hardware checksums? See System -> Advanced, click on Networking and scroll down to Network Interfaces, Hardware Checksum Offloading.