PFSense 2.1 & KVM VirtIO

JR

For those who run PFSense inside a KVM Virtual Machine, here are some easy steps to enable VirtIO for your PFSense VM.

Works-For-Me with:

PFSense 2.1-BETA0 SnapShot-20120603
Debian Testing - 20120603

qemu-kvm 1.0
libvirt 0.9.11
openvswitch 1.4.0
virt-manager 0.9.1

*** Step 1 - Create VirtIO Network Interfaces**

I'm using 4 interfaces that are connected to various virtual switches on the KVM host:

NIC1 - ISP1
NIC2 - ISP2
NIC3 - CARP
NIC4 - LAN

Before you can use a VirtIO device you need to load the VirtIO kernel modules, but before you can load the kernel modules you need to have at least one interface enabled during PFSense installation. Create all vm interfaces as VirtIO devices but leave just one default (realtek or intel), you're now able to finish the default install and reach the PFSense console menu.

*** Step 2 - Edit /boot/loader.conf.local**

Select < 8 Shell > from the PFSense console menu after the first boot with the single (default) interface. At the shell prompt edit the /boot/loader.conf.local file with vi and append the file with the VirtIO module settings.

vi /boot/loader.conf.local


virtio_load="YES"
virtio_pci_load="YES"
if_vtnet_load="YES"

If you only want to use VirtIO for your vm network interfaces you're finished now, just shutdown the vm, change the single default ethernet device used for installation to VirtIO and boot into PFSense. From here all VirtIO network interfaces should be visible and ready for final configuration ( interface name: vtnetX ).

*** Step 3 - Add VirtIO Memory Ballooning**

Memory ballooning allows you to have your guest dynamically change it's memory usage by evicting unused memory during runtime. Most useful for environments with large memory over-commit requirements.

vi /boot/loader.conf.local


virtio_balloon_load="YES"

*** Step 4 - Add VirtIO Block Device**

In addition to the network interfaces there also are VirtIO drivers for block devices (virtual harddisks). Just run the default installation with a virtual harddisk in IDE mode and change the /boot/loader.conf.local and /etc/fstab files afterwards.

vi /boot/loader.conf.local


virtio_blk_load="YES"

By default the PFSense installer creates a root and swap volume on IDE device /dev/ad0, before changing your virtual harddisk from IDE to VirtIO you need to edit the fstab file with the new VirtIO block device name ( /dev/vtbdX ).

vi /etc/fstab


/dev/vtbd0s1a     /          ufs          rw     1     1
/dev/vtbd0s1b     none     swap       sw     0     0

Shutdown the vm and change the virtual harddisk from IDE to VirtIO, boot and enjoy your PFSense KVM VirtIO Virtual Machine…

kldstat


#kldstat

Id Refs Address            Size     Name
 1   12 0xffffffff80100000 1540f20  kernel
 2    5 0xffffffff81641000 4ac0     virtio.ko
 3    1 0xffffffff81646000 5670     virtio_pci.ko
 4    1 0xffffffff8164c000 4d88     virtio_blk.ko
 5    1 0xffffffff81651000 2f80     virtio_balloon.ko
 6    1 0xffffffff81654000 b0e0     if_vtnet.ko

df


#df -h

Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/vtbd0s1a    4.4G    234M    3.8G     6%    /
devfs            1.0k    1.0k      0B   100%    /dev
/dev/md0         3.6M     46k    3.3M     1%    /var/run
devfs            1.0k    1.0k      0B   100%    /var/dhcpd/dev

swapinfo


#swapinfo

Device          1K-blocks     Used    Avail Capacity
/dev/vtbd0s1b      524288        0   524288     0%

Are

Hi, thanks for the info i have been looking for this and it worked right away.

However, it still is very cpu hungry compared to the ubuntu/shorewall
firewall running in the same environment. ( proxmox 2.1)

instead of eating 100% (2.01) cpu nat'ing 100/100 internet pipe at full speed it use about 70%.

Idle is at 6% instead of 7-8% in pfsense 2.01

So its a step in the right direction but still to high.
Linux idle at 0.1 % and max at 10% doing the same nat.

Thanks

dhatz

Thanks for the info JR.

I'd like to add a quick note reminding everybody that you should use /boot/loader.conf.local instead of /boot/loader.conf for tuning, since /boot/loader.conf is over-written by pfsense's startup scripts.

jimp

Should probably turn this into a doc wiki article since it's a somewhat popular topic/faq. If there are no objections I can just copy the text into the wiki.

eri--

@Are:

Hi, thanks for the info i have been looking for this and it worked right away.

However, it still is very cpu hungry compared to the ubuntu/shorewall
firewall running in the same environment. ( proxmox 2.1)

instead of eating 100% (2.01) cpu nat'ing 100/100 internet pipe at full speed it use about 70%.

Idle is at 6% instead of 7-8% in pfsense 2.01

So its a step in the right direction but still to high.
Linux idle at 0.1 % and max at 10% doing the same nat.

Thanks

Not sure on comparison side/fairness.

Helpful would be to show a top -SH and what is eating your CPU.

JR

@Are:

…

However, it still is very cpu hungry compared to the ubuntu/shorewall
firewall running in the same environment. ( proxmox 2.1)

...

Nice to hear it worked for you!

Regarding performance, yeah, that's still an issue but for now I believe the "best" you can get. From what I hear/read, didn't tested it myself, FreeBSD 9 and VirtIO should perform much better. For now PFSense is using 8.3, so I guess it will take some more time before you can benefit from that upgrade.

@dhatz: Thanks for mentioning, I'll edit the original post.

jimp

Added to the wiki (with some edits) here: http://doc.pfsense.org/index.php/VirtIO_Driver_Support

Thanks!

toomeek

What about perfromance test results?
I hit ~250Mbit/s limit with standard Intel E1000 emulation, what about virtio?

thulle

I'm not able to get it running; I have to run at least one interface as non-virtio, forwarding packets between two virtio-interfaces works for icmp, but any tcp-session just freezes.

Seems to be this bug; http://www.freebsd.org/cgi/query-pr.cgi?pr=165059

Edit; oh, and I've tried 5 different snapshots between PFSense 2.1-BETA0 SnapShot-20120701 to 0713

host is;

ubuntu 12.04, with kernel 3.2.0-26-generic
qemu-kvm 1.0
libvirt 0.9.12 - (tried with 0.9.11 and 0.9.8 too, had to upgrade for it to work nicely with openvswitch)
openvswitch 1.6.90 - tried with regular bridges too

tomcos

Hey everyone, been virtualizing PfSense for a while now and havn't run into any problem I couldn't figure out until now. I followed the directions and I believe everything works fine except for when I input the

<model type="virtio">from <model type="e1000">into my xml file. After, when PfSense beings to boot, no interfaces are found, so it cannot start.
I'm running
2.0.1-RELEASE (amd64)
FreeBSD 8.1-RELEASE-p6
on
Ubuntu 12.04 x64
latest versions of all virtualization packages

here is a snapshot of my xml file if it helps

<domain type="kvm"><name>Router</name>
<uuid>e2968164-4c0c-bc9e-3406-4043ea694ff4</uuid>
<description>PFSENSE</description>
<memory>3145728</memory>
<currentmemory>3145728</currentmemory>
<vcpu>2</vcpu>
<os><type arch="x86_64" machine="pc-1.0">hvm</type></os>
<features><acpi><apic><pae></pae></apic></acpi></features>
<clock offset="utc"><on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices><emulator>/usr/bin/kvm</emulator>
<disk type="file" device="disk"><driver name="qemu" type="raw"><source file="/pfSense/pfsense">
<target dev="hda" bus="ide"><address type="drive" controller="0" bus="0" unit="0">

and my loader.conf.local

virtio_load="YES"
virtio_pci_load="YES"
if_vtnet_load="YES"
virtio_balloon_load="YES"

any suggestions would be lovely :)</address></memballoon> </address></model></video></graphics> </address></model></mac></interface> </address></model></mac></interface> </address></controller> </address></readonly></target></driver></disk> </address></target></driver></disk></devices></clock></domain></model></model>

athurdent

@tomcos:

2.0.1-RELEASE (amd64)
FreeBSD 8.1-RELEASE-p6

I might be wrong here but wasn't VIRTIO support first added in 2.1-beta?
They work fine on Ubuntu 12.04 x64 for me. Throughput is also very nice, around 800 Mbit with iperf if I remember correctly.

jimp

Correct, the virtio drivers are only in 2.1, they aren't in 2.0.x

tomcos

ah! now it all makes sense. Thank you for the reply :)

athurdent

Don't know if it's related to VirtIO, but when I use
-smp cores=2
pfSense crashes after a few minutes. Without the smp Parameter everything's OK. Anyone else noticing this?
I'm running Ubuntu 12.04 LTS x64.

Startup Parameters:

/usr/bin/kvm \
-no-fd-bootchk \
-k de \
-cpu host \
-smp cores=2 \
-m 512 \
-machine type=pc,accel=kvm \
-drive file=/data/vms/wan-dmz-pfsense-qcow2.img,if=virtio,cache=writeback \
-net nic,model=virtio,macaddr=DE:AD:BE:EF:A0:88,vlan=88 \
-net tap,vlan=88,ifname=tap88,script=/etc/qemu-ifup-br1,downscript=/etc/qemu-ifdown-br1 \
-net nic,model=virtio,macaddr=DE:AD:BE:EF:AF:FE,vlan=89 \
-net tap,vlan=89,ifname=tap89,script=/etc/qemu-ifup-br2,downscript=/etc/qemu-ifdown-br2 \
-daemonize

wb-munzinger

Works great here, even with 2 cores.
I had to enable "Disable hardware checksum offload" again. Without I had no connection at all.

lothan

After activated "VirtIO Memory Ballooning" still does not automatically granted more memory from the available range, to be increased manually. Options are: CURRENT(256m) \ MAX(2048m)

snap2.png_thumb

snap3.png_thumb

athurdent

@wb-munzinger:

Works great here, even with 2 cores.

I think I have found the reason for the crashes. As soon as I turn on the SNMP CPU check in Cacti (running on my monitor host), pfSense crashes after some time. It's the same on VMWare, I have submitted the crash info for the VMWare host yesterday. Using only one virtual core or turning off SNMP CPU checks fixes the problem for me.

Mattz

This worked for me, as the vtnet was shown but not on my new installs with the latest snapshots anymore.

I can load virtio @ the commandline but it won't load everyting during boot. My lines are in /boot/loader.conf.local

Nachtfalke

Thank you JR for this great tutorial.

I just installed for testing a pfsense 2.1BETA1 snapshot from 21.03.2013 on a ProxmoxVE 2.3 installation.
Loading HDD VirtIO drivers and NIC VirtIO drivers is working.

Not sure about ballooning. On proxmox I can type "info balloon" into the "monitor" and it always shows me the MAX mem of 1024MB.
The same on another VM with Windows 7 x64 and the ballooning drivers shows me other values which are below the maximum.

Are there any other way/methods to test if ballooning is working or not ? I am no linus/freebsd expert so perhaps someone can explain it more in detail if possible ;)

pfsense 2.1 "System activity" shows me this line:

    7 root     -16    -     0K    16K vtbslp   0:01  0.00% virtio_balloon

Nevertheless thank you very much for that tutorial!

invitu

I also have a big problem with incorrect cksum packets

configuration :
host : Centos 6.4 64 bit (2.6.32-358.6.2.el6.x86_64) + qemu-kvm-0.12.1.2
nic hardware : e1000e
lan : eth0
vlan : eth0.254
br0 : bridge to eth0
br254 : bridge to eth0.254

guest : pfsense 2.1RC0 64 bits snapshot (31/05/2013) with virtio or 2.1BETA0
LAN : br0 (virtio)
WAN : br254 (virtio)

another guest : for example fedora12
LAN : br0

from the host or the other guest, I can icmp or udp:53 to Internet but tcp:80 or udp:123 are stuck…

here is tcpdump extract for "ntpdate us.pool.ntp.org" :

226.76.50.123.dsl.dyn.mana.pf.44299 > 192.168.254.253.rockwell-csp2: Flags [.], cksum 0xe925 (correct), seq 97, ack 432, win 320, options [nop,nop,TS val 3629676 ecr 27931684], length 0
10:00:13.398297 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 76)
192.168.254.253.35791 > kapu.skafari.com.ntp: [bad udp cksum b920!] NTPv4, length 48
Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 3s, precision -6
Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
Reference Timestamp: 0.000000000
Originator Timestamp: 0.000000000
Receive Timestamp: 0.000000000
Transmit Timestamp: 3579105613.398010432 (2013/06/01 10:00:13)
Originator - Receive Timestamp: 0.000000000
Originator - Transmit Timestamp: 3579105613.398010432 (2013/06/01 10:00:13)
192.168.254.253.59909 > housetree.sugarlabs.org.ntp: [bad udp cksum 290e!] NTPv4, length 48
Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 3s, precision -6
Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
Reference Timestamp: 0.000000000
Originator Timestamp: 0.000000000
Receive Timestamp: 0.000000000
Transmit Timestamp: 3579105613.998007059 (2013/06/01 10:00:13)
Originator - Receive Timestamp: 0.000000000
Originator - Transmit Timestamp: 3579105613.998007059 (2013/06/01 10:00:13)

with (net2k_pci driver for br0 and virtio driver for br254) or (net2k_pci for both), everything is fine

I also had no problem with fedora14 64 bits + qemu-kvm-0.13.0 (exactly same hardware) as the host and pfsense 2.1BETA0 as the guest (I just upgraded my host from fedora 14 to centos 6.4…)

I installed a linux-based firewall distrib (ipfire.org) as a guest with same settings and I have no problem...

do you think the problem is with qemu-kvm-0.12.1.2 + freebsd virtio drivers ?