PFSense 2.1 & KVM VirtIO



  • For those who run PFSense inside a KVM Virtual Machine, here are some easy steps to enable VirtIO for your PFSense VM.

    Works-For-Me with:

    • PFSense 2.1-BETA0 SnapShot-20120603

    • Debian Testing - 20120603

    • qemu-kvm 1.0
    • libvirt 0.9.11
    • openvswitch 1.4.0
    • virt-manager 0.9.1

    *** Step 1 - Create VirtIO Network Interfaces**

    I'm using 4 interfaces that are connected to various virtual switches on the KVM host:

    • NIC1 - ISP1

    • NIC2 - ISP2

    • NIC3 - CARP

    • NIC4 - LAN

    Before you can use a VirtIO device you need to load the VirtIO kernel modules, but before you can load the kernel modules you need to have at least one interface enabled during PFSense installation. Create all vm interfaces as VirtIO devices but leave just one default (realtek or intel), you're now able to finish the default install and reach the PFSense console menu.

    *** Step 2 - Edit /boot/loader.conf.local**

    Select < 8 Shell > from the PFSense console menu after the first boot with the single (default) interface. At the shell prompt edit the /boot/loader.conf.local file with vi and append the file with the VirtIO module settings.

    vi /boot/loader.conf.local

    
    virtio_load="YES"
    virtio_pci_load="YES"
    if_vtnet_load="YES"
    
    

    If you only want to use VirtIO for your vm network interfaces you're finished now, just shutdown the vm, change the single default ethernet device used for installation to VirtIO and boot into PFSense. From here all VirtIO network interfaces should be visible and ready for final configuration ( interface name: vtnetX ).

    *** Step 3 - Add VirtIO Memory Ballooning**

    Memory ballooning allows you to have your guest dynamically change it's memory usage by evicting unused memory during runtime. Most useful for environments with large memory over-commit requirements.

    vi /boot/loader.conf.local

    
    virtio_balloon_load="YES"
    
    

    *** Step 4 - Add VirtIO Block Device**

    In addition to the network interfaces there also are VirtIO drivers for block devices (virtual harddisks). Just run the default installation with a virtual harddisk in IDE mode and change the /boot/loader.conf.local and /etc/fstab files afterwards.

    vi /boot/loader.conf.local

    
    virtio_blk_load="YES"
    
    

    By default the PFSense installer creates a root and swap volume on IDE device /dev/ad0, before changing your virtual harddisk from IDE to VirtIO you need to edit the fstab file with the new VirtIO block device name ( /dev/vtbdX ).

    vi /etc/fstab

    
    /dev/vtbd0s1a     /          ufs          rw     1     1
    /dev/vtbd0s1b     none     swap       sw     0     0
    
    

    Shutdown the vm and change the virtual harddisk from IDE to VirtIO, boot and enjoy your PFSense KVM VirtIO Virtual Machine…

    kldstat

    
    #kldstat
    
    Id Refs Address            Size     Name
     1   12 0xffffffff80100000 1540f20  kernel
     2    5 0xffffffff81641000 4ac0     virtio.ko
     3    1 0xffffffff81646000 5670     virtio_pci.ko
     4    1 0xffffffff8164c000 4d88     virtio_blk.ko
     5    1 0xffffffff81651000 2f80     virtio_balloon.ko
     6    1 0xffffffff81654000 b0e0     if_vtnet.ko
    
    

    df

    
    #df -h
    
    Filesystem       Size    Used   Avail Capacity  Mounted on
    /dev/vtbd0s1a    4.4G    234M    3.8G     6%    /
    devfs            1.0k    1.0k      0B   100%    /dev
    /dev/md0         3.6M     46k    3.3M     1%    /var/run
    devfs            1.0k    1.0k      0B   100%    /var/dhcpd/dev
    
    

    swapinfo

    
    #swapinfo
    
    Device          1K-blocks     Used    Avail Capacity
    /dev/vtbd0s1b      524288        0   524288     0%
    
    


  • Hi, thanks for the info i have been looking for this and it worked right away.

    However, it still is very cpu hungry compared to the ubuntu/shorewall
    firewall running in the same environment. ( proxmox 2.1)

    instead of eating 100% (2.01) cpu nat'ing 100/100 internet pipe at full speed it use about 70%.

    Idle is at 6% instead of 7-8% in pfsense 2.01

    So its a step in the right direction but still to high.
    Linux idle at 0.1 % and max at 10% doing the same nat.

    Thanks



  • Thanks for the info JR.

    I'd like to add a quick note reminding everybody that you should use /boot/loader.conf.local instead of /boot/loader.conf for tuning, since /boot/loader.conf is over-written by pfsense's startup scripts.


  • Rebel Alliance Developer Netgate

    Should probably turn this into a doc wiki article since it's a somewhat popular topic/faq. If there are no objections I can just copy the text into the wiki.



  • @Are:

    Hi, thanks for the info i have been looking for this and it worked right away.

    However, it still is very cpu hungry compared to the ubuntu/shorewall
    firewall running in the same environment. ( proxmox 2.1)

    instead of eating 100% (2.01) cpu nat'ing 100/100 internet pipe at full speed it use about 70%.

    Idle is at 6% instead of 7-8% in pfsense 2.01

    So its a step in the right direction but still to high.
    Linux idle at 0.1 % and max at 10% doing the same nat.

    Thanks

    Not sure on comparison side/fairness.

    Helpful would be to show a top -SH and what is eating your CPU.



  • @Are:

    However, it still is very cpu hungry compared to the ubuntu/shorewall
    firewall running in the same environment. ( proxmox 2.1)

    ...

    Nice to hear it worked for you!

    Regarding performance, yeah, that's still an issue but for now I believe the "best" you can get. From what I hear/read, didn't tested it myself, FreeBSD 9 and VirtIO should perform much better. For now PFSense is using 8.3, so I guess it will take some more time before you can benefit from that upgrade.

    @dhatz: Thanks for mentioning, I'll edit the original post.


  • Rebel Alliance Developer Netgate

    Added to the wiki (with some edits) here: http://doc.pfsense.org/index.php/VirtIO_Driver_Support

    Thanks!



  • What about perfromance test results?
    I hit ~250Mbit/s limit with standard Intel E1000 emulation, what about virtio?



  • I'm not able to get it running; I have to run at least one interface as non-virtio, forwarding packets between two virtio-interfaces works for icmp, but any tcp-session just freezes.

    Seems to be this bug; http://www.freebsd.org/cgi/query-pr.cgi?pr=165059

    Edit; oh, and I've tried 5 different snapshots between PFSense 2.1-BETA0 SnapShot-20120701 to 0713

    host is;

    • ubuntu 12.04, with kernel 3.2.0-26-generic
    • qemu-kvm 1.0
    • libvirt 0.9.12 - (tried with 0.9.11 and 0.9.8 too, had to upgrade for it to work nicely with openvswitch)
    • openvswitch 1.6.90 - tried with regular bridges too


  • Hey everyone, been virtualizing PfSense for a while now and havn't run into any problem I couldn't figure out until now.  I followed the directions and I believe everything works fine except for when I input the

    <model type="virtio">from <model type="e1000">into my xml file.  After, when PfSense beings to boot, no interfaces are found, so it cannot start.
    I'm running
    2.0.1-RELEASE (amd64)
    FreeBSD 8.1-RELEASE-p6
    on
    Ubuntu 12.04 x64
    latest versions of all virtualization packages

    here is a snapshot of my xml file if it helps

    <domain type="kvm"><name>Router</name>
      <uuid>e2968164-4c0c-bc9e-3406-4043ea694ff4</uuid>
      <description>PFSENSE</description>
      <memory>3145728</memory>
      <currentmemory>3145728</currentmemory>
      <vcpu>2</vcpu>
      <os><type arch="x86_64" machine="pc-1.0">hvm</type></os>
      <features><acpi><apic><pae></pae></apic></acpi></features>
      <clock offset="utc"><on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>restart</on_crash>
      <devices><emulator>/usr/bin/kvm</emulator>
        <disk type="file" device="disk"><driver name="qemu" type="raw"><source file="/pfSense/pfsense">
          <target dev="hda" bus="ide"><address type="drive" controller="0" bus="0" unit="0">

    <disk type="block" device="cdrom"><driver name="qemu" type="raw"><target dev="hdc" bus="ide"><readonly><address type="drive" controller="0" bus="1" unit="0">

    <controller type="ide" index="0"><address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1">

    <interface type="bridge"><mac address="52:54:00:fb:14:eb"><source bridge="br1">
          <model type="e1000"><address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0">

    <interface type="bridge"><mac address="52:54:00:b2:2b:34"><source bridge="br2">
          <model type="e1000"><address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0">

    <serial type="pty"><target port="0"></target></serial>
        <console type="pty"><target type="serial" port="0"></target></console>

    <graphics type="vnc" port="-1" autoport="yes"><video><model type="cirrus" vram="9216" heads="1"><address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0">

    <memballoon model="virtio"><address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0">

    and my loader.conf.local

    virtio_load="YES"
    virtio_pci_load="YES"
    if_vtnet_load="YES"
    virtio_balloon_load="YES"

    any suggestions would be lovely  :)</address></memballoon> </address></model></video></graphics> </address></model></mac></interface> </address></model></mac></interface> </address></controller> </address></readonly></target></driver></disk> </address></target></driver></disk></devices></clock></domain></model></model>



  • @tomcos:

    2.0.1-RELEASE (amd64)
    FreeBSD 8.1-RELEASE-p6

    I might be wrong here but wasn't VIRTIO support first added in 2.1-beta?
    They work fine on Ubuntu 12.04 x64 for me. Throughput is also very nice, around 800 Mbit with iperf if I remember correctly.


  • Rebel Alliance Developer Netgate

    Correct, the virtio drivers are only in 2.1, they aren't in 2.0.x



  • ah! now it all makes sense.  Thank you for the reply :)



  • Don't know if it's related to VirtIO, but when I use
    -smp cores=2
    pfSense crashes after a few minutes. Without the smp Parameter everything's OK. Anyone else noticing this?
    I'm running Ubuntu 12.04 LTS x64.

    Startup Parameters:

    /usr/bin/kvm \
    -no-fd-bootchk \
    -k de \
    -cpu host \
    -smp cores=2 \
    -m 512 \
    -machine type=pc,accel=kvm \
    -drive file=/data/vms/wan-dmz-pfsense-qcow2.img,if=virtio,cache=writeback \
    -net nic,model=virtio,macaddr=DE:AD:BE:EF:A0:88,vlan=88 \
    -net tap,vlan=88,ifname=tap88,script=/etc/qemu-ifup-br1,downscript=/etc/qemu-ifdown-br1 \
    -net nic,model=virtio,macaddr=DE:AD:BE:EF:AF:FE,vlan=89 \
    -net tap,vlan=89,ifname=tap89,script=/etc/qemu-ifup-br2,downscript=/etc/qemu-ifdown-br2 \
    -daemonize
    


  • Works great here, even with 2 cores.
    I had to enable "Disable hardware checksum offload" again. Without I had no connection at all.



  • After activated "VirtIO Memory Ballooning" still does not automatically granted more memory from the available range, to be increased manually. Options are: CURRENT(256m) \ MAX(2048m)






  • @wb-munzinger:

    Works great here, even with 2 cores.

    I think I have found the reason for the crashes. As soon as I turn on the SNMP CPU check in Cacti (running on my monitor host), pfSense crashes after some time. It's the same on VMWare, I have submitted the crash info for the VMWare host yesterday. Using only one virtual core or turning off SNMP CPU checks fixes the problem for me.



  • This worked for me, as the vtnet was shown but not on my new installs with the latest snapshots anymore.

    I can load virtio @ the commandline but it won't load everyting during boot. My lines are in /boot/loader.conf.local



  • Thank you JR for this great tutorial.

    I just installed for testing a pfsense 2.1BETA1 snapshot from 21.03.2013 on a ProxmoxVE 2.3 installation.
    Loading HDD VirtIO drivers and NIC VirtIO drivers is working.

    Not sure about ballooning. On proxmox I can type "info balloon" into the "monitor" and it always shows me the MAX mem of 1024MB.
    The same on another VM with Windows 7 x64 and the ballooning drivers shows me other values which are below the maximum.

    Are there any other way/methods to test if ballooning is working or not ? I am no linus/freebsd expert so perhaps someone can explain it more in detail if possible ;)

    pfsense 2.1 "System activity" shows me this line:

        7 root     -16    -     0K    16K vtbslp   0:01  0.00% virtio_balloon
    

    Nevertheless thank you very much for that tutorial!



  • I also have a big problem with incorrect cksum packets

    configuration :
    host : Centos 6.4 64 bit (2.6.32-358.6.2.el6.x86_64) + qemu-kvm-0.12.1.2
    nic hardware : e1000e
    lan : eth0
    vlan : eth0.254
    br0 : bridge to eth0
    br254 : bridge to eth0.254

    guest : pfsense 2.1RC0 64 bits snapshot (31/05/2013) with virtio or 2.1BETA0
    LAN : br0 (virtio)
    WAN : br254 (virtio)

    another guest : for example fedora12
    LAN : br0

    from the host or the other guest, I can icmp or udp:53 to Internet but tcp:80 or udp:123 are stuck…

    here is tcpdump extract for "ntpdate us.pool.ntp.org"  :

    226.76.50.123.dsl.dyn.mana.pf.44299 > 192.168.254.253.rockwell-csp2: Flags [.], cksum 0xe925 (correct), seq 97, ack 432, win 320, options [nop,nop,TS val 3629676 ecr 27931684], length 0
    10:00:13.398297 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 76)
       192.168.254.253.35791 > kapu.skafari.com.ntp: [bad udp cksum b920!] NTPv4, length 48
           Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 3s, precision -6
           Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
             Reference Timestamp:  0.000000000
             Originator Timestamp: 0.000000000
             Receive Timestamp:    0.000000000
             Transmit Timestamp:   3579105613.398010432 (2013/06/01 10:00:13)
               Originator - Receive Timestamp:  0.000000000
               Originator - Transmit Timestamp: 3579105613.398010432 (2013/06/01 10:00:13)
       192.168.254.253.59909 > housetree.sugarlabs.org.ntp: [bad udp cksum 290e!] NTPv4, length 48
           Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 3s, precision -6
           Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
             Reference Timestamp:  0.000000000
             Originator Timestamp: 0.000000000
             Receive Timestamp:    0.000000000
             Transmit Timestamp:   3579105613.998007059 (2013/06/01 10:00:13)
               Originator - Receive Timestamp:  0.000000000
               Originator - Transmit Timestamp: 3579105613.998007059 (2013/06/01 10:00:13)

    with (net2k_pci driver for br0 and virtio driver for br254) or (net2k_pci for both), everything is fine

    I also had no problem with fedora14 64 bits + qemu-kvm-0.13.0 (exactly same hardware) as the host and pfsense 2.1BETA0 as the guest (I just upgraded my host from fedora 14 to centos 6.4…)

    I installed a linux-based firewall distrib (ipfire.org) as a guest with same settings and I have no problem...

    do you think the problem is with qemu-kvm-0.12.1.2 + freebsd virtio drivers ?



  • Suppose pfSense has hardware checksums enabled. A tcpdump on pfSense will show checksum errors because software doesn't compute the checksums because the hardware does it. But in a virtualised environment pfSense doesn't have access to real hardware. Perhaps the hypervisor doesn't correctly get the hardware to compute the checksums.

    What is the pfSense setting related to hardware checksums? See System -> Advanced, click on Networking and scroll down to Network Interfaces, Hardware Checksum Offloading.



  • Yes I tried the "Hardware Checksum Offloading" setting but it did not resolve the issue.

    tcpdump has been done on the host with "Hardware Checksum Offloading" unchecked. I will try to send tcpcump with the option checked



  • @invitu:

    Yes I tried the "Hardware Checksum Offloading" setting but it did not resolve the issue.

    What did you set it to? (Note the box should be TICKED to DISABLE hardware checksum calculation.) A reboot is probably required for a change in setting to take effect.



  • wallabybob, thanks for helping me

    I just ticked the box to disable hardware calculation, reboot pfsense and ….. IT WORKS

    Yesterday, I also did tick the box but I did not reboot the pfsense virtual machine as the web interface's message was "The changes have been applied successfully." + Close button

    Maybe pfsense developers should modify the message to "Your appliance must be rebooted" + Reboot button

    Thank you my problem is solved