Installing pfSense on KVM with OpenVswitch - a somewhat complete guide



  • This guide is intented as a reference for people who are trying to install pfSense(or any virtual firewall, for that matter) on a KVM system. Using this guide, you can essentially turn any pc you have lying around into a full blown firewall, and still use it for other purposes as well by using virtualisation for pfSense.

    It also somewhat encompasses the network part that is external to the linux-setup in the diagram. How you configure the switch and trunking there is up to you since this is so different from vendor to vendor.

    This guide is not recommended for your first setup with pfSense, but rather for people who have experimented with this and got stuck along the way, like I did. Putting all the lessons learned from random articles on the internet into one composite reference. I'm assuming you know basic linux administration.

    In this setup, for instance, the server that I'm using is also used as a platform to play video material using kodi. It also runs sonarr/jacket/transmission/plex on the same machine. But those services are outside of the scope of this. A simple NUC i3 will be more than powerful enough for this setup, just make sure it has enough RAM to accomodate the VM and all your services. I have 8GB and it's plenty for all of the above with 300 megabits/second throughput to the internet.

    Without further ado: the end-goal, see the picture in attachement

    This shows pretty much every component in the setup. The idea is that you want to trunk both vlan 100 & 200(your choice, obviously) to the server and using OVS to deliver the vlans to the firewall and server itself.

    Required:

    1 physical server with 1 or more network interfaces.

    1 linux machine with a GUI desktop, preferably.(can be a VM)

    1 managed .1Q switch where vlan 200 is the internet, vlan 100 is the LAN.

    So let's get going :)

    First, take the server and install linux on it. I'm using Ubuntu server for this, but obviously this works with any modern distro. And if you're using arch/gentoo, this guide should be rather redundant anyway…

    When installing ubuntu server, just install the standard system utilities and the openSSH server, the rest we will do manually.

    Copy your SSH-key to the server and connect to it. This will come in handy later when using the virt-manager.

    ssh-copy-id <localuser>@ <server>ssh <user>@ <server>It shouldn't ask for a password.

    If you don't have a public key yet, google it :)

    On the server, check of KVM is available:

    sudo apt-get install cpu-checker
    kvm-ok
    

    If you see:

    INFO: /dev/kvm exists
    KVM acceleration can be used
    

    You are good to go.

    Let's install KVM:

    sudo apt-get install qemu-kvm libvirt-bin
    

    And OVS:

    sudo apt-get install openvswitch-switch
    

    Let's first set the network in order, create the OVS bridge

    sudo ovs-vsctl add-br OVSBridge
    sudo ovs-vsctl set port OVSBridge tag=100
    

    Verify by running sudo ovs-vsctl show, you should see this:

    92a55ab-da19-4a6b-939f-3eb7ab74e0f9
        Bridge OVSBridge
            Port OVSBridge
                tag: 100
                Interface OVSBridge
                    type: internal
        ovs_version: "2.5.2"
    

    Next, let's make the OVSBridge ip configuration persistent after reboot.

    Edit /etc/network/interfaces, it should look like this:

    source /etc/network/interfaces.d/*
    
    # The loopback network interface
    auto lo
    iface lo inet loopback
    
    # The primary network interface
    auto ens3
    iface ens3 inet manual
    
    auto OVSBridge
    iface OVSBridge inet static
    address 192.168.122.150
    netmask 255.255.255.0
    gateway 192.168.122.1
    
    dns-nameservers 8.8.8.8
    

    Obviously, change it to your internal network settings.

    Now comes the "hold on to your butts"-moment:

    sudo ovs-vsctl add-port OVSBridge <physical interface=""> tag=100 trunk=200 && sudo reboot now</physical>
    

    This will set the interface as a untagged interface in vlan 100 and will also trunk vlan 200 as tagged traffic.

    This will add the port to the OVS, and you will lose all connectivity. This will also reboot the server. After reboot, it should come back with the IP above. If it doesn't, grab a keyboard, plug it in the server and try to figure out why…

    Verify again by running "sudo ovs-vsctl show", you should see this:

    
    292a55ab-da19-4a6b-939f-3eb7ab74e0f9
        Bridge OVSBridge
            Port OVSBridge
                tag: 100
                Interface OVSBridge
                    type: internal
            Port "ens3"
                tag: 100
                trunks: [200]
                Interface "ens3"
        ovs_version: "2.5.2"
    

    If you messed up, you can set any value with:

    sudo ovs-vsctl set port <portname> tag=XXX trunk=YYY</portname>
    

    This will overwrite whatever was there.

    Notice the "tag" and "trunks".

    This is rather a confusing naming convention in that "tag", is the equivalent of "mode access" in cisco-terms. It means untagged traffic is put in this vlan…

    "Trunks" is more straightforward: it allows tagged traffic over the link in the vlan mentioned. However, if no tag or trunk argument is given, the port is a trunk that allows all vlans tagged and uses native vlan 1.

    If all is well, it just comes back. Allowing us to move on. Notice that you are no longer connecting to the physical interface IP address, but to the OVSBridge interface IP address.

    Start up virt-manager on the linux-desktop GUI and go to file=> add connection. Select QEMO+SSH and your username and server IP. This will not work if you don't have your key on the remote server, you could install packages such as ask-pass but just uploading your key is easier.

    Copy the pfSense installer ISO to your server any way you want and select it as the install source. Name the vm "pfsense".

    Here comes the important bit: in the network selection, just take the OVSbridge and macvtap. This is wrong, but the GUI won't let go forward past this point otherwise.

    Using the virt-manager, Install pfsense the "quick and easy" way, standard kernel. After installation, shut down the pfSense VM. We have to make some changes first.

    type the following:

    virsh
    edit pfsense
    

    With the editor of you choice and go down to the network part of the XML file. In my case, this is what is generated:

    
        <interface type="direct"><mac address="52:54:00:b0:3a:a0"><source dev="OVSBridge" mode="bridge">
          <model type="virtio"><address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0">
    
    That's not right, let's change this to the following:
    

    <interface type="bridge"><source bridge="OVSBridge">
          <vlan trunk="yes"><tag id="100"><tag id="200"></tag></tag></vlan>
          <virtualport type="openvswitch"></virtualport>
          <target dev="pfSenseTrunk"><model type="virtio">!!copy the last line as it was generated

    <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0">

    After booting the VM, a quick glance now at the OVS config shows:

    
    292a55ab-da19-4a6b-939f-3eb7ab74e0f9
        Bridge OVSBridge
            Port pfSenseTrunk
                trunks: [100, 200]
                Interface pfSenseTrunk
            Port OVSBridge
                tag: 100
                Interface OVSBridge
                    type: internal
            Port "ens3"
                tag: 100
                trunks: [200]
                Interface "ens3"
        ovs_version: "2.5.2"
    

    Cool, this means virsh spawned a new interface, configured it as a trunk with 2 vlans and connected it to the pfSense Firewall. The last step before configuring pfsense would be to automatically start the pfsense vm on boot: in virtual machine manager in the GUI, just go to the lightbulb => boot options => check "boot virtual machine on host boot", also, check the "copy host cpu configuration" if your CPU has AES acceleration.

    Let's configure pfSense now

    Boot the firewall, when it asks if you want to configure vlans, say yes and configure vlan 100 and 200. Mark 200 as WAN and 100 as LAN. If you configured your switch correctly, you'll get a public IP on the WAN.

    The very last caveat: in the current state of VirtIO network drivers in FreeBSD, it is necessary to check the Disable hardware checksum offload box under System > Advanced on the Networking tab and to manually reboot pfSense after saving the setting. Not doing this will drop all traffic traversing the firewall…

    So there you have it :). The rest is up to you but once you managed this you should be fine, I think.

    Questions/comments don't hesitate

    Feel free to share
    ![pfsense drawings (1).png](/public/imported_attachments/1/pfsense drawings (1).png)
    ![pfsense drawings (1).png_thumb](/public/imported_attachments/1/pfsense drawings (1).png_thumb)</address></model></target></interface>

    ```</server></user></server></localuser>


  • I would like to follow this guide, but I can't see the attachments... would be possible to re-upload them?

    I'm just really testing things and trying to learn, and the diagram would be quite helpful

    Thanks!