Installing pfSense on KVM with OpenVswitch - a somewhat complete guide

SoulChild

This guide is intented as a reference for people who are trying to install pfSense(or any virtual firewall, for that matter) on a KVM system. Using this guide, you can essentially turn any pc you have lying around into a full blown firewall, and still use it for other purposes as well by using virtualisation for pfSense.

It also somewhat encompasses the network part that is external to the linux-setup in the diagram. How you configure the switch and trunking there is up to you since this is so different from vendor to vendor.

This guide is not recommended for your first setup with pfSense, but rather for people who have experimented with this and got stuck along the way, like I did. Putting all the lessons learned from random articles on the internet into one composite reference. I'm assuming you know basic linux administration.

In this setup, for instance, the server that I'm using is also used as a platform to play video material using kodi. It also runs sonarr/jacket/transmission/plex on the same machine. But those services are outside of the scope of this. A simple NUC i3 will be more than powerful enough for this setup, just make sure it has enough RAM to accomodate the VM and all your services. I have 8GB and it's plenty for all of the above with 300 megabits/second throughput to the internet.

Without further ado: the end-goal, see the picture in attachement

This shows pretty much every component in the setup. The idea is that you want to trunk both vlan 100 & 200(your choice, obviously) to the server and using OVS to deliver the vlans to the firewall and server itself.

Required:

1 physical server with 1 or more network interfaces.

1 linux machine with a GUI desktop, preferably.(can be a VM)

1 managed .1Q switch where vlan 200 is the internet, vlan 100 is the LAN.

So let's get going :)

First, take the server and install linux on it. I'm using Ubuntu server for this, but obviously this works with any modern distro. And if you're using arch/gentoo, this guide should be rather redundant anyway…

When installing ubuntu server, just install the standard system utilities and the openSSH server, the rest we will do manually.

Copy your SSH-key to the server and connect to it. This will come in handy later when using the virt-manager.

ssh-copy-id <localuser>@ <server>ssh <user>@ <server>It shouldn't ask for a password.

If you don't have a public key yet, google it :)

On the server, check of KVM is available:

sudo apt-get install cpu-checker
kvm-ok

If you see:

INFO: /dev/kvm exists
KVM acceleration can be used

You are good to go.

Let's install KVM:

sudo apt-get install qemu-kvm libvirt-bin

And OVS:

sudo apt-get install openvswitch-switch

Let's first set the network in order, create the OVS bridge

sudo ovs-vsctl add-br OVSBridge
sudo ovs-vsctl set port OVSBridge tag=100

Verify by running sudo ovs-vsctl show, you should see this:

92a55ab-da19-4a6b-939f-3eb7ab74e0f9
    Bridge OVSBridge
        Port OVSBridge
            tag: 100
            Interface OVSBridge
                type: internal
    ovs_version: "2.5.2"

Next, let's make the OVSBridge ip configuration persistent after reboot.

Edit /etc/network/interfaces, it should look like this:

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto ens3
iface ens3 inet manual

auto OVSBridge
iface OVSBridge inet static
address 192.168.122.150
netmask 255.255.255.0
gateway 192.168.122.1

dns-nameservers 8.8.8.8

Obviously, change it to your internal network settings.

Now comes the "hold on to your butts"-moment:

sudo ovs-vsctl add-port OVSBridge <physical interface=""> tag=100 trunk=200 && sudo reboot now</physical>

This will set the interface as a untagged interface in vlan 100 and will also trunk vlan 200 as tagged traffic.

This will add the port to the OVS, and you will lose all connectivity. This will also reboot the server. After reboot, it should come back with the IP above. If it doesn't, grab a keyboard, plug it in the server and try to figure out why…

Verify again by running "sudo ovs-vsctl show", you should see this:


292a55ab-da19-4a6b-939f-3eb7ab74e0f9
    Bridge OVSBridge
        Port OVSBridge
            tag: 100
            Interface OVSBridge
                type: internal
        Port "ens3"
            tag: 100
            trunks: [200]
            Interface "ens3"
    ovs_version: "2.5.2"

If you messed up, you can set any value with:

sudo ovs-vsctl set port <portname> tag=XXX trunk=YYY</portname>

This will overwrite whatever was there.

Notice the "tag" and "trunks".

This is rather a confusing naming convention in that "tag", is the equivalent of "mode access" in cisco-terms. It means untagged traffic is put in this vlan…

"Trunks" is more straightforward: it allows tagged traffic over the link in the vlan mentioned. However, if no tag or trunk argument is given, the port is a trunk that allows all vlans tagged and uses native vlan 1.

If all is well, it just comes back. Allowing us to move on. Notice that you are no longer connecting to the physical interface IP address, but to the OVSBridge interface IP address.

Start up virt-manager on the linux-desktop GUI and go to file=> add connection. Select QEMO+SSH and your username and server IP. This will not work if you don't have your key on the remote server, you could install packages such as ask-pass but just uploading your key is easier.

Copy the pfSense installer ISO to your server any way you want and select it as the install source. Name the vm "pfsense".

Here comes the important bit: in the network selection, just take the OVSbridge and macvtap. This is wrong, but the GUI won't let go forward past this point otherwise.

Using the virt-manager, Install pfsense the "quick and easy" way, standard kernel. After installation, shut down the pfSense VM. We have to make some changes first.

type the following:

virsh
edit pfsense

With the editor of you choice and go down to the network part of the XML file. In my case, this is what is generated:


    <interface type="direct"><mac address="52:54:00:b0:3a:a0"><source dev="OVSBridge" mode="bridge">
      <model type="virtio"><address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0">

That's not right, let's change this to the following:

<interface type="bridge"><source bridge="OVSBridge">
<vlan trunk="yes"><tag id="100"><tag id="200"></tag></tag></vlan>
<virtualport type="openvswitch"></virtualport>
<target dev="pfSenseTrunk"><model type="virtio">!!copy the last line as it was generated

After booting the VM, a quick glance now at the OVS config shows:


292a55ab-da19-4a6b-939f-3eb7ab74e0f9
    Bridge OVSBridge
        Port pfSenseTrunk
            trunks: [100, 200]
            Interface pfSenseTrunk
        Port OVSBridge
            tag: 100
            Interface OVSBridge
                type: internal
        Port "ens3"
            tag: 100
            trunks: [200]
            Interface "ens3"
    ovs_version: "2.5.2"

Cool, this means virsh spawned a new interface, configured it as a trunk with 2 vlans and connected it to the pfSense Firewall. The last step before configuring pfsense would be to automatically start the pfsense vm on boot: in virtual machine manager in the GUI, just go to the lightbulb => boot options => check "boot virtual machine on host boot", also, check the "copy host cpu configuration" if your CPU has AES acceleration.

Let's configure pfSense now

Boot the firewall, when it asks if you want to configure vlans, say yes and configure vlan 100 and 200. Mark 200 as WAN and 100 as LAN. If you configured your switch correctly, you'll get a public IP on the WAN.

The very last caveat: in the current state of VirtIO network drivers in FreeBSD, it is necessary to check the Disable hardware checksum offload box under System > Advanced on the Networking tab and to manually reboot pfSense after saving the setting. Not doing this will drop all traffic traversing the firewall…

So there you have it :). The rest is up to you but once you managed this you should be fine, I think.

Questions/comments don't hesitate

Feel free to share
![pfsense drawings (1).png](/public/imported_attachments/1/pfsense drawings (1).png)
![pfsense drawings (1).png_thumb](/public/imported_attachments/1/pfsense drawings (1).png_thumb)</address></model></target></interface>

```</server></user></server></localuser>

ballanux

I would like to follow this guide, but I can't see the attachments... would be possible to re-upload them?

I'm just really testing things and trying to learn, and the diagram would be quite helpful

Thanks!

ebedtang

Could it be done with wifi as wan? I have followed the steps above and its failed as my wan uses wifi. My wifi won't connect, i put wifi as trunked port with this command,

$ ovs-vsctl add-port ovsbr wls5 tag=100 trunk=200 && reboot
...
ovsbr is openvswitch bridge

After reboot my wifi failed to connect. Im using fedora 31 with openvswitch v2.12.

simon_lefisch

@SoulChild Thanks for the write-up, been looking for something like this. I do have a slightly different setup and would like to get your opinion on how to set it up correctly.

I installed a 2-port PCIe NIC card on my server so pfSense would have its own NIC ports....1 for WAN which is connected directly to my modem (we'll call it NIC port A) and 1 for LAN which is connected to port 1 on my managed switch (we'll call this NIC port B). I have the VLANs set as follows for port 1 on my switch....VLAN 1 for default/mgmt [untagged], VLAN 100 for LAN [tagged], VLAN 200 for WLAN [tagged], and VLAN 201 for Guest WLAN [tagged]. I have also setup the VLANs accordingly on pfSense.

The devices that are hardwired to the switch are able to get and IP from pfSense and can connect to the internet. I experimented with setting VLANs in OVS using other guides and my VMs were able to get an IP from pfSense as well, however they were not able to connect to the internet (I can ping the VMs from my hardware). I assume it has something to do the with the VLAN settings in OVS (trunking to the switch?).

What would your suggestion be on how to properly setup this network? I've scoured all over and have found things that say one thing and another. Looking for something more straightforward. Would my connections be the best way? Or should I go about it your way in the sense of having the modem connected to the managed switch on a VLAN vs having the pfSense NIC WAN port connect to it directly?