Advice needed on virtualising pfSense
-
Hi,
I am looking for some guidance on installing pfSense in a QEMU/KVM VM. I am a developer and networking and sys admin isn't my area of expertise.
We have a Dell PowerEdge Server with 4 Intel NICs. It will be used as a VM host and will be stored in a Data Centre. Since the budget is tight, we're looking to virtualise pfSense and make it our firewall/router/security gateway. All incoming and outgoing traffic will go through the pfSense VM. OpenVPN will be set up for remote users who will need to be able to ssh into the VMs on the host as well as into the host itself.
So far, I've been able to install pfSense in a QEMU/KVM VM running on CentOS 7. I created two bridges using NetworkManager:
- br0 with em1 as the slave. Configured with DHCP.
- br1 with em4 as the slave. Configured with DHCP.
When I created the pfSense VM, I attached both bridges as network interfaces to the VM. I used VirtIO in both cases. When I installed pfSense, I assigned the WAN to br0 and the LAN to br1. Since I was using the VirtIO drivers, I disabled Hardware Checksum Offloading.
I then created an OpenVPN server and tested it. I was able to connect to the OpenVPN server with no problems. I was able to ping and access the pfSense VM, however, I was not able to ping or access any other hosts on the LAN. I've seen other users report this problem, but it's usually related to the routing table. When I checked the routing table on the connected OpenVPN client, it looked correct to me. So, i then tried changing the br1 bridge from DHCP to static on the host. No joy. When I changed it back to DHCP, I was then able to ping and access all the hosts on the LAN. After that, it would continue to work with no problems. I have found that if I reinstall everything from scratch, I would sometimes run into the same problem, where OpenVPN clients can't access the rest of the LAN. If I fiddle with the network connection br1 on the host by changing it from DHCP to static and back again, eventually it will work, but I don't know why.
While this setup works for us, I am nervous about putting the server in the Data Centre. I am worried that a reboot may stop the OpenVPN client from working as it should, forcing me to make a unplanned trip to the Data Centre. Another thing I've noticed is that when I connect my laptop to em4 (br1) on the server, it seems to take a long time (~30 seconds) for a network connection to be detected. Sometimes, it won't detect it at all, but that usually happens after the initial pfSense installation. A reboot of pfSense and sometimes the host itself, usually fixes the problem, but it will always take up to 30 seconds for the connection to be detected. For reference, when I connect my laptop to my ASUS router at home, it takes no more than 5 seconds to detect the ethernet connection.
I am starting to wonder if the combination of pfSense running in a VM using linux bridges is the cause of my problems? This leads bring me back to my original question:
Is there a recommended way to install pfSense in a VM?
I've seen other posts where people are using MacVTap instead of traditional linux bridges. I am not all that familiar with MacVTap, but from my understanding, MacVTap connects directly to the network interface on the KVM host. There is also a limitation where the guest cannot talk to the host and vice versa. There is a work around apparently:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-troubleshooting-common_libvirt_errors_and_troubleshooting#sect-Guest_can_reach_outside_network_but_cannot_reach_host_when_using_macvtap_interface
but I don't know how this affects OpenVPN clients. We need remote users to have the ability to access both the VMs and the VM host.
I also read somewhere that e1000 would be better than VirtIO for pfSense. Is that true?
Apologies for the long winded post. I am just looking for some guidance or for some people to share their experience in this area.
Thanks for reading.