Help with Ubuntu + KVM + pfSense Configuration
-
Because of a missing driver I'm hosting pfS on a virtualized server, I am virtualizing it un Ubuntu until the driver is available in FreeBSD and then install it back as originally planned.
Since I am virtualized now, I am having problems:
Ubuntu install success, all drivers recognized
KVM up and configured
Added needed virtual networks (I think)
PfSense VM is up and running, its receiving a WAN connection and connected to the internet via ping in shell in pfS.
Internet works on host Ubuntu installHere is where it goes sideways:
I can't get the pfSense instance to start giving out IPs, there is a PC direct connected to 1 of the 2 LAN ports (for testing the LAN) is not getting an IP and stuck on 'unidentified network' on Win11.
I have read several blog posts on setup, mainly around setting up a Netplan, and this is where is seems to go wrong for me with everything I try.
What is the ideal way to set this up? I have 1 WAN connection via a 10G NIC, and a dual port SFP+ NIC with 2 x 10G connections to be the 2 separate LANs. I seem to be getting the virtual instance to be working ok, but it seems to fail when crossing over the virtualized network to the physical one.
-
@f4-0
How did you setup the virtual network in Ubuntu?How is the host connected to pfSense?
Did you disable "hardware checksum offloading" in pfSense?
Did you also try to connect another device to pfSense?
-
@viragomann said in Help with Ubuntu + KVM + pfSense Configuration:
@f4-0
How did you setup the virtual network in Ubuntu?I set them up in QEMU/KVM under Virtual Networks. There is default, then vmbr0 and vmbr1, and they both show connected to the 2 mac addresses and on a bridge which I named virbr2
How is the host connected to pfSense?
I dont understand, the host? Meaning the host OS? Via a KVM virtual machine.
Did you disable "hardware checksum offloading" in pfSense?
I turned off both LAN interfaces which are the problems, the WAN is connected fine. It has no impact but I will leave it off until its working otherwise.
Did you also try to connect another device to pfSense?
Yes, same result. pfSense is not handing out IPs successfully yet. I also had this connection working before I virtualized the instance, so it was also working before, it was the NIC on the WAN that doesn't have a driver, the LAN ports were fine before.
-
I used both of these as references, but they ended up getting confusing. There is a layer of knowledge I am missing to understand what this guy is doing, also, he gives different examples and I can't determine what to use or try.
https://getlabsdone.com/how-to-install-pfsense-on-kvm/
https://getlabsdone.com/how-to-configure-netplan-network/
Also referencing this one:
https://gist.github.com/RulerOf/12d8460d727b232525f0fdc1fbde1245
And I also watched this video, which was helpful getting the image booted, but then lost me on network setup and interop with the host OS:
https://www.youtube.com/watch?v=DKD--Egx39Q
This guy has the worst microphone ever invented and is also in a Spanish dialect, but I could follow some visually:
https://www.youtube.com/watch?v=fBctb_HcRU8
-
@f4-0 said in Help with Ubuntu + KVM + pfSense Configuration:
I set them up in QEMU/KVM under Virtual Networks.
QEMU/KVM needs a user interface for doing this. Do you use the graphical VMM or virsh?
There is default, then vmbr0 and vmbr1, and they both show connected to the 2 mac addresses and on a bridge which I named virbr2
Not clear, what you mean. "vmbrX" are virtual bridges on Linux. Each bridge must have at least one physical NIC as slave to connect to it from outside.
So if you have two NICs, vmbr0 is connected e.g. to eth0 and vmbr1 to eth1.
Internally you will have connected pfSense LAN interface to vmbr1, so if you connect an outside physical device to eth1 port it is connected to LAN of pfSense.How is the host connected to pfSense?
I dont understand, the host? Meaning the host OS? Via a KVM virtual machine.
The Ubuntu server. You said, it has internet, so I'm wondering how it gets it.
-
@viragomann said in Help with Ubuntu + KVM + pfSense Configuration:
@f4-0 said in Help with Ubuntu + KVM + pfSense Configuration:
I set them up in QEMU/KVM under Virtual Networks.
QEMU/KVM needs a user interface for doing this. Do you use the graphical VMM or virsh?
I used the graphical interface launched thru the command virtmanager in a terminal window.
There is default, then vmbr0 and vmbr1, and they both show connected to the 2 mac addresses and on a bridge which I named virbr2
Not clear, what you mean. "vmbrX" are virtual bridges on Linux. Each bridge must have at least one physical NIC as slave to connect to it from outside.
So if you have two NICs, vmbr0 is connected e.g. to eth0 and vmbr1 to eth1.
Internally you will have connected pfSense LAN interface to vmbr1, so if you connect an outside physical device to eth1 port it is connected to LAN of pfSense.This is where I am stuck, what is the mechanism to do this? My understanding was that in the VirtManager console, you define a the virtual NICs and then with NetPlan you connect them to the physical nics... is this wrong? If so, what is the process?
How is the host connected to pfSense?
I dont understand, the host? Meaning the host OS? Via a KVM virtual machine.
The Ubuntu server. You said, it has internet, so I'm wondering how it gets it.
The NIC is in a PCIe slot, host PC, Ubuntu base OS, so its getting internet from the cable connected to it from the WAN.
-
@f4-0 said in Help with Ubuntu + KVM + pfSense Configuration:
This is where I am stuck, what is the mechanism to do this? My understanding was that in the VirtManager console, you define a the virtual NICs and then with NetPlan you connect them to the physical nics... is this wrong? If so, what is the process?
There may be different way to configure bridges. I don't know, which is recommended for Ubuntu.
I'm running KVM (with pfSense on top) on OpenSUSE. SUSE uses wicked for networking and has a graphical interface to manage it. There I add bridges and connect them to the NICs.
In VMM I can connect VMs to these bridges then.However, you can also write the settings in config files in /etc/sysconfig/networks/.
The NIC is in a PCIe slot, host PC, Ubuntu base OS, so its getting internet from the cable connected to it from the WAN.
I connected the host to my LAN bridge. I.e. I set an IP on the bridge. So I can access it from the LAN without the need of running pfSense.
-
@f4-0 I'm not hosting a dhcpd guest with libvirt so can't comment on that issue, but I tried various bridging techniques including libvirt's virtual networks and openvswitch. All worked. But in the end I found the simplest (for me to implement and understand) was to bring up the bridges on the host using Ubuntu's netplan and networkd. I only have one NIC on this machine (desktop) and it receives tagged and untagged traffic.
#/etc/netplan/01.vmbr.yaml network: version: 2 renderer: networkd ethernets: enp0s31f6: {} vlans: vlan100: accept-ra: no id: 100 link: enp0s31f6 vlan200: accept-ra: no id: 200 link: enp0s31f6 bridges: br0: interfaces: [enp0s31f6] macaddress: 00:01:02:03:04:05 addresses: [192.168.30.11/24] routes: - to: default via: 192.168.30.1 nameservers: search: [local.lan] addresses: [192.168.30.10] br100: interfaces: [vlan100] link-local: [] br200: interfaces: [vlan200] link-local: []$ networkctl list IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 enp0s31f6 ether enslaved configured 3 br0 bridge routable configured 4 br100 bridge carrier configured 5 br200 bridge carrier configured 6 vlan200 vlan enslaved configured 7 vlan100 vlan enslaved configuredThen with libvirt, dispense with virtual network definitions and assign guest interfaces to the bridges:
<interface type='bridge'> <mac address='00:00:00:00:00:00'/> <source bridge='br200'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface>Also, check iptables configuration.