NIC passthrough vs OS bridge
-
@eiger3970-0 It's still difficult to follow and maybe why others have not responded. I take it that Ubuntu is the host machine and you have four bridges...I assumed that vtnet0 is pfSense WAN and vtnet1 is pfSense LAN...the only two that needed to passthrough. May I suggest to have a look here: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html
Normally, the host machine has a built-in Ethernet port that's used solely for the host and usually one adds a PCIe NIC, two of which are passthrough to pfSense. Then, most, like me, add a switch to pfSense LAN, then adds the host via Ethernet cable to LAN.
-
@eiger3970-0 If my reading of your set up is correct. Your physical hardware had 2 physical NIC.
If that is correct then
- passing through the pfsense WAN NIC and
- using a software bridge for pfsense LAN, other vitual machines and hypervistor access
Is reasonable. Doing so limits the attack surface of your WAN interface.
If you want to do more (perhaps to increase the LAN bandwidth or manage VLANs in pfsense) then you would need more physical NICs to pass through to pfsense and still support LAN access for the hyervistor an other virtual machines.
-
@Patch
Seems like a bridge is the simplest network.
I've drawn up this network topology.
Is there any easy to follow pfSense guide for the Ubuntu bridge setup?
-
@eiger3970-0 I can't figure out the passthrough setup.
I have the network working with the KVM router pfSense, bridged via the hypervisor.
The network topology isn't perfect too. -
@eiger3970-0 How can I set the network to have:
Ubuntu: 192.168.1.120
KVM router LAN: 192.168.1.170
KVM router WAN: whatever's needed there. -
@eiger3970-0 I have discovered that one doesn't need to passthrough the entire PCi NIC...just name pfsense WAN vtnet0 to the port, example ens0f0, and when you plug the cable from your ISP in, it will automatically passthrough.
I see or have some idea of what you are trying to accomplish; however, start simple by adding ISP cable to pfSense WAN...you can always set bridges to other network later...
-
@NollipfSense Thanks.
I found VMM (Virtual Machine Manager 4.0.0) has PCI passthrough, by Adding Hardware -> PCI Host Device -> Interface enp3s0 -> Finish.
I removed all VMM NICs, although later I guess I'll need to Add Hardware for 1 NIC, for LAN to this host machine, other KVMs and LAN devices.However, starting KVM pfSense shows error:
Error starting domain: unsupported conifguration: host doesn't support passthrough of host PCI devices
Details:Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 108, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/object/domain.py", line 1384, in startup self._backend.create() File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create raise libvirtError('virDomainCreate() failed') libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices -
@eiger3970-0 said in NIC passthrough vs OS bridge:
starting KVM pfSense shows error:
Error starting domain: unsupported conifguration: host doesn't support passthrough of host PCI devicesBe sure that if your host is using a latest Linus kernel, when passing through a PCI device, the kernel will write to denylist file and block. One got to create a vfio-pci.conf file in /etc/modprobe.d and add:
options vfio-pci disable_denylist=1But as I had said earlier if you setup pfsense WAN = vtnet0 and the corresponding port, example enp3s0, you just need to plug the cable from your modem in...it will automatically pass-through if the modem is in bridge mode...no need to pass-through the entire NIC.
Is your an onboard NIC?
-
@NollipfSense said in NIC passthrough vs OS bridge:
But as I had said earlier if you setup pfsense WAN = vtnet0 and the corresponding port, example enp3s0, you just need to plug the cable from your modem in...it will automatically pass-through if the modem is in bridge mode...no need to pass-through the entire NIC.
The pfSense router is a virtual router, so when I install the KVM (Kernel Virtual Machine) pfSense, there's only 1 recognised interface, being vtnet0: 52:54:00:81:05:8a 192.168.1.249.
However the WAN NIC0 has MAC address a8:a1:59:6e:1f:8b. -
@eiger3970-0
I unplugged NIC0 and plugged in.
KVM router terminal -> option 1 -> Connect the WAN interface now and make sure that the link is up.
I reconnected NIC0 -> No link-up detected.
Ubuntu Terminal shows enp2s0 and enp3s0 both UP. -
@eiger3970-0
So it's working, but I'm unable to connect directly to the KVM router pfSense.
Here's the best topology diagram I think represents the connection from the bridged ISP router to the KVM router pfSense.Just trying to clean up the Host which has a messy 3 Ethernet connections and 2 bridges...I think this can be simplified somehow?
-
@eiger3970-0 said in NIC passthrough vs OS bridge:
So it's working
Good news...despite not sure why you cannot connect directly, since like you , I have a coNatted ISP.
-
@NollipfSense
May I ask if you're using Hypervisor VMM and running pfSense as a KVM?
My current network is a bit messed up with 3 Ethernet connections and 2 bridges.
This is the network I'm trying to finalise, by cleaning up unnecessary Ethernet and Bridge connections. By my limited understanding, I should only need 1 bridge from the Ubuntu host to somewhere (the physical NIC1?)
-
@eiger3970-0 said in NIC passthrough vs OS bridge:
May I ask if you're using Hypervisor VMM and running pfSense as a KVM?
No, I am using Proxmox v8.04...I thought that's what you were using also.
-
@NollipfSense
Thanks, this explains the simplicity, which is a good feature of Proxmox. I used Proxmox for years.
I've moved on to VMM and KVM for several reasons. -
@eiger3970-0
Ok, I've figured out the topology of the network.
Any suggestions how to configure it please?Network topology: NIC0->KVM router WAN, NIC1->KVM router LAN.
First layer 2 network (Br0):
Modem==(Host-NIC0)---Br0---Net1---PFSenseWAN(NIC0)Second layer 2 network (Br1):
24 port SW==(Host-NIC1)---Br1---Net2---PFSenseLAN(NIC1)Virtual NIC 0:
this NIC should be assigned to Br1, so your host will get ip/gw from FW like other clients.Host Ubuntu will manually create Br0 and Br1.
I'm not sure how or where to create Net1 and Net2? -
@eiger3970-0 yep
That’s a standard pfsense install for a virtual machine.- Wan NIC connected to a virtual switch (a bridge). VM wan connected to the bridge by a virtual NIC.
- structure repeated for the LAN using a different physical NIC, virtual switch (bridge) and virtual nic for pfsense VM.
- hypervisor configuration added to connect it to the virtual switch (bridge) which pfSense LAN connects to.
The variant are:
- pass through the WAN NIC. Doing so means your hypervisor is not exposed to the internet.
- pass through both NICs to pfsense, doing so means the hypervisor will need a third physical NIC and external physical switch to connect to your LAN (and in turn connect to the internet via the pfsense VM)
-
@Patch Thank you.
So, will the host Ubuntu have LAN and Internet access?
My idea seems to enable the host Ubuntu to have LAN and Internet access via vNIC0.Your 1. variant is inline with my proposed setup right?
Your 2. variant is not an option, as I only want 2 physical NICs, rather than 3 physical NICs.So, this is my understanding so the host Ubuntu, KVMs and LAN devices can have communication and Internet:
NIC0, let's call it WAN (that is the interface which will access modem through Host's Br0)
NIC1, let's call it LAN (provides internet access to the LAN-Wired/Wireless and even the host)First layer 2 network (Br0):
Modem==(Host-NIC0)---Br0(Net1)---PFSenseWAN(NIC0)Second layer 2 network (Br1):
24 port SW==(Host-NIC1)---Br1(Net2)---PFSenseLAN(NIC1)Virtual NIC 0 (vNIC0) on HOST:
this vNIC should be assigned to Br1, so the host will get IP/GW from the FW like other clients. -
@eiger3970-0 said in NIC passthrough vs OS bridge:
Your 1. variant is inline with my proposed setup right?
No.
You have passed zero NIC to any virtual machines. Your virtual machines are only connected by virtual NIC to virtual switches (ie a bridge). That is the normal way of configuring virtual machines. I probably should have labelled it option zero.The alternative 1) listed above pass the hardware of one physical NIC to one VM. Doing so means no other VM or hypervisor can access that NIC while that VM is running. The system cost for the WAN NIC may not be that high as often only the software router should access that NIC.
Alternative 2) above passes the hardware of two physical NICs to one VM. Doing so means no other VM or hypervisor can access those 2 NICs while that VM is running. The system cost for which is your physical hardware needs at least 3 physical NIC. That cost is too high is some systems. I run Proxmox on a minicomputer with 6 physical NICs so I pass through a WAN and multiple LAN NICs to my pfsense VM.
-
@Patch said in NIC passthrough vs OS bridge:
Alternative 2) above passes the hardware of two physical NICs to one VM. Doing so means no other VM or hypervisor can access those 2 NICs while that VM is running. The system cost for which is your physical hardware needs at least 3 physical NIC. That cost is too high is some systems. I run Proxmox on a minicomputer with 6 physical NICs so I pass through a WAN and multiple LAN NICs to my pfsense VM.
Agree, and one can always link other VMs to the Linux bridge vmbr2...if the VM needs a port opened, then one can use HAproxy on pfSense. Alternative 2 is so flexible offering both external and internal switch expansion as well as control.
