NIC passthrough vs OS bridge

eiger3970 0

@eiger3970-0 I can't figure out the passthrough setup.
I have the network working with the KVM router pfSense, bridged via the hypervisor.
The network topology isn't perfect too.

eiger3970 0

@eiger3970-0 How can I set the network to have:
Ubuntu: 192.168.1.120
KVM router LAN: 192.168.1.170
KVM router WAN: whatever's needed there.

NollipfSense

@eiger3970-0 I have discovered that one doesn't need to passthrough the entire PCi NIC...just name pfsense WAN vtnet0 to the port, example ens0f0, and when you plug the cable from your ISP in, it will automatically passthrough.

I see or have some idea of what you are trying to accomplish; however, start simple by adding ISP cable to pfSense WAN...you can always set bridges to other network later...
Screenshot 2023-10-17 at 6.26.44 AM.png

eiger3970 0

@NollipfSense Thanks.
I found VMM (Virtual Machine Manager 4.0.0) has PCI passthrough, by Adding Hardware -> PCI Host Device -> Interface enp3s0 -> Finish.
I removed all VMM NICs, although later I guess I'll need to Add Hardware for 1 NIC, for LAN to this host machine, other KVMs and LAN devices.

However, starting KVM pfSense shows error:
Error starting domain: unsupported conifguration: host doesn't support passthrough of host PCI devices
Details:

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 108, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1384, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

starting KVM pfSense shows error:
Error starting domain: unsupported conifguration: host doesn't support passthrough of host PCI devices

Be sure that if your host is using a latest Linus kernel, when passing through a PCI device, the kernel will write to denylist file and block. One got to create a vfio-pci.conf file in /etc/modprobe.d and add:
options vfio-pci disable_denylist=1

But as I had said earlier if you setup pfsense WAN = vtnet0 and the corresponding port, example enp3s0, you just need to plug the cable from your modem in...it will automatically pass-through if the modem is in bridge mode...no need to pass-through the entire NIC.

Is your an onboard NIC?

eiger3970 0

@NollipfSense said in NIC passthrough vs OS bridge:

But as I had said earlier if you setup pfsense WAN = vtnet0 and the corresponding port, example enp3s0, you just need to plug the cable from your modem in...it will automatically pass-through if the modem is in bridge mode...no need to pass-through the entire NIC.

The pfSense router is a virtual router, so when I install the KVM (Kernel Virtual Machine) pfSense, there's only 1 recognised interface, being vtnet0: 52:54:00:81:05:8a 192.168.1.249.
However the WAN NIC0 has MAC address a8:a1:59:6e:1f:8b.

eiger3970 0

@eiger3970-0
I unplugged NIC0 and plugged in.
KVM router terminal -> option 1 -> Connect the WAN interface now and make sure that the link is up.
I reconnected NIC0 -> No link-up detected.
Ubuntu Terminal shows enp2s0 and enp3s0 both UP.

eiger3970 0

@eiger3970-0
So it's working, but I'm unable to connect directly to the KVM router pfSense.
Here's the best topology diagram I think represents the connection from the bridged ISP router to the KVM router pfSense.

Just trying to clean up the Host which has a messy 3 Ethernet connections and 2 bridges...I think this can be simplified somehow?

alt text

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

So it's working

Good news...despite not sure why you cannot connect directly, since like you , I have a coNatted ISP.

eiger3970 0

@NollipfSense
May I ask if you're using Hypervisor VMM and running pfSense as a KVM?
My current network is a bit messed up with 3 Ethernet connections and 2 bridges.
alt text

This is the network I'm trying to finalise, by cleaning up unnecessary Ethernet and Bridge connections. By my limited understanding, I should only need 1 bridge from the Ubuntu host to somewhere (the physical NIC1?)
alt text

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

May I ask if you're using Hypervisor VMM and running pfSense as a KVM?

No, I am using Proxmox v8.04...I thought that's what you were using also.

eiger3970 0

@NollipfSense
Thanks, this explains the simplicity, which is a good feature of Proxmox. I used Proxmox for years.
I've moved on to VMM and KVM for several reasons.

eiger3970 0

@eiger3970-0
Ok, I've figured out the topology of the network.
Any suggestions how to configure it please?

Network topology: NIC0->KVM router WAN, NIC1->KVM router LAN.
First layer 2 network (Br0):
Modem==(Host-NIC0)---Br0---Net1---PFSenseWAN(NIC0)

Second layer 2 network (Br1):
24 port SW==(Host-NIC1)---Br1---Net2---PFSenseLAN(NIC1)

Virtual NIC 0:
this NIC should be assigned to Br1, so your host will get ip/gw from FW like other clients.

Host Ubuntu will manually create Br0 and Br1.
I'm not sure how or where to create Net1 and Net2?

Patch

@eiger3970-0 yep
That’s a standard pfsense install for a virtual machine.

Wan NIC connected to a virtual switch (a bridge). VM wan connected to the bridge by a virtual NIC.
structure repeated for the LAN using a different physical NIC, virtual switch (bridge) and virtual nic for pfsense VM.
hypervisor configuration added to connect it to the virtual switch (bridge) which pfSense LAN connects to.

The variant are:

pass through the WAN NIC. Doing so means your hypervisor is not exposed to the internet.
pass through both NICs to pfsense, doing so means the hypervisor will need a third physical NIC and external physical switch to connect to your LAN (and in turn connect to the internet via the pfsense VM)

eiger3970 0

@Patch Thank you.
So, will the host Ubuntu have LAN and Internet access?
My idea seems to enable the host Ubuntu to have LAN and Internet access via vNIC0.

Your 1. variant is inline with my proposed setup right?
Your 2. variant is not an option, as I only want 2 physical NICs, rather than 3 physical NICs.

So, this is my understanding so the host Ubuntu, KVMs and LAN devices can have communication and Internet:
NIC0, let's call it WAN (that is the interface which will access modem through Host's Br0)
NIC1, let's call it LAN (provides internet access to the LAN-Wired/Wireless and even the host)

First layer 2 network (Br0):
Modem==(Host-NIC0)---Br0(Net1)---PFSenseWAN(NIC0)

Second layer 2 network (Br1):
24 port SW==(Host-NIC1)---Br1(Net2)---PFSenseLAN(NIC1)

Virtual NIC 0 (vNIC0) on HOST:
this vNIC should be assigned to Br1, so the host will get IP/GW from the FW like other clients.

Patch

@eiger3970-0 said in NIC passthrough vs OS bridge:

Your 1. variant is inline with my proposed setup right?

No.
You have passed zero NIC to any virtual machines. Your virtual machines are only connected by virtual NIC to virtual switches (ie a bridge). That is the normal way of configuring virtual machines. I probably should have labelled it option zero.

The alternative 1) listed above pass the hardware of one physical NIC to one VM. Doing so means no other VM or hypervisor can access that NIC while that VM is running. The system cost for the WAN NIC may not be that high as often only the software router should access that NIC.

Alternative 2) above passes the hardware of two physical NICs to one VM. Doing so means no other VM or hypervisor can access those 2 NICs while that VM is running. The system cost for which is your physical hardware needs at least 3 physical NIC. That cost is too high is some systems. I run Proxmox on a minicomputer with 6 physical NICs so I pass through a WAN and multiple LAN NICs to my pfsense VM.

NollipfSense

@Patch said in NIC passthrough vs OS bridge:

Alternative 2) above passes the hardware of two physical NICs to one VM. Doing so means no other VM or hypervisor can access those 2 NICs while that VM is running. The system cost for which is your physical hardware needs at least 3 physical NIC. That cost is too high is some systems. I run Proxmox on a minicomputer with 6 physical NICs so I pass through a WAN and multiple LAN NICs to my pfsense VM.

Agree, and one can always link other VMs to the Linux bridge vmbr2...if the VM needs a port opened, then one can use HAproxy on pfSense. Alternative 2 is so flexible offering both external and internal switch expansion as well as control.

eiger3970 0

@NollipfSense
Ok, well according to your description, the network architecture I am looking for seems to be
Option2: hardware NIC0 and NIC1 passes to the KVM router pfSense.
Doing so means no other VM or hypervisor (also no host?) can access NIC0 or NIC1 while KVM router pfSense is running.
Why is a 3rd physical NIC2 needed?
The host, hypervisor, KVMs and LAN devices could connect by virtual NIC to virtual switches (ie a bridge)?
Option0: the normal way of configuration virtual machines with
NIC0 connected to virtual NIC to virtual switches (ie a bridge) and
NIC1 connected to virtual NIC to virtual switches (ie a bridge).

I simply need my Host, LAN devices and KVMs to communicate and have Internet, preferably with the simplest/most efficient network architecture (currently with 2 NICs available).

How are you 'passing' your NICs to the KVM. You mentioned you're on Proxmox, so it may be a case of just plugging in, but I'm trying to figure out how on Host Ubuntu, Hypervisor VMM and KVM router pfSense.

Patch

@eiger3970-0 said in NIC passthrough vs OS bridge:

Option2: hardware NIC0 and NIC1 passes to the KVM router pfSense.
...
Why is a 3rd physical NIC2 needed?
The host, hypervisor, KVMs and LAN devices could connect by virtual NIC to virtual switches (ie a bridge)?

because while the hypervisior and other VM can be connected by virtual NIC to a virtual switch (bridge), that bridge can not be connected to a physical NIC or your physical switch without another physical NIC (which the hardware has not been passed through to a VM eg pfsense)

eiger3970 0

@Patch Ok.
I guess I'll have to setup something like this?