NIC passthrough vs OS bridge

eiger3970 0

@eiger3970-0 So, this is my current network topology:
alt text

My understanding is that this architecture is called 'User space device emulation' as per IBM source, Figure 2.
Complicated, however I have managed to bridge my host hardware platform the KVM pfSense router to access LAN devices, however if the KVM pfSense router is off, the host hardware platform can still access the Internet.

My issue is that I cannot remote access the network, probably due to the KVM pfSense router's WAN IP not receiving the public WAN IP, which a remote user needs to know to remotely access the network.

Therefore, I am considering changing the architecture to 'Passthrough within the hypervisor', as per IBM source Figure 3.
I think this might apply the KVM pfSense router with the public WAN IP?

I'm guessing I configure the KVM pfSense router for the passthrough, via the VMM controls?
Someone mentioned a NAT, but I've been unsuccessful with a NAT in all architectures.

Any feedback on my diagram, I might change it to IBM's diagram which are probably more ISO standard-like?

eiger3970 0

@eiger3970-0
I updated my network topology to be inline with IBM standards.

Pretty sure I have some of the connections wrong, but it's working, apart from Remote Access.

My understanding is my network is a User-space device emulation (not a Hyper-based device emulation, nor a Passthrough within the hypervisor).
alt text

Here's my understanding of how I might build a passthrough?
alt text

NollipfSense

@eiger3970-0 It's still difficult to follow and maybe why others have not responded. I take it that Ubuntu is the host machine and you have four bridges...I assumed that vtnet0 is pfSense WAN and vtnet1 is pfSense LAN...the only two that needed to passthrough. May I suggest to have a look here: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

Normally, the host machine has a built-in Ethernet port that's used solely for the host and usually one adds a PCIe NIC, two of which are passthrough to pfSense. Then, most, like me, add a switch to pfSense LAN, then adds the host via Ethernet cable to LAN.

Patch

@eiger3970-0 If my reading of your set up is correct. Your physical hardware had 2 physical NIC.

If that is correct then

passing through the pfsense WAN NIC and
using a software bridge for pfsense LAN, other vitual machines and hypervistor access

Is reasonable. Doing so limits the attack surface of your WAN interface.

If you want to do more (perhaps to increase the LAN bandwidth or manage VLANs in pfsense) then you would need more physical NICs to pass through to pfsense and still support LAN access for the hyervistor an other virtual machines.

eiger3970 0

@Patch
Seems like a bridge is the simplest network.
I've drawn up this network topology.
Is there any easy to follow pfSense guide for the Ubuntu bridge setup?
alt text

eiger3970 0

@eiger3970-0 I can't figure out the passthrough setup.
I have the network working with the KVM router pfSense, bridged via the hypervisor.
The network topology isn't perfect too.

eiger3970 0

@eiger3970-0 How can I set the network to have:
Ubuntu: 192.168.1.120
KVM router LAN: 192.168.1.170
KVM router WAN: whatever's needed there.

NollipfSense

@eiger3970-0 I have discovered that one doesn't need to passthrough the entire PCi NIC...just name pfsense WAN vtnet0 to the port, example ens0f0, and when you plug the cable from your ISP in, it will automatically passthrough.

I see or have some idea of what you are trying to accomplish; however, start simple by adding ISP cable to pfSense WAN...you can always set bridges to other network later...
Screenshot 2023-10-17 at 6.26.44 AM.png

eiger3970 0

@NollipfSense Thanks.
I found VMM (Virtual Machine Manager 4.0.0) has PCI passthrough, by Adding Hardware -> PCI Host Device -> Interface enp3s0 -> Finish.
I removed all VMM NICs, although later I guess I'll need to Add Hardware for 1 NIC, for LAN to this host machine, other KVMs and LAN devices.

However, starting KVM pfSense shows error:
Error starting domain: unsupported conifguration: host doesn't support passthrough of host PCI devices
Details:

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 108, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1384, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

starting KVM pfSense shows error:
Error starting domain: unsupported conifguration: host doesn't support passthrough of host PCI devices

Be sure that if your host is using a latest Linus kernel, when passing through a PCI device, the kernel will write to denylist file and block. One got to create a vfio-pci.conf file in /etc/modprobe.d and add:
options vfio-pci disable_denylist=1

But as I had said earlier if you setup pfsense WAN = vtnet0 and the corresponding port, example enp3s0, you just need to plug the cable from your modem in...it will automatically pass-through if the modem is in bridge mode...no need to pass-through the entire NIC.

Is your an onboard NIC?

eiger3970 0

@NollipfSense said in NIC passthrough vs OS bridge:

But as I had said earlier if you setup pfsense WAN = vtnet0 and the corresponding port, example enp3s0, you just need to plug the cable from your modem in...it will automatically pass-through if the modem is in bridge mode...no need to pass-through the entire NIC.

The pfSense router is a virtual router, so when I install the KVM (Kernel Virtual Machine) pfSense, there's only 1 recognised interface, being vtnet0: 52:54:00:81:05:8a 192.168.1.249.
However the WAN NIC0 has MAC address a8:a1:59:6e:1f:8b.

eiger3970 0

@eiger3970-0
I unplugged NIC0 and plugged in.
KVM router terminal -> option 1 -> Connect the WAN interface now and make sure that the link is up.
I reconnected NIC0 -> No link-up detected.
Ubuntu Terminal shows enp2s0 and enp3s0 both UP.

eiger3970 0

@eiger3970-0
So it's working, but I'm unable to connect directly to the KVM router pfSense.
Here's the best topology diagram I think represents the connection from the bridged ISP router to the KVM router pfSense.

Just trying to clean up the Host which has a messy 3 Ethernet connections and 2 bridges...I think this can be simplified somehow?

alt text

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

So it's working

Good news...despite not sure why you cannot connect directly, since like you , I have a coNatted ISP.

eiger3970 0

@NollipfSense
May I ask if you're using Hypervisor VMM and running pfSense as a KVM?
My current network is a bit messed up with 3 Ethernet connections and 2 bridges.
alt text

This is the network I'm trying to finalise, by cleaning up unnecessary Ethernet and Bridge connections. By my limited understanding, I should only need 1 bridge from the Ubuntu host to somewhere (the physical NIC1?)
alt text

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

May I ask if you're using Hypervisor VMM and running pfSense as a KVM?

No, I am using Proxmox v8.04...I thought that's what you were using also.

eiger3970 0

@NollipfSense
Thanks, this explains the simplicity, which is a good feature of Proxmox. I used Proxmox for years.
I've moved on to VMM and KVM for several reasons.

eiger3970 0

@eiger3970-0
Ok, I've figured out the topology of the network.
Any suggestions how to configure it please?

Network topology: NIC0->KVM router WAN, NIC1->KVM router LAN.
First layer 2 network (Br0):
Modem==(Host-NIC0)---Br0---Net1---PFSenseWAN(NIC0)

Second layer 2 network (Br1):
24 port SW==(Host-NIC1)---Br1---Net2---PFSenseLAN(NIC1)

Virtual NIC 0:
this NIC should be assigned to Br1, so your host will get ip/gw from FW like other clients.

Host Ubuntu will manually create Br0 and Br1.
I'm not sure how or where to create Net1 and Net2?

Patch

@eiger3970-0 yep
That’s a standard pfsense install for a virtual machine.

Wan NIC connected to a virtual switch (a bridge). VM wan connected to the bridge by a virtual NIC.
structure repeated for the LAN using a different physical NIC, virtual switch (bridge) and virtual nic for pfsense VM.
hypervisor configuration added to connect it to the virtual switch (bridge) which pfSense LAN connects to.

The variant are:

pass through the WAN NIC. Doing so means your hypervisor is not exposed to the internet.
pass through both NICs to pfsense, doing so means the hypervisor will need a third physical NIC and external physical switch to connect to your LAN (and in turn connect to the internet via the pfsense VM)

eiger3970 0

@Patch Thank you.
So, will the host Ubuntu have LAN and Internet access?
My idea seems to enable the host Ubuntu to have LAN and Internet access via vNIC0.

Your 1. variant is inline with my proposed setup right?
Your 2. variant is not an option, as I only want 2 physical NICs, rather than 3 physical NICs.

So, this is my understanding so the host Ubuntu, KVMs and LAN devices can have communication and Internet:
NIC0, let's call it WAN (that is the interface which will access modem through Host's Br0)
NIC1, let's call it LAN (provides internet access to the LAN-Wired/Wireless and even the host)

First layer 2 network (Br0):
Modem==(Host-NIC0)---Br0(Net1)---PFSenseWAN(NIC0)

Second layer 2 network (Br1):
24 port SW==(Host-NIC1)---Br1(Net2)---PFSenseLAN(NIC1)

Virtual NIC 0 (vNIC0) on HOST:
this vNIC should be assigned to Br1, so the host will get IP/GW from the FW like other clients.