NIC passthrough vs OS bridge

eiger3970 0

Hello, I run pfSense as a KVM gateway/router, bridged however it seems passing through the NIC is better.

The challenge in my network topology is that a bridged setup didn't expose the KVM gateway/router's WAN IP to the Internet, generating an internal WAN IP of 192.168.122.165. This was due to my host machine having a bridge within a bridge (nested bridge?) so if the KVM gateway/router is off, the host machine can access Internet. This is really only needed at startup to download the VMM and pfSense programs on site.

Could someone elucidate my basic and new understanding?

Here's how I think the packet traffic flows:
Bridged:
Modem (in bridge mode) -> NIC -> OS bridge -> hypervisor -> KVM -> gateway -> router.

Passthrough:
Modem (in bridge mode) -> NIC -> KVM passthrough -> gateway -> router.

According to several articles I read ```
https://www.iplocation.net/bridge-vs-passthrough


Another bonus for me is that now my gateway has the WAN exposed to the Internet, so OpenVPN can export a certificate (automatically populating the gateway WAN, now being the public WAN IP address) for remote control from a phone.

NollipfSense

@eiger3970-0 To me, it really doesn't matter as it depends on the use case. Personally, I prefer NIC passthrough because one can add a smart managed switch, especially one with L2, L3, L4 capabilities and giving administrative control as well as expanding one's network, keep in mind, expanding by virtual switches also.

eiger3970 0

@NollipfSense
Thanks.
I tried to setup passthrough but with the Ubuntu host's current bridged network to the KVM pfSense router, the changes on KVM pfSense didn't apply after a reboot.
Setting up a bridge from Ubuntu host to the KVM pfSense router seems easier than a passthrough from the Ubuntu host's NIC to the KVM pfSense router.

eiger3970 0

@eiger3970-0 So, this is my current network topology:
alt text

My understanding is that this architecture is called 'User space device emulation' as per IBM source, Figure 2.
Complicated, however I have managed to bridge my host hardware platform the KVM pfSense router to access LAN devices, however if the KVM pfSense router is off, the host hardware platform can still access the Internet.

My issue is that I cannot remote access the network, probably due to the KVM pfSense router's WAN IP not receiving the public WAN IP, which a remote user needs to know to remotely access the network.

Therefore, I am considering changing the architecture to 'Passthrough within the hypervisor', as per IBM source Figure 3.
I think this might apply the KVM pfSense router with the public WAN IP?

I'm guessing I configure the KVM pfSense router for the passthrough, via the VMM controls?
Someone mentioned a NAT, but I've been unsuccessful with a NAT in all architectures.

Any feedback on my diagram, I might change it to IBM's diagram which are probably more ISO standard-like?

eiger3970 0

@eiger3970-0
I updated my network topology to be inline with IBM standards.

Pretty sure I have some of the connections wrong, but it's working, apart from Remote Access.

My understanding is my network is a User-space device emulation (not a Hyper-based device emulation, nor a Passthrough within the hypervisor).
alt text

Here's my understanding of how I might build a passthrough?
alt text

NollipfSense

@eiger3970-0 It's still difficult to follow and maybe why others have not responded. I take it that Ubuntu is the host machine and you have four bridges...I assumed that vtnet0 is pfSense WAN and vtnet1 is pfSense LAN...the only two that needed to passthrough. May I suggest to have a look here: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

Normally, the host machine has a built-in Ethernet port that's used solely for the host and usually one adds a PCIe NIC, two of which are passthrough to pfSense. Then, most, like me, add a switch to pfSense LAN, then adds the host via Ethernet cable to LAN.

Patch

@eiger3970-0 If my reading of your set up is correct. Your physical hardware had 2 physical NIC.

If that is correct then

passing through the pfsense WAN NIC and
using a software bridge for pfsense LAN, other vitual machines and hypervistor access

Is reasonable. Doing so limits the attack surface of your WAN interface.

If you want to do more (perhaps to increase the LAN bandwidth or manage VLANs in pfsense) then you would need more physical NICs to pass through to pfsense and still support LAN access for the hyervistor an other virtual machines.

eiger3970 0

@Patch
Seems like a bridge is the simplest network.
I've drawn up this network topology.
Is there any easy to follow pfSense guide for the Ubuntu bridge setup?
alt text

eiger3970 0

@eiger3970-0 I can't figure out the passthrough setup.
I have the network working with the KVM router pfSense, bridged via the hypervisor.
The network topology isn't perfect too.

eiger3970 0

@eiger3970-0 How can I set the network to have:
Ubuntu: 192.168.1.120
KVM router LAN: 192.168.1.170
KVM router WAN: whatever's needed there.

NollipfSense

@eiger3970-0 I have discovered that one doesn't need to passthrough the entire PCi NIC...just name pfsense WAN vtnet0 to the port, example ens0f0, and when you plug the cable from your ISP in, it will automatically passthrough.

I see or have some idea of what you are trying to accomplish; however, start simple by adding ISP cable to pfSense WAN...you can always set bridges to other network later...
Screenshot 2023-10-17 at 6.26.44 AM.png

eiger3970 0

@NollipfSense Thanks.
I found VMM (Virtual Machine Manager 4.0.0) has PCI passthrough, by Adding Hardware -> PCI Host Device -> Interface enp3s0 -> Finish.
I removed all VMM NICs, although later I guess I'll need to Add Hardware for 1 NIC, for LAN to this host machine, other KVMs and LAN devices.

However, starting KVM pfSense shows error:
Error starting domain: unsupported conifguration: host doesn't support passthrough of host PCI devices
Details:

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 108, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1384, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

starting KVM pfSense shows error:
Error starting domain: unsupported conifguration: host doesn't support passthrough of host PCI devices

Be sure that if your host is using a latest Linus kernel, when passing through a PCI device, the kernel will write to denylist file and block. One got to create a vfio-pci.conf file in /etc/modprobe.d and add:
options vfio-pci disable_denylist=1

But as I had said earlier if you setup pfsense WAN = vtnet0 and the corresponding port, example enp3s0, you just need to plug the cable from your modem in...it will automatically pass-through if the modem is in bridge mode...no need to pass-through the entire NIC.

Is your an onboard NIC?

eiger3970 0

@NollipfSense said in NIC passthrough vs OS bridge:

But as I had said earlier if you setup pfsense WAN = vtnet0 and the corresponding port, example enp3s0, you just need to plug the cable from your modem in...it will automatically pass-through if the modem is in bridge mode...no need to pass-through the entire NIC.

The pfSense router is a virtual router, so when I install the KVM (Kernel Virtual Machine) pfSense, there's only 1 recognised interface, being vtnet0: 52:54:00:81:05:8a 192.168.1.249.
However the WAN NIC0 has MAC address a8:a1:59:6e:1f:8b.

eiger3970 0

@eiger3970-0
I unplugged NIC0 and plugged in.
KVM router terminal -> option 1 -> Connect the WAN interface now and make sure that the link is up.
I reconnected NIC0 -> No link-up detected.
Ubuntu Terminal shows enp2s0 and enp3s0 both UP.

eiger3970 0

@eiger3970-0
So it's working, but I'm unable to connect directly to the KVM router pfSense.
Here's the best topology diagram I think represents the connection from the bridged ISP router to the KVM router pfSense.

Just trying to clean up the Host which has a messy 3 Ethernet connections and 2 bridges...I think this can be simplified somehow?

alt text

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

So it's working

Good news...despite not sure why you cannot connect directly, since like you , I have a coNatted ISP.

eiger3970 0

@NollipfSense
May I ask if you're using Hypervisor VMM and running pfSense as a KVM?
My current network is a bit messed up with 3 Ethernet connections and 2 bridges.
alt text

This is the network I'm trying to finalise, by cleaning up unnecessary Ethernet and Bridge connections. By my limited understanding, I should only need 1 bridge from the Ubuntu host to somewhere (the physical NIC1?)
alt text

NollipfSense

@eiger3970-0 said in NIC passthrough vs OS bridge:

May I ask if you're using Hypervisor VMM and running pfSense as a KVM?

No, I am using Proxmox v8.04...I thought that's what you were using also.

eiger3970 0

@NollipfSense
Thanks, this explains the simplicity, which is a good feature of Proxmox. I used Proxmox for years.
I've moved on to VMM and KVM for several reasons.