Changing interfaces "names" affects throughput
-
I suppose I could set up another pfsense VM on my second Hvisor, which would match the HW for all parts that matter. But I thought I'd use the APU since it's sitting there in the rack.
I think I need to dig a bit deeper into this, like looking into setting up HA. But more importantly perhaps, trying to find out what is really happening when changing the name like that. It's not like I made some strange hack or something...
-
Well renaming the interfaces like that isn't supported. The result of it is untested/undefined.
-
Yes probably not tested, or at least reported, until now perhaps?
So, running ifconfig reveals some differences between the two setups.
Working setup on VM (no name change):
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN
options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>Not working on VM (name changed to igb0):
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>Don't know if this means anything though. And everything else in the output is exactly the same.
Looking at the APU2 where the name actually is igb0, the list looks like this:
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>Could it be a driver issue perhaps? Does it load based on the name??
-
Ok, yeah so it's enabled all the hardware off-loading features that igb has by default but the actual NIC probably doesn't support them or worse supports them very badly.
Try disabling those. pfSense disables all but checksum offload by default but by running that command outside the pfSense scripts none of that is applied.Steve
-
Yes, clearly driver related...
Running pfsense as VM, with default interface naming (vtnet0, 1...), I have no issues whith any form of HW acceleration enabled. (System/Advanced/Networking).
Same thing on the APU2E4 which runs fine with all HW acceleration enabled, almost reaching the same speeds as the VM, with some effort... (i210 NIC's).
On a VM with names changed as above, I get only 1-2Mbit/s UL when Hardware Checksum Offloading is enabled (not disabled in GUI). The other HW Accel-options don't seem to effect performance in any noticeable way.
The host machine is equipped with a PCI-card listed as having 4 Intel i350 NIC's. However, when setting it up the first time I had trouble passing through only some of the NIC's on the card, keeping the others for other VM's, and therefore ended up assigning them as VirtIO-devices. Having no trouble reaching gigabit speeds anyway, I stayed with that setting.
Now I have just tested changing the VM NICs to "Intel E1000" on the host machine, and this also works with all HW acceleration enabled. This is true even when changing the interface names to igb (from em0, 1 etc. this time).
There is a clear performance hit due to the emulation on the host, but I get the exact same performance regardless of interface names and/or HW acceleration on/off.So to conclude:
Assigning interfaces as VirtIO devices works fine, leaving them named vtnet0,1.. for pfsense.
Changing the names to igb0,1.. works fine as long as Harware Checksum Offloading is turned OFF. "Providing" a different NIC (e.g. through emulation) also works fine, even if interface name is changed.Would be interesting to understand what "device" is actually used by Proxmox when selecting VirtIO (paravirtualized). There is clearly something that messes things up.
I'm guessing that if I could manage to actually pass through the 2(3) NIC's, it will also work, with excellent performance since they are i350's? Perhaps worth trying again...
-
Yes, if you can pass though the i350 successfully then both configs will be igbX and all the problems go away.
-
I thought with some changes I made (like using Q35 instead of ix440 machine) that I could pass individual NIC's to pfsense. But even though they show up as different IMMU groups, the entire card gets passed to the VM.
Unfortunately, this means that I can't use the fourth port for anything else... I could put all other VM's on the mobo port but it basically means I would loose one port.I made some tests though, and things turn out as expected. The card shows up with igb0, 1, 2 and 3 and things work fine from a performance perspective, also with HW offloading activated.
I might do some more testing but it seems it's either renaming and shutting off HW offloading (also large receive), and be able to experiment with sync. Or keep things default and reconfigure interfaces if I ever copy config.xml. No HA Sync in that case though...
Actually it would be nice if one could select not just one, but any combination of "restore areas" in Backup&Restore. That would actually make things easier... Then I could reload everything but interfaces...
-
Never thought the solution could be so simple...
It turns out that the only thing I had to do be able to pass each indivdual NIC to one VM, was to NOT to tick the box "All functions"... ?!?!?
I thought, and probably everyone else that I have watched or read, that you have to tick this in order to get all the functionality that the i350 could offer. Apparently, it meant all the i350s, together from the card. A bit misleading but hey, now it works.
So now I finally have things exactly as I had hoped for, with three i350 NICs nicely configured on my pfsense VM (igb0, igb1 and igb2) with HW acceleration enabled.
Just like it is on the APU2... -
It might have a PCI bridge on the card that it tries to pass?
Anyways that's a much solution that renaming the interface for force them to match.
-
The method is the same for any PCI-device you want to pass through. In the documentation, which I now suddenly understand, they have command line examples using a graphics card with sound chipset on it. They may show up as separate devices (IOMMU groups) but you might want to pass them both together... hence "All functionality"...
Anyway, as you say, now I have a much better solution than renaming devices. Which is actually the way I tried to set it up a long time ago...
