4 port NIC with mixed IDs

johnpoz

I have an old server I have to setup proxmox on sometime today, been putting it off.. It has 2 4 port nics in it.. Will make sure to note order of ports.. I have been out of the server biz for quite some time but I don't recall them ever being out of order like that.. Sometimes they go in opposite direction than you might think.. But I don't recall them being out of order or skipping a number like he says where there is no 2..

I had a dual port nic in my old machine - but with only 2 its hard to tell if anything is off like being stated..

AndyRH

My first one like this was my first pfSense build and it took a bit to figure out why the WAN was getting an inside address. I thought it was a pfSense thing. Then I got an identical card for my PC and it did the same thing. Those are the only two I have seen do that and I have had many at work.

bingo600

Wonder what is causing the naming.
First detected or lowest mac or ???

My QOTOM-Q355G4 have messed up ports.
My newer 6-ports doesn't

stephenw10

The port numbering order is entirely down the way FreeBSD parses the PCI tree.

You can force it to reorder using PCI Device Wiring but I don't recommend that. If you have to reinstall those settings will be lost and the ports would be re-ordered.

Steve

johnpoz

Best solution is prob just a label maker and label them hehe

AndyRH

On mine it is not the MAC, I checked. Just a guess, but maybe they crossed 2 wires. The 4 ports are really 4 NICs on 1 board. They would be found in order as the OS scans the bus.

As for the label maker idea, be sure it is a little one.

stephenw10

I bet they are not in order on the bus. Try running: pciconf -lv

The NICs will almost certainly appear in the correct order for the PCI device locations. As seen by FreeBSD at least. It's not uncommon to see the BIOS pass different values for FreeBSD that it does for Windows.

Steve

Patch

@johnpoz said in 4 port NIC with mixed IDs:

server I have to setup proxmox on

For me with pfsense running in a VM under Proxmox, nic order is determined by the order they are defined in the VM definition created in Proxmox (I use nic pass through)

johnpoz

@patch well got proxmox running - and its showing the interfaces freaking odd, ie get the 1-4, but then 49-52

Haven't setup any vms yet just got the thing booted.. Old DL360 gen9..

stephenw10

Interesting. I'd check the PCI locations there too. If they're not the same card perhaps one set of NICs is behind a bridge so appears on a different bus?

Patch

@johnpoz
What is happening would probably be shown by

lspci -v

and if you have set up pass through

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

btw
When you create your VM I find it easiest to assign no NIC
then go to pve -> VM -> Hardware
and add nic in the order you want them to apear in pfsense ie WAN, LAN, Op1 etc
In pfsense then they will be given sequential numbers in that order.

darkcorner

@johnpoz

My error.

igp1
igp0
igp3
igp2

darkcorner

@stephenw10 said in 4 port NIC with mixed IDs:

I bet they are not in order on the bus. Try running: pciconf -lv

pcib4@pci0:3:0:0:	class=0x060400 card=0x00000000 chip=0x230412d8 rev=0x05 hdr=0x01
    vendor     = 'Pericom Semiconductor'
    device     = 'PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:4:1:0:	class=0x060400 card=0x00000000 chip=0x230412d8 rev=0x05 hdr=0x01
    vendor     = 'Pericom Semiconductor'
    device     = 'PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib6@pci0:4:2:0:	class=0x060400 card=0x00000000 chip=0x230412d8 rev=0x05 hdr=0x01
    vendor     = 'Pericom Semiconductor'
    device     = 'PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
igb0@pci0:5:0:0:	class=0x020000 card=0xa02f8086 chip=0x10c98086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82576 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb1@pci0:5:0:1:	class=0x020000 card=0xa02f8086 chip=0x10c98086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82576 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb2@pci0:6:0:0:	class=0x020000 card=0xa02f8086 chip=0x10c98086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82576 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb3@pci0:6:0:1:	class=0x020000 card=0xa02f8086 chip=0x10c98086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82576 Gigabit Network Connection'
    class      = network
    subclass   = ethernet

Even so I would have expected them to be in order.

darkcorner

@johnpoz said in 4 port NIC with mixed IDs:

Best solution is prob just a label maker and label them hehe

I had thought of that too.
However, if the numbering derives from FreeBSD the important thing is that this does not change at each restart because otherwise it would skip the assignment of the networks and therefore of the connected cables.

darkcorner

@patch
Is there lspci on FreeBSD / pfSense?
I can't even find a package, I also tried with pciutils.

Patch

@darkcorner said in 4 port NIC with mixed IDs:

Is there lspci on FreeBSD / pfSense?

No
It is a linux command relevant to Proxmox not pfsense install

stephenw10

pciconf is the FreeBSD equivalent for most functions.

You can see though how the ports are numbered in the order they are seen on the bus:

igb0@pci0:5:0:0:
igb1@pci0:5:0:1:
igb2@pci0:6:0:0:
igb3@pci0:6:0:1:

There's nothing you can do to change that.

Steve

darkcorner

Interpreting the pciconf data, I believe the problem is due to the card having two buses for two NICs each.
Therefore, the double sequence igp1-0 and igp3-2 is explained.
OK. I'll put a label indicating the ports ID.
Thanks everyone.