How to fix interface assignment at multiple identical NICs?
-
For the last days I have been fighting with this. I want to clone a machine, have more identical boxes. Backup/Restore does a fine job, except that for reasons unknown to me, on one box at start ue0 comes up as LAN, so that I can configure ue1 as DMZ. Backup and Restore on another in principle identical hardware fails, because ue1 comes up as LAN for reasons unknown to me.
Therefore, it is close to impossible for me to produce a drop-in replacement: the cables will have to go into different plugs, and even the firewall rules don't fit any longer.
And with bad luck, I can't even use the webconfigurator because I end up in the wrong subnet.
What is the trick, and which is the hook, that allows me to define a certain interface to ue0 (re0, em0, etc.) as 'second' interface, that is LAN, despite of what the boot process finds first?
I wouldn't bet, but I even seem to have noticed that the allocation can be changed during a simple reboot. At least, that how it looked to me; once I had a working box reboot, and after the reboot somehow LAN and DMZ (ue0 and ue1) had changed places. -
The short answer there is: don't use USB NICs.
That's unhelpful but true. If you use PCI NICs they always come up in the same order. And since 2.5 you can even set the order they come up in it you want.
With USB NICs it's all about timing and that can change. I don't think there's a quirk you can apply even.The assigned interface doesn't change in the config though and the rules etc all follow that. So only the physical NIC would change, if it did.
What hardware are you actually using there?
Steve
-
Thanks a lot. Sad. Actually, I received a bunch of former thin clients, fanless, no moving parts, sufficiently powerful, small SSD, relatively low power requirements. And USB 3.0! Naturally, one NIC. And I also have a handful of lenovo USB3 to Gigabit Ethernet adapters. I thought their logical marriage was to continue as network appliances.
I am not yet clear about the inherent logic of allocation at reboot. I had thought that they'd just switch physical network interface, and one only had to swap cables. But that doesn't seem to be the case. Once they change order, neither from LAN nor DMZ can I ping or at any way connect to the box, though the clients in both networks receive their IPs from there, very properly and correctly. Consequently no SSH. So I have to go to the box, plug in a monitor and keyboard, and just change the interface allocation (1) by swapping LAN and DMZ. Then everything comes up. That's the part that I hate.
-
It could be some other timing issue, like the USB NICs are not ready when the interfaces are assigned at boot perhaps.
If the DHCP is working and handing out IPs in the correct range though that seems unlikely.
In the config the rules, subnet and dhcp are all linked to the same logical interface. If one is working they all should so I would look for something else preventing it. An error loading the ruleset at all for example.
I imagine if you simply resaved the interfaces with the same order they would also come back up.Steve
-
(I very much appreciate your input, a thanks in between!)
Okay, alas worse. I tried under 'controlled' conditions: power off - remove power - put power back - power on. (Remove power simply because I don't know how far some NIC is still on for WoL.) And then, the interfaces came up configured in reverse order.
I had never used USB-NICs before, and never ran into this. Though, I can't think I'm the one and first human who ever touched this as 'terra incognita'? :-)
This is what dmesg makes from it:.... uhub5: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 Trying to mount root from ufs:/dev/ada0s1a [rw,noatime]... Root mount waiting for: usbus0 CAM usbus1 usbus2 usbus3 usbus4 usbus5 uhub5: 4 ports with 4 removable, self powered .... TSC: P-state invariant ure0 on uhub5 ure0: <Lenovo Thinkpad USB LAN, class 0/0, rev 3.00/30.00, addr 1> on usbus0 ure1 on uhub5 ure1: <Lenovo Thinkpad USB LAN, class 0/0, rev 3.00/30.00, addr 2> on usbus0 miibus1: <MII bus> on ure1 miibus2: <MII bus> on ure0 rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus1 rgephy2: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus2 rgephy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto rgephy2: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto ue0: <USB Ethernet> on ure1 ue0: Ethernet address: 3c:18:a0:07:75:7a ue1: <USB Ethernet> on ure0 ue1: Ethernet address: 3c:18:a0:08:58:40 ....Looks like first-came-first-served to me.
-
Mmm, I'm not sure there's anything you can do about that.
There are some scripts for FreeBSD I see people have written to address this but nothing in pfSense directly.
-
@stephenw10 Yep. Found some scripts, like id your card, and then run some check at boot up. I mean, for a client machine that might be okay, but I don't feel all too happy on a mission-critical box.
I can already see myself, in a few month's time, sleepy or half-drunk, pulling out that USB, plug another one, and nothing works.
By the way, I even tried another usbhub, 100 Mbps, but it didn't make it 100% predictable either. And then I saw one chap having similar problems with re-s.
I think it's a RFE for FreeBSD. Strange enough, many years ago on my Soekrisbox with m0n0wall there was a 1000% clear allocation of WAN, LAN, DMZ to the Ethernet ports. -
@digard said in How to fix interface assignment at multiple identical NICs?:
Strange enough, many years ago on my Soekrisbox with m0n0wall there was a 1000% clear allocation of WAN, LAN, DMZ to the Ethernet ports.
Yes, if you have PCI(e) based NICs there is no problem, they always come up in the same order.
There can be an issue if you add more NICs with the same driver but in FreeBSD 12 you can work around that with PCI device wiring to force it.
Steve
-
Since you're using USB NICs, why not just swap them, so the order matches what you want?
-
Because at the next boot that order may have changed.
-
@jknott Yep. I had tried, but that didn't work. No connection. No ping, no nothing. Reported elsewhere. ARP or what, no clue yet. Had to (1) re-allocate interfaces (in this case, swapping). Trouble is also physical access, after a reboot. Crawling into some dungeons. The box has neither monitor nor keyboard.
No, it must be pre-set.
(Don't want to whine about 'good old days', and yet, my former Soekris/m0n0wall was just running along. Power off - power on and I had access through the web interface for everything else. Have tried hard, but not found anything on the same level of ease and reliability. Well, updates and performance made it a no-go.)
