Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    How to fix interface assignment at multiple identical NICs?

    Scheduled Pinned Locked Moved General pfSense Questions
    11 Posts 3 Posters 969 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      digard
      last edited by

      For the last days I have been fighting with this. I want to clone a machine, have more identical boxes. Backup/Restore does a fine job, except that for reasons unknown to me, on one box at start ue0 comes up as LAN, so that I can configure ue1 as DMZ. Backup and Restore on another in principle identical hardware fails, because ue1 comes up as LAN for reasons unknown to me.

      Therefore, it is close to impossible for me to produce a drop-in replacement: the cables will have to go into different plugs, and even the firewall rules don't fit any longer.

      And with bad luck, I can't even use the webconfigurator because I end up in the wrong subnet.
      What is the trick, and which is the hook, that allows me to define a certain interface to ue0 (re0, em0, etc.) as 'second' interface, that is LAN, despite of what the boot process finds first?
      I wouldn't bet, but I even seem to have noticed that the allocation can be changed during a simple reboot. At least, that how it looked to me; once I had a working box reboot, and after the reboot somehow LAN and DMZ (ue0 and ue1) had changed places.

      JKnottJ 1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        The short answer there is: don't use USB NICs. ๐Ÿ˜‰

        That's unhelpful but true. If you use PCI NICs they always come up in the same order. And since 2.5 you can even set the order they come up in it you want.
        With USB NICs it's all about timing and that can change. I don't think there's a quirk you can apply even.

        The assigned interface doesn't change in the config though and the rules etc all follow that. So only the physical NIC would change, if it did.

        What hardware are you actually using there?

        Steve

        1 Reply Last reply Reply Quote 1
        • D
          digard
          last edited by

          Thanks a lot. Sad. Actually, I received a bunch of former thin clients, fanless, no moving parts, sufficiently powerful, small SSD, relatively low power requirements. And USB 3.0! Naturally, one NIC. And I also have a handful of lenovo USB3 to Gigabit Ethernet adapters. I thought their logical marriage was to continue as network appliances.

          I am not yet clear about the inherent logic of allocation at reboot. I had thought that they'd just switch physical network interface, and one only had to swap cables. But that doesn't seem to be the case. Once they change order, neither from LAN nor DMZ can I ping or at any way connect to the box, though the clients in both networks receive their IPs from there, very properly and correctly. Consequently no SSH. So I have to go to the box, plug in a monitor and keyboard, and just change the interface allocation (1) by swapping LAN and DMZ. Then everything comes up. That's the part that I hate.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            It could be some other timing issue, like the USB NICs are not ready when the interfaces are assigned at boot perhaps.
            If the DHCP is working and handing out IPs in the correct range though that seems unlikely.
            In the config the rules, subnet and dhcp are all linked to the same logical interface. If one is working they all should so I would look for something else preventing it. An error loading the ruleset at all for example.
            I imagine if you simply resaved the interfaces with the same order they would also come back up.

            Steve

            1 Reply Last reply Reply Quote 0
            • D
              digard
              last edited by

              (I very much appreciate your input, a thanks in between!)
              Okay, alas worse. I tried under 'controlled' conditions: power off - remove power - put power back - power on. (Remove power simply because I don't know how far some NIC is still on for WoL.) And then, the interfaces came up configured in reverse order.
              I had never used USB-NICs before, and never ran into this. Though, I can't think I'm the one and first human who ever touched this as 'terra incognita'? :-)
              This is what dmesg makes from it:

              ....
              uhub5: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
              Trying to mount root from ufs:/dev/ada0s1a [rw,noatime]...
              Root mount waiting for: usbus0 CAM usbus1 usbus2 usbus3 usbus4 usbus5
              uhub5: 4 ports with 4 removable, self powered
              ....
                TSC: P-state invariant
              ure0 on uhub5
              ure0: <Lenovo Thinkpad USB LAN, class 0/0, rev 3.00/30.00, addr 1> on usbus0
              ure1 on uhub5
              ure1: <Lenovo Thinkpad USB LAN, class 0/0, rev 3.00/30.00, addr 2> on usbus0
              miibus1: <MII bus> on ure1
              miibus2: <MII bus> on ure0
              rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus1
              rgephy2: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus2
              rgephy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
              rgephy2:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
              ue0: <USB Ethernet> on ure1
              ue0: Ethernet address: 3c:18:a0:07:75:7a
              ue1: <USB Ethernet> on ure0
              ue1: Ethernet address: 3c:18:a0:08:58:40
              ....
              

              Looks like first-came-first-served to me.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by stephenw10

                Mmm, I'm not sure there's anything you can do about that. ๐Ÿ˜•

                There are some scripts for FreeBSD I see people have written to address this but nothing in pfSense directly.

                D 1 Reply Last reply Reply Quote 1
                • D
                  digard @stephenw10
                  last edited by

                  @stephenw10 Yep. Found some scripts, like id your card, and then run some check at boot up. I mean, for a client machine that might be okay, but I don't feel all too happy on a mission-critical box.
                  I can already see myself, in a few month's time, sleepy or half-drunk, pulling out that USB, plug another one, and nothing works.
                  By the way, I even tried another usbhub, 100 Mbps, but it didn't make it 100% predictable either. And then I saw one chap having similar problems with re-s.
                  I think it's a RFE for FreeBSD. Strange enough, many years ago on my Soekrisbox with m0n0wall there was a 1000% clear allocation of WAN, LAN, DMZ to the Ethernet ports.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    @digard said in How to fix interface assignment at multiple identical NICs?:

                    Strange enough, many years ago on my Soekrisbox with m0n0wall there was a 1000% clear allocation of WAN, LAN, DMZ to the Ethernet ports.

                    Yes, if you have PCI(e) based NICs there is no problem, they always come up in the same order.

                    There can be an issue if you add more NICs with the same driver but in FreeBSD 12 you can work around that with PCI device wiring to force it.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • JKnottJ
                      JKnott @digard
                      last edited by

                      @digard

                      Since you're using USB NICs, why not just swap them, so the order matches what you want?

                      PfSense running on Qotom mini PC
                      i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                      UniFi AC-Lite access point

                      I haven't lost my mind. It's around here...somewhere...

                      D 1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Because at the next boot that order may have changed.

                        1 Reply Last reply Reply Quote 0
                        • D
                          digard @JKnott
                          last edited by

                          @jknott Yep. I had tried, but that didn't work. No connection. No ping, no nothing. Reported elsewhere. ARP or what, no clue yet. Had to (1) re-allocate interfaces (in this case, swapping). Trouble is also physical access, after a reboot. Crawling into some dungeons. The box has neither monitor nor keyboard.
                          No, it must be pre-set.
                          (Don't want to whine about 'good old days', and yet, my former Soekris/m0n0wall was just running along. Power off - power on and I had access through the web interface for everything else. Have tried hard, but not found anything on the same level of ease and reliability. Well, updates and performance made it a no-go.)

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.