Disable start up interface reassignment

Patch

How do I configure pfsense to disable interface reassignment during start up or does it require a new pfsense feature?

To be clear I would like to retain

the initial interface assignment when running pfsense initial set up
the ability to manually invoke a interface reassignment from the console when I have changed the interfaces to pfsense

It is only want to stop the reassignment when pfsense restarts for other reasons and pfsense decides the currently defined interfaces need to be reassigned.

Use case: Without this change fault finding efficiency is compromised during failure elsewhere in the network

Restarting software during fault correction / isolation is a relatively common procedure
During fault finding the automatic interface reassignment means a fault in any interface will break all interfaces to pfsense.
I could not find a way of disabling the interface reassignment once triggered at the console
Secondary loss off all interfaces during fault conditions complicates uploading a prior saved configuration with the pre-existing correct interface assignment
Manual interface re-assignment is some times not possible when there is a fault on one of the interfaces (eg WAN during start up)

Jarhead

@patch No configuration necessary.
pfSense doesn't reassign interfaces.

Are you using a VM maybe?

bmeeks

pfSense does not, as a matter of routine, require interface reassignment with each reboot. The only reason this would happen is if the interface is not present at reboot or it has changed its name as presented to the OS.

One way to have this happen is with USB NICs. When using USB NICs, it is possible the USB driver has not finished initializing completely before the scan for active interfaces is performed. In that scenario, the "missing USB NIC" would kick off the interface reassignment routine.

There may be some hackery you could do in the pfSense startup scripts to stop this, but it would not be a supported configuration and would also not survive system future system updates.

I would determine the root cause for the reassignment routine kicking off at each reboot. If it is USB NICs, then swap those out to actual PCIe or similar hardware NICs.

Patch

@bmeeks said in Disable startup interface reassignment:

pfSense does not, as a matter of routine, require interface reassignment with each reboot. The only reason this would happen is if the interface is not present at reboot

You are correct but the use case I am discussing is behaviour during a fault.

Faults can occur in many locations.
A simple fix for a transient fault is to restart the software such as pfsense
If the fault was within pfsense then doing so may well resolve the transient error.

However if the fault was with an interface to pfsense, (such as it not being available a boot time) then we have a problem

The pre existing fault is still there
pfsense forces an interface reassignment as it now notices something has changed (the initial fault being debugged)
My desired pfsense behaviour is for pfsense the leave the interface assignments alone. I'm happy for the console to show an alert "Interface configuration not valid, please use option to re assign interfaces". That way debugging of the initial fault can continue or if I have intentionally changed the interfaces I can update pfsense configuration.
Unfortunately at this point pfsense forces an interface reassignment which creates a secondary fault. The degradation in pfsense functionality is most acute if the initial fault was in a low numbered interface (such as the WAN) and before the pfsense management interface. In that case there is no valid reassignment possible and the normal Management interface to pfsense is lost complicating backup configuration restoration.

So yes I'm aware in normal circumstances pfsense does not force interface reassignment however for fault tolerant behaviour, the only time I want to be required to do an interface reassignment is:

When pfsense if first installed from the console and there is no pre existing configuration
When I'm at the console and select the reassign interfaces option, because I have changed the interfaces

When pfsense notices the interface configuration is not identical to what was configured then it should use flag an issue on the console but continue to use the configuration of the interfaces which have not changed. Doing so minimises secondary failure.

@bmeeks said in Disable startup interface reassignment:

I would determine the root cause for the reassignment

Yes that was what I was trying to do.
In this particular case it is probably a ISP cable modem fault or upstream ISP fault / reconfiguration.

But that's not the issue I'm trying to address here. The issue is fault tolerance and associated system repair time for future faults.

SteveITS

@patch If a NIC dies the interfaces may get renumbered, so pfSense doesn’t want to connect devices to the wrong interface. It doesn’t know what to do at that point, so stops.

Patch

@steveits
I do not think the primary fault is in the WAN NIC on pfsense computer.

The primary problem is the more likely to be ISP cable modem is flapping and temporally takes it's NIC off line while the cable modem restarts. When this happens while pfsense is booting then pfsense sees no connection to the wan and forces an interface reassignment. Doing so takes out every other interface.

Looks like this is a fundamental limitation of pfsense.

SteveITS

@patch Hmm. Can’t say I’ve tried. The timing would have to be right if that’s the case. I’d have expected pfSense to maintain the assignment.

A $20 switch would fix that, though introduce another failure point.

bmeeks

@patch:
I'm not sure that what you want is possible due to how the underlying OS responds to missing interfaces.

pfSense works with the physical interface name supplied by FreeBSD. So if your firewall has 3 Intel e1000 interfaces, the FreeBSD OS will number them em0, em1, and em2. These are the interface names passed to pfSense and are what it eventually stores in config.xml and would assign to the friendly names WAN, LAN and DMZ.

But if for some reason one of those Intel NICs disappears - for example assume it was em1 - then the FreeBSD OS will, on boot, likely renumber the interfaces to be em0 and em1 since it sees only two now with the failure of the former em1.

But in reality, in this example because em1 died, what FreeBSD is now calling em1 is really em2 when all the NICs are present. So, now any firewall rules based on the old em1 interface would actually be applied to the em2 interface. That's likely not what is desired. Exactly how the renumbering pans out will be governed by the precise NIC failure. But if the NIC totally disappears to the system, then the sequential numbering FreeBSD does would be a potential issue. That's why pfSense stops when the number of NICs returned by FreeBSD at boot-up does not match what is configured in config.xml, or if any of the physical names have changed (say moved from emX to igcX, for example).

Here is a write-up from Netgate describing how the interface numbering and auto-detection works: https://docs.netgate.com/pfsense/en/latest/install/assign-interfaces.html.

Gertjan

@patch said in Disable start up interface reassignment:

there is a fault on one of the interfaces

That happens often ?

I do remember, as every body else, having a realtek interface dying on me.
happens ones, in my entire live.
You have interfaces failing all over ?

For me, when you add remove hardware on the system, you should also make the system aware what you want with the 'new' configuration. The first comes with the second.

So, when you do 'hardware' you confirm that in the software (setup).
While doing hardware things, the console access isn't far anyway.

This will also show you that newly added hardware is recognized at boot, etc.

See it like this :
We, as humans, give interfaces 'labels or names, like WAN and LAN, DMZ etc.
Internally, driver names are used em0, em1, igc0, ix1 etc.
At a lower level, numbers are used.

When you add or remove an interface, actually any device, these can get renumbered,, and more as 1 can exist for a type of device (interface).
Drivers are loaded at random time, as soon as the hardware detection method found them.
So, internal numbering can change. And here comes the issue :
What happens when WAN is now 1 and LAN is now 0 ?
They get turned around, with the firewall rules and everything.

Now you have a huge security issue.
( because you didn't re assign )

If you really want to stop the re assignment : stop it.
It reading start here : /etc/rc.bootup, you'll find what you need.

Just be warned.

Patch

@gertjan said in Disable start up interface reassignment:

That happens often ?

I had changed no hardware.
The fault was with my external internet provider / cable connection resulting in the ISP software restarting the ISP modem on my premises a lot.

To investigate if the loss of internet was an issue with my equipment or the ISP & their wholesaler (NBN) I looked on the ISP phone app to see it they could see their modem on my premises (they could not). After I used their phone app to reset the port they used to access their modem on my premises their connection to their modem on my premises improved by my internet access was still down.

Restarting pfsense at that time (no hardware changed and external fault actually corrected unbeknown to me), pfsense could not restart with the known good configuration. As a result, rebooting pfsense resulted in loss of all lans as wall as the prior wan fault. An inconvenient situation during fault finding.

@gertjan said in Disable start up interface reassignment:

Internally, driver names are used em0, em1, igc0, ix1 etc.
At a lower level, numbers are used.

Which is the crux of the problem during fault finding with a pfsense router.
The numbering of the internal names are some what random but at least constant if the system starts up in a constant fashion. Which is fine until it is not (a fault somewhere or a USB drive with a start up race condition).

@gertjan said in Disable start up interface reassignment:

At a lower level, numbers are used

For pfsense resilience in fault conditions a very useful enhancement would be for the configuration to also record the lower level address information.

That way functioning interfaces could be assigned to the appropriate rules / configuration within pfsense. Doing so stops a single transient fault on one interface stopping all other interfaces. Resulting in a dramatic improvement in fault tolerance and subsequent reduction in debug time.

Btw
For now, my internet is restored.

A switch was added between the ISP modem & pfsense as pfsense locks up if the line fluctuates an the "wrong" time.
My pfsense runs on a Proxmox single board computer. All nic pfsense uses are passed through to pfsense. Restarting Proxmox resulted in a very slow pfsense start up but now works again.

So current fault is resolve however the fragility of pfsense to fault conditions and the resulting debug time still concerns me.

bmeeks

@patch:
What model NIC is in your firewall on the WAN interface?

I have an SG-5100 with 6 ports, but I only use three of them currently. The other 3 ports have nothing plugged in and thus no link. But on boot they all show up as available ports, but just with no active link.

You seem to say that your WAN NIC disappeared from the pfSense system when there was no active link. That would be unusual if I am correctly understanding your description of your problem.

If you have a total and complete hardware failure such that the NIC does not even show up during the POST hardware scan, then upsetting the interface numbering scheme is expected. But if the NIC shows up in the hardware scan, it should not get "lost". It should only show with no link if the other side of the connection is down (such as your ISP modem).

Gertjan

@patch

Like @bmeeks, I can start or reboot my pfSense with the WAN cable connected - or connected to the ISP router beeing kept off line or not switched on.
pfSense will still boot, find all it's network interfaces, and work just fine.

There will be two issues :

all LAN's have no Internet connectivity.
when you visit https://192.168.1.1 to check up with the pfSense dashboard, you'll see that the WAN is down = no link. The WAN interface is still there, of course.

You will notice that the access to the man dashboard is slow.
This is because some of info showed on the dashboard comes from the Internet, like the list with packages and their 'upgradable' state.
Just be patient, the dashboard will show up.

If the WAN interface isn't there any more .... then you have a (hardware) issue with the pfSense device itself.
In this case there won't be a GUI dashboard (web interface) that works, as all network interfaces have to be re assigned first.

stephenw10

The decision to drop to the interfaces assignment prompt is deliberate and for security reasons.

As others have said if interfaces are added or removed resulting in an assigned interface not being present at boot the order of the other interfaces cannot be guaranteed. In that circumstance it's preferable to fail to boot rather than incorrectly connect network segments that should not be.

But, also yes, simply disconnecting a WAN cable should not trigger that. It's only triggered by an assigned interface not being present in the firewall.

Steve

Justaguy 0

This happens to me a lot and it's from using sr-iov assigned interfaces. Basically I can't reboot as the interfaces mac addresses are assigned with the interface from a pool of VFs. 1 mac address changes and I'm having to remap all of the interfaces! Also, the setup script includes tunnel, tap, bridge and ovpns interfaces, some of which weren't mapped to a physical address.

Is there a way to find the old interface configuration? Sometimes I'm lucky and remember the interface name to which then pfsense loads the correct configuration and rule set. If not I'm starting from scratch again which is becoming a pain.

SteveITS

@Justaguy-0 You can find them in a saved config file, in <interfaces>.

Justaguy 0

Thank you,

That should be enough to recreate them. Is there anyway to see past mappings of optX to mac address?

stephenw10

Unlikely since they would all have been replaced by the current values.

I'm unclear exactly what you are seeing here though. You have to re-assign the interfaces at each boot? I'd still expect the same number of interfaces with the same names even if all the MACs change?

Justaguy 0

Same number of interfaces. It's not every reboot but every now and again the assignment script starts and I have to reassign the interfaces. Some of my interfaces are bridges so the mac won't change. Some are pooled VFs when they are dynamically assigned. Why my hypervisor doesn't keep assigning the same VFs to pfSense I have yet to figure out.

stephenw10

The assignment script only starts if it fails interface check. That means that at least one interface that is assigned in the config file doesn't exist on the system.

So it shouldn't matter how the hypervisor presents them or what MAC it uses as long as the correct number of NICs of any type are present.

Justaguy 0

Thanks, good to know. I'm going to keep an eye on the number of interfaces between reboots.