Prevent pfSense to restart every packages

Samlink

Hello, I have a secondary WAN on a 4G network that changes the IP every 9 and 1/2 hours (ISP policy). Every time pfSense senses an IP change, it restarts every package, included NUT, which then sends me a notification on telegram saying the UPS was disconnected, and after a couple more seconds it reconnect back (after NUT had completed the restart).
How can I avoid that? I undestand that package restarts are a good thing on wan changes, but I don't see the point on doing that on NUT, can I avoid that in some ways?

stephenw10

Does the WAN go down entirely when it changes? What is logged at the time?

Samlink

@stephenw10 Yes the WAN goes down for 30 to 40 seconds, everything starts with these lines on the syslog:

Jul 30 00:24:52 link check_reload_status[419]: Linkup starting igb2
Jul 30 00:24:52 link kernel: igb2: link state changed to DOWN
Jul 30 00:24:53 link php-fpm[78242]: /rc.linkup: Hotplug event detected for WAN2(opt1) dynamic IP address (4: dhcp)
Jul 30 00:24:53 link php-fpm[78242]: /rc.linkup: DEVD Ethernet detached event for opt1
Jul 30 00:24:54 link check_reload_status[419]: Reloading filter
Jul 30 00:24:56 link check_reload_status[419]: Linkup starting igb2
Jul 30 00:24:56 link kernel: igb2: link state changed to UP
Jul 30 00:24:57 link php-fpm[22432]: /rc.linkup: Hotplug event detected for WAN2(opt1) dynamic IP address (4: dhcp)
Jul 30 00:24:57 link php-fpm[22432]: /rc.linkup: DEVD Ethernet attached event for opt1
Jul 30 00:24:57 link php-fpm[22432]: /rc.linkup: HOTPLUG: Configuring interface opt1
Jul 30 00:25:26 link kernel: igb2: link state changed to DOWN
Jul 30 00:25:26 link check_reload_status[419]: Linkup starting igb2
Jul 30 00:25:30 link kernel: igb2: link state changed to UP
Jul 30 00:25:30 link check_reload_status[419]: Linkup starting igb2
Jul 30 00:25:34 link check_reload_status[419]: rc.newwanip starting igb2

And with these ones on the dhcpd log (the WAN is configured as DHCP and the IP is assigned by the modem setted on bridged mode, so on the WAN2 I received the CGNAT IP of my ISP):

Jul 30 00:24:52 link dhclient[91362]: igb2 link state up -> down
Jul 30 00:24:53 link dhclient[74886]: connection closed
Jul 30 00:24:53 link dhclient[74886]: exiting.

After some seconds the process of restarting all the packages begins and I receive the alerts:

Jul 30 00:25:42 link php-fpm[44191]: /rc.start_packages: Restarting/Starting all packages.
Jul 30 00:25:42 link php-fpm[44191]: /rc.start_packages: Stopping service nut
Jul 30 00:25:42 link upsmon[19789]: Signal 15: exiting
Jul 30 00:25:42 link upsd[25741]: User local-monitor@::1 logged out from UPS [CyberPower]
Jul 30 00:25:42 link upsmon[18871]: upsmon parent: read
Jul 30 00:25:42 link upsd[25741]: mainloop: Interrupted system call
Jul 30 00:25:42 link upsd[25741]: Signal 15: exiting
Jul 30 00:25:42 link php-fpm[44191]: /rc.start_packages: Starting service nut
Jul 30 00:25:42 link usbhid-ups[25569]: Signal 15: exiting
Jul 30 00:25:42 link upsmon[43303]: Startup successful
Jul 30 00:25:42 link upsmon[43445]: UPS [CyberPower]: connect failed: Connection failure: Connection refused
Jul 30 00:25:42 link upsmon[43445]: Communications with UPS CyberPower lost
Jul 30 00:25:42 link usbhid-ups[44245]: using 'battery.charge' to set battery low state
Jul 30 00:25:42 link usbhid-ups[44245]: using 'battery.runtime' to set battery low state
Jul 30 00:25:42 link usbhid-ups[45413]: Startup successful
Jul 30 00:25:43 link upsd[67979]: listening on ::1 port 3493
Jul 30 00:25:43 link upsd[67979]: listening on 127.0.0.1 port 3493
Jul 30 00:25:43 link upsd[67979]: Connected to UPS [CyberPower]: usbhid-ups-CyberPower
Jul 30 00:25:43 link upsd[68131]: Startup successful

stephenw10

Ah, it actually loses link, that's interesting. I can see why it might be restarted it can send and listen on any interface, including the one that is bounced.

I would probably ask in the UPS sub directly. The NUT package developer is very active there. There may not be an option there though if all packages are restarted.

Gertjan

@Samlink

See also here :

https://forum.netgate.com/topic/189406/wan-periodically-rebooting

Do you also use a modem WAN 4G and is it under "Chinese control" ?

As said over there : if you know that the mdoem pulled its LAN (== WAN pfSEnse) down :

ask it not to do so, or
put a switch between pfSense and the modem.

Btw : many process are bound to (all) interfaces. When their setup changes, or they just vanish, processes are restarted.
Your mission, as an admin, is trying to chose the right equipment that doesn't pull down its connection.
Be aware that pfSense will pull the plug if the connection because bad : you can disable this if you want to.

Samlink

@Gertjan said in Prevent pfSense to restart every packages:

Do you also use a modem WAN 4G and is it under "Chinese control" ?

What do you mean with chinese control?
Yes, my 4G modem is chinese (Dlink) but what does that mean? It's the ISP that renovates the IP on a fixed schedule, and that causes the interface to go down for a bit.

I cannot change this behaviour on the modem, and I want to avoid putting a switch in between just for this.
I understand that is better to let pfsense restart the packages, so I think I'm going to ditch the bridge mode and instead go with the double NAT and set pfsense as a normal dhcp client on the modem lan interface. This way it's the modem that is getting disconnected for the IP renewal.

Gertjan

@Samlink said in Prevent pfSense to restart every packages:

What do you mean with chinese control?

Or 'radio wave interference' ^^

@Samlink said in Prevent pfSense to restart every packages:

I cannot change this behaviour on the modem, and I want to avoid putting a switch in between just for this.

So, again, discover who's doing what, and why, use a switch for some time and figure it out.

Samlink

@Gertjan I think I'm not following you...
Is this a troll or what? Chinese radio interference, what are you talking about?

I know who is doing that, and it is my internet service provider, it's a known fact that every 9.5 hours the IP changes.
The thread was about the behavior of pfsense when the IP changes. Nothing strange about it. I've also thought about a workaround without writing on the UPS sub-thread, so this can be considered closed for me

Gertjan

@Samlink said in Prevent pfSense to restart every packages:

Is this a troll or what? Chinese radio interference, what are you talking about?

Not a troll.
I'm trying to understand myself why on a big scale all huwaie equipment (5G) is removed. For me, as a consumer, I've no issues what so ever. But if ISPs and phone companies start to ditch multi xxxx worth of equipment "just like that", I'm wondering.
This is what I see here (France).
In the US things are different, but they have issues with the same equipment.
Elsewhere : I don't know.
Why : I don't know. I run a hotel, not some telecom company.

Radio wave interference : two of my best friends use 4G 'routers', and the notion of - after one hour, or several days : all lights are green, the GUI says :"connected" but nothing comes in, nothings goes out.
I've solved the issue somewhat by setting up an auto reboot every night a 3 am.
The thing is : these kind of connection are hard to debug, and consumer routers like this don't have any 'ssh' access with details carrier information.

@Samlink said in Prevent pfSense to restart every packages:

every 9.5 hours the IP changes.

That's a horror.
If possible, don't even argue with them. Leave them.

@Samlink said in Prevent pfSense to restart every packages:

The thread was about the behavior of pfsense when the IP changes.

I get it, I understand your question.
pfSense isn't your off the shelves SOHO 'get me at Marties' router, like so many dlink, tplink, etc etc routers out there.
Its a full FreeBSD OS, and overloaded with a max of features - probably to many.
Real routers/firewalls normally never have their interfaces going down. Up times for weeks, months or more are possible.
All my LAN's have main switches, and from these witches I cabled up the entire building. Must Ethernet plugs are "connected always". These devices can reboot, or power down, but that will not influence pfSense.
My ISP router is, like my pfSense, is fed by an UPS. It's stays up and is connected for weeks if not months.
It's pretty common that many process restart when a physical interface goes up/down, especially for firewall routers.

Samlink

@Gertjan you are completely offtopic with the Huawei thing here. No one mentioned Huawei in this thread so why bring it up now?
I had a "problem" with a Dlink consumer 4G router and pfSense, nothing compared with ISP grade Huawei equipment, and surely it has nothing to do with some conspiracy theories. So please stop with it.

@Gertjan said in Prevent pfSense to restart every packages:

That's a horror.
If possible, don't even argue with them. Leave them.

As for this, it doesn't bother me at all, it's a fallback/secondary connection and I don't mind if it changes every 9 hours or so. It's for my home connection, not for a company one.

michmoor

@Gertjan Is it common to put a switch in between firewalls(at least not ones that are set up for HA).
I understand why it restarts processes but its terribly annoying that a bounced link causes system-wide issues. This honestly feels like a problem. I can deploy (insert vendor) firewall and drop links happen and dhcpd doesn't restart.
Worse - I have an open redmine where IPsec tunnels restart when it senses an interface change although there isn't one. Like i said, this feels like a way bigger issue.

Gertjan

@michmoor said in Prevent pfSense to restart every packages:

I can deploy (insert vendor) firewall and drop links happen and dhcpd doesn't restart.

dhcpd - worlds most known one, written by ISC, is 'bound' to an interface, always a LAN , and will stop running when the interface goes down, as it doesn't make sense to run a (this) process on a device that doesn't exist on a system.
When this LAN goes up again, and a dhcpd was set up to 'serve' it, it will start.
If these events follow each other quickly, then you might call this a restart. But it isn't. The device goes down = process goes away. The the other way around when the device comes up.
Milli seconds for us is 'years' later for pfSense ^^

That said, I know what you mean.
When pfSense is sensing an interface device event, down or up, a lot happens. And yes, again, I agree, without any thoughts about it, I do qualify - just a feeling - this as 'to much' as I can see race conditions lurking ahead. So I said to myself : all this is needed to construct a connection in a correct defined order. This is a router after all, not a switch.
I've made it a non issue by protecting all devices surrounding pfSense (included), with a UPS. Case closed for me ^^

stephenw10

It could definitely use a more nuanced approach to interface handling. Much of that code is from a time when there were fewer options and the big hammer of restart everything was all that could be done. Unfortunately it's tied into so much that improving or replacing it is.... non-trivial!

michmoor

@stephenw10 I get it completely. Legacy code. Technical debt. Limited resources. If we have all the time in the world then all the things can be done