Random disconnect

[michmoor]

Howdy everyone,
Attaching a log file here: https://pastebin.com/FdgY7MbZ

The situation is that i was going down for bed early this morning where I started getting monitoring alerts that all my servers went down even pfSense (SG-6100). My monitoring server sits in the cloud so i figured it was some cloud network connectivity issue but then i started seeing gateway monitoring alerts come through email with latency spikes so when i loged into my pfsense (ssh) i saw all the log messages. Some ive never seen before.
A worrisome message i kept seeing is "Restarting packages." This indicates there was some software failure that occurred.

Somehow the firewall had a blip but im trying to determine the cause. Any insight would be helpful.

[bmeeks]

@michmoor said in Random disconnect:

A worrisome message i kept seeing is "Restarting packages." This indicates there was some software failure that occurred.

Not necessarily software. Packages are auto-restarted for several reasons, but the most common is a firewall network interface cycling (going down and coming back up, for example). Because many packages tie themselves to a firewall interface (the two IDS/IPS packages are examples, but there are many others including unbound), they will usually need to be restarted when something changes on an interface. The assumption when an interface cycles is that perhaps the previous IP and/or gateway changed and the packages would need to see that update.

dpinger will also initiate this restart of packages sequence if it thinks the monitored interface went down. Super long ping reply times could fool dpinger into thinking an interface was offline, so it would start the recycling in an attempt to restore connectivity.

The fact you saw gateway monitoring alerts in your pfSense logs indicates that something happened between your firewall and the rest of the world. That loss of connectivity would result in the "restart all packages" command getting issued automatically by pfSense.

[michmoor]

@bmeeks said in Random disconnect:

The fact you saw gateway monitoring alerts in your pfSense logs indicates that something happened between your firewall and the rest of the world. That loss of connectivity would result in the "restart all packages" command getting issued automatically by pfSense.

Interesting. Is there a way to check if my WAN interface went down? Some kind of status that says "up for x amount of time"?

[michmoor]

@bmeeks

Yep...Interface status change.

Nov 13 01:16:20 GAFW kernel: ix3: link state changed to DOWN
Nov 13 01:16:20 GAFW check_reload_status[2674]: Linkup starting ix3
Nov 13 01:16:30 GAFW check_reload_status[2674]: Linkup starting ix3
Nov 13 01:16:30 GAFW kernel: ix3: link state changed to UP

Is there a place to see this in the GUI ?

[bmeeks]

@michmoor said in Random disconnect:

Is there a way to check if my WAN interface went down? Some kind of status that says "up for x amount of time"?

The only way I know of is to check the system log. The gateway alarms will be logged there. You can also seem them with details under the Gateways tab of the STATUS > SYSTEM LOGS page.

[bmeeks]

@michmoor said in Random disconnect:

Is there a place to see this in the GUI ?

There is a GUI page with several tabs to view system logs. So "yeah", there is a way to see this in the GUI the way I would interpret it.

[michmoor]

@bmeeks
I see it now.

Thanks Bill. Also i didnt know about the packages tied to interface status part so thats really good to know.

[bmeeks]

@michmoor said in Random disconnect:

i didnt know about the packages tied to interface status

Think about it logically -- many services (packages) need to know about active interfaces and what their status and IP settings are. So, there has to be a mechanism to let packages know something has changed with interfaces. pfSense uses a sort of big hammer here -- simply restart all the packages when an interface changes. That causes them to behave like an initial boot-up and they all read the information they need/require again and configure themselves accordingly.

[stephenw10]

Yup there could be more finesse there. But most pfSense installs rarely see an interface link status change so not often a big issue.

[bmeeks]

You can also see a series of "restarting packages" commands over time if your dpinger-monitored interface is down for an extended period. The interface will be cycled, but if dpinger still sees no connectivity to the monitored IP, then it will restart the interface again. This will continue until the interface's connectivity to the monitor IP is restored, or you tell dpinger and pfSense to always consider the gateway as "up".

[michmoor]

@stephenw10 @bmeeks
Is that with ANY interface state change or only when an interface is a wan-type?

A flapping interface is not uncommon. So if i have a DMZ leg that is flapping does that mean my LAN -> WAN flows will be impacted? Essentially 2x interfaces that have nothing to do with DMZ will see an outage?

[bmeeks]

@michmoor said in Random disconnect:

@stephenw10 @bmeeks
Is that with ANY interface state change or only when an interface is a wan-type?

A flapping interface is not uncommon. So if i have a DMZ leg that is flapping does that mean my LAN -> WAN flows will be impacted? Essentially 2x interfaces that have nothing to do with DMZ will see an outage?

I believe it is either a physical link status change or the execution of ifconfig up or ifconfig down that triggers the restart all packages command. And I think dpinger will trigger that ifconfig up/down command when it fails to reach the monitored IP within the configured time window. I have never examined all the PHP and shell script code for this in pfSense to find every possible trigger.