Multiple issues, firewall freezes and whole network goes down.

Laxarus

@stephenw10 so far nothing crashed yet with static IP.
I am guessing this is a very rare problem since you have to have pppoe DSL (most uses fiber these days) and configured the network to be able to access modem behind WAN via DHCP.
I will continue monitoring.

There is no option that suggests to mark the gateway as internal network only. Maybe checking the boxes for "Disable Gateway Monitoring", "Disable Gateway Monitoring Action", and "Do not add static route for gateway monitor IP address via the chosen interface" might achieve a similar result but I dont want to try these for now.
Still no idea for the exact use case of "Use non-local gateway" option.

stephenw10

You would have to use the DHCP advanced options field to force the dhcp client to ignore the gateway passed by the server. So adding supersede routers in the Option modifiers field should do it.

But I would just use a static IP here.

JonathanLee

@Laxarus Your traffic shaper is that required? That could be configured incorrectly.

https://forum.netgate.com/topic/171842/queue-management-algorithms-differences

Laxarus

@JonathanLee said in Multiple issues, firewall freezes and whole network goes down.:

https://forum.netgate.com/topic/171842/queue-management-algorithms-differences

My main interface is on 2x25G LAGG and LAGG is not supported with traffic shaper so other than bufferbloat nothing is set there.

Laxarus

@stephenw10 After setting the static IP, I did not get this for a long while.
However, today Jan 9th at 6 AM, the modem performed its weekly scheduled reboot. The same issue again. Everything went down. It stayed down until I performed a hard power reset since the console over IPMI was unresponsive.

At this point, the only thing suspicious is the problem with multicast on the logs but I don't see how it is relevant to the WAN.

system.log.0

an  9 06:00:44 FIREWALL check_reload_status[635]: Linkup starting igb1
Jan  9 06:00:44 FIREWALL kernel: igb1: link state changed to DOWN
Jan  9 06:00:45 FIREWALL php-fpm[78698]: /rc.linkup: Hotplug event detected for MODEM(opt2) static IP address (4: 192.168.0.2)
Jan  9 06:00:45 FIREWALL php-fpm[78698]: /rc.linkup: DEVD Ethernet detached event for opt2
Jan  9 06:00:46 FIREWALL dhcpleases[64223]: Could not deliver signal HUP to process 66132: No such process.
Jan  9 06:00:48 FIREWALL check_reload_status[635]: Linkup starting igb1
Jan  9 06:00:48 FIREWALL kernel: igb1: link state changed to UP
Jan  9 06:00:55 FIREWALL ppp[99256]: caught fatal signal TERM
Jan  9 06:00:55 FIREWALL ppp[99256]: [wan] IFACE: Close event
Jan  9 06:00:55 FIREWALL ppp[99256]: [wan] IPCP: Close event
Jan  9 06:00:55 FIREWALL ppp[99256]: [wan] IPCP: state change Opened --> Closing
Jan  9 06:00:55 FIREWALL ppp[99256]: [wan] IPCP: SendTerminateReq #4
Jan  9 06:00:55 FIREWALL ppp[99256]: [wan] IPCP: LayerDown
Jan  9 06:00:56 FIREWALL php-cgi[91601]: rc.kill_states: rc.kill_states: Removing states for IP {redacted}/32
Jan  9 06:00:58 FIREWALL check_reload_status[635]: Reloading filter
Jan  9 06:00:58 FIREWALL check_reload_status[635]: Reloading filter
Jan  9 06:00:58 FIREWALL php-fpm[592]: /rc.linkup: Hotplug event detected for MODEM(opt2) static IP address (4: 192.168.0.2)
Jan  9 06:00:58 FIREWALL php-fpm[592]: /rc.linkup: DEVD Ethernet attached event for opt2
Jan  9 06:00:58 FIREWALL php-fpm[592]: /rc.linkup: HOTPLUG: Triggering address refresh on opt2 (igb1)
Jan  9 06:00:58 FIREWALL check_reload_status[635]: rc.newwanip starting igb1
Jan  9 06:00:58 FIREWALL ppp[98752]: Multi-link PPP daemon for FreeBSD
Jan  9 06:00:58 FIREWALL ppp[98752]:  
Jan  9 06:00:58 FIREWALL ppp[98752]: process 98752 started, version 5.9
Jan  9 06:00:58 FIREWALL ppp[98752]: waiting for process 99256 to die...
Jan  9 06:00:59 FIREWALL ppp[98752]: waiting for process 99256 to die...
Jan  9 06:00:59 FIREWALL php-fpm[14172]: /rc.newwanip: rc.newwanip: Info: starting on igb1.
Jan  9 06:00:59 FIREWALL php-fpm[14172]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.0.2) (interface: MODEM[opt2]) (real interface: igb1).
Jan  9 06:01:00 FIREWALL rc.gateway_alarm[11168]: >>> Gateway alarm: VPNAC_WG (Addr:10.11.0.1 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Jan  9 06:01:00 FIREWALL check_reload_status[635]: updating dyndns VPNAC_WG
Jan  9 06:01:00 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Jan  9 06:01:00 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Jan  9 06:01:00 FIREWALL check_reload_status[635]: Reloading filter
Jan  9 06:01:00 FIREWALL ppp[98752]: waiting for process 99256 to die...
Jan  9 06:01:01 FIREWALL ppp[98752]: waiting for process 99256 to die...
Jan  9 06:01:01 FIREWALL php-cgi[91601]: rc.kill_states: rc.kill_states: Removing states for interface pppoe0
Jan  9 06:01:01 FIREWALL check_reload_status[635]: Rewriting resolv.conf
Jan  9 06:01:01 FIREWALL ppp[99256]: [wan] IFACE: Removing IPv4 address from pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address
Jan  9 06:01:01 FIREWALL ppp[99256]: [wan] IFACE: Down event
Jan  9 06:01:01 FIREWALL ppp[99256]: [wan] IFACE: Rename interface pppoe0 to pppoe0
Jan  9 06:01:01 FIREWALL ppp[99256]: [wan] IFACE: Set description "WAN"
Jan  9 06:01:01 FIREWALL php-fpm[41860]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Jan  9 06:01:01 FIREWALL php-fpm[41860]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPNAC_WG.
Jan  9 06:01:01 FIREWALL php-fpm[592]: /rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' 
Jan  9 06:01:01 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Jan  9 06:01:02 FIREWALL ppp[98752]: waiting for process 99256 to die...
Jan  9 06:01:02 FIREWALL ppp[99256]: [wan] IPCP: SendTerminateReq #5
Jan  9 06:01:02 FIREWALL dhcpleases[60027]: Could not deliver signal HUP to process 68165: No such process.
Jan  9 06:01:03 FIREWALL ppp[98752]: waiting for process 99256 to die...
Jan  9 06:01:03 FIREWALL ppp[99256]: [wan] Bundle: Shutdown
Jan  9 06:01:03 FIREWALL ppp[99256]: [wan_link0] Link: Shutdown
Jan  9 06:01:03 FIREWALL ppp[99256]: process 99256 terminated
Jan  9 06:01:04 FIREWALL rc.gateway_alarm[80528]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Jan  9 06:01:04 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Jan  9 06:01:04 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Jan  9 06:01:04 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Jan  9 06:01:04 FIREWALL check_reload_status[635]: updating dyndns wan
Jan  9 06:01:04 FIREWALL php-fpm[592]: /rc.linkup: The command '/sbin/ifconfig 'pppoe0' description 'WAN'' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist' 
Jan  9 06:01:04 FIREWALL php-fpm[592]: /rc.linkup: The command '/sbin/ifconfig 'pppoe0' -staticarp ' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist' 
Jan  9 06:01:04 FIREWALL php-fpm[592]: /rc.linkup: The command '/usr/sbin/arp -d -i 'pppoe0' -a > /dev/null 2>&1 ' returned exit code '1', the output was '' 
Jan  9 06:01:04 FIREWALL ppp[98752]: web: web is not running
Jan  9 06:01:04 FIREWALL ppp[98752]: [wan] Bundle: Interface ng0 created
Jan  9 06:01:04 FIREWALL ppp[98752]: [wan_link0] Link: OPEN event
Jan  9 06:01:04 FIREWALL ppp[98752]: [wan_link0] LCP: Open event
Jan  9 06:01:04 FIREWALL ppp[98752]: [wan_link0] LCP: state change Initial --> Starting
Jan  9 06:01:04 FIREWALL ppp[98752]: [wan_link0] LCP: LayerStart
Jan  9 06:01:04 FIREWALL ppp[98752]: [wan_link0] PPPoE: Connecting to ''
Jan  9 06:01:04 FIREWALL kernel: ng0: changing name to 'pppoe0'

stephenw10

Hmm, those logs really don't show an error. The upstream device dropped the link and re-linked 4s later. The PPP session reconnected.

If it stops responding even via IPMI it doesn't seem like a networking issue though. Like it's triggering some other problem.

Do any logs show anything after 6:01:04?

Laxarus

@stephenw10 I have attached the related full log file in my previous post.

stephenw10

So despite the fact it was still logging connection attempts it stops responding entirely even at the local console? At what time did you try to connect?

Laxarus

@stephenw10 around 19:30, I tried to connect to see what is going on but IPMI console display was unresponsive. I had to perform a hard power reset over IPMI.

stephenw10

So ~12hrs later? And it was still logging connection attempts?

I have no idea what could cause it to be unresponsive at the console but continue logging like that.

Laxarus

@stephenw10 it might be a weird ipmi gimmick. I need to confirm the actual display by connecting it directly to a monitor if and when this happens again (hopefully not). I have not yet updated to the latest. There is a chance this issue might get fixed totally. I dont want to update remotely so this has to wait.

Other than that, any idea why when WAN is interrupted, the whole network goes down as well? (Sometimes not every time)

stephenw10

Hmm, could the IPMI module be crashing? Does it have a shared port with the WAN?

Laxarus

@stephenw10 no it is a separate interface.

stephenw10

Maybe check any logs the IPMI module has anyway.

Laxarus

@stephenw10 I don't mind the IPMI console not being functional when this occurs at the moment.
The real problem is why my whole network goes down when the WAN is interrupted until a hard reset. Assigning a static ip to "Modem" interface as you have suggested previously, has drastically reduced the frequency of this issue but the real reason still remains a mystery. I will try your other suggestion of changing the "modem" interface from igb to ixl but this will also as to wait until I go back.

Do you think LAGG might be the reason? I know that it has no relation with WAN but I am grasping straws right now.

stephenw10

Unlikely to be related to LAGG IMO. Many people run LAGG without issue.

Are you able to check the physical local console?

If it stop responding at the console it's not that surprising to me that it also stops passing traffic. It's more surprising that it appears to keep running and logging.

Laxarus

@stephenw10 said in Multiple issues, firewall freezes and whole network goes down.:

Are you able to check the physical local console?

Not right now. Only when I get back.

If it stop responding at the console it's not that surprising to me that it also stops passing traffic. It's more surprising that it appears to keep running and logging.

I honestly cannot tell. Maybe if I can physically connect, it will respond but due to IPMI video console, it somehow has a gimmick where it cannot register the user inputs through IPMI console web interface.

Laxarus

@stephenw10 I now have physical access to the firewall now (will leave in a week again). Since then, I had the same problem of pfsense bringing the whole network going down when the WAN pppoe connection has a problem occasionally.

I did try to connect to a physical monitor but I am not getting any signal on my monitor. It might be possible that the VGA port is broken or my monitor is broken or there is some configuration error on pfsense which causes no video output to vga. Anyway I am stuck with the IPMI console for management for now.

As you suggested previously, I have switched the WAN port from igb to ixl but it did not help. Shortly after, I had the same problem. So, we can rule that out.

stephenw10

Is it possible it's setup for serial console?

If you reboot it do you see anything on the monitor?

Laxarus

@stephenw10 nope, serial console is not set. My monitor is old, that is probably why. I remember when I was first setting up and installing pfsense, it also did not work back then. I had to remove my old TV with VGA input from its wall mount, carry it 3 floors up, and only then I have managed to install pfsense.