Multiple issues, firewall freezes and whole network goes down.

stephenw10

So it was completely unresponsive again? You had to power cycle it?

You are still using igb1 for WAN. Are you able to try a different NIC?

It's a interesting log. The igb NIC bounces twice causing a bunch of service restarts but then remains stable.
The PPP daemon then tries to reconnect but initially it fails with:

Oct 14 06:02:17 FIREWALL ppp[73450]: [wan_link0] PAP: rec'd NAK #1 len: 27
Oct 14 06:02:17 FIREWALL ppp[73450]: [wan_link0]   MESG: Authentication failed!
Oct 14 06:02:17 FIREWALL ppp[73450]: [wan_link0] LCP: authorization failed

But then after ~40s:

Oct 14 06:02:45 FIREWALL ppp[73450]: [wan_link0] PAP: rec'd ACK #1 len: 39
Oct 14 06:02:45 FIREWALL ppp[73450]: [wan_link0]   MESG: Authentication Successful,Welcome!
Oct 14 06:02:45 FIREWALL ppp[73450]: [wan_link0] LCP: authorization successful

That implies the NIC is passing traffic and the remote server is responding at that point. The PPPoE link comes up correctly.
But only for ~90s, then:

Oct 14 06:03:54 FIREWALL ppp[73450]: [wan_link0] LCP: no reply to 1 echo request(s)

After that the ppp daemon times out and cannot reconnect again. But it keeps trying.

Nothing is logged that seems be anything that would cause it to stop passing traffic.

Laxarus

@stephenw10 said in Multiple issues, firewall freezes and whole network goes down.:

So it was completely unresponsive again? You had to power cycle it?

Yeah, exactly the same thing

@stephenw10 said in Multiple issues, firewall freezes and whole network goes down.:

You are still using igb1 for WAN. Are you able to try a different NIC?

I have not been able to return back to this site to make this change yet. (will try to get someone over there to switch the ports as soon as possible)

is it possible that there is some configuration mistake on my part for pppoe?

stephenw10

I doubt it's a pppoe issue. If it was it would either fail to connect entirely or disconnect consistently. This seems like something happens to the upstream device. Somehow it then fails. What's totally unclear though is why the firewall stops responding completely. The logs show it just keeps trying to connect. It makes me wonder if it's actually an IPMI issue somehow.

One thing I would do here is chnage the 'Modem' interface to a static IP with no gateway if you can. Assuming you are using that only to access the modem? When it's configured as dhcp pfSense treats it as a WAN and runs all the link scrips when igb1 bounces.

Laxarus

@stephenw10 hmm, I will try that during the weekend. No time to test it right now. Thanks for the suggestion. However, by logic, does it matter whether it is dhcp or static?

According to this recipe, it does recommend "static".

https://docs.netgate.com/pfsense/en/latest/recipes/modem-access.html

Now that I think about it:

My WAN IP is 88.....
and my WAN_PPPOE is 10......
For some reason, they are different and I am not behind CGNAT.
Does enabling "Use non-local gateway" on one of these gateways will make a difference? I am not exactly sure what this option does and which gateway it should be enabled?

stephenw10

Yes, use static there if you can. When you set it as dhcp the server passes it a gateway to use and pfSense sets that on the interface turning it into a WAN. Then it triggers all the WAN IP scripts when it bounces.

Laxarus

@stephenw10 Okay, I set it up as static now. Let's see if it will survive the scheduled restart of Modem on Monday.
Now that you have reminded me, I have another interface (network) which is not WAN but with a DHCP gateway. It is connected to a switch (This switch connects all IPMI devices together in one network with elevated privilidges). The switch acts as a DHCP server. By this logic, should I also set this up as static?

The MNG interface below.

stephenw10

You could also change that but it won't be nearly as impactful because it's on a different NIC. It shouldn't lose link at the same time.

Laxarus

@stephenw10 Survived the Monday but will monitor it for changes.

Just curious is there a way to mark the DHCP gateway as local only instead of WAN or this default behavir cannot be changed?

stephenw10

The only way you can do that is to add a gateway separately to an interface config like you might with an internal router for example. But you can't do that for a dynamic interface type like DHCP. The server passes a gateway to the client to use and it is always added to the gateway.
You could maybe override the gateway that is passed in the advanced dhcp options. I'm not sure I've ever tried that.

Laxarus

@stephenw10 so far nothing crashed yet with static IP.
I am guessing this is a very rare problem since you have to have pppoe DSL (most uses fiber these days) and configured the network to be able to access modem behind WAN via DHCP.
I will continue monitoring.

There is no option that suggests to mark the gateway as internal network only. Maybe checking the boxes for "Disable Gateway Monitoring", "Disable Gateway Monitoring Action", and "Do not add static route for gateway monitor IP address via the chosen interface" might achieve a similar result but I dont want to try these for now.
Still no idea for the exact use case of "Use non-local gateway" option.

stephenw10

You would have to use the DHCP advanced options field to force the dhcp client to ignore the gateway passed by the server. So adding supersede routers in the Option modifiers field should do it.

But I would just use a static IP here.

JonathanLee

@Laxarus Your traffic shaper is that required? That could be configured incorrectly.

https://forum.netgate.com/topic/171842/queue-management-algorithms-differences

Laxarus

@JonathanLee said in Multiple issues, firewall freezes and whole network goes down.:

https://forum.netgate.com/topic/171842/queue-management-algorithms-differences

My main interface is on 2x25G LAGG and LAGG is not supported with traffic shaper so other than bufferbloat nothing is set there.