Crashdump Master FW rebooting
-
@stephenw10 This issue started happening after the sync started complaining and we started getting lots of messages like this.
14:52:52 A communications error occurred while attempting to call XMLRPC method restore_config_section:We then compared the primary and secondary firewall and noticed several config bit being out of sync (especially the users being duplicated on the backup FW and the openvpn setting on the same FW not being identical)
My colleague then disabled users/group sync, deleted the items from the secondary FW and re enabled the sync. All then seemed to synch correctly. But after a while the Master FW started rebooting by itself.
I did compare the configuration and found that some Openvpn interfaces were slightly different between the two FWs. The Master FW was showing a MAC address for the ovpns2/ovpns3 interface whilst the seconday FW did not show a MAC address:
Master FW opvns3 interface:
Secondary FW opvns2 interface:
Maybe above is unrelated but that was a strange difference also cause those interfaces on the master FW were showing a "Speed and Duplex" drop down menu (not shown on the Backup FW).
What we have done now is this:
- Delete those ovpns2/3 interfaces (TAP)
- Upgraded the FW networkin inerfaces connecting to our core distribution switches to 10Gbps SFP interfaces
We are monitoring to see if we get random reboots.
-
Mmm, the config should be identical otherwise the state sync will be incorrect. That's unlikely to really be an issue on OpenVPN servers though as only one can ever be active and clients have to reconnect at failover anyway.
The presence of a MAC address indicates the server is running in TAP mode. They should both have one if both are TAP mode servers and both are running.
Steve
-
@stephenw10 The backup server never had the MAC for the TAP interfaces even when it was MASTER. I dunno why...
The other funny thing is this:
We took another Dell R210 server and used the SSD with pfSense from the primary FW, the primary FW would reboot anyway. So it was not an HW issue cause it was a completely "new" server running the exact same pfSense. install.
The secondary FW has been running for 114 days straight (since the updated to 2.5.2 think), on the same Server HW model.
I think we are planning to buy Netgate HW next year (new budget). But it would be really great to get to the bottom of this.
-
Hmm, it looks almost identical to this: https://redmine.pfsense.org/issues/5473
But that was fixed years ago.I assume you had been running 2.5.2 for a while before this started?
Did you make any other sort of change that coincided with it starting?
Steve
-
@stephenw10 yeah we have been running 2.5.2 for months before this issue started.
By looking at the auto backup service on the primary FW, I cannot spot any substantial changes (I think I only see a change to the ovpns interface).
On the seconday FW instead i had added the ovpns interfaces and bridges for the respective LAN interfaces.
-
How often are you seeing the crashes?
You might try changing isrdispatch from direct to deferred since that's the code path that seems to be triggering it.
See: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nicsSteve
-
@stephenw10 The crashes were random, sometimes happened after hours some times after minutes, it crashed about 8 times within say 24h.
The crashes stopped after removing the TAP interfaces and after upgrading the network cards to SFP hence I did not follow your suggetion re isrdispatch.
-
Hmm, interesting.
Good to know for future reference. Thanks for following up.Steve
-
@stephenw10 I might re add the TAP interfaces after the xmas break and report back any issues
-
@stephenw10 One question, a bit unrelated perhaps to this issue. Is it normal for the firewall WEBUI to stall when adding VLAN and/or interfaces in a HA setup?
-
It depends what you mean by stall. Adding a new interface can trigger quite a few things especially in an HA pair but I wouldn't expect it to take very much longer than any other change.
Steve
-
@stephenw10 Next time i add an interface/VLAN I will time it and let you know
