Anomaly in the Pfsense 2.6 restoration process?

[peterkrautle]

I finally was able to reproduce a Pfsense 2.6 anomaly consistently and am not sure whether it is a bug or user error (am not perfect :( ). We have a customer with two sites that had 10 year old ASA firewalls that were intermittently failing. We loaned them older NUCs and replaced the ASAs with Pfsense running 2.6. The new NUCs are in my office and being staged for deployment - each NUCi5 is running Esxi 8.0 with Pfsense 2.6 as a virtual machine. Esxi is configured with the WAN interface assigned to VLAN 2 and the switch uses 802.1Q to separate the LAN traffic from the WAN traffic. To test that the switch, Pfsense, and Esxi are configured correctly, the NUC and switch is staged behind our office network and a private IP address (in this case 10.20.4.5) is provisioned in the WAN interface of the customer Pfsense - a slide of network topology is here https://www.dropbox.com/s/g9rkuofxsjotok3/Topology.pptx?dl=0. A laptop is placed on the LAN side of Pfsense (192.168.61.83) and then pings are issued to the office WAN gateway, other nodes in the office, 8.8.8.8, www.google.com, and nodes on our colo server to ascertain everything is configured correctly - we have done this multiple times.

As there already was working Pfsense configurations, backups were taken from Pfsense on the loaner NUCs and restored on Pfsense running on the new NUCs. There were successful ping responses from 8.8.8.8, www.google.com, and nodes on one of our colo servers (e.g. 10.20.2.25), but not to the office network or gateway (10.20.4.1). A pcap of the LAN interface shows successful ping responses from everywhere except the 10.20.4.x office network https://www.dropbox.com/s/mwz6s66jsv6v0ac/PfsenseLANTrace.pcap?dl=0. An analysis of the PCAP taken on the WAN is that ICMP pings to the office Pfsense router (10.20.4.1) was never delivered by Pfsense from the LAN interface https://www.dropbox.com/s/d5bn84ybx0022o1/PfsenseWANTrace.pcap?dl=0.

Pfsense was factory reset and the Pfsense configuration was manually rebuilt - pings to the office network gateway and other office nodes then worked correctly. Pfsense was then factory reset again and this configuration file was restored https://www.dropbox.com/s/r60o0hverc0ux8o/config-ABC-CT.Office-20230106140502.xml?dl=0. Pings to office network nodes (10.20.4.x) would not work. Factory reset Pfsense and built the configuration manually. Everything worked correctly. This cycle was repeated a third time with similar results.

Either I am missing something obvious or there is anomaly in the Pfsense restore process.

Any insights appreciated - many thanks in advance.

All the best
Peter

[SteveITS]

@peterkrautle I suggest comparing the .xml config file backups from each to see if there’s a difference.