Static routes no longer working after upgrade

[lfwhy]

Hello,

We have 2 pfsense/netgate appliances configured identical to one another. One (a.k.a., PF0) is about a year older than the other (a.k.a., PF1) and exists as a backup for the production system (PF1). In other words PF0 is the older unit and PF1 is the production unit, but they are configured identically.

Static routes worked on PF1 while it was running 21.02-RELEASE-p1 (amd64).

Upgraded PF1 from 21.02-RELEASE-p1 (amd64) to 21.05-RELEASE (amd64). After that, static routing is not working.

PF0 was not upgraded and is still running 21.02-RELEASE-p1 (amd64).

Moved the network cables from PF1 over to PF0 and static routing works.

The only configuration difference between PF0 and PF1 is the version of pfSense.

Anybody else running into this issue?

Do static routes in 21.05-RELEASE (amd64) work differently than 21.02-RELEASE-p1 (amd64)?

Other information:
Default gateway is igb1 (WAN)
igb0 (LAN)
igb2 (LAN2)

The routing information in Diagnostics >> Routes is no longer the same between the two. For instance, the below entry exists on PF0 but no longer exists on PF1 (actual IP addresses have been changed):

Destination Gateway Flags Netif
1.2.0.0/16 1.2.3.4 UGS igb2

When I use Diagnostics >> Traceroute on PF1 to reach 1.2.3.70, it shows the traffic going through the default gateway of igb1 rather than the gateway for igb2.

When I use Diagnostics >> Traceroute on PF0 to reach 1.2.3.70, it shows the traffic going through the gateway for igb2.

Thinking something went wrong with the upgrade, I did a factory reset of PF1 and rebuilt it from scratch making sure that all the settings implemented on PF0 were configured on PF1 (it is a pretty simple setup). The same problem exists.

[lfwhy]

@lfwhy Quick update: I have confirmed that the missing entry in the routing table is the problem.

If I log into PF1 via SSH, enter the shell, and enter the below command, my static route works:

route add -net 1.2.0.0/16 1.2.3.4

However, it is my understanding that entering routes this way is temporary and will be lost if the appliance is rebooted.

Anybody know of a way to make the change permanent via command line? Everything I am finding online has instructions using the GUI which is not working for me.

[jimp]

It's probably due to https://redmine.pfsense.org/issues/11296

You can revert the change there or disable gateway monitoring/monitoring actions for the gateway which is the target of the route and see if it helps.