@stephenw10
OK I think i understand what happened. At some point I had 3 gateways in a gateway group.
1 of these gateways was removed from the gateway group, and the interface was disabled, however, looking at my routing table I found 2 "default" gateways at the same time.
Destination Gateway Flags Netif Expire
default XXX.XXX.XXX.XXX UGS igb4
default "supposedly.disabled.if.ip" UGS igb5
.......
I issued a /sbin/route -n get 'default' command manually, showing:
route to: 0.0.0.0
destination: 0.0.0.0
mask: 0.0.0.0
gateway: >>>>GATEWAY THAT WAS DISABLED IN THE UI IP ADDR HERE<<<<
fib: 0
interface: igb5
flags: <UP,GATEWAY,DONE,STATIC>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 0 0
In the UI -> I enabled and disabled the interface for the gateway in the UI.
after doing that, then in ifconfig -a output, I noticed that the interface no longer shows an IP Address and is not in "UP" status.
I subsequently issued a route delete default command which removed both default routes (the correct one, and the stagnant one for the now down interface), followed by adding a default route for the correct interface gateway.
I believe the issue is now resolved. since netstat -rn only shows 1 entry as 'default' now rather than two, and the route -n get default command now returns the correct gw ip addr.
FYI, this issue has been plaguing me for quite some time, it may be worth adding some logic to check for the presence of this issue if the issue is non-deterministic and/or non-reproducible. I unfortunately cannot provide reproduction steps that would lead to the loss of configuration sync between the UI and the OS, but I would note that the offending 3rd gateway interface was disabled in the UI, and unfortunately, it's interface was still up and had an ip addr, and the routing table had two routes set to "default". Not sure what here could be extrapolated as either a bug or an enhancement request to prevent the issue from reoccurring for others. It appears that the "disabled" state for the interface didn't quite make it down to the OS level bringing the interface down for the gateway. The presence of two routes both "default" I think might not be errant in load balancing scenarios (but definitely a bad deal if the interface is disabled in the UI, also, I think enabling/disabling the IF didn't seem to remove the duplicate default route entry corresponding to it)?
I would note that my gateway group is configured as "failover" rather than a Load Balancing configuration, i.e. interfaces in my gateway groups are usually categorized as tier1 & tier 2 & tier 3. So perhaps this config sync mixup between the ui and OS happened at some point during a failover, and the config "mixup" remained in this state indefinitely until manual intervention was required.
Anyhow, Thank You kindly for your help and for responding to my forum post!