pfSense flushes default route when second gateway is starting

Yves_

@stephenw10 here you have those two logs:
System General Logs

System Gateway logs do not notice anything... so nothing at this time in the logs

Here is how my default gateway is configured

Here are the empty static routes (i think this i correct since both wan connections are dhcp)

And here is how the Gateway group which is used as default gateway is configured

stephenw10

Hmm, so in that log you end up with no default gateway between the hotplug on OPT3 event and rc.linkup setting the default to the UPCCablecom gateway?

If you just set the default gateway to UPCCablecom rather than the failover group does it still happen?

Is there anything in the routing log at that time?

There must be something in the gateways log when that happens because the Starlink gateway becomes available.

Steve

Yves_

@stephenw10 As far as I can see. Yes. Nothing else.

I just did that, as you can see here:

Unfortunalty it has had the same effect:

Gateway Logs like I said:

stephenw10

Hmm, OPT3 is the starlink WAN dircetly? igb1?

I expect to see the Starlink gateway come up when it's connected? Or at least a dpinger instance started for it.

Yves_

@stephenw10 OPT3 / igb1 is starlink directly without the stupid starlink router part. POE Box from starlink directly to the igb1.

Well it never does... (because the antenna is misplaced currently) so it does not get an IP and it is always on 0.0.0.0. And according to a guide the 192.168.100.1 dhcp offer will be as described in this guide rejected by the pfsense. This I can see on the DHCP log.

But it does not explain why it drops the default route to UPC...

stephenw10

What pfSense version is this?

Yves_

@stephenw10 22.05-RELEASE

stephenw10

Hmm, I think I see what might have happened. Though I can't replicate. Yet.

Download the config file and look to see what's in the <gateways> section.

stephenw10

Oh actually it's probably this: https://redmine.pfsense.org/issues/12922

Did you check the routing log? Does it show those routes being passed?

stephenw10

Hmm, though that would only happen if it was actually seeing a dhcp response and the 60s delay implies it is not....

Yves_

@stephenw10

	<gateways>
		<defaultgw4>UPCCABLECOM_DHCP</defaultgw4>
		<defaultgw6>-</defaultgw6>
		<gateway_item>
			<interface>wan</interface>
			<gateway>dynamic</gateway>
			<name>UPCCABLECOM_DHCP</name>
			<weight>1</weight>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[UPC Business Uplink]]></descr>
			<data_payload>0</data_payload>
			<latencylow>100</latencylow>
			<latencyhigh>250</latencyhigh>
			<losslow>5</losslow>
			<losshigh>10</losshigh>
		</gateway_item>
		<gateway_item>
			<interface>opt3</interface>
			<gateway>dynamic</gateway>
			<name>STARLINK_DHCP</name>
			<weight>1</weight>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[Starlink Uplink]]></descr>
		</gateway_item>
		<gateway_group>
			<name>UPC_Preferred</name>
			<item>UPCCABLECOM_DHCP|1|address</item>
			<item>STARLINK_DHCP|2|address</item>
			<trigger>downlosslatency</trigger>
			<descr><![CDATA[Failover between UPC Business and Starlink]]></descr>
		</gateway_group>
		<gateway_group>
			<name>Starlink_Preferred</name>
			<item>UPCCABLECOM_DHCP|2|address</item>
			<item>STARLINK_DHCP|1|address</item>
			<trigger>downlosslatency</trigger>
			<descr><![CDATA[Failover between Starlink and UPC Business]]></descr>
		</gateway_group>
	</gateways>

Yves_

@stephenw10
My routing log looks quite boring....

stephenw10

@yves_ said in pfSense flushes default route when second gateway is starting:

<defaultgw6>-</defaultgw6>

That looks a bit odd. But apart from that it's almost exactly what I was testing and still haven't managed to replicate it.
Do you have any IPv6 config on either of those WANs?

Steve

Yves_

@stephenw10 I guess thats because I don't have an IPv6 gateway configured. Since I don't need it so far...

Might it be because of this dhcp lease stuff described in the problem you posted above?

stephenw10

That seemed likely since it's specific to starlink but you would see something in the routing log. And it has to actually receive a new dhcp lease to get that and your issue looks to be during the timeout where it's failing to pull a lease.

So you have no IPv6 configured on any interface? They are all set as 'none'?