pfSense flushes default route when second gateway is starting

Yves_

Hi there,

I don't know what I am doing wrong. But I am pretty sure that it is me :-)

Pretty basic setup:
I got two gateways. A COAX Cable modem (DHCP) and a Starlink modem (DHCP CGNAT) for testing purpose.

I have both in the System -> Routing -> Gateways Tab
Cable modem is Tier 1 and weight 1
Starlink modemis Tier 2 and weight 10

As soon as I give power to the Starlink Dishy it boots up and even doh there is no network ip and anything... It flushes / removes the default route in the routing table and kills connection like that. After another 30-60 second default route comes back.

If you need more infos I am happy to provide. But I tried and tried and did not find the right setting to solve this behavior.

Thanks for your help
Yves

stephenw10

What's shown in the system and gateways logs when that happens?

I certainly wouldn't expect that if you have Cable modem gateway set as default. If the default gateway is set as automatic then a WAN interface changing link state can trigger a change.

Steve

Yves_

@stephenw10 here you have those two logs:
System General Logs

System Gateway logs do not notice anything... so nothing at this time in the logs

Here is how my default gateway is configured

Here are the empty static routes (i think this i correct since both wan connections are dhcp)

And here is how the Gateway group which is used as default gateway is configured

stephenw10

Hmm, so in that log you end up with no default gateway between the hotplug on OPT3 event and rc.linkup setting the default to the UPCCablecom gateway?

If you just set the default gateway to UPCCablecom rather than the failover group does it still happen?

Is there anything in the routing log at that time?

There must be something in the gateways log when that happens because the Starlink gateway becomes available.

Steve

Yves_

@stephenw10 As far as I can see. Yes. Nothing else.

I just did that, as you can see here:

Unfortunalty it has had the same effect:

Gateway Logs like I said:

stephenw10

Hmm, OPT3 is the starlink WAN dircetly? igb1?

I expect to see the Starlink gateway come up when it's connected? Or at least a dpinger instance started for it.

Yves_

@stephenw10 OPT3 / igb1 is starlink directly without the stupid starlink router part. POE Box from starlink directly to the igb1.

Well it never does... (because the antenna is misplaced currently) so it does not get an IP and it is always on 0.0.0.0. And according to a guide the 192.168.100.1 dhcp offer will be as described in this guide rejected by the pfsense. This I can see on the DHCP log.

But it does not explain why it drops the default route to UPC...

stephenw10

What pfSense version is this?

Yves_

@stephenw10 22.05-RELEASE

stephenw10

Hmm, I think I see what might have happened. Though I can't replicate. Yet.

Download the config file and look to see what's in the <gateways> section.

stephenw10

Oh actually it's probably this: https://redmine.pfsense.org/issues/12922

Did you check the routing log? Does it show those routes being passed?

stephenw10

Hmm, though that would only happen if it was actually seeing a dhcp response and the 60s delay implies it is not....

Yves_

@stephenw10

	<gateways>
		<defaultgw4>UPCCABLECOM_DHCP</defaultgw4>
		<defaultgw6>-</defaultgw6>
		<gateway_item>
			<interface>wan</interface>
			<gateway>dynamic</gateway>
			<name>UPCCABLECOM_DHCP</name>
			<weight>1</weight>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[UPC Business Uplink]]></descr>
			<data_payload>0</data_payload>
			<latencylow>100</latencylow>
			<latencyhigh>250</latencyhigh>
			<losslow>5</losslow>
			<losshigh>10</losshigh>
		</gateway_item>
		<gateway_item>
			<interface>opt3</interface>
			<gateway>dynamic</gateway>
			<name>STARLINK_DHCP</name>
			<weight>1</weight>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[Starlink Uplink]]></descr>
		</gateway_item>
		<gateway_group>
			<name>UPC_Preferred</name>
			<item>UPCCABLECOM_DHCP|1|address</item>
			<item>STARLINK_DHCP|2|address</item>
			<trigger>downlosslatency</trigger>
			<descr><![CDATA[Failover between UPC Business and Starlink]]></descr>
		</gateway_group>
		<gateway_group>
			<name>Starlink_Preferred</name>
			<item>UPCCABLECOM_DHCP|2|address</item>
			<item>STARLINK_DHCP|1|address</item>
			<trigger>downlosslatency</trigger>
			<descr><![CDATA[Failover between Starlink and UPC Business]]></descr>
		</gateway_group>
	</gateways>

Yves_

@stephenw10
My routing log looks quite boring....

stephenw10

@yves_ said in pfSense flushes default route when second gateway is starting:

<defaultgw6>-</defaultgw6>

That looks a bit odd. But apart from that it's almost exactly what I was testing and still haven't managed to replicate it.
Do you have any IPv6 config on either of those WANs?

Steve

Yves_

@stephenw10 I guess thats because I don't have an IPv6 gateway configured. Since I don't need it so far...

Might it be because of this dhcp lease stuff described in the problem you posted above?

stephenw10

That seemed likely since it's specific to starlink but you would see something in the routing log. And it has to actually receive a new dhcp lease to get that and your issue looks to be during the timeout where it's failing to pull a lease.

So you have no IPv6 configured on any interface? They are all set as 'none'?