Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN
-
Thank you for the reply. I have tried a few different monitoring addresses. Doesn't seem to make a difference. And the problem ONLY exists when both interfaces are active. I can have the routing set to only one of the gateways...no fail-over....and it still cycles a down member due to packet loss. As soon as I disable one of the two interfaces everything works fine again.
1 <1 ms <1 ms <1 ms fw1.xxxxxx.localdomain [192.168.1.1] 2 19 ms 20 ms 19 ms 100.64.0.1 3 16 ms 20 ms 23 ms 172.16.252.90 4 19 ms 19 ms 23 ms undefined.hostname.localhost [206.224.65.136] 5 17 ms 27 ms 16 ms undefined.hostname.localhost [206.224.64.173] 6 60 ms 47 ms 20 ms 140.248.126.222 7 17 ms 21 ms 20 ms 151.101.67.5 -
@jimeez said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Thank you for the reply. I have tried a few different monitoring addresses. Doesn't seem to make a difference. And the problem ONLY exists when both interfaces are active. I can have the routing set to only one of the gateways...no fail-over....and it still cycles a down member due to packet loss. As soon as I disable one of the two interfaces everything works fine again.
My issue is exactly the same as jimeez. I too have tried different monitoring addresses wiht no change. Simply enabling the Century Link interface causes 100% packet loss after 15 minutes on the Starlink WAN. After a minute or so, the Starlink WAN will return online, but then fail with 100% packet loss every 15 minutes.
-
But look at the ultimate latency of both links. One is satellite which by default will have a higher latency.. and the other is DSL which with interleaving will generally have 38ms or so.. (educated guess)
If you do not have the second faster (latency) interface then the system will simply stay on the only gateway is see's.
If (and I have not had the opportunity yet to play with Starlink although at work we will be soon..) your Starlink interface see's a change in latency that is drastic enough then I can see your system trying to switch to a more stable link..
Try this. From a command prompt.. c> Ping -t 8.8.8.8 and let that run for an hour or so. Watch the latency there and see if it changes much. If it does not then I am probably barking up the wrong tree.. But my SWAG says you will probably see some latency swings. Of coarse take your second link down and only allow it on the Starlink.
-
Thanks for the reply.
Here are stats from my Starlink for the last 24 hours. The Starlink app statistics also match the pfSense stats.
-
I don't want to get my hopes up, but it's been 62 minutes and I have not lost the Starlink WAN. Here is what I did today:
-
Deleted the Centurylink Gateway and Centurylink interface.
-
Assigned the Centurylink interface and gateway.
-
Power cycled the Centurylink modem.
-
Disabled the kea-dhcp6 service
-
Under System/Routing/Gateways: Changed default IPv6 to NONE.
I haven't added any Gateway groups and failover settings yet, but so far the Starlink WAN is staying up. For now (testing) I have "Block private networks and loopback addresses" and "Block bogon networks" both checked. I also haven't set up a monitor IP or DNS server for Centurylink (one thing at a time).
.
.I really think this alone might have done the trick:
-
-
Oh wow. No kidding? That would be amazing if this solved things. If it does, I wonder what that means in terms of why this started happening. Something with CenturyLink perhaps?
Also, general question regarding the IP address of your CenturyLink WAN. I've seen this in a lot of the hot-to videos I watch. Why is the IP address of the CL WAN 192.168.0.1 rather than a CL-assigned IP address?
-
That screenshot above is showing the 192.168.0.1 as a monitor IP. I was trying to make as few changes as possible to see what would break it so I did not have a monitor IP or DNS server set.
I have now added a DNS server and monitor address of 8.8.8.8 to the CL connection and it is now showing the CL IP address on the dashboard correctly. After doing so, I had to disable and re-enable the CL interface to get to pull a proper IP.
!!! After adding the DNS server to the CL connection I lost Starlink at the 15 minute mark! D@mn! !!!
Maybe I'm getting closer to the answer since Starlink stayed online for several hours and only went down when I made DNS changes to the CL connection.
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Thanks for the reply.
Here are stats from my Starlink for the last 24 hours. The Starlink app statistics also match the pfSense stats.
Actually I appreciate you posting those numbers.. It will help me with my day job when we get our setup for a remote site we have.. ;)
-
Curious how you're making out over these last 24 hours. Planning to tackle this later this afternoon. Was hoping to see that you've maintained solid connections before embarking on a fresh config. ;-)
-
No luck. I thought I had it, but adding the DNS server to the CL connection, it broke the Starlink connection.
How are you running your DNS servers for the dual wan? I am wondering if that is somehow causing Starlink to drop offline.
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
How are you running your DNS servers for the dual wan? I am wondering if that is somehow causing Starlink to drop offline.
I wish I knew how to answer that, but sadly I don't. I followed a guide a year and a half ago and it's been working ever since....until recently. I don't recall doing anything specific directly related to DNS. I do recall though thinking how simple it was to set up.
One question for you: are you running your CL modem in transparent bridge mode?
-
I am running the CL modem in transparent bridge mode. My modem is the Zyxel C1100Z.
I am using DNS forwarder. I have tried different combinations of 1.1.1.1 for Starlink and 8.8.8.8 for Centurylink. I have also tried using the DNS servers supplied by Starlink and Centurylink. I may be barking up the wrong tree with the DNS thing, but I'm at my wits end.
I too remember how easy and painless it was to set up the dual wans and like you it ran fine for a long time.
-
Ok. Thanks. Yep, my setup is identical.
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
!!! After adding the DNS server to the CL connection I lost Starlink at the 15 minute mark! D@mn! !!!
So I think you're on to something here with the 15 minute thing. I never really paid attention to the time intervals before but made sure to time it tonight. It's 15 minutes on the nose! Literally.
What in the world could cause a dual interface setup to kill one of them due to measured/perceived packet loss every 15 minutes AND kill the NUT service? Very weird.
-
If anyone has any ideas, I am still working this problem.
Here are my DHCP log entries from about the time I enabled the Centurylink WAN 2 (ix2) interface 11:55 to to the time that Starlink WAN1 goes offline with 100% packet loss 15 minutes later. I hope there are some 'log whisperers' out there that can help. Am I barking up the wrong tree thinking it's a DHCP issue?
The correlation I see here is that at 11:55:30 dhc client binds to the Centurylink IP with a 900 second renewal. Exactly 900 seconds later, Starlink WAN1 goes offline with 100% packet loss. It takes Starlink WAN1 about 1-2 minutes to come back online and then the 15 minute cycle repeats.
Thank-you.
-
I haven't given up yet. While I have had zero success getting it to work on pfSense, I figured I'd give OPNsense a try next. Planning to work on it this coming weekend. Will report back with my findings.
Surely we can be the only two having this issue.
-
Agreed. Two people with working dual WANs that suddenly stops working.
Some kind of change happened with Centurylink, Starlink ,or pfSense.
-
Having basically the same issue as well. Dual WAN in a gateway group, Starlink as Tier 1 and DSL as Tier 2. No issues for the last 2+ years until around Aug. 24th when Starlink suddenly started dropping out about every 2 hours.
Will be back on site this Thursday to more troubleshooting and will see if disabling the DSL connection provides the same results that you guy saw. I also have a second Starlink dish that I am going to add into the mix just for fun.
-
Just some of my thoughts.
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
The correlation I see here is that at 11:55:30 dhc client binds to the Centurylink IP with a 900 second renewal.
CenturyLink = interface ix3 - a renewal of 150 seconds ? Right ?
For IPv4, 900 sec or 15 minutes is already very low, but ok, as this include 'new' technology, why not.
Then what is this Century Link ?900 seconds = the Starlink, right ?
... Sep 9 11:55:33 kea-dhcp4 42003 INFO [kea-dhcp4.lease-cmds-hooks.0x3156f3012000] LEASE_CMDS_DEINIT_OK unloading Lease Commands hooks library successful Sep 9 11:55:33 kea-dhcp4 42003 INFO [kea-dhcp4.dhcp4.0x3156f3012000] DHCP4_SHUTDOWN server shutdown Sep 9 11:55:30 kea-dhcp4 42003 INFO [kea-dhcp4.dhcp4.0x3156f3012000] DHCP4_STARTED Kea DHCPv4 server version 2.4.1 started Sep 9 11:55:30 kea-dhcp4 42003 WARN [kea-dhcp4.dhcp4.0x3156f3012000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 ...(from bottom to top) : ... and a DCHP LAN server also restarts .... why ?
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Exactly 900 seconds later, Starlink WAN1 goes offline with 100% packet loss.
Here :
Sep 9 12:10:57 dhclient 86826 bound to 76.0.28.79 -- renewal in 900 seconds. Sep 9 12:10:57 dhclient 47261 Creating resolv.conf Sep 9 12:10:57 dhclient 46263 RENEW Sep 9 12:10:57 dhclient 86826 DHCPACK from 71.33.5.2 Sep 9 12:10:56 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:45 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:39 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:36 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:34 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:32 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:31 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:30 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67(from bottom to top)
At 12:10:30 its reewal time .... DHCPREQUEST on ix2 but no answer.
So one second delay : ... DHCPREQUEST on ix2 but no answer.
2 seconds deklay ... DHCPREQUEST on ix2 but no answer.
4 seconds delay DHCPREQUEST on ix2 but no answer.
8 seconds DHCPREQUEST on ix2 but no answer.etc everything is fine here, the stand-off delay doubles at every request - that's normal.
and suddenly :
Sep 9 12:10:57 dhclient 86826 DHCPACK from 71.33.5.2
An answer from the 'startlink' DHCP server came back 27 seconds later - ouf !!Not to bad, I guess, as I don't know where the DHCP 'starlink' server is, how many inter linked laser hops between satellites the packet made .... where the ground station is etc.
Let say .... the links was bad for a moment ? Chinese space junk in the way ? The link was overloaded ?
(we'll never know)At this moment, the same Ipv4 = 76.0.2x.79 - came back, thus renew.
Still, you said : "2 minutes later", counting from the start of the DHCP renewal, the connection is 'dead'.
My question : is this related to the fact that a a simple 'DHCPREQUEST' request packet took 30 seconds to be answered ? If the conenctuion is that bad at that moment, then yeah, the connection will be considered as very bad by dpinger (huge pings) .... and it will 'reset' the connection for sure.
edit : wait : satellites are not geo locke din the sky, they really do move ... was the disk syncing to a new satellite ? How much should that take ?
Does that change the DHCP server - does the gateway change ?
I know, sorry, more questions as answers.Btw : if the IPv6 gateway has been shut down, why not also silence the LAN IPv6 DHCP server ?
Also : Why not testing with the good old 'ISC-DHCP' stuff instead of KEA, just to be sure ?
-
Thanks for the response.
- I can rule out Starlink as a bad connection as I can monitor it's stability via the app. It also remains up 99.99% of the time when it is the only interface enabled.
- I have tried reverting to ISC-DHCP with the same results.
- I tried disabling IPv6 everywhere with the same results.
- The 900 seconds is for the CenturyLink DSL (ix2) connection.
- I've tried DNS resolver and DNS forwarder with the same results.
- I'm not 100% certain it's a DHCP issue...just guessing since I found the 900 second entry in the log which is exactly how long it takes for the Starlink WAN to go down.
- I've factory reset and changed the CenturyLink modem's address to 172.16.0.1 (instead of the default 192.168.0.1) with the same results.
- When Starlink WAN1 goes down, it takes about 2 minutes for it to return and then the 15 minute cycle repeats.
- @jimeez also found some reddit posts with people having a similar issue.
Crazy thing is, every thing was working fine for a long time, and I didn't make any changes (no updates, no new packages, nothing) when the failure began.
