Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN
-
Oh wow. No kidding? That would be amazing if this solved things. If it does, I wonder what that means in terms of why this started happening. Something with CenturyLink perhaps?
Also, general question regarding the IP address of your CenturyLink WAN. I've seen this in a lot of the hot-to videos I watch. Why is the IP address of the CL WAN 192.168.0.1 rather than a CL-assigned IP address?
-
That screenshot above is showing the 192.168.0.1 as a monitor IP. I was trying to make as few changes as possible to see what would break it so I did not have a monitor IP or DNS server set.
I have now added a DNS server and monitor address of 8.8.8.8 to the CL connection and it is now showing the CL IP address on the dashboard correctly. After doing so, I had to disable and re-enable the CL interface to get to pull a proper IP.
!!! After adding the DNS server to the CL connection I lost Starlink at the 15 minute mark! D@mn! !!!
Maybe I'm getting closer to the answer since Starlink stayed online for several hours and only went down when I made DNS changes to the CL connection.
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Thanks for the reply.
Here are stats from my Starlink for the last 24 hours. The Starlink app statistics also match the pfSense stats.
Actually I appreciate you posting those numbers.. It will help me with my day job when we get our setup for a remote site we have.. ;)
-
Curious how you're making out over these last 24 hours. Planning to tackle this later this afternoon. Was hoping to see that you've maintained solid connections before embarking on a fresh config. ;-)
-
No luck. I thought I had it, but adding the DNS server to the CL connection, it broke the Starlink connection.
How are you running your DNS servers for the dual wan? I am wondering if that is somehow causing Starlink to drop offline.
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
How are you running your DNS servers for the dual wan? I am wondering if that is somehow causing Starlink to drop offline.
I wish I knew how to answer that, but sadly I don't. I followed a guide a year and a half ago and it's been working ever since....until recently. I don't recall doing anything specific directly related to DNS. I do recall though thinking how simple it was to set up.
One question for you: are you running your CL modem in transparent bridge mode?
-
I am running the CL modem in transparent bridge mode. My modem is the Zyxel C1100Z.
I am using DNS forwarder. I have tried different combinations of 1.1.1.1 for Starlink and 8.8.8.8 for Centurylink. I have also tried using the DNS servers supplied by Starlink and Centurylink. I may be barking up the wrong tree with the DNS thing, but I'm at my wits end.
I too remember how easy and painless it was to set up the dual wans and like you it ran fine for a long time.
-
Ok. Thanks. Yep, my setup is identical.
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
!!! After adding the DNS server to the CL connection I lost Starlink at the 15 minute mark! D@mn! !!!
So I think you're on to something here with the 15 minute thing. I never really paid attention to the time intervals before but made sure to time it tonight. It's 15 minutes on the nose! Literally.
What in the world could cause a dual interface setup to kill one of them due to measured/perceived packet loss every 15 minutes AND kill the NUT service? Very weird.
-
If anyone has any ideas, I am still working this problem.
Here are my DHCP log entries from about the time I enabled the Centurylink WAN 2 (ix2) interface 11:55 to to the time that Starlink WAN1 goes offline with 100% packet loss 15 minutes later. I hope there are some 'log whisperers' out there that can help. Am I barking up the wrong tree thinking it's a DHCP issue?
The correlation I see here is that at 11:55:30 dhc client binds to the Centurylink IP with a 900 second renewal. Exactly 900 seconds later, Starlink WAN1 goes offline with 100% packet loss. It takes Starlink WAN1 about 1-2 minutes to come back online and then the 15 minute cycle repeats.
Thank-you.
-
I haven't given up yet. While I have had zero success getting it to work on pfSense, I figured I'd give OPNsense a try next. Planning to work on it this coming weekend. Will report back with my findings.
Surely we can be the only two having this issue.
-
Agreed. Two people with working dual WANs that suddenly stops working.
Some kind of change happened with Centurylink, Starlink ,or pfSense.
-
Having basically the same issue as well. Dual WAN in a gateway group, Starlink as Tier 1 and DSL as Tier 2. No issues for the last 2+ years until around Aug. 24th when Starlink suddenly started dropping out about every 2 hours.
Will be back on site this Thursday to more troubleshooting and will see if disabling the DSL connection provides the same results that you guy saw. I also have a second Starlink dish that I am going to add into the mix just for fun.
-
Just some of my thoughts.
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
The correlation I see here is that at 11:55:30 dhc client binds to the Centurylink IP with a 900 second renewal.
CenturyLink = interface ix3 - a renewal of 150 seconds ? Right ?
For IPv4, 900 sec or 15 minutes is already very low, but ok, as this include 'new' technology, why not.
Then what is this Century Link ?900 seconds = the Starlink, right ?
... Sep 9 11:55:33 kea-dhcp4 42003 INFO [kea-dhcp4.lease-cmds-hooks.0x3156f3012000] LEASE_CMDS_DEINIT_OK unloading Lease Commands hooks library successful Sep 9 11:55:33 kea-dhcp4 42003 INFO [kea-dhcp4.dhcp4.0x3156f3012000] DHCP4_SHUTDOWN server shutdown Sep 9 11:55:30 kea-dhcp4 42003 INFO [kea-dhcp4.dhcp4.0x3156f3012000] DHCP4_STARTED Kea DHCPv4 server version 2.4.1 started Sep 9 11:55:30 kea-dhcp4 42003 WARN [kea-dhcp4.dhcp4.0x3156f3012000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 ...(from bottom to top) : ... and a DCHP LAN server also restarts .... why ?
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Exactly 900 seconds later, Starlink WAN1 goes offline with 100% packet loss.
Here :
Sep 9 12:10:57 dhclient 86826 bound to 76.0.28.79 -- renewal in 900 seconds. Sep 9 12:10:57 dhclient 47261 Creating resolv.conf Sep 9 12:10:57 dhclient 46263 RENEW Sep 9 12:10:57 dhclient 86826 DHCPACK from 71.33.5.2 Sep 9 12:10:56 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:45 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:39 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:36 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:34 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:32 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:31 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67 Sep 9 12:10:30 dhclient 86826 DHCPREQUEST on ix2 to 71.33.5.2 port 67(from bottom to top)
At 12:10:30 its reewal time .... DHCPREQUEST on ix2 but no answer.
So one second delay : ... DHCPREQUEST on ix2 but no answer.
2 seconds deklay ... DHCPREQUEST on ix2 but no answer.
4 seconds delay DHCPREQUEST on ix2 but no answer.
8 seconds DHCPREQUEST on ix2 but no answer.etc everything is fine here, the stand-off delay doubles at every request - that's normal.
and suddenly :
Sep 9 12:10:57 dhclient 86826 DHCPACK from 71.33.5.2
An answer from the 'startlink' DHCP server came back 27 seconds later - ouf !!Not to bad, I guess, as I don't know where the DHCP 'starlink' server is, how many inter linked laser hops between satellites the packet made .... where the ground station is etc.
Let say .... the links was bad for a moment ? Chinese space junk in the way ? The link was overloaded ?
(we'll never know)At this moment, the same Ipv4 = 76.0.2x.79 - came back, thus renew.
Still, you said : "2 minutes later", counting from the start of the DHCP renewal, the connection is 'dead'.
My question : is this related to the fact that a a simple 'DHCPREQUEST' request packet took 30 seconds to be answered ? If the conenctuion is that bad at that moment, then yeah, the connection will be considered as very bad by dpinger (huge pings) .... and it will 'reset' the connection for sure.
edit : wait : satellites are not geo locke din the sky, they really do move ... was the disk syncing to a new satellite ? How much should that take ?
Does that change the DHCP server - does the gateway change ?
I know, sorry, more questions as answers.Btw : if the IPv6 gateway has been shut down, why not also silence the LAN IPv6 DHCP server ?
Also : Why not testing with the good old 'ISC-DHCP' stuff instead of KEA, just to be sure ?
-
Thanks for the response.
- I can rule out Starlink as a bad connection as I can monitor it's stability via the app. It also remains up 99.99% of the time when it is the only interface enabled.
- I have tried reverting to ISC-DHCP with the same results.
- I tried disabling IPv6 everywhere with the same results.
- The 900 seconds is for the CenturyLink DSL (ix2) connection.
- I've tried DNS resolver and DNS forwarder with the same results.
- I'm not 100% certain it's a DHCP issue...just guessing since I found the 900 second entry in the log which is exactly how long it takes for the Starlink WAN to go down.
- I've factory reset and changed the CenturyLink modem's address to 172.16.0.1 (instead of the default 192.168.0.1) with the same results.
- When Starlink WAN1 goes down, it takes about 2 minutes for it to return and then the 15 minute cycle repeats.
- @jimeez also found some reddit posts with people having a similar issue.
Crazy thing is, every thing was working fine for a long time, and I didn't make any changes (no updates, no new packages, nothing) when the failure began.
-
I tried disabling the DSL interface (WAN01 - Tier 2) and wouldn't you know, Starlink interface (WAN02 - Tier 1) starts to work without issue. Re-enable DSL interface (WAN01 - Tier 2) and within an hour I am seeing the same issue with packet loss shooting up to 100% on the Starlink connection
The DSL connection has a static IP address, but for years now I have just left the interface IPv4 Configuration Type as "DHCP" without issue. As a quick test I switched it over to "Static IPv4" along with it's assigned IP address. Hours now with both the DSL and Starlink interfaces active with no issues. Everything is running like it was a couple weeks back. Will continue to monitor for the rest of the day.
For now, while I monitor I need to sit here and think about why this appears to be the solution for me, and why it is only a recent problem.
@jimeez or @preston do either of you have a static IP for your respective DSL connections?
-
@knoppolis said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
@jimeez or @preston do either of you have a static IP for your respective DSL connections?
My CenturyLink DSL connection is not static. This is a good data point though, thanks. Let us know how it does.
-
@knoppolis said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
@jimeez or @preston do either of you have a static IP for your respective DSL connections?
Also a no here.
I am very curious to see if this holds up for you. Although, if it does, my and @preston's issue will be an even bigger mystery.
-
So @preston mentioned something to me in a private chat that got my wheels turning. He brought up the fact that, prior to this issue, his StarLink connection would drop out around 4AM most days then come right back up. Mine did this too. Like clockwork. I always thought the reason was that the StarLink unit was receiving an update and restarting or something. But now I'm wondering if that 24 hour cycle is somehow related to this problem. Only now instead of every 24 hours it's happening every 15 minutes.
I went back and checked my notification logs. This 24 hour drop out was very consistent. Then on August 24th the 15 minute dropout started happening.
-
So I lobbed a support ticket to StarLink. Referenced this thread. Their response as follows:
Would you be able to confirm how you currently have your health checks set up for a failover to occur? The typical recommendation we provide our enterprise customers is to relax heath checks (i.e. pings, etc.) to deal with occasional connection drops from Starlink. Checking every 10 seconds & getting 5 fails in a row would be a good threshold to start with.
Would anyone be able to tell me where to go look in pfsense to find the answer to their question?
