Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN
-
@chpalmer said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
What happens if you put your Centurylink as tier 3?
As @preston already stated, both of us have only two WANs.
Have any of you put your Centurylink modem back to modem mode and tried to let it handle the PPP?
Sounds like this may have worked for @preston . Assuming it did/does, I have a couple questions:
- With the DSL modem no longer in transparent bridge mode I assume that it will assign that WAN interface a local IP address of 192.168.1.xxx. If this assumption is correct is this connection now sitting behind a double NAT?
- If that's the case, I guess we can no longer use pfSense to resolve our Dynamic DNS clients as that interface will no longer have an outside IP address.
You'll have to pardon my ignorance here. I only know enough to be slightly dangerous.
-
Things are still working here with the CenturyLink modem out of transparent bridging mode.
Here is what I did ( I connected my laptop via ethernet to the CenturyLink modem for the setup):
-
Factory reset the CenturyLink modem (again), disabled the CL modem WiFi, reset the admin password and so on.
-
My CenturyLink modem's default GUI address is 192.168.0.1 - I left that as is.
-
I connected LAN 1 on CL modem to my WAN2 pfSense port.
-
Under DHCP reservations in the CL modem, I assigned my pfSense box an IP of 192.168.0.2.
-
I disabled the DHCP server on the CL modem.
-
Rebooted the CL modem
-
Unplugged my laptop from the CL modem.
-
I reconnected to my pfSense network and set up the CL interface and gateway.
-
My DNS servers and monitor IPs are 1.1.1.1 for Starlink and 8.8.8.8 for CenturyLink respectively.
-
My pfSense LAN is in the 192.168.1.xxx range
-
The pfSense dashboard shows the CL WAN IP as 192.168.0.2, but when I check sites like infosniper.net I can see the CL IP address.
-
As an added bonus I can now access the CL modem GUI (192.168.0.1) via the pfSense network without having to fiddle with additional pfSense settings.
-
I'm not sure about Dynamic DNS, but I have been using Tailscale with Starlink and it has worked great.
-
@chpalmer may just be our hero!
As far as Double NAT while using the CL WAN, I really don't know (or understand it completely), but here is my Traceroute from the CL WAN to www.google.com:
1 192.168.0.1 0.544 ms 0.425 ms 0.402 ms 2 184.102.159.254 28.701 ms 28.475 ms 28.817 ms 3 71.33.4.9 28.078 ms 28.604 ms 28.296 ms 4 4.68.144.169 59.685 ms 46.815 ms 42.017 ms 5 4.68.127.114 44.359 ms 55.718 ms 63.480 ms 6 * * * 7 142.251.60.10 42.169 ms 216.239.51.116 43.287 ms 209.85.255.172 42.109 ms 8 209.85.247.117 42.379 ms 192.178.249.234 43.372 ms 209.85.247.117 42.327 ms 9 142.251.233.230 43.373 ms 44.324 ms 142.250.190.4 41.627 ms -
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
I disabled the DHCP server on the CL modem.
Ahhhhh....there it is. This makes sense now. Perfect! I'll give this a try tonight and see if I get the same result.
-
My guess is that pfsense is re-authenticating with C-Link every 15 minutes and something occurs to cause the issue at that time.
Though I am unsure why this hasn't come up before with other users trying to utilize similar setups.. I use Astound and Verizon here and have no issues. Neither of my ISPs use any kind of PPP.
-
Generally speaking, CenturyLink (now called Brightspeed) has been the worst ISP I have ever had. Until Starlink, they were the only option in my area.
That being said, it worked fine in Transparent Bridging for a long time. Not sure what changed, but it sure broke things. So far, so good. Things seem to be back to normal. Hope it works for you @jimeez.
Thank-you again!
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Generally speaking, CenturyLink (now called Brightspeed) has been the worst ISP I have ever had. Until Starlink, they were the only option in my area.
That being said, it worked fine in Transparent Bridging for a long time. Not sure what changed, but it sure broke things.
This was my exact experience as well. Only option available until StarLink (I mean I choose to live in the middle of nowhere). Worked just fine in transparent bridge mode forever....and still does when it's the only active interface. But something changed on or about August 22, 2024.
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Hope it works for you @jimeez.
I'm not quite there yet. Although it does seem promising. I spent more hours on this last night than I will admit and am still struggling with the CL modem settings. I have an old Protectli 4-port device with which I decided to start fresh. Got StarLink up and running no problem on the main WAN interface. Adding the CL interface is another story for some reason. I must be doing something wrong.
(i disconnected the StarLink interface while setting up the CL interface)
- I initially connected a laptop to the factory-reset CL modem via the WAN port (laptop to WAN port).
- After initial config of the CL modem (turn off WiFi etc.) I connected the pfSense device OPT1 interface to Port 1 on the CL modem and reserved an IP for it. In this case I was not able to use 192.168.0.2 because the laptop already took it, so I gave it 192.168.0.5.
- When the DHCP service is active both the laptop and pfSense see the modem and have internet.
- As soon as I disable the DHCP server on the CL modem I can no longer resolve DNS addresses. The laptop and pfSense devices both now show that they no longer have internet.
- I can ping actual IP addresses on both devices (like 8.8.8.8), but can't resolve addresses (say google.com).
Basically I'm stuck here. Grateful for any input on the likely obvious thing I'm doing wrong. ;-)
-
Short response right now is that this works for me too. Thank you so much!!
Will post back later with more detail.
-
@stephenw10 any comment from the balcony seats?
This seems to be reproducible but the particulars need to be understood a bit more I think.
-
A bit of bad news here. After about 24 hours, I lost the Centurylink connection. pfSense shows the Centurylink WAN as "pending" and will not reconnect. Restarted dpinger, rebooted pfSense, and it is still offline. I also have lost the ability to connect to the CenturyLink modem interface.
Perhaps disabling the DHCP server on the CL modem caused the lease to time out even though I assigned an IP address to the pfSense connection. I had to factory reset to get back to the CL interface.
I'm going to try it with the CenturyLink DHCP server enabled to see what happens. Back online now.
I'm going to play with lease times and see what happens.
EDIT: The lease expire time seemed to be the culprit. The default lease expire was 24 hours. I left the CenturyLink DHCP server enabled and changed the lease expire time to 5 minutes. It made it past the 5 minute mark.
More testing to come.
-
@preston Yes.. you have to set your pfsense CL WAN to static and use something like 192.168.0.5 as its address and 192.168.0.1 as its gateway.
-
@chpalmer said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
@preston Yes.. you have to set your pfsense CL WAN to static and use something like 192.168.0.5 as its address and 192.168.0.1 as its gateway.
I did try those settings in pfSense but couldn't get the CenturyLink WAN connection to show online. So, for now at least, I have CenturyLink as WAN2 with the IPv4 config type as DHCP.
-
@chpalmer said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
@preston Yes.. you have to set your pfsense CL WAN to static and use something like 192.168.0.5 as its address and 192.168.0.1 as its gateway.
I think in @preston 's and my setup it's OK to leave it as DHCP if we are reserving the address in the CL modem for the MAC address of the pfSense interface. It's worked for me so far. I have the DHCP service active on the CL modem, reserved an IP address of 192.168.0.2 for the pfSense MAC on WAN 2, and kept the pfSense CL WAN set to DHCP. So far so good. The connection has been solid other than the daily 4:00 AM EST brief down time.
The only problem that remains for me is that now I have CGNAT on both of my connections. Used to be able to use the CenturyLink connection for Dynamic DNS ....which gave me a remote gateway into my network via WireGaurd. So this is now toast as well as some other port forwarding like XBOX open NAT and a handful of others.
Would really LOVE to better understand what happened that caused the transparent bridge mode to just stop working after it worked for nearly two years.
-
@jimeez said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Would really LOVE to better understand what happened that caused the transparent bridge mode to just stop working after it worked for nearly two years.
I agree. Things worked just fine for a long time on my end too.
-
@jimeez said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
The only problem that remains for me is that now I have CGNAT on both of my connections. Used to be able to use the CenturyLink connection for Dynamic DNS ....which gave me a remote gateway into my network via WireGaurd. So this is now toast as well as some other port forwarding like XBOX open NAT and a handful of others.
-
I have had great success with Tailscale to access my network while away. Its free and gets around CGNAT.
-
Not sure what DYN DNS service you are using, but I noticed that there are Dynamic DNS settings in my CL modem interface.
-
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
I have had great success with Tailscale to access my network while away. Its free and gets around CGNAT.
Are you running that on your pfSense device?
-
Yes, there is a Tailscale pfSense package. I am able to access the home (pfsense) network with my phone and laptop when I'm away.
-
P preston referenced this topic on
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Yes, there is a Tailscale pfSense package. I am able to access the home (pfsense) network with my phone and laptop when I'm away.
Yep. Got that up and running no problem. I really like having it instralled on the pfSense device rather than a client machine like how I was using WireGuard (on an unRAID box). But I don't see how this is going to help get around the CGNAT specific to port forwarding for things like the XBOX and say a bittorrent client. I still cannot get an open NAT on the XBOX.
But anyway, I'm (mostly) very satisfied with this current solution. It's got me back to a solid stable dual WAN failover setup and has helped me iron out a couple other kinks in my network as I started fresh from scratch on a new device. Can't thank @chpalmer enough for his suggestion.
-
@jimeez said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
The only problem that remains for me is that now I have CGNAT on both of my connections. Used to be able to use the CenturyLink connection for Dynamic DNS ....which gave me a remote gateway into my network via WireGaurd. So this is now toast as well as some other port forwarding like XBOX open NAT and a handful of others.
I was thinking more about this issue today. Is your CenturyLink really a CGNAT connection?
Can you use a policy routing rule to send the device you want out through the CenturyLink WAN? For example, I set up a rule where my Synology NAS uses only the CenturyLink connection. I use Synology's DYNDNS service, opened a port on the CenturyLink WAN, and use that for an OpenVPN connection. Would something like that work for your setup?
-
@preston said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
Is your CenturyLink really a CGNAT connection?
You know, I'm not really sure anymore. I just spent some time reading up on it and running some tests. Apparently it's not. And it seams like the StarLink connection no longer is either.
I'm pulling a 98.97.xx.x IP address for the SL connection and a 75.165.xx.xxx IP address for the CL connection. pfSense sees them as 100.64.x.x and 192.168.0.1 respectively, but when I check my IP address on "what'smyIP" that's what I get. The XBox now shows open NAT to boot. So I'm not quite what I did (if anything) to fix this. But it's working now. Maybe the TailScale settings I applied did something? I also enabled UPnP & NAT-PMP.
Whatever happened, everything is back to normal. Better than normal actually.
-
@jimeez said in Dual WAN Fail-over Issue - Tier 1 WAN frequently failing upon activation of the second Tier 2 WAN:
I also enabled UPnP & NAT-PMP.Whatever happened, everything is back to normal. Better than normal actually.
Good deal. Just a guess but I would think that UPnP and/or NAT-PMP would help.
Thanks to you and @chpalmer for solving this issue!
