Multipathing 3 routers when one isn't pfSense
-
@michmoor great so its not me having a brain fart ;) heheh
-
My suggestion would be to create a clearer diagram that shows how things are routed today and what you want your design to look like tomorrow.
It seems that the 75.60.X.X/29 is your public assignment. The 10.10.111.X is your transit subnet (you're using /30s). -
@michmoor @johnpoz All good. It was a late night drawing that made sense to my tired brain.
Here's an updated diagram of my setup.
Due to some peculiarities with the UniFi Talk application, I am forced to use this split configuration for the dual ISPs. My original configuration had the 7100 dual homed, but despite rules to block all VoIP related traffic out the Spectrum interface, the UDM SE would still pick up that WAN IP on occasion and would result in loss of service due to no route from Spectrum to the AT&T provided /29.
I want to ensure I route traffic between the physical and virtual routers using the transit networks so I don't have to deal with as many NAT rules. AT&T provides a single /29, with each router getting one IP, and several VMs taking the remainder. If AT&T goes down, traffic should only route via transit nets, not point to point via the public IP interfaces. NAT would be handled by the UDM SE going out the Spectrum interface.
If Spectrum were to go down, I'd want the UDM SE to route traffic over the public IP interface, to the 7100, then out via AT&T. This would ensure the VoIP traffic still has a public IP available, and NAT is minimized.
Hopefully this clarifies what I'm trying to do. If not, let me know and I'll try to further explain.
Thanks!
-
@pokrifchakd so you advertise this network you own.. You have your own ASN, and your providers allow you to advertise and if one goes down you can come in the other one.
-
@johnpoz No, and that's the problem. The /29 is "owned" by AT&T and I'm just "renting" it. Also, since I only have a /29 to use, I am unable to get an ASN from ARIN (minimum is /24). I also don't have an extra $13K-$17K sitting around.
Here's the original setup I had.
The 2 ISPs connected to the 7100 were in Load Balancing mode to begin with, then I switched to Failover. I had rules to prevent STUN and VoIP traffic from exiting the Spectrum interface, but something in the way the UDM checks it's public IP would still occasionally pick up the Spectrum IP as the Gateway. Since the UDM's WAN was using AT&T provided /29, traffic would fail to route.
This led me to the current configuration, and the desire to make the back end as robust as possible.
-
@pokrifchakd said in Multipathing 3 routers when one isn't pfSense:
The /29 is "owned" by AT&T and I'm just "renting" it
Well how would you have that network come in spectrum? So you want outbound traffic only to be natted out the ATT connection, what is the network that comes in the spectrum connection?
You list 2 providers but only 1 network.. If you do not own the network - how would that work? It wouldn't..
you could for sure nat out either connection for outbound traffic. But you sure can not do anything inbound across providers with only one network that you do not own.
This /29 is routed to you? Or your just attached to it on att.. How exactly are you getting 75.60.x.140 to pfsenseVM? Or those 137-139 to elsewhere. Are you just doing VIPs and nat, do you have a bridge setup?
-
@johnpoz Spectrum provides me with a single IP of 173.175.98.X/22 to the WAN interface of my UDM SE, or in the old config to WAN2 of the 7100.
For the /29 from AT&T, it is routed on their end to the public IP on my 7100 WAN, and is assigned to a LAN interface, which provides the connectivity to the UDM and the VMs (servers and pfSense+) via VLAN.
The more I think about it, it may be easier to just remove the AT&T provided IP from the UDM SE and focus on how to utilize the transit networks to move things around if either ISP or associated router goes down.
If AT&T goes down, then all traffic would have to go out Spectrum. I'd need a secondary route to the Internet on the 7100 that points to the UDM SE. If any of the virtual enclave with an AT&T /29 address needs to go out, it would hit the 7100, as the Gateway, then route to the UDM SE for NAT out. If the 7100 is down as well, the pfSense VM would need to route via transit network to the UDM SE. My hosted services would go down, but I would still have core functionality with VoIP, DNS, and browsing.
-
@pokrifchakd said in Multipathing 3 routers when one isn't pfSense:
tilize the transit networks to move things around if either ISP or associated router goes down.
Yeah that would be easier for sure..
-
Are these different sites that the provider comes into you or same location?
-
@michmoor Single office location with a central PoP.
-
@johnpoz Yeah, the public IP from my /29 made sense in the old scenario, since the 7100 was doing the load balancing/failover. The VoIP system still needed a public IP to function optimally.
One other thing to consider is actually leaving the public IP, but configuring it as a WAN2 connection. Ubiquiti recommended only to use failover, not load balancing, so it may work if Spectrum goes down.
