Multi-WAN IPv4 failover with BGP + NAT without dropping connections

[davidbalbert]

tl;dr: is it possible to have seamless failover (no dropped connections) using v4 with BGP if you have to NAT because you don't have enough public v4 addresses?

The details

I'm in the process of planning a multi-WAN failover setup. We'll have two connections, a dedicated 1Gbps fiber as a primary, and a cable modem as a backup. We'll have BGP sessions on both connections (technically, our cable ISP won't speak BGP with us, but all traffic on the cable modem will be routed over a tunnel to an ISP that will).

It's important that our users experience zero downtime when our primary connection goes down (no connection drops, etc.). This is easy with v6: when the primary connection goes down, we'll announce our v6 address space over our backup, and traffic will keep flowing (in this scenario, each LAN user has a publicly routable v6 address).

For v4 this seems more complicated. We'll have a /24, but this isn't enough addresses for all the users on our network. This means we'll have to NAT our LAN. So here's the problem:

• Assume our /24 is 3.2.1.0/24.
• ISP1 gives us an address of 1.1.1.2 for WAN1 (the primary)
• ISP2 gives us an address of 2.2.2.2 for WAN2 (the backup)

Under normal operation, LAN traffic leaving WAN1 will have its source address set to 1.1.1.2 by NAT. When failover happens, LAN traffic will start having its source address set to 2.2.2.2 when it leaves on WAN2. This is a problem. Any traffic that's connection oriented will fail when the remote recipient (or a firewall/NAT on the path to the recipient) sees a source address that it hasn't seen before, and the connection will have to be reestablished.

Is there a way to work around this problem?

The only thing I can think of is having some sort of virtual interface with a public IP address from our /24 (let's say 3.2.1.1) sitting between the LAN interface and the WAN interfaces. NAT would then happen on this virtual interface instead of the WAN interfaces. This way, the source address of all packets leaving our network would be 3.2.1.1, regardless of whether they left on WAN1 or WAN2.

Questions:

• Is this possible on pfSense? It requires some sort of custom topology within the router than I'm not sure how to put together.
• Is this the right solution to the problem?
• How would you set this up?

Unrelated BGP question:

• Is it possible to use addresses from our /24 for WAN1 and WAN2? I think the answer is "no" because that would require announcing a subnet smaller than a /24 to each of our ISPs, but I want to double check my thinking.

Thanks,
Dave

[dotdash]

I would set this up with a separate box doing the BGP connection to ISP1 and ISP2. This box would present your 3.2.1.0/24 network on it's 'LAN' side, which would be the gateway for the firewall, which would have a 3.2.1.x WAN. My 2c, YMMV.

[davidbalbert]

That makes sense, and it's a reasonable suggestion.

I'd like to do it with one box if I can, though. I'm the only person in our (small) org who does network administration, and it's only a small part of my job. The cost in complexity of having another box in our network closet is something I'd like to avoid.

[Derelict]

Yup. BGP in that case should be handled by a router that doesn't care what interface a packet arrives on because it is not maintaining firewall states.

ISP1    ISP2
 +        +
 +        +
 BGP ROUTER
     +
     |
     |
     +
  FIREWALL