CARP + Multi-WAN



  • Is CARP+Multi-WAN possible?

    We have a successful CARP implementation (2x Netgate boxes running v2.1.3 AMD64) and wish to implement multi-WAN for fault tolerance/load-balancing.

    My questions are:

    1. Can pfSense do CARP along with Multi-WAN?
    2. If so, how does that work since the WAN gateways are in different subnets from each other? (two completely different public IPs)

    thank-you

    edit: added network diagram for future reference // Google indexing
    ![CARP plus Multi-WAN.png](/public/imported_attachments/1/CARP plus Multi-WAN.png)
    ![CARP plus Multi-WAN.png_thumb](/public/imported_attachments/1/CARP plus Multi-WAN.png_thumb)


  • Rebel Alliance Developer Netgate

    1. Yes
    2. They must be on separate physical interfaces or VLANs, not two subnets on the same interface.

    Though using two subnets on one interface does work with CARP, it does not work well for multi-wan even without CARP.



  • @jimp:

    1. Yes
    2. They must be on separate physical interfaces or VLANs, not two subnets on the same interface.

    Though using two subnets on one interface does work with CARP, it does not work well for multi-wan even without CARP.

    Thank-you. So I'm assuming the switch on the other side of the link must support VLAN trunking to the pfsense interface, if only one physical NIC is used?

    Just for future reference + Google indexing, I attached the network diagram to the original post. For ease of configuration we're leaning towards the option on the left (no VLANs).



  • Hi vocatus,

    I'm planing on implementing the same thing, multiple WAN with CARP.
    How did your implementation go? Was it successful?
    Hope to hear from you soon.

    Cheers!

    novice



  • @network.novice:

    Hi vocatus,

    I'm planing on implementing the same thing, multiple WAN with CARP.
    How did your implementation go? Was it successful?
    Hope to hear from you soon.

    Cheers!

    novice

    Hi network.novice,

    It actually went really well. I posted a diagram of the final setup here on Reddit.

    Let me know if you have any questions.



  • That's awesome vocatus! Congratulations mate!

    Is it possible to have 2 different ISPs as WAN right? Each ISP would belong a separate VLAN?
    Did you implement Traffic Shaping/QoS on your network design?
    I would greatly appreciate some tips from your current network.

    Regards,

    novice



  • @network.novice:

    That's awesome vocatus! Congratulations mate!

    Is it possible to have 2 different ISPs as WAN right? Each ISP would belong a separate VLAN?
    Did you implement Traffic Shaping/QoS on your network design?
    I would greatly appreciate some tips from your current network.

    Regards,

    novice

    Yes, it's possible to have 2 different ISPs. All that matters is that both pfSense boxes can see both WAN connections. So the way we did it was have a smart switch in front of the pfSenses, with three ports per network (1 for uplink, and one for each pfSense box) in their own private VLAN. This way we can use the same switch for all five networks (WAN, DMZ, WIFI, LAN, SERVERS). You could of course do it without using VLANs to break up the switch ports, but then you'd need a separate physical switch per network.

    For each network you need:

    • 1 IP address per pfSense box (total: 2)
    • 1 "virtual IP address" on the same subnet to share between them (total: 1)

    Basically the active firewall "holds" the virtual IP address and that's what all hosts talk to. If it fails, the other firewall "grabs" that virtual IP and starts answering to it.



  • Hello vocatus and a big thank you for the great diagram you posted on reddit.

    I am in the planning phase for such a setup. Some questions I could not find answers to:

    1. CARP runs active/passive or active/active? I am asking since I will run into problems with PPPoE dial up connections (obviously I cannot have both firewalls dialed in simultaneously…)
    2. I guess from your diagram you assigned a public IPv4 address to the virtual WAN IP. Any reason to do so? For my Multiwan Setup I can only use /32 subnets.

    Thanks!



  • @helge000:

    Hello vocatus and a big thank you for the great diagram you posted on reddit.

    I am in the planning phase for such a setup. Some questions I could not find answers to:

    1. CARP runs active/passive or active/active? I am asking since I will run into problems with PPPoE dial up connections (obviously I cannot have both firewalls dialed in simultaneously…)
    2. I guess from your diagram you assigned a public IPv4 address to the virtual WAN IP. Any reason to do so? For my Multiwan Setup I can only use /32 subnets.

    Thanks!

    Hi helge000,

    1. CARP runs active/passive - one primary firewall does all the routing, while the secondary just sits there waiting for a failover. If the primary comes back online after failing, they will "fail back" to the primary (at least as of v2.1.5-RELEASE of pfSense).

    2. Yes, in order to use dual firewalls you must use a Virtual IP address. If you read my above comment, each physical interface will have an IP address, and then there is a third, shared "virtual IP" that both firewalls use. The primary firewall "holds" the address and handles all traffic destined for it, and the secondary firewall will only "hold" that address when the primary firewall fails. So you cannot have a /32 on your WAN connection - you must purchase additional public IP addresses.



  • vocatus, again, thank you for your insight.

    I hope I can tap on it a little further, since some things are unclear. I hope you excuse my ignorance ;)

    While the concept of having a virtual IP for HA purposes is clear for my intranet, it is not so much for the external connections. This would implicate the following things:

    • Get more IP addresses
    • Redo all my external DNS A records, pointing services running in my DMZ to these virtual IPs

    ATM, I cannot see the point on the WAN side. I read in case of a failover WAN interfaces will be offline for about 10min anyway (pfSense) and then the backup PPPoE connections would need to dail in again; getting the same IP's from the ISP.

    In this case firewall states will be lost and connections reset. My best guess is after +10min they will have timed out anyway.

    This behavior is completely fine with me. pfSense is very reliable and I only really need it in case of updates or a major power failure.

    Thanks again,
    Helge



  • @helge000:

    vocatus, again, thank you for your insight.

    I hope I can tap on it a little further, since some things are unclear. I hope you excuse my ignorance ;)

    While the concept of having a virtual IP for HA purposes is clear for my intranet, it is not so much for the external connections. This would implicate the following things:

    • Get more IP addresses
    • Redo all my external DNS A records, pointing services running in my DMZ to these virtual IPs

    ATM, I cannot see the point on the WAN side. I read in case of a failover WAN interfaces will be offline for about 10min anyway (pfSense) and then the backup PPPoE connections would need to dail in again; getting the same IP's from the ISP.

    In this case firewall states will be lost and connections reset. My best guess is after +10min they will have timed out anyway.

    This behavior is completely fine with me. pfSense is very reliable and I only really need it in case of updates or a major power failure.

    Thanks again,
    Helge

    Hi Helge,

    OK, no problem, happy to help.

    Yes, you will need more external IP addresses on the WAN. This is because, as I stated before, each interface must have an IP address, and there must be a third IP address that all systems communicate with. The interface-specific IP addresses are never used directly - the Virtual IP is what all traffic gets sent to, and only the active firewall responds to it. Now, I don't have any experience with PPPoE, so I might be giving you information that only applies to a regular coax or fiber connection, but AFAIK the requirement is still the same.

    Basically the idea is that everything talks to the Virtual IP address, and it never changes - one of the two firewalls will respond to it (the active one).

    Using CARP+pfsync the firewall states are maintained during a failover event - the slave firewall is constantly updating its state table with a copy of the primary state table, so in the event it fails it can pick back up where it left off.



  • OK, no problem, happy to help.

    Really appreciated.

    In the meantime I read more about CARP and 'getting' the picture a bit better. Using vIPs for WAN makes perfect sense
    Sadly because of this requirement I have to bury the CARP way; I checked with my ISP and there is no way I will get a bigger subnet for a reasonable price with them.

    ATM I can only think of a more complicated setup involving a separate intermediate private subset for the WAN part. Though I think I will end up in a troubleshooting hell I will try this in a lab.

    Cheers,
    Helge



  • @helge000:

    OK, no problem, happy to help.

    Really appreciated.

    In the meantime I read more about CARP and 'getting' the picture a bit better. Using vIPs for WAN makes perfect sense
    Sadly because of this requirement I have to bury the CARP way; I checked with my ISP and there is no way I will get a bigger subnet for a reasonable price with them.

    ATM I can only think of a more complicated setup involving a separate intermediate private subset for the WAN part. Though I think I will end up in a troubleshooting hell I will try this in a lab.

    Cheers,
    Helge

    Yeah, at that point you might better off investing in some quality hardware that will be reliable rather than a dual-box setup. Good luck!



  • Hi vocatus,

    Congrats for the complicated setup.

    It is possible you to make us a tutorial with step by step or have screenshots of what is needed to configure this? Don't forget to hide the public IP's.

    Thanks.



  • @jnevestdl:

    Hi vocatus,

    Congrats for the complicated setup.

    It is possible you to make us a tutorial with step by step or have screenshots of what is needed to configure this? Don't forget to hide the public IP's.

    Thanks.

    Hi jnevestdl,

    It's been quite a while since I designed this and I'm at a different position now, so I don't have access to the GUI to take screenshots. I can try to answer questions for you though if you have any.


Log in to reply