1x pfSense+PPPoE to 2x pfSense+CARP+?

vbentley

I need some advice for an upgrade on a single pfSense host.

The WAN interface uses PPPoE to a directly connected ADSL router. On successful connection a single static public IP address is provided on the pfSense WAN interface. The LAN and all other interfaces are private IP subnets rfc1918. What changes do I need to make on the 'internet connection' when upgrading to a redundant CARP installation?

I have been reading the old pfSense book this morning and I think that what I need is something that is between the diagrams on page 386 and page 400 of the book.

I have to submit a detailed request form to RIPE for the IP block required and I want to be certain that what I think I will be implementing is actually possible.

I think I need six public IP addresses as follows:

Public IP address for CARP VIP
Public IP address for pfSense Primary host
Public IP address for pfSense Backup host
Public IP address for 1:1 NAT host 1
Public IP address for 1:1 NAT host 2 (spare)
Public IP address for Ethernet interface on ADSL router

Instead of setting up the WAN interface on pfSense to PPPoE to the ISP, this function will now be configured as PPPoA from the ADSL router to ISP and a static IP from the the block provided by RIPE is configured on the router's Ethernet interface. An additional Ethernet switch is required with at least three ports for connecting the two pfSense hosts to the router.

Does this sound correct? Am I missing something?

vbentley

I have attached a PNG diagram to illustrate what I am hoping to put in place. The key to the annotation is as follows:

For WAN1
1. Public IP address for CARP VIP
2. Public IP address for pfSense Primary host
3. Public IP address for pfSense Backup host
4. Public IP address for 1:1 NAT host 1
5. Public IP address for 1:1 NAT host 2 (spare)
6. Public IP address for Ethernet interface on ADSL router

For WAN2
7. Public IP address for CARP VIP
8. Public IP address for pfSense Primary host
9. Public IP address for pfSense Backup host
10. Public IP address for 1:1 NAT host 1
11. Public IP address for 1:1 NAT host 2 (spare)
12. Public IP address for Ethernet interface on ADSL router

Colour coding:
RED = Public IP, WAN networks
BLACK = Private IP rfc1918, Backbone and sync networks
YELLOW = Private IP rfc1918, DMZ host network for 1:1 NAT to host
BLUE = Private IP rfc1918, Database Host Network
GREEN = Private IP rfc1918, LAN

Dotted lines = normally backup connections

This will be my first production CARP install. I am grateful for any comments from those that have done something similar before.

![CARP Multi-WAN.png](/public/imported_attachments/1/CARP Multi-WAN.png)
![CARP Multi-WAN.png_thumb](/public/imported_attachments/1/CARP Multi-WAN.png_thumb)

jimp

PPPoE cannot be made to work with a proper CARP setup. The PPP layer would have to be handled at the modem, exposing a routed subnet to the WAN side of pfSense with sufficient IP addresses for CARP to function. pfSense itself cannot have PPPoE WAN interfaces if you want a proper, fully functional, HA configuration.

pigbrother

@jimp:

PPPoE cannot be made to work with a proper CARP setup. The PPP layer would have to be handled at the modem, exposing a routed subnet to the WAN side of pfSense with sufficient IP addresses for CARP to function. pfSense itself cannot have PPPoE WAN interfaces if you want a proper, fully functional, HA configuration.

Thus if my ISP connection is Ethernet with PPPOE authorization there's no way to use CARP?

jmilne

@jimp :

I'm replying here instead of opening yet another CARP with PPPoE thread as there are loads of them and the answer is always the same. I am sure it is annoying constantly having to read the same questions over and over again. I would just like to check I understand the answer.

I understand your answer means: configure the modem (not pfsense) with the details (username, password, etc) supplied by the ISP, ensure the LAN interface on the modem has a subnet with at least 4 usable private IP addresses (1 for the modem interface and 3 for CARP on pfsense), set the WAN interface on pfsense to be "Static" and then set up CARP as normal? No further config on the modem required?

And the resulting HA would then be no different to if pfsense were connected to a leased line WAN? As far as pfsense is concerned, it doesn't care? This setup would still work if added to a gateway group (a VDSL line in a group with a leased line)?

And this is the same for any PPPoE connection (doesn't matter if it is ADSL/VDSL/whatever)?

Any further info would be greatly appreciated. I'll put this in a new thread if preferred.

netblues

Yes, this is correct.
All carp interfaces require a lan interface and three ip's.
If the xdsl router routes a public subnet then pfsense will be doing nat.
If xdsl provides a private lan, then the router will be doing nat, and most probably pf would be doing a second nat too.

jmilne

@netblues : Thank you very much for your reply - so appreciated - I'm on a crazy steep learning curve - happy to google (been googling) but finding search terms not specific enough because I don't know the terminology - no expectations on anyone though.

I've seen the phrase "routes a public subnet" on several answers - what does this mean exactly? Is this what is happening for our leased line i.e. the public subnet is visible to pfsense and I can set the WAN interface on pfsesne to be a static IP (the first useable address in the range from the ISP)?

I think our scenario for our secondary ISP will be the "xsdl provides a lan". Our secondary ISP provided a free router. However, the configuration is locked down e.g. we can't turn the firewall on the device off. So I think we need our own modem that we control fully. As one of the setup steps involves setting up the LAN interface with a private subnet, I think I this means we are in "xsdl provides a lan" category.

Is it possible to choose between "routing the public subnet" and the "xsdl router providing a private LAN"? If there is a choice, is one way preferable to the other? Apologies if that is a dumb question but I'd like to be sure I understand all my options.

Also, just checking, by "xsdl router" do you mean router or modem? Do we actually need our own xsdl router to replace the one the ISP sent? Or will a modem suffice?

My understanding from the docs is that pfsense will do auto-NAT on any interface that has a gateway (which it uses as an indication that the interface is a WAN interface). I'm assuming that in this case, I set the gateway to be the interface address of the LAN interface on the modem/router that PPPoE has been configured on...?

netblues

@jmilne A public, or not rfc1819 or routable address is one that is assigned explicitly to you by your isp. Same goes for a subnet.
Apart from that, all ip's are equal.
The basic thing is that ppp and carp doesn't work well together.
Keep that in mind
So in all cases we need something to turn ppp to lan before it hits pf.
And this is always a router device, never a modem.(we don't want any form of ppp bridging on ha pf too)
If your provider supplies a public lan then use it
If not use the private one supplied by practically all free routers.
A private lan is natted to the public wan ip of the router
A public lan is routed to the lan interface of the router.
Forget pf auto nat with carp.
You will need to configure it manually.

ANd correct, yoyr gateway is always your isp router ip

jmilne

@netblues : Thank you again - that's cleared up the IP address confusion (and yes I had read in the book that auto-nat wasn't supported with CARP - forgot that in the confusion with PPPoE).

Our usual networking hardware supplier recommended NETGEAR DM200-100EUS ADSL/VDSL Modem to replace the ISP supplied router. Reading the manual shows it appears to have routing features. Does this qualify it as a "router device" even though it's being called a "modem"? The constant interchangeability of the two terms is driving me nuts. Once I've nailed down what actual type of device I need, I can order one and start an actual experiment.

Appreciate your replies very much - thank you for your time and patience