PPPoE is no longer always dynamic

[MrPete]

CenturyLink (aka Lumen) is a reasonably large ISP (about 5% of the ISP market - 5+ million customers.)

They provide gigabit fiber service with static IP via PPPoE with a specific VLAN tag. It's actually not too bad: I get a bare Cat6 Ethernet to plug into my system, and it "just works" when properly config'd.

From an HA/CARP perspective on pfSense, this is pretty painful. To make it work, I had to:
a) Create a VLAN using the WAN hardware interface and the needed VLAN tag. [SAME as non-HA]
b) Create a CARP2WAN static IPv4 interface using (a), specifying the IP for this server (e.g. x.y.z.2)
c) Create a VIP for the "real" WAN static IP
d) Create a WAN PPPoE interface using a random temporary physical interface (the VIP is not available in the Interfaces Assignment UI)
e) Go to the PPP tab to edit the pppoe0 interface:
- Change interface to the VIP
- Re-enter the password (same bug as recently reported: there's no way to save a form without specifying both PW's in pfSense, if it has PW/PW-confirm fields.)

THAT appears to work...

Hopefully someday this can be simplified to be not much more complex than setting up other CARP interfaces ;)

(REALITY: I had to work out how to do the above. It's not exactly documented ;) ... I've put a lot of hours in on this. Hopefully this workaround will be helpful! :-D

[MrPete]

[UPDATE March 2022. This is working even better. Thanks to @viragomann who was quite close on the needed gateway setup.]

Revised instructions. This actually works for me, under most conditions. Exceptions/anomalies to follow:
a) Create a VLAN using the WAN hardware interface and the needed VLAN tag. [SAME as non-HA]
b) Create a CARP2WAN static IPv4 interface using (a), specifying the IP for this server (e.g. x.y.z.2 or 3 - private IP ok even for WAN)
c) Create a CARP VIP for the "real" WAN static IP, based on CARP2WAN interface (skew 0 on primary, 100 on secondary)
d) Create a WAN PPPoE interface using a random temporary physical interface (the CARP VIP is not available in the Interfaces Assignment UI)
e) Go to the PPP tab to edit the pppoe0 interface, changing the interface to the VIP created in (c)

• Make SURE the underlying interfaces match! (LAN-LAN, OPT1-OPT1 etc)
• Fix all "WAN" IP or interface references to point to the actual static IP...
  • Outbound NAT (manual)
  • Inbound firewall and port forwards
• Fix all interface references if/as needed
  • DNS

f) Now take care of secondary CARP access to Internet

• In System->Routing create a gateway to LAN CARP interface (on my setup it is *.1.1). This one is NOT monitored and is assumed to be 'up'.

• Now make a gateway group, with normal internet gateway as Tier 1, and the new gw Tier 2.

• Save, then set the gateway to be this new GW group.

• Adjust DNS Resolver: it needs to include accessing upstream via the LAN CARP IP. Strange but necessary ;)

• Test sync. Ensure the above changes are visible to secondary.

• Test secondary: it should have Internet access.

• Test failover.

• Is the skew adjusting correctly? (ifconfig on both primary and secondary)

At this point my main anomaly is: further adjustments to primary cause secondary to lose the Tier 1 setting in the GW group... maybe due to different names.

[MrPete]

Followup a few months later:

• This does appear to be working reliably

Note that with a single WAN connection, I find NO way for the secondary pfSense box to access the Internet for updates, etc (unless it is made Master temporarily). I've tried many potential workarounds with no success. (If someone has a solution to that, I'm all ears )

[viragomann]

@mrpete
On the master go to System > High Availability Sync and disable the sync of "Static Route configuration" if it's enabled to avoid that the following settings get overwritten.

On the secondary go to System > Routing > Gateways and add the primarys LAN IP as gateway on LAN interface.
Then add a new gateway group, set the WAN GW as Tier 1 and the primarys LAN you've created before as Tier 2.

Go back to the gateways tab and set this gateway group as default gateway.

You may also do the same on the primary, but with the secondarys LAN IP instead, to get internet access when it's in CARP maintenance mode.

[MrPete]

@viragomann That's what I did, or at least thought I did. Never could get it to work -- not even a ping.

I'll try again when I get some more round tuits :)

[MrPete]

@mrpete @viragomann
I've got it working close to 100% now :)