Multi-WAN with dynamic PPPoE looses addresses

TGX03

I want to start out by stating that I have highly unusual setup. I'm running pfSense with two upstream DSL connections from Inexio in Germany which use PPPoE. One connection is used by my parents, the other one by me. We have this setup so we each have a dedicated line, but we can still have fast traffic between our networks. Since it's the same ISP in the same house, I had to solve the usual problems with gateways and dpinger. I also had the additional issue that Inexio does not like when multiple DHCPv6 request come with the same DUID. I fixed that by patching dhcp6c to use different DUIDs on different interfaces. I also know that will never work in bare pfSense, as it violates the official standard.

Now, the confusing part: When I boot pfSense up, everything works. The downstream interfaces get the prefixes from their respective upstream interface, and routing works.

However, if one of the line reconnects for whatever reason, it breaks. The line that didn't reconnect survives it pretty unaffected, but the line that reconnected behaves very strangely:

At first, it gets assigned its normal link-local address and a gateway.
A bit later, it gets its GUA over SLAAC. Inexio does not give out addresses over DHCPv6.
Then, dhcp6c starts and assigns addresses to the downstream interfaces. It also reassigns addresses to the downstream interfaces of the other upstream interface, as currently, only one dhpc6c daemon at any time is supported.*
Shortly after, the GUA and gateway of the upstream interface vanish again. Afterwards, the IPv6 addresses of the downstream interface slowly start to disappear as well. Not all at the same time, and sometimes, one or two survive completely. They are however unable to connect to the internet, because the gateway disappeared.

This happens for both lines. Meaning, if one of them goes down during operation, I have to fully reboot pfSense to get it working again. Inexio does not do forced daily reconnects, so that wouldn't be a problem, but the line to our house is heavily damaged, and since fiber is currently being built, Deutsche Telekom, who owns the line, can't be bothered to fix it. So currently, anytime one of the two lines resyncs, I have to reboot pfSense. A "reroot" does not suffice.

*I also modified the calling code for dhcp6c. For some reason, when reconnecting a line, pfSense only gives one of the pppoe connections as an argument to dhcp6c, so it only requests a prefix on one line. I did modifiy that behavior so it would always request for both lines, because otherwise the downstream interfaces wouldn't get any addresses.

So the core issue currently is the gateway and GUA of the PPPoE connection disappearing. I'm really out of ideas why that happens, because there is no report anywhere on it. Not in pfSense's logs nor in dmesg. I tried route monitor, and it does show the removal of the address, but it reports its own PID as the PID responsible for that action, meaning the kernel did it.

And, since this works without issues directly after a reboot, it really doesn't make sense to me why it would loose everything it got over RA when reconnecting later.

Does someone have an idea where I could go look to find out more why this happens? I understand that it is a really unusual setup, but I guess it's worth a try to ask here.

netblues

You have already created a support mess, since each time something is published upstream, you need to patch AGAIN.
And all this in a public facing security device.

What you are running isn't that uncommon, but here are your options..
a. Disable ipv6 Unless your provider isn't offering anything else, thats the way to go
Ipv6 fails miserably by design in multihomed environments.
b. Mess with nat64 and split traffic accordingly
c. Just upgrade your line, and forget multihome.
Nowdays getting 1G for a home connection is normal and its quite difficult to saturate. Use fqcodel if needed.

If you insist
Finding what goes on wire between a reboot and a reroot requires another device with rmon and wireshark.

(or a virtualized pfsense)