@Magoogle
Check Status > Gateways.
Is the tier2 the default now?

@netblues said in Configure an PPPoE on an CARP IF:

This never really worked. pppoe running on a carp interface isn't an option.

It sure is. We have a few customers set up that way and working well - within boundaries. Of course in such a setup the secondary node of a CARP setup won't easily have internet which is/can be a problem and as such the setup isn't really recommended. But it IS working though. It's important to check though that both nodes on it's WAN "carrier" interface are connected to each other and the DSL modem correctly so both have access to dial-in if needed. If that's set up correctly it's a relatively simple setup:

either node gets the physical interface for the PPPoE connection assigned with its own IP, say 10.12.34.251 and .252 check pinging from one to the other and back (allow ICMP on that interface first) then add a CARP VIP to it, e.g. .254 - that one should now be active on the primary node anad backup on the secondary node. If that is not the case you don't need to proceed with PPPoE stuff. That's basic CARP that should be working first! If that's running you can now add the PPPoE interface but as carrier you don't choose your physical interface BUT the NEW CARP VIP you created (yes, that .254 one from above!) This ensures the PPPoE connection switches from node 1 to 2 and back if needed. Then set up PPPoE as usual. When finished assign that interface (pppoe0) as your WAN_PPPoE or something else like it. THAT one is your actual WAN, the other physical interface and the VIP on it are only a sort-of transfer/carrier network.

Cheers

@reberhar Yes of course. Why would you want to choose a gateway for every rule.

I was just caught in the verbage.

This seems to be the same issue as https://redmine.pfsense.org/issues/13569 -- I'd love to debug this further but I am not sure what else to look into.

contacted support, reinstalled from scratch, same errors again.
provided diagnosis data, support reproduced issue, now we have this issue in redmine: bug

it happened to me several times,
don't change the port, delete the backend and redo it

@jimp

These two pfSenses are in the middle of network, the issue didn't affect interfaces faced to syslog server, syslog source set as local pfSense interface, not as CARP VIP. We see in syslog other messages like FW rules actions during the issue period, but not CARP-related ones.

@damianhl If it has ZFS there is a Disks widget that can expand to show details:
e80dceed-465d-4da2-9b03-30e91c0a4dcd-image.png

Not sure about hardware RAID, have never used it. Unless FreeBSD/pfSense includes a driver the pfSense OS will probably only be able to see what the BIOS shows it.

DHCPD LOG.txt

@decibel83 A problem at Layer 2 is the most common cause.

FYI- You can disable NAT and route without also disabling the firewall.

Firewall > NAT, Outbound tab, set it to Disable Outbound NAT and save/apply.

@james92 Yes a dumb switch is fine.

@davidredekop Interesting. I have never had to change anything in proxmox for CARP.

As an aside, while fc00::/7 is the ULA network space, fc00::/8 is currently undefined. fd00::/8 is proper ULA addressing. Recommend implementing RFC 4193 and randomly selecting a /48 for ULA usage.

i was able to track down a bit of a solution
we had disabled hardware offloads , this is now turned back on which make xmlrpc sync much quicker and lower load and cpu.

also

we have two wans, on each wan we had two openvpn servers listening for different purposes, 7-8 years ago we were told that its best to listen on localhost with each vpn server, then nat port forward each external port so that each wan can listen on the same server, it appears if we do this now, each time an xmlproc sync occurs it causes pfctl and the reload scripts to thrash and loop 3 or more times.
we this this occuring over and over with localhost
php-fpm[6973]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use VPN

the solution now is listening on a single carp ip, this means were not able to openvpn in the backup wan, but atleast vpn works on master and backup servers, just not the backup wan

all xmlrpc sync is re-enabled and no CARP timeouts so far.......

@nocternal Yup, I'll be doing just that, super nice we can do "micro" patches like this.

Thanks again!!

Just to add for anyone else coming across this issue.

Adding a vlan and therefore triggering a configuration reload and mini failover, caused exactly the same issue. Which was not fixable with restoring a configuration backup or even a restart of both firewalls.

Applying this patch:
Fix automatic firewall rules for HA DHCP server failover (Requires reboot or filter reload to activate, Redmine #13965)

Fixed the issue with the DHCP server.

The issue showed in Status / DHCP Leases a permanent status of My State - 'Recover', as well as previously mentioned 'communication-interrrupted'

@derelict At the moment, I only have one firewall.

I will add the other one later when I have more WAN addresses.