Just a bit of help for anyone still dealing with this issue. Here in Chattanooga, TN we have EPB Internet that times out vip's after 4 hours of no arp. This thread has been extremely helpful. It is a bit easier to implement now. If you install the Filer and Cron package from package manager, you can drop this script right into a file and edit if needed. Schedule right from the GUI. No more ssh needed.

The only hiccup I ran into was when I copied the above script, I didn't notice that the <? was missing at the beginning and it kept failing until I hit the shell to see what was happening.

BTW, @rightnow version works perfectly on 2.5.1-RELEASE

@steveits Yes I need Snort on both interfaces, I like to know for example who from my family wants to download torrents.
I have Zabbix and FreeRADIUS package install along with Snort.

@joezyz
think about A Records later, first make the network work
configure it step by step
the gateway to the network will be the shared IP
it's easy only after you understand it

@bp81
Exactly. You configure the secondary WANs as private network, so that they can talk together. Then you hook up the CARP VIP on this interface on the master and add the WAN gateway in System > Routing > Gateways to this interface.

Internet access over the secondary WAN has only the router which has the master role. I.e. in case of failover to the secondary box it takes over the WAN2 CARP and gets access to the internet over WAN2.

In normal state when the secondary box is backup it can access the internet over WAN1. So WAN1 GW has to be set as default gateway.

There is also a workaround to get internet on the backup router over a single WAN connection and a single IP over the master, but that makes no sense in a Multi-WAN setup.

@binary_bandit I went with a solution roughly as explained by Mike here.. the advice came from elsewhere, but the comments were basically the same.

Have two sites both routable always, each with its own carp cluster (no pfsync across sites, not necessary for me), but only one is routed to at a time. I allow my upstream provider to route for me, but could do this myself later by enabling/disabling an IP at either site and have them route to that instead. Each site is completely independent and although they advertise 3 public ranges they both have their own native/local range of public ips too.

Really the concensus from everyone I've spoken to is to do this with switches and bgp not pfsense, which is a huge bottleneck - but it does work.

@ronrn18 I have a domain with cloudflare, that points to my wan IP. And I use haproxy to do ssl offloading of this service because its just a docker and https is not really supported.

I am not having any issues with this. I use a acme cert..

I can bounce off the proxy both internally, and externally my users are able to access it. I even share the outside 443 port being used with openvpn and have not problems.

@magnus-maximus said in VIP addresses stop working:

https://datatracker.ietf.org/doc/html/rfc3768#section-8.2

That seems to indicate what is included in the ARP IS AT response in the ARP protocol itself. It is silent about the source MAC address of the frame containing the ARP response.

8.2 pretty much describes what CARP does. The MAC address in the ARP response for a CARP VIP is always the virtual CARP MAC address.

What, exactly, is the ISP doing that is breaking things? Why are they not issuing another ARP request when they have traffic for an IP address after the ARP cache has expired?

@viktor_g

Thank you very much for your help and all details @viktor_g . I have successfully implemented the changes :)

Regards

@iptvcld
Interestingly. Didn't know that. Was assuming only the master is handing out DHCP leases and only the lease state is synced to the other node.

@iptvcld said in HA/CARP DHCP Lease Hand Out:

showing the DHCP server IP of my backup node interface IP

Normal:
https://forum.netgate.com/topic/166542/pfsense-ha-lan-interfaces-only/27

@bp81 To sync states (for a transparent changeover) you need identical interfaces, see this. Otherwise the hardware shouldn't matter so much. Presumably the secondary would be using the same memory and packages as the primary.

@steveits them this is not for a mortal like me...thanks buddy.

@viragomann Thank you. Working perfectly now.😊

Solved it.

Diagnostic Method:

Review ALL the basics... Interfaces are same, same order (easiest for me: check the Interfaces menu item links :) ) XMLRPC Sync setup is correct: correct IP, login, pw on Master. NONE of those on Backup. Sync setup is correct in other packages (depends on pkg) Fix any errors Now make a change in the area(s) that were not syncing

In my case:

Oops: I had an IP still in "Sync Config to IP" Then, make small changes as needed... changing one static DNS assign-> All transferred changing one HAproxy item -> All transferred changing one Cert item -> all sync'd incl old/bad certs gone

etc.

@mrpete FWIW,
This effect has simply disappeared over time.

I have no idea what the root cause was.

I can't say failover is 100% reliable yet, but it "mostly works" and that's good enough for now. I need to move on to more urgent issues.

@honest_matt Not documented, but here are some hints that may help. I'm still in the process:

Convert your existing setup to use CARP VIP's (Virtual IP's - Firewall->Virtual IP) as the primary IP, and an alternate IP for direct access to that box. The CARP IP should be the gateway for any VLAN etc. It also should be what is provided as DHCP and DNS IP in DHCP server.

Change your WebGUI to specify a specific port for SSL instead of the default 443. You'll want this later with HAproxy. (It's set in System->Advanced->TCP Port)

Add an additional interface just for sync. I call mine HA. Give it its own subnet, and add a rule for HA that allows the HA net to talk to HA net freely.

Do NOT yet define HA sync stuff.

Do a backup. Save the XML file. Examine the XML. Record the interfaces assigned to WAN, LAN, and OPT1-n -- the new/mirror box must have the exact same interfaces assigned the same way.

Put the backup XML on a USB stick, name the file config.xml

Set up the mirror pfSense. Reboot with the USB stick in place, and NOT connected to your WAN or LAN. It should auto-configure itself with everything from the primary box.

Attach directly to the new box. Change the interface IP's to be different from the other box. Leave WAN undefined for now. Once that's done you should be able to attach the HA Sync ethernet.

Follow standard instructions to define HA Sync / XML-RPC on Primary and Secondary. At this point, any changes on primary should propagate to secondary.

You're on your way... there is still more to configure.

WAN DNS and DHCP sync/failover Any other failovers

It's a pretty big deal ;)

@jimp Yep. A password plugin was injecting a PW into one of the two fields.

Thanks.

@mrpete said in Shell/Cmd line for HA/CARP/VIP troubleshooting and config?:

I'm realizing that getting HA/CARP/VIP running properly is quite the detailed process, in which it as raaather easy to break any ability to access pfSense through the GUI / (V)LAN / WAN.

So (of course) I went looking for shell equivalents to the various GUI configuration and status screens related to HA/CARP/VIP.

At the very least, I want to learn how to accomplish the following from the shell:

List and Remove CARP IP's (if busted, they will conflict with the other box) List and change (or at least clear) sync settings Turn off HAproxy Create a straightforward way to keep a backup HA machine connected enough to access the Internet

Answers:

ifconfig inter.face ip.add.re.ss -alias (temporarily removes any IP, including CARP VIP) See below for a console script that can disable CARP No solution found for sync settings, but a workaround: with CARP IP's removed from the backup pfSense, I could again plug in the ethernet and access the web GUI A console "svc" script (see below) can start, stop and restart services.

Many console scripts are documented here

@yo-mismo CARP (a "variant" of VRRP) packets are sent using MAC 00:00:5e:00:01:vhid (unicast) to 01:00:5e:00:01:vhid (multicast). ARP of CARP VIP should be 00:00:5e:00:01:vhid.

Can you post a Wireshark capture that shows that every packets are sent to the multicast 01:00:5e:00:01:vhid ?

It shouldn't be multicasting traffic other than the CARP heartbeats. If it does, there may be issues with ARP proxies or something something that would interfere with it.

I have uninstalled unused packages and added via shellcmd afterfilter /erc/rc.start_packages even if it should be launched without aid. Much better, haproxy+net-snmp+ntopng autostart, but freeradius is the only one left not starting automatically.

Is this "rule" still valid, that the VIP should be the lowest IP?

Additional: without NAT, how do I attach OpenVPN to the CARP IP, doesn't it also have to be mapped/rewritten to the CARP IP?

I try to set up a CARP cluster and have issues assigning fw rules etc, because I don't see the CARP IP in the Destination dropdown.

@gpfsenser I've also added a 'gateway group' to see if this helps - seems to be a requirement to the fix. Even if it works, this certainly falls into the 'super ugly' category. Appreciate the quick reply. Everyone is using their home internet more than they ever expected - a good opportunity I hope for pfsense to polish off some of the rough edges.

@mecjay12
This solution only works for the secondary, but you still not able to access the primary, when the secondary is the master and runs the OpenVPN server.

What I suggested works for both.