Hi bahsig,

my problem was that I tried to start a 3rd party script (ElasticSearch Beats binary in my case) via the shellcmd package.
As Beats is not a service by default it ran as a program and didn't provide an exit code to shellcmd.
So in the end shellcmd waited to infinity for Beats exit code.
Due to the daisy chaining of shellcmd in the PfSense / FreeBSD boot process it blocked the machine from booting. ;-)

Once I killed the binary from the console / SSH booting finished and PfSense worked as expected with syncing, etc.

Sadly there is no alert or system stat that shows you the FWs "boot state". ;-(

Hope that helps.

If you're not NATing to a CARP IP, your sessions will be lost on failover, which is why he asked.

Other likely cause, if the server is pointing to the primary's IP rather than a CARP IP for its gateway.

i already found the problem..
it is due to i used windows text editor to make the script (CR LF)
after saving the script file using UNIX (LF only) , the script successfully executed..

thx

The VIP MAC is used if VIP is used.
Have you set your outbound NAT to translate to VIP?

Because CARP VIPs are static and those are the addresses that "swing" over to the secondary in the event of a failure. This means that Layer 3 stays intact for states, routes, client gateways, DNS servers, etc.

@jnevestdl:

Hi vocatus,

Congrats for the complicated setup.

It is possible you to make us a tutorial with step by step or have screenshots of what is needed to configure this? Don't forget to hide the public IP's.

Thanks.

Hi jnevestdl,

It's been quite a while since I designed this and I'm at a different position now, so I don't have access to the GUI to take screenshots. I can try to answer questions for you though if you have any.

This appears fixed in 2.3.1-RELEASE-p5.  (I can't spot anything that could be related in the release notes.)

That explains it. CARP works on every interface where a CARP VIP exists, not the sync interface. They'll switch over completely where the secondary knows it needs to take over and the primary sees that fact, but they have to be able to communicate on all the VLANs and interfaces for that to function.

That sounds like the interface stayed up but would not pass traffic any more, either due to something on either interface (primary or secondary) or something in layer 1 (bad pair out but not in maybe) or layer 2. It is not possible for HA to know what to do in that case. Disable CARP on the malfunctioning master or unplug the failed interface / shutdown the switch port and HA will shutdown CARP on all interfaces and swing to the backup.

The answer is more redundancy like LAGG interfaces to stacked switches so traffic will continue to pass in a carrier-up-but-no-traffic-passing situation on one interface.

This image is what I get when I change the VLAN on one interface's switch port so carrier stays up but traffic (including CARP) no longer passes between nodes.


Simply ended up using VLANs for this situation.  Previously had been told that the switch did not support VLANs, found out otherwise.

Further… configured VLANs on pfSense under: Interfaces > Assign > VLANs and Interface Assignments.

Hello,
Have you checked that the OpenVPN Interface is the VIP and not the physical Interface?

Sure, but what if the primary (now a backup) is not reachable through any interfaces, and you must make changes to the firewall (secondary, now acting as master) right away? What do you do? Write down every config change, then execute them in the primary as soon as it comes back online?

There might be an even worse scenario: what if I don't even realize that I'm making changes to the secondary? Don't people usually complain about this potential issue? I'm not sure whether this is not very common, or if there's an alternative which I'm not aware of.

Please, don't get me wrong, I don't mean to offend (plus, english is not my native language), that's just out of curiosity, but I was told that Cisco ASA works as I thought pfSense should: whoever's the master, becomes the config replication source. Is it really that complex to implement such a feature?

Maybe I'm just looking at things from the wrong point of view, but I'm afraid people will frown at this if config replication might become an issue. So I was looking for a solution

Thanks for your time and patience anyway.

Hi,

I don't think so! Because my system already have a default from 1631000. My server has 16GB RAM.

Today active states in my system is 400000

May be I have to change something on /usr/local/etc/php.ini

However, I have no idea if this will work.

Im lost!!

Thanks a lot

Cesar

It looks like (as expected) carp: interface down is an actual carrier loss on the interface. Here are two CARP events on a backup node. The first is a unplug/plug of the patch cable. The second is removing the interface from the same VLAN as the master node, which leaves the interface up but stops CARP advertisement reception.

Jun 16 07:08:03 pfSenseB kernel: carp: demoted by 240 to 240 (interface down)
Jun 16 07:08:17 pfSenseB kernel: carp: VHID 66@igb1: INIT -> BACKUP
Jun 16 07:08:17 pfSenseB kernel: carp: demoted by -240 to 0 (interface up)

Jun 16 07:13:02 pfSenseB kernel: carp: VHID 66@igb1: BACKUP -> MASTER (master down)
Jun 16 07:13:36 pfSenseB kernel: carp: VHID 66@igb1: MASTER -> BACKUP (more frequent advertisement received)

What is your switch logging on the switchport in question? I'd take a good look at your switching and cabling.