HAproxy service cannot start - pfSense 2.4.4

phoadm

Hello!!

We operate two pfSense boxes under v2.4.4 update 3 in HA mode.

We have installed haproxy_devel package to the first node, we configured it and it runs with no issues.

We performed the same procedure on the second box in the HA cluster but.. alas, haproxy refused to start either manually or after a reboot

After removing, reinstalling etc. the haproxy-devel package, we verified that the problem lies elsewhere. By using https://forum.netgate.com/topic/115222/haproxy-will-hangs-when-i-upgrade-to-pfsense-2-3-4/ & as an inspiration, we created a haproxy_manual.sh and we placed it into /usr/local/etc/rc.d/ in order for pfSense to run it after boot as per https://docs.netgate.com/pfsense/en/latest/development/executing-commands-at-boot-time.html .
Alas, the haproxy service remained stopped! We also tried the shellcmd option, with the same effect (service still stopped). HOWEVER, if we run the file from the command line ( /usr/local/etc/rc.d/haproxy_manual.sh start) then the haproxy daemon starts and is fully operational.

Thus, we started to think that there is a permission issue somewhere, as we operate some HAproxy backends using the Client-IP directive, which requires elevated rights for the HAproxy daemon. By adding some umask parameters (as per https://forums.freebsd.org/threads/why-wont-my-new-style-rc-script-start-at-boot.56923/ ), voila! The HAProxy service was started using the “manual” script. Of course, if we stop the service through either the command-line or the Web GUI, then we must re-launch HaProxy manually and not through the controls.

We tried to enable all logging possible (to our knowledge) to hunt down the issue, we managed it up to a point however the root cause remains unsolved. Can anyone help us to locate this issue regarding the proper (not manually, as we devised) service launch? The other pfSense box has no such issues with exactly the same configuration!

The (modified) haproxy_manual.sh :

#!/bin/sh

# PROVIDE: haproxy
# REQUIRE: LOGIN
# KEYWORD: FreeBSD

. /etc/rc.subr

# rc_fast=yes workaround for pfSense that calls start when it
# means restart (for a wan-ip change for example..)
# this way it doesnt check for a already running process 
# and just fires of the start procedure again. which will
# take care to restart haproxy gracefully
rc_fast=yes

name="haproxy_manual"
rcvar="haproxy_enable"
haproxy_enable=${haproxy-"YES"}

load_rc_config haproxy

start_precmd="/usr/bin/umask 022"
start_cmd="haproxy_manual_start"
stop_postcmd="haproxy_manual_stop"
restart_cmd="haproxy_manual_restart"

haproxy_manual_start()
{
    echo "Starting HAProxy"
    /usr/bin/umask 022;/usr/local/sbin/haproxy -D -- /var/etc/haproxy/haproxy.cfg
}

haproxy_manual_check () {
        echo "Checking HAProxy."
        /usr/bin/env \
        PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
		/usr/local/bin/php-cgi -q -d auto_prepend_file=config.inc <<ENDOFF
<?php
			require_once("globals.inc");
			require_once("functions.inc");
			require_once("haproxy/haproxy.inc");
			haproxy_check_run(0);
?>
ENDOFF
}


haproxy_manual_stop () {
        echo "Stopping HAProxy."
        killall haproxy
}

haproxy_manual_restart () {
	echo "Restarting HAProxy."
	haproxy_manual_start
}


run_rc_command "$1"

The section that is included in the original haproxy.sh but was replaced by the script above (where the permission issue possibly exists) is:

haproxy_start () {
        echo "Starting haproxy."
        /usr/bin/env \
        PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
		/usr/local/bin/php-cgi -q -d auto_prepend_file=config.inc <<ENDOFF
<?php
			require_once("globals.inc");
			require_once("functions.inc");
			require_once("haproxy/haproxy.inc");
			haproxy_configure();
?>
ENDOFF
}

PiBa

@phoadm
Have you configured the haproxy webgui to 'monitor' a carp interface? If so it wont start on that node until that node becomes master.