HAProxy causes failure to operate properly after 2.4.4 upgrade

wolrah

I just upgraded from 2.4.3_1 to 2.4.4 on my SG-2440 and the HAProxy package caused the device to end up in a technically booted but effectively useless state.

There were a few errors displayed while the package tried to update itself as part of the install. Unfortunately I was not logging the serial console but I did have it up so I was able to see the final state.

>>> Removing unnecessary packages... done.
>>> Cleanup pkg cache... done.
 Starting package pfBlockerNG...done.
 Starting package iperf...done.
 Starting package LADVD...done.
 Starting package OpenVPN Client Export Utility...done.
 Starting package mtr-nox11...done.
 Starting package acme...done.
 Starting package Netgate Coreboot Upgrade...done.
 Starting package suricata...done.
 Starting package FRR...done.
 Starting package haproxy-devel...
Fatal error: Cannot redeclare getarraybyref() (previously declared in /etc/inc/pfsense-utils.inc:3428) in /usr/local/pkg/haproxy/haproxy_utils.inc on line 105
PHP ERROR: Type: 1, File: /usr/local/pkg/haproxy/haproxy_utils.inc, Line: 105, Message: Cannot redeclare getarraybyref() (previously declared in /etc/inc/pfsense-utils.inc:3428)pfSense 2.4.4-RELEASE (Patch 1) amd64 Thu Sep 20 09:03:12 EDT 2018
Bootup complete

FreeBSD/amd64 (redacted.dns.name.here) (ttyu1)

login:

I do not have the password enabled on the serial console, nor was the device actually passing traffic or even responding to ping at this point.

I was able to log in with my normal credentials and of course first tried rebooting the machine, but that just left me in the same spot. The same PHP error also appeared during the "Starting firewall" stage which I presume is why the firewall seemed to be left in its initial startup lockdown state.

I don't actually use HAProxy in production and was just experimenting with it, so I decided to try removing the package with a "pkg remove haproxy" before giving in and reinstalling the system. After a reboot this did in fact resolve the issue and I'm back online through the device as I type this.

pfSense did report a crash log, but it contained just the same PHP error a few times over. It's short so I've pasted it here anyways, but I don't expect it really adds anything more to the equation.

[28-Sep-2018 09:52:28 America/Detroit] PHP Fatal error:  Cannot redeclare getarraybyref() (previously declared in /etc/inc/pfsense-utils.inc:3428) in /usr/local/pkg/haproxy/haproxy_utils.inc on line 105
[28-Sep-2018 09:55:36 America/Detroit] PHP Fatal error:  Cannot redeclare getarraybyref() (previously declared in /etc/inc/pfsense-utils.inc:3428) in /usr/local/pkg/haproxy/haproxy_utils.inc on line 105
[28-Sep-2018 09:55:41 America/Detroit] PHP Fatal error:  Cannot redeclare getarraybyref() (previously declared in /etc/inc/pfsense-utils.inc:3428) in /usr/local/pkg/haproxy/haproxy_utils.inc on line 105
[28-Sep-2018 09:55:49 America/Detroit] PHP Fatal error:  Cannot redeclare getarraybyref() (previously declared in /etc/inc/pfsense-utils.inc:3428) in /usr/local/pkg/haproxy/haproxy_utils.inc on line 105

This seems to be basically the exact opposite problem of what happened to user prewest a few days ago: https://forum.netgate.com/topic/135892/update-package-ha-proxy-fails-0-59-11

For that user the function was undeclared, for me it was declared twice.

Since the location of that function was changed with this commit: https://github.com/pfsense/FreeBSD-ports/commit/8e7865144b54262d07eefc31703cd11be217ecd1#diff-c6419a82c000bf48c3e7c12345353db6 it seems to me that there's a sort of race condition here where the package update needs to happen at the same time as the system. If the package post-that-commit is installed on 2.4.3 you get what happened to prewest, if the package has not been updated before installing 2.4.4 you get my issue.

Considering this can leave a system useless without manual local console interaction, this is probably worth warning anyone using HAProxy who's looking to upgrade. Make sure you update the package first.