SG-1100 : HAproxy fails after upgrade to 23.01

sgw

Upgraded an SG-1100 via shell, didn't come up again by itself. (remote) customer reapplied power, box started.

Looks OK, only HAproxy fails with this error:

Crash report begins.  Anonymous machine information:

arm64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:28:37 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/aarch64/Z3hsU8Fs/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/Free

Crash report details:

PHP Errors:
[04-Apr-2023 16:35:54 Europe/Vienna] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(781) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(781): eval()
#5 /etc/inc/pkg-utils.inc(899): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158



No FreeBSD crash data found.

Tried to reinstall, to remove and then install again, no change.

HAproxy is only used to run that good old small lua-script for the ACME-challenge, by the way.

rcoleman-netgate

@sgw Install system patches package https://docs.netgate.com/pfsense/en/latest/development/system-patches.html and apply all the recommended patches.

Reboot.

Uninstall HAP

Reinstall

See if the errors clear.

sgw

@rcoleman-netgate hmm, ok. I might try that ... but ... why didn't I have to do that on multiple other SG-1100 appliances? thanks

sgw

I hit this issue on another SG-1100 again.
Looking through "System Patches" I don't find anything matching exactly.
Additionally I don't find a matching ticket in the pfsense-redmine ...
Applying ALL patches feels like overkill ... Is there a fix for this already that I missed somehow? thanks

rcoleman-netgate

@sgw Not overkill -- covers all sorts of things you could come across later.

sgw

@rcoleman-netgate applied all recommended patches, tried to reinstall haproxy then, still the same error.

sgw

uninstalling and installing from scratch: same error
reboot: not yet possible during the day

sgw

Even now after 23.05 haproxy fails.
Uninstalled, re-installed. Same error(s).

EDIT: SG3100

SteveITS

@sgw said in SG-1100 : HAproxy fails after upgrade to 23.01:

didn't come up again by itself. (remote) customer reapplied power

Describe this...? If they didn't wait 10-15 minutes or more they probably booted it during the upgrade. If that's the case, interesting it was apparently working after and upgraded to 23.05, but if you can't find someone else to duplicate this then a next step might be install a clean copy of pfSense.

sgw

Unfortunately I can't remember exactly what happened there. In fact I can't remember exactly which of the appliances had that issue first.

Right now I see it on two appliances: a SG1100 and a SG3100.

From the 1100:

Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(781) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(781): eval()
#5 /etc/inc/pkg-utils.inc(899): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158
PHP ERROR: Type: 1, File: s string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(781) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(781): eval()
#5 /etc/inc/pkg-utils.inc(899): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrownpkg-static: POST-INSTALL script failed
>>> Cleaning up cache... done.
Success

Status: I pinned it to 23.01 to not install packages from 23.05 before the release upgrade. Tried to reinstall pfSense-pkg-haproxy-0.61_10. haproxy gets installed somehow, but doesn't show up in the GUI. And it doesn't run according to "ps".

sgw

Still seeing this issue. As far as I see the patches in redmine don't cover this, right?
How can I proceed? @jimp may I ask you to take a look, pls? thanks in advance.

sgw

The "fun": I have one SG3100 without this issue and one WITH this issue.

Executing custom_php_install_command()...
Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(783) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(783): eval()
#5 /etc/inc/pkg-utils.inc(901): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158
PHP ERROR: Type: 1, File: /usr/local/pkg/haproxy/haproxy.inc, Line: 2158, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(783) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(783): eval()
#5 /etc/inc/pkg-utils.inc(901): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrownpkg-static: POST-INSTALL script failed
>>> Cleaning up cache... done.
Success

The function in the error:

function use_transparent_clientip_proxying() {
        global $config;
        $a_backends = getarraybyref($config,'installedpackages','haproxy','ha_pools','item');
        if (is_array($a_backends)) {
                foreach ($a_backends as $backend) {
                        if ($backend["transparent_clientip"] == 'yes') {   #### the line from the error message: line 2158
                                return true;
                        }
                }
        }
        return false;
}

sgw

contacted support, reinstalled from scratch, same errors again.
provided diagnosis data, support reproduced issue, now we have this issue in redmine: bug