SG-1100 : HAproxy fails after upgrade to 23.01
-
Upgraded an SG-1100 via shell, didn't come up again by itself. (remote) customer reapplied power, box started.
Looks OK, only HAproxy fails with this error:
Crash report begins. Anonymous machine information: arm64 14.0-CURRENT FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:28:37 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/aarch64/Z3hsU8Fs/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/Free Crash report details: PHP Errors: [04-Apr-2023 16:35:54 Europe/Vienna] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158 Stack trace: #0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying() #1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...') #2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1) #3 /etc/inc/pkg-utils.inc(781) : eval()'d code(1): haproxy_custom_php_install_command() #4 /etc/inc/pkg-utils.inc(781): eval() #5 /etc/inc/pkg-utils.inc(899): eval_once('haproxy_custom_...') #6 /etc/rc.packages(76): install_package_xml('haproxy') #7 {main} thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158 No FreeBSD crash data found.Tried to reinstall, to remove and then install again, no change.
HAproxy is only used to run that good old small lua-script for the ACME-challenge, by the way.
-
@sgw Install system patches package https://docs.netgate.com/pfsense/en/latest/development/system-patches.html and apply all the recommended patches.
Reboot.
Uninstall HAP
Reinstall
See if the errors clear.
-
@rcoleman-netgate hmm, ok. I might try that ... but ... why didn't I have to do that on multiple other SG-1100 appliances? thanks
-
I hit this issue on another SG-1100 again.
Looking through "System Patches" I don't find anything matching exactly.
Additionally I don't find a matching ticket in the pfsense-redmine ...
Applying ALL patches feels like overkill ... Is there a fix for this already that I missed somehow? thanks -
@sgw Not overkill -- covers all sorts of things you could come across later.
-
@rcoleman-netgate applied all recommended patches, tried to reinstall haproxy then, still the same error.
-
uninstalling and installing from scratch: same error
reboot: not yet possible during the day -
Even now after 23.05 haproxy fails.
Uninstalled, re-installed. Same error(s).EDIT: SG3100
-
@sgw said in SG-1100 : HAproxy fails after upgrade to 23.01:
didn't come up again by itself. (remote) customer reapplied power
Describe this...? If they didn't wait 10-15 minutes or more they probably booted it during the upgrade. If that's the case, interesting it was apparently working after and upgraded to 23.05, but if you can't find someone else to duplicate this then a next step might be install a clean copy of pfSense.
-
Unfortunately I can't remember exactly what happened there. In fact I can't remember exactly which of the appliances had that issue first.
Right now I see it on two appliances: a SG1100 and a SG3100.
From the 1100:
Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158 Stack trace: #0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying() #1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...') #2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1) #3 /etc/inc/pkg-utils.inc(781) : eval()'d code(1): haproxy_custom_php_install_command() #4 /etc/inc/pkg-utils.inc(781): eval() #5 /etc/inc/pkg-utils.inc(899): eval_once('haproxy_custom_...') #6 /etc/rc.packages(76): install_package_xml('haproxy') #7 {main} thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158 PHP ERROR: Type: 1, File: s string on string in /usr/local/pkg/haproxy/haproxy.inc:2158 Stack trace: #0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying() #1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...') #2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1) #3 /etc/inc/pkg-utils.inc(781) : eval()'d code(1): haproxy_custom_php_install_command() #4 /etc/inc/pkg-utils.inc(781): eval() #5 /etc/inc/pkg-utils.inc(899): eval_once('haproxy_custom_...') #6 /etc/rc.packages(76): install_package_xml('haproxy') #7 {main} thrownpkg-static: POST-INSTALL script failed >>> Cleaning up cache... done. SuccessStatus: I pinned it to 23.01 to not install packages from 23.05 before the release upgrade. Tried to reinstall pfSense-pkg-haproxy-0.61_10. haproxy gets installed somehow, but doesn't show up in the GUI. And it doesn't run according to "ps".
-
Still seeing this issue. As far as I see the patches in redmine don't cover this, right?
How can I proceed? @jimp may I ask you to take a look, pls? thanks in advance. -
The "fun": I have one SG3100 without this issue and one WITH this issue.
Executing custom_php_install_command()... Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158 Stack trace: #0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying() #1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...') #2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1) #3 /etc/inc/pkg-utils.inc(783) : eval()'d code(1): haproxy_custom_php_install_command() #4 /etc/inc/pkg-utils.inc(783): eval() #5 /etc/inc/pkg-utils.inc(901): eval_once('haproxy_custom_...') #6 /etc/rc.packages(76): install_package_xml('haproxy') #7 {main} thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158 PHP ERROR: Type: 1, File: /usr/local/pkg/haproxy/haproxy.inc, Line: 2158, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158 Stack trace: #0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying() #1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...') #2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1) #3 /etc/inc/pkg-utils.inc(783) : eval()'d code(1): haproxy_custom_php_install_command() #4 /etc/inc/pkg-utils.inc(783): eval() #5 /etc/inc/pkg-utils.inc(901): eval_once('haproxy_custom_...') #6 /etc/rc.packages(76): install_package_xml('haproxy') #7 {main} thrownpkg-static: POST-INSTALL script failed >>> Cleaning up cache... done. SuccessThe function in the error:
function use_transparent_clientip_proxying() { global $config; $a_backends = getarraybyref($config,'installedpackages','haproxy','ha_pools','item'); if (is_array($a_backends)) { foreach ($a_backends as $backend) { if ($backend["transparent_clientip"] == 'yes') { #### the line from the error message: line 2158 return true; } } } return false; } -
contacted support, reinstalled from scratch, same errors again.
provided diagnosis data, support reproduced issue, now we have this issue in redmine: bug