SG-1100 : HAproxy fails after upgrade to 23.01

sgw

@rcoleman-netgate hmm, ok. I might try that ... but ... why didn't I have to do that on multiple other SG-1100 appliances? thanks

sgw

I hit this issue on another SG-1100 again.
Looking through "System Patches" I don't find anything matching exactly.
Additionally I don't find a matching ticket in the pfsense-redmine ...
Applying ALL patches feels like overkill ... Is there a fix for this already that I missed somehow? thanks

rcoleman-netgate

@sgw Not overkill -- covers all sorts of things you could come across later.

sgw

@rcoleman-netgate applied all recommended patches, tried to reinstall haproxy then, still the same error.

sgw

uninstalling and installing from scratch: same error
reboot: not yet possible during the day

sgw

Even now after 23.05 haproxy fails.
Uninstalled, re-installed. Same error(s).

EDIT: SG3100

SteveITS

@sgw said in SG-1100 : HAproxy fails after upgrade to 23.01:

didn't come up again by itself. (remote) customer reapplied power

Describe this...? If they didn't wait 10-15 minutes or more they probably booted it during the upgrade. If that's the case, interesting it was apparently working after and upgraded to 23.05, but if you can't find someone else to duplicate this then a next step might be install a clean copy of pfSense.

sgw

Unfortunately I can't remember exactly what happened there. In fact I can't remember exactly which of the appliances had that issue first.

Right now I see it on two appliances: a SG1100 and a SG3100.

From the 1100:

Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(781) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(781): eval()
#5 /etc/inc/pkg-utils.inc(899): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158
PHP ERROR: Type: 1, File: s string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(781) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(781): eval()
#5 /etc/inc/pkg-utils.inc(899): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrownpkg-static: POST-INSTALL script failed
>>> Cleaning up cache... done.
Success

Status: I pinned it to 23.01 to not install packages from 23.05 before the release upgrade. Tried to reinstall pfSense-pkg-haproxy-0.61_10. haproxy gets installed somehow, but doesn't show up in the GUI. And it doesn't run according to "ps".

sgw

Still seeing this issue. As far as I see the patches in redmine don't cover this, right?
How can I proceed? @jimp may I ask you to take a look, pls? thanks in advance.

sgw

The "fun": I have one SG3100 without this issue and one WITH this issue.

Executing custom_php_install_command()...
Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(783) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(783): eval()
#5 /etc/inc/pkg-utils.inc(901): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrown in /usr/local/pkg/haproxy/haproxy.inc on line 2158
PHP ERROR: Type: 1, File: /usr/local/pkg/haproxy/haproxy.inc, Line: 2158, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/haproxy/haproxy.inc:2158
Stack trace:
#0 /usr/local/pkg/haproxy/haproxy.inc(1490): use_transparent_clientip_proxying()
#1 /usr/local/pkg/haproxy/haproxy.inc(2353): haproxy_writeconf('/var/etc/haprox...')
#2 /usr/local/pkg/haproxy/haproxy.inc(653): haproxy_check_run(1)
#3 /etc/inc/pkg-utils.inc(783) : eval()'d code(1): haproxy_custom_php_install_command()
#4 /etc/inc/pkg-utils.inc(783): eval()
#5 /etc/inc/pkg-utils.inc(901): eval_once('haproxy_custom_...')
#6 /etc/rc.packages(76): install_package_xml('haproxy')
#7 {main}
  thrownpkg-static: POST-INSTALL script failed
>>> Cleaning up cache... done.
Success

The function in the error:

function use_transparent_clientip_proxying() {
        global $config;
        $a_backends = getarraybyref($config,'installedpackages','haproxy','ha_pools','item');
        if (is_array($a_backends)) {
                foreach ($a_backends as $backend) {
                        if ($backend["transparent_clientip"] == 'yes') {   #### the line from the error message: line 2158
                                return true;
                        }
                }
        }
        return false;
}

sgw

contacted support, reinstalled from scratch, same errors again.
provided diagnosis data, support reproduced issue, now we have this issue in redmine: bug