After upgrading to HAProxy 0.59_2 nothing works anymore!!!!

willywonka

@lavenetz Nope, i do not know how to get back online ... i might need to review my upgrade policy , because this update kicked me in the lulz hard.

LAVenetz

@willywonka thanks, anyway the contributor of the package is responsible to bring a new version with a patch. But I cannot wait until next year, by the way. I have an old pfSense with 2.3.5-RELEASE-p2 (i386), an there is the version 0.54_2 installed and running,but it's 32-bit version! What do you think?
Regards. LAV

willywonka

@lavenetz this is the 0.54_2 package. But howto restore this version, i do not know at this moment .....

alexwitherspoon

@willywonka yeah - downgrades for packages have never been supported, so I'd be cautious. - I get it though, I am also impacted, and wished I had looked extra hard at the changelog before hitting the button. Most of these upgrades are soo smooth.

LAVenetz

@alexwitherspoon Perhaps, you don't believe it, but I have an extra test pfSense, exactly the same hardware. The problem is, althought I did a backup and the update log showed me successful

Installing pfSense-pkg-haproxy-devel...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
pfSense-pkg-haproxy-devel: 0.59_2 [pfSense]
haproxy-devel: 1.8.12 [pfSense]

Number of packages to be installed: 2

The process will require 2 MiB more space.
727 KiB to be downloaded.
[1/2] Fetching pfSense-pkg-haproxy-devel-0.59_2.txz: .......... done
[2/2] Fetching haproxy-devel-1.8.12.txz: .......... done
Checking integrity... done (2 conflicting)

pfSense-pkg-haproxy-devel-0.59_2 [pfSense] conflicts with pfSense-pkg-haproxy-0.59_2 [installed] on /usr/local/pkg/haproxy.xml
haproxy-devel-1.8.12 [pfSense] conflicts with haproxy-1.7.11 [installed] on /usr/local/man/man1/haproxy.1.gz
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 4 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
haproxy-1.7.11
pfSense-pkg-haproxy-0.59_2

New packages to be INSTALLED:
haproxy-devel: 1.8.12 [pfSense]
pfSense-pkg-haproxy-devel: 0.59_2 [pfSense]

Number of packages to be removed: 2
Number of packages to be installed: 2
[1/4] Deinstalling pfSense-pkg-haproxy-0.59_2...
Removing haproxy components...
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.
Syslog entries... done.
[1/4] Deleting files for pfSense-pkg-haproxy-0.59_2: .......... done
Removing haproxy components...
Syslog entries... done.
Configuration... done.
[2/4] Deinstalling haproxy-1.7.11...
[2/4] Deleting files for haproxy-1.7.11: ........ done
[3/4] Installing haproxy-devel-1.8.12...
[3/4] Extracting haproxy-devel-1.8.12: ........ done
[4/4] Installing pfSense-pkg-haproxy-devel-0.59_2...
[4/4] Extracting pfSense-pkg-haproxy-devel-0.59_2: .......... done
Saving updated package information...
done.
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.

Cleaning up cache... done.
Erfolgreich

I didn't see that the HAProxy showed me backend_server_ipvANY in red! So I executed the update on the productive pfSense as well. Interesting accidents!

By the way, I replaced on the test system the haproxy 0.59_ by haproxy-devel 0.59_2 but with no effect.

alexwitherspoon

@lavenetz yeah I didn't actually have any issues running the upgrade, that went fine. My HAProxy shows all green status pages , and no issues, except that SNI isn't working. Only the primary certificate is issued, no other certificates are issued despite being in the crt_list.

That makes this one tricky to detect, though I could have tested ALL urls for proper 200 status and valid certs.

willywonka

@alexwitherspoon Ok so i managed to revert to v0.54_2 successfully with again my ssl offloading (SNI) working, this is how:

On pfSense console i insert 8 followed by enter (to choose Console).
i type in pkg remove haproxy-0.59_2
i got asked, are you sure? Insert yes
then i type pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
after installation reboot pfSense and voila everything working again and package manager says: haProxy v0.54-2
Party!

alexwitherspoon

@willywonka I owe you a beer. That's magic, works here too!

[2.4.3-RELEASE][admin@edge.atwlab.com]/root: pkg remove pfSense-pkg-haproxy-0.59_2
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        pfSense-pkg-haproxy-0.59_2

Number of packages to be removed: 1

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling pfSense-pkg-haproxy-0.59_2...
Removing haproxy components...
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.
Syslog entries... done.
[1/1] Deleting files for pfSense-pkg-haproxy-0.59_2: 100%
Removing haproxy components...
Syslog entries... done.
Configuration... done.
[2.4.3-RELEASE][admin@edge.atwlab.com]/root: pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
Fetching pfSense-pkg-haproxy-0.54_2.txz: 100%   69 KiB  70.5kB/s    00:01    
Installing pfSense-pkg-haproxy-0.54_2...
Extracting pfSense-pkg-haproxy-0.54_2: 100%
Saving updated package information...
done.
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.
[2.4.3-RELEASE][admin@edge.atwlab.com]/root:

willywonka

@alexwitherspoon said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

@willywonka I owe you a beer. That's magic, works here too!

No magic here, just pure desperation

PiBa

As a workaround you can probably use the haproxy-devel package, it functions on the same configuration. and seems to work properly with SNI and offloading with multiple certificates..

For haproxy 'stable' ive send a preliminary 'quick fix'.., should be easy to apply the 2 changed lines manually for those who want need it 'now': https://github.com/pfsense/FreeBSD-ports/pull/542/files#diff-eb226b2eb58fc682fb444d554fb6bab8
That seems to fix the SNI behaviour.. but im not sure about the first report from @kdillen is actually a SNI issue.? Can you @kdillen try the patch?

Sorry for the trouble guys..

kdillen

@PiBa Correct in my case it is not SNI because I am using the ssl/https (TCP Mode ) . This is done because I needed the HTTP/2 support which was not yet in Haproxy at moment I first installed the Firewall.

If you want I can try the patch but that will be during the weekend. I actually was lucky to have also a backup for my standby firewall with the older Package version so I did a restore on that one also. Normally on Saturday morning 7:00 CET I make full image backup's of my firewalls so I can easy upgrade the standby node and apply the patch

Can you provide me with the instructions on how to do the patching ? Thanks in advance.

LAVenetz

@kdillen hi I've checked also haproxy 0.59_4 on my main pfSense (normally I don't do this), but it also did not work, same as 0.59_2. I did the same workaround like Micha (many thnks, see https://forum.netgate.com/user/nonick):

deinstallation of current version
pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/lua53-5.3.4_1.txz
pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/haproxy-1.7.10.txz
pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
check box Encrypt (SSL) in HAProxy Frontend(s) and ev. reboot
check HTTPS and/or HSTS with
6 a) https://www.ssllabs.com/ssltest/analyze.html?d=www.xxx.yy (should be at least a green A+)
6 b) https://www.sslshopper.com/ssl-checker.html#hostname=www.xxx.yy (all should be green)
Regards, LAV, sorry about my English!

netech

Try adding SNI Filter in front end config "*.company.com" matching the following certificate. That's how I got mine to work again.

Steve_B

The package maintainer pushed several updates last night. They should become available very soon after the next snapshot builds. Watch for the updated versions in System->Package manager->Installed packages, or on the dashboard packages widget.

PiBa

@kdillen
Can you check how the 'servers' are configured in the haproxy backend? I expect yours do want 'https' but dont need haproxy to do the encryption though do have the 'Encrypt(SSL)' checkbox checked while probably they shouldn't now?

For others.:
Well 0.59_4 should be available for the 'haproxy' package (haproxy-devel does not need that particular change/fix..) this should have SNI certificate selection for people who are using ssl-offloading with haproxy, and fixed the files tab..

Smoothrunnings

@lavenetz hi, am I doing this pkg add from shell or is there away to do it from the web GUI?

Thanks,

kdillen

@piba said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

ckbox checked while probably they shouldn't now?

@PiBa I did the work around with going back to a previous version. Today I did the upgrade to the latest version of pfsense (2.4.4) and guess what the issue is back. I worked around it by using some extra public IP's and NAT and going back to the pfSense Loadbalancer.

Afterwards I tried to recreate a new TCP based loadbalancer. I played with all the settings for front en backend but or I get nothing passing through it or I get everything encrypted and not readable for my reverse proxies. I am not using any ACL's or anything just TCP forwarding towards 2 backends.

Anybody else has this problem ? Because it should not cause these kind of problems I guess ?

Thanks in advance.

PiBa

@kdillen
Can you post your haproxy.cfg ? Tried enabling/disabling the ssl options on the backend server?

netech

Make sure there is nothing in "SSL Offloading - client certificates" in the Frontend. I noticed that the latest version had a change to the way this was handled.

kdillen

I found the solution myself and yes it is strange behavior, something that used to work but suddenly not work anymore.

First part is the SSL checkbox in the backends that solved 1 part of the issue in my case.
Second part: Health check method. In my case I put it to HTTP and that made that my hosts where not available ( failed health check ) This is something that used to work in TCP mode but now suddenly not anymore. So i put them on basic.

After fixing both of these all my TCP forwarding problems are gone. But still this should be a big notification in the upgrade notes because it really changes in a big way things that used to work to not working at all anymore.