After upgrading to HAProxy 0.59_2 nothing works anymore!!!!

LAVenetz

@alexwitherspoon Perhaps, you don't believe it, but I have an extra test pfSense, exactly the same hardware. The problem is, althought I did a backup and the update log showed me successful

Installing pfSense-pkg-haproxy-devel...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
pfSense-pkg-haproxy-devel: 0.59_2 [pfSense]
haproxy-devel: 1.8.12 [pfSense]

Number of packages to be installed: 2

The process will require 2 MiB more space.
727 KiB to be downloaded.
[1/2] Fetching pfSense-pkg-haproxy-devel-0.59_2.txz: .......... done
[2/2] Fetching haproxy-devel-1.8.12.txz: .......... done
Checking integrity... done (2 conflicting)

pfSense-pkg-haproxy-devel-0.59_2 [pfSense] conflicts with pfSense-pkg-haproxy-0.59_2 [installed] on /usr/local/pkg/haproxy.xml
haproxy-devel-1.8.12 [pfSense] conflicts with haproxy-1.7.11 [installed] on /usr/local/man/man1/haproxy.1.gz
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 4 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
haproxy-1.7.11
pfSense-pkg-haproxy-0.59_2

New packages to be INSTALLED:
haproxy-devel: 1.8.12 [pfSense]
pfSense-pkg-haproxy-devel: 0.59_2 [pfSense]

Number of packages to be removed: 2
Number of packages to be installed: 2
[1/4] Deinstalling pfSense-pkg-haproxy-0.59_2...
Removing haproxy components...
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.
Syslog entries... done.
[1/4] Deleting files for pfSense-pkg-haproxy-0.59_2: .......... done
Removing haproxy components...
Syslog entries... done.
Configuration... done.
[2/4] Deinstalling haproxy-1.7.11...
[2/4] Deleting files for haproxy-1.7.11: ........ done
[3/4] Installing haproxy-devel-1.8.12...
[3/4] Extracting haproxy-devel-1.8.12: ........ done
[4/4] Installing pfSense-pkg-haproxy-devel-0.59_2...
[4/4] Extracting pfSense-pkg-haproxy-devel-0.59_2: .......... done
Saving updated package information...
done.
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.

Cleaning up cache... done.
Erfolgreich

I didn't see that the HAProxy showed me backend_server_ipvANY in red! So I executed the update on the productive pfSense as well. Interesting accidents!

By the way, I replaced on the test system the haproxy 0.59_ by haproxy-devel 0.59_2 but with no effect.

alexwitherspoon

@lavenetz yeah I didn't actually have any issues running the upgrade, that went fine. My HAProxy shows all green status pages , and no issues, except that SNI isn't working. Only the primary certificate is issued, no other certificates are issued despite being in the crt_list.

That makes this one tricky to detect, though I could have tested ALL urls for proper 200 status and valid certs.

willywonka

@alexwitherspoon Ok so i managed to revert to v0.54_2 successfully with again my ssl offloading (SNI) working, this is how:

On pfSense console i insert 8 followed by enter (to choose Console).
i type in pkg remove haproxy-0.59_2
i got asked, are you sure? Insert yes
then i type pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
after installation reboot pfSense and voila everything working again and package manager says: haProxy v0.54-2
Party!

alexwitherspoon

@willywonka I owe you a beer. That's magic, works here too!

[2.4.3-RELEASE][admin@edge.atwlab.com]/root: pkg remove pfSense-pkg-haproxy-0.59_2
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        pfSense-pkg-haproxy-0.59_2

Number of packages to be removed: 1

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling pfSense-pkg-haproxy-0.59_2...
Removing haproxy components...
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.
Syslog entries... done.
[1/1] Deleting files for pfSense-pkg-haproxy-0.59_2: 100%
Removing haproxy components...
Syslog entries... done.
Configuration... done.
[2.4.3-RELEASE][admin@edge.atwlab.com]/root: pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
Fetching pfSense-pkg-haproxy-0.54_2.txz: 100%   69 KiB  70.5kB/s    00:01    
Installing pfSense-pkg-haproxy-0.54_2...
Extracting pfSense-pkg-haproxy-0.54_2: 100%
Saving updated package information...
done.
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.
[2.4.3-RELEASE][admin@edge.atwlab.com]/root:

willywonka

@alexwitherspoon said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

@willywonka I owe you a beer. That's magic, works here too!

No magic here, just pure desperation

PiBa

As a workaround you can probably use the haproxy-devel package, it functions on the same configuration. and seems to work properly with SNI and offloading with multiple certificates..

For haproxy 'stable' ive send a preliminary 'quick fix'.., should be easy to apply the 2 changed lines manually for those who want need it 'now': https://github.com/pfsense/FreeBSD-ports/pull/542/files#diff-eb226b2eb58fc682fb444d554fb6bab8
That seems to fix the SNI behaviour.. but im not sure about the first report from @kdillen is actually a SNI issue.? Can you @kdillen try the patch?

Sorry for the trouble guys..

kdillen

@PiBa Correct in my case it is not SNI because I am using the ssl/https (TCP Mode ) . This is done because I needed the HTTP/2 support which was not yet in Haproxy at moment I first installed the Firewall.

If you want I can try the patch but that will be during the weekend. I actually was lucky to have also a backup for my standby firewall with the older Package version so I did a restore on that one also. Normally on Saturday morning 7:00 CET I make full image backup's of my firewalls so I can easy upgrade the standby node and apply the patch

Can you provide me with the instructions on how to do the patching ? Thanks in advance.

LAVenetz

@kdillen hi I've checked also haproxy 0.59_4 on my main pfSense (normally I don't do this), but it also did not work, same as 0.59_2. I did the same workaround like Micha (many thnks, see https://forum.netgate.com/user/nonick):

deinstallation of current version
pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/lua53-5.3.4_1.txz
pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/haproxy-1.7.10.txz
pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
check box Encrypt (SSL) in HAProxy Frontend(s) and ev. reboot
check HTTPS and/or HSTS with
6 a) https://www.ssllabs.com/ssltest/analyze.html?d=www.xxx.yy (should be at least a green A+)
6 b) https://www.sslshopper.com/ssl-checker.html#hostname=www.xxx.yy (all should be green)
Regards, LAV, sorry about my English!

netech

Try adding SNI Filter in front end config "*.company.com" matching the following certificate. That's how I got mine to work again.

Steve_B

The package maintainer pushed several updates last night. They should become available very soon after the next snapshot builds. Watch for the updated versions in System->Package manager->Installed packages, or on the dashboard packages widget.

PiBa

@kdillen
Can you check how the 'servers' are configured in the haproxy backend? I expect yours do want 'https' but dont need haproxy to do the encryption though do have the 'Encrypt(SSL)' checkbox checked while probably they shouldn't now?

For others.:
Well 0.59_4 should be available for the 'haproxy' package (haproxy-devel does not need that particular change/fix..) this should have SNI certificate selection for people who are using ssl-offloading with haproxy, and fixed the files tab..

Smoothrunnings

@lavenetz hi, am I doing this pkg add from shell or is there away to do it from the web GUI?

Thanks,

kdillen

@piba said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

ckbox checked while probably they shouldn't now?

@PiBa I did the work around with going back to a previous version. Today I did the upgrade to the latest version of pfsense (2.4.4) and guess what the issue is back. I worked around it by using some extra public IP's and NAT and going back to the pfSense Loadbalancer.

Afterwards I tried to recreate a new TCP based loadbalancer. I played with all the settings for front en backend but or I get nothing passing through it or I get everything encrypted and not readable for my reverse proxies. I am not using any ACL's or anything just TCP forwarding towards 2 backends.

Anybody else has this problem ? Because it should not cause these kind of problems I guess ?

Thanks in advance.

PiBa

@kdillen
Can you post your haproxy.cfg ? Tried enabling/disabling the ssl options on the backend server?

netech

Make sure there is nothing in "SSL Offloading - client certificates" in the Frontend. I noticed that the latest version had a change to the way this was handled.

kdillen

I found the solution myself and yes it is strange behavior, something that used to work but suddenly not work anymore.

First part is the SSL checkbox in the backends that solved 1 part of the issue in my case.
Second part: Health check method. In my case I put it to HTTP and that made that my hosts where not available ( failed health check ) This is something that used to work in TCP mode but now suddenly not anymore. So i put them on basic.

After fixing both of these all my TCP forwarding problems are gone. But still this should be a big notification in the upgrade notes because it really changes in a big way things that used to work to not working at all anymore.

PiBa

@kdillen
I doubt putting a note in some upgrade text would have helped your case.. And stating 'found the solution myself' like 4 days after having been pointed at the solution. Which really is configuring the 2 ssl checkboxes on the backend/server really doesn't justify shouting out 'nothing works'.. Lots of things do work, and yes there is a minor upgrade issue in some cases for users that use sni.. which is really simple to solve if you understand what haproxy is doing. Excuse my response.. I do my best to keep upgrades 'seamless' but i cannot validate every possible configuration for every situation before sending a PR on github.

So anyhow:
Enable http healthchecks again like they used to be and enable the "SSL checks" and it will verry likely work as it used to..

kdillen

@piba Well in that case how come that a configuration that worked perfectly with a haproxy 0.54.2 ( including older versions ) and with an upgrade to haproxy 0.59_2 or 0.59_4 it suddenly does not work anymore ?

But the haproxy below is the same version at least for 0.54_2 and 0.59_2 for 0.59_4 it is a new version of haproxy but still a minor upgrade.

And for your information the configuration was converted from haproxy 1.5 to haproxy 1.6 to haproxy 1.7.10 and never had issues with pfsense packages 0.54_2 or lower just after going to 1.7.10 or higher since pfSense haproxy package 0.59.x became available!!!!

So it means the configuration generated with the old package and new package is not compatible anymore in certain conditions... I am not that stupid to understand this. But at least it should have been noted that some parameters have other meanings and could create strange behavior after upgrade to the new packages.. And if you do not understand that. Then I wonder what would qualify for you as a requirement to make an upgrade note available. I have seen Note's for application upgrades which caused far less damage than this.

And 4 days pointing out the solution.. What solution the ssl checkboxes... Well they are not the solution.. They are just a part of the solution but also the cause, because that is the part that is not backwards compatible between 0.54_x and 0.59_x without any notice or remark!!! Secondly the real solution of the problem in my case was the "Health Check Method" chance. Something that used to work and suddenly not work anymore in TCP mode. I wonder what has changed there without any NOTIFICATION!!!!

And I did not even see your message with your solution until I wrote my last message.. What you think everybody is waiting constantly for their messages. I just did a nice workaround. Used the build in pfsense load balancer and added some of the destinations behind some free spare public NAT addresses directly. But I know then I not have sticky sessions. But that gave me the time to look into the problem and try to solve it before to update my last firewall.

PiBa

Okay i might have stepped out of bed with the wrong foot first, and would like to apologize for my response.

@kdillen said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

the real solution of the problem in my case was the "Health Check Method" chance

For that part changing the health-check from HTTP to TCP is not the 'correct' solution, though it would 'pass' the check, it might then also pass the check if the server is still listening but only producing http errors. The health check and its parameters are (or at least should) still be working as it used to. If it fails due to the upgrade of the package, then the only thing to change to fix that is to enable or disable the "SSL checks" checkbox.

Nick2253

@piba I believe you are saying this, but just to clarify for other who were confused like me:

The reason you need to change the health check, is because unchecking the "SSL" checkbox changes how HAProxy connects to the server. Without that checked, the HTTP check is only HTTP, and does not properly negotiate the SSL connection. The correct fix for this is to change the check method to "SSL".

So, if you are like me, and get a SSL_ERROR_RX_RECORD_TOO_LONG error after you upgrade, it's because of a change in how HAProxy handles intranet SSL encryption, so you need to do the following:

Go to Services -> HAProxy -> Backend
Select each HTTPS backend in turn, and change the following:
- In the server list, select the server, click the edit pencil, and UNcheck the "Encrypt(SSL)" box.
- Under the "Health checking" section, change the "Health check method" to "SSL".
- Scroll to the bottom, and click Save
Once you're done with all the servers, apply the configuration.
You can double check that the health checking is working by checking the "Stats" page.