After upgrading to HAProxy 0.59_2 nothing works anymore!!!!
-
Awesome, good thing I held off, sadly, I was waiting for 1.7.11 to resolve https://redmine.pfsense.org/issues/8580
-
I can confirm, I also got affected by upgrading, SNI is not working - only the main cert is being issued. No way to downgrade the physical appliance right now. I am dead in the water.
I was able to replicate at work + at home. Here is a home config set up similarly to production config with same structure/approach.
https://gist.github.com/alexwitherspoon/7bfe371ae532e791231caacc03a8ffee
-
Found this in the changelog - https://redmine.pfsense.org/issues/8670
Devel branch was copied over, so that's why the massive upgrade got pushed out.
-
@alexwitherspoon said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:
Found this in the changelog - https://redmine.pfsense.org/issues/8670
Devel branch was copied over, so that's why the massive upgrade got pushed out.Really nice and all but how do we get this issue registered in the first place? Right now it seems to stall as a forum post, i tried to login with my account to the issue tracker but no luck. What is the next step?
-
@willywonka
Hi, I have the same issue, 6 sites offline! Backup file gives me a tag error. Do you know how to go back to haproxy 0.54_2?
Regards, LAV -
@lavenetz Nope, i do not know how to get back online ... i might need to review my upgrade policy
, because this update kicked me in the lulz hard. -
@willywonka thanks, anyway the contributor of the package is responsible to bring a new version with a patch. But I cannot wait until next year, by the way. I have an old pfSense with 2.3.5-RELEASE-p2 (i386), an there is the version 0.54_2 installed and running,but it's 32-bit version! What do you think?
Regards. LAV -
-
@willywonka yeah - downgrades for packages have never been supported, so I'd be cautious. - I get it though, I am also impacted, and wished I had looked extra hard at the changelog before hitting the button. Most of these upgrades are soo smooth.
-
@alexwitherspoon Perhaps, you don't believe it, but I have an extra test pfSense, exactly the same hardware. The problem is, althought I did a backup and the update log showed me successful
Installing pfSense-pkg-haproxy-devel...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 2 package(s) will be affected (of 0 checked):New packages to be INSTALLED:
pfSense-pkg-haproxy-devel: 0.59_2 [pfSense]
haproxy-devel: 1.8.12 [pfSense]Number of packages to be installed: 2
The process will require 2 MiB more space.
727 KiB to be downloaded.
[1/2] Fetching pfSense-pkg-haproxy-devel-0.59_2.txz: .......... done
[2/2] Fetching haproxy-devel-1.8.12.txz: .......... done
Checking integrity... done (2 conflicting)- pfSense-pkg-haproxy-devel-0.59_2 [pfSense] conflicts with pfSense-pkg-haproxy-0.59_2 [installed] on /usr/local/pkg/haproxy.xml
- haproxy-devel-1.8.12 [pfSense] conflicts with haproxy-1.7.11 [installed] on /usr/local/man/man1/haproxy.1.gz
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 4 package(s) will be affected (of 0 checked):
Installed packages to be REMOVED:
haproxy-1.7.11
pfSense-pkg-haproxy-0.59_2New packages to be INSTALLED:
haproxy-devel: 1.8.12 [pfSense]
pfSense-pkg-haproxy-devel: 0.59_2 [pfSense]Number of packages to be removed: 2
Number of packages to be installed: 2
[1/4] Deinstalling pfSense-pkg-haproxy-0.59_2...
Removing haproxy components...
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.
Syslog entries... done.
[1/4] Deleting files for pfSense-pkg-haproxy-0.59_2: .......... done
Removing haproxy components...
Syslog entries... done.
Configuration... done.
[2/4] Deinstalling haproxy-1.7.11...
[2/4] Deleting files for haproxy-1.7.11: ........ done
[3/4] Installing haproxy-devel-1.8.12...
[3/4] Extracting haproxy-devel-1.8.12: ........ done
[4/4] Installing pfSense-pkg-haproxy-devel-0.59_2...
[4/4] Extracting pfSense-pkg-haproxy-devel-0.59_2: .......... done
Saving updated package information...
done.
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.Cleaning up cache... done.
ErfolgreichI didn't see that the HAProxy showed me backend_server_ipvANY in red! So I executed the update on the productive pfSense as well. Interesting accidents!
By the way, I replaced on the test system the haproxy 0.59_ by haproxy-devel 0.59_2 but with no effect.
-
@lavenetz yeah I didn't actually have any issues running the upgrade, that went fine. My HAProxy shows all green status pages , and no issues, except that SNI isn't working. Only the primary certificate is issued, no other certificates are issued despite being in the crt_list.
That makes this one tricky to detect, though I could have tested ALL urls for proper 200 status and valid certs.
-
@alexwitherspoon Ok so i managed to revert to v0.54_2 successfully with again my ssl offloading (SNI) working, this is how:
- On pfSense console i insert 8 followed by enter (to choose Console).
- i type in pkg remove haproxy-0.59_2
- i got asked, are you sure? Insert yes
- then i type pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
- after installation reboot pfSense and voila everything working again and package manager says: haProxy v0.54-2
- Party!
-
@willywonka I owe you a beer. That's magic, works here too!
[2.4.3-RELEASE][admin@edge.atwlab.com]/root: pkg remove pfSense-pkg-haproxy-0.59_2 Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 1 packages (of 0 packages in the universe): Installed packages to be REMOVED: pfSense-pkg-haproxy-0.59_2 Number of packages to be removed: 1 Proceed with deinstalling packages? [y/N]: y [1/1] Deinstalling pfSense-pkg-haproxy-0.59_2... Removing haproxy components... Menu items... done. Services... done. Loading package instructions... Deinstall commands... done. Syslog entries... done. [1/1] Deleting files for pfSense-pkg-haproxy-0.59_2: 100% Removing haproxy components... Syslog entries... done. Configuration... done. [2.4.3-RELEASE][admin@edge.atwlab.com]/root: pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz Fetching pfSense-pkg-haproxy-0.54_2.txz: 100% 69 KiB 70.5kB/s 00:01 Installing pfSense-pkg-haproxy-0.54_2... Extracting pfSense-pkg-haproxy-0.54_2: 100% Saving updated package information... done. Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_install_command()...done. Menu items... done. Services... done. Writing configuration... done. [2.4.3-RELEASE][admin@edge.atwlab.com]/root: -
@alexwitherspoon said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:
@willywonka I owe you a beer. That's magic, works here too!
No magic here, just pure desperation
-
As a workaround you can probably use the haproxy-devel package, it functions on the same configuration. and seems to work properly with SNI and offloading with multiple certificates..
For haproxy 'stable' ive send a preliminary 'quick fix'.., should be easy to apply the 2 changed lines manually for those who want need it 'now': https://github.com/pfsense/FreeBSD-ports/pull/542/files#diff-eb226b2eb58fc682fb444d554fb6bab8
That seems to fix the SNI behaviour.. but im not sure about the first report from @kdillen is actually a SNI issue.? Can you @kdillen try the patch?Sorry for the trouble guys..
-
@PiBa Correct in my case it is not SNI because I am using the ssl/https (TCP Mode ) . This is done because I needed the HTTP/2 support which was not yet in Haproxy at moment I first installed the Firewall.
If you want I can try the patch but that will be during the weekend. I actually was lucky to have also a backup for my standby firewall with the older Package version so I did a restore on that one also. Normally on Saturday morning 7:00 CET I make full image backup's of my firewalls so I can easy upgrade the standby node and apply the patch
Can you provide me with the instructions on how to do the patching ? Thanks in advance.
-
@kdillen hi I've checked also haproxy 0.59_4 on my main pfSense (normally I don't do this), but it also did not work, same as 0.59_2. I did the same workaround like Micha (many thnks, see https://forum.netgate.com/user/nonick):
- deinstallation of current version
- pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/lua53-5.3.4_1.txz
- pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/haproxy-1.7.10.txz
- pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
- check box Encrypt (SSL) in HAProxy Frontend(s) and ev. reboot
- check HTTPS and/or HSTS with
6 a) https://www.ssllabs.com/ssltest/analyze.html?d=www.xxx.yy (should be at least a green A+)
6 b) https://www.sslshopper.com/ssl-checker.html#hostname=www.xxx.yy (all should be green)
Regards, LAV, sorry about my English!
-
Try adding SNI Filter in front end config "*.company.com" matching the following certificate. That's how I got mine to work again.
-
The package maintainer pushed several updates last night. They should become available very soon after the next snapshot builds. Watch for the updated versions in System->Package manager->Installed packages, or on the dashboard packages widget.
-
@kdillen
Can you check how the 'servers' are configured in the haproxy backend? I expect yours do want 'https' but dont need haproxy to do the encryption though do have the 'Encrypt(SSL)' checkbox checked while probably they shouldn't now?For others.:
Well 0.59_4 should be available for the 'haproxy' package (haproxy-devel does not need that particular change/fix..) this should have SNI certificate selection for people who are using ssl-offloading with haproxy, and fixed the files tab..
