After upgrading to HAProxy 0.59_2 nothing works anymore!!!!



  • Hello all,

    Just to let you all know to be cautious about upgrading HAProxy to the latest version: 0.59_2 because everything just stops working.

    Nothing is getting through anymore and all my sites where down. Restoring my backup of the firewall to have back the version 0.54_2 and all worked again as before.

    Content of the log on 1 of the backend nodes:

    10.xxx.xxx.236 - - [26/Jul/2018:08:59:23 +0200] "HEAD / HTTP/1.1" 302 0 "-" "-"
    10.xxx.xxx.236 - - [26/Jul/2018:08:59:53 +0200] "HEAD / HTTP/1.1" 302 0 "-" "-"
    10.xxx.xxx.236 - - [26/Jul/2018:09:00:23 +0200] "HEAD / HTTP/1.1" 302 0 "-" "-"
    10.xxx.xxx.236 - - [26/Jul/2018:09:00:32 +0200] "\x16\x03\x01\x00\xCC\x01\x00\x00\xC8\x03\x03\xBF0\xBERO\x1A\x9D\x05z\xCF\xD2N\xAE\x125\x89\xEB\xC1\x05\xE8\x8E\xCF&\xA8\xA1_\xA87\xC5\xA9\x18\xA1\x00\x00\x1C**\xC0+\xC0/\xC0,\xC00\xCC\xA9\xCC\xA8\xC0\x13\xC0\x14\x00\x9C\x00\x9D\x00/\x005\x00" 400 166 "-" "-"
    10.xxx.xxx.236 - - [26/Jul/2018:09:00:32 +0200] "\x16\x03\x01\x00\xCC\x01\x00\x00\xC8\x03\x030\x07\x1C\x0E\x04!\xFC\x5CI\xC9\x02\x8C\xB2b\x8F\xBC\xB2>\xE3\x15Dw#DKe\xB7u\xC8\xFAC=\x00\x00\x1CZZ\xC0+\xC0/\xC0,\xC00\xCC\xA9\xCC\xA8\xC0\x13\xC0\x14\x00\x9C\x00\x9D\x00/\x005\x00" 400 166 "-" "-"
    10.xxx.xxx.236 - - [26/Jul/2018:09:00:53 +0200] "HEAD / HTTP/1.1" 302 0 "-" "-"
    10.xxx.xxx.236 - - [26/Jul/2018:09:01:23 +0200] "HEAD / HTTP/1.1" 302 0 "-" "-"

    10.xxx.xxx.235 - - [26/Jul/2018:09:01:43 +0200] "HEAD / HTTP/1.1" 302 0 "-" "-"
    10.xxxx.xxx.235 - - [26/Jul/2018:09:02:13 +0200] "HEAD / HTTP/1.1" 302 0 "-" "-"
    10.xxxx.xxx.235 - - [26/Jul/2018:09:02:40 +0200] "GET / HTTP/2.0" 302 109 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:40 +0200] "GET /users/sign_in HTTP/2.0" 200 3510 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:40 +0200] "GET /assets/application-3699df5421217cf3678b3fccba46be0eb9ba5f72488c2eece3cf7ee2e8e8b284.css HTTP/2.0" 200 127854 "https://gitlab.xxxxx.eu/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:40 +0200] "GET /assets/webpack/runtime.9fcb75d4.bundle.js HTTP/2.0" 200 1861 "https://gitlab.xxxxx.eu/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:40 +0200] "GET /assets/webpack/pages.sessions.new.6dbf9c97.chunk.js HTTP/2.0" 200 1735 "https://gitlab.xxxxx.eu/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:40 +0200] "GET /assets/webpack/main.a66b6c66.chunk.js HTTP/2.0" 200 222178 "https://gitlab.xxxxx.eu/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:40 +0200] "GET /assets/webpack/commons~pages.ldap.omniauth_callbacks~pages.omniauth_callbacks~pages.sessions~pages.sessions.new.432e20dc.chunk.js HTTP/2.0" 200 3601 "https://gitlab.xxxxx.eu/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:40 +0200] "GET /assets/print-c8ff536271f8974b8a9a5f75c0ca25d2b8c1dceb4cff3c01d1603862a0bdcbfc.css HTTP/2.0" 200 384 "https://gitlab.xxxxx.eu/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:41 +0200] "GET /assets/webpack/emoji.04b9c3fd.chunk.js HTTP/2.0" 200 109348 "https://gitlab.xxxxx.eu/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
    10.xxx.xxx.235 - - [26/Jul/2018:09:02:41 +0200] "GET /assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png HTTP/2.0" 200 1611 "https://gitlab.xxxxx.eu/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"

    Same request. The source IP ending with 236 is the standby node who was active during the restore of the primary node 235. The request was identically and haproxy configation also ( Replicated configuration ). Fw 236 runs haproxy package: 0.59_2 . Fw 235 runs haproxy package: 0.54_2

    Package maintainers can you please make package 0.54_2 back available so that my secondary node can downgrade to this working version ?

    Thanks in advance.

    Kurt



  • Same for me here, upgraded from 0.54_2 to 0.59_2 yesterday evening and all my SSL offloading endpoints stopped functioning completely! No backup here, so now I am trying to figure out how to go back to 0.54_2 ....

    I'd say do NOT upgrade to version 0.59_2 of HAPROXY package.



  • Since I installed the version 0.59_2 is the SNI functionality not longer working. The browser is receiving only the main certificate. The version 0.54_2 was working fine.



  • Awesome, good thing I held off, sadly, I was waiting for 1.7.11 to resolve https://redmine.pfsense.org/issues/8580



  • I can confirm, I also got affected by upgrading, SNI is not working - only the main cert is being issued. No way to downgrade the physical appliance right now. I am dead in the water.

    I was able to replicate at work + at home. Here is a home config set up similarly to production config with same structure/approach.

    https://gist.github.com/alexwitherspoon/7bfe371ae532e791231caacc03a8ffee



  • Found this in the changelog - https://redmine.pfsense.org/issues/8670

    Devel branch was copied over, so that's why the massive upgrade got pushed out.



  • @alexwitherspoon said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

    Found this in the changelog - https://redmine.pfsense.org/issues/8670
    Devel branch was copied over, so that's why the massive upgrade got pushed out.

    Really nice and all but how do we get this issue registered in the first place? Right now it seems to stall as a forum post, i tried to login with my account to the issue tracker but no luck. What is the next step?



  • @willywonka
    Hi, I have the same issue, 6 sites offline! Backup file gives me a tag error. Do you know how to go back to haproxy 0.54_2?
    Regards, LAV



  • @lavenetz Nope, i do not know how to get back online ... i might need to review my upgrade policy 😭 , because this update kicked me in the lulz hard.



  • @willywonka thanks, anyway the contributor of the package is responsible to bring a new version with a patch. But I cannot wait until next year, by the way. I have an old pfSense with 2.3.5-RELEASE-p2 (i386), an there is the version 0.54_2 installed and running,but it's 32-bit version! What do you think?
    Regards. LAV



  • @lavenetz this is the 0.54_2 package. But howto restore this version, i do not know at this moment .....



  • @willywonka yeah - downgrades for packages have never been supported, so I'd be cautious. - I get it though, I am also impacted, and wished I had looked extra hard at the changelog before hitting the button. Most of these upgrades are soo smooth.



  • @alexwitherspoon Perhaps, you don't believe it, but I have an extra test pfSense, exactly the same hardware. The problem is, althought I did a backup and the update log showed me successful

    Installing pfSense-pkg-haproxy-devel...
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up to date.
    Updating pfSense repository catalogue...
    pfSense repository is up to date.
    All repositories are up to date.
    The following 2 package(s) will be affected (of 0 checked):

    New packages to be INSTALLED:
    pfSense-pkg-haproxy-devel: 0.59_2 [pfSense]
    haproxy-devel: 1.8.12 [pfSense]

    Number of packages to be installed: 2

    The process will require 2 MiB more space.
    727 KiB to be downloaded.
    [1/2] Fetching pfSense-pkg-haproxy-devel-0.59_2.txz: .......... done
    [2/2] Fetching haproxy-devel-1.8.12.txz: .......... done
    Checking integrity... done (2 conflicting)

    • pfSense-pkg-haproxy-devel-0.59_2 [pfSense] conflicts with pfSense-pkg-haproxy-0.59_2 [installed] on /usr/local/pkg/haproxy.xml
    • haproxy-devel-1.8.12 [pfSense] conflicts with haproxy-1.7.11 [installed] on /usr/local/man/man1/haproxy.1.gz
      Checking integrity... done (0 conflicting)
      Conflicts with the existing packages have been found.
      One more solver iteration is needed to resolve them.
      The following 4 package(s) will be affected (of 0 checked):

    Installed packages to be REMOVED:
    haproxy-1.7.11
    pfSense-pkg-haproxy-0.59_2

    New packages to be INSTALLED:
    haproxy-devel: 1.8.12 [pfSense]
    pfSense-pkg-haproxy-devel: 0.59_2 [pfSense]

    Number of packages to be removed: 2
    Number of packages to be installed: 2
    [1/4] Deinstalling pfSense-pkg-haproxy-0.59_2...
    Removing haproxy components...
    Menu items... done.
    Services... done.
    Loading package instructions...
    Deinstall commands... done.
    Syslog entries... done.
    [1/4] Deleting files for pfSense-pkg-haproxy-0.59_2: .......... done
    Removing haproxy components...
    Syslog entries... done.
    Configuration... done.
    [2/4] Deinstalling haproxy-1.7.11...
    [2/4] Deleting files for haproxy-1.7.11: ........ done
    [3/4] Installing haproxy-devel-1.8.12...
    [3/4] Extracting haproxy-devel-1.8.12: ........ done
    [4/4] Installing pfSense-pkg-haproxy-devel-0.59_2...
    [4/4] Extracting pfSense-pkg-haproxy-devel-0.59_2: .......... done
    Saving updated package information...
    done.
    Loading package configuration... done.
    Configuring package components...
    Loading package instructions...
    Custom commands...
    Executing custom_php_install_command()...done.
    Menu items... done.
    Services... done.
    Writing configuration... done.

    Cleaning up cache... done.
    Erfolgreich

    I didn't see that the HAProxy showed me backend_server_ipvANY in red! So I executed the update on the productive pfSense as well. Interesting accidents!

    By the way, I replaced on the test system the haproxy 0.59_ by haproxy-devel 0.59_2 but with no effect.



  • @lavenetz yeah I didn't actually have any issues running the upgrade, that went fine. My HAProxy shows all green status pages , and no issues, except that SNI isn't working. Only the primary certificate is issued, no other certificates are issued despite being in the crt_list.

    That makes this one tricky to detect, though I could have tested ALL urls for proper 200 status and valid certs.



  • @alexwitherspoon Ok so i managed to revert to v0.54_2 successfully with again my ssl offloading (SNI) working, this is how:

    1. On pfSense console i insert 8 followed by enter (to choose Console).
    2. i type in pkg remove haproxy-0.59_2
    3. i got asked, are you sure? Insert yes
    4. then i type pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
    5. after installation reboot pfSense and voila everything working again and package manager says: haProxy v0.54-2
    6. Party!


  • @willywonka I owe you a beer. That's magic, works here too!

    [2.4.3-RELEASE][admin@edge.atwlab.com]/root: pkg remove pfSense-pkg-haproxy-0.59_2
    Checking integrity... done (0 conflicting)
    Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):
    
    Installed packages to be REMOVED:
            pfSense-pkg-haproxy-0.59_2
    
    Number of packages to be removed: 1
    
    Proceed with deinstalling packages? [y/N]: y
    [1/1] Deinstalling pfSense-pkg-haproxy-0.59_2...
    Removing haproxy components...
    Menu items... done.
    Services... done.
    Loading package instructions...
    Deinstall commands... done.
    Syslog entries... done.
    [1/1] Deleting files for pfSense-pkg-haproxy-0.59_2: 100%
    Removing haproxy components...
    Syslog entries... done.
    Configuration... done.
    [2.4.3-RELEASE][admin@edge.atwlab.com]/root: pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
    Fetching pfSense-pkg-haproxy-0.54_2.txz: 100%   69 KiB  70.5kB/s    00:01    
    Installing pfSense-pkg-haproxy-0.54_2...
    Extracting pfSense-pkg-haproxy-0.54_2: 100%
    Saving updated package information...
    done.
    Loading package configuration... done.
    Configuring package components...
    Loading package instructions...
    Custom commands...
    Executing custom_php_install_command()...done.
    Menu items... done.
    Services... done.
    Writing configuration... done.
    [2.4.3-RELEASE][admin@edge.atwlab.com]/root:
    


  • @alexwitherspoon said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

    @willywonka I owe you a beer. That's magic, works here too!

    No magic here, just pure desperation 😲



  • As a workaround you can probably use the haproxy-devel package, it functions on the same configuration. and seems to work properly with SNI and offloading with multiple certificates..

    For haproxy 'stable' ive send a preliminary 'quick fix'.., should be easy to apply the 2 changed lines manually for those who want need it 'now': https://github.com/pfsense/FreeBSD-ports/pull/542/files#diff-eb226b2eb58fc682fb444d554fb6bab8
    That seems to fix the SNI behaviour.. but im not sure about the first report from @kdillen is actually a SNI issue.? Can you @kdillen try the patch?

    Sorry for the trouble guys..



  • @PiBa Correct in my case it is not SNI because I am using the ssl/https (TCP Mode ) . This is done because I needed the HTTP/2 support which was not yet in Haproxy at moment I first installed the Firewall.

    If you want I can try the patch but that will be during the weekend. I actually was lucky to have also a backup for my standby firewall with the older Package version so I did a restore on that one also. Normally on Saturday morning 7:00 CET I make full image backup's of my firewalls so I can easy upgrade the standby node and apply the patch

    Can you provide me with the instructions on how to do the patching ? Thanks in advance.



  • @kdillen hi I've checked also haproxy 0.59_4 on my main pfSense (normally I don't do this), but it also did not work, same as 0.59_2. I did the same workaround like Micha (many thnks, see https://forum.netgate.com/user/nonick):

    1. deinstallation of current version
    2. pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/lua53-5.3.4_1.txz
    3. pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/haproxy-1.7.10.txz
    4. pkg add http://files00.netgate.com/pfSense_v2_4_2_amd64-pfSense_v2_4_2/All/pfSense-pkg-haproxy-0.54_2.txz
    5. check box Encrypt (SSL) in HAProxy Frontend(s) and ev. reboot
    6. check HTTPS and/or HSTS with
      6 a) https://www.ssllabs.com/ssltest/analyze.html?d=www.xxx.yy (should be at least a green A+)
      6 b) https://www.sslshopper.com/ssl-checker.html#hostname=www.xxx.yy (all should be green)
      Regards, LAV, sorry about my English!


  • Try adding SNI Filter in front end config "*.company.com" matching the following certificate. That's how I got mine to work again.


  • Developer Netgate

    The package maintainer pushed several updates last night. They should become available very soon after the next snapshot builds. Watch for the updated versions in System->Package manager->Installed packages, or on the dashboard packages widget.



  • @kdillen
    Can you check how the 'servers' are configured in the haproxy backend? I expect yours do want 'https' but dont need haproxy to do the encryption though do have the 'Encrypt(SSL)' checkbox checked while probably they shouldn't now?

    For others.:
    Well 0.59_4 should be available for the 'haproxy' package (haproxy-devel does not need that particular change/fix..) this should have SNI certificate selection for people who are using ssl-offloading with haproxy, and fixed the files tab..



  • @lavenetz hi, am I doing this pkg add from shell or is there away to do it from the web GUI?

    Thanks,



  • @piba said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

    ckbox checked while probably they shouldn't now?

    @PiBa I did the work around with going back to a previous version. Today I did the upgrade to the latest version of pfsense (2.4.4) and guess what the issue is back. I worked around it by using some extra public IP's and NAT and going back to the pfSense Loadbalancer.

    Afterwards I tried to recreate a new TCP based loadbalancer. I played with all the settings for front en backend but or I get nothing passing through it or I get everything encrypted and not readable for my reverse proxies. I am not using any ACL's or anything just TCP forwarding towards 2 backends.

    Anybody else has this problem ? Because it should not cause these kind of problems I guess ?

    Thanks in advance.



  • @kdillen
    Can you post your haproxy.cfg ? Tried enabling/disabling the ssl options on the backend server?



  • Make sure there is nothing in "SSL Offloading - client certificates" in the Frontend. I noticed that the latest version had a change to the way this was handled.



  • I found the solution myself and yes it is strange behavior, something that used to work but suddenly not work anymore.

    • First part is the SSL checkbox in the backends that solved 1 part of the issue in my case.
    • Second part: Health check method. In my case I put it to HTTP and that made that my hosts where not available ( failed health check ) This is something that used to work in TCP mode but now suddenly not anymore. So i put them on basic.

    After fixing both of these all my TCP forwarding problems are gone. But still this should be a big notification in the upgrade notes because it really changes in a big way things that used to work to not working at all anymore.



  • @kdillen
    I doubt putting a note in some upgrade text would have helped your case.. And stating 'found the solution myself' like 4 days after having been pointed at the solution. Which really is configuring the 2 ssl checkboxes on the backend/server really doesn't justify shouting out 'nothing works'.. Lots of things do work, and yes there is a minor upgrade issue in some cases for users that use sni.. which is really simple to solve if you understand what haproxy is doing. Excuse my response.. I do my best to keep upgrades 'seamless' but i cannot validate every possible configuration for every situation before sending a PR on github.

    So anyhow:
    Enable http healthchecks again like they used to be and enable the "SSL checks" and it will verry likely work as it used to..



  • @piba Well in that case how come that a configuration that worked perfectly with a haproxy 0.54.2 ( including older versions ) and with an upgrade to haproxy 0.59_2 or 0.59_4 it suddenly does not work anymore ?

    But the haproxy below is the same version at least for 0.54_2 and 0.59_2 for 0.59_4 it is a new version of haproxy but still a minor upgrade.

    And for your information the configuration was converted from haproxy 1.5 to haproxy 1.6 to haproxy 1.7.10 and never had issues with pfsense packages 0.54_2 or lower just after going to 1.7.10 or higher since pfSense haproxy package 0.59.x became available!!!!

    So it means the configuration generated with the old package and new package is not compatible anymore in certain conditions... I am not that stupid to understand this. But at least it should have been noted that some parameters have other meanings and could create strange behavior after upgrade to the new packages.. And if you do not understand that. Then I wonder what would qualify for you as a requirement to make an upgrade note available. I have seen Note's for application upgrades which caused far less damage than this.

    And 4 days pointing out the solution.. What solution the ssl checkboxes... Well they are not the solution.. They are just a part of the solution but also the cause, because that is the part that is not backwards compatible between 0.54_x and 0.59_x without any notice or remark!!! Secondly the real solution of the problem in my case was the "Health Check Method" chance. Something that used to work and suddenly not work anymore in TCP mode. I wonder what has changed there without any NOTIFICATION!!!!

    And I did not even see your message with your solution until I wrote my last message.. What you think everybody is waiting constantly for their messages. I just did a nice workaround. Used the build in pfsense load balancer and added some of the destinations behind some free spare public NAT addresses directly. But I know then I not have sticky sessions. But that gave me the time to look into the problem and try to solve it before to update my last firewall.



  • Okay i might have stepped out of bed with the wrong foot first, and would like to apologize for my response.

    @kdillen said in After upgrading to HAProxy 0.59_2 nothing works anymore!!!!:

    the real solution of the problem in my case was the "Health Check Method" chance

    For that part changing the health-check from HTTP to TCP is not the 'correct' solution, though it would 'pass' the check, it might then also pass the check if the server is still listening but only producing http errors. The health check and its parameters are (or at least should) still be working as it used to. If it fails due to the upgrade of the package, then the only thing to change to fix that is to enable or disable the "SSL checks" checkbox.



  • @piba I believe you are saying this, but just to clarify for other who were confused like me:

    The reason you need to change the health check, is because unchecking the "SSL" checkbox changes how HAProxy connects to the server. Without that checked, the HTTP check is only HTTP, and does not properly negotiate the SSL connection. The correct fix for this is to change the check method to "SSL".

    So, if you are like me, and get a SSL_ERROR_RX_RECORD_TOO_LONG error after you upgrade, it's because of a change in how HAProxy handles intranet SSL encryption, so you need to do the following:

    • Go to Services -> HAProxy -> Backend
    • Select each HTTPS backend in turn, and change the following:
      • In the server list, select the server, click the edit pencil, and UNcheck the "Encrypt(SSL)" box.
      • Under the "Health checking" section, change the "Health check method" to "SSL".
      • Scroll to the bottom, and click Save
    • Once you're done with all the servers, apply the configuration.
    • You can double check that the health checking is working by checking the "Stats" page.


  • @nick2253
    Sorry but that was not what i'm saying.

    To make haproxy health-checks use SSL you should enable the "SSL checks" checkbox behind each server.

    0_1540496969032_cb3883c2-37b0-4062-a4bb-094704716f28-image.png