HAProxy will hangs when I upgrade to pfsense 2.3.4



  • My haproxy  version is 1.7.4.Packages version is 0.52_7.Do you need what information?


  • Banned

    Messing with this via GUI won't produce much useful info. Run

    
    /usr/local/etc/rc.d/haproxy.sh start
    
    

    from console and post the output.



  • OK,I will try this command.Thanks a lot.



  • I run this command.It's hang.
    Please see attachment.




  • I have test.If I stop haproxy and start haproxy.It's can start.But still can't open stats tab.

    [2.3.4-RELEASE][root@pfSense.aspa.idv.tw]/root: /usr/local/etc/rc.d/haproxy.sh stop
    Stopping haproxy.
    Waiting for PIDS: 43914.
    Stopping haproxy.
    No matching processes were found
    [2.3.4-RELEASE][root@pfSense.aspa.idv.tw]/root: /usr/local/etc/rc.d/haproxy.sh start
    Starting haproxy.

    Could have any log need check it?



  • After struggling with the console for the whole day, it seems that the haproxy-1.7.4 provided in pfSense 2.3.4 does not work in daemon mode.

    The haproxy is listening, but it does not response.

    
    [2.3.4-RELEASE][root@***]/root: sockstat | grep haproxy
    www      haproxy    2567  1  udp4   127.0.0.1:33846       127.0.0.1:53
    www      haproxy    2567  5  stream /tmp/haproxy.socket.2497.tmp
    www      haproxy    2567  6  tcp4   *:80                  *:*
    www      haproxy    2567  7  tcp4   *:443                 *:*
    www      haproxy    2567  10 stream /var/run/php-fpm.socket
    
    [2.3.4-RELEASE][root@***]/root: curl 127.0.0.1:80
    ^C
    // the curl 'hangs'.
    
    

    Force haproxy to run in foreground mode

    haproxy -V -db -- /var/etc/haproxy/haproxy.cfg
    

    or debug mode

    haproxy -d -- /var/etc/haproxy/haproxy.cfg
    

    restores it.

    Tried reinstall pfSense 2.3.4 and import the config back to the system, but the haproxy still does not work.

    Manually downgraded to haproxy-1.7.2 provided in pfSense 2.3.3 "fix" the service.

    pkg add https://pkg.pfsense.org/pfSense_v2_3_3_amd64-pfSense_v2_3_3/All/haproxy-1.7.2.txz
    

    WARNING: Running the command above may break your package dependency and break your firewall. Do not run the command on production environment.

    I am not sure if it is a config / local problem or not.
    Needs more confirmation.


    EDIT: format



  • @Cow:

    Manually downgraded to haproxy-1.7.2 provided in pfSense 2.3.3 "fix" the service.

    pkg add https://pkg.pfsense.org/pfSense_v2_3_3_amd64-pfSense_v2_3_3/All/haproxy-1.7.2.txz
    

    WARNING: Running the command above may break your package dependency and break your firewall. Do not run the command on production environment.

    I am not sure if it is a config / local problem or not.
    Needs more confirmation.

    I'm having the same problem.
    If I try to install the older version I get a notification that haproxy is already installed.
    Is there a way to install the older version while keeping the config?



  • I also am having the same issue.  It hangs the pfsense box.  Any idea when we could expect a fix so it starts working again in 2.3.4?

    haproxy is a great product and we use it extensively, I presume it has to be upgraded to work properly with 2.3.4.

    Running :  Pfsense 2.3.4
                    Haproxy: 1.7.4  pfsense package 0.52_7

    Thanks



  • The same problem at my site, also having pfSense 2.3.4, HAProxy 1.7.4.

    To be exact, just WebGUI hangs and the WebGUI restart or php-fpm restart (console option 11 or 16) returns it to be responsive again.

    And as @Cow mentioned, running HAProxy in foreground or debug mode manually is a quick workaround, but no long-term solution.


  • Rebel Alliance Developer Netgate

    Those of you experiencing this problem, can you post more about both your GUI and your HAProxy configurations?

    Is HAProxy handling your GUI connections in some way? Or is your GUI on an alternate port? Do you have the HAProxy dashboard widget active?

    Need some more specifics about the HAProxy end of things as well, general config info, frontend/backend config, etc.

    Since the same version of HAProxy (1.7.4) is also on 2.4, I'd be curious to know if anyone has a problem with that as well, or if it's working as expected.



  • @jimp;

    HAProxy is not handling any GUI connections for pfSense (it is only active as a reverse proxy and installed through the packages supplied in pfSense).
    I had the dashboard widget active, but I have removed it cause it was noticed that it could possibly be a solution.

    I do not feel confortable sharing all my config from haproxy, but I'm using a shared frontend on port 443 for multiple backends.
    The backend consists of multiple different web servers on different ports (some connections are plain http, some are https).

    Other common settings:
    Enable HAProxy: ticked
    Maximum connections: 1024
    Carp monitor: disabled
    Internal stats port: 2200
    Syslog has been setup.
    DNS servers have been entered in the Global DNS resolvers list.
    No mail configurated
    Max SSL Diffe-Hellman size: 2048

    If there is anything else you would like to know, just post here and I'll try to reply asap.

    ** Typo on the Diffe-Hellman size… **


  • Rebel Alliance Developer Netgate

    That's fine, I don't need all of your specifics, mostly what I mentioned: Listening port(s) for the GUI and haproxy and if they are connected in some way, and answers to my other questions.

    I setup a simple haproxy instance on 2.4 with the widget, SSL offloading to a backend server, and it works fine there. I'll have to setup another web server to test 2.3.4, but I'd like to know more about how you have the haproxy and GUI daemons set to listen/bind on the firewall at least.


  • Banned

    Hmmm. With a pretty simple setup with SSL offloading here, this works just fine as always. Then I have another fairly complicated one with lots of backends, multiple frontends and the pfSense GUI itself behind HAproxy plus the LUA ACME plugin, this works perfectly fine as well.

    Both have HAproxy on 80/443 and GUI at 4443, the HTTP => HTTPS redirect disabled for webGUI.



  • here is my binding if it can help.

    Automaticaly generated, dont edit manually.

    Generated on: 2017-05-08 11:51

    global
    maxconn 10000
    stats socket /tmp/haproxy.socket level admin
    uid 80
    gid 80
    nbproc 1
    chroot /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param 2048
    server-state-file /tmp/haproxy_server_state

    listen HAProxyLocalStats
    bind 127.0.0.1:8080 name localstats
    mode http
    stats enable
    stats refresh 10
    stats admin if TRUE
    stats uri /haproxy/haproxy_stats.php?haproxystats=1
    timeout client 5000
    timeout connect 5000
    timeout server 5000

    frontend httpWEBSites
    bind 127.0.0.1:8080 name 127.0.0.1:8080 
    mode http
    log global
    option socket-stats
    option dontlog-normal
    option log-separate-errors
    option httplog
    option http-keep-alive
    option forwardfor
    acl https ssl_fc
    http-request set-header X-Forwarded-Proto http if !https
    http-request set-header X-Forwarded-Proto https if https
    timeout client 30000
    errorfile /var/etc/haproxy/errorfile_httpWEBSites__
    #remove header that expose security-sensitive information
    rspidel ^Server:.*S
    rspidel ^X-Powered-By:.*S
    rspidel ^X-AspNet-Version:.*S

    redirect scheme https if (hdr(Host) -i www.filopto.com ) !{ssl_fc }

    acl nas_acl hdr(host) -i famille.accra.ca
    acl syncbox_acl hdr(host) -i syncbox.accra.ca
    acl syncbox_acl hdr(host) -i securebackup.accra.ca
    acl remotehelp_acl hdr(host) -i remotehelp.accra.ca
    acl ftpserver_acl hdr(host) -i ftpweb.accra.ca
    acl demofilopto_acl hdr(host) -i demo.filopto.com
    acl accra_acl hdr_end(host) -i accra.ca
    acl filopto_acl hdr_end(host) -i filopto.com
    acl dragondreams_acl hdr_end(host) -i dragondreams.ca
    acl dragondoodles_acl hdr_end(host) -i dragondoodles.ca
    acl ajefnb_acl hdr_end(host) -i ajefnb.nb.ca
    use_backend NasWEBServer4_http_ipvANY  if  nas_acl
    use_backend Securebackup16_http_ipvANY  if  syncbox_acl
    use_backend RemoteHelp25_http_ipvANY  if  remotehelp_acl
    use_backend FiloptoDemoWEBSite103_http_ipvANY  if  demofilopto_acl
    use_backend WEBServer14_http_ipvANY  if  filopto_acl
    use_backend WEBServer14_http_ipvANY  if  dragondreams_acl
    use_backend WEBServer14_http_ipvANY  if  dragondoodles_acl
    use_backend WEBServer14_http_ipvANY  if  ajefnb_acl
    default_backend WEBServer14_http_ipvANY



  • I'm having the same issue.  I run haproxy on port 4343 which doesn't conflict with any other ports.

    I'd also like to know more about these awesome domains:
      acl        dragondreams_acl  hdr_end(host) -i dragondreams.ca
      acl        dragondoodles_acl  hdr_end(host) -i dragondoodles.ca


  • Rebel Alliance Developer Netgate

    @cjbujold:

    […]
    listen HAProxyLocalStats
    bind 127.0.0.1:8080 name localstats
    […]
    frontend httpWEBSites
    bind 127.0.0.1:8080 name 127.0.0.1:8080 
    […]

    Should your stats and a live frontend really be bound to the same port? Try moving the stats to port 2200. HAProxy may be smart enough to do the right thing there, but it's better not to tempt fate.


  • Rebel Alliance Developer Netgate

    @gjurriens:

    Max SSL Diffe-Hellman size: 2018

    Is that a typo? That should probably be 2048. Otherwise it seems sane.


  • Rebel Alliance Developer Netgate

    Here is my basic test setup that works OK:

    # Automaticaly generated, dont edit manually.
    # Generated on: 2017-05-08 15:05
    global
    	maxconn			1000
    	stats socket /tmp/haproxy.socket level admin
    	uid			80
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    	tune.ssl.default-dh-param	2048
    	server-state-file /tmp/haproxy_server_state
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:2200 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats uri /haproxy/haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend doc-front
    	bind			0.0.0.0:4443 name 0.0.0.0:4443 ssl  crt /var/etc/haproxy/doc-front.pem  
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	default_backend doc-back_http_ipvANY
    
    backend doc-back_http_ipvANY
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk GET / 
    	server			doctor 10.20.0.10:80 check inter 1000  
    
    


  • @jimp:

    @gjurriens:

    Max SSL Diffe-Hellman size: 2018

    Is that a typo? That should probably be 2048. Otherwise it seems sane.

    Yeah, it was a typo  :-\



  • I searched my log files from the long saturday night - the only message that I could find related and suspect are problems before reboot like

    2017-05-06 23:40:30	Error (3)	PHP-CGI rc.initial.reboot: The command '/usr/local/etc/rc.d/haproxy.sh stop' returned exit code '1', the output was 'Stopping haproxy. Waiting for PIDS: 76571\. Stopping haproxy. No matching processes were found'
    

    My config is running fine after degrading to 1.7.2 - it's a very basic setup with two backends online and ssl/https frontend with sni.



  • I reproduced the issue on a fresh install. It seems that there is a problem with DNS resolvers of haproxy.

    1. Install a fresh pfSense 2.3.4-RELEASE (amd64)
    2. Install haproxy from package manager
    3. Add a backend with your favourite web server
    3. Add a HTTP frontend, listen to any:8080, set the default backend.
    4. Check Enable HAProxy, Maximum connections: 500, Internal stats port: 2200
    5. In the DNS servers section: click new server, Name: local, DNSserver:127.0.0.1, DNSport:53
    6. Apply changes.
    7. HAProxy looks like up and running.
    8. Setup firewall rules and visit port 8080 / 2200. No response.
    9. Visit haproxy stats page few times, and webConfigurator should hang now.


    EDIT: amd64.


  • Rebel Alliance Developer Netgate

    OK, that I can reproduce on 2.3.4 and 2.4. The key is having the DNS resolver configured inside haproxy. It appears to get hung up attempting to query the UNIX socket for stats. Even if I try to hit that manually, it never receives a response. If I start haproxy manually in the foreground (Disable daemon mode), the stats command succeeds.



  • I can also confirm that if I remove the DNS entries in Haproxy it no longer hangs pfsense.  However, it is still not routing the calls to any internal server.  I presume the daemon issue still exist.



  • @cjbujold:

    However, it is still not routing the calls to any internal server.

    Does it show a 503 service unavailable error?

    –-
    EDIT: Could you try disabling ssl offloading (if enabled) and see if it work or not?



  • Got it to work, Had a typo (SSL) and once fixed everything seems to work like normal as long as I do not add any DNS entries into haproxy.

    Thanks for all the help, much appreciated.


  • Rebel Alliance Developer Netgate

    FYI- Same problem with haproxy 1.7.4 on plain FreeBSD, so not specific to our package. I did just copy over the config and adjust though, so it could be in one of the directives there but nothing seems obvious yet. Still trying things out.


  • Rebel Alliance Developer Netgate

    Same problem with 1.7.5 also on FreeBSD.


  • Rebel Alliance Developer Netgate

    OK, I reported the issue upstream and I have a lead on fixing our local copy until they can work on a fix. Won't be long now, at least for a temporary fix.


  • Rebel Alliance Developer Netgate

    New version of HAProxy is up which has the problematic commits removed. Works OK here with resolvers configured in HAProxy.

    Update and give it a try.

    https://redmine.pfsense.org/issues/7533



  • Thank you @jimp for the update, it works great for me!


  • Rebel Alliance Developer Netgate

    @Cow:

    Thank you @jimp for the update, it works great for me!

    Thank YOU for finding the way to reproduce it!  ;D



  • Thank you all for resolving this issue.
    It seemed I had the exact same problem with the DNSes in the HAproxy config.

    I'm glad it is resolved, now I don't need to keep a putty session open anymore to keep HAproxy running.


  • Rebel Alliance Developer Netgate

    I updated the haproxy again with a new patch that seems to be a better fix. If anyone has problems with the new version, let me know.



  • Thank you @jimp. The new version works as expected, no hanging anymore. Great job!



  • Sorry about the noob question in advance, but can someone please advise or point me in the right direction on how to update to the fix? I have tried a reinstall and it hasn't worked.


Log in to reply