HAProxy will hangs when I upgrade to pfsense 2.3.4
-
Here is my basic test setup that works OK:
# Automaticaly generated, dont edit manually. # Generated on: 2017-05-08 15:05 global maxconn 1000 stats socket /tmp/haproxy.socket level admin uid 80 gid 80 nbproc 1 chroot /tmp/haproxy_chroot daemon tune.ssl.default-dh-param 2048 server-state-file /tmp/haproxy_server_state listen HAProxyLocalStats bind 127.0.0.1:2200 name localstats mode http stats enable stats admin if TRUE stats uri /haproxy/haproxy_stats.php?haproxystats=1 timeout client 5000 timeout connect 5000 timeout server 5000 frontend doc-front bind 0.0.0.0:4443 name 0.0.0.0:4443 ssl crt /var/etc/haproxy/doc-front.pem mode http log global option http-keep-alive timeout client 30000 default_backend doc-back_http_ipvANY backend doc-back_http_ipvANY mode http log global timeout connect 30000 timeout server 30000 retries 3 option httpchk GET / server doctor 10.20.0.10:80 check inter 1000 -
Max SSL Diffe-Hellman size: 2018
Is that a typo? That should probably be 2048. Otherwise it seems sane.
Yeah, it was a typo :-\
-
I searched my log files from the long saturday night - the only message that I could find related and suspect are problems before reboot like
2017-05-06 23:40:30 Error (3) PHP-CGI rc.initial.reboot: The command '/usr/local/etc/rc.d/haproxy.sh stop' returned exit code '1', the output was 'Stopping haproxy. Waiting for PIDS: 76571\. Stopping haproxy. No matching processes were found'My config is running fine after degrading to 1.7.2 - it's a very basic setup with two backends online and ssl/https frontend with sni.
-
I reproduced the issue on a fresh install. It seems that there is a problem with DNS resolvers of haproxy.
1. Install a fresh pfSense 2.3.4-RELEASE (amd64)
2. Install haproxy from package manager
3. Add a backend with your favourite web server
3. Add a HTTP frontend, listen to any:8080, set the default backend.
4. Check Enable HAProxy, Maximum connections: 500, Internal stats port: 2200
5. In the DNS servers section: click new server, Name: local, DNSserver:127.0.0.1, DNSport:53
6. Apply changes.
7. HAProxy looks like up and running.
8. Setup firewall rules and visit port 8080 / 2200. No response.
9. Visit haproxy stats page few times, and webConfigurator should hang now.–
EDIT: amd64. -
OK, that I can reproduce on 2.3.4 and 2.4. The key is having the DNS resolver configured inside haproxy. It appears to get hung up attempting to query the UNIX socket for stats. Even if I try to hit that manually, it never receives a response. If I start haproxy manually in the foreground (Disable daemon mode), the stats command succeeds.
-
I can also confirm that if I remove the DNS entries in Haproxy it no longer hangs pfsense. However, it is still not routing the calls to any internal server. I presume the daemon issue still exist.
-
However, it is still not routing the calls to any internal server.
Does it show a 503 service unavailable error?
–-
EDIT: Could you try disabling ssl offloading (if enabled) and see if it work or not? -
Got it to work, Had a typo (SSL) and once fixed everything seems to work like normal as long as I do not add any DNS entries into haproxy.
Thanks for all the help, much appreciated.
-
FYI- Same problem with haproxy 1.7.4 on plain FreeBSD, so not specific to our package. I did just copy over the config and adjust though, so it could be in one of the directives there but nothing seems obvious yet. Still trying things out.
-
Same problem with 1.7.5 also on FreeBSD.
-
OK, I reported the issue upstream and I have a lead on fixing our local copy until they can work on a fix. Won't be long now, at least for a temporary fix.
-
New version of HAProxy is up which has the problematic commits removed. Works OK here with resolvers configured in HAProxy.
Update and give it a try.
https://redmine.pfsense.org/issues/7533
-
Thank you @jimp for the update, it works great for me!
-
-
Thank you all for resolving this issue.
It seemed I had the exact same problem with the DNSes in the HAproxy config.I'm glad it is resolved, now I don't need to keep a putty session open anymore to keep HAproxy running.
-
I updated the haproxy again with a new patch that seems to be a better fix. If anyone has problems with the new version, let me know.
-
Thank you @jimp. The new version works as expected, no hanging anymore. Great job!
-
Sorry about the noob question in advance, but can someone please advise or point me in the right direction on how to update to the fix? I have tried a reinstall and it hasn't worked.
