HAProxy will hangs when I upgrade to pfsense 2.3.4

akong · May 5, 2017, 8:36 AM

Hello,
I have install haproxy on pfsense 2.3.3_p1.It's can use it.
But I upgrade to 2.3.4.It's can't see my web server and can't see haproxy stats.
I has reinstall haproxy and restart haproxy.It's can't solved it.
How to fix it?

doktornotor · May 5, 2017, 8:38 AM

Works just fine here on multiple boxes. Some real info required.

akong · May 5, 2017, 8:55 AM

How to find more information?When I click Stats tab.It's will hangs all pfsense system.

akong · May 5, 2017, 9:10 AM

I found stop haproxy also hang.

akong · May 5, 2017, 9:16 AM

My haproxy version is 1.7.4.Packages version is 0.52_7.Do you need what information?

doktornotor · May 5, 2017, 9:33 AM

Messing with this via GUI won't produce much useful info. Run


/usr/local/etc/rc.d/haproxy.sh start

from console and post the output.

akong · May 5, 2017, 10:07 AM

OK,I will try this command.Thanks a lot.

akong · May 5, 2017, 1:11 PM

I run this command.It's hang.
Please see attachment.

akong · May 5, 2017, 1:14 PM

I have test.If I stop haproxy and start haproxy.It's can start.But still can't open stats tab.

[2.3.4-RELEASE][root@pfSense.aspa.idv.tw]/root: /usr/local/etc/rc.d/haproxy.sh stop
Stopping haproxy.
Waiting for PIDS: 43914.
Stopping haproxy.
No matching processes were found
[2.3.4-RELEASE][root@pfSense.aspa.idv.tw]/root: /usr/local/etc/rc.d/haproxy.sh start
Starting haproxy.

Could have any log need check it?

ringo · May 5, 2017, 4:24 PM

After struggling with the console for the whole day, it seems that the haproxy-1.7.4 provided in pfSense 2.3.4 does not work in daemon mode.

The haproxy is listening, but it does not response.


[2.3.4-RELEASE][root@***]/root: sockstat | grep haproxy
www      haproxy    2567  1  udp4   127.0.0.1:33846       127.0.0.1:53
www      haproxy    2567  5  stream /tmp/haproxy.socket.2497.tmp
www      haproxy    2567  6  tcp4   *:80                  *:*
www      haproxy    2567  7  tcp4   *:443                 *:*
www      haproxy    2567  10 stream /var/run/php-fpm.socket

[2.3.4-RELEASE][root@***]/root: curl 127.0.0.1:80
^C
// the curl 'hangs'.

Force haproxy to run in foreground mode

haproxy -V -db -- /var/etc/haproxy/haproxy.cfg

or debug mode

haproxy -d -- /var/etc/haproxy/haproxy.cfg

restores it.

Tried reinstall pfSense 2.3.4 and import the config back to the system, but the haproxy still does not work.

Manually downgraded to haproxy-1.7.2 provided in pfSense 2.3.3 "fix" the service.

pkg add https://pkg.pfsense.org/pfSense_v2_3_3_amd64-pfSense_v2_3_3/All/haproxy-1.7.2.txz

WARNING: Running the command above may break your package dependency and break your firewall. Do not run the command on production environment.

I am not sure if it is a config / local problem or not.
Needs more confirmation.

–
EDIT: format

gjurriens · May 6, 2017, 3:34 PM

@Cow:

Manually downgraded to haproxy-1.7.2 provided in pfSense 2.3.3 "fix" the service.
pkg add https://pkg.pfsense.org/pfSense_v2_3_3_amd64-pfSense_v2_3_3/All/haproxy-1.7.2.txz
WARNING: Running the command above may break your package dependency and break your firewall. Do not run the command on production environment.

I am not sure if it is a config / local problem or not.
Needs more confirmation.

I'm having the same problem.
If I try to install the older version I get a notification that haproxy is already installed.
Is there a way to install the older version while keeping the config?

cjbujold · May 6, 2017, 4:03 PM

I also am having the same issue. It hangs the pfsense box. Any idea when we could expect a fix so it starts working again in 2.3.4?

haproxy is a great product and we use it extensively, I presume it has to be upgraded to work properly with 2.3.4.

Running : Pfsense 2.3.4
Haproxy: 1.7.4 pfsense package 0.52_7

Thanks

DuSt · May 7, 2017, 6:58 AM

The same problem at my site, also having pfSense 2.3.4, HAProxy 1.7.4.

To be exact, just WebGUI hangs and the WebGUI restart or php-fpm restart (console option 11 or 16) returns it to be responsive again.

And as @Cow mentioned, running HAProxy in foreground or debug mode manually is a quick workaround, but no long-term solution.

jimp · May 8, 2017, 1:09 PM

Those of you experiencing this problem, can you post more about both your GUI and your HAProxy configurations?

Is HAProxy handling your GUI connections in some way? Or is your GUI on an alternate port? Do you have the HAProxy dashboard widget active?

Need some more specifics about the HAProxy end of things as well, general config info, frontend/backend config, etc.

Since the same version of HAProxy (1.7.4) is also on 2.4, I'd be curious to know if anyone has a problem with that as well, or if it's working as expected.

gjurriens · May 8, 2017, 8:02 PM

@jimp;

HAProxy is not handling any GUI connections for pfSense (it is only active as a reverse proxy and installed through the packages supplied in pfSense).
I had the dashboard widget active, but I have removed it cause it was noticed that it could possibly be a solution.

I do not feel confortable sharing all my config from haproxy, but I'm using a shared frontend on port 443 for multiple backends.
The backend consists of multiple different web servers on different ports (some connections are plain http, some are https).

Other common settings:
Enable HAProxy: ticked
Maximum connections: 1024
Carp monitor: disabled
Internal stats port: 2200
Syslog has been setup.
DNS servers have been entered in the Global DNS resolvers list.
No mail configurated
Max SSL Diffe-Hellman size: 2048

If there is anything else you would like to know, just post here and I'll try to reply asap.

** Typo on the Diffe-Hellman size… **

jimp · May 8, 2017, 2:20 PM

That's fine, I don't need all of your specifics, mostly what I mentioned: Listening port(s) for the GUI and haproxy and if they are connected in some way, and answers to my other questions.

I setup a simple haproxy instance on 2.4 with the widget, SSL offloading to a backend server, and it works fine there. I'll have to setup another web server to test 2.3.4, but I'd like to know more about how you have the haproxy and GUI daemons set to listen/bind on the firewall at least.

doktornotor · May 8, 2017, 2:33 PM

Hmmm. With a pretty simple setup with SSL offloading here, this works just fine as always. Then I have another fairly complicated one with lots of backends, multiple frontends and the pfSense GUI itself behind HAproxy plus the LUA ACME plugin, this works perfectly fine as well.

Both have HAproxy on 80/443 and GUI at 4443, the HTTP => HTTPS redirect disabled for webGUI.

cjbujold · May 8, 2017, 3:05 PM

here is my binding if it can help.

Automaticaly generated, dont edit manually.

Generated on: 2017-05-08 11:51

global
maxconn 10000
stats socket /tmp/haproxy.socket level admin
uid 80
gid 80
nbproc 1
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
bind 127.0.0.1:8080 name localstats
mode http
stats enable
stats refresh 10
stats admin if TRUE
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend httpWEBSites
bind 127.0.0.1:8080 name 127.0.0.1:8080
mode http
log global
option socket-stats
option dontlog-normal
option log-separate-errors
option httplog
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
errorfile /var/etc/haproxy/errorfile_httpWEBSites__
#remove header that expose security-sensitive information
rspidel ^Server:.*S
rspidel ^X-Powered-By:.*S
rspidel ^X-AspNet-Version:.*S

redirect scheme https if (hdr(Host) -i www.filopto.com ) !{ssl_fc }

acl nas_acl hdr(host) -i famille.accra.ca
acl syncbox_acl hdr(host) -i syncbox.accra.ca
acl syncbox_acl hdr(host) -i securebackup.accra.ca
acl remotehelp_acl hdr(host) -i remotehelp.accra.ca
acl ftpserver_acl hdr(host) -i ftpweb.accra.ca
acl demofilopto_acl hdr(host) -i demo.filopto.com
acl accra_acl hdr_end(host) -i accra.ca
acl filopto_acl hdr_end(host) -i filopto.com
acl dragondreams_acl hdr_end(host) -i dragondreams.ca
acl dragondoodles_acl hdr_end(host) -i dragondoodles.ca
acl ajefnb_acl hdr_end(host) -i ajefnb.nb.ca
use_backend NasWEBServer4_http_ipvANY if nas_acl
use_backend Securebackup16_http_ipvANY if syncbox_acl
use_backend RemoteHelp25_http_ipvANY if remotehelp_acl
use_backend FiloptoDemoWEBSite103_http_ipvANY if demofilopto_acl
use_backend WEBServer14_http_ipvANY if filopto_acl
use_backend WEBServer14_http_ipvANY if dragondreams_acl
use_backend WEBServer14_http_ipvANY if dragondoodles_acl
use_backend WEBServer14_http_ipvANY if ajefnb_acl
default_backend WEBServer14_http_ipvANY

Salient · May 8, 2017, 3:48 PM

I'm having the same issue. I run haproxy on port 4343 which doesn't conflict with any other ports.

I'd also like to know more about these awesome domains:
acl dragondreams_acl hdr_end(host) -i dragondreams.ca
acl dragondoodles_acl hdr_end(host) -i dragondoodles.ca

jimp · May 8, 2017, 7:42 PM

@cjbujold:

[…]
listen HAProxyLocalStats
bind 127.0.0.1:8080 name localstats
[…]
frontend httpWEBSites
bind 127.0.0.1:8080 name 127.0.0.1:8080
[…]

Should your stats and a live frontend really be bound to the same port? Try moving the stats to port 2200. HAProxy may be smart enough to do the right thing there, but it's better not to tempt fate.