Completely stumped as to why HAProxy does not connect me to backend.



  • Hi,

    I'm currently having issues connecting to my HAProxy frontend from a remote destination.

    When I connect from a remote address, I do not get a response. However, if I enter in my WAN address into my browser on LAN, I get a 503 service unavailable request.

    
    global
    	maxconn			1000
    	log			/var/run/log	local0	info
    	stats socket /tmp/haproxy.socket level admin
    	uid			80
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    	tune.ssl.default-dh-param	2048
    	#Modern browser compatibility only as mentioned here:
    	# https://wiki.mozilla.org/Security/Server_Side_TLS
    	ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
    	tune.ssl.default-dh-param 2048# Time-to-first-Byte (TTFB) value needs to be optimized based on
    	# the actual public certificate chain see
    	# https://www.igvita.com/2013/10/24
    	# /optimizing-tls-record-size-and-buffering-latency/
    	tune.ssl.maxrecord 1370
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:2200 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats uri /haproxy/haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend front_http
    	bind			#Censored#:80 name #Censored#:80   
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		30000
    	default_backend backend_http_ipvANY
    
    frontend front_https
    	bind			#Censored#:443 name #Censored#:443 ssl  crt /var/etc/haproxy/front_https.pem  
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	acl			aclcrt_front_https	hdr_reg(host) -i ^butterantoast\.com(:([0-9]){1,5})?$
    	acl			aclcrt_front_https	hdr_reg(host) -i ^www\.butterantoast\.com(:([0-9]){1,5})?$
    	use_backend backend_http_ipvANY  if   aclcrt_front_https
    
    backend backend_http_ipvANY
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	redirect scheme https if !{ ssl_fc }
    	server			owncloud 192.168.1.109:80
    
    

    I have not yet tried connecting using my domain name because I don't think that is the issue here and I have not yet gotten DNS forwarding setup yet(with my Domain provider).

    The main Issue that I want to attack is not being able to connect via remote access.

    I am completely stumped.

    Any ideas where I should start troubleshooting?

    I would be grateful for your help,

    Thank you in advance,
    Michael L.



  • Your site looks to be 'working' over :80 but sending wrong content.? And because of that shows a empty page.?

    The fixed ip in the frame isnt right..?

    As for the :443 frontend it will only respond when one of the two domain names is requested.. Testing with a wan-ip wont work there.. As no backend will be selected due to the certificate acl's..



  • @PiBa:

    Your site looks to be 'working' over :80 but sending wrong content.? And because of that shows a empty page.?

    The fixed ip in the frame isnt right..?

    As for the :443 frontend it will only respond when one of the two domain names is requested.. Testing with a wan-ip wont work there.. As no backend will be selected due to the certificate acl's..

    Oh, are you talking about the areas where I replaced my IP with #censored#? I did that out of paranoia, just so I wouldn't be posting my public ip. The part in there is a ip address and not a name or domain name.

    Hmm, okay I see.

    by 'working' are you implying that the server at the endpoint is at fault here?

    I apologise, I'm not quite sure what you mean by frame isn't right. I don't see any part of that code you posted shown in my config.

    –-----------------

    On a different note, I've tried connecting to the http frontend using my free duckdns.org DNS (remotely) and doesn't even give me a response.

    But, I suppose because I've added in the line

    
    redirect scheme https if !{ ssl_fc }
    
    

    it automatically redirects me to HTTPS and therefore by default will not work with ip or different domain name.

    However, I've tried removing that line and testing again, with the same results. (getting no response)

    –---------

    I am stumped, Is this happening because pfsense will not even accept incomming connections through port 443 or 80 since it is being used by PFsense GUI?

    Diagnostics>Sockets list shows port 80 and 443 being taken up by nginx(which I assume is the pfsense server for the GUI).

    
    root	nginx	32282	8	tcp4	*:80	*:*
    root	nginx	32100	6	tcp4	*:443	*:*
    
    


  • Haproxy should be able to bind on the wanip:443 even while nginx is also listening on *:443 , though imho that would be a security risk if for some reason haproxy isnt running your webgui would be exposed to the internet..

    Few things i would propose to change for now.
    -enable http checks on the backend, and make the server is shown in 'green' on the stats page.
    -disable the checkbox for the certificate acl's .
    -make sure your request actually passes through the frontend/backend by checking counters on the stats, or maybe better gather the syslogs it can send..



  • @PiBa:

    Haproxy should be able to bind on the wanip:443 even while nginx is also listening on *:443 , though imho that would be a security risk if for some reason haproxy isnt running your webgui would be exposed to the internet..

    Few things i would propose to change for now.
    -enable http checks on the backend, and make the server is shown in 'green' on the stats page.
    -disable the checkbox for the certificate acl's .
    -make sure your request actually passes through the frontend/backend by checking counters on the stats, or maybe better gather the syslogs it can send..

    By default, I'm assuming that PFSense does not allow connections to the GUI via WAN interface. (right?) I was a bit confused at first when I saw that nginx in the sockets list listening to port 80 and 443. My install is mostly default (as-is:out of the box)with a few small changes made (port forwarding) and (local DNS resolver).

    I'm assuming to allow pfsense GUI to be accessible on WAN, that the admin needs to add a Firewall rule to open :80/:443 on WAN right? And it seems like you're implying the same.

    I guess that's my newbie mistake, my appologies.
    (I had that though in the back of my mind, but I never thought about it).

    –----------------------

    So, if that were the case,

    How could I go about allowing :80/443 traffic to my firewall but prevent PFSense GUI from listening on WAN:80/:443 and only permit HAProxy?


    Also, I noticed that the status of the server (in HAProxy stats) is indeed down. I guess that explains half of my problem... T.T

    The backend server is an Owncloud server using Nginx. Is there any specific configuration I need to add Nginx to allow HAProxy?

    Edit: the error shows as "Layer 7 wrong status: not allowed"

    -Man, I feel like a goof for missing all these simple details, my apologies.



  • Normally pfSense will block all traffic from the WAN and as such be secure.. However if you want to run a website you will need to open the 80 and or 443 ports..

    Webgui will (sadly) always listen on 0.0.0.0, so only choice there is moving it to a different port.. (to get rid of the :80 nginx you could disable the webgui redirect)

    The 'not allowed' might be solvable by choosing a different check method like GET or HEAD instead of OPTIONS for example. Or perhaps sending a domain name in the check. https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_troubleshooting



  • I have just changed my webGUI port to 8081 and added a rule for WAN on 80 and 443 also, I've changed the health check to GET rather than OPTION and it works!!

    Connecting to my fileserver now works when trying to access HAProxy from the local network (I enter in www.butterantoast.com from my web browser and it proxies me to my server)

    Feels great that progress is being made!! :D Thank you!

    –----

    One last(possibly) issue. I still have a problem connecting to my backend via Remote access. I'm testing this using my cellular network. I can connect to the backend server on the WAN url (remotely) but not when there is a path after it. IE: https://butterantoast.com/owncloud

    I locally use the path '<serverip>/owncloud' to connect to the file server.

    HAProxy shouldn't have any problems accepting extra paths right?</serverip>



  • The https://www.butterantoast.com/owncloud/ does end up on owncloud but then tells it isnt trusted/known/configured on the owncloud configuration..

    The other url https://butterantoast.com/owncloud/ however results in a 302 redirect to a 'http' page:

    Request URL:https://butterantoast.com/owncloud/
    Request Method:GET
    Status Code:302 Found
    
    Response Headers
    Location:http://butterantoast.com/owncloud/login
    Server:nginx/1.10.1
    

    If that would have been 'https://butterantoast.com/owncloud/login' then i do get the login page.
    It looks like this is something that needs to be configured on the owncloud configuration owncloud documentation mentions  "overwriteprotocol"=>"https" could be set, i 'think' that might solve the issue.. https://doc.owncloud.org/server/8.2/admin_manual/configuration_server/reverse_proxy_configuration.html

    p.s. it seems :80 isnt working/listening..



  • This is absolutely perfect! I have added HAProxy as a trusted proxy. It's working flawlessly now! Thanks for your help!

    Today I learned… :)