HAProxy as SSL Reverse Proxy Behind Single IP



  • Hello,

    I'd like to implement a reverse proxy that can make all of my webserver-lan-IPs:Ports and be directed appropriately by using a ddnshost-wan-IP.ca:443 from the outside. HTTPS to start, but other protocols are welcome later if HAProxy can make sense of them.

    I have a single IP from my ISP since statics are $15/month! :o

    Mirroring this inside my network would be the preference, but one step at a time here.

    I have not been successful at implementing this and am looking for some help. PfSense support told me that one could make HAProxy work for me but it would be 6 hours to build. I'm not going to say the rate, but it is not cheap. That wasn't a big surprise, so I've been attempting this on my own.

    This is only for my networking desire at home. Nothing major…or I would buy statics no question.

    Do any of you have this working in some capacity? Currently, I do not have much built as I keep blowing it all away.

    One example:

    LAN IP        LAN PORT      PUBLIC IP                      PUBLIC PORT
    10.0.0.1      5001              https://my-ddns1.ca    443
    10.0.0.2      80                  https://my-ddns2.ca    443
    10.0.03      443                https://my-ddns3.ca    443

    Eventually I'd like to mirror the my-ddns#.ca:443 (or port 80 to relief SSL processing in the LAN) hostname to my LAN even though the service may be on 5001. Forward and Reverse Proxy. No load balancing.

    Am I crazy?



  • Hi Brailyn,

    Proxying HTTPS and or HTTP to multiple backends from one public-ip should be possible with haproxy. Doing that myself, so no your probably not crazy :o , not that i'm legally allowed to make such judgement.

    Anyway for HTTPS you have a few choices what to do:
    1-use haproxy in HTTPS/SSL mode and use SNI information from the ssl handshake to decide.
    2-load certificates on haproxy and decypher HTTPS on haproxy then decide by host header where to send the traffic
    If you go with option 2, then the question is on the backend side if you want to encrypt traffic again or perhaps keep it as http.. Re-encrypting costs more cpu.. But might avoid some trouble with absolute urls returned by the webapplication..

    Use haproxy-devel package untill 1.6 stable package arrives :).. Haproxy 1.6 itself is stable so nothing much to worry there..

    Create 3 backends pointing to your servers:
    Create one frontend. And add acl's and actions there to direct traffic to the proper backend.

    Check in stats that the servers are seen 'UP' when using healthchecks. And fix settings where needed..

    If you still have trouble please post the content of haproxy.cfg so i can advice further.

    Regards,
    PiBa-NL



  • Sorry for the huge delay, I've been dreading how difficult this is going to be.

    I'm just going to walk through the steps as I do them so you can point out my mistakes. If it is a waste of your time, just say so and I'll post my config file that you mentioned (if I can find it)…

    BACKENDS

    I'll start with the DSM subdomain.

    Balance=none
    Transparent ClientIP = yes and LAN for backend.

    ACL1
    Name= DSM
    Host starts with: https://dsm.root-domain.ca

    ACL2
    Name= root
    Host starts with: https://root-domain.ca

    FRONTEND

    External Address

    WAN address, 443, SSL-offloading=yes

    ACL1
    Name= DSM
    Host starts with: https://dsm.root-domain.ca

    ACL2
    Name= root
    Host starts with: https://root-domain.ca

    not sure if it matters, but saving everything gives this error:

    
    [WARNING] 347/215713 (62182) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048\. Please set a value >= 1024 to make this warning disappear.
    
    

    Firewall, I allowed "any source, 192.168.1.100 dest" on 443 so I could test. I'm behind a 2wire gateway so you know. 443 is port forwarded.

    I jumped onto a remote network to test. Going to subdomain DSM resolved in an HSTS errrr. Going to my root domain shows a HSTS error too.

    Edit: I see that this morning from work, I get a "503 Service Unavailable | No server is available to handle this request."  :D so it looks like HAProxy is listening to some extent.

    My questions so far:

    1. How do I specify the ports of each service within my LAN?

    2. Where is the haproxy.cfg file?

    3. Is there any guides on how HAProxy is configured to do something similar to this? Or is it supposed to be straight forward? (It sure isn't that obvious to me :o )

    Again, thank you for your help!



  • I think your acl 'host starts with' should not include 'https://'

    1- On the backend you should add servers that point to the apropriate ports.
    2-On the config tab at the bottom there is a link to show the config.

    3- the zip file attached here has some info: https://forum.pfsense.org/index.php?topic=93766.msg527268#msg527268
    Also creating a new wiki here https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki but thats a work in progress..



  • Hello again,

    I appreciate your guides, PiBa. I am a lot further ahead than last night, and have spend a lesser amount of time attempting it.

    I decided to go with SSL offloading…I do not have many web servers... and I like the ability to manage their certificates (and CAs?) accordingly.

    I've got one and only one website barely working. Issues seem to be:

    HSTS Browser Security Limitation

    1)On my OSX via separate LAN, both Chrome and Safari give the HSTS error and man-in-the-middle attacks (Proxy in the middle I would say)

    1. I created a certificate with the CN matching the hostname in an attempt to resolve. Same issue.
      My iPhone allows me to connect via it's safari:) I changed the LAN server a couple of times to prove it was not just a cached site.

    2. What's going on here?

    Self-signed Internal Certificates

    1. I created them a few times with different CNs. No luck there either. Still URL matching issues.

    2. I don't want to buy a legit wildcard CA until I know that I can make it work with a self-signed one. At which point I would probably just add it to the trusted list on mine/friends machines.

    3. The alternative names section seems like the way to go….just plug in all of the domains pointing to my single IP address right? Type =DNS, Value=DNS hostnames.

    4. The CA and Cert thing always seems to cause issues with us noobs.

    My Config

    
    /var/etc/haproxy.cfg file contents:
    global
    	stats socket /tmp/haproxy.socket level admin
    	uid			80
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:444 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats uri /haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend websites
    	bind			192.168.1.100:443 name 192.168.1.100:443 ssl  crt /var/etc/haproxy/websites.pem  
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	acl			dsm	hdr(host) -i dsm.my-domain.ca
    	default_backend dsm_http_ipvANY
    
    backend dsm_http_ipvANY
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			dsm 10.0.0.1:5001 ssl check inter 1000  weight 1 verify none
    
    

    General Questions Again

    Q1) How do I add the same frontend to multiple backends? It didn;t seem to allow this for SSL offloading if I remember correctly.

    Q2) When is it good to use the Transparent ClientIP option in the backend servers? I know my dsm server has it's own firewall which could cause issues here.
    Edit: Just tried it from work this morning, and it is indeed working on chrome with a Transparent IP. I'd bet it's safer that way it doesn't keep accounts logged in for 10.0.0.254 even though it could be from another public IP outside my network. Could this actually be an issue when not using Transparent ClientIPs?

    Q3) Once I get multiple sites going visible to the outside world, do I just add alias hostnames to pfsense and HAProxy will send them to the appropriate LAN devices from within the LAN, or is it not that easy?

    Q4) To prevent the Man-in-the-Middle errors, does the proxy need some sort of intermediate cert? or is it actually the "end" point to the outside world?

    Thank you for the guides! I'd love to help if I was capable.



  • Haproxy does not need the CA for sending it to the client, the client should already have the ca stored in the trusted certificate store.

    1. HSTS is a security measure which makes browsers verify that a valid and trusted certificate is used for the connection. So if CN does not match, intermediate certificate is missing, or the CA that signed the certs is not already trusted the browser will refuse to connect.

    2. adding 'your' ca to the trusted list is important for getting the certificate checked as valid.

    3. yes using alternate names is possible, or create a wildcard cert.

    Config)
    -The acl currently does nothing, it needs a use_backend action (this 'recently' changed in the package..)

    Webserver shows up in green on the stats page?

    Q1) use shared frontends, or simply configure multiple acl's  and use_backend actions.
    Q2) if the backend needs the client-ip and cannot use the x-forward-for header, but im not sure what issue your referring to exactly.? It does come with its own set of problems…
    Q3) every server haproxy will use needs to be defined in a backend, aliases wont do much for this purpose.
    Q4) if the servercert loaded into haproxy was signed by a intermediateca then that does need to be loaded into the pfSense certificate manager.



  • Thank you for the concise answers. I'm glad I have been numbering my convoluted rambles :)

    Before I attempt some of these things, I would like to clarify a few things and respond to what you asked.

    "Webserver shows up in green on the stats page?" Yes, it is green. Port 10.0.0.254:444 does not work for some reason though.

    1. Any tips on making a wildcard cert? Or is it simply *.domain.ca for the CN?

    2. How/where do I add "use_backend" to the config? via SSH?

    3. I think I've sorted out the transparent IP thing enough. I am aware that it does come with it's own set of problems. Discussion for later maybe.

    4. (Q3) so once HAProxy works for the WAN, the LAN will work for the entries as well? It sort of seemed to work last night without horrible loopback issues, but I didn't test it thoroughly.



  • The 444 port that is confgured for HAProxyLocalStats can only be accessed by pfSense itself, and is used 'internally' by the stats tab and Status/haproxy-stats of the haproxy package.

    1. yes just put "*.domain.ca" as the CN, and perhaps also "domain.ca" as an alternative name.

    2. below the 'acl' on the frontend you can add a 'action' that should allow to choose use_backend

    3. yup later 8)

    4. yes the haproxy wan-ip frontends can be accessed by lan users and it should work properly.

    However if you then enable 'transparent-client-ip' the horrible loopback issues will happen if client and server are on the same subnet. Server will try to reply directly to client, while haproxy waits for response but doesnt get it, and client expects response from haproxy..



  • Hello again,

    Took a break from this for a while.

    Been having trouble getting a single frontend to point to multiple backends. Any request gets forwarded to the first default in any case.

    Here's my config:

    
    /var/etc/haproxy.cfg file contents:
    global
    	stats socket /tmp/haproxy.socket level admin
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:444 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats uri /haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend websites
    	bind			192.168.1.100:443 name 192.168.1.100:443 ssl  crt /var/etc/haproxy/websites.pem  
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	acl			dsm	hdr(host) -i dsm.my-domain.ca
    	acl			pfsense	hdr(host) -i pfsense.my-domain.ca
    	acl			webroot	hdr(host) -i my-domain.ca
    	default_backend dsm_http_ipv4
    	default_backend webroot_http_ipv4
    	default_backend pfsense_http_ipv4
    	default_backend dsm_http_ipv4
    
    backend dsm_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			dsm 10.0.0.1:5001 ssl check inter 1000  weight 1 verify none 
    
    backend webroot_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			webroot 10.0.0.1:443 ssl check inter 1000  weight 1 verify none 
    
    backend pfsense_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			pfsense 10.0.0.254:443 ssl check inter 1000  weight 1 verify none
    
    

    I created a *.my-domain.ca CA (with my-domain.ca as a alternate name) and cert and I keep getting HSTS errors on a lot of browsers. Seems as though I have not created the SSL stuff correctly. I also cannot add the CA to my trusted CA list without a "password" in my OSX keychain–that is not a parameter in PfSense cert configurator... Haven't tried adding the CA to windows machines yet.

    Thanks for the help!



  • Looks like in each frontend you configured the 'default backend' so you ended up with 4 defaults.

    Please set it to 'none' on 3 and add a 'action' to look like this:

    action: Use Backend
    parameter backend: dsm
    condition acl name: dsm

    action: Use Backend
    parameter backend: pfsense
    condition acl name: pfsense



  • p.s. the 'Templates' tab contains a link for creating a example for "Serving multiple domains from 1 frontend."
    If you checkout the configuration that link creates maybe it helps a little..



  • Okay,

    I finally got a few backend servers running via one front end from the outside :)

    1. Most of my servers have their own SSL settings… Would this explain the HSTS error from some browsers? I don't mind removing LAN SSL from them, it just takes a bit of time... I hate to see efforts wasted if I know they wouldn't help.

    2.1) Is it normal for servers to be bound to their specific backend from the WAN? Once I create the "dsm" subdomain for my synology, I can no longer access it via its old port (5001). Is this normal?

    2.2) If you use synology products at all... I will also include that I am having trouble getting mobile applications to work through the rProxy on 443 from outside my LAN or "on-the-road". This doesn't surprise me, but I am also positive it did work for me in the early stages... Maybe it is just a cert based issue that will be solved by proceeding with (1).



  • That you got a few running is great :)

    1. HSTS is a header send by the server, and is cached for X amount of time (1 year is the usual setting..) , and can even be persisted if submitted to a online list, then you might never get rid of it…

    Its actually good to have, as it 'forces' all future connections from clients to be over https with a VALID certificate for the url they request in the address bar. (You will need to get a valid certificate, this means installing the CA you used to sign the certificate on all client computers, or get a certificate from a real CA like LetsEncrypt or buy one..)

    If you already have a valid certificate then perhaps you forgot to load the intermediate certificate into pfSense that could cause issues.. Check with for example https://www.ssllabs.com/ssltest/ if indeed the chain not complete.?.

    2.1) This is due to the setting "transparent client ip", sadly the implementation is not 'perfect' all reply traffic is 'captured' from the server and send to haproxy.. Even if the inital request did not go through haproxy.. (The webgui does warn for this effect, sorry..)

    Workaround is possible by making the server listen on a second port or a second ip, but depending on the machine running the website that might be difficult to configure on that side..

    2.2) not using synology myself.. But yes if the certificates are not 'valid' that could cause issues..



  • Before I let you go for now,

    Is there any way to do HTTPS redirects using HAProxy?



  • Depending what you intend to do exactly my first thought is, yes it can.

    Use either a 'action' or put this into advanced passthrough:

    redirect scheme https if !{ ssl_fc }
    

    And make the frontend listen on :80 as well as the current :443 with offloading.



    1. I believe that works! (Edit: jk, tried IE and no luck…) I used the advanced option and am listening on :80, not sure how to do the 'action' one... I know the advanced statement includes "if", but does this allow HTTP traffic through when I have some eventually, or does it force redirect any urls to HTTPS?

    2. Is there any way to have a single frontend do SSL offloading as well as HTTPS where the SSL Handshake is done by the specific servers (SNI I believe)?

    I setup two front ends on :443 and it shut everything down...



    1. The "!{ ssl_fc }"matches all traffic that was not offloaded by haproxy. So effectively makes it impossible to perform a plain http request without redirecting to https..

    It is of course possible to make the criteria more selective..

    acl MyPlainHttpHost hdr(Host) www.plain.http.example.com
    acl issecure ssl_fc
    
    redirect scheme https if !issecure !MyPlainHttpHost
    
    1. Its not possible to have two frontends listen on the same port and get consistent results..
      It is possible to first use sni for a few domains, and then forward some other traffic to a second frontend.

    Frontend1 wanip:443 (using SNI)

    • backend1: webserver 192.168.10.10:443
    • backend2: (forward to frontend2 over unix socket)

    Frontend2 localhost:10443 with certificate offloading

    • backend3 : webserver 192.168.10.30:80
    • backend4 : webserver 192.168.10.40:80


  • "(forward to frontend2 over unix socket)" would that just be 127.0.0.1:10443 or localhost:10443? Do I have to config anything else to use this port?



  • That would work to.
    But i prefer the for the field "Forwardto" of the server definition to not set it to "adress+port" but to "Frontend2"



  • Okay, now this is starting to get out of hand… :o

    1. Is the new frontend1 shared with frontend2-- with frontend1 being the primary?
      1.1) If so, the backend2 "forwarder" only sees frontend1, rather than frontend2...so I'm stuck with that case. OR,
      1.2) If not, do I set frontend2 to listen on 10443 only and frontend1 to be the main :443 :80 listener? When I tried this, at it allows SNI to work, but the forwarding to SSLfrontend2 does not work.
      1.25) In the backend that forwards to the frontend2, does the SSL box have to be checked to the right of the "Forwardto" box? It seems to make everything not work when it is checked.

    1.3) It's just a chain in my small mind...      outside-https-request -> SNIfrontend1 -> backend "forwarder"  -> SSLfrontend2 -> server (not working after 1.2, or 1.25)
    or the non-offloading scenario, still HTTPS... outside-https-request -> SNIfrontend1 -> server (working after 1.2, not after 1.25)

    Is that the correct way of looking at it?

    Second issue
    2.1) When attempting to make an HTTP request, it says "Server Hangup" which leads me to believe that my Frontend2 is sort of working and your advanced config code is doing something.

    2.2) This may resolve itself once the first mess is fixed up. We can work on issue one first… cause it seems like a doozie :)

    2.3) Again, I really appreciate all the help with my complex desires ;D This is probably the most ridiculious thing that it's been used for ;)

    My config after 1.2)

    Worked before changes to allow SNI and SSL offloading
    https://my-domain.ca:443 -> 10.0.0.6:443 (TransparentIP)
    https://dsm.my-domain.ca:443 -> 10.0.0.6:5001 (TransparentIP)
    https://pfsense.my-domain.ca:443 -> 127.0.0.1:443
    these are in frontend2 (called rProxy)

    Worked after changes to allow SNI and SSL offloading
    https://ubnt.my-domain.ca:443 -> 10.1.1.40:443
    http://photo.my-domain.ca:80 -> 10.0.0.3:8080  (non-ssl)
    These are in frontend1.
    No requests appear to make it to frontend2, all http requests show server hangup.

    
    /var/etc/haproxy/haproxy.cfg file contents:
    global
    	stats socket /tmp/haproxy.socket level admin
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:444 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats uri /haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend rProxy
    	bind			192.168.1.100:10443 name 192.168.1.100:10443 ssl  crt /var/etc/haproxy/rProxy.pem  
    	bind /tmp/haproxy_chroot/rProxy.socket name unixsocket accept-proxy ssl  crt /var/etc/haproxy/rProxy.pem 
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	acl photo hdr(Host) ubnt.my-domain.ca
    	acl issecure ssl_fc
    
    	redirect scheme https if !issecure !photo
    	acl			dsm	hdr(host) -i dsm.my-domain.ca
    	acl			webroot	hdr(host) -i my-domain.ca
    	acl			pfsense	hdr(host) -i pfsense.my-domain.ca
    	acl			photo	hdr(host) -i photo.my-domain.ca
    	use_backend dsm_http_ipv4  if  dsm 
    	use_backend webroot_http_ipv4  if  webroot 
    	use_backend pfsense_http_ipv4  if  pfsense 
    	use_backend photo_http_ipv4  if  photo 
    
    frontend Frontend1
    	bind			192.168.1.100:443 name 192.168.1.100:443   
    	bind			192.168.1.100:80 name 192.168.1.100:80   
    	mode			tcp
    	log			global
    	maxconn			10
    	timeout client		30000
    	tcp-request inspect-delay	5s
    	acl			ubnt	req.ssl_sni -i ubnt.my-domain.ca
    	tcp-request content accept if { req.ssl_hello_type 1 }
    
    	use_backend ubnt_https_ipvANY  if  ubnt 
    	default_backend forward2rProxy_https_ipvANY
    
    backend dsm_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			dsm 10.0.0.6:5001 ssl check inter 1000  weight 1 verify none 
    
    backend webroot_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			webroot 10.0.0.6:443 ssl check inter 1000  weight 1 verify none 
    
    backend pfsense_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			pfsense 127.0.0.1:443 ssl check inter 1000  weight 1 verify none 
    
    backend photo_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			photo 10.0.0.3:8080 check inter 1000  weight 1 
    
    backend ubnt_https_ipvANY
    	mode			tcp
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			ubnt 10.1.1.40:443 check-ssl check inter 1000  weight 1 verify none 
    
    backend forward2rProxy_https_ipvANY
    	mode			tcp
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			forward-to-rProxy /rProxy.socket send-proxy-v2-ssl-cn check inter 1000
    
    


    1. All 3 frontends should be 'primary'.
      Using 1 frontend for both 80 and 443, while using them both in TCP mode, means the backend will receive mixed connections.. Some with plain http other with ssl traffic.. That wont work..
      1.1/1.2 checkout my new wiki page :)
      1.3) its indeed a chain.

    2.1) When requesting a HTTP page, it will first wait 5 seconds in the first fronted for the SSL 'hello'.. Then its forwarded to the second frontend, which also waits for the client to send the 'SSL-HELLO'.. The client never sends this, and the haproxy cannot 'decrypt' the traffic.. caused by 1)

    2.3) you are not the first to attempt this ;)
    Because it is kinda complicated ive added a page to my 'wiki', based on 2.3 but same principles.. maybe it helps a bit:
    https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/pfsense_2_3_haproxy_sni_plus_offloading_backends



  • I greatly appreciate that wiki page :) Hopefully others discover it soon!

    It does exactly what I want it to, but I couldn't quite make mine do it. Very close though!

    SSL and SNI works. HTTP does not.

    1. If I make a request on a fresh browser (say pfsense.my-domain.com) it does not forward to https, but rather the 503 service not available page.

    2. I cannot get one of my servers to UP. photo.my-domain.com. That is my http test server. I think it has to do with what frontends it is in.
      You kept your www (http) page as the default in the  http-frontend1 and for the SSL-offloading-Frontend3… My default www page (called webroot) is HTTPS, so I wasn't quite sure how to implement the HTTP page into the frontends without making it default... regardless it needs to be UP first.

    3. I am okay with abandoning HTTP if it is easier...  8)

    4. Where did you get the theme from in your wiki??

    My config:

    
    /var/etc/haproxy/haproxy.cfg file contents:
    global
    	stats socket /tmp/haproxy.socket level admin
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:444 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats uri /haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend Frontend3-offload
    	bind			127.0.0.1:1443 name 127.0.0.1:1443 ssl  crt /var/etc/haproxy/Frontend3-offload.pem  
    	bind /tmp/haproxy_chroot/Frontend3-offload.socket name unixsocket accept-proxy ssl  crt /var/etc/haproxy/Frontend3-offload.pem 
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	acl			dsm-ssl	        hdr(host) -i dsm.my-domain.ca
    	acl			webroot-ssl	hdr(host) -i my-domain.ca
    	acl			pfsense-ssl	hdr(host) -i pfsense.my-domain.ca
    	acl			photo-nonssl	hdr(host) -i photo.my-domain.ca
    	use_backend dsm_http_ipv4  if  dsm-ssl 
    	use_backend webroot_http_ipv4  if  webroot-ssl 
    	use_backend pfsense_http_ipv4  if  pfsense-ssl 
    	use_backend photo-http_http_ipv4  if  photo-nonssl 
    	default_backend webroot_http_ipv4
    
    frontend Frontend2-SNI
    	bind			192.168.1.100:443 name 192.168.1.100:443   
    	mode			tcp
    	log			global
    	maxconn			10
    	timeout client		30000
    	tcp-request inspect-delay	5s
    	acl			ubntsni1	req.ssl_sni -i ubnt.my-domain.ca
    	tcp-request content accept if { req.ssl_hello_type 1 }
    
    	use_backend ubnt_https_ipvANY  if  ubntsni1 
    	default_backend Frontend3offload_https_ipvANY
    
    frontend Frontend1-http
    	bind			192.168.1.100:80 name 192.168.1.100:80   
    	mode			http
    	log			global
    	option			http-keep-alive
    	maxconn			10
    	timeout client		30000
    	acl			httpRedirectACL	hdr(host) -i photo.my-domain.ca
    	http-request redirect scheme https  if  httpRedirectACL 
    	default_backend photo-http_http_ipvANY
    
    backend dsm_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			dsm 10.0.0.6:5001 ssl check inter 1000  weight 1 verify none 
    
    backend webroot_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			webroot 10.0.0.6:443 ssl check inter 1000  weight 1 verify none 
    
    backend pfsense_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			pfsense 127.0.0.1:443 ssl check inter 1000  weight 1 verify none 
    
    backend photo-http_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			photo-http 10.0.0.3:8080 check inter 1000  weight 1 
    
    backend ubnt_https_ipvANY
    	mode			tcp
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			ubnt 10.1.1.40:443 check-ssl check inter 1000  weight 1 verify none 
    
    backend Frontend3offload_https_ipvANY
    	mode			tcp
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			Frontend3-srv /Frontend3-offload.socket send-proxy-v2-ssl-cn check inter 5  
    
    backend photo-http_http_ipvANY
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			photo-http 10.0.0.3:8080 check inter 1000  weight 1
    
    


    1. only photo.my-domain.ca is redirected to https, all other requests go to backend photo-http_http_ipvANY
      which doesnt seem logical to me.

    Perhaps you should add a ! before the aclname?
    http-request redirect scheme https  if  !httpRedirectACL
    So that 'photo' can be retrieved over http and everything else like pfsense.my-domain.com causes the redirect.?

    1. The frontend should not matter for getting a backend 'up'.
      What you could try is changing the check method to "GET" and send a version+host header. Or if that fails try the "basic" check..
      https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki#troubleshooting

    2. http should be the easier part :)

    3. It seems to be the default layout for a github based wiki.. I didnt choose anything special.



    1. I put a "!" before "httpRedirectACL" under Condition acl names (in Frontend1-http) and now they all seem to be redirecting to https… which is better than before as the majority of my servers are https only.

    2. Changing to GET didn't help, although I didn't understand what you mean by "send a version+host header"... so I changed to "basic" check and that makes the server turn green. To get to it I had to remove the recently added "!" in 1). But now it seems to be stuck as HTTPS again.

    ....hold on... after getting the server UP (with basic health check) and keeping the "!" everything seems to be working!!!!  ;D though I haven't tested fully yet.

    1. It was about the same amount of difficulty as I started with HTTPS.

    2. The PfSense theme, not the Git theme. Oops... I want that theme.

    3. You've been too much help! Is there anything I can help you with?

    My config "as-is" working FYI.

    
    /var/etc/haproxy/haproxy.cfg file contents:
    global
    	stats socket /tmp/haproxy.socket level admin
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:444 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats uri /haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend Frontend3-offload
    	bind			127.0.0.1:1443 name 127.0.0.1:1443 ssl  crt /var/etc/haproxy/Frontend3-offload.pem  
    	bind /tmp/haproxy_chroot/Frontend3-offload.socket name unixsocket accept-proxy ssl  crt /var/etc/haproxy/Frontend3-offload.pem 
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	acl			dsm-ssl	hdr(host) -i dsm.my-domain.ca
    	acl			webroot-ssl	hdr(host) -i my-domain.ca
    	acl			pfsense-ssl	hdr(host) -i pfsense.my-domain.ca
    	acl			photo-nonssl	hdr(host) -i photo.my-domain.ca
    	use_backend dsm_http_ipv4  if  dsm-ssl 
    	use_backend webroot_http_ipv4  if  webroot-ssl 
    	use_backend pfsense_http_ipv4  if  pfsense-ssl 
    	use_backend photo-http_http_ipv4  if  photo-nonssl 
    	default_backend webroot_http_ipv4
    
    frontend Frontend2-SNI
    	bind			192.168.1.100:443 name 192.168.1.100:443   
    	mode			tcp
    	log			global
    	maxconn			10
    	timeout client		30000
    	tcp-request inspect-delay	5s
    	acl			ubntsni1	req.ssl_sni -i ubnt.my-domain.ca
    	tcp-request content accept if { req.ssl_hello_type 1 }
    
    	use_backend ubnt_https_ipvANY  if  ubntsni1 
    	default_backend Frontend3offload_https_ipvANY
    
    frontend Frontend1-http
    	bind			192.168.1.100:80 name 192.168.1.100:80   
    	mode			http
    	log			global
    	option			http-keep-alive
    	maxconn			10
    	timeout client		30000
    	acl			httpRedirectACL	hdr(host) -i photo.my-domain.ca
    	http-request redirect scheme https  if  !httpRedirectACL 
    	default_backend photo-http_http_ipvANY
    
    backend dsm_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			dsm 10.0.0.6:5001 ssl check inter 1000  weight 1 verify none 
    
    backend webroot_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk OPTIONS / 
    	server			webroot 10.0.0.6:443 ssl check inter 1000  weight 1 verify none 
    
    backend pfsense_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			pfsense 127.0.0.1:443 ssl check inter 1000  weight 1 verify none 
    
    backend photo-http_http_ipv4
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			photo-http 10.0.0.3:8080 check inter 1000  weight 1 
    
    backend ubnt_https_ipvANY
    	mode			tcp
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			ubnt 10.1.1.40:443 check-ssl check inter 1000  weight 1 verify none 
    
    backend Frontend3offload_https_ipvANY
    	mode			tcp
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			Frontend3-srv /Frontend3-offload.socket send-proxy-v2-ssl-cn check inter 5  
    
    backend photo-http_http_ipvANY
    	mode			http
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			photo-http 10.0.0.3:8080 check inter 1000  weight 1
    
    


    1. with current config photo.my-domain.ca should be reachable over http:// without being redirected by haproxy. Maybe the backend itself also sends a https redirect?

    2. in the healthcheck field "Http check version" try it with the following value

    HTTP/1.1\r\nHost:\ www.yourdomain.com\r\nAccept:\ */*
    

    Also do check what the chkresult in the stats is. It should tell if a unexpected response is retrieved or maybe it just takes more than 2 seconds to check the server response?

    Perhaps its a 'permission denied' response? Workaround for that could be checking a different url or accepting 404 as a 'valid' response.. http-check expect status 404 http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#4.2-http-check%20expect

    1. ok

    2. its the default theme of pfSense 2.3 beta snapshots. Ive converted haproxy to bootstrap for usage on 2.3 only recently..

    3. if you have some time install pfSense 2.3 on a virtual machine, add haproxy package and report any issues that might still exist in the package :).

    p.s. The the 5 millisecond on the offloading backend i intended to be a 5000 millisecond timeout.. It might currently be eating some more cpu than needed.. (going to change my wiki screenshot as well..)



    1. It's just a web service running on a rPi2. I turned off https to test http with the rProxy, so it shouldn't be redirecting to https itself.
      It appears that the issue is resolved (now that I've moved to Firefox… Chrome loves remembering broken things)... I can access photo (when it's UP) via http without it redirecting. It appears to use my SSL offloading when I type "https://" into the url though... I'm not sure that I want it to do that. All of my other servers are redirecting to https fine, even if I try http:)

    2. With the "Http check version" set to:

    
    HTTP/1.1\r\nHost:\ photo.my-domain.ca\r\nAccept:\ */*
    
    

    the server goes to "down".

    I cannot find chkresult, but here are a few stats that stuck out on the down server:

    under photo-http_http_ipv4 and photo-http_http_ipvANY (they are red)
    Server Lastchk=L7STS/401 in 10ms
    Server chk=1

    Is there any issues in running the check method as basic?

    "Perhaps its a 'permission denied' response? Workaround for that could be checking a different url or accepting 404 as a 'valid' response.. http-check expect status 404"

    I'm not sure If I need to worry about this anymore as the http request does through the frontend and looks for a backend.

    1. Bootstrap FTW!

    2. I'll deploy another VM and give it a go. I'm not to certain how I'll test it all without interrupting others in my house… I need to get better at running a "network lab"

    Q1) Can I make https requests to http servers deadend to nothing or an error page?
    Q2) Can I have no defaults so that incorrect domain names also go nowhere, or is this poor practice?



    1. if you dont want photo to be reachable over https then remove it from 'Frontend3'

    2. ok so the issue indicated by LastChk is that haproxy gets a 401 response, this is by default considered invalid. But you could configure it to expect that status. Put "http-check expect status 401" into advanced setting of the backend. 'Chkresult' indeed does not exist, i ment the one you found.. Even though basic health check works, doesnt check if the webserver is 'properly working' it only checks if the connection can be made. If a cgi or database backend is not working that might go unnoticed.. And haproxy would declare the backend healthy while it is unable to respond to requests.. Even so the impact is probably small it can mean the diffence between no response, a error response from the backend depending on how functional it is.. And a page from haproxy 'no server available' which could trigger a email alert, and is easy to diagnose. In environments where load-balancing the same domain to multiple servers it is more important to properly detect if 1 server is in a bad state so it can be taken offline and requests will be balanced across the remaining servers.

    Q1) It is possible to deny requests using acls+actions. The error returned by haproxy can be changed using the errorfiles. (example on template tab)
    Q2) Its possible to have no default. Whether or not that is good practice, i don't really know, it might confuse search-engine crawlers if it finds 10 (mis spelled) url's leading to the same website, but i don't have much experience with that. I personally kinda like to always return 'some' response. Perhaps put redirect location action at the bottom towards the main webroot url..



  • Alright,

    Great info yet again, thanks PiBa.

    I can see how the health check can be very important, but at this point I do not desire notifications or for the health do even show up correctly for that matter.

    This may be just my last post Q1, but is there a reasonably easy way to disable servers to resolve in an error page? I honesty haven't checked the docs, but I've got a bunch of rProxy config, and I want to simply disable a few backends without affecting others running… Is there an error page for "temporarily unavailable" I can quick deploy? You can point me to docs if it's a lot to explain here, and we may be getting a bit off topic 8)



  • Q1) 'Disabling' a server can easily be done from the haproxy widged, or the stats page.
    By default haproxy will then send a 503 error, but you can change that page using the error files.
    On the 'files' tab add a new errorfile (or use the example from the 'Templates' tab as a starting point)
    Then assign the errorfile to the backend by adding a line with code 503 and select file available.

    There are other more 'advanced' solutions involving sticktables and acls, but those are probably not easy to recreate through the current webgui options.

    This is an example of such a 'maintenance' page: https://gist.github.com/sts/62d8dd59221ab68661aa
    As-is it requires to run some commands to the /tmp/haproxy.socket to put it into / out of maintenance mode..



  • Cool, it looks like there is a lot of functionality that I will be able to enjoy once I get rolling with this.

    I am beginning to wonder about firewall now… Is there an easy way to allow/deny certain IP ranges from accessing certain backends?



  • As haproxy is only listening on one port firewall rules cannot make a much of a difference depending on the domain used.. (maybe thats not truly correct ;) , if using transparent-client-ip you could technically block haproxy from reaching the backend with some floating rules..)

    Other way would be to use a acl in haproxy 'source matches ip or alias' and then perform a 'http-requests block' action on that acl.



  • I'll fiddle with the firewall later. I shouldn't need much there, but I'll put thought into it another day.

    The widget thing you embedded into the package works very well! :)



  • Hello again,

    In time if come up with a few more questions.

    1. Can the document root be specified in the backends? I have a website that is accessed via my-domain.ca/file1/here.

    2. Is my rProxy config mirrored to within my network at all? As in I have 4 web addresses running on one server via different ports (not vhosts), and I'd like their public domain names, from within my LAN, to resolve to LANIP:WANPUBLICPORT… Is this crazy talk? I do not need this feature, it's just something that would make HAProxy very seamless for me.



  • Hi Brialyn,

    1. If required it is possible to rewrite the request url.. But depending on the web-application and the urls it generates/uses that might be tricky. Checkout reqirep and its syntax, it might help..  http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#4.2-reqirep

    Perhaps these 'examples' will help a little?: https://gist.github.com/PiBa-NL/8ad6c222354cbd7a5af5

    1. HAProxy can be used from the LAN network, but do make sure you keep routing both request and response traffic through pfSense.. This is especially required when using 'transparent client ip'. When doing so the client and server may not be on the same subnet.

    Regards,
    PiBa-NL



  • PiBa,

    1. Was having a bunch of issues with accessing my Synology NAS as it was using vhosts to redirect standard websites… After much grief, it appears that disabling vhosts and applying your first example to the "Backend pass thru" works!
    reqirep  ^([^\ :]*)\ /(.*) \1\ /folder-name/\2
    

    From the WAN side:

    • now by accessing photo.root-domain.ca I get redirected by the index.html in the root folder. Which is how I get redirected to PhotoStation for all you Synology fan boys out there.

    • by accessing root-domain.ca I get direct access the the index.html file found in the /app1 folder without having to specify it in the original URL. It is sort of hidden if you will. Not sure if navigation to other folders is possible now, but I would like to explore for any introduced security/functionality issues.

    1. In a business situation, if I were running a reverse proxy like this, I would most definitely run it on a VM in a completely different subnet with all of my backend clients in that subnet. I would use routing to make my production LAN talk with the rProxy server in the other LAN. I would imagine VLANs could be configured to do this as well, but I do not know much about configuring them. Maybe someday.

    Thanks for all of your help!



  • Hello again,

    Just got most of the web services working on the synology (DSM 5.2 latest as of this post). Then I upgraded it to the newest software (DSM 6.0 Beta2) and most of my web services on that box behind HAProxy broke.  :o

    I sort of assumed it was due to settings not porting over after upgrading… Now, I've been fiddling with it for 2 days and still have had no luck getting things back online. I'm sort of glad I did have it working a couple of days ago, cause that made me understand that my crazy setup did indeed work as it was intended.

    To my point,

    I can access my websites by going to LAN-IP:443 and LAN:443/sub-root-dir and it takes me to the document roots on the NAS, and executes the appropriate index.html files in the specified directory. That's good.

    The stats show the websites as DOWN and when attempting to access them from the WAN, it shows "503 Service Unavailable". I have two LAN IP's on the NAS. Both of them work identically from the LAN, but only one is being used behind this rProxy to prevent the weird DNS issues I was getting at the top of this thread. Really just brute force and ignorance there.

    1. Is there any way to see the logging on such issues? Note, the main NAS landing page is on :5001 and it continues to work fine behind the HAproxy:443 from the outside.

    2. Any other advice to get these pages to roll again? There isn't really anything special about the two troublesome websites, other than the box they are on. vhosts are disabled (as far as I know...) but maybe there are some issues there on this beta version.



  • The "503 Service Unavailable" is normal when the server is down.
    Check what LastChk says in the stats.



  • photo reads Layer6 timeout. L6TOUT in 1004ms

    webroot reads Layer7 wrong status: Bad request. L7STS/400 in 4ms



  • L6 is a problem with SSL, sure the ip:port properly serves a certificate?
    A HTTP 400 status is probably due to the requested page or method used for the checks, try a different one.



  • The SSL issue was just cause I had the Backend pointing to the incorrect port.

    Then it was simply an issue of switching the Health Checks to Basic….

    Not sure why I didn't try that before before. Sorry for all the background... that's what I do when I flustered.

    Thanks again:)