HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck



  • I had HAProxy working great as a reverse proxy with SSL Offloading.

    FrontEnd:

    • Bound to 443

    • SSL Offloading setup.

    • 2 domains with SSL

    • Configured with SSL cert and 'Add ACL for certificate Subject Alternative Names

    • Mutliple ACL where if a path starts with particular value, it will go to relevant backend

    This was working great and had minimal problems.

    I then stumbled upon HAProxy's ability to inspect traffic and being able to then have SSH and SSL on the same 443 port. So I tried setting this up with

    FrontEnd [Call this Front_SSL_SSH:443] set to listen to new 443.

    • Type is tcp

    • Set with inspect delay 5s

    • Set acl 'is_ssl' if req_ssl_ver 2:3.1

    • Configured to use the new back end if is_ssl true - BackEnd_SSL_SSH (see below)

    • User ssh_server backend (essentially to a ssh port on a server in my network)

    BackEnd_SSL_SSH

    • This points to the original FrontEnd at the very top

    SSH works fine, but the web requests fail. The HAProxy logs shows a 'SSL handshake failure' when I try and access the server via a browser. I suspect that the new front end that is doing the detection has done the SSL handshake already, so when it comes the web server, this fails as the browser does not expect a second SSL? Not sure - but would like to keep the SSL Offloading from the original if possible.

    Any ideas? Or have I misunderstood the problem. I have googled but all examples I have found does not use the SSL Offloading for the web part.



  • Can you share the haproxy.conf from bottom of settings tab?



  • I have sanitised the config to the below and changed ip etc. Let me know if anything does not make sense.
    Note that

    • 192.168.2.1 - this is the WAN address of pfsense

    • 192.168.1.1 - this is the LAN address of pfsense

    • 192.168.1.0/24 - this is the LAN network hosting the webservices

    Automaticaly generated, dont edit manually.

    Generated on: 2017-12-10 11:24

    global
    maxconn 128
    log /var/run/log local0 info
    stats socket /tmp/haproxy.socket level admin
    uid 80
    gid 80
    nbproc 1
    chroot /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param 2048
    server-state-file /tmp/haproxy_server_state

    Time-to-first-Byte (TTFB) value needs to be optimized based on

    the actual public certificate chain see

    https://www.igvita.com/2013/10/24

    /optimizing-tls-record-size-and-buffering-latency/

    tune.ssl.maxrecord 1370

    Modern browser compatibility only as mentioned here:

    https://wiki.mozilla.org/Security/Server_Side_TLS

    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

    listen HAProxyLocalStats
    bind 127.0.0.1:2200 name localstats
    mode http
    stats enable
    stats admin if TRUE
    stats uri /haproxy/haproxy_stats.php?haproxystats=1
    timeout client 5000
    timeout connect 5000
    timeout server 5000

    frontend personal-domain.com
    bind 192.168.2.1:443 name 192.168.2.1:443 ssl  crt /var/etc/haproxy/personal-domain.com.pem crt /var/etc/haproxy/personal-domain.com 
    mode http
    log global
    option socket-stats
    option http-keep-alive
    maxconn 10
    timeout client 30000
    redirect scheme https code 301 if !{ ssl_fc }
    acl PersonalDomainBackend hdr_end(host) -i personal-domain.com
    acl AppOneBackend path_beg -i /secure/appone
    acl AppTwoBackend path_beg -i /secure/apptwo
    acl BusinessDomainBackend hdr_end(host) -i business-domain.com
    use_backend appone_http_ipvANY  if  PersonalDomainBackend AppOneBackend aclcrt_personal-domain.com
    use_backend apptwo_http_ipvANY  if  PersonalDomainBackend AppTwoBackend aclcrt_personal-domain.com
    use_backend businessdomain_http_ipvANY  if  BusinessDomainBackend aclcrt_personal-domain.com
    use_backend personaldomain_http_ipvANY  if  aclcrt_personal-domain.com

    frontend personal-dimain.com.http
    bind 192.168.2.1:80 name 192.168.2.1:80 
    mode http
    log global
    option http-keep-alive
    timeout client 30000
    redirect scheme https code 301 if !{ ssl_fc }

    frontend top_level_ssl_ssh_splitter
    bind 192.168.2.1:1443 name 192.168.2.1:1443 ssl  crt /var/etc/haproxy/top_level_ssl_ssh_splitter.pem crt /var/etc/haproxy/top_level_ssl_ssh_splitter 
    mode tcp
    log global
    option log-separate-errors
    option tcplog
    timeout client 30000
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    acl is_ssh payload(1,7) -m bin 5353482d322e30
    tcp-request content accept if is_ssh
    use_backend main_ssl_web_to_frontend_tcp_ipvANY  if  !is_ssh
    use_backend ssh_server_tcp_ipvANY  if  is_ssh
    default_backend ssh_server_tcp_ipvANY

    backend appone_http_ipvANY
    mode http
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server appone 192.168.1.9:9091 check inter 10000

    backend apptwo_http_ipvANY
    mode http
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server apptwo 192.168.1.9:8113 check inter 10000

    backend personaldomain_http_ipvANY
    mode http
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server personal-domain.com 192.168.1.9:10080 check inter 1000

    backend businessdomain_http_ipvANY
    mode http
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    acl businessdomain hdr_end(host) -i business-domain.com
    server business-domain.com 192.168.1.9:10080 check inter 1000

    backend main_ssl_web_to_frontend_tcp_ipvANY
    mode tcp
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server ssl_web_server 192.168.2.1:443

    backend ssh_server_tcp_ipvANY
    mode tcp
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server ssh 192.168.1.9:22



  • The https requests can only be decrypted once.. So after "top_level_ssl_ssh_splitter" decrypts the traffic with the configured certificate. The resulting traffic is plain ssh or http.

    As such the "main_ssl_web_to_frontend_tcp_ipvANY" should probably be pointing to a internal/private/localhost:80 port on haproxy that does not decrypt ssl again and further processes the desired acl's to split domains for appone/apptwo/business/personal apart tp the backends.



  • Not had an opportunity to test this out but just wanted to clarify the config

    • The entry point is port 1443 - as 443 was the original and I didn't want to change the setup to drastically. So the entry port via mapping is to port 1443, this then decides to hit the 443 or the ssh server

    • The https website is working fine. The SSH attempt shows in the haproxy.log an 'SSL handshake failure'

    • Tested via a ssh client on IOS



  • Seems to be the opposite of your previous problem? "SSH works fine, but the web requests fail." and yes i was assuming usage of the 1443.. So please check again which of the two is actually the problem.?.



  • As for the ssh connection make sure to wrap it inside a ssl layer.?

    https://blog.chmd.fr/ssh-over-ssl-episode-4-a-haproxy-based-configuration.html

    Connecting from an SSH client
    
    To connect to your server from linux, just drop this in your ~/.ssh/config:
    
    Host server.com
        ProxyCommand openssl s_client -connect server.com:443 -quiet
    
    If you are on windows and you cannot install anything client side, there is also a solution for you.
    Download socat and putty (none of them requires admin rights).
    Then, with socat, run:
    
    socat TCP-LISTEN:8888 OPENSSL:server.com:443,verify=0
    And with putty, direct your client to 127.0.0.1 on the port 8888.
    


  • thanks - I can't remember what setting I had for 'ssh working but ssl not'. I had some different acl testing entries but as I needed the website up and running I reverted back to the current set of acl above.

    I will try your suggestions. I think there is too much going on so will start from scratch over the holiday period and get basics working before moving up the level of complexity.

    Just a quick question - if I have a splitter that decrypts SSL to divert between the web traffic and SSH traffice, will I be able to use SNI to run two websites on different URL's or have I lost that ability on the split as the SSL is decrypted by that time and its only HTTP sent to the web server/frontend?



  • Using SNI does not need decryption of the traffic, so it should be possible to not configure any certificate on the 1443 frontend, and keep the is_ssh payload check and have it working like that.

    Or perhaps this helps?: https://marc.info/?l=haproxy&m=132375969032305&w=2