HAProxy Empty Responses



  • Apologies if this has been posted elsewhere before, but I am having trouble finding solid guidelines for setting up HAProxy. Below is my haproxy.cfg that is generated by PFSense after configuring everything. HAProxy seems to be accepting the traffic, but only sends back empty responses immediately. Does anyone have any thoughts on what I configured incorrectly?

    global
    maxconn 10000
    stats socket /tmp/haproxy.socket level admin
    gid 80
    nbproc 1
    hard-stop-after 15m
    chroot /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param 2048
    server-state-file /tmp/haproxy_server_state

    frontend Front-HTTP
    bind <my_public_IP>:80 name <my_public_IP>:80
    mode http
    log global
    option http-keep-alive
    timeout client 30000
    acl httpRedirectACL var(txn.txnhost) -m str -i drive.mydomain.com
    acl httpRedirectACL var(txn.txnhost) -m str -i mydomain.com
    acl httpRedirectACL var(txn.txnhost) -m str -i www.mydomain.com
    http-request set-var(txn.txnhost) hdr(host)
    http-request redirect scheme https if httpRedirectACL

    frontend Front-HTTPS-merged
    bind <my_public_IP>:443 name <my_public_IP>:443 ssl crt-list /var/etc/haproxy/Front-HTTPS.crt_list
    mode tcp
    log global
    timeout client 30000
    tcp-request inspect-delay 5s
    acl drivename req.ssl_sni -i drive.mydomain.com
    acl wordpressACL req.ssl_sni -i mydomain.com
    acl wordpressACL req.ssl_sni -i www.mydomain.com
    tcp-request content accept if { req.ssl_hello_type 1 }
    use_backend drive_ipv4 if drivename
    use_backend wordpress_ipv4 if wordpressACL
    default_backend drive_ipv4
    default_backend wordpress_ipv4

    backend drive_ipv4
    mode tcp
    id 10100
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    source ipv4@ usesrc clientip
    option ssl-hello-chk
    server drive 192.168.1.11:443 id 10101 ssl check inter 1000 ca-file /var/etc/haproxy/ca_5e714af3e9f3a.pem verifyhost drive.mydomain.com

    backend wordpress_ipv4
    mode tcp
    id 10102
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    source ipv4@ usesrc clientip
    option ssl-hello-chk
    server wordpress 192.168.1.12:443 id 10103 ssl check inter 1000 verify none



  • @mhanding
    I would change the frontend mode to 'HTTP'. Which provides better statistics/and diagnostics for http traffic. (will then need to create new acl's based on host header)
    Perhaps also disable the transparent-client-ip feature.(at least for testing)
    Are the servers 'up' according to the stats page?



  • Hi @PiBa, thanks for the response. Excuse if I am ignorant about the settings, I have only started to dabble in HAProxy yesterday. Would changing the front end so it is HTTP affect HTTPS traffic? One of the servers is a Nextcloud server and I would definitely prefer that it runs over HTTPS.

    Thanks for the tip about the stats page, I had not yet set it up and not realized its usefulness yet. Turns out my attempt to setup SSL checks caused the servers to show as down so the traffic was not being forwarded. Once I removed the check it began forwarding the traffic without needing to make any adjustments to the frontend.

    Now my next challenge popped up: it looks like my check for mydomain.com is overwriting my check for drive.mydomain.com. I will start looking into that; do you have any suggestions for this issue?



  • @mhanding
    As you have a certificate on the frontend, the https-traffic is being decrypted by haproxy. After which it basically is plain http traffic, so mode http would be the best fit, it can then inspect/modify headers, count separate http requests, send http errors if a backend is down, provide better logging. As you can see it has several advantages..

    As for the 'wrong' backend being selected, this might be a sideeffect of the use of SNI and perhaps even a single certificate valid for both names? This would be solved as well by checking the 'Host' header (not possible in tcp-mode). Another thing is that you seem to have selected a 'default backend' in multiple frontend, i would try and remove both and only use acl's+actions for now, and later add a single default, that way its easier to say if/what acl matched.



  • @PiBa
    On my backend, I have both servers checked for Encrypt(SSL); wouldn't this re-encrypt the traffic after HAProxy decrypts for ACL checks? If that checkbox/setting does not actually re-encrypt the traffic, then I can definitely see your point in having the mode HTTP. However, I do not see a "mode" option for the frontend other than the type, which removes certificate options. I am assuming you are not mentioning removing the HTTP redirect; could you be more specific about the option you are suggesting changes?

    After some testing, I actually determined it looks like my ACLs are not actually doing anything; potentially because of SNI? The forwarded backend seems to only change when I change the default backend and returns an empty response when no default is selected. The expression I am using in the ACLs is "Server Name Indication TLS extension matches:". I pasted an updated set of settings below.

    # Automaticaly generated, dont edit manually.

    Generated on: 2020-03-22 05:15

    global
    maxconn 10000
    stats socket /tmp/haproxy.socket level admin
    uid 80
    gid 80
    nbproc 1
    hard-stop-after 15m
    chroot /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param 2048
    server-state-file /tmp/haproxy_server_state

    listen HAProxyLocalStats
    bind 127.0.0.1:2200 name localstats
    mode http
    stats enable
    stats admin if TRUE
    stats show-legends
    stats uri /haproxy/haproxy_stats.php?haproxystats=1
    timeout client 5000
    timeout connect 5000
    timeout server 5000

    frontend Front-HTTP
    bind <my_public_ip>:80 name <my_public_ip>:80
    bind 10.1.1.1:80 name 10.1.1.1:80
    mode http
    log global
    option http-keep-alive
    timeout client 30000
    acl httpRedirectACL var(txn.txnhost) -m str -i drive.mydomain.com
    acl httpRedirectACL var(txn.txnhost) -m str -i mydomain.com
    acl httpRedirectACL var(txn.txnhost) -m str -i www.mydomain.com
    http-request set-var(txn.txnhost) hdr(host)
    http-request redirect scheme https if httpRedirectACL

    frontend Front-HTTPS
    bind <my_public_ip>:443 name <my_public_ip>:443 ssl crt-list /var/etc/haproxy/Front-HTTPS.crt_list
    bind 10.1.1.1:443 name 10.1.1.1:443 ssl crt-list /var/etc/haproxy/Front-HTTPS.crt_list
    mode tcp
    log global
    timeout client 30000
    tcp-request inspect-delay 5s
    acl driveACL req.ssl_sni -i drive.mydomain.com
    acl wordpressACL req.ssl_sni -i www.mydomain.com
    acl wordpressACL req.ssl_sni -i mydomain.com
    tcp-request content accept if { req.ssl_hello_type 1 }
    use_backend drive_ipvANY if driveACL
    use_backend wordpress_ipvANY if wordpressACL
    default_backend drive_ipvANY

    backend drive_ipvANY
    mode tcp
    id 100
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server drive 192.168.1.11:443 id 101 ssl check inter 1000 verify none

    backend wordpress_ipvANY
    mode tcp
    id 102
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server wordpress 192.168.1.12:443 id 103 ssl check inter 1000 verify none

    I also found that due to VLAN seperation, HAProxy was interfering with the interVLAN traffic so you will notice I have another bound interface on the frontend.



  • @mhanding
    Yes the 'encrypt(ssl)' does re-encrypt traffic. Also in the config you can see it shows the 'ssl' keyword behind the servers. But for haproxy 'in memory' processing it can use mode http.
    The change i propose is for the type parameter of the 'Front-HTTPS' frontend to change that to 'http / https(offloading)'. (and yes due to the re-encryption before sending to the backend there is no 'offloading' regarding the ssl cpu usage) it will allow usage of different acl's, it doesn't remove certificate options for 'offloading' it does remove the checks for SNI values as checking for the host header is actually better indicator of what the end-user really requested.

    As for the inter-vlan-communication issue, that was likely caused by the 'transparent-client-ip' feature you have disabled in latest config.



  • @PiBa
    Awesome! Thank you again @PiBa and apologies again for the ignorance on my part. Your insight was extremely helpful and the proxy is now behaving as I had hoped.

    Just a note in case anyone wonders, I tried disabling the transparent IP and that did not help with the interVLAN traffic. The problem only affected the ports defined in HAProxy (80,443) and did not resolve until I added the binded interface in the frontend. I did not expect to have to do that as I thought HAProxy would only listen on the defined ports but it happened to me.


Log in to reply