HAProxy Empty Responses

mhanding · Mar 21, 2020, 6:01 PM

Apologies if this has been posted elsewhere before, but I am having trouble finding solid guidelines for setting up HAProxy. Below is my haproxy.cfg that is generated by PFSense after configuring everything. HAProxy seems to be accepting the traffic, but only sends back empty responses immediately. Does anyone have any thoughts on what I configured incorrectly?

global
maxconn 10000
stats socket /tmp/haproxy.socket level admin
gid 80
nbproc 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state

frontend Front-HTTP
bind <my_public_IP>:80 name <my_public_IP>:80
mode http
log global
option http-keep-alive
timeout client 30000
acl httpRedirectACL var(txn.txnhost) -m str -i drive.mydomain.com
acl httpRedirectACL var(txn.txnhost) -m str -i mydomain.com
acl httpRedirectACL var(txn.txnhost) -m str -i www.mydomain.com
http-request set-var(txn.txnhost) hdr(host)
http-request redirect scheme https if httpRedirectACL

frontend Front-HTTPS-merged
bind <my_public_IP>:443 name <my_public_IP>:443 ssl crt-list /var/etc/haproxy/Front-HTTPS.crt_list
mode tcp
log global
timeout client 30000
tcp-request inspect-delay 5s
acl drivename req.ssl_sni -i drive.mydomain.com
acl wordpressACL req.ssl_sni -i mydomain.com
acl wordpressACL req.ssl_sni -i www.mydomain.com
tcp-request content accept if { req.ssl_hello_type 1 }
use_backend drive_ipv4 if drivename
use_backend wordpress_ipv4 if wordpressACL
default_backend drive_ipv4
default_backend wordpress_ipv4

backend drive_ipv4
mode tcp
id 10100
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4@ usesrc clientip
option ssl-hello-chk
server drive 192.168.1.11:443 id 10101 ssl check inter 1000 ca-file /var/etc/haproxy/ca_5e714af3e9f3a.pem verifyhost drive.mydomain.com

backend wordpress_ipv4
mode tcp
id 10102
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4@ usesrc clientip
option ssl-hello-chk
server wordpress 192.168.1.12:443 id 10103 ssl check inter 1000 verify none

PiBa · Mar 21, 2020, 9:05 PM

@mhanding
I would change the frontend mode to 'HTTP'. Which provides better statistics/and diagnostics for http traffic. (will then need to create new acl's based on host header)
Perhaps also disable the transparent-client-ip feature.(at least for testing)
Are the servers 'up' according to the stats page?

mhanding · Mar 21, 2020, 9:45 PM

Hi @PiBa, thanks for the response. Excuse if I am ignorant about the settings, I have only started to dabble in HAProxy yesterday. Would changing the front end so it is HTTP affect HTTPS traffic? One of the servers is a Nextcloud server and I would definitely prefer that it runs over HTTPS.

Thanks for the tip about the stats page, I had not yet set it up and not realized its usefulness yet. Turns out my attempt to setup SSL checks caused the servers to show as down so the traffic was not being forwarded. Once I removed the check it began forwarding the traffic without needing to make any adjustments to the frontend.

Now my next challenge popped up: it looks like my check for mydomain.com is overwriting my check for drive.mydomain.com. I will start looking into that; do you have any suggestions for this issue?

PiBa · Mar 21, 2020, 9:57 PM

@mhanding
As you have a certificate on the frontend, the https-traffic is being decrypted by haproxy. After which it basically is plain http traffic, so mode http would be the best fit, it can then inspect/modify headers, count separate http requests, send http errors if a backend is down, provide better logging. As you can see it has several advantages..

As for the 'wrong' backend being selected, this might be a sideeffect of the use of SNI and perhaps even a single certificate valid for both names? This would be solved as well by checking the 'Host' header (not possible in tcp-mode). Another thing is that you seem to have selected a 'default backend' in multiple frontend, i would try and remove both and only use acl's+actions for now, and later add a single default, that way its easier to say if/what acl matched.

mhanding · Mar 21, 2020, 10:30 PM

@PiBa
On my backend, I have both servers checked for Encrypt(SSL); wouldn't this re-encrypt the traffic after HAProxy decrypts for ACL checks? If that checkbox/setting does not actually re-encrypt the traffic, then I can definitely see your point in having the mode HTTP. However, I do not see a "mode" option for the frontend other than the type, which removes certificate options. I am assuming you are not mentioning removing the HTTP redirect; could you be more specific about the option you are suggesting changes?

After some testing, I actually determined it looks like my ACLs are not actually doing anything; potentially because of SNI? The forwarded backend seems to only change when I change the default backend and returns an empty response when no default is selected. The expression I am using in the ACLs is "Server Name Indication TLS extension matches:". I pasted an updated set of settings below.

# Automaticaly generated, dont edit manually.

Generated on: 2020-03-22 05:15

global
maxconn 10000
stats socket /tmp/haproxy.socket level admin
uid 80
gid 80
nbproc 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend Front-HTTP
bind <my_public_ip>:80 name <my_public_ip>:80
bind 10.1.1.1:80 name 10.1.1.1:80
mode http
log global
option http-keep-alive
timeout client 30000
acl httpRedirectACL var(txn.txnhost) -m str -i drive.mydomain.com
acl httpRedirectACL var(txn.txnhost) -m str -i mydomain.com
acl httpRedirectACL var(txn.txnhost) -m str -i www.mydomain.com
http-request set-var(txn.txnhost) hdr(host)
http-request redirect scheme https if httpRedirectACL

frontend Front-HTTPS
bind <my_public_ip>:443 name <my_public_ip>:443 ssl crt-list /var/etc/haproxy/Front-HTTPS.crt_list
bind 10.1.1.1:443 name 10.1.1.1:443 ssl crt-list /var/etc/haproxy/Front-HTTPS.crt_list
mode tcp
log global
timeout client 30000
tcp-request inspect-delay 5s
acl driveACL req.ssl_sni -i drive.mydomain.com
acl wordpressACL req.ssl_sni -i www.mydomain.com
acl wordpressACL req.ssl_sni -i mydomain.com
tcp-request content accept if { req.ssl_hello_type 1 }
use_backend drive_ipvANY if driveACL
use_backend wordpress_ipvANY if wordpressACL
default_backend drive_ipvANY

backend drive_ipvANY
mode tcp
id 100
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server drive 192.168.1.11:443 id 101 ssl check inter 1000 verify none

backend wordpress_ipvANY
mode tcp
id 102
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server wordpress 192.168.1.12:443 id 103 ssl check inter 1000 verify none

I also found that due to VLAN seperation, HAProxy was interfering with the interVLAN traffic so you will notice I have another bound interface on the frontend.

PiBa · Mar 21, 2020, 10:54 PM

@mhanding
Yes the 'encrypt(ssl)' does re-encrypt traffic. Also in the config you can see it shows the 'ssl' keyword behind the servers. But for haproxy 'in memory' processing it can use mode http.
The change i propose is for the type parameter of the 'Front-HTTPS' frontend to change that to 'http / https(offloading)'. (and yes due to the re-encryption before sending to the backend there is no 'offloading' regarding the ssl cpu usage) it will allow usage of different acl's, it doesn't remove certificate options for 'offloading' it does remove the checks for SNI values as checking for the host header is actually better indicator of what the end-user really requested.

As for the inter-vlan-communication issue, that was likely caused by the 'transparent-client-ip' feature you have disabled in latest config.

mhanding · Mar 22, 2020, 12:46 AM

@PiBa
Awesome! Thank you again @PiBa and apologies again for the ignorance on my part. Your insight was extremely helpful and the proxy is now behaving as I had hoped.

Just a note in case anyone wonders, I tried disabling the transparent IP and that did not help with the interVLAN traffic. The problem only affected the ports defined in HAProxy (80,443) and did not resolve until I added the binded interface in the frontend. I did not expect to have to do that as I thought HAProxy would only listen on the defined ports but it happened to me.