HAProxy / SSL Offloading / OpenVPN / SSH
-
I'm looking for some HAProxy config guidance.
I'm looking to end up with traffic coming in to WAN on 443, the first frontend sends HTTPS traffic to a backend that forwards to a HTTP/HTTPS SSL Offloading front end on localhost:4443. The first front end has a default port for OpenVPN on localhost:443. The SSL Offloading front end then decrypts incoming traffic by cert, passes the option fordwarfor so Cloudflare client IP gets send with the traffic to the web server. I encrypt the outgoing traffic with a self signed ssl client cert to make things easier on the web server (NGINX).
I had the HTTPS Offloading fully functional with the domains decrypting and encrypting properly and getting sent to the web server. I'm having problems trying to add the new front end in HAProxy to get the OpenVPN working. I don't want to use OpenVPN as the main 443 server (using the port-sharing) to send HTTPS traffic to HAProxy because I'm using the port sharing feature on OpenVPN already to pass on SSH traffic. I did have an SSLH server running on the web server, but that meant all OpenVPN and SSH traffic was also routing through the web server, which slowed those connections down. That's why I'm trying to offload the work to the pfsense.
I thought I had the two front ends and the various back ends setup correctly, but I'm getting an error 503 Service Unavailable trying to connect to the domains. The OpenVPN is working and so is the SSH. The trouble is getting the HTTPS traffic to the right endpoint.
Here is my HAProxy config. I'm new to figuring out the HAProxy configurations.
# Automaticaly generated, dont edit manually. # Generated on: 2022-09-13 21:13 global maxconn 10000 stats socket /tmp/haproxy.socket level admin expose-fd listeners gid 80 nbproc 1 nbthread 1 hard-stop-after 15m chroot /tmp/haproxy_chroot daemon server-state-file /tmp/haproxy_server_state listen HAProxyLocalStats bind 127.0.0.1:2200 name localstats mode http stats enable stats admin if TRUE stats show-legends stats uri /haproxy/haproxy_stats.php?haproxystats=1 timeout client 5000 timeout connect 5000 timeout server 5000 frontend WAN_HTTP_Bypass bind <WAN IP>:80 name <WAN IP>:80 mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 default_backend Web-80_ipv4 frontend WAN_HTTPS_SSL_Offload bind 127.0.0.1:4443 name 127.0.0.1:4443 ssl crt-list /var/etc/haproxy/WAN_HTTPS_SSL_Offload.crt_list mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 acl Web_<domain 1> var(txn.txnhost) -m sub -i <domain 1> acl Web_<domain 2> var(txn.txnhost) -m sub -i <domain 2> acl Web_<domain 3> var(txn.txnhost) -m sub -i <domain 3> acl Web_<domain 4> var(txn.txnhost) -m sub -i <domain 4> acl Web_<domain 5> var(txn.txnhost) -m sub -i <domain 5> acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^direct\.<domain 2>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<domain 2>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<domain 2>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<domain 5>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<domain 5>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<domain 1>\.net(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<domain 1>\.net(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<domain 1>\.org(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<domain 1>\.org(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<domain 1>\.us(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<domain 1>\.us(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^mr<domain 4>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.mr<domain 4>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^mrdodge<domain 4>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.mrdodge<domain 4>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<domain 3>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<domain 3>\.com(:([0-9]){1,5})?$ http-request set-var(txn.txnhost) hdr(host) use_backend Web-443_ipv4 if Web_<domain 1> aclcrt_WAN_HTTPS_SSL_Offload use_backend Web-443_ipv4 if Web_<domain 2> aclcrt_WAN_HTTPS_SSL_Offload use_backend Web-443_ipv4 if Web_<domain 3> aclcrt_WAN_HTTPS_SSL_Offload use_backend Web-443_ipv4 if Web_<domain 4> aclcrt_WAN_HTTPS_SSL_Offload use_backend Web-443_ipv4 if Web_<domain 5> aclcrt_WAN_HTTPS_SSL_Offload frontend WAN_OpenVPN_Bypass bind <WAN IP>:443 name <WAN IP>:443 mode tcp log global timeout client 30000 tcp-request inspect-delay 5s acl Web_<domain 1> req.ssl_sni -m sub -i <domain 1> acl Web_<domain 2> req.ssl_sni -m end -i <domain 2>.com acl Web_<domain 3> req.ssl_sni -m end -i <domain 3>.com acl Web_<domain 4> req.ssl_sni -m sub -i <domain 4> acl Web_<domain 5> req.ssl_sni -m end -i <domain 5>.com tcp-request content accept if { req.ssl_hello_type 1 } use_backend Web-4443_ipvANY if Web_<domain 1> use_backend Web-4443_ipvANY if Web_<domain 2> use_backend Web-4443_ipvANY if Web_<domain 3> use_backend Web-4443_ipvANY if Web_<domain 4> use_backend Web-4443_ipvANY if Web_<domain 5> default_backend OpenVPN-TCP_ipvANY backend Web-80_ipv4 mode http id 10104 log global timeout connect 30000 timeout server 30000 retries 3 source ipv4@ usesrc clientip server Web-80 10.10.33.2:80 id 10103 backend Web-443_ipv4 mode http id 10102 log global timeout connect 30000 timeout server 30000 retries 3 source ipv4@ usesrc clientip server Web-443 10.10.33.2:443 id 10103 ssl verify none crt /var/etc/haproxy/server_clientcert_631f9f6f8fc2a.pem backend Web-4443_ipvANY mode tcp id 105 log global timeout connect 30000 timeout server 30000 retries 3 server localhost 127.0.0.1:4443 id 106 backend OpenVPN-TCP_ipvANY mode tcp id 100 log global timeout connect 30000 timeout server 30000 retries 3 server 127.0.0.1 127.0.0.1:443 id 101 -
I could also try and figure out a solution using SSLH if anyone has any recommendations for how to get SSLH to work on the pfsense with transparent working for the web server?
Still no luck passing the SSL traffic through the first TCP frontend to the second http/https frontend that handles SSL offloading.
-
For anyone else looking, I finally found a link that got me in the right direction. I now have my setup working. This is what helped me, from post 117557, https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/pfsense_2_3_haproxy_sni_plus_offloading_backends
WAN:443 -> TCP Mode Front End
TCP Mode Front End -> Redirect backend to HTTP/HTTPS Off Loading Front End
-> Default OpenVPN Backend (OpenVPN using port-sharing with ssh server)HTTP/HTTPS Off Loading Front End -> Multiple domain backends (host based), off loading SSL and sending plain text to secure VLAN NGINX Server
If anyone needs it, I can put up some images, just send me a message. Here is my haproxy config file:
# Automaticaly generated, dont edit manually. # Generated on: 2022-09-14 20:20 global maxconn 10000 stats socket /tmp/haproxy.socket level admin expose-fd listeners gid 80 nbproc 1 nbthread 1 hard-stop-after 15m chroot /tmp/haproxy_chroot daemon server-state-file /tmp/haproxy_server_state listen HAProxyLocalStats bind 127.0.0.1:2200 name localstats mode http stats enable stats admin if TRUE stats show-legends stats uri /haproxy/haproxy_stats.php?haproxystats=1 timeout client 5000 timeout connect 5000 timeout server 5000 frontend WAN_HTTP_Bypass bind <WAN IP>:80 name <WAN IP>:80 mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 default_backend Web-80_ipv4 frontend WAN_HTTPS_SSL_Offload bind 127.0.0.1:4443 name 127.0.0.1:4443 ssl crt-list /var/etc/haproxy/WAN_HTTPS_SSL_Offload.crt_list bind /tmp/haproxy_chroot/WAN_HTTPS_SSL_Offload.socket name unixsocket uid 80 accept-proxy ssl crt-list /var/etc/haproxy/WAN_HTTPS_SSL_Offload.crt_list mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 acl Web_<DOMAIN 1> var(txn.txnhost) -m sub -i <DOMAIN 1> acl Web_<DOMAIN 2> var(txn.txnhost) -m sub -i <DOMAIN 2> acl Web_<DOMAIN 3> var(txn.txnhost) -m sub -i <DOMAIN 3> acl Web_<DOMAIN 4> var(txn.txnhost) -m sub -i <DOMAIN 4> acl Web_<DOMAIN 5> var(txn.txnhost) -m sub -i <DOMAIN 5> acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^direct\.<DOMAIN 2>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<DOMAIN 2>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<DOMAIN 2>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<DOMAIN 5>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<DOMAIN 5>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<DOMAIN 1>\.net(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<DOMAIN 1>\.net(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<DOMAIN 1>\.org(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<DOMAIN 1>\.org(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<DOMAIN 1>\.us(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<DOMAIN 1>\.us(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^mr<DOMAIN 4>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.mr<DOMAIN 4>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^mrdodge<DOMAIN 4>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.mrdodge<DOMAIN 4>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^<DOMAIN 3>\.com(:([0-9]){1,5})?$ acl aclcrt_WAN_HTTPS_SSL_Offload var(txn.txnhost) -m reg -i ^www\.<DOMAIN 3>\.com(:([0-9]){1,5})?$ http-request set-var(txn.txnhost) hdr(host) use_backend Web-443_ipv4 if Web_<DOMAIN 1> aclcrt_WAN_HTTPS_SSL_Offload use_backend Web-443_ipv4 if Web_<DOMAIN 2> aclcrt_WAN_HTTPS_SSL_Offload use_backend Web-443_ipv4 if Web_<DOMAIN 3> aclcrt_WAN_HTTPS_SSL_Offload use_backend Web-443_ipv4 if Web_<DOMAIN 4> aclcrt_WAN_HTTPS_SSL_Offload use_backend Web-443_ipv4 if Web_<DOMAIN 5> aclcrt_WAN_HTTPS_SSL_Offload frontend WAN_OpenVPN_Bypass bind <WAN IP>:443 name <WAN IP>:443 mode tcp log global timeout client 30000 tcp-request inspect-delay 5s acl Web_<DOMAIN 1> req.ssl_sni -m sub -i <DOMAIN 1> acl Web_<DOMAIN 2> req.ssl_sni -m end -i <DOMAIN 2>.com acl Web_<DOMAIN 3> req.ssl_sni -m end -i <DOMAIN 3>.com acl Web_<DOMAIN 4> req.ssl_sni -m sub -i <DOMAIN 4> acl Web_<DOMAIN 5> req.ssl_sni -m end -i <DOMAIN 5>.com tcp-request content accept if { req.ssl_hello_type 1 } use_backend Web-4443_ipv4 if Web_<DOMAIN 1> use_backend Web-4443_ipv4 if Web_<DOMAIN 2> use_backend Web-4443_ipv4 if Web_<DOMAIN 3> use_backend Web-4443_ipv4 if Web_<DOMAIN 4> use_backend Web-4443_ipv4 if Web_<DOMAIN 5> default_backend OpenVPN-TCP_ipv4 backend Web-80_ipv4 mode http id 10104 log global timeout connect 30000 timeout server 30000 retries 3 source ipv4@ usesrc clientip server Web-80 <NGINX IP>:80 id 10103 backend Web-443_ipv4 mode http id 10102 log global timeout connect 30000 timeout server 30000 retries 3 source ipv4@ usesrc clientip server Web-443 <NGINX IP>:443 id 10103 backend Web-4443_ipv4 mode tcp id 10105 log global timeout connect 30000 timeout server 30000 retries 3 source ipv4@ usesrc clientip server localhost /WAN_HTTPS_SSL_Offload.socket send-proxy-v2-ssl-cn id 10106 check inter 5000 backend OpenVPN-TCP_ipv4 mode tcp id 10100 log global timeout connect 30000 timeout server 30000 retries 3 server 127.0.0.1 127.0.0.1:443 id 10101 -
S sgnoc referenced this topic on Sep 18, 2022, 8:35 PM
-
Hi there,
Can you post pfsense Haproxy screenshots for SSL passthrough?
Thanks
-
@deeztek Sorry for the delay, it took a little longer to get time to sit and screen shot these. I didn't snap an image, but the SSL_Offload_FrontEnd and piWeb-80 backends have the "Use Client-IP to connect to backend servers." option selected in the advanced section. Let me know if you need any other sections or anything else. Hope this helps! I also set up the OpenVPN to port share with my SSH server, so I have my WAN router doing the SSL offloading, passing decrypted traffic to my web server, the OpenVPN sharing the same 443 port, and SSH getting passed from the router to the OpenVPN server and then off to the SSH server also on port 443. Works great and haven't had any issues serving the multiple domains from my web server.
Front Ends:
Back Ends:
OpenVPN-TCP Back End:
SSL_Offload_FrontEnd Back End:
piWeb-80 Back End:
