Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    HAProxy / SSL Offloading / OpenVPN / SSH

    Scheduled Pinned Locked Moved Cache/Proxy
    5 Posts 2 Posters 2.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      sgnoc
      last edited by

      I'm looking for some HAProxy config guidance.

      I'm looking to end up with traffic coming in to WAN on 443, the first frontend sends HTTPS traffic to a backend that forwards to a HTTP/HTTPS SSL Offloading front end on localhost:4443. The first front end has a default port for OpenVPN on localhost:443. The SSL Offloading front end then decrypts incoming traffic by cert, passes the option fordwarfor so Cloudflare client IP gets send with the traffic to the web server. I encrypt the outgoing traffic with a self signed ssl client cert to make things easier on the web server (NGINX).

      I had the HTTPS Offloading fully functional with the domains decrypting and encrypting properly and getting sent to the web server. I'm having problems trying to add the new front end in HAProxy to get the OpenVPN working. I don't want to use OpenVPN as the main 443 server (using the port-sharing) to send HTTPS traffic to HAProxy because I'm using the port sharing feature on OpenVPN already to pass on SSH traffic. I did have an SSLH server running on the web server, but that meant all OpenVPN and SSH traffic was also routing through the web server, which slowed those connections down. That's why I'm trying to offload the work to the pfsense.

      I thought I had the two front ends and the various back ends setup correctly, but I'm getting an error 503 Service Unavailable trying to connect to the domains. The OpenVPN is working and so is the SSH. The trouble is getting the HTTPS traffic to the right endpoint.

      Here is my HAProxy config. I'm new to figuring out the HAProxy configurations.

      # Automaticaly generated, dont edit manually.
# Generated on: 2022-09-13 21:13
global
	maxconn			10000
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend WAN_HTTP_Bypass
	bind			<WAN IP>:80 name <WAN IP>:80   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	default_backend Web-80_ipv4

frontend WAN_HTTPS_SSL_Offload
	bind			127.0.0.1:4443 name 127.0.0.1:4443   ssl crt-list /var/etc/haproxy/WAN_HTTPS_SSL_Offload.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			Web_<domain 1>	var(txn.txnhost) -m sub -i <domain 1>
	acl			Web_<domain 2>	var(txn.txnhost) -m sub -i <domain 2>
	acl			Web_<domain 3>	var(txn.txnhost) -m sub -i <domain 3>
	acl			Web_<domain 4>	var(txn.txnhost) -m sub -i <domain 4>
	acl			Web_<domain 5>	var(txn.txnhost) -m sub -i <domain 5>
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^direct\.<domain 2>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<domain 2>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<domain 2>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<domain 5>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<domain 5>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<domain 1>\.net(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<domain 1>\.net(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<domain 1>\.org(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<domain 1>\.org(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<domain 1>\.us(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<domain 1>\.us(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^mr<domain 4>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.mr<domain 4>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^mrdodge<domain 4>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.mrdodge<domain 4>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<domain 3>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<domain 3>\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend Web-443_ipv4  if  Web_<domain 1> aclcrt_WAN_HTTPS_SSL_Offload
	use_backend Web-443_ipv4  if  Web_<domain 2> aclcrt_WAN_HTTPS_SSL_Offload
	use_backend Web-443_ipv4  if  Web_<domain 3> aclcrt_WAN_HTTPS_SSL_Offload
	use_backend Web-443_ipv4  if  Web_<domain 4> aclcrt_WAN_HTTPS_SSL_Offload
	use_backend Web-443_ipv4  if  Web_<domain 5> aclcrt_WAN_HTTPS_SSL_Offload

frontend WAN_OpenVPN_Bypass
	bind			<WAN IP>:443 name <WAN IP>:443   
	mode			tcp
	log			global
	timeout client		30000
	tcp-request inspect-delay	5s
	acl			Web_<domain 1>	req.ssl_sni -m sub -i <domain 1>
	acl			Web_<domain 2>	req.ssl_sni -m end -i <domain 2>.com
	acl			Web_<domain 3>	req.ssl_sni -m end -i <domain 3>.com
	acl			Web_<domain 4>	req.ssl_sni -m sub -i <domain 4>
	acl			Web_<domain 5>	req.ssl_sni -m end -i <domain 5>.com
	tcp-request content accept if { req.ssl_hello_type 1 }
	use_backend Web-4443_ipvANY  if  Web_<domain 1> 
	use_backend Web-4443_ipvANY  if  Web_<domain 2> 
	use_backend Web-4443_ipvANY  if  Web_<domain 3> 
	use_backend Web-4443_ipvANY  if  Web_<domain 4> 
	use_backend Web-4443_ipvANY  if  Web_<domain 5> 
	default_backend OpenVPN-TCP_ipvANY

backend Web-80_ipv4
	mode			http
	id			10104
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	server			Web-80 10.10.33.2:80 id 10103  

backend Web-443_ipv4
	mode			http
	id			10102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	server			Web-443 10.10.33.2:443 id 10103 ssl  verify none crt /var/etc/haproxy/server_clientcert_631f9f6f8fc2a.pem 

backend Web-4443_ipvANY
	mode			tcp
	id			105
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			localhost 127.0.0.1:4443 id 106  

backend OpenVPN-TCP_ipvANY
	mode			tcp
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			127.0.0.1 127.0.0.1:443 id 101
      1 Reply Last reply Reply Quote 0
      • S
        sgnoc
        last edited by

        I could also try and figure out a solution using SSLH if anyone has any recommendations for how to get SSLH to work on the pfsense with transparent working for the web server?

        Still no luck passing the SSL traffic through the first TCP frontend to the second http/https frontend that handles SSL offloading.

        1 Reply Last reply Reply Quote 0
        • S
          sgnoc
          last edited by

          For anyone else looking, I finally found a link that got me in the right direction. I now have my setup working. This is what helped me, from post 117557, https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/pfsense_2_3_haproxy_sni_plus_offloading_backends

          WAN:443 -> TCP Mode Front End

          TCP Mode Front End -> Redirect backend to HTTP/HTTPS Off Loading Front End
          -> Default OpenVPN Backend (OpenVPN using port-sharing with ssh server)

          HTTP/HTTPS Off Loading Front End -> Multiple domain backends (host based), off loading SSL and sending plain text to secure VLAN NGINX Server

          If anyone needs it, I can put up some images, just send me a message. Here is my haproxy config file:

          # Automaticaly generated, dont edit manually.
# Generated on: 2022-09-14 20:20
global
	maxconn			10000
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend WAN_HTTP_Bypass
	bind			<WAN IP>:80 name <WAN IP>:80   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	default_backend Web-80_ipv4

frontend WAN_HTTPS_SSL_Offload
	bind			127.0.0.1:4443 name 127.0.0.1:4443   ssl crt-list /var/etc/haproxy/WAN_HTTPS_SSL_Offload.crt_list  
	bind /tmp/haproxy_chroot/WAN_HTTPS_SSL_Offload.socket name unixsocket uid 80 accept-proxy   ssl crt-list /var/etc/haproxy/WAN_HTTPS_SSL_Offload.crt_list 
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			Web_<DOMAIN 1>	var(txn.txnhost) -m sub -i <DOMAIN 1>
	acl			Web_<DOMAIN 2>	var(txn.txnhost) -m sub -i <DOMAIN 2>
	acl			Web_<DOMAIN 3>	var(txn.txnhost) -m sub -i <DOMAIN 3>
	acl			Web_<DOMAIN 4>	var(txn.txnhost) -m sub -i <DOMAIN 4>
	acl			Web_<DOMAIN 5>	var(txn.txnhost) -m sub -i <DOMAIN 5>
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^direct\.<DOMAIN 2>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<DOMAIN 2>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<DOMAIN 2>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<DOMAIN 5>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<DOMAIN 5>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<DOMAIN 1>\.net(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<DOMAIN 1>\.net(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<DOMAIN 1>\.org(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<DOMAIN 1>\.org(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<DOMAIN 1>\.us(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<DOMAIN 1>\.us(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^mr<DOMAIN 4>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.mr<DOMAIN 4>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^mrdodge<DOMAIN 4>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.mrdodge<DOMAIN 4>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^<DOMAIN 3>\.com(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS_SSL_Offload	var(txn.txnhost) -m reg -i ^www\.<DOMAIN 3>\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend Web-443_ipv4  if  Web_<DOMAIN 1> aclcrt_WAN_HTTPS_SSL_Offload
	use_backend Web-443_ipv4  if  Web_<DOMAIN 2> aclcrt_WAN_HTTPS_SSL_Offload
	use_backend Web-443_ipv4  if  Web_<DOMAIN 3> aclcrt_WAN_HTTPS_SSL_Offload
	use_backend Web-443_ipv4  if  Web_<DOMAIN 4> aclcrt_WAN_HTTPS_SSL_Offload
	use_backend Web-443_ipv4  if  Web_<DOMAIN 5> aclcrt_WAN_HTTPS_SSL_Offload

frontend WAN_OpenVPN_Bypass
	bind			<WAN IP>:443 name <WAN IP>:443   
	mode			tcp
	log			global
	timeout client		30000
	tcp-request inspect-delay	5s
	acl			Web_<DOMAIN 1>	req.ssl_sni -m sub -i <DOMAIN 1>
	acl			Web_<DOMAIN 2>	req.ssl_sni -m end -i <DOMAIN 2>.com
	acl			Web_<DOMAIN 3>	req.ssl_sni -m end -i <DOMAIN 3>.com
	acl			Web_<DOMAIN 4>	req.ssl_sni -m sub -i <DOMAIN 4>
	acl			Web_<DOMAIN 5>	req.ssl_sni -m end -i <DOMAIN 5>.com
	tcp-request content accept if { req.ssl_hello_type 1 }
	use_backend Web-4443_ipv4  if  Web_<DOMAIN 1> 
	use_backend Web-4443_ipv4  if  Web_<DOMAIN 2> 
	use_backend Web-4443_ipv4  if  Web_<DOMAIN 3> 
	use_backend Web-4443_ipv4  if  Web_<DOMAIN 4> 
	use_backend Web-4443_ipv4  if  Web_<DOMAIN 5> 
	default_backend OpenVPN-TCP_ipv4

backend Web-80_ipv4
	mode			http
	id			10104
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	server			Web-80 <NGINX IP>:80 id 10103  

backend Web-443_ipv4
	mode			http
	id			10102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	server			Web-443 <NGINX IP>:443 id 10103  

backend Web-4443_ipv4
	mode			tcp
	id			10105
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	server			localhost /WAN_HTTPS_SSL_Offload.socket send-proxy-v2-ssl-cn id 10106 check inter 5000  

backend OpenVPN-TCP_ipv4
	mode			tcp
	id			10100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			127.0.0.1 127.0.0.1:443 id 10101
          1 Reply Last reply Reply Quote 0
          • S sgnoc referenced this topic on
          • D
            deeztek
            last edited by

            Hi there,

            Can you post pfsense Haproxy screenshots for SSL passthrough?

            Thanks

            S 1 Reply Last reply Reply Quote 0
            • S
              sgnoc @deeztek
              last edited by sgnoc

              @deeztek Sorry for the delay, it took a little longer to get time to sit and screen shot these. I didn't snap an image, but the SSL_Offload_FrontEnd and piWeb-80 backends have the "Use Client-IP to connect to backend servers." option selected in the advanced section. Let me know if you need any other sections or anything else. Hope this helps! I also set up the OpenVPN to port share with my SSH server, so I have my WAN router doing the SSL offloading, passing decrypted traffic to my web server, the OpenVPN sharing the same 443 port, and SSH getting passed from the router to the OpenVPN server and then off to the SSH server also on port 443. Works great and haven't had any issues serving the multiple domains from my web server.

              Front Ends:
              Frontends.jpg

              Back Ends:
              Backends.jpg

              OpenVPN-TCP Back End:
              Backend1.jpg

              SSL_Offload_FrontEnd Back End:
              Backend2.jpg

              piWeb-80 Back End:
              Backend3.jpg

              1 Reply Last reply Reply Quote 0
              • First post
                Last post
              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.