HAProxy 502 bad gateway with Cloudflare Proxy
-
Hello Netgate community,
not long ago I build my own pfSense machine and it works great besides one thing.
I try to get HAProxy to work with the web domains of my cloudflare account, but it only works, when I disable the Proxy function for my a records (The image is from the cloudflare configuration interface with censored names and addresses).
I use cloudflare for dynamic dns and the domain management (I got my domains from there). I tried different settings in the cloudflare configuration panel from encryption modes to different tls version or not forcing https, but to no avail.
I also added the cloudflare ips as pass rules with pfblockerng to my firewall. This changed the error message from an 522 to an 502:
My HAProxy Config is as follows (also censored):
global maxconn 500 stats socket /tmp/haproxy.socket level admin expose-fd listeners uid 80 gid 80 nbproc 1 nbthread 1 hard-stop-after 15m chroot /tmp/haproxy_chroot daemon tune.ssl.default-dh-param 4096 server-state-file /tmp/haproxy_server_state listen HAProxyLocalStats bind 127.0.0.1:2200 name localstats mode http stats enable stats admin if TRUE stats show-legends stats uri /haproxy/haproxy_stats.php?haproxystats=1 timeout client 5000 timeout connect 5000 timeout server 5000 frontend services bind 1.2.3.4:443 name 1.2.3.4:443 ssl crt-list /var/etc/haproxy/services.crt_list mode http log global option socket-stats option http-keep-alive timeout client 30000 acl service1 var(txn.txnhost) -m sub -i service1.domain.tld acl service2 var(txn.txnhost) -m sub -i service2.domain.tld acl service3 var(txn.txnhost) -m sub -i service3.domain.tld acl aclcrt_services var(txn.txnhost) -m reg -i ^domain\.tld(:([0-9]){1,5})?$ acl aclcrt_services var(txn.txnhost) -m reg -i ^service1\.domain\.tld(:([0-9]){1,5})?$ acl aclcrt_services var(txn.txnhost) -m reg -i ^service2\.domain\.tld(:([0-9]){1,5})?$ acl aclcrt_services var(txn.txnhost) -m reg -i ^service3\.domain\.tld(:([0-9]){1,5})?$ http-request set-var(txn.txnhost) hdr(host) http-request deny if { req.hdr_cnt(content-length) gt 1 } http-response deny if { res.hdr_cnt(content-length) gt 1 } use_backend service1_ipvANY if service1 aclcrt_services use_backend service2_ipvANY if service2 aclcrt_services use_backend service3_ipvANY if service3 aclcrt_services use_backend service1_ipvANY if aclcrt_services backend service1_ipvANY mode http id 100 log global timeout connect 30000 timeout server 30000 retries 3 server service1 5.6.7.6:45678 id 101 check inter 1000 backend service2_ipvANY mode http id 102 log global timeout connect 30000 timeout server 30000 retries 3 server service2 5.6.7.7:9999 id 103 check inter 1000 backend service3_ipvANY mode http id 104 log global timeout connect 30000 timeout server 30000 retries 3 server service3 5.6.7.11:49998 id 105 check inter 1000I am currently out of ideas why the proxy feature of cloudflare does break my config. I hope someone here can help me to get this stuff working. If there is any missing info, I am happy to provide it.
Just as a notice: I followed the guide from lawrence systems to set this all up.
-
As I cannot edit my post anymore:
The used version of pfSense on my machine is 2.6.0.
The certificates where created with the acme plugin and they where added to my cloudflare domains. -
Good and bad news:.
The good news:
I could get the haproxy setup kinda working. From my linux desktop, all services can be called with the setup domains. These tutorials really helped me:
https://www.youtube.com/watch?v=FWodNSZXcXs
https://www.youtube.com/watch?v=KkL3QyYlNUIThe bad news:
This setup only works on my desktop. When I try the services on my smartphone or another desktop in my local network or LTE, I get a timeout. Internet access works fine on these devices.
Here is my current haproxy configuration (again with some stuff modified/redacted):
global maxconn 1000 stats socket /tmp/haproxy.socket level admin expose-fd listeners uid 80 gid 80 nbproc 1 nbthread 1 hard-stop-after 15m chroot /tmp/haproxy_chroot daemon tune.ssl.default-dh-param 2048 server-state-file /tmp/haproxy_server_state listen HAProxyLocalStats bind 127.0.0.1:2200 name localstats mode http stats enable stats admin if TRUE stats show-legends stats uri /haproxy/haproxy_stats.php?haproxystats=1 timeout client 5000 timeout connect 5000 timeout server 5000 frontend http_redirect bind 1.2.3.252:80 name 1.2.3.252:80 mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 http-request redirect scheme https unless { ssl_fc } http-request deny if { req.hdr_cnt(content-length) gt 1 } http-response deny if { res.hdr_cnt(content-length) gt 1 } frontend https_443-copy bind 1.2.3.252:443 name 1.2.3.252:443 ssl crt-list /var/etc/haproxy/https_443-copy.crt_list mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 acl service1 var(txn.txnhost) -m beg -i service1.domain.tld acl service2 var(txn.txnhost) -m beg -i service2.domain.tld acl service3 var(txn.txnhost) -m beg -i service3.domain.tld acl aclcrt_https_443-copy var(txn.txnhost) -m reg -i ^spelunker\.tech(:([0-9]){1,5})?$ acl aclcrt_https_443-copy var(txn.txnhost) -m reg -i ^service1\.spelunker\.tech(:([0-9]){1,5})?$ acl aclcrt_https_443-copy var(txn.txnhost) -m reg -i ^service2\.spelunker\.tech(:([0-9]){1,5})?$ acl aclcrt_https_443-copy var(txn.txnhost) -m reg -i ^service3\.spelunker\.tech(:([0-9]){1,5})?$ http-request set-var(txn.txnhost) hdr(host) http-request deny if { req.hdr_cnt(content-length) gt 1 } http-response deny if { res.hdr_cnt(content-length) gt 1 } use_backend service1_ipvANY if service1 aclcrt_https_443-copy use_backend service2_ipvANY if service2 aclcrt_https_443-copy use_backend service3_ipvANY if service3 aclcrt_https_443-copy backend service1_ipvANY mode http id 100 log global timeout connect 30000 timeout server 30000 retries 3 server service1 1.2.3.4:25565 id 101 backend service2_ipvANY mode http id 102 log global timeout connect 30000 timeout server 30000 retries 3 server service2 1.2.3.5:7777 id 103 backend service3_ipvANY mode http id 104 log global timeout connect 30000 timeout server 30000 retries 3 server service3 1.2.3.6:48886 id 105Seems like I am close to make it work
. Again, any input is appreciated! -
System referenced this topic on
-
Try to follow videos step by step. First, make sure one that domain is working then add some more.
Did you use virtual IP, change firewall NAT rules and added Host Overrides under DNS Resolver?
If you open Diagnostics ==> HAProxy Stats do you see service as UP (green)?
My CloudFlare subdomains are type CNAME. I don't know if that is going to make any difference.
-
@te7 Thank you for the reply!
I followed the videos closely but it only works on my desktop, not on my other devices. I can ping the virtual ip and all domains just fine but get a timeout everything else then my desktop. I use a names for all my domains but this seems to work just fine. I shouldn't be able to ping them otherwise. I have the suspicion that some thing with my firewall rules is wrong. I will show them here now. Note that I use a bridge consisting of all my available interfaces aside from the WAN interface and also use fq_codel traffic shaping, which I could confirm is working probably.
Here are the WAN rules. The first two rules are for the virtual ip. The other three rules for opening ports to my services.
The rules for the virtual ip look like this:
The rules for the nat port forward of my services looks like this (the destination ip is the local ip of the service):
I use pfblockerng to allow all cloudflare ip spaces and let them update automatically. Otherwise the dns proxy did not work:
Next are the rules for the other interfaces. They just allow all kinds of traffic:
Sorry for all the images, but I want to be as thorough as possible with my data to make debugging this possible. If there is something missing, just let me know!
-
Here are images of my current haproxy front- and backend configs. The external address is the virtual ip I created.
Here the virtual ip and dns resolver config (the ip of the dns config is the virtual ip):
When I set the health check of the backend to basic, it gets recognized as up.
-
@klaussemmler
The only difference between your and mine setup that I can see is that you are using LAN bridge. I'm not familiar with bridging and from documentation it appears that it might cause issues with proxy.
"For services running on the firewall, bridging can be problematic. Features such as limiters, Captive Portal, and transparent proxies require special configuration and handling to work on bridged networks. Specifically, the bridge itself must be assigned and the only interface on the bridge with an IP address must be the assigned bridge. Also, in order for these functions to work, the IP address on the bridge must be the address used by clients as their gateway. These issues are discussed more in-depth in Bridging interoperability."
-
@te7 I read the documentation about this topic and I setup the bridge with its ip and all member interfaces have none. With what I am struggling a bit is the gateway remark. Where can I set this?
-
@te7 I read everything in the linked documentation thoroughly and reconfigured the interface assigment and assgned the bridge to the lan interface See the quick but tricky part. The bridge is also the only interface of all LAN Interfaces with an IP. All bridge members do not have an IP. I also always referred to the bridge, when the LAN Interface was requested in the configuration.
That still sadly did not fix my issues.
-
I noticed something else very odd:
I use KDE Connect on my Desktop and Smartphone and I cannot get them to connect with one another. KDE Connect uses the Ports 1716-1764. It gets more and more odd. Aside from that my Smartphone has normal internet connectivity and even the smb connection to my nas works. I am at a loss at the moment.
-
@klaussemmler
In general gateways are set up under System/Routing/Gatways
-
@te7 What should I configure here? As I understand it the gateway on the devices, for example my wifi access point should be the same ip as the ip of the bridge and that is the case. I checked it in the configuration interface of the wifi access point.
-
@klaussemmler so, jumping over from my thread to hopefully help you.
One huge problem I had (other than not recognizing that WAN is not internet ;) ) was that I had too many proxies working against each other. If you are working with cloudflare (only one I can speak about as it’s the only one I use) is to set cloudflare to dns forwarder only, do not proxy) doing that alone solved all of the fight I was having
-
@menethoran I should expand…
Cloud flare with proxy on is 1 proxy…
Your PfSense (I believe) is another (when you’re using haproxy) (2)
If you have a NAS that creates applications your working with, that’s ANOTHER proxy (3) (I use trueNAS personally)And 3 is too many proxies ;)
-
@menethoran can you elaborate on the WAN stuff a bit more? What had you changed?
Disabling the Cloudflare proxy does not seem to fix my issue with my services only working locally on one pc.
-
Okay, I disabled all the virtual ip stuff and the cloudflare proxy and the services are working again. I still would like to get that working ^^.
-
@klaussemmler
I would get rid of the bridge altogether and use switch. I don't see a reason not to use one. Switches are affordable and if you need managed switch you can get one from eBay.
Dell PowerConnect 2724 24 Port Gigabit Ethernet Managed Switch
-
@te7 This was just my missing experience with pfsense to try the LAN Bridge. I learned, that a dedicated switch is the way to go. Thanks for the advice! I will likely get a switch with multiple sfp+ ports as I have multiple devices with such ports. When I get one, I will come back to this topic, but this will take a bit of time. Here money is my limit ^^.
-
@menethoran said in HAProxy 502 bad gateway with Cloudflare Proxy:
@klaussemmler so, jumping over from my thread to hopefully help you.
One huge problem I had (other than not recognizing that WAN is not internet ;) ) was that I had too many proxies working against each other. If you are working with cloudflare (only one I can speak about as it’s the only one I use) is to set cloudflare to dns forwarder only, do not proxy) doing that alone solved all of the fight I was having
I don't think that disabling CloudFlare proxy is a sound advice. That will expose your IP address to the Internet and at the same time you will lose all of the benefits of using CloudFlare's cashing and protection against domain attacks. The whole point of using CloudFlare DNS proxy is hide your IP and let CloudFlare handle all bad guys out there trying to hack your web site or whatever you are hosting behind proxy.
See small example of CloudFlare firewall activity for my domain in the past 24hrs.
-
@te7 That is true. I will revisit this when I got my switch and see if it works better then. With the cloudflare proxy feature I always get timeouts when using my services. Pining them works fine though. I have neither an idea nor an solution for this problem yet.