HAProxy 502 bad gateway with Cloudflare Proxy
-
@te7 Thank you for the reply!
I followed the videos closely but it only works on my desktop, not on my other devices. I can ping the virtual ip and all domains just fine but get a timeout everything else then my desktop. I use a names for all my domains but this seems to work just fine. I shouldn't be able to ping them otherwise. I have the suspicion that some thing with my firewall rules is wrong. I will show them here now. Note that I use a bridge consisting of all my available interfaces aside from the WAN interface and also use fq_codel traffic shaping, which I could confirm is working probably.
Here are the WAN rules. The first two rules are for the virtual ip. The other three rules for opening ports to my services.
The rules for the virtual ip look like this:
The rules for the nat port forward of my services looks like this (the destination ip is the local ip of the service):
I use pfblockerng to allow all cloudflare ip spaces and let them update automatically. Otherwise the dns proxy did not work:
Next are the rules for the other interfaces. They just allow all kinds of traffic:
Sorry for all the images, but I want to be as thorough as possible with my data to make debugging this possible. If there is something missing, just let me know!
-
Here are images of my current haproxy front- and backend configs. The external address is the virtual ip I created.
Here the virtual ip and dns resolver config (the ip of the dns config is the virtual ip):
When I set the health check of the backend to basic, it gets recognized as up.
-
@klaussemmler
The only difference between your and mine setup that I can see is that you are using LAN bridge. I'm not familiar with bridging and from documentation it appears that it might cause issues with proxy.
"For services running on the firewall, bridging can be problematic. Features such as limiters, Captive Portal, and transparent proxies require special configuration and handling to work on bridged networks. Specifically, the bridge itself must be assigned and the only interface on the bridge with an IP address must be the assigned bridge. Also, in order for these functions to work, the IP address on the bridge must be the address used by clients as their gateway. These issues are discussed more in-depth in Bridging interoperability."
-
@te7 I read the documentation about this topic and I setup the bridge with its ip and all member interfaces have none. With what I am struggling a bit is the gateway remark. Where can I set this?
-
@te7 I read everything in the linked documentation thoroughly and reconfigured the interface assigment and assgned the bridge to the lan interface See the quick but tricky part. The bridge is also the only interface of all LAN Interfaces with an IP. All bridge members do not have an IP. I also always referred to the bridge, when the LAN Interface was requested in the configuration.
That still sadly did not fix my issues.
-
I noticed something else very odd:
I use KDE Connect on my Desktop and Smartphone and I cannot get them to connect with one another. KDE Connect uses the Ports 1716-1764. It gets more and more odd. Aside from that my Smartphone has normal internet connectivity and even the smb connection to my nas works. I am at a loss at the moment.
-
@klaussemmler
In general gateways are set up under System/Routing/Gatways
-
@te7 What should I configure here? As I understand it the gateway on the devices, for example my wifi access point should be the same ip as the ip of the bridge and that is the case. I checked it in the configuration interface of the wifi access point.
-
@klaussemmler so, jumping over from my thread to hopefully help you.
One huge problem I had (other than not recognizing that WAN is not internet ;) ) was that I had too many proxies working against each other. If you are working with cloudflare (only one I can speak about as it’s the only one I use) is to set cloudflare to dns forwarder only, do not proxy) doing that alone solved all of the fight I was having
-
@menethoran I should expand…
Cloud flare with proxy on is 1 proxy…
Your PfSense (I believe) is another (when you’re using haproxy) (2)
If you have a NAS that creates applications your working with, that’s ANOTHER proxy (3) (I use trueNAS personally)And 3 is too many proxies ;)
-
@menethoran can you elaborate on the WAN stuff a bit more? What had you changed?
Disabling the Cloudflare proxy does not seem to fix my issue with my services only working locally on one pc.
-
Okay, I disabled all the virtual ip stuff and the cloudflare proxy and the services are working again. I still would like to get that working ^^.
-
@klaussemmler
I would get rid of the bridge altogether and use switch. I don't see a reason not to use one. Switches are affordable and if you need managed switch you can get one from eBay.
Dell PowerConnect 2724 24 Port Gigabit Ethernet Managed Switch
-
@te7 This was just my missing experience with pfsense to try the LAN Bridge. I learned, that a dedicated switch is the way to go. Thanks for the advice! I will likely get a switch with multiple sfp+ ports as I have multiple devices with such ports. When I get one, I will come back to this topic, but this will take a bit of time. Here money is my limit ^^.
-
@menethoran said in HAProxy 502 bad gateway with Cloudflare Proxy:
@klaussemmler so, jumping over from my thread to hopefully help you.
One huge problem I had (other than not recognizing that WAN is not internet ;) ) was that I had too many proxies working against each other. If you are working with cloudflare (only one I can speak about as it’s the only one I use) is to set cloudflare to dns forwarder only, do not proxy) doing that alone solved all of the fight I was having
I don't think that disabling CloudFlare proxy is a sound advice. That will expose your IP address to the Internet and at the same time you will lose all of the benefits of using CloudFlare's cashing and protection against domain attacks. The whole point of using CloudFlare DNS proxy is hide your IP and let CloudFlare handle all bad guys out there trying to hack your web site or whatever you are hosting behind proxy.
See small example of CloudFlare firewall activity for my domain in the past 24hrs.
-
@te7 That is true. I will revisit this when I got my switch and see if it works better then. With the cloudflare proxy feature I always get timeouts when using my services. Pining them works fine though. I have neither an idea nor an solution for this problem yet.
-
@klaussemmler
That points to your pfSense rules and HAProxy settings. You are close and using switch will make it easier to troubleshoot everything.
-
@te7 I also tried the packet capture feature and get tcp traffic but it seems like it never reached the service defined in the haproxy backend. Without the cloudflare proxy, everything works, but this is not optimal. This could also be a funky issue with lan bridging and if that is the case the switch could fix that. It is fascinating how many tools pfsense provides for debugging out of the box. I do not understand all tools but I try to learn about them step by step.
-
I now tried some other things:
- Using the cloudflare origin certificate on the haproxy frontend and as webConfigurator certificate -> did not change anything.
- Setting the SSL/TLS encryption mode in cloudflare to flexible intsead of full or full (strict) -> did also not fix it. Without the cloudflare proxy, the full (strict) mode works fine.
These where some things I saw as suggestion to fix the problem. Sadly without success.
-
@klaussemmler Have you found an solution of this issue? I had the same setup and the same problem as you had.