Haproxy to nginx backend

thewismit

Hi folks,

I'm not sure how to track anything in the firewall so I can't give you much more info regarding the route tracing other than the fact that Cloudflare reports a 522 timeout to the a record pointing at the public IP associated with my WAN connection. And also that I've followd this guide to try and get this working https://gainanov.pro/eng-blog/linux/installing-haproxy-pfsense/

I can say that I've tested the connectivity to the FQDN using a custom NextDNS entry to the servers private IP and the connection works fine from inside.

Lastly, I'm not configuring Haproxy with a subdomain, as shown in the guide. I'm trying to resolve to domain.com.

I hope that makes sense. Can anyone help me out? Thank you.

Edit: I did forget to mention that at one point I was using what I assume are the equivalent of A records in the DNS resolver to point clients to FQDNs using the private IP of my server. They are still in place and not being used due to having installed the NextDNS app for pfsense. I'm not sure if they will interfere but they don't seem to.

PiBa

@thewismit
Google tels me: "Code 522 stands for 'Connection timed out', which occurs whenever the TCP handshake between the web server and Cloudflare fails."

So that means one of a few things:
1- either the request never reached your public-pfSense-ip
2- the firewall blocked the request
3- haproxy was not listening on the ip:port

for 3 you can see under diagnostics/sockets if haproxy is currently listening
for 2 you could check the firewall logs..
for 1 you would need to check with packetcapture or preferably tcpdump on a ssh console if the syn [S] packet from the connection request arrives on the wan interface

thewismit

@piba thank you.

www haproxy 70155 7 tcp4 10.8.48.229:443 :
I can't see any 443 traffic being blocked. I also don't know how to see what's being passed.
I'm fairly certain I don't understand how to do that but I'll try.

PiBa

@thewismit
that 10.8.48.229:443 doesnt not look like a public ip.. the 10.x.x.x. range are private ip's just like 192.168.x.x and 172.16.x.x.. is pfSense behind another router? does that perform natting and portforwarding and if so is the portforward configured there towards pfSense?

thewismit

@piba it's not my router if it is. It is connected to the fiber connection that my ISP installed. I do have a public IP registered with cloudflare using ddns and I suspect traffic is being forwarded somehow to pfsense as I can see tons of UDP and IPv6 traffic being blocked in the logs.

PiBa

@thewismit
ok.. well lets try the tcpdump. you know how to connect to pfSense with SSH? Note the interface name of your wan interface something like em0, rtl1, xn2, eth3 or something like that.. Then go to 8) Shell.

Then type this (replacing em0 with your actual wan interface name..):

tcpdump -i em0 dst host 10.8.48.229 and dst port 443 or src host 10.8.48.229 and src port 443

And then try to visit the webpage again.
Also try connecting from somewhere 'outside' to that IP:port directly bypassing a possible cloudflare config issue.

thewismit

@piba I don't think I did this right. The WAN interface is called mvneta0.4090 and while it seemed to monitor it, I had to ctrl-c to stop it.

I went to the web server URL from "inside" with proxy turned off on the A record.

[2.4.5-RELEASE] tcpdump -i mvneta0.4090 dst host 10.8.48.229 and dst port 443 or src host 10.8.48.229 and src port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mvneta0.4090, link-type EN10MB (Ethernet), capture size 262144 bytes

^C
0 packets captured
21383 packets received by filter
0 packets dropped by kernel

If you know what I did wrong, I'll try it again from the "outside".

thewismit

@thewismit said in Haproxy to nginx backend:

If you know what I did wrong, I'll try it again from the "outside".

I don't think this can be done. I don't know how traffic would be routed from the public IP to the 10.x.x.x address. I wonder if it was configured this way to prevent hosting on that public IP.

PiBa

@thewismit
I think you did the tcpdump correctly.. but it just didn't see any traffic arrive that matches the filter, which i think is the right filter.. Which would mean that no traffic arrived at pfSense/haproxy requesting the intended website.

I think the big question is what the ISP modem and network allows/supports.. Does it have a login page, or perhaps you can call the ISP and tell you want to configure port-forwarding maybe they have a explanation/webpage that tells how or perhaps that it cant be done at all.. Or perhaps they can configure the modem in 'bridge mode' for you, so pfSense would get the real public ip which would probably be the preferred configuration.. (you might need to reconfigure the pfSense WAN interface to get it to work again if they change it..)

What is the modem brand/type that the ISP provided, what country do live and which ISP / service do you use? Maybe i can find what it can and/or cannot do.

thewismit

@piba FYI https://forum.netgate.com/topic/159754/private-wan-address

thewismit

Hi @PiBa

So I have my public IP address now and I've configured haproxy as in the article above, exactly, except using my domain names and cloudflare DNS.

When I connect internally using internal DNS zone, everything works fine but when trying to come in through the WAN address, I get this SSL error: Error code: SSL_ERROR_NO_CYPHER_OVERLAP

Would you know if this is caused by something in the Cloudflare DNS or if it could be HAproxy?

Thank you!

PiBa

@thewismit
It might be haproxy, dns by itself shouldn't cause this error.. Unless if its pointing to a wrong/different server/(caching)proxy..

If DNS is pointing to the cloudflare 'proxy', then you need to make sure that they have the proper certificate and encryption cipher options to accept the connection from the browser.. If DNS is pointing directly to the WAN ip, then it has to be haproxy that is sending the wrong allowable ciphers. Perhaps you could try with SSLlabs to see if/what ciphers are currently shown when visiting your wanip and/or domainname.?

Can you share the haproxy.cfg file perhaps? (with obfuscated ip/domain names)

Can you maybe share the domainname and your public ip? Or send me a PM, maybe i can see something hinky.?.