Please help setup HAProxy with name-based virtual hosts
-
I have a Synology running WebStation with name-based virtual hosts which I believe are name-based Apache virtual hosts. For simplicity let's say I have the following (sub)domains as virtual hosts:
111.domain.com
222.domain.com
333.domain.comWhen I was port-forwarding to Synology its web server worked. The Synology handled Letsencrypt certificates. Now the certificates are on my pfSense and I am struggling to setup HAProxy so the Synology web server can work the same way.
I created an Address+Port backend pointing to Synology port 443 without SSL checks (tried 80, too). Very simple.
I created a frontend with SSL offloading pointing to the correct certificate, with an ACL Host ends with: domain.com and an action to use the above backend. I checked the Use "forwardfor" option.
This mostly works, i.e., going to 111.domain.com or 222.domain.com etc. loads the correct pages but I often get 500 Internal Server Error it seems only (or mostly) on POST. On simple pages I get intermittent 500 errors, if I reload I can get the correct page. On more complex pages I always get 500 when it comes to POST.
Can anyone advise how to properly set it up? I am not finding advice or guides by googling. I thought about changing to port-based hosts but I would have to create backends and fronends for all subdomains. I think this universal frontend for *.domain.com should work but it's not.
P.S. For the subdomain with Matomo I get "The page isn’t redirecting properly".
-
@pfpv said in Please help setup HAProxy with name-based virtual hosts:
111.domain.com
222.domain.com
333.domain.comAre all websites running in the root of the respective virtual server, no virtual directory?
Find out, where the 500 error is coming from. It's not implicitly from HAproxy. To be honest, I think, I haven't never seen it from HAproxy.
If it comes from the backend, you should see dedicated entries in its log.
In HAproxy you can enhance the log level to get closer to what's wrong.I thought about changing to port-based hosts but I would have to create backends and fronends for all subdomains.
If you just forward all requests for *.domain.com to the backend server to the same IP and port, the backend has to do the determination, to which virtual the packet belongs to.
It does this on host header bases. HAproxy doesn't change anything in the host header, as long as you didn't create a rule for it. -
@viragomann said in Please help setup HAProxy with name-based virtual hosts:
@pfpv said in Please help setup HAProxy with name-based virtual hosts:
111.domain.com
222.domain.com
333.domain.comAre all websites running in the root of the respective virtual server, no virtual directory?
I am not sure if it is about semantics but the server is not virtual, hosts are virtual. Like here: https://httpd.apache.org/docs/2.4/vhosts/examples.html
The directories for each hosts are defined as "Document root" and they are not virtual.Find out, where the 500 error is coming from. It's not implicitly from HAproxy. To be honest, I think, I haven't never seen it from HAproxy.
If it comes from the backend, you should see dedicated entries in its log.
I didn't think I could be getting this 500 error from HAProxy but you prompted me to check. It does indeed come from HAProxy! In the backend logs there are no errors but I see the following:
"[200] GET (path) HTTP/1.1 from - [bytes sent : 224]"
But then after POST I see this:
"[499] POST (path) HTTP/1.1 from (URL)(path) [bytes sent : 0]"Then in HAProxy stats, when I mouse over my backend Sessions Total I see increase in Internal error counts after every 500 server error. [Update] I see internal errors in the frontend, too.
That 499 status code is from nginx, though. "An Nginx HTTP server extension. This code is introduced to log the case when the connection is closed by client while HTTP server is processing its request, making server unable to send the HTTP header back."
I believe Synology runs nginx on top of Apache for security reasons. I believe it shouldn't matter as it shows that HAProxy closes the connection before the server can respond.
In HAproxy you can enhance the log level to get closer to what's wrong.
Unfortunately, I haven't found a way. There are settings for externalScratch that. I just read the instructions and will activate logging on pfSense. I will update with what I see there.I thought about changing to port-based hosts but I would have to create backends and fronends for all subdomains.
If you just forward all requests for *.domain.com to the backend server to the same IP and port, the backend has to do the determination, to which virtual the packet belongs to.
It does this on host header bases. HAproxy doesn't change anything in the host header, as long as you didn't create a rule for it.That's what I thought. I checked the Use "forwardfor" option, though. I suppose it shouldn't hurt?
-
After I entered /var/run/log into the Remote syslog host field in Settings as instructed I see entries in Status/Package Logs/haproxy-devel but they are only "Connect from IP_A to IP_B (my WAN)". No other info. I tried Debugging log level and others, Syslog facility 'local0' and 'syslog' but I see no other info in the logs. Is there any way to obtain more informative logs?
-
I noticed another peculiar thing. I set up my backend on port 443, Encrypt(SSL) - yes and SSL checks - no. But when I hover over my backend in the HAProxy stats it shows IP:80, not 443. I re-did the backend taking precautions described here: HAProxy backend port changes are not applied but it still shows port 80, not 443. I have two other backends on non-conventional ports and they show correct ports when hovered over.
My web server responds on both 443 and 80, so it shouldn't be a problem (and it works until POST) but I wonder why the hover over shows wrong port.
And in general, I am new to HAProxy. I still wonder if I set it up properly as I described in the first post here.
-
I simplified the frontend by removing the ACL and set the default backend to my backend. It's my web server anyway and it should process all my virtual hosts. Nothing changed after doing that. Still works for GET but not for POST, with internal errors for both the frontend and the backend. I am at a loss.
-
@pfpv are you maybe trying to use client IP, and in the advanced settings?
I do ssl offload, and have haproxy send to different backends via the sni sent by the client, ie the host headers. I have never had any problem.
Normally if you were doing ssl offload, your proxy connection to the backend would be in the clear, ie 80 or some other port not via 443 with the server trying to serve up ssl.
I am by no means a haproxy expert.. But if me, and I was wanting haproxy to do the offload, ie wouldn't do any sort of virtual host on the webserver sending too.. Use different ports or even IPs for your different sites you want to serve up.
And I don't do anything with client IP passthru or having the proxy use the clients ip in the advanced settings.
-
@johnpoz,
I am not sure what you meant by me using client IP. I see a mention of client IP in the Use "forwardfor" option. I disabled this option and nothing changed. [EDIT] I think I found what you meant in the backend Advanced settings. No, I am not using Transparent ClientIP.I changed the backend to port 80. I actually started with 80, then as part of troubleshooting I changed it to 443 as the backend web server serves on both. This had no effect on the observed behavior.
My Synology web server has worked for years with current configuration. As I mentioned I used to port forward to it and it handled the Letsencrypt certificates. I wanted to use wildcard certificates and the Synology couldn't handle them (at least not easily), so I decided to offload on pfSense that can handle wildcards. Name-based virtual hosts are typical for Apache configurations, and they are easy to handle. Seemingly, only one backend and one frontend are needed in HAProxy to handle all hosts on the Apache web server. And it works but only partially. GET but not POST, and POST works sometimes for simple web pages.
I thought about switching to port-based virtual hosts on Apache but each host would require its own backend and frontend. I bet there are servers with name-based virtual hosts behind HAProxy. Maybe HAProxy needs some extra configuration to work this way?
-
I switched one of my virtual hosts to port-based on some random 50k port. I created a new backend for it in HAProxy and created a new shared frontend.
The same. I can reach this host, let's say it's 111.domain.com, so HAProxy properly connects to the new port on the backend but POST gives a 500 error.
Now I am thinking this may have something to do with the fact that Synology runs nginx as frontend by default with Apache behind it. There is at least one mention of it elsewhere: Web Station uses nginx instead of Apache
Any advice?
-
I have an update to this. I had been testing my configuration from my PC on LAN. I use Pure NAT and never had a problem (and don't have now) but I tried connecting to my web server from outside and didn't have issues with POST. I started looking into what could be causing this issue on my PC and found that disabling AdGuard app resulted in no 500 error.
Going into its settings I found that setting "Adjust size of fragmentation of initial TLS packet" and "Plain HTTP request fragment size" back to default 1 resulted in no 500 errors anymore. Because I am on PPPoE I had set those values to 1492.
Now, why would this this crash HAProxy? Is there any way to find out the exact HAProxy error? The logging was set to debugging and I could only see connection attempts in the logs, no errors.
Why would anything crash HAProxy? Shouldn't it fail gracefully? Refuse connections maybe if it's not happy about something? I was using those settings for many months and I don't think I ever saw error 500 on the internet. I never crashed any other reverse proxy.
Can it be related to that I am using haproxy-devel per recommendations here? I could not find any meaningful description why the devel package was created and what's changed in it compared to the non-devel.
https://cgit.freebsd.org/ports/commit/?id=acb561a07356b92137b8388c668b2c622638acb6
https://cgit.freebsd.org/ports/commit/?id=c958e9dfd9b3bdefd1d53b28dc5882ca061ccb16
https://github.com/pfsense/FreeBSD-ports/commits/devel/net/pfSense-pkg-haproxy
https://github.com/pfsense/FreeBSD-ports/commits/devel/net/pfSense-pkg-haproxy-develIs there any chance HAProxy will stop crashing if I move to the non-devel package? Both of them use the dependency from Sept. 2023 while the latest non-devel haproxy is from June 2024: https://www.freshports.org/net/haproxy
