Remote Nginx Reverse proxy in front of Pfsense HAproxy : weird behavior

[Universal2688]

Hi everyone ,

This is my first post so thanks for correcting me if I'm missing something or posting at the wrong place. I have an intermediate level in networking and security and I'll try to explain everything in details but that might no be precise enough, don't hesitate to correct me, thanks.

This is a complicated bug so it's going to be a long post, sorry for that.

I have a HomeLab setup with multiple VLANs and everything is working fine for now, my architecture looks like this :

• ISP router in bridge mode (transparent)

• Netgate SG-1100 with Pfsense plus

  • VLAN management with a manageable switch (OPT interface isn't used)
    • VLAN 1 : Public pages (isolated) -> let's say 10.0.10.0/24
    • VLAN 2 : Proxmox admin access -> let's say 10.0.20.0/24
    • others VLANs : admin PC, iot wifi ...
  • PfBlockerNG (GeoIP filtering)
    • I have a custom inbound rule that allows french IPs only to my HAProxy backend 443
  • HAProxy
    • Frontend : my ISP public address 443, SSL offloading with ACME cert, ACL conditions ok
    • Backend 1 : matches gitlab.mydomain.ovh -> Gitlab instance 10.0.10.50
      • Check : GET /-/health (as described in Gitlab docs)
    • Backend 2 : matches pages.mydomain.ovh -> Nginx instance 10.0.10.60
    • Check GET /
    • For testing, I set nginx backend as default to exclude ACL problems from my bug tracking
    • Note : I do http offloading because these websites contains only public pages and projects so there's nothing to hide really, nobody connects to the gitlab

• Proxmox VE

  • 2 NICs, one for admin access (vlan 2) and one for the public pages (vlan 1)
  • Gitlab container in http mode (external_url 'http://gitlab.mydomain.ovh')
  • Nginx container listening on http port

As I said, it works OK in this setup, flawlessly.

But in my DNS records, I point directly at my home IP address and I'd like to hide it behind a remote proxy. This would also allow me to not open 443 on pfsense WAN but a custom port for the backend.

I've been trying to achieve this with a rented VPS (with a french IP). On this VPS I run another Nginx server, but this one as a reverse proxy, I'll call it nginx-proxy to avoid confusion.

The bug is, in short, when I try to proxy requests from my remote nginx-proxy to my HAProxy frontend, it seems to block the IP from the VPS somewhere in Pfsense, and then mess up with some of my backends, in this case gitlab.

I reproduced the same loop on two VPS instance separately, with different IPs :

• Test a curl -v https://pages.mydomain.ovh from the VPS -> works fine
• Test a curl -v https://gitlab.mydomain.ovh from the VPS -> works fine
• Modify my DNS record to point to the new VPS IP instead of my WAN IP
• Install nginx on the VPS
• Configure nginx proxy to listen for incoming *.mydomain.ovh on port 443 ssl with cert provided (copied from the Pfsense ACME certs) and forward to the WAN IP on 443
• Test a request from a different network (my phone on 4G data) to https://pages.mydomain.ovh or https://gitlab.mydomain.ovh -> No answer

First bug after this loop

• The gitlab suddenly isn't reachable anymore from HAProxy, neither from an allowed VLAN (like promox) or from pfsense console, though the nginx container works fine.
  Gitlab responds locally from the container shell but simply can't be reached. Restarting the container or HAProxy does not help, I must restart Pfsense so that the backend is accessible again.

Second bug after this loop

• After reseting DNS and testing the initial setup, I can't curl -v https://pages.mydomain.ovh anymore from the VPS although the nginx container is accessible from elsewhere
• After rebooting then completely reinstalling the VPS OS, still can't curl my websites
• After rebooting pfsense, still can't
• After reseting pfsense to factory defaults (and losing lots of time re-configuring everything) my VPS can again do a curl -v https://pages.mydomain.ovh
• From there I can start the loop again and reproduce the bug

My log analysis :

• The nginx-proxy receives the initial requests and sends it
• The request from the nginx-proxy is allowed on WAN address 443 -> I can see the PfBlockerNG allow rule (mentioned above) in the logs, so the firewall isn't the issue
• HAProxy does not see anything, nothing on logs (I normally see logs with the working setup) and won't tell me anything, I don't even know if the frontend really receives requests

My analysis

• There's maybe a problem with my Gitlab config in plain http. I tried using http://10.0.10.50 directly as external_url but it is not meant to be configured like that, I causes a lot of problems
• HAProxy seems to do stuff behind my back and won't let me log it, I can't see any options to fully log frontend requests through the GUI, and I can't modify the haproxy.cfg because it gets overwritten at restart.

Other tests :

• PfBlockerNG could be the culprit, but I tried disabling it and adding a simpler firewall rules (any source IP to 443 on my WAN, not ideal but only for a quick test) and it did not help.

So ... Thanks if you took time to read, and if you have any idea what could be causing these issues, or how could I troubleshoot what's going on because I already read all the logs I can (at least I know of) and it's driving me crazy

Again, thank a lot if you can help.

U2688

[Universal2688]

Here's my network diagram, it will be easier :

[viragomann]

@Universal2688
Is HAproxy running in transparent mode?

Can you show your firewall and NAT rules on pfSense, please?
There is no reason for obscuring local IP addresses, as long as they are private ones. Nobody outside your network will be able to access them. Private IPs are not routed through the internet.

[Universal2688]

Thanks for your reply!

I'm not trying to hide any local IP, that's because I have weird random IP address for my VLAN, using 10.0.10.0 is easier to read imo.

This is my WAN firewall config :

The Proxy135 rule is a backup to pass the VPS IP directly in case it does not appear as a french IP.

I have not NAT rules configured. The pfblockerng rule pass to 443 is enough when I don't enable the VPS. As you will see in the next screenshots, the request is accepted.

On the gitlab/nginx VLAN, no rules are configured for now as these containers should not access anything (I will add specific rules for updating soon).

This is my pfblockerng config :

This is an example of the VPS connecting to pfsense :

This is the same example from the pfblockerng logs

So as you can see everything seems fine from the firewall POV.

[viragomann]

@Universal2688 said in Remote Nginx Reverse proxy in front of Pfsense HAproxy : weird behavior:

Test a request from a different network (my phone on 4G data) to https://pages.mydomain.ovh or https://gitlab.mydomain.ovh -> No answer

Just no answer?
If the request hits the HAproxy frontend I expect to get anything at least.

What if you set the Gitlab as default backend for testing purposes?

The gitlab suddenly isn't reachable anymore from HAProxy, neither from an allowed VLAN (like promox) or from pfsense console, though the nginx container works fine.
Gitlab responds locally from the container shell but simply can't be reached. Restarting the container or HAProxy does not help, I must restart Pfsense so that the backend is accessible again.

Anything regarding to this in the pfSense log?

HAProxy does not see anything, nothing on logs (I normally see logs with the working setup) and won't tell me anything, I don't even know if the frontend really receives requests

Check the HAproxy stats.

There's maybe a problem with my Gitlab config in plain http.

I don't know Gitlab, but if it requires TLS connections you can configure HAproxy accordingly. You can use a self-singed certificate on the backend server and disable SSL checks in HAproxy.

[Universal2688]

@viragomann said in Remote Nginx Reverse proxy in front of Pfsense HAproxy : weird behavior:

Just no answer?

Sadly yes, simply no answer from HAProxy no matter which logs I'm reading, it's like the request simply vanishes once it passes the firewall filter and HAProxy never gets it (but it certainly does if it's listening on 443).

When I pass a request from anything but the VPS but the backend is not up, I receive a 503 service unavailable. When I pass a request from the VPS it times out even tho the request appears in the pfsense firewalls logs.

It would be great if there was a log option like "Show all incoming packets on frontend" and show more detailed infos.

What if you set the Gitlab as default backend for testing purposes?

I did at first before building the nginx container for simpler tests. Same results, no answer from HAProxy even tho the default backend is healthy according to the checks.

Anything regarding to this in the pfSense log?

Nope, I looked for an error in the system logs too and there's nothing. I'm currently trying to build a remote syslog on my proxmox maybe that will help.

Check the HAproxy stats.

You're right, there's something wrong on the stats, see the screenshot below. I still can't find the associated logs. Am I missing something ?

I don't know Gitlab, but if it requires TLS connections you can configure HAproxy accordingly. You can use a self-singed certificate on the backend server and disable SSL checks in HAproxy.

You're right, and also the SG-1100 isn't that powerful so offloading isn't the best idea.

Thanks again.

[viragomann]

@Universal2688
Yes, you have view errors, but the frontend also shows some sessions total.

You can enhance the log level in the HAproxy settings to get more details on what it does.

Also you can sniff the traffic on the DMZ interface to ensure that the packets are forwarded to the correct backend.
But since the Gitlab backend is showing some traffic and sessions, I presume that the traffic is forwarded to it.
So possibly there is something wrong with the Gitlab backend. I don't use it, so I cannot tell you, how to configure.