Haproxy Backend Outbound Interface

[rpm5099]

I don't think this is really an HAProxy question because I have that part working fine. I have a need to proxy a domain:port through the WAN of pfsense via HAProxy, and have the HAProxy backend forward the request outbound via a specific VPN1 interface. I've tried this a number of ways, but no matter how I configure it the backend always just sends the request back out through the WAN instead of through the VPN1 interface I need it to.

My current configuration summarized (oldfuncoollaugh.neverssl.com is a non-ssl test site):

• Virtual IP on unused subnet created 10.7.0.107
• HAProxy front end listens on custom address 10.7.0.107:12345 with ssl offloading enabled, uses hostname acl xyz.mydomain.com to select backend - this all works fine
• HAProxy Backend server list set to address+port oldfuncoollaugh.neverssl.com:80, no-SSL, no checks.
• Backend passthrough: http-request set-header Host oldfuncoollaugh.neverssl.com:80
• NAT Port forward from WAN:12345 to 10.7.0.107:12345, default NAT reflection; WAN rule to allow TCP port 12345
• Outbound NAT rule source 10.7.0.107 translation to (dynamic) VPN1 interface address - this is what does not work I believe
• Firewall/Rules/LAN - source 10.7.0.107:* dest *:* Gateway: VPN1
• The site can be successfully accessed via https://xyz.mydomain.com:12345

I've also tried this same complete setup using a local LAN ip instead of a virtual IP on a different subnet with the exact same results. HAProxy backend will NOT use the VPN1 interface for outbound traffic. I've static routes, nat reflection settings, new gateways, every firewall rule configuration I can think of, and several passthrough settings for the HAProxy backend including source 10.7.0.107 which just broke it entirely and I'm not sure why.

This seems like it should be really simple to do, I got things to the point of done within 15 minutes and then have been stuck trying to somehow force HAProxy to use the VPN1 interface for outbound traffic. None of the routing or firewall rules seem to make any difference, I think because the traffic never leaves the pfsense "local host" - HAProxy seems to be turning the traffic right around and sending it out the default gateway without any other settings being evaluated at all. Any help is greatly appreciated.

[rpm5099]

Made quick diagram.

[viragomann]

@rpm5099 said in Haproxy Backend Outbound Interface:

Virtual IP on unused subnet created 10.7.0.107
HAProxy front end listens on custom address 10.7.0.107:12345 with ssl offloading enabled, uses hostname acl xyz.mydomain.com to select backend - this all works fine

Not clear, which device used 10.7.0.107.
Here you wrote, that HAproxy is listening on it, but your diagram implies that it is used by the backend server.

Backend passthrough: http-request set-header Host oldfuncoollaugh.neverssl.com:80

The port is not part of the host header.

Which type of VPN is this one in question?

[rpm5099]

@viragomann - sorry if it was unclear, was my first time using that diagram site and struggled a bit. The frontend is listening on the WAN and NAT'ing to the haproxy frontend at 10.7.0.107:12345, which breaks SSL and forwards to the backend. The backend is then set to send an outbound request to oldfuncoollaugh.neverssl.com on port 80. In this case the server I believe does inspect the host field but probably ignores the port. Whether the port is included in the host header field or not [per RFC] the connection works fine.

The sole problem here is that I need the backend to not use the WAN interface for outbound traffic, and instead use one of the VPN interfaces which I have set in the firewall rules as the gateway. It does not, it sends it right back out the non-VPN WAN it came in on. This happens to be an OpenVPN interface, but I don't think that should matter. I just can't seem to figure out how to force the HAProxy backend to use the correct outbound interface. I could point it to another LAN host and have that host redirect it out through the VPN interface, but that's an unnecessary complication with an additional point of failure and I know there has to be a simple way to do this.

Thanks for responding, let me know if there's any other info I can provide.

[viragomann]

@rpm5099
It's still not clear to me. From your drawing, HAproxy and the VPN is running on pfSense itself.
But what is the backend server? Is it running at the remote site and you need to access it via the VPN?
If so, just add a DNS host override for oldfuncoollaugh.neverssl.com and map it to the local IP of the server, presuming the VPN has proper routes.

I need the backend to not use the WAN interface for outbound traffic, and instead use one of the VPN interfaces which I have set in the firewall rules as the gateway.

This only affects packets entering on an interface, but packets from HAproxy running on pfSense doesn't

[rpm5099]

@viragomann I think I realized where our wires were crossed, the backend server is the actual external http server. It's pointed directly at it via address:port - there's no other configuration options set on the backend except for the host header modification and health. Maybe this isn't the way to make this work with haproxy, but I assume there must be a way to make an Haproxy backend use a specific gateway when connecting outside of the LAN.

My outbound NAT rules are setup with the virtual IP of the frontend - the source IP on the packet should be changed to that after traversing the front end right?

Do I need to create an additional virtual IP and have the backend forward to that possibly?

[viragomann]

@rpm5099 said in Haproxy Backend Outbound Interface:

It's pointed directly at it via address:port - there's no other configuration options set on the backend except for the host header modification and health. Maybe this isn't the way to make this work with haproxy

I'd expect, that it should work that way.

but I assume there must be a way to make an Haproxy backend use a specific gateway when connecting outside of the LAN.

No, outbound packets from HAproxy strictly follows the routing table of pfSense. You cannot policy route them or whatever.

As I suggested above, you should add an DNS host override for the host name mapping it to its local IP. This has to be routed over the VPN then.

Alternatively you can route the public IP over the VPN.

There should no outbound NAT rule be necessary for this, as long as HAproxy is not in transparent mode.
Then the packets will arrive the remote site with the source of the VPN endpoint anyway.

I suspect you have issues with your VPN.

[rpm5099]

@viragomann I'm not sure how else to say it but oldfuncoollaugh.neverssl.com is not a local host, it doesn't have a local IP. It's a domain on the internet outside of my control. I'm using it for testing because I don't have the final host:port info of the actual domain yet. I can't hard code the IP address into the pfsense routing table because the IP of oldfuncoollaugh.neverssl.com could change at any time - haproxy updates it regularly when you point a backend at a domain.

I have 20+ other devices on my LAN that solely use this same VPN as their internet gateway - I configure them using an alias in the source of an outbound NAT entry with the VPN as the interface. I have added 10.7.0.107 to this alias, but it makes no difference. The haproxy backend connects outbound to oldfuncoollaugh.neverssl.com using the WAN as the gateway instead of the VPN.

[viragomann]

@rpm5099 said in Haproxy Backend Outbound Interface:

I have 20+ other devices on my LAN that solely use this same VPN as their internet gateway - I configure them using an alias in the source of an outbound NAT entry with the VPN as the interface.

I don't think, that this does the job. The outbound NAT does just NAT, no routing.
Maybe you've used the same alias in a policy routing rule.

The haproxy backend connects outbound to oldfuncoollaugh.neverssl.com using the WAN as the gateway instead of the VPN.

That's expected, if there is no other route for this domain.

As far as I remember, OpenVPN in recent pfSense versions accepts host names or aliases in the "remote network" settings, but not sure at all. Maybe it was just a feature request, I've read in the past. But you can try it with an alias.

Without routing the destination address to the VPN remote endpoint, it will ever go out to the WAN gateway.

[rpm5099]

@viragomann Yes, the outbound firewall rule for the VPN does apply policy based routing. Under the firewall rules for the VPN I have a pass rule with source=All LAN subnets, destination=Any, Gateway=VPNV4 (that sets the routing for you). The gateway field in firewall rules has a note:

Leave as 'default' to use the system routing table. Or choose a gateway to utilize policy based routing. Gateway selection is not valid for "IPV4+IPV6" address family.

I'm pretty sure I tried a configuration where the haproxy frontend was on the 10.0.0.0/24 with the rest of my LAN and got the same results, but if you think this is where the problem lies I will make an explicit rule here for the 10.7.0.107 front end and see if that makes a difference.

[viragomann]

@rpm5099
Again: You cannot policy route traffic from HAproxy. And the frontend address makes no sense anyway.
You need to route packets from HAproxy to the backend device, and this can only be done by the routing table of pfSense.

I've repeated this here at least three times obviously to no avail.

[rpm5099]

@viragomann

I've repeated this here at least three times obviously to no avail.

I appreciate you trying to help me, but if had been been able to successfully "route packets from HAproxy to the backend device" I wouldn't have made this post in the first place. I'm not a moron, I'm asking for help from the community because I'm not an expert specifically in haproxy and pfsense.

I understand that pfsense does not use iptables, but I am not familiar with how it differs from "packet filter" and how that relates to policy based routing, why HAproxy cannot be directed by policy based routing, and how policy based routing translates to the pfsense routing table. I think we are talking past each other on networking terminology semantics.

Is it possible to configure pfsense settings to direct all hosts from any interface and all subnets to use a certain VPN gateway for specific domains on a pfsense device?

[viragomann]

@rpm5099
The only setting, which is capable of ding this in pfSense is a floating rule.
You will need a policy routing floating rule on WAN for outgoing packets to the specific host name.
I've never used something like that.

But might work this way:
Create an alias for the destination and state the host name (because it's not static).

Add a floating rule:
Action: pass
Quick checked
Interface: WAN
Direction: out
Protocol: TCP, or whatever you need
Source: any might be fine for you.
Destination: the alias you've created above
Advanced options > Gateway: the desired VPN gateway

[rpm5099]

@viragomann

Sorry I never replied back with what I found out. It turns out that you can do this but it requires a setting in the OpenVPN client and Outbound NAT settings. I only figured this out through many iterations of trying every possible reasonable setting.

Create a firewall Host alias that contains the domains you want to force through a specific gateway. Enter the domains/URL's there. When you save it will fetch the URL's, so do this first or your list might not be updated with resolved IP's.

Add your firewall "host" alias (that contains the list of fqdn's you wish to force through the specific VPN Gateway) to the "IPv4 Remote Networks" field in the OpenVPN client settings. This is what updates the routing table. Note that the remote networks field does not auto-complete the alias when you type it (unlike most fields), but it'll fail if it's the wrong type or non-existent.

For the outbound NAT settings: set the interface to the VPN you want to force the domains through, IPV4, any protocol, source any, destination -> network or alias - use an alias created in pfblockerng OR the one you use above, [gateway] translation [your vpn client name] Address, and place it at the very top of the Outbound NAT rules

For pfblockerng alias, it can be used for the outbound NAT config but cannot be used for the OpenVPN client config (you can probably ignore pfblockerng and just use the built in URL alias option). In the pfblockerng settings create IPV4 alias and set the domains for Whois with state On and enter the domains. Under settings use Alias Native and pick your update settings (For local alias I'm pretty sure you can just leave it disabled, but it doesn't really matter).

It ended up not working for what I wanted to use it for because the domains in question end up forwarding, but at least there is a way. I did not find this documented anywhere.