2.3 New Setup, HAProxy with Transparent Client IP + Outbound NAT

Borkness

I recently setup a pfSense VM at the recommendation of a trusted friend to replace an Ubuntu Server setup that I was using as a load balancer. My old setup was using Pound and HAProxy (though not together). and my HAProxy was fixed up to allow passing through of the client IP to the backend server behind HAProxy. Things worked great until I fubared it by upgrading Ubuntu and then client IP forwarding stopped working and I don't remember all the steps I took to get it working.

Anyways, so fresh pfSense VM, configured it, installed HAProxy, had it working, got the transparent client IP working, life was good. Then I started messing with Outboud NAT and I could ping, tracert, etc but I couldn't get out with a web browser on port 80.

Well, after messing with it for two days I was able to isolated the issue down to the Transparent Client IP, once that is toggled things work/don't work so that's the culprit.

Are there any tricks to getting Transparent Client IP with Outbound NAT? I'm using HAProxy as a TCP proxy for an Exchange box in my current Ubuntu environment and Pound for HTTP/HTTPS traffic and I planned to just setup HAProxy to handle all of it going forward. Unfortunately, I NEED Transparent Client IP so Exchange can do SCL ratings and Block Lists and at the same time need Outbound NAT so the Exchange box can "send" mail as a specific IP.

I was able to see that other people had a similar issue but since they were using HAProxy as a HTTP/S proxy they could use X-Forwarded-For but since I'm doing TCP I can't. This topic lead me to be able to confirm my issue: https://forum.pfsense.org/index.php?topic=74042.0

PiBa

Hi Borkness,

The part i don't understand from your situation description is how outbound nat is related to the traffic passing through haproxy.

These two traffic flows are completely separate in my opinion.

1: Exchange:40000+ > pfsense(outbound nat) > internet > remote mailserver:25
2: Mail sender:40000+ > internet > pfsense(haproxy) > Exchange:25

The first uses outbound-nat, the second haproxy. If one of those flows don't work that specific problem should be dissected to the root cause and a solution/workaround found..

One weird thing haproxy package does when set to use transparent clientip is direct ALL traffic coming from exchange:25 to localhost/haproxy when it passes through pfSense(a ipfw rule is used for that). Even when the request was send directly without the request passing though haproxy process itself..

It doesn't sound like you where doing that? Or are you?

Can you elaborate a bit on the actual traffic stream as seen from tcpdump and where it 'breaks' ?

Regards
PiBa-NL

Borkness

Thanks for the reply. Both of the traffic flow examples you've given are accurate, to the best of my knowledge, with some exceptions.

On flow #2 we're also allowing HTTP (80) and HTTPS (443) to flow through to the Exchange box. Full list:

80
443
25
465
110
135
139
60000 - 60001
6001 - 6004
993 - 995

Outbound NAT should not have any relation to inbound traffic, yet one specific setting does cause outbound NAT to fail. That setting is "Transparent Client IP" on a HAProxy backend server. Once that is enabled, HTTP/HTTPS Out from the Exchange box stops working.

In all honesty, I wasn't doing the testing from my Exchange box, I had a test Windows 2008 R2 VM that has IIS installed and I was testing everything before I moved the the IP from Ubuntu to pfSense.

The Transparent Client IP was working, the traffic in the logs in IIS showed the remote IP. This is important because if that works then Exchange can also see the real client IP and our Spam Content Filtering and Tarpit would work correctly.

With all of that said, on this VM that was set to use pfSense as the gateway for outbound NAT, once Transparent Client IP was enabled, HTTP traffic OUT from the VM failed. I could still ping, tracert, etc, just no HTTP/HTTPS. Toggle the Transparent Client IP and it started working again.

When the "Transparent Client IP" is enabled, something behind the scenes is happening in terms of routing that is breaking.

From the other thread I linked to:

That creates a 'ipfw' rule to catch all 'reply traffic' and send it to the local system.. Not caring about if it was send out through haproxy or not..

When I look in the firewall to see the blocked connections it's clear that the inbound reply is being blocked from the server back to the client, exactly the symptom of the above mentioned catch all rule.

For now I just have NAT 1:1 setup with Transparent Client IP off. It doesn't allow me to load balance Exchange but at this point I only have one box so that isn't so important.

PiBa

The 'catch all' is limited to the actual ports that are forwardedby the haproxy backend servers.

Can you try a 'ipfw -x 4000 show' on a ssh console and show the output?

Should result in something like this:

64000    4945    642772 fwd 127.0.0.1 tcp from 192.168.0.40 80 to any in recv em0

This forwards reply-traffic from the server .0.40 port 80 to haproxy. If that same server tries to visit a remote webserver on port 80 it should not be caught by this rule and pass normally.

Do you have a captive portal active as well? (it shouldnt interfere.. but uses ipfw as well so might..)

Borkness

Toggling the transparent Client IP adds/removes these two lines to ipfw rules:

64000 0 0 fwd ::1 tcp from 192.168.3.133 80 to any in recv em0
65535 125 53458 allow ip from any to any

For what ever reason, as of right this second, outbound NAT is working correctly but I don't have faith that my "out of the box" configuration was causing a conflict before. To get it working I just went with NAT 1:1 for the Exchange VIP and moved Exchange behind pfSense. That is all working now so I moved on to setting up HAProxy for SSL Offloading for some other websites.

I tried to revert most of my settings to test but for whatever reason NAT is working as it should when Transparent Client IP is enabled, which is good, but not good in terms of figuring out what the root of the problem is.

I'm going to setup another pfSense VM tonight and attempt to get it "broken" again to troubleshoot. I literally had setup a fresh pfSense, installed HAProxy, gotten transparent IP working, tried to NAT out and it failed so it should be fairly easy to replicate.

PiBa

Hmm one thing that is peculiar is that your rule seems to be mixing IPv6 (::1) with IPv4(192.168.3.133) not sure if that might cause anything strange..

Anyway most of it 'should' function properly. (besides the known issue with a attempted 'direct' server connection..)

p.s. your only using transparent-client-ip for the :80 port currently right? otherwise it would show more rules.. also means your :25 is presented the haproxy ip.. (if it go's through haproxy..)

Borkness

Yes, I'm "testing" HAProxy balancing "tcp" instead of "http" and my test server only has IIS on it so just port 80.

I setup another VM from scratch and I can't, for the life of me, reproduce the issue. It would have to be user error of some sort but it's odd that I'm not the only one who has experienced the same thing where you can't access "out" on port 80 and 443. Perhaps a bug with the order that things happen in some configurations but since I can no longer reproduce the issue I can't say.

I appreciate your help though.