Load balancing + Carp issue on LAN
I'm trying to set up a load labancer (virtual server) for a cluster of 2 tcp servers residing on the LAN.
I have no problems when the load balancer works between WAN (carp IP) and the servers on the LAN.
However, when I set up a LAN->LAN load balancer, I cannot connect to the load balancer IP.
My setup is :
1. Cluster of 2 pfsense 2.0.2 boxes (cannot upgrade them yet).
2. The 2 pfsense boxes have a shared IP, via CARP, set on the LAN interface. I can ping the CARP ip form the lan
3. A pool of 2 servers, on the LAN, and a TCP service. The pool shows both servers UP (green).
4. A virtual server, set on the CARP ip and port and pointing to the pool of 2 servers.
5. Firewall rule on LAN to allow all to all.
6. Advanced outbound NAT on WAN for packets originating from LAN to WAN (for the needs of the lan servers).
If I set up the same load balancer from WAN to LAN all works properly.
If I set it up LAN->LAN I cannot connect to the service.
Any ideas are welcome. Is this s known bug?
I'm connecting on the LAN ip address. The balancer is LAN->LAN. It must be something else.
I need to change this topic, as this is clearly an issue with relayd…. for which I'll open a new topic.
In such a scenario you rely entirely on icmp redirect messages, so make sure you are not blocking something you shouldn't.
I fired up some tcpdump and found the issue.
What happens is this:
1. the SYN packet comes in the LAN interface
2. pfsense sends it back over lan to one of the destination servers, with a changed dest address
3. the server replies back, directly to the originating client
4. The client sees a good packet, with wrong source address (being the one of the server and not the carp ip), and thus kills the connection with a RST packet.
So, to work, you have to keep the two networks (client and server) separate or a RST will kill the connection.
Now, this is very unfortunate, as it means I cannot use pfsense for the load balancing, but have to resort to some other means (corosync/heartbeat)
Pfsense is not special case :) Any load balancing with the same topology will work the same. You just have to play some more with NAT to make it work.
The way that I have been able to get this working is to Create a VIP on a separate subnet, then I have a Rule that allows any traffic to that Subnet to the VIP.
Since the LAN Traffic is on the same private space interface the traffic can flow between the two. Just make sure you allow ANY ANY Traffic between the two DMZ's then any Server/Client on DMZ-1 can talk to the VIP on DMZ-2 that is a Load Balancer that points to Servers in DMZ-1 Subnet… seems a bit wonky but it works.. the biggest problem with allowing traffic to flow out the same interface and then back in.. NO Load balancer will allow this.. so you have to create another subnet for it to route to... even if the servers behind the Load Balancer are on the same subnet.