Help! I give up…. I need to setup Load Balancing *locally* and it only works a

WolphFang · Feb 19, 2010, 12:44 AM

Help! I give up…. I need to setup Load Balancing locally and it only works against the local HTTP port!

Goal: See attached pic.

rkelleyrtp · Feb 19, 2010, 1:05 AM

What have you done so far? Have you created your VIP?

Have you considered using haproxy instead?

WolphFang · Feb 19, 2010, 1:36 AM

Virtual IP address Type Description
192.168.0.2/32 [Proxy ARP]

Load Balancer

Pool: Server-5
Type: Server
Port 3389
Monitor: TCP
192.168.0.5

Virtual Servers:
192.168.0.2
Port: 3390
Virtual Server Pool: Server-5
Pool Down Server: 127.0.0.2 (intentional to a bad number)

============

Active States during connection attempt:


tcp	192.168.0.1:46428 -> 192.168.0.1:80	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:80 <- 192.168.0.1:46428	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:56830 -> 192.168.0.6:3389	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:28813 -> 192.168.0.5:3389	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:80 <- 192.168.0.2:3392 <- 24.129.75.203:42434	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:80 <- 192.168.0.2:3392 <- 24.129.75.203:15469	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:41123 -> 192.168.0.1:80	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:80 <- 192.168.0.1:41123	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:54026 -> 192.168.0.6:3389	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:35586 -> 192.168.0.5:3389	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:44511 -> 192.168.0.1:80	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:80 <- 192.168.0.1:44511	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:19584 -> 192.168.0.6:3389	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:8813 -> 192.168.0.5:3389	TIME_WAIT:TIME_WAIT
tcp	192.168.0.5:3389 <- 192.168.0.2:3390 <- 192.168.0.5:32570	CLOSED:SYN_SENT
tcp	192.168.0.5:32570 -> 192.168.0.5:3389	SYN_SENT:CLOSED
tcp	192.168.0.1:50965 -> 192.168.0.1:80	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:80 <- 192.168.0.1:50965	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.1:10358 -> 192.168.0.5:3389	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:44662 -> 192.168.0.6:3389	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:80 <- 192.168.0.2:3392 <- 24.129.75.203:6031	ESTABLISHED:ESTABLISHED

WolphFang · Feb 19, 2010, 1:40 AM

@rkelleyrtp:

What have you done so far? Have you created your VIP?

Have you considered using haproxy instead?

I tried haproxy with mostly non-results. I could get a single connection going.. then after disconnecting the RDP session, I get connection refused afterwards…. I also don't like the way it disconnects all active connections when an 'apply' is done to implement a change.

WolphFang · Feb 19, 2010, 1:46 AM

@WolphFang:

Help! I give up…. I need to setup Load Balancing locally and it only works against the local HTTP port!

Here are the states when configuring it with ICMP check:


tcp	192.168.0.1:80 <- 192.168.0.2:3392 <- 24.129.75.203:5290	TIME_WAIT:TIME_WAIT
tcp	192.168.0.1:80 <- 192.168.0.2:3392 <- 24.129.75.203:14106	FIN_WAIT_2:FIN_WAIT_2
tcp	192.168.0.6:3389 <- 192.168.0.2:3391 <- 192.168.0.5:32592	CLOSED:SYN_SENT
tcp	192.168.0.5:32592 -> 192.168.0.6:3389	SYN_SENT:CLOSED
icmp	192.168.0.1:57146 -> 192.168.0.1	0:0
tcp	192.168.0.1:80 <- 192.168.0.2:3392 <- 24.129.75.203:40333	ESTABLISHED:ESTABLISHED
icmp	192.168.0.1:59706 -> 192.168.0.6	0:0
icmp	192.168.0.1:60474 -> 192.168.0.5	0:0
icmp	192.168.0.1:61242 -> 192.168.0.1	0:0
icmp	192.168.0.1:65338 -> 192.168.0.6	0:0
icmp	192.168.0.1:571 -> 192.168.0.5	0:0
icmp	192.168.0.1:1339 -> 192.168.0.1	0:0

rkelleyrtp · Feb 19, 2010, 1:53 AM

It is hard to tell from your picture, but what exactly is your network topology? Are your clients plugged into the same network (LAN), or are the coming from the WAN side? Can you give a better picture?

Also, why is your PROXY ARP set with a /32 mask? As far as I know, it should be whatever mask is applied to your LAN subnet.

WolphFang · Feb 19, 2010, 2:12 AM

@rkelleyrtp:

It is hard to tell from your picture, but what exactly is your network topology?

pfSense in VM, two Term Serves in VM, Sonic Wall firewall to Internet.

Are your clients plugged into the same network (LAN), or are the coming from the WAN side?

All clients are in LAN, same subnet, no routers involved.

Can you give a better picture?

See new pic.

Also, why is your PROXY ARP set with a /32 mask? As far as I know, it should be whatever mask is applied to your LAN subnet.

a) Because the interface does not permit any change for the mask.
b) If I remember correctly, additional IPs set on an interface in the same same subnet as the parent ip# have to have a /32 to prevent some sort of internal routing issue with *BSD in general.

WolphFang · Feb 19, 2010, 2:27 AM

Here is /var/etc/slbd.conf:

I am currently trying to setup one server per "cluster" for testing purposes….

The pfSenseMGMT one seems to work fine.


pfSenseMGMT:\
	:poolname=pfSenseMGMT:\
	:vip=192.168.0.2:\
	:vip-port=3392:\
	:sitedown=127.0.0.2:\
	:sitedown-port=3392:\
	:method=round-robin:\
	:services=1:\
	:service-port=80:\
	:0=192.168.0.1:\
	:ping:
Server-0-6:\
	:poolname=Server-0-6:\
	:vip=192.168.0.2:\
	:vip-port=3391:\
	:sitedown=127.0.0.2:\
	:sitedown-port=3391:\
	:method=round-robin:\
	:services=1:\
	:service-port=3389:\
	:0=192.168.0.6:\
	:ping:
Server-0-5:\
	:poolname=Server-0-5:\
	:vip=192.168.0.2:\
	:vip-port=3390:\
	:sitedown=127.0.0.2:\
	:sitedown-port=3390:\
	:method=round-robin:\
	:services=1:\
	:service-port=3389:\
	:0=192.168.0.5:\
	:ping:

Status -> Load Balancer -> Virtual Servers shows all Servers Online.

rkelleyrtp · Feb 19, 2010, 2:30 AM

Sorry, dumb question. Are you specifying port 3390 as per the conf file when connecting to the remote machines via RDP?

WolphFang · Feb 19, 2010, 2:41 AM

@rkelleyrtp:

Sorry, dumb question. Are you specifying port 3390 as per the conf file when connecting to the remote machines via RDP?

Using 192.168.0.2:3390 and/or 192.168.0.2:3391
When I http to 192.168.0.2:3392, that works just fine.

Just noticed something though in the states:


tcp	192.168.0.6:3389 <- 192.168.0.2:3391 <- 192.168.0.5:32700	CLOSED:SYN_SENT
tcp	192.168.0.5:32700 -> 192.168.0.6:3389	SYN_SENT:CLOSED

Why is the second line missing the load balancer in the middle?

Addendum: Downloading and installing wireshark to look for *.0.2 traffic on one the TS's.

rkelleyrtp · Feb 19, 2010, 2:41 AM

Sorry, don't have the answer to your question. But, I am going to try this on my own pfSense firewall right now. I will let you know what I find.

BTW - I am sure you have seen this, eh? http://doc.pfsense.org/index.php/Setup_Incoming_Load_Balancing

WolphFang · Feb 19, 2010, 2:56 AM

Addendum: Downloading and installing wireshark to look for *.0.2 traffic on one the TS's.

Beyond the ICMP checks, I am not receiving any traffic from pfSense when connecting to the VIP:PORT combination.

WolphFang · Feb 19, 2010, 3:14 AM

Running a packet capture on the LAN interface for target host of 192.168.0.6 when connecting from 192.168.0.5 to 192.168.0.2:3391 reveals:

ICMPs…


08:06:38.846401 arp who-has 192.168.0.6 tell 192.168.0.1
08:06:38.846453 IP 192.168.0.1 > 192.168.0.6: ICMP echo request, id 17533, seq 0, length 64
08:06:38.846914 arp reply 192.168.0.6 is-at 00:0c:29:50:8d:aa
08:06:38.847029 IP 192.168.0.6 > 192.168.0.1: ICMP echo reply, id 17533, seq 0, length 64

Malformed attempt at connection?


08:06:43.727167 IP 192.168.0.5.32976 > 192.168.0.6.3389: tcp 0

ICMPs…


08:06:43.866523 IP 192.168.0.1 > 192.168.0.6: ICMP echo request, id 20093, seq 0, length 64
08:06:43.866850 IP 192.168.0.6 > 192.168.0.1: ICMP echo reply, id 20093, seq 0, length 64

Malformed attempt at connection?


08:06:46.764640 IP 192.168.0.5.32976 > 192.168.0.6.3389: tcp 0

ICMPs…


08:06:48.886760 IP 192.168.0.1 > 192.168.0.6: ICMP echo request, id 22653, seq 0, length 64
08:06:48.887107 IP 192.168.0.6 > 192.168.0.1: ICMP echo reply, id 22653, seq 0, length 64

Malformed attempt at connection?


08:06:52.800578 IP 192.168.0.5.32976 > 192.168.0.6.3389: tcp 0

ICMPs


08:06:53.906960 IP 192.168.0.1 > 192.168.0.6: ICMP echo request, id 42365, seq 0, length 64
08:06:53.907377 IP 192.168.0.6 > 192.168.0.1: ICMP echo reply, id 42365, seq 0, length 64
08:06:58.928464 IP 192.168.0.1 > 192.168.0.6: ICMP echo request, id 47229, seq 0, length 64
08:06:58.928866 IP 192.168.0.6 > 192.168.0.1: ICMP echo reply, id 47229, seq 0, length 64
08:07:03.946814 IP 192.168.0.1 > 192.168.0.6: ICMP echo request, id 49789, seq 0, length 64
08:07:03.947293 IP 192.168.0.6 > 192.168.0.1: ICMP echo reply, id 49789, seq 0, length 64
08:07:08.967876 IP 192.168.0.1 > 192.168.0.6: ICMP echo request, id 52349, seq 0, length 64
08:07:08.968258 IP 192.168.0.6 > 192.168.0.1: ICMP echo reply, id 52349, seq 0, length 64

Why is the load balancer using the connecting IP# as its source IP# for a generic TCP connection?

rkelleyrtp · Feb 19, 2010, 3:19 AM

Unfortunately, no help here. In fact, after configuring my LB pool, I get an "Offline" message under Status–>Load Balancer-->Virtual Servers. In addition, I don't get any servers listed under the Status-->Load balancer -->Pools tab. The only thing I have not done is reboot my firewall yet...

WolphFang · Feb 19, 2010, 3:27 AM

@rkelleyrtp:

Unfortunately, no help here. In fact, after configuring my LB pool, I get an "Offline" message under Status–>Load Balancer-->Virtual Servers. In addition, I don't get any servers listed under the Status-->Load balancer -->Pools tab. The only thing I have not done is reboot my firewall yet...

I thought pools were only for load balancing outbound across ISPs?

My status under virtual servers shows:


Name	Port	Servers	Status	Description

pfSenseMGMT 	3392 192.168.0.1 Online 	Last change Feb 19 2010 08:21:37

Server-0-6 	3391 192.168.0.6 Online 	Last change Feb 19 2010 08:21:37

Server-0-5 	3390 192.168.0.5 Online 	Last change Feb 19 2010 08:21:37

WolphFang · Feb 19, 2010, 3:54 AM

I found sufficient documentation to realize that this is not a TCP reconnecting daemon. It is a pf rules modifier for NAT reflection to produce the load balancing.

That is why I am getting those bad entries for the connection attempts.

The arrangement I am trying to setup is not possible with slbd.

The only reason it works for the MGMT port is because the pfSense machine is both IPs so that the response packets get processed before being sent back to the web-browser and get de-natted/re-natted the way they are needed.

When slbd redirects the connection attempts from the local machine to a local machine, it rewrites the request NATted, which causes the response to be transmitted directly to the originator, therefore it can't get re-munged into a proper format for the initiator to know it is a response, so I am sure it gets dropped and a TCP session is never even setup. The initial hand-shake fails.

WolphFang · Feb 19, 2010, 4:05 AM

I figured out a way to get it to rewrite EVERYTHING going in/out.

Firewall -> NAT
Manual Mode
Edit WAN rule: make it WAN interface, NAT, any <-> any
Create LAN rule: make it LAN interface, NAT, any <-> any

Testing operations now both locally and remotely.

WolphFang · Feb 19, 2010, 4:19 AM

@WolphFang:

I figured out a way to get it to rewrite EVERYTHING going in/out.

Firewall -> NAT
Manual Mode
Edit WAN rule: make it WAN interface, NAT, any <-> any
Create LAN rule: make it LAN interface, NAT, any <-> any

Testing operations now both locally and remotely.

ARGH! Someone went and OBEYED that stupid https management warning message on the Sonic Wall while I was finagling the rules!

All locked out now, everything, no more testing tonight. :(

rkelleyrtp · Feb 19, 2010, 11:38 AM

If it were me, I would just use haproxy for this. Check out this blog:
http://blog.loadbalancer.org/load-balancing-windows-terminal-server-–-haproxy-and-rdp-cookies/#more-296