Sticky Incoming Load Balancer problem when primary pool responds with TCP reset

[adam65535]

To begin with I realize that sticky is useless when you only have 1 IP in both the primary and backup pool.  Assume that sticky is enabled because of the need for sticky in other Load Balancer entries that are created where you have more than 1 IP in a pool.

Setup:
Primary pool1 with 1 IP
Backup pool2 with 1 IP
Sticky enabled in System/Advanced/Misc
I have a test app making a GET / connection request every 2 seconds to the external Load Balancer IP from a system outside the external WAN with example IP address we will call ip1.

If a client keeps making requests every few seconds and the primary backedn pool server starts responding with TCP reset (backend web service stopped) the Source Tracking entry for that client IP will never be removed.  This will keep the client going to the primary pool that is down(GUI shows primary down).  The main problem seems to be that the Source Tracking table entry never gets removed because the TCP reset from the server creates a TIME_WAIT state in the firewall. With sticky as long as a state exists for that client the Source Tracking will keep the client going to the same backend pool.  If the client stops sending requests the TIME_WAIT states created from the TCP resets will eventually expire so that after that the client IP will get directed to the backup pool.

1. Connection #1 from ip1 comes in and goes to primary pool.
2. State created as well as entry in Source Tracking table
3. Primary pool1 goes down (server will respond with TCP reset)
4. GUI shows Primary pool1 as down
5. Connection #2 from ip1 comes in and because of Source Tracking entry goes to primary pool
6. Server on primary pool responds with TCP reset (This seems to keep the Source Tracking table from timing out because it creates a TIME_WAIT:TIME_WAIT state )
7. Connection #3 from ip1 comes in and because of Source Tracking entry goes to primary pool
8. Server on primary pool responds with TCP reset (This seems to keep the Source Tracking table from timing out because it creates a TIME_WAIT:TIME_WAIT state)
–- repeat 7 and 8 until infinity... the client will never go to the backup pool as long as the backend primary pool server responds with TCP reset and the client keeps making connection requests before the TIME_WAIT states time out.

Showing trace on LAN side where the server is responding with TCP reset because service is stopped on primary pool1 IP:

17:37:08.079419 IP 128.z.z.z.53835 > 192.168.x.x.80: Flags [s], seq 926449648, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2776012600 ecr 0], length 0
17:37:08.079766 IP 192.168.x.x.80 > 128.z.z.z.53835: Flags [R.], seq 0, ack 926449649, win 0, length 0
17:37:10.099465 IP 128.z.z.z.53836 > 192.168.x.x.80: Flags [s], seq 2621468179, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2776014619 ecr 0], length 0
17:37:10.099675 IP 192.168.x.x.80 > 128.z.z.z.53836: Flags [R.], seq 0, ack 2621468180, win 0, length 0

Showing states on firewall associated with the TCP resets.

[code]tcp	192.168.x.x:80 <- 97.y.y.y:80 <- 128.z.z.z:53835	TIME_WAIT:TIME_WAIT	
tcp	128.z.z.z:53835 -> 192.168.x.x:80	TIME_WAIT:TIME_WAIT	
tcp	192.168.x.x:80 <- 97.y.y.y:80 <- 128.z.z.z:53836	TIME_WAIT:TIME_WAIT	
tcp	128.z.z.z:53836 -> 192.168.x.x:80	TIME_WAIT:TIME_WAIT[/code]
(a lot of these because connection attempts every 2 seconds)

To fix it I wonder if when a pool goes down (or individual IP in a pool goes down) all states associated with the downed backend services should be killed with pfctl -k and also any Source Tracking table entries associated with that IP should be killed with pfctl -K (capital K for Source Tracking table).

In specific cases like this where you only have 1 IP in a pool for primary and backup there is no need for sticky on that connection.  Maybe set those specifically to never use sticky under that situation but leave others as sticky if the user has configured that option.  This does not work when you have multiple IPs in a pool though.  It is only a workaround for this one case where you have 1 IP in a pool on both primary and backup pool.

I was hoping to stick with the built-in relayd because connections will be kept up when failing over from Primary and Backup firewalls with relayd load balancing.[/s][/s]