Load Balancing with sticky sessions

Legia

Hello,

I am using pfsense as a load balancer for multiple web applications deployed on 3+ web servers (https access only). Applications are not designed for load balancing and require each user to maintain their connection with individual web servers.

Due to the nature of the service it is necessary to maintain idle session state for at least 45 minutes. I have noticed that states are lost before then and users are redirected to alternative web server on their next request.

I have sticky sessions enabled with 2700 seconds source tracking time out. Firewall Adaptive Timeouts at 6000-12000, Firewall Optimization Options set to Conservative.
There is no single setting for connection timeout and I do not know what change should be made. It is also expected that traffic will grow about 6 times in withing next three months.

pfSence version: 2.1.3-RELEASE (amd64) built on Thu May 01 15:52:13 EDT 2014 FreeBSD 8.3-RELEASE-p16

pfInfo output:

Status: Enabled for 68 days 20:45:40 Debug: Urgent

Hostid: 0x7c986363
Checksum: 0xe5346ecc0b6704a908e03d422e37bfb0

Interface Stats for em1 IPv4 IPv6
Bytes In 862405958763 0
Bytes Out 98268604810 304
Packets In
Passed 697418872 0
Blocked 41437 0
Packets Out
Passed 497496072 4
Blocked 0 0

State Table Total Rate
current entries 2704
searches 2370268910 398.4/s
inserts 22523960 3.8/s
removals 22521256 3.8/s
Source Tracking Table
current entries 103
searches 10424221 1.8/s
inserts 58220 0.0/s
removals 58117 0.0/s
Counters
match 22527146 3.8/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 2 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 85632 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
divert 0 0.0/s
Limit Counters
max states per rule 0 0.0/s
max-src-states 0 0.0/s
max-src-nodes 0 0.0/s
max-src-conn 0 0.0/s
max-src-conn-rate 0 0.0/s
overload table insertion 0 0.0/s
overload flush states 0 0.0/s

states hard limit 100000
src-nodes hard limit 100000
frags hard limit 5000
tables hard limit 3000
table-entries hard limit 200000

tcp.first 3600s
tcp.opening 900s
tcp.established 432000s
tcp.closing 3600s
tcp.finwait 600s
tcp.closed 180s
tcp.tsdiff 60s
udp.first 300s
udp.single 150s
udp.multiple 900s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 6000 states
adaptive.end 12000 states
src.track 2700s

all
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 3 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
carp
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
em0
Cleared: Thu May 29 16:22:04 2014
References: [ States: 1319 Rules: 25 ]
In4/Pass: [ Packets: 486576520 Bytes: 96514393117 ]
In4/Block: [ Packets: 47372 Bytes: 2174727 ]
Out4/Pass: [ Packets: 688292290 Bytes: 861156329162 ]
Out4/Block: [ Packets: 11 Bytes: 440 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 5 Bytes: 368 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
em1
Cleared: Thu May 29 16:22:04 2014
References: [ States: 1373 Rules: 9 ]
In4/Pass: [ Packets: 697418872 Bytes: 862404250194 ]
In4/Block: [ Packets: 41437 Bytes: 1708569 ]
Out4/Pass: [ Packets: 497496073 Bytes: 98268604850 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 4 Bytes: 304 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
enc
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
enc0
Cleared: Thu May 29 16:22:04 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
lo
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
lo0
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 12 Rules: 5 ]
In4/Pass: [ Packets: 197990 Bytes: 156023856 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 197990 Bytes: 156023856 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
pflog
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
pflog0
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
pfsync
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
pfsync0 (skip)
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]
plip0
Cleared: Tue Aug 5 12:07:12 2014
References: [ States: 0 Rules: 1 ]
In4/Pass: [ Packets: 0 Bytes: 0 ]
In4/Block: [ Packets: 0 Bytes: 0 ]
Out4/Pass: [ Packets: 0 Bytes: 0 ]
Out4/Block: [ Packets: 0 Bytes: 0 ]
In6/Pass: [ Packets: 0 Bytes: 0 ]
In6/Block: [ Packets: 0 Bytes: 0 ]
Out6/Pass: [ Packets: 0 Bytes: 0 ]
Out6/Block: [ Packets: 0 Bytes: 0 ]