Multi WAN seems to be poorly implemented

frater

Horde gives me the explicit message that I'm coming from another IP all of a sudden….
What does "sticky" mean?
I mean... what's the timeframe?

jimp

Sticky keeps a client_ip+gateway_ip association for as long as a connection state is active. So as long as the browser holds the connection open.

On 2.1 I have added a box to let the user specify the length of time after the state expires to hold the association, so it can be made longer.

Supermule

When does 2.1 go live??

jimp

It will probably be sometime after March for that. Still lots of work to go there.

2.0.1 will be coming out shortly (maybe today) but it doesn't include that change.

dhatz

jimp,

Is the upcoming change to "sticky" in pfsense v2.1 still using pf's "sticky-address" feature?

In the *bsd mailing lists there have been several requests over the years to increase the sticky timeout to e.g. 30min, but no clear solution.

jimp

Yes, it is done the same way. The timeout for that is controller by pf's src.track timer (see pfctl -st)

dhatz

Thx jimp.

What would happen in a load-balancing pfsense setup with two gateways with sticky enabled, if one gateway failed? What would prevent pf from remembering the old (finished) sessions tracks for src.track (e.g. 30 minutes) and keep sending traffic its way?

I am having in mind the issue raised in
http://lists.freebsd.org/pipermail/freebsd-pf/2006-December/002860.html
http://lists.freebsd.org/pipermail/freebsd-pf/2006-December/002861.html

In that example they suggest using pfctl -K to remove source tracking entries.

jimp

I believe the way our multi-wan works none of that info applies, as it does some behind-the-scenes magic when a WAN fails that may be doing the right thing here already. Ermal would have to say for sure though.

frater

But what code does that checkbox execute?
Can I do something from the console….?

dhatz

To see the code, do a search, e.g.

fgrep lb_use_sticky /etc/inc/*

Basically it enables the "sticky-address" feature of pf.

jimp

The code triggered by the checkbox to activate sticky can be found in /etc/inc/interfaces.inc

As for the GUI field to set the timeout manually, here is what does that:
https://github.com/bsdperimeter/pfsense/commit/4573641589d50718b544b778cea864cfd725078a

frater

I really ment…
can I write some value to some pseudo-file in /proc or doesn't it work that way?

The checkbox / input field in the php-site does something...
Can you tell us what it does or can this be backported to 2.0

I'm insisting because it's a bit of a show stopper for my implementation....

jimp

Read my last post again, I provided everything you'd need to know to make the change on 2.0.1 by hand.

frater

Did you edit your previous post?….
Anyhow

I applied those 2 patches (I wasn't able to create a patch file from it, but downloaded the 2 raw files with fetch)
I set the timeout to 3600 seconds which should be sufficient.
If this works I will have the best of both worlds (equal spreading of bandwidth and stability)

Tomorrow I will know if it does its job better.

sticky.pfsense.png
sticky.pfsense.png_thumb

dhatz

@frater:

Tomorrow I will know if it does its job better.

So, what's your impression of the new src.track feature (i.e. sticky with a timeout)?

And have you tested how it handles failure of one of the two gateways? (see my question in the previous page)

TIA.

frater

Because there was still one site that gave problems I didn't dare to post immediately.
But I never had any problems accessing the 2 hosting servers of our own.

I think it's working…
I assume it's properly working when your line goes down. The behaviour will depend a lot on what it should do with the current states (flush them or keep them open).

I just updated to 2.01, so I needed to patch it again.
I now did it in a way that is easy to share with the community.
It's a bit less work now....

I do think it should have been backported to 2.01

Cheers and happy new year


cd /usr/local/www
cp -p system_advanced_misc.php system_advanced_misc.php.2.01
patch system_advanced_misc.php <system_advanced_misc.php.patch<br>cd /etc/inc
cp -p filter.inc filter.inc.201
patch filter.inc <filter.inc.patch< pre="">/etc/inc/filter.inc.patch

281a282,284

if (isset($config['system']['lb_use_sticky']) && is_numeric($config['system']['srctrack']) && ($config['system']['srctrack'] > 0))
$rules .= "set timeout src.track {$config['system']['srctrack']}\n";


/usr/local/www/system_advanced_misc.php.patch

58a59

$pconfig['srctrack'] = $config['system']['srctrack'];
105c106
< if($_POST['lb_use_sticky'] == "yes")

if($_POST['lb_use_sticky'] == "yes") {
107c108,109
< else

$config['system']['srctrack'] = $_POST['srctrack'];
} else
192a195,200
function sticky_checked(obj) {
if (obj.checked)
jQuery('#srctrack').attr('disabled',false);
else
jQuery('#srctrack').attr('disabled','true');
}
269c277
< />

onClick="sticky_checked(this)" />
278a287,292

" class="formfld unknown" >

> "By default this is 0, so source tracking is removed as soon as the state expires. " .
"Setting this timeout higher will cause the source/destination relationship to persist for longer periods of time."); ?>
387c401
< ---

frater

I upgraded to 2.01 and re-applied the patch….
It's not working :-(

I am getting kicked from our hosting server from time to time. 'tcpdump' is running at the same time and as you can see it all of a sudden switches to the other connection.
This is not sticky at all......

I did think it was working properly before I upgraded, although I did get some strange behaviour on other servers....

Can I check if srctrack is really set properly?
I have little knowledge of FreeBSD / pfilter

tcpdump -i eth0 -n 'port 8443'

14:33:33.041319 IP 89.250.179.117.29754 > 46.243.24.60.pcsync-https: P 1949:2786(837) ack 2746 win 16425
14:33:33.041658 IP 46.243.24.60.pcsync-https > 89.250.179.117.29754: . 2746:4206(1460) ack 2786 win 96
14:33:33.041676 IP 46.243.24.60.pcsync-https > 89.250.179.117.29754: P 4206:4254(48) ack 2786 win 96
14:33:33.092396 IP 89.250.179.117.19784 > 46.243.24.60.pcsync-https: . ack 98209 win 16425
14:33:33.102207 IP 89.250.179.117.19784 > 46.243.24.60.pcsync-https: P 8928:9765(837) ack 98209 win 16425
14:33:33.102225 IP 89.250.180.164.13586 > 46.243.24.60.pcsync-https: S 929859587:929859587(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.102315 IP 46.243.24.60.pcsync-https > 89.250.180.164.13586: S 1120370230:1120370230(0) ack 929859588 win 5840 <mss 7="" 1460,nop,nop,sackok,nop,wscale="">14:33:33.102667 IP 46.243.24.60.pcsync-https > 89.250.179.117.19784: . 98209:99669(1460) ack 9765 win 228
14:33:33.102681 IP 46.243.24.60.pcsync-https > 89.250.179.117.19784: P 99669:99685(16) ack 9765 win 228
14:33:33.108718 IP 89.250.180.164.38584 > 46.243.24.60.pcsync-https: S 1307358421:1307358421(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.108751 IP 46.243.24.60.pcsync-https > 89.250.180.164.38584: S 2972341496:2972341496(0) ack 1307358422 win 5840 <mss 7="" 1460,nop,nop,sackok,nop,wscale="">14:33:33.114151 IP 89.250.180.164.16897 > 46.243.24.60.pcsync-https: S 2970339047:2970339047(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.114183 IP 46.243.24.60.pcsync-https > 89.250.180.164.16897: S 1688705918:1688705918(0) ack 2970339048 win 5840 <mss 7="" 1460,nop,nop,sackok,nop,wscale="">14:33:33.120833 IP 89.250.180.164.48746 > 46.243.24.60.pcsync-https: S 794117234:794117234(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.120865 IP 46.243.24.60.pcsync-https > 89.250.180.164.48746: S 3021423565:3021423565(0) ack 794117235 win 5840 <mss 7="" 1460,nop,nop,sackok,nop,wscale="">14:33:33.126519 IP 89.250.180.164.52196 > 46.243.24.60.pcsync-https: S 1506349235:1506349235(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.126550 IP 46.243.24.60.pcsync-https > 89.250.180.164.52196: S 3390423845:3390423845(0) ack 1506349236 win 5840</mss></mss></mss></mss></mss></mss></mss></mss></mss>

again…. can't this feature be backported?????

dhatz

@frater:

Can I check if srctrack is really set properly?

You can check if there is a rule. From shell prompt run:

pfctl -sr | fgrep src.track

jimp

Also check:

pfctl -st

To see the current timer values.

frater

pfctl -st | grep src.track
src.track 1800s
pfctl -sr | grep src.track

So, I guess I didn't implement the patch properly this time….

Can you supply a patch file or just 2 the files already patched?