Multi WAN seems to be poorly implemented

frater · Dec 15, 2011, 12:18 PM

I'm using pfsense 2.0 in a multi-LAN / multi-WAN scenario.
Quite often, especially on https connections (shopping sites or our own hosting server) we're getting kicked.
This is typical behaviour for badly implemented multi-WAN which I didn't expect on pfsense.

My general feeling of pfsense is that it's robust and flexible, but this is a bit of a downside….

I believe it's a bug that has been reported a long time ago and hasn't been solved yet.
I don't understand why it takes that long and even less why pfsense 2.0 is considered to be stable.

For the time being I created a gateway group where the different gateways are in different tiers.
I'm using that gateway group for port 80,443 and some other ports on which we're using http-traffic.

To avoid load-balancing problems it would be best to always use the same gateway for the same IP for at least several hours.

Any info regarding this issue is welcome.
I would like to know if anyone is at least trying to solve it.

Perry · Dec 15, 2011, 12:34 PM

Did it also happen with sticky connection enable?

cmb · Dec 15, 2011, 6:30 PM

Best to never load balance HTTPS, but sticky connections will work around that most of the time.

frater · Dec 16, 2011, 3:07 PM

Yes, sticky connections is/was enabled….

But, what's that bug that's mentioned in the list?
I haven't read about it being solved.

cmb · Dec 16, 2011, 4:33 PM

It's not an issue in 2.0 with multi-WAN.

frater · Dec 19, 2011, 4:47 PM

Horde gives me the explicit message that I'm coming from another IP all of a sudden….
What does "sticky" mean?
I mean... what's the timeframe?

jimp · Dec 19, 2011, 6:11 PM

Sticky keeps a client_ip+gateway_ip association for as long as a connection state is active. So as long as the browser holds the connection open.

On 2.1 I have added a box to let the user specify the length of time after the state expires to hold the association, so it can be made longer.

Supermule · Dec 19, 2011, 6:47 PM

When does 2.1 go live??

jimp · Dec 19, 2011, 6:49 PM

It will probably be sometime after March for that. Still lots of work to go there.

2.0.1 will be coming out shortly (maybe today) but it doesn't include that change.

dhatz · Dec 19, 2011, 7:24 PM

jimp,

Is the upcoming change to "sticky" in pfsense v2.1 still using pf's "sticky-address" feature?

In the *bsd mailing lists there have been several requests over the years to increase the sticky timeout to e.g. 30min, but no clear solution.

jimp · Dec 19, 2011, 7:25 PM

Yes, it is done the same way. The timeout for that is controller by pf's src.track timer (see pfctl -st)

dhatz · Dec 19, 2011, 7:46 PM

Thx jimp.

What would happen in a load-balancing pfsense setup with two gateways with sticky enabled, if one gateway failed? What would prevent pf from remembering the old (finished) sessions tracks for src.track (e.g. 30 minutes) and keep sending traffic its way?

I am having in mind the issue raised in
http://lists.freebsd.org/pipermail/freebsd-pf/2006-December/002860.html
http://lists.freebsd.org/pipermail/freebsd-pf/2006-December/002861.html

In that example they suggest using pfctl -K to remove source tracking entries.

jimp · Dec 19, 2011, 7:51 PM

I believe the way our multi-wan works none of that info applies, as it does some behind-the-scenes magic when a WAN fails that may be doing the right thing here already. Ermal would have to say for sure though.

frater · Dec 20, 2011, 4:03 PM

But what code does that checkbox execute?
Can I do something from the console….?

dhatz · Dec 20, 2011, 7:14 PM

To see the code, do a search, e.g.

fgrep lb_use_sticky /etc/inc/*

Basically it enables the "sticky-address" feature of pf.

jimp · Dec 20, 2011, 7:38 PM

The code triggered by the checkbox to activate sticky can be found in /etc/inc/interfaces.inc

As for the GUI field to set the timeout manually, here is what does that:
https://github.com/bsdperimeter/pfsense/commit/4573641589d50718b544b778cea864cfd725078a

frater · Dec 21, 2011, 9:10 AM

I really ment…
can I write some value to some pseudo-file in /proc or doesn't it work that way?

The checkbox / input field in the php-site does something...
Can you tell us what it does or can this be backported to 2.0

I'm insisting because it's a bit of a show stopper for my implementation....

jimp · Dec 21, 2011, 12:30 PM

Read my last post again, I provided everything you'd need to know to make the change on 2.0.1 by hand.

frater · Dec 21, 2011, 8:07 PM

Did you edit your previous post?….
Anyhow

I applied those 2 patches (I wasn't able to create a patch file from it, but downloaded the 2 raw files with fetch)
I set the timeout to 3600 seconds which should be sufficient.
If this works I will have the best of both worlds (equal spreading of bandwidth and stability)

Tomorrow I will know if it does its job better.

dhatz · Dec 28, 2011, 1:31 PM

@frater:

Tomorrow I will know if it does its job better.

So, what's your impression of the new src.track feature (i.e. sticky with a timeout)?

And have you tested how it handles failure of one of the two gateways? (see my question in the previous page)

TIA.