Sticky connections not working with dual WAN
-
Well there's no major rush as I'm not exactly down so I'll just hang on for an update and hopefully, he'll see this soon.
Thanks for your help so far Johnpoz :)
-
If the application doesn't work with load balancing it doesn't work with load balancing.
That's pretty much what I have. Talk to the application side about accepting sessions from multiple IP addresses.
Chattanooga, Tennessee, USA
A comprehensive network diagram is worth 10,000 words and 15 conference calls.
DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
Do Not Chat For Help! NO_WAN_EGRESS(TM) -
I'm using this exact scenario, with dual wan, banking sites and quite a few users accessing them. No issues
I did have issues in the beginning and I had to raise stickiness to 2500.I also have raised the default weight to 2, so no line has a weight for 1.
I recall reading somewhere about an issue with load balancing, and this as a suggested workaround, but I can't recall it.In any case, it doesn't hurt anything to use a default weight of 2 and adjust smaller lines accordingly.
I'm on 2.4.5 and this also worked flawlesly on 2.4.4.p3
-
@Derelict I’m sorry but I don’t understand your reply.
The application does work with loadbalancing (Google Chrome, Microsoft Edge etc...) but the security of these websites being visited require that the IP doesn’t change. Isn’t that the exact purpose of sticky connections to work around this?
Plus if someone else is now reporting the issue surely it warrants being looked into?
Thank you.
-
Look at the states when you are connected. If there are two different IP addresses being connected to, but all connections to the same IP address use the same WAN, then load balancing is doing what it is designed to do and you will need to policy route all traffic for that application out the same WAN or Failover gateway group, not a load balance gateway group.
Chattanooga, Tennessee, USA
A comprehensive network diagram is worth 10,000 words and 15 conference calls.
DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
Do Not Chat For Help! NO_WAN_EGRESS(TM) -
^ great point... But my take on him saying his server was logging 2 different IPs connecting is that he was only connecting to 1 destination IPv4 address..
But your point is very valid for many of these sites that are hosted on cdn where www.whatever.com could end up being 2 different destination ips for the same site..
-
-
@Derelict I tried as you suggested.... killed all states, went to my own server and logged in via the website (as said the server only has 1 IP). I was almost immediately logged out so logged in again.
Checked the states and noticed it's using both WANs as suspected:
VLAN1_TRUSTED tcp 192.168.1.126:64519 -> 62.3.XXX.XXX:3334 TIME_WAIT:TIME_WAIT 8 / 8 2 KiB / 936 B
WAN1 tcp 217.45.XXX.XXX:8341 (192.168.1.126:64519) -> 62.3.XXX.XXX:3334 TIME_WAIT:TIME_WAIT 8 / 8 2 KiB / 936 B
VLAN1_TRUSTED tcp 192.168.1.126:64522 -> 62.3.XXX.XXX:3334 FIN_WAIT_2:FIN_WAIT_2 8 / 8 2 KiB / 4 KiB
WAN2 tcp 5.70.XXX.XXX:59341 (192.168.1.126:64522) -> 62.3.XXX.XXX:3334 FIN_WAIT_2:FIN_WAIT_2 8 / 8 2 KiB / 4 KiBSticky connections are on and the timeout is set to 1200.
Thanks.
-
Well your states are showing fin_wait.. and time_wait
Those states are being closed..
I would sniff this traffic and who is sending the fin?
-
I'm not sure what you mean? I'm the only person connecting.
It took me a few minutes to find those details in the states so that's probably why it shows they connections are closing. But I just grabbed any 2 connections in the logs showing it was using more than 1 WAN. There were many other lines of logs showing connections on both WANs.
These connections were made over the timeframe of 1 minute and after killing states so shouldn't there only be 1 WAN IP in the logs regardless?
-
@Daskew78 said in Sticky connections not working with dual WAN:
t took me a few minutes to find those details
You can filter states.. My point was that those states are closed..
This statement "Once the states for that source expire" means what exactly... If any state, even closed states that are just waiting to time out.. Or does that state have to actually be active?
This where I thought maybe @Derelict could help..
Lets look at this scenario... You create a connection to IP X, now that state has been set to be closed.. fin.. and you enter a time_wait state.. Is that state considered expired - so a new session which is what you show there from a different source port would that go out the same wan, or would it round robin to the other wan?
You could look at it both ways.. Since the the state is just waiting to close, and you have this new session coming fro a different source port maybe I should round robin that connection.. Or you could look at it as hey there is ANY state from IP your rfc1918 address to this public IP 62.3 - so always use that wan? I am not exactly sure how it is looked at?
I could see both ways being valid ways of looking at.. Hey this client has an active session to x, any new sessions it creates will go out the same wan.. Or hey this session is closed or closing... Since this is a new session "different source port.. Maybe it should go out the other wan to load balance.
-
@johnpoz said in Sticky connections not working with dual WAN:
w that state has been set to be closed.. fin.. and you enter a time_wait state.. Is that state considered expired - so a new session which is what you show there from a different source port would that go out the same wan, or would it round robin to the other wan?
Would you be willing to do a remote session with me and I can show you all the evidence? I really think there's a bug here.
-
I am not saying its not a bug or that there isn't a problem - I just don't know which specifics pfsense is using to know keep a connection sticky.. I made a bit of edit addition - on my previous post.
You can look at it both ways, I don't know exactly what "Once the states for that source expire" means.. Maybe once there has been a fin, that state is no longer looked at - I am not sure..
-
Well I'm at a loss as to what to do next.
I think it comes down to @Derelict needs to advise what further testing I can do or accept it may be a possible bug?
I hope he replies!
-
I still think is out for a bit, my understand is he wouldn't be back for a few more days... So his check into the thread was a bit unexpected to me..
We can see if @jimp has any advice as well.. This is just a bit out of my comfort level, since I do not use multiple wans in a load balancing setup.. I don't really see the point to it to be honest ;) If you need to load balance tells me your connections are undersized ;) hehehe
I have more experience with this sort of thing on fortinet load balancing to servers behind them, and how their sticky connections work.. And even then its not a day to day sort of thing, only get called into consult on issues - normally they give me sniffs to work with and help them figure out what is going wrong ;)
If you could show state that is clearly active, and then another state being opened - then I would agree that is not how I would understand sticky to work.
You know who might be good as well would be @stephenw10
-
@johnpoz I don't deny my upload is undersized of for my needs... it's the best i can get at the moment though until they upgrade the infrastructure around here. It has many other advantages though such as redundancy.
Hopefully one of the people you tagged can chip in :)
I do appreciate all the help so far... thanks pal!
-
Sure failover I get. but that wouldn't need to be a in load balancing setup to do that ;) heheh
What I would suggest is try and validate if this other connection is being created after original state is closed.. You could just sniff on your client.. Do you see or send a fin at any time?
And that is when the wan changes.
-
It's getting a bit above me this now which is why I was hoping I could let you teamviewer in and maybe take a look?
You'd have all the answers in 5 minutes rather than going back and forth through this monkey ;)
-
@Daskew78 You shouldnt really care about states being closed when you have a stickiness of 1200.
As documentation says, if you have stickiness 0, then load balancing path is re evaluated when connectios are closed. (and we could discuss if this means fin wait etc)
But stickiness of 1200 Means 1200 seconds AFTER connections is closed, if a new request comes from the same ip to the same host it will leave from the same gateway.I insist. stickiness works fine on multiwan ssl load balancing scenario.
And consider this workaround too
https://redmine.pfsense.org/issues/6025quoting
Also of note, when the weights differ, even though the gateways have a specific order with repetition in the rule, pf seems to still flip back and forth, though the general ratio of the weights is respected. For example with WAN1=3, WAN2=2:I had the same issues as you do until I made 2 the default weight on both load balancing connections.
Deeper issues are suspected, as redmine says.
Please consider testing the workaround. -
Yeah sure seems that issue is exactly what your seeing... I would do what @netblues says and that should fix up your issue I would hope.
