HAProxy with 1 or 2 frontends and dual wan - cookies not working on wan2
-
Using pfSense 2.2.1 (dev) and either HAProxy-dev or full. I've been beating my head against a wall here. This is a test environment that I am trying to set up so I can see how it will function when we implement this into a live environment.
I have 2 wan connections from 2 different providers. Wan failover is set up and working correctly. When I take out one of the wan connections, the second one gets set to the default gateway, my Route 53 DNS record gets updated, and now everything comes in on the second provider.
I have HAProxy set up with a frontend and a backend that has 2 servers in it. I have it set up to insert a cookie via insert indirect. This works great. A cookie is inserted, I see it, and it's getting sent back and I can verify that it's sending it to the same server each time. Works great. Until the Wan connection fails over to the second provider. The cookie is not being inserted into the response anymore.
I've tried with a single front end that listens on both public IPs, I've tried a single front end just listening on localhost, then I tried 2 front ends each listening to a single public IP. In all instances, I can get to the primary wan connection with a cookie inserted every time, but I can never get one inserted on the secondary wan connection.
A search of the documentation for HAProxy showed that I can specify an interface option on the bind, but when I do that, pfsense reports that the interface option was not compiled into haproxy.So, I'm guessing that HAProxy is only working with the primary Wan interface? I can probably verify this by reinstalling pfsense and/or changing which provider is on the first wan interface… Anyone have any ideas on how I can get around this?
An option I've been throwing around now is having 2 pfSense instances for this, but that would mean splitting our pool into 2 pools and having one pool for one provider and another pool for the second provider. I'd rather not do that because if one provider goes down, that's going to put a tremendous load on a pool.
Any advice or even tips on what to try next would be appreciated.
Here is the haproxy.cfg that is generated - http://pastebin.com/MQC1G2ub
The error I get when I use the Interface option in the bind command: Errors found while starting haproxy [ALERT] 075/090747 (49958) : parsing [/var/etc/haproxy_test/haproxy.cfg:32] : 'bind x.x.x.x:80' : 'interface' option is not implemented in this version (check build options). [ALERT] 075/090747 (49958) : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg [ALERT] 075/090747 (49958) : Fatal errors found in configuration.
-
pfSense runs on FreeBSD see below text from the manual:
–---
interface <interface>Restricts the socket to a specific interface. When specified, only packets
received from that particular interface are processed by the socket. This is
currently only supported on Linux.
–---Why do you have 2 frontends? A single frontend can also listen on multiple ip's or 'any'.
Another option could be to bind it to localhost and use a portforward on both wan interfaces to forward to 127.0.0.1:XYZ.
Even so the cookie insert option should work either way..
BUT the indirect option does mean the cookie is not inserted if the browser did send a usable cookie in the request. Are you sure the browser didn't send a valid cookie?</interface>
-
pfSense runs on FreeBSD see below text from the manual:
–---
interface <interface>Restricts the socket to a specific interface. When specified, only packets
received from that particular interface are processed by the socket. This is
currently only supported on Linux.
–---Why do you have 2 frontends? A single frontend can also listen on multiple ip's or 'any'.
Another option could be to bind it to localhost and use a portforward on both wan interfaces to forward to 127.0.0.1:XYZ.
Even so the cookie insert option should work either way..
BUT the indirect option does mean the cookie is not inserted if the browser did send a usable cookie in the request. Are you sure the browser didn't send a valid cookie?</interface>
I originally had only 1 frontend (like the title says) and tried 2 just in case that's what was wrong. Thank you for the information about the interface option. I didn't see that it was only on linux.
As stated in the post, I did have it on localhost as well and it didn't work on the second wan either.
I am 1000% sure that the browser is not sending a cookie with the request. I am using fiddler and wireshark to look at the streams, requests, and responses.
-
This isn't a problem. I figured out what I was doing wrong and it had nothing to do with the HAProxy or pfsense setup.
-
Could you share the underlying problem & solution ? Perhaps others running into the same situation can benefit from the solution you found.?
-
Could you share the underlying problem & solution ? Perhaps others running into the same situation can benefit from the solution you found.?
I'm embarrassed to say…. :-X
I had another load balancer set up on another pfsense install and I was hitting that IP instead of the haproxy server IP... ::)
-
Ok thanks for sharing. Sometimes those problems are the most difficult, where a small assumption was made incorrect.