HAProxy on 1.2.3-RELEASE - Successful!!!

rkelleyrtp

Rock solid with no problems, but I am running no where near 100Mbit speeds (yet). We have a pair of web servers getting hit about 820K connections per day, and I have "monit" monitoring the haproxy daemon on the pfSense boxes. To date, I have seen no issues with haproxy. Both pfSense servers are running less than 5% CPU utilization (2.6Ghz QuadCore CPU, 4G RAM, Intel GigE NICs running 1.2.3-RELEASE).

Hope this helps.

kamus

Hey @rkelleyrtp thanks for your last reply and by the way greate job!. Well I can tell me we are working/planning our cluster/failover for our company under pfsense (1.2.3 release) and just right now are bought some servers with identical specs. I would like to question two thinks, one is why we can't use remote syslog? because we didn't have another form to check activity of haproxy, local logs do not register nothing about it (this take me to second point) and second, I will try to use haproxy statistics but always got an 404 error (not found), if you can explain me a litle bit detailed how we can enable and configure correctly these option we would be so appreciated.

Best regards ;)

rkelleyrtp

Hi. To make this happen, you need to do the following things:

Configure/enable a central syslog server (linux, etc) accessible via your LAN segment.
Configure your pfsense server to send all logs to a remote syslog server.
Login via SSH (as admin) on your pfSense machine
Edit the /etc/syslog file
Comment out all lines starting with "local"
Add a single like like this:


# -----------------------------------------------
# Send all local.* logs to remote syslog server
# -----------------------------------------------
local.*					@192.168.100.25

Next, edit your haproxy configuration (via pfSense GUI) and make sure the logging section is set for your remote syslog server

NOTE, the last time I tried this, haproxy would not start because the GUI does not supply the proper syntax for remote syslog servers. The workaround is to hand-edit the haproxy.cfg file, make the correction, then manually restart haproxy. If you have a failover pair of pfSense boxes, you will need to update this on both servers. If you make changes to your haproxy configuration later you, you will need to hand-edit the file again.

As to why you are unable to to get haproxy stats, I don't know because I don't use that feature. Your best bet would be to look at the haproxy main website - perhaps someone over there can help you.

rumptis

I installed it today without doing anything special and it seems to be working just fine so far.

I have it setup for a transparent proxy.

The only thing that doesn't seem to be working is there doesn't seem to be a Antivirus log I can find, I have logging enabled. I don't know if that worked before though, this is the first time I have used it.

This was also on a fresh install with no other packages installed yet.

rkelleyrtp

Just found some additional information.

By default, the haProxy configuration file hard-codes a value of ONE SECOND when checking the status of your backend servers
The configuration file also uses a variable named "$numprocs" to identify how many haProxy threads get run. This value is derived from the number of CPUs installed in your system

While these may not seem like major issues, these default values can wreak havoc with your back-end servers. In our case, we have two pfSense machines running haProxy for a web service with two back-end servers. Our firewalls have 4 CPUs each. Using the default configuration, EACH web sever was getting hit 8 times per second with haproxy status check queries ("HEAD /index.html HTTP/1.0" 200") causing noticeable performance issues with our application (and accounting logs).

Unfortunately, the current version of the haProxy package does not allow you to change these values. And, if you hand-modify the haproxy.cfg file, the file will get overwritten when you restart the haproxy service.

The only way to change these values is to modify the /usr/local/pkg/haproxy.inc file

To change the timeout for checking your back-end servers, look for "1000" (the only line with 1000) and change it accordingly. I changed mine to 10000
To change the number of threads, look for the line like "numprocs = trim(/sbin/sysctl kern.smp.cpus | cut -d" " -f2)" - comment it out using the "# sign at the front, then add a line below specifying the number you want. In my case, I added a new line that read, "$numprocs=1;" (don't forget the trailing semi-colon).

After making these changes, I restarted the haproxy service (/usr/local/etc/rc.d/haproxy.sh restart) and immediately noticed the log files on our back-end servers cleaned up quite a bit.

Hopefully this makes sense. Please let me know if you have any questions…

BTW - I really like pfSense and the haproxy package. Both are rock-solid applications and I swear by them (I put my name on the line every day). My goal is to help improve this project by providing as much feedback as I can.

kamus

thank you so much for shared this information, would be nice if in future versions of haproxy could be able to change these values. by the way I have noticed that current package, service restart function do not reload haproxy, I have to do manually and then run start from web interface and well, menwhile, I will continue working to up haproxy statistics in web interface. regards

rkelleyrtp

No problem - glad to help.

One clarification to my earlier post - it appears you CAN set the value when checking the status of your backend servers. I completely missed it! Look for the "Check inter" entry box on the Servers tab (double click on the server name then look toward the bottom). In fact, you can set this value per server.

BTW - I am hoping to catch up with the author of haProxy so we can discuss some of these issues. I ran into another bug this morning…

rkelleyrtp

@kamus:

by the way I have noticed that current package, service restart function do not reload haproxy, I have to do manually and then run start from web interface and well, menwhile, I will continue working to up haproxy statistics in web interface. regards

Make sure the /usr/local/etc/rc.d/haproxy.sh file has the correct "enable" setting. By default, it is disabled and won't start. Edit the file and make sure the line reads:

haproxy_enable=${haproxy-"YES"}

That should fix your start/stop problems.

kamus

@rkelleyrtp:

No problem - glad to help.

One clarification to my earlier post - it appears you CAN set the value when checking the status of your backend servers. I completely missed it! Look for the "Check inter" entry box on the Servers tab (double click on the server name then look toward the bottom). In fact, you can set this value per server.

BTW - I am hoping to catch up with the author of haProxy so we can discuss some of these issues. I ran into another bug this morning…

great! heheh. I have been searching "Check Inter" entrybox but I can't see it, maybe we are using different versions (currently I'm working on 1.2.3 and haproxy 0.26)
–---------------
I haven't see it that a new version has been released! hehe, now I can see it :). I will try some tests with this new version. Thanks!

Itwerx

Quick question for anyone hosting multiple websites, (and web servers). It appears that if two back-end servers are used that there is a bug in the way the inbound requests are handled? (FYI we are using multi-WAN config and CARP).

Here's an example:

Frontends:
abcco.com-WAN1
xyzco.com-WAN1
abcco.com-WAN2
xyzco.com-WAN2

Servers:
abcco.com-DMZ 10.1.1.1: active abcco.com-WAN1 1
abcco.com-WAN2
abcco.com-DMZ 10.2.2.1: active xyzco.com-WAN1 1
xyzco.com-WAN2

The home page of each site is proxied properly but pages linked within the site all seem to go to one back-end server, (and get 404s if they're for the site not hosted on that server of course.)

Ideas?

rkelleyrtp

Can you post your full haproxy config file?

For what its worth, I changed our config to host two separate back-end clusters and it has been working beautifully. Here is our config:

more haproxy.cfg

global
maxconn 1000
uid 80
gid 80
nbproc 1
chroot /var/empty
daemon

listen Cluster1
bind 20.30.40.51:80
mode http
log global
option dontlognull
option forwardfor
maxconn 10000
clitimeout 3000
balance roundrobin
contimeout 6000
srvtimeout 6000
retries 2
option httpchk HEAD /index.html HTTP/1.0
server WEBSVR-1 10.1.1.11:80 check inter 10000 weight 1
server WEBSVR-2 10.1.1.12:80 check inter 10000 weight 1

listen Cluster2
bind 20.30.40.52:80
mode http
log global
option dontlognull
option forwardfor
maxconn 10000
clitimeout 3000
balance roundrobin
contimeout 6000
srvtimeout 6000
retries 2
option httpchk HEAD /index.html HTTP/1.0
server WEBSVR-3 10.1.1.21:80 check inter 10000 weight 1
server WEBSVR-410.1.1.22:80 check inter 10000 weight 1

BTW - if you look closely, you will see a global option called "nbproc". The current pfsense haproxy package calculates this based upon the # of CPUs on your firewall, and spawns an haproxy thread for each CPU. As I mentioned earlier in this thread, this can cause undue logging to your back-end servers. Just something to keep in mind…

Itwerx

@rkelleyrtp
Except those are listening on different public IPs. (Sorry, I should have clarified this is for a single public IP address. :)
To clarify further, we currently are using Apache reverse proxy to handle inbound requests to a single IP address assigned to multiple sites. Apache does a good job of reading the site address out of the request header and redirecting to the correct internal server. Only problem is it is slo-o-ow, (and HAproxy is rly-fst! :)
Is this not a good job for HAproxy…?

rkelleyrtp

Itwerx:

So, are you asking if haproxy can read the incoming http header and forward to the appropriate back-end server(s) when a single IP Address is used? I think what you are looking for are ACLs inside haproxy. Here is a quick link:

http://agiletesting.blogspot.com/2009/02/load-balancing-in-amazon-ec2-with.html

Essentially, you have a single IP address on the outside (your "listen" address) and multiple servers on the back-end. Since haproxy can read the http header request, it can make intelligent decisions as to where the requests should go. From that link above, you define your frontend section like this:

frontend myfrontend *:80
log global
maxconn 25000
option forwardfor
acl acl_example1 url_sub example1
acl acl_example2 url_sub example2
use_backend example1_farm if acl_example1
use_backend example2_farm if acl_example2
default_backend default_farm

This creates ACLs (access-control-lists) that match the incoming HTTP header. Next, setup your back-end section to route the requests:

backend example1_farm
mode http
balance roundrobin
server server1 192.168.1.1:80 check
server server2 192.168.1.2:80 check
backend example2_farm
mode http
balance roundrobin
server server3 10.0.0.3:80 check
server server4 10.0.0.4:80 check
backend default_farm
mode http
balance roundrobin
server server5 192.168.1.5:80 check
server server6 192.168.1.6:80 check

Hope this helps…

-Ron

Koti

Yup its working fine for me also.

SyRenity

Hi.

Can anyone using haproxy comment how well it compares to inbound Load Balancing?

What is the improvement in concurrent connections, used CPU, etc…

Thanks!

Itwerx

For anyone installing this on a clean load of pfSense there's actually one additional rule needed allowing HTTP (or whatever), in to the internal servers. I.e. the same rule you would have for a standard NAT even though there is no NAT defined. (This is contrary to what is stated in the first post in this thread.)

Arjen

Any support for full transparent proxy mode ? :

aka: source 0.0.0.0 use src client

Regards.

Arjen

sam_son

Single IP address - multiple back end servers.

Is it possible for me to do the following.

I thought it best to add it to this post rather than start another.

Basically the 192.168.5.2 server has a combination of both http & https for authentication. It is an appliance box and when initially logging in via an http url having done so this then changes to https.

I have no clue at this point how to proceed with the configuration so if someone could enlighten me with some configuration example that would be great. I appreciate you are all busy but getting the answer to this question would help my little office out no end.

I would ideally like if possible the url access to be

http://81.81.81.81/server1 to redirect to 192.168.5.1 back end server
http://81.81.81.81/server2 to redirect to 192.168.5.2 back end server

Is this similar to port forwarding or can haproxy forward all ports using the url. This may be on the vague side but I'm sure there will be someone equally as clueless as myself with haproxy.

![HAproxy single IP address.png](/public/imported_attachments/1/HAproxy single IP address.png)
![HAproxy single IP address.png_thumb](/public/imported_attachments/1/HAproxy single IP address.png_thumb)

sam_son

Ok having read the initial post would this be correct? I have a little test rig in the office and between my daily work I sometimes have time to tinker with this kit. I have been scouring google and not come up with anything concrete with regards to overcoming the https question. Other posts have mentioned about using stunnel but the link here

http://agiletesting.blogspot.com/2009/02/load-balancing-in-amazon-ec2-with.html

says

SSL

To handle SSL traffic in HAProxy, you need 3 things:

Define a frontend with a unique name which handles *:443
Send traffic to real_server_IP_1:443 through real_server_IP_N:443 in the backend(s) associated with the frontend
Specify 'mode tcp' instead of 'mode http' both in the frontend section and in the backend section(s) which handle port 443. Otherwise you won't see any SSL traffic hitting your real servers, and you'll wonder why….

------ Would the follwing be correct?

Configure HAProxy using default values except the following:
--> On Settings Tab:
--> Enable HAProxy
--> DO NOT USE REMOTE SYSLOG HOST (bug in current version of pfSense haproxy package)

--> On Frontend Tab:
--> 10.10.10.1 (fictitious VIP)

--> On Servers Tab:
--> 192.168.5.1
--> 192.168.5.2

Configure the firewall rule - very impt!
--> On the Firewall-->Rules tab
--> New rule -
--> Protocol=any
--> Source=Any
--> Port=80
--> Destination=CARP VIP 10.10.10.1
--> On the Firewall-->Rules tab
--> New rule -
--> Protocol=any
--> Source=Any
--> Port=443
--> Destination=CARP VIP 10.10.10.1

ACL
frontend myfrontend *:80
log global
maxconn 25000
option forwardfor
acl acl_example1 server1 example1
acl acl_example2 server2 example2
use_backend example1_farm if acl_example1
use_backend example2_farm if acl_example2
default_backend default_farm

I’m a little confused, will haproxy take the http header and pass both http&https requests.
I have assumed that I would need to add the 443 port rule to pfsense is this correct?

xeress

I'm also trying haproxy out. Though for some reason it's very slow looking at download speeds. A file of 2 mb is downloading with 15 kb/s thru haproxy though when i enter a simple NAT rule it's downloads with 1 MB/s . Latency on haproxy is very low (15ms).

For now it's an experimental cluster so no other traffic is going thru. Can someone help me out on this? HaProxy seems to be alot better then the build in load balancer looking at features.

(i do use a carp vip to the internet, and my webservers are using apache on a 192.168.2.0/24 network. All settings are pretty basic.)