HAProxy on 1.2.3-RELEASE - Successful!!!

rkelleyrtp

Greetings all. Today, I was successful in getting haproxy (v1.3.22) working on pfSense 1.2.3-RELEASE. I scoured the forums looking for a how-to and came up empty. So today, I spent some time getting this working and thought I would write what I did. It took a while (there is a bug in the remote syslog setting), but I am now able to leverage my pfSense box as an haproxy front-end to my backend servers. In summary, here is what I did:

Create a new VIP for the front-end server - this MUST BE A CARP ADDRESS otherwise haproxy will not start! Even if you don't have a standby pfSense firewall, you MUST create the VIP using the CARP setting (use bogus password as needed)
Download and install the haproxy package on your pfSense machine(s)
Configure HAProxy using default values except the following:
–> On Settings Tab:
--> Enable HAProxy
--> DO NOT USE REMOTE SYSLOG HOST (bug in current version of pfSense haproxy package)

--> On Frontend Tab:
--> External address of the VIP you created before

--> On Servers Tab:
--> Add your backend server(s) as necessary

Configure the firewall rule - very impt!
--> On the Firewall-->Rules tab
--> New rule -
--> Protocol=any
--> Source=Any
--> Port=80
--> Destination=CARP VIP <very important!!!>

After you save everything, you should be able to now leverage haproxy with your pfSense box(es). Note that no NAT addresses are required for this to work properly. I will follow-up this thread with some screenshots later.

Abacabb

Curious how much traffic you have going through haproxy.

Me after doing 100 mbit/s for a while, haproxy starts crashing. How is it running for you ?

rkelleyrtp

Rock solid with no problems, but I am running no where near 100Mbit speeds (yet). We have a pair of web servers getting hit about 820K connections per day, and I have "monit" monitoring the haproxy daemon on the pfSense boxes. To date, I have seen no issues with haproxy. Both pfSense servers are running less than 5% CPU utilization (2.6Ghz QuadCore CPU, 4G RAM, Intel GigE NICs running 1.2.3-RELEASE).

Hope this helps.

kamus

Hey @rkelleyrtp thanks for your last reply and by the way greate job!. Well I can tell me we are working/planning our cluster/failover for our company under pfsense (1.2.3 release) and just right now are bought some servers with identical specs. I would like to question two thinks, one is why we can't use remote syslog? because we didn't have another form to check activity of haproxy, local logs do not register nothing about it (this take me to second point) and second, I will try to use haproxy statistics but always got an 404 error (not found), if you can explain me a litle bit detailed how we can enable and configure correctly these option we would be so appreciated.

Best regards ;)

rkelleyrtp

Hi. To make this happen, you need to do the following things:

Configure/enable a central syslog server (linux, etc) accessible via your LAN segment.
Configure your pfsense server to send all logs to a remote syslog server.
Login via SSH (as admin) on your pfSense machine
Edit the /etc/syslog file
Comment out all lines starting with "local"
Add a single like like this:


# -----------------------------------------------
# Send all local.* logs to remote syslog server
# -----------------------------------------------
local.*					@192.168.100.25

Next, edit your haproxy configuration (via pfSense GUI) and make sure the logging section is set for your remote syslog server

NOTE, the last time I tried this, haproxy would not start because the GUI does not supply the proper syntax for remote syslog servers. The workaround is to hand-edit the haproxy.cfg file, make the correction, then manually restart haproxy. If you have a failover pair of pfSense boxes, you will need to update this on both servers. If you make changes to your haproxy configuration later you, you will need to hand-edit the file again.

As to why you are unable to to get haproxy stats, I don't know because I don't use that feature. Your best bet would be to look at the haproxy main website - perhaps someone over there can help you.

rumptis

I installed it today without doing anything special and it seems to be working just fine so far.

I have it setup for a transparent proxy.

The only thing that doesn't seem to be working is there doesn't seem to be a Antivirus log I can find, I have logging enabled. I don't know if that worked before though, this is the first time I have used it.

This was also on a fresh install with no other packages installed yet.

rkelleyrtp

Just found some additional information.

By default, the haProxy configuration file hard-codes a value of ONE SECOND when checking the status of your backend servers
The configuration file also uses a variable named "$numprocs" to identify how many haProxy threads get run. This value is derived from the number of CPUs installed in your system

While these may not seem like major issues, these default values can wreak havoc with your back-end servers. In our case, we have two pfSense machines running haProxy for a web service with two back-end servers. Our firewalls have 4 CPUs each. Using the default configuration, EACH web sever was getting hit 8 times per second with haproxy status check queries ("HEAD /index.html HTTP/1.0" 200") causing noticeable performance issues with our application (and accounting logs).

Unfortunately, the current version of the haProxy package does not allow you to change these values. And, if you hand-modify the haproxy.cfg file, the file will get overwritten when you restart the haproxy service.

The only way to change these values is to modify the /usr/local/pkg/haproxy.inc file

To change the timeout for checking your back-end servers, look for "1000" (the only line with 1000) and change it accordingly. I changed mine to 10000
To change the number of threads, look for the line like "numprocs = trim(/sbin/sysctl kern.smp.cpus | cut -d" " -f2)" - comment it out using the "# sign at the front, then add a line below specifying the number you want. In my case, I added a new line that read, "$numprocs=1;" (don't forget the trailing semi-colon).

After making these changes, I restarted the haproxy service (/usr/local/etc/rc.d/haproxy.sh restart) and immediately noticed the log files on our back-end servers cleaned up quite a bit.

Hopefully this makes sense. Please let me know if you have any questions…

BTW - I really like pfSense and the haproxy package. Both are rock-solid applications and I swear by them (I put my name on the line every day). My goal is to help improve this project by providing as much feedback as I can.

kamus

thank you so much for shared this information, would be nice if in future versions of haproxy could be able to change these values. by the way I have noticed that current package, service restart function do not reload haproxy, I have to do manually and then run start from web interface and well, menwhile, I will continue working to up haproxy statistics in web interface. regards

rkelleyrtp

No problem - glad to help.

One clarification to my earlier post - it appears you CAN set the value when checking the status of your backend servers. I completely missed it! Look for the "Check inter" entry box on the Servers tab (double click on the server name then look toward the bottom). In fact, you can set this value per server.

BTW - I am hoping to catch up with the author of haProxy so we can discuss some of these issues. I ran into another bug this morning…

rkelleyrtp

@kamus:

by the way I have noticed that current package, service restart function do not reload haproxy, I have to do manually and then run start from web interface and well, menwhile, I will continue working to up haproxy statistics in web interface. regards

Make sure the /usr/local/etc/rc.d/haproxy.sh file has the correct "enable" setting. By default, it is disabled and won't start. Edit the file and make sure the line reads:

haproxy_enable=${haproxy-"YES"}

That should fix your start/stop problems.

kamus

@rkelleyrtp:

No problem - glad to help.

One clarification to my earlier post - it appears you CAN set the value when checking the status of your backend servers. I completely missed it! Look for the "Check inter" entry box on the Servers tab (double click on the server name then look toward the bottom). In fact, you can set this value per server.

BTW - I am hoping to catch up with the author of haProxy so we can discuss some of these issues. I ran into another bug this morning…

great! heheh. I have been searching "Check Inter" entrybox but I can't see it, maybe we are using different versions (currently I'm working on 1.2.3 and haproxy 0.26)
–---------------
I haven't see it that a new version has been released! hehe, now I can see it :). I will try some tests with this new version. Thanks!

Itwerx

Quick question for anyone hosting multiple websites, (and web servers). It appears that if two back-end servers are used that there is a bug in the way the inbound requests are handled? (FYI we are using multi-WAN config and CARP).

Here's an example:

Frontends:
abcco.com-WAN1
xyzco.com-WAN1
abcco.com-WAN2
xyzco.com-WAN2

Servers:
abcco.com-DMZ 10.1.1.1: active abcco.com-WAN1 1
abcco.com-WAN2
abcco.com-DMZ 10.2.2.1: active xyzco.com-WAN1 1
xyzco.com-WAN2

The home page of each site is proxied properly but pages linked within the site all seem to go to one back-end server, (and get 404s if they're for the site not hosted on that server of course.)

Ideas?

rkelleyrtp

Can you post your full haproxy config file?

For what its worth, I changed our config to host two separate back-end clusters and it has been working beautifully. Here is our config:

more haproxy.cfg

global
maxconn 1000
uid 80
gid 80
nbproc 1
chroot /var/empty
daemon

listen Cluster1
bind 20.30.40.51:80
mode http
log global
option dontlognull
option forwardfor
maxconn 10000
clitimeout 3000
balance roundrobin
contimeout 6000
srvtimeout 6000
retries 2
option httpchk HEAD /index.html HTTP/1.0
server WEBSVR-1 10.1.1.11:80 check inter 10000 weight 1
server WEBSVR-2 10.1.1.12:80 check inter 10000 weight 1

listen Cluster2
bind 20.30.40.52:80
mode http
log global
option dontlognull
option forwardfor
maxconn 10000
clitimeout 3000
balance roundrobin
contimeout 6000
srvtimeout 6000
retries 2
option httpchk HEAD /index.html HTTP/1.0
server WEBSVR-3 10.1.1.21:80 check inter 10000 weight 1
server WEBSVR-410.1.1.22:80 check inter 10000 weight 1

BTW - if you look closely, you will see a global option called "nbproc". The current pfsense haproxy package calculates this based upon the # of CPUs on your firewall, and spawns an haproxy thread for each CPU. As I mentioned earlier in this thread, this can cause undue logging to your back-end servers. Just something to keep in mind…

Itwerx

@rkelleyrtp
Except those are listening on different public IPs. (Sorry, I should have clarified this is for a single public IP address. :)
To clarify further, we currently are using Apache reverse proxy to handle inbound requests to a single IP address assigned to multiple sites. Apache does a good job of reading the site address out of the request header and redirecting to the correct internal server. Only problem is it is slo-o-ow, (and HAproxy is rly-fst! :)
Is this not a good job for HAproxy…?

rkelleyrtp

Itwerx:

So, are you asking if haproxy can read the incoming http header and forward to the appropriate back-end server(s) when a single IP Address is used? I think what you are looking for are ACLs inside haproxy. Here is a quick link:

http://agiletesting.blogspot.com/2009/02/load-balancing-in-amazon-ec2-with.html

Essentially, you have a single IP address on the outside (your "listen" address) and multiple servers on the back-end. Since haproxy can read the http header request, it can make intelligent decisions as to where the requests should go. From that link above, you define your frontend section like this:

frontend myfrontend *:80
log global
maxconn 25000
option forwardfor
acl acl_example1 url_sub example1
acl acl_example2 url_sub example2
use_backend example1_farm if acl_example1
use_backend example2_farm if acl_example2
default_backend default_farm

This creates ACLs (access-control-lists) that match the incoming HTTP header. Next, setup your back-end section to route the requests:

backend example1_farm
mode http
balance roundrobin
server server1 192.168.1.1:80 check
server server2 192.168.1.2:80 check
backend example2_farm
mode http
balance roundrobin
server server3 10.0.0.3:80 check
server server4 10.0.0.4:80 check
backend default_farm
mode http
balance roundrobin
server server5 192.168.1.5:80 check
server server6 192.168.1.6:80 check

Hope this helps…

-Ron

Koti

Yup its working fine for me also.

SyRenity

Hi.

Can anyone using haproxy comment how well it compares to inbound Load Balancing?

What is the improvement in concurrent connections, used CPU, etc…

Thanks!

Itwerx

For anyone installing this on a clean load of pfSense there's actually one additional rule needed allowing HTTP (or whatever), in to the internal servers. I.e. the same rule you would have for a standard NAT even though there is no NAT defined. (This is contrary to what is stated in the first post in this thread.)

Arjen

Any support for full transparent proxy mode ? :

aka: source 0.0.0.0 use src client

Regards.

Arjen

sam_son

Single IP address - multiple back end servers.

Is it possible for me to do the following.

I thought it best to add it to this post rather than start another.

Basically the 192.168.5.2 server has a combination of both http & https for authentication. It is an appliance box and when initially logging in via an http url having done so this then changes to https.

I have no clue at this point how to proceed with the configuration so if someone could enlighten me with some configuration example that would be great. I appreciate you are all busy but getting the answer to this question would help my little office out no end.

I would ideally like if possible the url access to be

http://81.81.81.81/server1 to redirect to 192.168.5.1 back end server
http://81.81.81.81/server2 to redirect to 192.168.5.2 back end server

Is this similar to port forwarding or can haproxy forward all ports using the url. This may be on the vague side but I'm sure there will be someone equally as clueless as myself with haproxy.

![HAproxy single IP address.png](/public/imported_attachments/1/HAproxy single IP address.png)
![HAproxy single IP address.png_thumb](/public/imported_attachments/1/HAproxy single IP address.png_thumb)