Strange Test results…..

maunded · Dec 2, 2005, 2:07 AM

Hi All,
Im very y new to FreeBSD and pf, but I need to upgrade my corporate firewall and pfSense looked like it had a lot of good features.
I did a bit of a comparison test of pfSense with Smoothwall and Checkpoint.
They are all running in a VMWare GSX environment on a Dell PE2850 3.2Ghz Xeon w/512Mb RAM. ALso running on VMWare are 2xCentOS4.1 servers to provide Apache.
The results are really odd to me, admittedly I know very little about FreeBSD, but it seems that Smoothwall (RH9 I think) outperformed everything else. The ruleset was very simple, an external interface NAT'd to a webserver in the DMZ. I used Webbench to ramp up the connections to the firewalls over 3 mintues. Heres the results:
Smoothwall: 279.77 req/sec - Errors 0
Checkpoint NGX: 234.466 req/sec - Errors 0
pfSense (2 Load balanced Apache servers): 31.9083 req/sec - Errors 10+
pfSense(Single NAT'd APache server) - 18.0167 req/sec - Errors 10+

Can someone tell me why my results were so bad for pf? i think its a great firewall, and has many features I would like to use, but considering we run a very busy website, I dont think it would handle the traffic, especially once I start putting 25-30 rules in there.

Comment, questions, suggestions, criticism welcome

D.

sullrich · Dec 2, 2005, 5:15 AM

How are you determining that there was errors? Is this part of the client software?

It may come down to a bug in LB. It's a brand new feature…

The errors have me wondering.

Also, do you have the vmware tools loaded in each? Are you using the vmnet drivers?

maunded · Dec 2, 2005, 10:34 PM

Yes, the client software reports the errors, I believe they were all request timeouts, when I say 10+ I mean there were on average about 10-15 errors in 9000 requests

I dont have the vmtools loaded on any of the servers, I will try that next week.

Any idea why the request/sec was so low for pf? I thought that it may be because the client software I am using is sending all requests from one machine? Does pf have some sort of connection throttling? Is it trying to defend itself against a SYN flood? Is there anywhere I might start to look for errors?

D.

sullrich · Dec 2, 2005, 10:39 PM

@maunded:

Yes, the client software reports the errors, I believe they were all request timeouts, when I say 10+ I mean there were on average about 10-15 errors in 9000 requests

I dont have the vmtools loaded on any of the servers, I will try that next week.

Any idea why the request/sec was so low for pf? I thought that it may be because the client software I am using is sending all requests from one machine? Does pf have some sort of connection throttling? Is it trying to defend itself against a SYN flood? Is there anywhere I might start to look for errors?

D.

In a nutshell: VMWare + FreeBSD networking performance sucks. I would try these tests with real hardware. I know this is not what you want to hear but its true.

maunded · Dec 2, 2005, 10:55 PM

In a nutshell: VMWare + FreeBSD networking performance sucks. I would try these tests with real hardware. I know this is not what you want to hear but its true.

Thats completely understandable, and I have a Dell PE850 waiting to install pfSense on, which takes me back to our emails re pe850s and sata drives :)
Until the next release comes out I'll run pfSense from the LiveCD/USBKey and do some more testing using real hardware on Monday.
I'll post the results back here.

billm · Dec 7, 2005, 5:49 AM

@sullrich:

@maunded:

Yes, the client software reports the errors, I believe they were all request timeouts, when I say 10+ I mean there were on average about 10-15 errors in 9000 requests

I dont have the vmtools loaded on any of the servers, I will try that next week.

Any idea why the request/sec was so low for pf? I thought that it may be because the client software I am using is sending all requests from one machine? Does pf have some sort of connection throttling? Is it trying to defend itself against a SYN flood? Is there anywhere I might start to look for errors?

D.

In a nutshell: VMWare + FreeBSD networking performance sucks. I would try these tests with real hardware. I know this is not what you want to hear but its true.

There's also a possibility that it's state table collisions (pf flushes expired states every 10 seconds by default). In the real world you'll see connections from a larger number of IP addresses so this tends to be less of an issue. This may, or may not be the problem here, just offering up another suggestion ;)

FWIW, I've got hosts that do 1000 state table insertions and removals / second with 90K active states w/ no problems. This is on PF's native platform though, I can't speak for FreeBSD although a number of people have mentioned similar numbers to me personally.

–Bill