Strange Test results…..



  • Hi All,
    Im very y new to FreeBSD and pf, but I need to upgrade my corporate firewall and pfSense looked like it had a lot of good features.
    I did a bit of a comparison test of pfSense with Smoothwall and Checkpoint.
    They are all running in a VMWare GSX environment on a Dell PE2850 3.2Ghz Xeon w/512Mb RAM.  ALso running on VMWare are 2xCentOS4.1 servers to provide Apache.
    The results are really odd to me, admittedly I know very little about FreeBSD, but it seems that Smoothwall (RH9 I think) outperformed everything else.  The ruleset was very simple, an external interface NAT'd to a webserver in the DMZ.  I used Webbench to ramp up the connections to the firewalls over 3 mintues.  Heres the results:
    Smoothwall: 279.77 req/sec - Errors 0
    Checkpoint NGX: 234.466 req/sec - Errors 0
    pfSense (2 Load balanced Apache servers): 31.9083 req/sec - Errors 10+
    pfSense(Single NAT'd APache server) - 18.0167 req/sec - Errors 10+

    Can someone tell me why my results were so bad for pf?  i think its a great firewall, and has many features I would like to use, but considering we run a very busy website, I dont think it would handle the traffic, especially once I start putting 25-30 rules in there.

    Comment, questions, suggestions, criticism welcome

    D.



  • How are you determining that there was errors?  Is this part of the client software?

    It may come down to a bug in LB.  It's a brand new feature…

    The errors have me wondering.

    Also, do you have the vmware tools loaded in each?  Are you using the vmnet drivers?



  • Yes, the client software reports the errors, I believe they were all request timeouts, when I say 10+ I mean there were on average about 10-15 errors in 9000 requests

    I dont have the vmtools loaded on any of the servers, I will try that next week.

    Any idea why the request/sec was so low for pf?  I thought that it may be because the client software I am using is sending all requests from one machine?  Does pf have some sort of connection throttling?  Is it trying to defend itself against a SYN flood? Is there anywhere I might start to look for errors?

    D.



  • @maunded:

    Yes, the client software reports the errors, I believe they were all request timeouts, when I say 10+ I mean there were on average about 10-15 errors in 9000 requests

    I dont have the vmtools loaded on any of the servers, I will try that next week.

    Any idea why the request/sec was so low for pf?  I thought that it may be because the client software I am using is sending all requests from one machine?  Does pf have some sort of connection throttling?  Is it trying to defend itself against a SYN flood? Is there anywhere I might start to look for errors?

    D.

    In a nutshell: VMWare + FreeBSD networking performance sucks.  I would try these tests with real hardware.  I know this is not what you want to hear but its true.



  • In a nutshell: VMWare + FreeBSD networking performance sucks.  I would try these tests with real hardware.  I know this is not what you want to hear but its true.

    Thats completely understandable, and I have a Dell PE850 waiting to install pfSense on, which takes me back to our emails re pe850s and sata drives :)
    Until the next release comes out I'll run pfSense from the LiveCD/USBKey and do some more testing using real hardware on Monday.
    I'll post the results back here.



  • @sullrich:

    @maunded:

    Yes, the client software reports the errors, I believe they were all request timeouts, when I say 10+ I mean there were on average about 10-15 errors in 9000 requests

    I dont have the vmtools loaded on any of the servers, I will try that next week.

    Any idea why the request/sec was so low for pf?  I thought that it may be because the client software I am using is sending all requests from one machine?  Does pf have some sort of connection throttling?  Is it trying to defend itself against a SYN flood? Is there anywhere I might start to look for errors?

    D.

    In a nutshell: VMWare + FreeBSD networking performance sucks.  I would try these tests with real hardware.  I know this is not what you want to hear but its true.

    There's also a possibility that it's state table collisions (pf flushes expired states every 10 seconds by default).  In the real world you'll see connections from a larger number of IP addresses so this tends to be less of an issue.  This may, or may not be the problem here, just offering up another suggestion ;)

    FWIW, I've got hosts that do 1000 state table insertions and removals / second with 90K active states w/ no problems.  This is on PF's native platform though, I can't speak for FreeBSD although a number of people have mentioned similar numbers to me personally.

    –Bill


Log in to reply