How Far Have You Scaled Your PFS Box?



  • I'm just personally curious.  I've been using PFS off and on for a few years now, and for the past 8 months have been running it at home in an old Celeron 1 GHz tower with 4 NICs, 2 of them WAN's, one of them my LAN, and the last interface provides bandwith to a couple AP's for about a dozen or so people on my street.  (see www.socalfreenet.org)

    I've enjoyed watching the project mature over the years, and am about to put my first PFS box in a commercial application.  Nothing serious, just Dual WAN failover and basic firewall services for a small office.

    That being said, I'd like to open the floor to discussion on scalability.  How far have you pushed your PFS?  How many NICs, what CPU, how much RAM, what kind of bandwidth, how many users are you supporting?  Got any photos?

    And of course, my apologies if this has been done before.



  • I'll start small and let people go from there…

    We have 30 people behind a P3 1.7ghz running squid attached to two bonded T1s (3mb/s).  Two 3com 905 NICs.



  • Just one out of many examples I have:

    We have a 266mhz firewall running all of the pfSense servers 5+.  Our servers take a royal pounding and are always pushing ATLEAST 1.5 megabit.  It spikes up and down all day to 10 megabit (limited by network interface 10 megabit link).

    Never had any issues and has been running strong for many years.



  • I have a 100meg link to the internet Pfsense is setup in a small lab network off of this. Have 3 servers behind it. Two linux, and one Windows box. Below you can see my bandwdith use since starting pfsense 1.2 beta1. The only time I have had trouble is with Tor. It seems it can be a little hard on connection use. It will kill a linksys good! Only problem that I have run into is I need to clear the state table sometimes. I blaim Tor for this however, it should close them a little faster by default. I raised the number of states to 20,000 for my little box.

    Beta's http://stashbox.org/77126/pfsense.bmp

    1.2 release http://shup.com/Shup/37288/108319191050-gw.adotnet.net-Status-RRD-Graphs-Windows-Internet-Explorer.png

    Every time I have had a lockup it has been HARD. Had to power cycle pfsense box but it has always back come up. 1.2rc4 has been the most stable to date for me, it has yet to crash. Going to load it down hard and let tor just go nutts in my little network.

    Forgot to add that I use a Via 1ghz mini-itx board, 256megs ram, IDE solid state drive.



  • WOw, that's some bandwidth  :o    BUt err….what happened in October? lol

    -M@



  • I have more than 30 pfsense runing at my clients office or datacenter. From 1 to 100 mbits. Every site has a two nodes cluster, built on HP proliant system or IBM Xseries, form 100 to 5 000 users behind the boxes.
    First machine has been started in october 2005 (v0.69 I don't mistake myself).
    All systems are runing basic features (IPSEC, pptp, captive portal and multi wan). For example, I have a cluster protecting medical applications (9 J2EE clusters) processing data for 750 000+ patients.

    I'am starting a new project wich will use the squid+squidguard packages in addition to the other basic features, in this project I will deploy pfsense boxes all over the world (europe,us,asia,africa), that's why some of you have already heard me speaking about a management host on the IRC channel. On another project, I am going to change openBSD based pf firewalls to pfsense (European country military context).

    I love pfsense and its developers team. The project is awesome.



  • 1. Firewall for 100+ staff, and 50+ public wireless. Load balances 2x 6MB DSL connections, handles incoming PPTP and IPSEC (just a handful). 
    -Running on a Dell Poweredge 1550, Pentium III, 1Ghz, 512MB RAM.
    -CPU is never more than 2% except for spikes, and RAM is never more than 20%
    -Solid as a freaking rock.

    2. Firewall for IIS web server. Serves streaming media on a 100MB internet connection, sometimes topping 40MB/sec. Many thousands of hits per day.
    -Running on a Dell Poweredge 1650, Pentium III, 1.13Ghz, 512MB RAM
    -RAM is never more than 30%
    -Also solid as a freaking rock.



  • i work for purdue university,

    we've been using it for about 3 or 4 months in testing.  I have some pictures ill upload later of our setup.  We setup a failover system with 4 gateway e1600 systems (2 on one subnet and 2 on another) with about 2556mb of ram, 30gb hd, 1ghz or so processing.  We have not put the 200 plus machines behind the firewall yet but we'll see how far it can go.  We took the subnet given to use by the university and split it into our local lan of /22 (about 1000 machines or so support).

    We use another e1600 for a windows 2003 server ias server (radius) and another e1600 for our GHOST.

    We use NUT to manage our two UPS for the machines (APC Smart UPS 1500).  We tested it today and it worked, it shutdown once the battery got to very low status (took about 2-3 hours to do)

    We have not gotten captive portal to work with mac authentication yet but that will be a future testing phase.

    We use the DNS, OpenNTPD, PPTP, DNS, DHCP, Carp, and a few other services i cant remember.  Theres only a about a 10% load max on the machine!

    we are still playing with squid to try to get  atransparent proxy working as well to help keep the T3 bandwidth usage low.

    so yea its been a pretty intense and eventful testing.  PFSense works great and we are very pleased and appreciate the community for the program.



  • We use it at a wireless community setup as firewall/gateway to the internet and also in some cases as wireless client/ap router.

    The internet box is a Compaq Deskpro SFF (PII 400Mhz | 512MB RAM | 80GB HDD | 3 x Intel Pro 100 NIC's), manages  6Mbps/512Kbps PPPoE ADSL connection, soon to add a 2nd ADSL connection… It runs pfSense with squid and lightsquid and provides access to internet to 30-40 users.

    It's been rocksolid for months, specially after the upgrade from 256MB to 512MB of RAM.

    As a wireless client/ap router we use the same Compaq boxes with atheros cards and only 64MB RAM and booting from CF cards. The boxes provide access to the wireless community network and internet through their wireless card (WAN) and basic NAT/firewall/DHCP to the user's home network. Sometimes a 2nd atheros card is added to provide wifi in the area. Also works fine, although recently we've been using beta versions of m0n0wall to this function due to lack PC100 SDRAM to this boxes, and m0n0 works better with 64MB in this setup.

    Overall, m0n0wall/pfSense are great projects, we've been using them for 3 years in this network.



  • @KiFFuSeR:

    We use it at a wireless community setup as firewall/gateway to the internet and also in some cases as wireless client/ap router.

    The internet box is a Compaq Deskpro SFF (PII 400Mhz | 512MB RAM | 80GB HDD | 3 x Intel Pro 100 NIC's), manages  6Mbps/512Kbps PPPoE ADSL connection, soon to add a 2nd ADSL connection… It runs pfSense with squid and lightsquid and provides access to internet to 30-40 users.

    Ya, I used to run Squid, until I got the 2nd internet feed (I've got one 6 Mb/s DSL line, and one 15 Mb/s cable line load balancing and failover) but found out the hard way that squid doesn't work in dual wan mode.  Actually, its seems most add-on packages break when you add a 2nd gateway.  But forced to choose between 21 Mb/s combined bandwidth and squid, I'll choose 21 Mb/s lol ….tho I do miss tailing the squid logs and watching the random URL's go by.  Maybe I'll get a 2nd box for squid....who knows, I could always use a higher power bill :)

    -M@



  • @dnky_bones:

    Ya, I used to run Squid, until I got the 2nd internet feed (I've got one 6 Mb/s DSL line, and one 15 Mb/s cable line load balancing and failover) but found out the hard way that squid doesn't work in dual wan mode.

    My goal with 2nd WAN is to make all high priority traffic (http/dns/pop3/voip) go through the WAN1 and all other traffic go to WAN2. I'm not trying to aggregate bandwidth or do failover, just simple routing policy, and really need squid/lightsquid statistics. Shouldn't it work this way?

    PS: sorry if this is a bit offtopic…



  • From my experience….sorta.  Its been about 6 months since I stopped using squid, but if my memory serves me, clients would be directed over the transparent proxy if I directed them to go over the default gateway.  If you add a 2nd gateway, squid has no idea about it, it doesn't really know it exists.

    -M@



  • I suppose you get set up 3 PFsense boxes:

    2 boxes connect to each DSL line respectively, both run Squid (both only have one connection, right?)

    1 box is the gateway for your LAN, and it load balances to the other two boxes.



  • I have Foxnews.com and Foxbusiness.com behind two redundant pfsense firewalls running on Dell 2950's.



  • @foomanjee:

    Foxnews.com and Foxbusiness.com

    Just out of curiosity: how much traffic do they generate on average?



  • Obviously most of the http requests are Akamaized, but not all of it.  There's ads and everything else we deal with, plus odd projects, etc.  So, it's not so much 'bandwidth/traffic' as it is 'packets'.

    I don't have a whole lot behind them yet, I'm in the process of moving more services from other datacenters.

    At any rate, currently they're only doing about 60mbps.  In 3 months time, I expect to be doing about 1gpbs consistently.



  • @foomanjee:

    In 3 months time, I expect to be doing about 1gpbs consistently.

    What sort of hardware you planning on using for THAT ?



  • @sai:

    @foomanjee:

    In 3 months time, I expect to be doing about 1gpbs consistently.

    What sort of hardware you planning on using for THAT ?

    I've already got 2 Dell 6850's allocated for it.  2950's will do the job easily, but you always want to be prepared for future growth.  6850's will let me ignore any firewall related hardware upgrades in the future.



  • @foomanjee:

    I've already got 2 Dell 6850's allocated for it.  2950's will do the job easily, but you always want to be prepared for future growth.  6850's will let me ignore any firewall related hardware upgrades in the future.

    Are you serious?  You are my new hero if you are!



  • Yep, I'll be at the datacenter tomorrow, I'll take some pictures of our cage with my phone.  We've got 6 6850's in production right now, mostly for database servers.  Then another 40 or so 1950's/1850's in our cage, all behind the firewalls - which again, are 1950's for the time being.

    1950's will surprise you, before I went live with the pfsense firewalls, I got around 600mbps through them in testing, stable.  Bursts up to around 800mbps.

    I really don't want to do anymore changes to the firewalls until pfsense 1.2 is released.  FreeBSD 7 is going to help things a lot more than you might think.



  • @sullrich:

    Are you serious?  You are my new hero if you are!

    Tell me about it.  I was pretty stoked about my Epia LN10000EG 1 GHz fanless ITX deployment….."was" being the operative word lol..

    But seriously, I think its beautiful that the software scales so dramatically.  I'm curious, tho, are there any commercial gateway/firewalls that can handle that kind of load and have a similar feature set as PFS?.... that are within the same price range as a poweredge 6850?

    Why would one choose PFS over a Cisco or Foundry, etc?

    Once again, out of sheer morbid curiosity ;P

    -M@



  • I'm actually in the middle of this argument with one of my bosses.  He wants Cisco, mainly because of paid support - which I completely understand.  I told him I'm more comfortable with pfsense, I know what it can and can't do.  I don't know anything about Cisco IOS.

    Plus the Cisco ASA, if you want to get anywhere near 1gpbs, you're looking at $190,000.  I'm sorry, it's just not worth it.

    He may eventually overrule me on this, and make me dump pfsense, however I really, really don't think he's going to.



  • @foomanjee:

    I'm actually in the middle of this argument with one of my bosses.  He wants Cisco, mainly because of paid support - which I completely understand.  I told him I'm more comfortable with pfsense, I know what it can and can't do.  I don't know anything about Cisco IOS.

    Plus the Cisco ASA, if you want to get anywhere near 1gpbs, you're looking at $190,000.  I'm sorry, it's just not worth it.

    He may eventually overrule me on this, and make me dump pfsense, however I really, really don't think he's going to.

    We have paid commercial support.  See the front page of pfsense.org :)



  • I have been running pfSense for a while now and I have it running everywhere.

    At work I have 3 sets of pfSense firewalls.  Primary and failover.  They work beautifully.  They are all running on Dell Optiplex GX260's which if my memory serves me right, they are around 2.4Ghz each with 1GB RAM and 40GB HD.  I originally had 3COM 3c2000 NIC's all over the place, but I had lots of issues so now I run Intel PRO1000's across the board.

    Each box runs rock solid and has about 300 computers behind them.  We run a lot services from the company, primarly an offsite backup service for a few hundred clients so we have a ton of traffic all the time.

    We recently upgraded to a 100mbit internet connection and our ISP recommended we purchase a Cisco 7204 for somethign like $9,000.  Well that didn't fly so I slapped pfSense on another GX260 and turned off the firewall so it was just a router and we stress tested the bad boy and were able to achieve a solid 300mbit, which was more than enough.  So I ended up paying a grand total of $0, which is just amazing.

    I would be able to replace my Cisco PIX's if pfSense could do Policy NAT because we have a few hundred IPSEC Tunnels and as you can imagine, subnets get claimed really fast, so policy NAT is a must.

    I recently made a purchase on eBay of 50 Intel Pro 100's, so now whenever one of my coworkers, friends, relatives is in need of a firewall I just tell them to go find a working peice of crap computer and I will set them up an awesome firewall.  Needless to say I have a few dozen pfSense boxes runnning at there homes and an IPSEC tunnel to each, for helping them out with comptuer problems, file sharing, etc.  I have running at my house an old school P2 300Mhz Overclocked to 450Mhz (thats such an insane increase if you think about it!) with 256mb RAM, 6gb HD.  It runs flawlessly.  My record uptime was 290 something days, but ofcourse the power went out and killed my record (time for a ups right?).

    My only complaint is about the PPTP GRE NAT issue, but really, I love pfSense and have been nothing but pleased over and over and over.  Whenever I speak with other IT guys and friends I always promote pfSense, it is simply amazing and well on its way to becoming a Cisco/Checkpoint killer, the other boys cant really hold a candle to pfSense.

    Kudos to all you guys who help make pfSense what it is, you rock!!!



  • Our office has a single PII 400 with 128MB RAM and 5 3COM 3C905-TX NICs, 3 of which are currently in use. We have about 50 constant users, and our average bandwidth usage is around 6MBps of our cable connection and 1-1.5 of our DSL. The only service we use so far is ntop, so it doesn't seem to be overloaded yet. This machine was supposed to be just a demo for the bosses, but ended up working so well that we put it in production and it stayed there. Within a few months I'm hoping we'll get permission to buy a new system for it so I can get better traffic filtering in place.



  • @foomanjee:

    Yep, I'll be at the datacenter tomorrow, I'll take some pictures of our cage with my phone.  We've got 6 6850's in production right now, mostly for database servers.  Then another 40 or so 1950's/1850's in our cage, all behind the firewalls - which again, are 1950's for the time being.

    So where are those pictures at foo!  ???    ;D

    -M@



  • Was in a different post!

    http://box.nevernet.com/~foo/IMG00036.jpg

    10 6850's, a bunch of 1950's, and a few 1750's.  Few Sun boxes, too.  In about a week we're going to have a gigantic 3Par (san) cabinet that everything pulls from.



  • @foomanjee:

    Was in a different post!

    http://box.nevernet.com/~foo/IMG00036.jpg

    10 6850's, a bunch of 1950's, and a few 1750's.  Few Sun boxes, too.  In about a week we're going to have a gigantic 3Par (san) cabinet that everything pulls from.

    /drool



  • After few day of installing, testing, transferring existing rules/routes, and some testing again I can proudly say: New pfsense firewall/router is working instead of Alliedtelesyn Rapier 24i.

    I have installed pfsense on Intel sr1350ahlx (dual core xeon, 2 GB RAM, 2x500 GB HDD in RAID 1, 2 integrated Gb NICs and additional PCI-X 2 port Intel Gb NIC). This machine is serving 1 WAN connection (E1, soon double E1) 4 VLANs (in one of VLAN's I have internal routers for 9 other networks), 1 DMZ and one admin network. On VLAN port VLANs are distributed trough gigabit port on AT8000s/24 switch and fiber optics.

    Altogether there is around 400 PC's, 30+ servers (Windows and UNIX) few-dozen print-servers, etc.

    For now load is minimal. Tomorrow I am adding some IPSec tunnels.

    So far this is my biggest pfSense installation.

    Sasa



  • My biggest pfSense installation to date is a pair of Dell 1950s that have been running 1.01 flawlessly for something over a year now.  They're in a cabinet at a data center, supporting a handful of web servers, a mail and DNS server and a few (non-public) database servers.  95th %ile traffic is around 2 megabits, but we get spikes up to 10 during the day.

    I had hoped to use pfSense for the firewall when we moved into our new office space last summer, too, but the Pentium III box I had handy to install the firewall on wouldn't boot either 1.01 or the 1.2 beta, or anything else with a FreeBSD kernel.  I never did find out why.  Fortunately, it boots OpenBSD just fine, so at least I still get pf, even without the web interface and the additional features I like in pfSense.



  • @gthornock:

    I had hoped to use pfSense for the firewall when we moved into our new office space last summer, too, but the Pentium III box I had handy to install the firewall on wouldn't boot either 1.01 or the 1.2 beta, or anything else with a FreeBSD kernel.  I never did find out why.  Fortunately, it boots OpenBSD just fine, so at least I still get pf, even without the web interface and the additional features I like in pfSense.

    You should retry with the upcoming 1.3 release which will be based on freebsd 7. Also make sure your bios is up2date and maybe exchange the cdrom. freebsd is sometimes picky about cdroms.



  • Will take a few more pictures on Monday.  We've grown considerably since I originally commented on this thread.



  • @hoba:

    @gthornock:

    I had hoped to use pfSense for the firewall when we moved into our new office space last summer, too, but the Pentium III box I had handy to install the firewall on wouldn't boot either 1.01 or the 1.2 beta, or anything else with a FreeBSD kernel.  I never did find out why.  Fortunately, it boots OpenBSD just fine, so at least I still get pf, even without the web interface and the additional features I like in pfSense.

    You should retry with the upcoming 1.3 release which will be based on freebsd 7. Also make sure your bios is up2date and maybe exchange the cdrom. freebsd is sometimes picky about cdroms.

    I already tried with a FreeBSD 7 release candidate CD, and again with RELEASE.  Those won't boot either.  (Neither will 6.2 or Dragonfly, and neither will a hard drive with pfSense installed on it and moved from another box.)

    If I were ever to replace that firewall with a different machine, I would try pfSense again.  It's been outstanding in our data center cabinet.  But for now, the OpenBSD installation we have is working fine.



  • Just for the record, which PIII hardware is that? Motherboard and chipset mf'rer?



  • My setup providing Internet for about 500 people policy based load balancing on 8 x 8/2mbit ADSLs. I left out a lot of details about equipments for the ADSLs, the LAN and layer-2 failover.

    | A1  |A2  |A3  |A4  |A5  |A6  |A7  |A8
    |        |      |      |      |      |      |      |


    *                  VLAN Switch                    *


    |                                        |
    ******************  sync    *******************

    • PFSense 1.2 * - - - - - - * PFSense 1.2 *
      ******************              *******************
                        \                    /
                      *******************
                      * VLAN Switch *
                      *******************
                        /                   
      *******************  sync  *******************
    • PFSense 1.2 * - - - - - - * PFSense 1.2 *
      *******************            *******************
                        \                    /
                      *******************
                      * VLAN Switch * –----------- DMZ (Through the firewall setup)
                      *******************
                                |
                        500 people LAN

    ./Thomas



  • @jahonix:

    Just for the record, which PIII hardware is that? Motherboard and chipset mf'rer?

    Looks like it's not a PIII, it's a Duron.  According to dmidecode, the motherboard is an Asus A7N8X2.0 with Phoenix BIOS dated 2003-03-19.



  • …and with an nVidia nForce2 chipset.

    The nve(4) driver supports the NVIDIA MCP onboard adapters of mainboards with the following chipsets:

    *      nForce
        *      nForce2
        *      nForce3
        *      nForce4

    So I guess the chipset should be supported. The AMD Duron is in this list as well. Hm…

    What about your BIOS settings? Have you tried to disable ACPI / flash latest firmware etc.?

    Found this from a quick google search: http://marc.info/?l=freebsd-questions&m=111719084502712&w=2

    Try disabling firewire in the bios.

    I had the same problem with my system (A7N8X). Worked until 4.9 then
    stopped working.

    Disabling firewire allowed me to boot.

    Thanks for the tip on disabling firewire to get this board to work.
    Do you have the sata raid going as well?

    Never tried it.



  • qdk

    why do you have 2 sets of failover pairs of firewalls?



  • @sai:

    qdk

    why do you have 2 sets of failover pairs of firewalls?

    Just to keep it simple… The first layer of firewalls isnt needed when decent Internet is provided like a 100mbit fiber terminated in ethernet. futhermore there are some limits with captive portal and didnt go well with the policy based routing.

    ./Thomas



  • So its been a little over a year since this thread began, I figure there's got to be some new stories out there.  Since I started this thread, I've deployed 4 more PFS's, and working on my 5th tonight.  I've been following the same template as I outlined earlier, VIA EPIA LN 10000EG mini-ITX boards running of CF's.  Though I did have my first CF failure (I've been running them in x86 mode rather than embedded.  Fortunately, solid state drives have come down in price enough now that I was able to replace that CF with a 32 GB SSD.  The nice thing is however, that when a CF fails, the vast majority of the data is still intact and accessible, so I was able to pull the XML file off the old CF, and put it on a brand new install on the new SSD.  Only picture I have, I took with my cellphone.

    The box I'm installing tomorrow is just an Alix board, will be used as a remote IPsec gateway….actually connecting to the PFS box in the above picture as luck would have it.

    So....any new stories out there?  Is FoxNews guy still lurking about?  Its always interesting to see what others have done/are doing with this stuff!!

    -M@


Locked