Help in finding a rackmount server or build one for a new pfsense box



  • Hey there,

    So I think my current pfsense box is going AWOL (hardware wise) and is currently in need of replacement.  I have been looking on newegg and several forums (of course including this one) for people in the same position  as I am, but the field changing so fast, pfsense builds (at least rackmount ones) sometimes dates months if not years which makes the threads of little value due to obsolescence..

    Anyways, I currently use a stoopid Foxconn D510 box with 4GB 667 RAM and a SATA drive.  Worked OK initially, but the PSU is dying on me (literally), the machine freezes randomly and there are strange smells coming from the box (cant be too good).  My phone also relies on this box as well as my local server and my internet connection.  So I'm borrowing time here and need to find a replacement ASAP.

    My requirements (in green are the most important to me):
    -Rackmount.  I am soon to move my server to a 14U rack (if I can find one that will not require remortgaging my house) so the pfsense box will also move to the rack.
    -Current internet connection 30/10 soon to be upgraded to 50/20, thinking this pfsense box will have to last 5 years +, I think its reasonable to imagine an optic fiber connection some time in the future
    -I use Snort, Squid, SG, NTOP, HAVP, Clamd, pfblocker with several custom long lists, etc.  Current RAM is maxed out, bootup takes 10 minutes + (from powering up until I hear the pfsense chimes).
    -LAN is Gigabit and I do snort sniffing on LAN as well as WAN
    -A little bit of extra power is mandatory as I will not upgrade once again in 2 years (not even in 5 years).  Quite frankly, if that box can last 10 years, so be it.
    -"Fairly" silent.  I put silent in quotes because I do realize that rackmount machines are usually server (enterprise oriented) machines located in server rooms.  Mine will be in my home office next to my bedroom so ultra loud hissing 80,000RPM fans are out of the question.
    -PowerD is mandatory.  When idling (maybe 85% of the time) the CPU will throttle down to save power, when peak power is required, it will scale up.  My current D510 does that perfectly.
    -Intel integrated Gigabit NICs
    -Some PCIE slots for additional Intel Dual ethernet controllers (probably 2)
    -Max budget is around $250-300USD

    I looked at building a box from parts ordered online, could work well if I go for a i3 CPU for the extra power but I have a hard time finding a good mobo that will fit in a 1U case, and that has 2 integrated Intel NICS… Most have the Realtek NICs.  My current box has Realtek NIC's and they drop constantly.

    Seems Intel has some interesting motherboards (like the DQ77KB) but I cant find a seller anywhere or they are all $150+ insane..

    I also extensively looked at used servers on fleebay.  I am reluctant to pull the trigger on that one since I cant see them and Im not sure how noisy they will be.  Also, for lets say <$250 there is only older generations of servers (slow RAM, power hungry, noisy)  Maybe someone here can recommend a little dual core server that will do what I want?

    Looking forward to hearing what you guys are recommending!
    Thanks!


  • Rebel Alliance Developer Netgate

    It's louder than hell but it'll do the job well.

    http://www.alixbox.com/dellR200



  • Thanks Jimp for suggesting something!

    Louder than hell!

    Thats because of the high speed fans, any ways to reduce noise?  Perhaps install an aftermarket fan?

    Other than that, I have a supermicro server at home, I like their quality a lot.  Are Dell or IBM servers just as good?  There is a LOT of second hand servers on ebay…  Are they usually good to get?  I mean, are they usually at end of life (about to die, not about to be obsolete performance wise)?

    Last thing I'd wanna do is buy a used server online,  install it and a few months later, it just quits on me.

    I have 0 experience in second hand servers.  Can you tell me what you think?



  • We run a Dell server shop and they're usually loud at power up.  Once the OS is running the fans calm down a bit but wouldn't want the server out in the livingroom.  Yeah alot of those servers on ebays are pretty decent for firewall setups but they are noisy since they are designed to run 24/7 in an enclosed A/C controlled environment.  Not sure about aftermarket replacement fans since these are specially designed for Dell servers.  They sound like a jet taking off during initial power up. lol


  • Netgate Administrator

    Your bandwidth requirements cover a very wide range, 50/20 up to a potential Gigabit connection. This may not be such an issue if you are asking for 6 (?) interfaces and need Gigabit routing internally anyway.

    Steve



  • This may not be such an issue if you are asking for 6 (?) interfaces and need Gigabit routing internally anyway.

    No!  I mispelled the requirements.. Or I wasnt clear.. Sorry.  2 NIC's for WAN & LAN, with the addition of at least another LAN interfance (DMZ).  I was thinking future expansion for in case I reuse the motherboard 5 years down the road for a server or anything else.. More slots is better (hope this was not confusing either ;)

    My biggest issue is heat.  In the summer, the office where the server is located (and where the pfsense box will be located) becomes quite warm (with my server, 2 monitors, my desktop, switches and modem, its not rare to see it going in the 30-34C range…)

    I could AC the office too.  Planning to do that next summer... perhaps just a AC window shaker.  Also, I was not having blinds or curtains in the window (because I was renovating) and the window is facing straight south so lots of sun.  Now I have nice blinds that will hide the sun a bit.

    Its a tough call.  I'd rather put that stuff in the garage like everyone else, but my garage is detached (condo unit) and running cables to there is unfeasible (unless I drill holes in my neighbor's condo)

    Noise is more or less an issue.  I know what servers sound like when powered up.  THat I cant deal with, my bedroom is literally next door to my office.  And closing the office door will only aggravate the heat issue in the summer.

    wow... If OS controls the fans, wouldnt it be reasonable to think that noise levels be decent since CPU wouldnt be that loaded?


  • Netgate Administrator

    Thanks for clarifying that. I read "Intel integrated Gigabit NICs. Some PCIE slots for additional Intel Dual ethernet controllers (probably 2)" and thought 2 + 2 further dual. 3 NICs is much easier to accommodate.

    So do you need gigabit routing (with Snort) between LAN and DMZ?
    Is your future fibre connection likely to be Gigabit?

    Steve



  • Hey Steve,

    Yes, probably going  to need Gb B/W LAN & DMZ….  DMZ will most likely be used at first for my VoIP adapters (so Snort, squid or whatever else will not interfere with the phone service) and eventually DMZ will be used for whoever comes here with their laptops, cell phones, tablets, etc... So they dont "contaminate" my LAN but still have access to the internet.  A straight connection to the outside world while still monitoring with Snort whats going on.

    What I always keep in mind is the fact that , out of probably 100+ people, I am the only one PROUDLY running GNU/Linux on ALL of my LAN machines, and from that, I consider my IT knowledge still to be way ahead of most of my folks..  Most of them run bloated contaminated (trojans, worms, etc) windowz machines, and I want to segregate them completely.  The goal is to protect my LAN, and still prevent illegal activities (hacking, etc) from the DMZ..

    All that is not likely to happen, I reckon.

    Mmmmm  good question on the Fibre connection.  Currently, my ISP is planning to upgrade to fibre in the next 18 months (at least offer it, will I buy it?  Not sure, its currently 190$/month here in Quebec).  Again, wouldnt be surprised if WAN goes up to 50/20 that I'd be willing to get...

    $190/month  :-[ :-\ :'( ??? >:( :( :o >:(

    Did I forget a smiley??!


  • Netgate Administrator

    $190 a month. Ouch!

    I worded that badly. Whilst you may well be connecting everything using Gigabit ethernet it sounds like you won't actually need anywhere near 1Gbps between lan and dmz. Would you say that's correct?

    Steve



  • Would you say that's correct?

    Perhaps.  I cannot really say for sure since I am not sure how to evaluate that…  If you have in mind that a device connected to the DMZ (re2) will not be accessing my LAN (re1) then u'r right!

    Other than that, may happen that that same device connects to my local server PROVIDED (and there's a BIG IF!!!!) its clean and I trust it enough...

    I looked on ebay for used servers.  There are thousands.... Im lost!  If you can further help, oh that will be nice!


  • Netgate Administrator

    Well I'm just trying to determine what the maximum throughput is you will need the box to handle. Since your WAN connection is relatively slow the throughput from WAN to LAN or from WAN to DMZ will not be very large. The throughput between LAN and DMZ though could potentially be at wirespeed if, for example, you have a server on LAN and you're moving data from it onto a guest machine on DMZ. Given a sufficiently large file you may want as much speed as you can get. If you're running all those packages against that traffic you're going to need a high powered box to keep things flowing at near 1Gbps. If however you're never going to do that, or so infrequently it doesn't matter, then you might accept a maximum throughput of, say, a few hundred Mbps in which case a lower powered (and hence cheaper and quieter) machine will suffice.

    Either way I would recommend getting a socket 1155 board and fitting it with something cheap like a Celeron G1610. That will be far faster than your Atom, you can load it with RAM too but it provides access to a huge number of CPUs for suture upgrades if you need them. The DQ77KB was a great board but it seems to have been discontinued, I'm not sure what else to recommend. There are plenty of threads here with similar requirements though. The next gen Atom also looks great but possibly a bit bleeding edge for pfSense 2.1. If you can wait…

    Steve



  • Thanks Steve!

    I understand what you're saying.  That being said, I already looked at everything, except used servers.

    I started by looking at embedded devices (Atom, etc) but couldn't find anything interesting.  Then I moved my search to custom builds… Couldn't find an interesting motherboard (finding a good mobo with dual Intel NIC proved to be a challenge more than I expected).  Finally, I oriented my search toward used servers..

    This is where I am right now.

    I had already looked at the DQ77KB, but yeah, its discontinued, or if you're willing to spit $150+ for it, its available on ebay. (Im not!)

    There are plenty of threads here with similar requirements though.

    Yes & No.  Yes there are, but most of the threads are ending up with solutions that the OP initially didn't intent to go for (all inclusive solutions at several hundreds $$) or simply ended up recommending something that has long been phased out or is now discontinued.



  • Actually I forgot to ask:

    Based on my requirements, do you figure I need a powerful CPU (more than 2 cores, or more than 2.4GHz)?

    Do I need RAM faster than DDR2?

    If not, then perhaps this?

    http://www.e b ay .ca/itm/Supermicro-1U-AMD-3200-dual-core-server-8GB-WD-500GB-SATA-2-x-GBit-lan-/261152895660?pt=COMP_EN_Servers&hash=item3ccdecf2ac

    ??



  • @lpallard:

    Actually I forgot to ask:

    Based on my requirements, do you figure I need a powerful CPU (more than 2 cores, or more than 2.4GHz)?

    Do I need RAM faster than DDR2?

    If not, then perhaps this?

    http://www.e b ay .ca/itm/Supermicro-1U-AMD-3200-dual-core-server-8GB-WD-500GB-SATA-2-x-GBit-lan-/261152895660?pt=COMP_EN_Servers&hash=item3ccdecf2ac

    ??

    Looking through the messages I have noticed you didn't say anything about VPNs so if you are not going to utilize heavy VPN connections like OpenVPN then the bottleneck is going to be the IO of the bus that the network cards ride on not the CPU.  Obviously as you know PCI is a shared resource so you will be limited on the bus speed.  PCI express is alot faster.

    SNORT takes up CPU resources but it's RAM intensive.

    So you could get a very beefy server but you will end up wasting electricity and heat since it'll be idle most of the time.  Some have virtualized PfSesne to take advantage of the beefy hardware while it runs other VM guests to balance things out.


  • Netgate Administrator

    Hard to make a judgement on that Supermicro box. I haven't tried running pfSense on an Opteron. I can't actually find out what CPU it has either. Check the forum for reports on those Nvidia NICs.
    It's probably going to be fine for your current and near future requirement but for the next 5 years?  :-\ That's going to be looking quite old in 5 years time and it will have goobled up a lot of kWh in that time.

    Steve



  • Some have virtualized PfSesne to take advantage of the beefy hardware while it runs other VM guests to balance things out.

    Funny you talk about virtualization… Last July when I built my new home server, I initially planned to do virtualization to reduce the power consumption and facilitate maintenance...

    The server I built has plenty of horsepower to vurtualize pfsense, but lately, I have been debating if virtualize pfsense was a good move at all.. I am thinking about reliability..

    Its like putting all your eggs in the same basket.  What if my server dies?

    Then I'd lose the server itself with its services, I would lose the internet and the phone...  All at once

    If I keep pfsense a separate machine, going back in operation would require only purchasing new parts and get it back online fast since it would be a fairly simple machine (mobo, CPU RAM and PSU)

    Statistically speaking, the chance a computer or another dies are pretty much the same so I am not sure virtualizing it or not really inpacts the "reliability"..

    The biggest drawback of virtualizing it is the need of moving my current server to a virtualized machine... I could start a 1000 page thread on how and what not, but its off topic..

    Darkk, maybe you are right.  Virtualization is the way to go.??



  • My personal recommendation for running all the things you need would be a supermicro 1155 motherboard. You said you need it to be 1U but state that it might need 2 extra PCIE adapters in the future. I'm guessing you meant a dual port nic. If you still need additional interfaces, the X9SCi-LN4F (specs here:http://www.supermicro.com/products/motherboard/Xeon/C202_C204/X9SCi-LN4F.cfm) is excellent. Bare in mind when looking for the price, that it has 4 intel nics, and you don't need to purchase any other PCIE adapters. Otherwise if you plan to still use 1U but go up to a quad port PCIE (with raiser) then the X9SCM (specs here:http://www.supermicro.com/products/motherboard/Xeon/C202_C204/X9SCM-F.cfm). The -F models have IPMI (for remotely controlling the motherboard without the need to hook up keyboard/monitor, even for installing OS). You could find those on ebay with a -O or a -B on ebay. -O is retail (motherboard shield, sata cables) and -B is bulk (only motherboard)

    That box should last well in excess of 5 years.

    What I would do is this: Get a 4U case, the X9SCM-F, a corsair CX model for the PSU, and a CPU of your choice. You don't need to go crazy with the CPU, even the lowest xeon would handle all you want without a sweat. Get pretty much any heatsink/fan for the CPU, and if your case has room 2 large quiet fans. I've seen you mention that the HDD is ok, so you could reuse that, and whatever is left, spend it on quality (ECC) RAM. Those motherboards take up to 32GB, but one has to be insane to use that much RAM for pfsense, up to 8GB (and that's a very extreme scenario, ie, highly unlikely you need it) is OK.
    When the time comes, you can install up to 4 PCIE adapters. that's 4x4+2=10 interfaces.

    The systems recommended are enterprise grade (yes corsair's CXs are excellent).Going from enterprise grade to consumer quality is a different story.
    In that case, I would get an asrock 775 motherboard (*scrathes head…what's the model...,G41M-VS3 ah there it is) a celeron E3300 and a quad port intel nic used from ebay. Again you would go with a CX PSU and a 4u case with large quiet fans. Can't vouch for the reliability of this system in this particular scenario, since I'm using them for other purposes, not pfsense, and in my case so far they are working flawlessly. The motherboard takes up to 8GB RAM.

    I know you said you need 1U but that limits your future expandability. Going with a 4U case allows you to use "consumer grade" PSUs (corsair or seasonic, all other makes are a bunch of wires and solder dumped in a metal case, NO arguments about this,ever) and also allows you to install more PCIE adapters. Considering the intel CTs are dirt cheap on ebay, I don't see a reason you shouldn't go 4U. Oh and you have the choice of a $5 "Sata hotswap tray" (ebay) for an extremely easy HDD replacement. 2 of those and cheap used HDDs (properly stress tested before being installed) and you are pretty close to systems I use in production.

    What my point is, in case anyone missed it, is if you go with a larger case you open up yourself to more choices down the road. If you don't really truely need 1U cases, stay away from them.



  • Yeah, for enterprise environments I would always use server class hardware for anything including firewalls and virtualize some servers as backup.  Eventually I may do something like this at work.



  • @lpallard:

    -Rackmount.  I am soon to move my server to a 14U rack (if I can find one that will not require remortgaging my house) so the pfsense box will also move to the rack.
    -"Fairly" silent.  I put silent in quotes because I do realize that rackmount machines are usually server (enterprise oriented) machines located in server rooms.  Mine will be in my home office next to my bedroom so ultra loud hissing 80,000RPM fans are out of the question.
    -Max budget is around $250-300USD

    I have a 1m depth fully enclosed 14U at home, and it is actually quiet in the living room no less, I've managed to hide it like a piece of furniture. (ebay/craigslist are your friend, found an unused HP 10062 in a box on a pallet, got it shipped to the loading dock at work for $300)

    However to get to that point you are going to have to give up some things:
    -1U cases are not quiet, ever. Go to 4U, come to the dark side.
    -That is too cheap for an expandable system that won't have tiny screaming fans and can scale to gigabit routing. Not only do I think the hardware would be underpowered at $300, I think you need to budget at least $50-100 per system for aftermarket heatsink and fan considerations.
    -It is possible to go fanless, but far more expensive and little to no expansion



  • @jimp:

    It's louder than hell but it'll do the job well.

    http://www.alixbox.com/dellR200

    Now http://store.pfsense.org/R200/


Log in to reply