Datacenter Firewall Pair
-
Hey all,
So I've been using pfsense as an affordable way to get VPN connectivity to customer sites when they can't budget for dedicated gear. I also use it at home as a router/wireless gateway/vpn/etc..
I've recently been plagued with network issues at my Data Center, specifically… with the provider firewalls as it's leased space and I don't control the WAN edge at all.
However, after begging them to let me take over routing and filtering, I decided to build out my own enterprise level HA pair using the following:
2x HP DL 360 G5's
2x Quad E5420's @2.5GHZ
8GB of ram (4GB usable in memory mirror)
2x Intel dual port gig nics (WAN/LAN) + 2 port Broadcom on board (CARP/Management)
Raid 1 with hot spare
Dual PSU'sSo.. highly available and redundant hardware.
Both servers are configured with pfsense 2.0.2 and are running CARP.
The provider WAN is dual 1Gb network internet connections, with two separate gateways and IP blocks. I am using policy based firewall rules to route everything correctly.
I set up test boxes on both networks internal and external, and was able to completely saturate the WAN links through HTTP and FTP downloads. Since I don't have the hardware, or the disk speed to get better performance, I'm not sure where the box is limited in terms of throughput, but it certainly handles 2Gb/s aggregate just fine. (Nothing is more thrilling than downloading giant HD video files in seconds.)
So now that you have an introduction.. here's what I'm looking for:
1. A good way to test IMIX scenarios over both connections simultaneously. It would be great if I could simulate a lot of different traffic, packet sizes, types, good/bad packets, etc.. I really want to give this thing a thorough performance test. No need to worry about test hardware, I have enough to create most configurations so throw those ideas out.
2. I also want to kick the crap out of it until it keels over and dies. Currently the only issue I've found is logging and the box being unable to write to disk fast enough and locks up slightly. I'm going to be implementing a central logging server, and pushing the log messages out over the management interface, so knowing how to prevent messages from being logged locally should clear this issue up. The logging issue arose when I was writing between 75,000 and 100,000 lines of logging per second, so you can see why there was an issue. The other option is to attach some SAN storage for logging only, this would need to be over iSCSI so I hope the OS is capable of that. I like to be able to log blocks... and since this was a flooding scenario a way to suppress duplicate messages would be wonderful.
3. General performance tuning and tips... since these are purely routed firewalls, the priority is that they are able to handle potentially millions of packets per second, and still be able to filter correctly. Anything you can suggest for high performance environments would be great!
Overall.. I'm extremely impressed with pfsense and a high performance environment. We were considering purchasing a pair of Juniper SRX 240's to do this same job, but if the pfsense boxes work out... We are going to table that budget item. ;)
Also.. feel free to ask questions, or call me names for over-provisioning these firewalls. The hardware was part of our old XenServer pool and was either going to sit in a closet or be re-purposed.
-
Here's a screen cap of the dashboard for those who are interested. I have all of the WAN/LAN testing hardware powered down right now so I'm just sending traffic over the Management link.
-
Hi Mike:
Your needs are way above my paygrade! But you might consider commercial support if you don't get the answers your looking for. https://portal.pfsense.org/index.php/support-subscription
Are your interfaces bridged or the subnets kept small?
:)
-
Good hardware choice for that kind of environment. It's well beyond the performance of a SRX240, especially in new connections per second (only 8500 vs. you're getting 100K/sec if you're getting 100K logs/sec), and maximum states (128K or 256K on the Juniper depending on which memory, vs. easily 3 million in 4 usable GB RAM). Millions of packets per second may be beyond what the packet filter can handle (and is WAY beyond what a SRX240 will handle) depending on how many millions. May get 3 Mpps or so out of that.
re: load generation, it depends on the profile of traffic the network will push in production. A network with primarily web servers generates a much different load profile than one of primarily VoIP PBXes, for instance. There are a variety of load generating tools out there to simulate various things, use those to generate traffic similar to what you'll have on the network.
But in short, if you think a SRX240 would have sufficed for you, at least in max new connections per second and max states, you've proved you're at least one order of magnitude more scalable than a SRX240 would be. And more throughput in general since you pegged 2 Gbps, where the best case maximum for a SRX240 is 1.5 Gbps.
re: tuning: Bump your maximum states up to 3000000 (very important in a DC, and you'll spend as much as a house on a commercial firewall that can handle that many connections). The 128-256K maximum connections on the SRX240 would get blown away in a second under a small DDoS, heck a single host on a DSL line could knock that over pretty quickly. Hitting maximum states or maximum state insertions (new connections) per second is where DC firewalls generally get melted down.
Also check:
http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cardsother than that, defaults are good.
-
First.. thanks for your replies!
I've been quickly confirming your expectation that this meets and exceeds the ability of an SRX 240. I feel extremely fortunate at this point.
So far I've done the following:
Increase MBUF to 655536
Increase States to 3,000,000
Increased no. of Tables to 10,000
Increased no. of Table Entries to 500,000
Increased the IP input queue to 3000
Set firewall mode to 'conservative' << which is best here? By the description, I would assume conservative since I have the memory.I was able to drain all 4GB of memory by running a half-open UDP flood… but what was using all of the memory was ntop. Anyone have an idea why? I like ntop so far, but if it's going to hog all of the memory and crash, then that isn't good. :)
As far as remote logging and performance... will I take a hit if I'm forwarding out 100,000+ lines of logging out of the Management interface to a collector? I'd think there would be some performance penalty, but I can't imagine much. Also, does anyone have a good estimate on how much 1 log entry is in disk space? Or.. 10.. 100,000... I'm assuming it's either the size of or smaller than a 4KB disk block so. I just want to estimate how much disk I'm going to have to allocate.
Also might be kicking these up to 8GB of mirrored memory as I just sent two more of these blades out to pasture. Should be interesting. :)
Can anyone recommend any good IDS settings for Snort or any DDoS suppression-type programs? I know there's not much you can do if someone throws 10Gb/s of crap traffic at you, but this is going to be development for a few more weeks, if not months, before I deploy.
I am also going to look into commercial support! I'm sure jimp lurks around there as well.. ;)
-
ntop is a resource hog, I wouldn't run it on a firewall that needs to be highly scalable. Netflow is a better solution in such networks.
I probably wouldn't put state keeping to conservative, only if you're running VoIP with clients you can't easily control that may not have very sane keepalive settings.
-
So I was able to achieve 70,000 pps sustained with an IMIX of UDP packets.
However when I run: sysctl net.inet.ip.intr_queue_drops
I see the number incrementing quite quickly, net.inet.ip.intr_queue_drops: 645011625 :)
Any idea on what I can do here? I booted the IP Input Queue to 3000, then 10000, and stopped at 25000 with no real results.
It could be that some of the traffic I'm generating is being dropped anyway.
Any opinions on whether or not I should use the traffic shaper? PriQ has always served me well with slower connections, but I'd be apprehensive about having queues that might not be optimized for 1Gb links.
