Recommendations - large firewall sandwich deployment



  • Hello,

    I just finished implementing two decent sized pfsense boxes between a large development/qa test lab and the rest of my company.  Our company makes a product that does load balancing / traffic management, so I was able to set the firewalls up independently (not CARP) and am load balancing ingoing and outgoing connections across them intelligently.  We call this a firewall sandwich.

    We chose pfsense because a lot of us use it for home and other situations, and it's rock solid.  Keep up the good development work!

    I'm looking for any feedback on tuning or suggestions when using pfsense in a corporate environment.  This system isn't connected to the internet, yet I'm using the WAN interface as my default route back out to the rest of our corporate network.

    I also am not using the LAN interface, but rather created a secondary interface so I could use policy based routing for inbound packets into the lab.  Otherwise, I would have been forced to create routes for every subnet inside our lab.  I tried using the LAN interface but could never get this working quite right.

    The main goal isn't to prevent people from getting into the lab, but prevent bad traffic from getting out.  So, I'm using pfsense in sort of a backwards way here - WAN traffic is all allowed in, traffic to the LAB interface is where the rules are that block various different things.

    Hardware is: Dell 1950 Dual Quad Core 2.66Ghz CPUs, 16GB RAM (I know pfsense doesn't care about more than 4GB, but this is the smallest config I could find), dual 4 port Intel Pro/1000 PCI-x NICs, 2x on board Broadcom NICs, RAID-5 array of 5 73GB 15k RPM disks.

    Traffic pattern is currently about 400Mb/sec across both of the firewalls, and bursts up to 4Gb/sec depending on user load.

    A few of my questions:

    1.  Any way to take advantage of the additional disk space on these systems?  I'd like to keep more logs, but things seem to truncate rather quickly.  Yes, I know I can use syslog to a remote server, but I like having them locally too.

    2.  Any recommendations for an "admin" interface - one I can use for administering the system only, that isn't in the normal packet path?  I'd like to use the LAN interface for this, but sometimes traffic is routed through this - I suppose I have to sit down and think more about my routing architecture.

    3.  Other than CARP, any recommendations for sync'ing the configs across firewalls?  Right now I'm doing all of my rule editing on one, and then exporting the config, going to the second, and importing it.  Works, but I'd like something more automatic from the CLI, etc.

    4.  What is the maximum state table size for pfsense?  I increased the default to 1M on both of these boxes, and it didn't complain, but I'd like to know what the max is - I assume it's a factor of memory…

    Thanks



  • 1.  Not so much.  We use a circular logging type device (CLOG).  There has been talk of maybe changing this around for v3.0.  v2.0 will be hitting beta around the 25th of December.

    2. Use the LAN interface.  First allow the webConfigurator port in firewall rules and then reject/block anything else.

    3. Firewall -> Virtual IP -> CARP Settings.  See the various Synchronize items there.  It will automatically sync changes from the primary to the secondary, etc.

    4. General rule of thumb is about 1k per state.  http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pfvar.h?rev=1.234&content-type=text/x-cvsweb-markup

    Spread the word about pfS in that company of yours.  We are honored that you are using it :)



  • Thanks for the notes!

    @sullrich:

    3. Firewall -> Virtual IP -> CARP Settings.  See the various Synchronize items there.   It will automatically sync changes from the primary to the secondary, etc.

    Ah ok - I did not know you could use CARP just to sync config, and not do active/standby.

    @sullrich:

    4. General rule of thumb is about 1k per state.  http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pfvar.h?rev=1.234&content-type=text/x-cvsweb-markup

    1K per TCP state?  Woah…. I'm already way beyond that.  We usually have between 1-2M concurrent connections in and out of this lab.  I think my two pfsense boxes aren't going to be enough.

    I actually am just trying to figure out why we just had a failure - it looks like I may have gone beyond the limits somewhere.... will know more soon.



  • Maybe you can add some more memory to each of the boxes.  8 gigabytes or so.



  • @sullrich:

    Maybe you can add some more memory to each of the boxes.  8 gigabytes or so.

    I have 16GB in each already.  Should I go up from there?



  • No, that is more than plenty for 1-2 million states.



  • @sullrich:

    No, that is more than plenty for 1-2 million states.

    Is there something I need to do to allow pfsense to utilize that memory?  I see message at bootup where it's ignoring anything above 4GB… and "top" output doesn't show it using any more that I can tell...



  • Well, that requires a kernel with PAE which we do not ship with.  But I am looking into this for 2.0.

    4 gigs should allow for 4 million or so states if my math servers me correctly.



  • @sullrich:

    Well, that requires a kernel with PAE which we do not ship with.  But I am looking into this for 2.0.

    4 gigs should allow for 4 million or so states if my math servers me correctly.

    Ah ok.  I'm not too worried then, especially since we plan on scaling up to 4 boxes for our final deployment.  The most recent issue I was having appears to be because of my choice to use policy based routing, and I now have some sort of odd routing loop that is essentially DOSing everything with ICMP redirects.  Impressive that everything is still running with the connection counts currently bouncing back and forth through the firewalls :)



  • Right now its is active/passive.  With 2.0 it might be possible to change it to passive/passive but we have not tested if the CARP ARP changes in the newer version of PF make it worthwhile.  On the older codebase there was some kind of issue with poor ARP when passive/passive was used.

    The 2.0 builds have FreeBSD 8 which runs a lot better than previous versions.



  • @sullrich:

    Right now its is active/passive.   With 2.0 it might be possible to change it to passive/passive but we have not tested if the CARP ARP changes in the newer version of PF make it worthwhile.   On the older codebase there was some kind of issue with poor ARP when passive/passive was used.

    The 2.0 builds have FreeBSD 8 which runs a lot better than previous versions.

    Ah so for now I'm probably best sticking with standalone even - I'm actually sending traffic to both of the two firewalls, so CARP would limit that to just one that is active, correct?



  • Correct.  Maybe you can have 2 CARP clusters with 4 machines total.

    Send traffic to both, guessing with a load balancer of some sorts.  I bet you all know a lot about those :)



  • Ok color me stupid for not thinking of that.  Jeez.

    Yup - I think we can handle the LB side of things :)



  • If you get creative you could control it all from the primary machine.

    Syncing master -> secondary, third, fourth.   Gets complicated but doable.



  • Good idea - I saw some of that in the tutorial by Holger Bauer that I just watched.  Seems pretty straightforward.  Starting the CARP configuration now…



  • One other question - does CARP replicate or synchronize state of sessions between the two firewalls?  It's implied in a couple of places, but I want to make sure that is the case…



  • It does if "Synchronize Enabled" and "pfSync sync peer IP" is enabled aka pfsync.

    The rest of the checkboxes + Synchronize to IP + Remote System Password is configuration sync.



  • Management seems to be working, but my test of failing over to the second firewall failed miserably.  I simply rebooted the master, and the standby took over the CARP IPs - I could ping them, and things looked OK there, but all 100K existing sessions were either disconnected or simply hung, and no new sessions were allowed.

    I know the upstream and downstream load balancers saw the IPs correctly - CARP uses a form of mac masquerading as far as I can tell, and the MAC address didn't change, so the LB's didn't care - they just kept sending data to the MAC in question.

    I have to do some more digging, but something is definitely not working right here.  I'm not sure if there is something I've overlooked, but I think I've got everything setup correctly…

    The only thing I see on both systems, the master when it just comes up after preempting, and the slave after it takes over, is:

    kernel: arp_rtrequest: bad gateway x.x.x.x (!AF_LINK)

    But this message seems to be one that's been around a while.  Nothing else in the logs...

    One other question - could I send traffic to the non-CARP IP addresses and would it still get processed?  I realize the session replication would only be from the master to the slave, but I could see a configuration where sending traffic to the slave that I didn't need any session data replicated would be useful...  Or, after further reading, do all firewalls share state with eachother?

    By the way, I bought the pfsense book :)  I am sure some of this is in there, but I'd like to know as much now since it's the weekend, and I have a bit of flexibility to configure things before the book arrives on Monday.



  • @sullrich:

    Well, that requires a kernel with PAE which we do not ship with.  But I am looking into this for 2.0.

    4 gigs should allow for 4 million or so states if my math servers me correctly.

    PAE may not get you what you think it will. PAE alters the page table layout to map a 32 bit virtual address to a 36 bit physical address potentially allowing 64GB of RAM. (Standard page table table maps a 32 bit virtual address to a 32 bit physical address.) The 32 bit virtual address still has to accommodate both kernel and user mode software. If I recall correctly the default FreeBSD build splits the 32 bit (4GB) virtual address space into an upper part of about 1GB for the kernel and a lower part of about 3GB for applications. The split can be tweaked at kernel build time to allow a 2GB chunk for kernel and 2GB for applications, 3GB for kernel, 1GB for applications. Many other combinations are possible. But however the space is split, 3GB is probably the practical limit of the kernel size. (This is true regardless of whether or not PAE is used.) If you want the kernel to grow to more than about 3GB of memory you have to go to 64bit mode which significantly increases the virtual address size.

    PAE doesn't allow for a bigger kernel address space size nor for a bigger application size but it does allow for more applications to be memory resident at the same time, thus potentially reducing paging and/or swapping.

    I haven't looked recently, but the PAE man page used to warn that a number of device drivers haven't been tested in a PAE configuration.

    When last I tried it a few years, ago the 64 bit kernel was very easy to boot, a number of 32 bit applications I tried worked without problem (no tweaks required) and a number of applications recompiled under the 64 bit kernel worked without requiring any tweaks. (Two didn't, one had incorrectly declared a system call parameter as unsigned and the other was a networking application that exchanged a data structure containing a type (maybe it was int) that was 32 bits on the 32 bit system and 64 bits on the 64 bit system and so the 32 bit version didn't correctly converse with the 64 bit version.) There are a lot of 64 bit capable CPUs around (and have been for at least a couple of years) so there shouldn't be any serious impediments to putting together a development system. I'd be happy to converse with the pfSense developers if they think my experience might be helpful.



  • stevemitchell: sounds like pfsync is not syncing the states.  might want to double check on the backup node that you see the state size increasing to match close to the primary.  In terms of backup nodes, no, they will not process traffic in backup mode.

    wallabybob: we are always looking for help.



  • @sullrich:

    stevemitchell: sounds like pfsync is not syncing the states.  might want to double check on the backup node that you see the state size increasing to match close to the primary.   In terms of backup nodes, no, they will not process traffic in backup mode.

    I will check that - I had to take things out of the mix for now until I can figure out how I can maintain the source/internal address on the WAN side…



  • Use advanced outbound nat.

    Also might want to use static port depending on your apps in use internally.



  • @sullrich:

    Also might want to use static port depending on your apps in use internally.

    I did switch to the advanced outbound NAT and things seem to be showing up source address-wise correctly.

    Why would I want to use static ports?  Are you thinking of apps that don't like to be translated or routed?



  • By default PF scrambles the source port so if your applications are expecting the traffic on a specific source port it might  confuse them.



  • A gotcha.  Any way to disable that?  I saw various options that sounded like they might do that…



  • Advanced outbound nat, edit interface entry, click use static port.



  • I am on vacation now and will be slow to respond.  Hopefully the book can help you in the meantime.


Log in to reply