Recommendations - large firewall sandwich deployment
-
Thanks for the notes!
3. Firewall -> Virtual IP -> CARP Settings. See the various Synchronize items there. It will automatically sync changes from the primary to the secondary, etc.
Ah ok - I did not know you could use CARP just to sync config, and not do active/standby.
4. General rule of thumb is about 1k per state. http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pfvar.h?rev=1.234&content-type=text/x-cvsweb-markup
1K per TCP state? Woah…. I'm already way beyond that. We usually have between 1-2M concurrent connections in and out of this lab. I think my two pfsense boxes aren't going to be enough.
I actually am just trying to figure out why we just had a failure - it looks like I may have gone beyond the limits somewhere.... will know more soon.
-
Maybe you can add some more memory to each of the boxes. 8 gigabytes or so.
-
Maybe you can add some more memory to each of the boxes. 8 gigabytes or so.
I have 16GB in each already. Should I go up from there?
-
No, that is more than plenty for 1-2 million states.
-
No, that is more than plenty for 1-2 million states.
Is there something I need to do to allow pfsense to utilize that memory? I see message at bootup where it's ignoring anything above 4GB… and "top" output doesn't show it using any more that I can tell...
-
Well, that requires a kernel with PAE which we do not ship with. But I am looking into this for 2.0.
4 gigs should allow for 4 million or so states if my math servers me correctly.
-
Well, that requires a kernel with PAE which we do not ship with. But I am looking into this for 2.0.
4 gigs should allow for 4 million or so states if my math servers me correctly.
Ah ok. I'm not too worried then, especially since we plan on scaling up to 4 boxes for our final deployment. The most recent issue I was having appears to be because of my choice to use policy based routing, and I now have some sort of odd routing loop that is essentially DOSing everything with ICMP redirects. Impressive that everything is still running with the connection counts currently bouncing back and forth through the firewalls :)
-
Right now its is active/passive. With 2.0 it might be possible to change it to passive/passive but we have not tested if the CARP ARP changes in the newer version of PF make it worthwhile. On the older codebase there was some kind of issue with poor ARP when passive/passive was used.
The 2.0 builds have FreeBSD 8 which runs a lot better than previous versions.
-
Right now its is active/passive. With 2.0 it might be possible to change it to passive/passive but we have not tested if the CARP ARP changes in the newer version of PF make it worthwhile. On the older codebase there was some kind of issue with poor ARP when passive/passive was used.
The 2.0 builds have FreeBSD 8 which runs a lot better than previous versions.
Ah so for now I'm probably best sticking with standalone even - I'm actually sending traffic to both of the two firewalls, so CARP would limit that to just one that is active, correct?
-
Correct. Maybe you can have 2 CARP clusters with 4 machines total.
Send traffic to both, guessing with a load balancer of some sorts. I bet you all know a lot about those :)
-
Ok color me stupid for not thinking of that. Jeez.
Yup - I think we can handle the LB side of things :)
-
If you get creative you could control it all from the primary machine.
Syncing master -> secondary, third, fourth. Gets complicated but doable.
-
Good idea - I saw some of that in the tutorial by Holger Bauer that I just watched. Seems pretty straightforward. Starting the CARP configuration now…
-
One other question - does CARP replicate or synchronize state of sessions between the two firewalls? It's implied in a couple of places, but I want to make sure that is the case…
-
It does if "Synchronize Enabled" and "pfSync sync peer IP" is enabled aka pfsync.
The rest of the checkboxes + Synchronize to IP + Remote System Password is configuration sync.
-
Management seems to be working, but my test of failing over to the second firewall failed miserably. I simply rebooted the master, and the standby took over the CARP IPs - I could ping them, and things looked OK there, but all 100K existing sessions were either disconnected or simply hung, and no new sessions were allowed.
I know the upstream and downstream load balancers saw the IPs correctly - CARP uses a form of mac masquerading as far as I can tell, and the MAC address didn't change, so the LB's didn't care - they just kept sending data to the MAC in question.
I have to do some more digging, but something is definitely not working right here. I'm not sure if there is something I've overlooked, but I think I've got everything setup correctly…
The only thing I see on both systems, the master when it just comes up after preempting, and the slave after it takes over, is:
kernel: arp_rtrequest: bad gateway x.x.x.x (!AF_LINK)
But this message seems to be one that's been around a while. Nothing else in the logs...
One other question - could I send traffic to the non-CARP IP addresses and would it still get processed? I realize the session replication would only be from the master to the slave, but I could see a configuration where sending traffic to the slave that I didn't need any session data replicated would be useful... Or, after further reading, do all firewalls share state with eachother?
By the way, I bought the pfsense book :) I am sure some of this is in there, but I'd like to know as much now since it's the weekend, and I have a bit of flexibility to configure things before the book arrives on Monday.
-
Well, that requires a kernel with PAE which we do not ship with. But I am looking into this for 2.0.
4 gigs should allow for 4 million or so states if my math servers me correctly.
PAE may not get you what you think it will. PAE alters the page table layout to map a 32 bit virtual address to a 36 bit physical address potentially allowing 64GB of RAM. (Standard page table table maps a 32 bit virtual address to a 32 bit physical address.) The 32 bit virtual address still has to accommodate both kernel and user mode software. If I recall correctly the default FreeBSD build splits the 32 bit (4GB) virtual address space into an upper part of about 1GB for the kernel and a lower part of about 3GB for applications. The split can be tweaked at kernel build time to allow a 2GB chunk for kernel and 2GB for applications, 3GB for kernel, 1GB for applications. Many other combinations are possible. But however the space is split, 3GB is probably the practical limit of the kernel size. (This is true regardless of whether or not PAE is used.) If you want the kernel to grow to more than about 3GB of memory you have to go to 64bit mode which significantly increases the virtual address size.
PAE doesn't allow for a bigger kernel address space size nor for a bigger application size but it does allow for more applications to be memory resident at the same time, thus potentially reducing paging and/or swapping.
I haven't looked recently, but the PAE man page used to warn that a number of device drivers haven't been tested in a PAE configuration.
When last I tried it a few years, ago the 64 bit kernel was very easy to boot, a number of 32 bit applications I tried worked without problem (no tweaks required) and a number of applications recompiled under the 64 bit kernel worked without requiring any tweaks. (Two didn't, one had incorrectly declared a system call parameter as unsigned and the other was a networking application that exchanged a data structure containing a type (maybe it was int) that was 32 bits on the 32 bit system and 64 bits on the 64 bit system and so the 32 bit version didn't correctly converse with the 64 bit version.) There are a lot of 64 bit capable CPUs around (and have been for at least a couple of years) so there shouldn't be any serious impediments to putting together a development system. I'd be happy to converse with the pfSense developers if they think my experience might be helpful.
-
stevemitchell: sounds like pfsync is not syncing the states. might want to double check on the backup node that you see the state size increasing to match close to the primary. In terms of backup nodes, no, they will not process traffic in backup mode.
wallabybob: we are always looking for help.
-
stevemitchell: sounds like pfsync is not syncing the states. might want to double check on the backup node that you see the state size increasing to match close to the primary. In terms of backup nodes, no, they will not process traffic in backup mode.
I will check that - I had to take things out of the mix for now until I can figure out how I can maintain the source/internal address on the WAN side…
-
Use advanced outbound nat.
Also might want to use static port depending on your apps in use internally.