State Table Sizing and RAM and the Kernel…
-
I just visit the pfSense hardware store and noticed the "max 1.8M active connections" spec for a 2GB model; 8M connection for an 8GB model
That raises a few questions for me…
AFAIR, each connection takes two states, one inbound and one outbound state. AFAIK, each state consumes about 1kB of RAM. Yes, the structure of a state table entry is quite complex, AFAIR. It has been quite a while since I've looked into the kernel source, but the 1kB is probably more realistic than, let's say, 500 bytes. Well - it could be that the marketing concept of a connection differs from that of a TCP connection. Like some firewall manufacturers specify the switching bandwidth of their firewalls instead of their filtering bandwidth.
What really made me thinking was that I thought to remember that the state table is kept in kernel memory. And I thought that I remembered that the kernel memory size limit was 512MB or so. If the kernel tries to allocate more RAM than allowed, than it would crash (more or lees, don't exactly remember the details).
Obviously, the kernel memory limit can be raised. My, um "knowledge" above is probably from some FreeBSD 8.x version and the 2.0.x pfSense versions. However, I assume that going past 32 bit of memory addresses is probably not an option for kernel memory.
So how does pfSense achieve the mentioned maximum numbers of active connections? Or is it just that my brain hit some memory limit and is now producing garbage?
-
And I thought that I remembered that the kernel memory size limit was 512MB or so.
Checking the FreeBSD documentation I found:
By default, the kernel address space is 1 GB (2 GB for PAE) for i386.
However, before reading the documentation, I had tried it out already. Test platform was my main pfSense box with the i386 version 2.1.5-RELEASE installed. It has 2GB of physical RAM installed.
I set the firewall state limit to 2.000.000 states (2M), set the irewall Optimization Options to "conservative" and disabled scrubbing (all options located on the System: Advanced: Firewall and NAT page for our convenience). Then I launched nmap on my Win7 box, choosing quick SYN pings and "insane" parallelism (like, dunno, 4k or 32k simultaneous hosts or so) with timeouts of one second or so, over a public class A subnet. Yes, this would be completely nuts if I'd really intend to receive results from nmap, but I chose this setup because I just wanted to flood the router with open connections. Note that the connection throttling which was introduced in WinXP SP2 was removed from Windows in Vista SP1 or SP2 and is also not present in Win7. The "SYN ping scan" apparently did not disrupt Windows networking too much (had I tried this with a tool like "Advanced IP Scanner", Windows networking would have been DOSed).
After connecting the VGA console to my pgSense box, I launched nmap and closely observed the dashboard and the VGA console. State count and memory utilization went up nicely, until it reached something around 1.2 million states and perhaps 60+% memory utilization and I noticed a kernel panic on the VGA console. I tried to produce a printout from the (now stuck) pfSense dashboard, but I had to learn that using MS IE had been a dumb mistake. So the exact numbers are now lost.
Of course, pfSense rebooted automatically and was backup soon. The crash report mentions:
root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8
Panic String: kmem_malloc(8192): kmem_map too small: 435548160 total allocatedSo, guessing from the memory utilization of somewhere between 60 and 70% on a 2GB machine (with the typical "idle" value being 14%), it looks indeed like the kernel limit is 1GB.
So, 1.200.000 (1.2M) states appear to work fine on a standard i386 pfSnese install. That's estimated to be enough for 10.000 concurrent users, of the "average office worker kind". Well, some firewall/router guys, probably from marketing go as low at 10 connection/user, but as I cannot possibly cram not even 10.000 users into my home, I do not need to think about how I could serve 120.000 users.
Hm, per VPN perhaps. They wouldn't need to be in my home. I'd need at least a /15 subnet to provide enough IP addresses…set firewall optimization to aggressive, use Squid to reduce WANl connection...yes, I really thing I'm going mental now.
I set the state limit to 1M now. Just because it sounds better to talk nerdily about the option of "1 million concurrent connection" than the boooring default 201.000 setting (default for a 2GB macine, YMMV).
Obviously, the above is true for an i386 install. A 64 bit install might fare much better.
I wonder if an 8GB machine, perhaps with a custom kernel, could accommodate 10M states? That would be pretty close to the edge, but should work.
-
Klaws, thanks for the informative write-up. Based on what I could quickly find, it seems KVA_PAGES is an i386 only compile time setting. I assume 64bit in unaffected and will make use of whatever you throw at it. There are so many knobs to tweak FreeBSD for extreme cases.
Again, thanks!
-
I know this is an old thread but it kinda helped me so I want to put my final solution here for anyone else who happens along. I had to delete the bogonsv6 file to solve the continuing errors. When throwing the errors the state table size was only a couple of percent. The MBuf was less than 10% so not sure where it went sideways. I did have tcpv6 enabled but then restored an older configuration file then noticed the errors growing. Deleting the file made all well again.
-
This post is deleted! -
A lot has changed since 2014. If you're seeing the bogonsv6 table limit make sure the Firewall Maximum Table Entries is set to 400000 in System > Advanced > Firewall & NAT. That is the default value in current pfSense.
Steve
