10Gb/s connected pfSense firewalls

bubble1975 · Nov 9, 2011, 5:36 AM

Hi Y'all,

I'm exploring the 10Gb/s possibilities on several firewalls I have. I'm seeing some performance issues that seem to point to the need for some kernel tuning on the box. I see this website:

http://fasterdata.es.net/fasterdata/host-tuning/freebsd/

Which suggests, for 10Gb/s connected hosts:

set to at least 16MB for 10GE hosts

kern.ipc.maxsockbuf=16777216
# set autotuning maximum to at least 16MB too
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
# enable send/recv autotuning
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
# increase autotuning step size
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
# turn off inflight limitting
net.inet.tcp.inflight.enable=0
# set this on test/measurement hosts
net.inet.tcp.hostcache.expire=1

I looked at a 2.0-RELEASE pfSense firewall (amd64) and saw:

kern.ipc.maxsockbuf: 262144
net.inet.tcp.sendbuf_max: 262144
net.inet.tcp.recvbuf_max: 262144
net.inet.tcp.sendbuf_auto: 1
net.inet.tcp.recvbuf_auto: 1
net.inet.tcp.sendbuf_inc: 8192
net.inet.tcp.recvbuf_inc: 16384
net.inet.tcp.inflight.enable: 1

As you can see, some of those numbers are a lot smaller than the recommended (according to that website, at least). And tcp.inflight.enable is enabled, when they say disable it.

The million dollar question: Can anyone think of any reason why I shouldn't modify sysctl.conf to reflect the "recommended" items listed above? Seems like I could get a big performance boost from it, maybe…

bubble1975 · Nov 9, 2011, 7:18 AM

Or rather, modify the corresponding "System Tunables" via the web configurator? Probably the better method…

Nachtfalke · Nov 9, 2011, 6:28 PM

Just try with system tunables. If you get a boost it is ok if not than the "problem" is something else.
System tunables are a good way to try that.

bubble1975 · Nov 11, 2011, 8:42 PM

I dropped in my tunables and it didn't seem to help my problem. I'll leave the tunables in there anyway just because it seems like they should be there for 10Gb/s connectivity.

I seem to be hitting a ceiling of about 2Gb/s transfer rates incoming. I can't see any dropped packets and I can't see any buffer problems on the firewall or my transfer servers. Looking at the live traffic graph, I see a sawtooth patten that is usually indicative of TCP scaling up and down. My understanding is that this is because of dropped packets somewhere.

If I have 4 cores in my firewall server, and I see 10% CPU utilization on my pfSense dashboard, does that mean 10% of total possible CPU power on the box? In other words, 10% means like 40% of one core? Or does 10% just mean 10% of one core, and not 10% of the total power of the 4 cores?

Nachtfalke · Nov 11, 2011, 9:03 PM

As far as I know system tunables need a reboot. Not sure about that.

If dashboard shows 10% that means that one CPU uses 40% or 2 CPUs use 2x20%.

I changed that - but not sure if this improved performance or not:

kern.ipc.somaxconn

The kern.ipc.somaxconn sysctl variable limits the size of the listen queue for accepting new TCP connections. The default value of 128 is typically too low for robust handling of new connections in a heavily loaded web server environment. For such environments, it is recommended to increase this value to 1024 or higher.

2048

kern.ipc.nmbclusters

The NMBCLUSTERS kernel configuration option dictates the amount of network Mbufs available to the system. A heavily-trafficked server with a low number of Mbufs will hinder FreeBSD's ability. Each cluster represents approximately 2 K of memory, so a value of 1024 represents 2 megabytes of kernel memory reserved for network buffers. A simple calculation can be done to figure out how many are needed. If you have a web server which maxes out at 1000 simultaneous connections, and each connection eats a 16 K receive and 16 K send buffer, you need approximately 32 MB worth of network buffers to cover the web server. A good rule of thumb is to multiply by 2, so 2x32 MB / 2 KB = 64 MB / 2 kB = 32768\. We recommend values between 4096 and 32768 for machines with greater amounts of memory.

65536

net.inet.tcp.sendbuf_max 

16777216

net.inet.tcp.recvbuf_max 

16777216

kern.maxfilesperproc

Set maximum files allowed open per process 

32768

kern.maxfiles

Set maximum files allowed open 

262144

net.inet.ip.intr_queue_maxlen 

Maximum size of the IP input queue 

3000

bubble1975 · Nov 11, 2011, 9:28 PM

Ah, OK… So, if the packet filtering process is single threaded and uses one core, then I am CPU bound if my dashboard reads 25% (25% being 1 core of 4 being fully pegged)? But, even if I were CPU bound, wouldn't I begin to see dropped packets on the interfaces?

Thanks!

bubble1975 · Nov 12, 2011, 4:56 AM

OK, here's some interesting stuff on tuning FreeBSD firewalls:

https://calomel.org/network_performance.html

They say to do this in loader.conf:

/boot/loader.conf

autoboot_delay="3" # reduce boot menu delay from 10 to 3 seconds
inet.tcp.tcbhashsize=4096 # tcb hash size
loader_logo="beastie" # old FreeBSD logo menu
net.inet.tcp.syncache.hashsize=1024 # syncache hash size
net.inet.tcp.syncache.bucketlimit=100 # syncache bucket limit
net.isr.bindthreads=0 # do not bind threads to CPUs
net.isr.direct=1 # interrupt handling via multiple CPU
net.isr.direct_force=1 # "
net.isr.maxthreads=3 # Max number of threads for NIC IRQ balancing (4 cores in box)
vm.kmem_size=1G # physical memory available for kernel (320Mb by default)

It looks like those tunables can't be set by sysctl (and hence not by the "System Tunables" in pfSense). I'm hesitant to change loader.conf under pfSense (and then reboot, I assume) - but the last four items look like they might help 10G performance a lot, possibly. Anyone heard of this stuff?

Nachtfalke · Nov 12, 2011, 12:23 PM

I checked which are the default settings on my system:

sysctl net.isr.direct
net.isr.direct: 1

sysctl vm.kmem_size
vm.kmem_size: 435544320

sysctl net.isr.maxthreads
net.isr.maxthreads: 1

sysctl net.isr.direct_force
net.isr.direct_force: 1

sysctl net.isr.bindthreads
net.isr.bindthreads: 0

You have to be careful on which version of BSD this tuning guides are based. pfsense 2.0 is using freebsd 8.1.

In the past I found much tuning tip which based on old versions.

Instead of putting the tuning parameters in loader.conf put the im loader.conf.local (perhaps you need to create it). This file will not be overwritten after a firmware update. This was nice to know when I used the beta and RC versions of pfsense with daily updates.

But I think you can create new system tunables in GUI because the tuning parameters above can all bet set by sysctl. I think this is the same as system tunables are doing. But a reboot will be neccessary I think.

bubble1975 · Nov 12, 2011, 6:41 PM

I was able to set most the the loader.conf stuff, but net.isr.maxthreads seems to always be set to "1" even if I try to set it higher. I did some digging and it seems like it is set on boot and is based on the number of CPU cores you have available. Which is weird, because I see 8 cores:

[2.0-RELEASE][admin@server]/root(8): dmesg | grep maxthreads
netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling

[2.0-RELEASE][admin@server]/root(9): dmesg | grep CPU
CPU: Intel(R) Xeon(R) CPU X5677 @ 3.47GHz (3458.02-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
cpu4: <acpi cpu="">on acpi0
cpu5: <acpi cpu="">on acpi0
cpu6: <acpi cpu="">on acpi0
cpu7: <acpi cpu="">on acpi0
est: CPU supports Enhanced Speedstep, but is not recognized.
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
p4tcc4: <cpu frequency="" thermal="" control="">on cpu4
est: CPU supports Enhanced Speedstep, but is not recognized.
p4tcc5: <cpu frequency="" thermal="" control="">on cpu5
est: CPU supports Enhanced Speedstep, but is not recognized.
p4tcc6: <cpu frequency="" thermal="" control="">on cpu6
est: CPU supports Enhanced Speedstep, but is not recognized.
p4tcc7: <cpu frequency="" thermal="" control="">on cpu7
SMP: AP CPU #1 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #5 Launched!

Seems like netisr_init is overriding my preferences at boot time… Anyone know a way around this? It seems like netisr_init thinks I have one CPU, if I am reading this right...

I can't set net.isr.maxthreads manually either:

[2.0-RELEASE][admin@server]/root(10): sysctl net.isr.maxthreads=3
sysctl: oid 'net.isr.maxthreads' is read only</cpu></cpu></cpu></cpu></cpu></cpu></cpu></cpu></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi>

dhatz · Nov 12, 2011, 6:57 PM

I suggest that you also post in FreeBSD-net mailing list (http://lists.freebsd.org/mailman/listinfo/freebsd-net)

Nachtfalke · Nov 12, 2011, 10:38 PM

Did you set these parameters using loader.conf or by system tuanbles (GUI) ?

bubble1975 · Nov 13, 2011, 3:55 PM

I set the following tunables via loader.conf.local:

inet.tcp.tcbhashsize=4096
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100
net.isr.bindthreads=0
net.isr.direct=1
net.isr.direct_force=1
net.isr.maxthreads=3
vm.kmem_size_max=12G
vm.kmem_size=10G

They all worked on a reboot except net.isr.maxthreads=3. I contacted the FreeBSD folks about that one, and they said that there is a limitation right now in the kernel where you cannot set net.isr.maxthreads above 1 if DEVICE_POLLING support is compiled into the kernel. Which it seems to be with pfSense 2.0-RELEASE. They are working on fixing that limitation, but as far as pfSense goes, it's not an option, unless pfSense 3.0 has it. ;)

It's a shame though, because it seems like it would help parallel interrupt handling and as such increase the max bandwidth of the box. And I'm not about to rebuild a kernel for pfSense, somehow I think I'd end up trashing more stuff than I was "fixing". So for now I'm just playing with my buffer sizes for various tunables via sysctl (ala "System Tunables"), now that I've learned how to increase the kernel memory space.

clarknova · Nov 14, 2011, 4:28 PM

Try 'top -SH' in a shell to see detailed CPU usage. My experience with the Intel 'em' driver is that it threads well, so throughput should benefit from multiple cores, at least with that driver.

It has been said on this forum that you should turn hyperthreading off, although last time I looked into that I couldn't really see a good reason why.