10Gb/s connected pfSense firewalls



  • Hi Y'all,

    I'm exploring the 10Gb/s possibilities on several firewalls I have.  I'm seeing some performance issues that seem to point to the need for some kernel tuning on the box.  I see this website:

    http://fasterdata.es.net/fasterdata/host-tuning/freebsd/

    Which suggests, for 10Gb/s connected hosts:

    set to at least 16MB for 10GE hosts

    kern.ipc.maxsockbuf=16777216
      # set autotuning maximum to at least 16MB too
      net.inet.tcp.sendbuf_max=16777216 
      net.inet.tcp.recvbuf_max=16777216
      # enable send/recv autotuning
      net.inet.tcp.sendbuf_auto=1
      net.inet.tcp.recvbuf_auto=1
      # increase autotuning step size
      net.inet.tcp.sendbuf_inc=16384
      net.inet.tcp.recvbuf_inc=524288
      # turn off inflight limitting
      net.inet.tcp.inflight.enable=0
      # set this on test/measurement hosts
      net.inet.tcp.hostcache.expire=1

    I looked at a 2.0-RELEASE pfSense firewall (amd64) and saw:

    kern.ipc.maxsockbuf: 262144
    net.inet.tcp.sendbuf_max: 262144
    net.inet.tcp.recvbuf_max: 262144
    net.inet.tcp.sendbuf_auto: 1
    net.inet.tcp.recvbuf_auto: 1
    net.inet.tcp.sendbuf_inc: 8192
    net.inet.tcp.recvbuf_inc: 16384
    net.inet.tcp.inflight.enable: 1

    As you can see, some of those numbers are a lot smaller than the recommended (according to that website, at least).  And tcp.inflight.enable is enabled, when they say disable it.

    The million dollar question:  Can anyone think of any reason why I shouldn't modify sysctl.conf to reflect the "recommended" items listed above?  Seems like I could get a big performance boost from it, maybe…



  • Or rather, modify the corresponding "System Tunables" via the web configurator?  Probably the better method…



  • Just try with system tunables. If you get a boost it is ok if not than the "problem" is something else.
    System tunables are a good way to try that.



  • I dropped in my tunables and it didn't seem to help my problem.  I'll leave the tunables in there anyway just because it seems like they should be there for 10Gb/s connectivity.

    I seem to be hitting a ceiling of about 2Gb/s transfer rates incoming.  I can't see any dropped packets and I can't see any buffer problems on the firewall or my transfer servers.  Looking at the live traffic graph, I see a sawtooth patten that is usually indicative of TCP scaling up and down.  My understanding is that this is because of dropped packets somewhere.

    If I have 4 cores in my firewall server, and I see 10% CPU utilization on my pfSense dashboard, does that mean 10% of total possible CPU power on the box?  In other words, 10% means like 40% of one core?  Or does 10% just mean 10% of one core, and not 10% of the total power of the 4 cores?



  • As far as I know system tunables need a reboot. Not sure about that.

    If dashboard shows 10% that means that one CPU uses 40% or 2 CPUs use 2x20%.

    I changed that - but not sure if this improved performance or not:

    kern.ipc.somaxconn
    
    The kern.ipc.somaxconn sysctl variable limits the size of the listen queue for accepting new TCP connections. The default value of 128 is typically too low for robust handling of new connections in a heavily loaded web server environment. For such environments, it is recommended to increase this value to 1024 or higher.
    
    2048 
    
    kern.ipc.nmbclusters
    
    The NMBCLUSTERS kernel configuration option dictates the amount of network Mbufs available to the system. A heavily-trafficked server with a low number of Mbufs will hinder FreeBSD's ability. Each cluster represents approximately 2 K of memory, so a value of 1024 represents 2 megabytes of kernel memory reserved for network buffers. A simple calculation can be done to figure out how many are needed. If you have a web server which maxes out at 1000 simultaneous connections, and each connection eats a 16 K receive and 16 K send buffer, you need approximately 32 MB worth of network buffers to cover the web server. A good rule of thumb is to multiply by 2, so 2x32 MB / 2 KB = 64 MB / 2 kB = 32768\. We recommend values between 4096 and 32768 for machines with greater amounts of memory.
    
    65536
    
    net.inet.tcp.sendbuf_max 
    
    16777216
    
    net.inet.tcp.recvbuf_max 
    
    16777216 
    
    kern.maxfilesperproc
    
    Set maximum files allowed open per process 
    
    32768 
    
    kern.maxfiles
    
    Set maximum files allowed open 
    
    262144 
    
    net.inet.ip.intr_queue_maxlen 
    
    Maximum size of the IP input queue 
    
    3000 
    


  • Ah, OK…  So, if the packet filtering process is single threaded and uses one core, then I am CPU bound if my dashboard reads 25% (25% being 1 core of 4 being fully pegged)?  But, even if I were CPU bound, wouldn't I begin to see dropped packets on the interfaces?

    Thanks!



  • OK, here's some interesting stuff on tuning FreeBSD firewalls:

    https://calomel.org/network_performance.html

    They say to do this in loader.conf:

    /boot/loader.conf

    autoboot_delay="3"                    # reduce boot menu delay from 10 to 3 seconds
    inet.tcp.tcbhashsize=4096            # tcb hash size
    loader_logo="beastie"                # old FreeBSD logo menu
    net.inet.tcp.syncache.hashsize=1024  # syncache hash size
    net.inet.tcp.syncache.bucketlimit=100 # syncache bucket limit
    net.isr.bindthreads=0                # do not bind threads to CPUs
    net.isr.direct=1                      # interrupt handling via multiple CPU
    net.isr.direct_force=1                # "
    net.isr.maxthreads=3                  # Max number of threads for NIC IRQ balancing (4 cores in box)
    vm.kmem_size=1G                      # physical memory available for kernel (320Mb by default)

    It looks like those tunables can't be set by sysctl (and hence not by the "System Tunables" in pfSense).  I'm hesitant to change loader.conf under pfSense (and then reboot, I assume) - but the last four items look like they might help 10G performance a lot, possibly.  Anyone heard of this stuff?



  • I checked which are the default settings on my system:

    sysctl net.isr.direct
    net.isr.direct: 1
    
    
    sysctl vm.kmem_size
    vm.kmem_size: 435544320
    
    
    sysctl net.isr.maxthreads
    net.isr.maxthreads: 1
    
    
    sysctl net.isr.direct_force
    net.isr.direct_force: 1
    
    
    sysctl net.isr.bindthreads
    net.isr.bindthreads: 0
    
    

    You have to be careful on which version of BSD this tuning guides are based. pfsense 2.0 is using freebsd 8.1.

    In the past I found much tuning tip which based on old versions.

    Instead of putting the tuning parameters in loader.conf put the im loader.conf.local (perhaps you need to create it). This file will not be overwritten after a firmware update. This was nice to know when I used the beta and RC versions of pfsense with daily updates.

    But I think you can create new system tunables in GUI because the tuning parameters above can all bet set by sysctl. I think this is the same as system tunables are doing. But a reboot will be neccessary I think.



  • I was able to set most the the loader.conf stuff, but net.isr.maxthreads seems to always be set to "1" even if I try to set it higher.  I did some digging and it seems like it is set on boot and is based on the number of CPU cores you have available.  Which is weird, because I see 8 cores:

    [2.0-RELEASE][admin@server]/root(8): dmesg | grep maxthreads
    netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling

    [2.0-RELEASE][admin@server]/root(9): dmesg | grep CPU
    CPU: Intel(R) Xeon(R) CPU          X5677  @ 3.47GHz (3458.02-MHz K8-class CPU)
    FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    cpu4: <acpi cpu="">on acpi0
    cpu5: <acpi cpu="">on acpi0
    cpu6: <acpi cpu="">on acpi0
    cpu7: <acpi cpu="">on acpi0
    est: CPU supports Enhanced Speedstep, but is not recognized.
    p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
    est: CPU supports Enhanced Speedstep, but is not recognized.
    p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
    est: CPU supports Enhanced Speedstep, but is not recognized.
    p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
    est: CPU supports Enhanced Speedstep, but is not recognized.
    p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
    est: CPU supports Enhanced Speedstep, but is not recognized.
    p4tcc4: <cpu frequency="" thermal="" control="">on cpu4
    est: CPU supports Enhanced Speedstep, but is not recognized.
    p4tcc5: <cpu frequency="" thermal="" control="">on cpu5
    est: CPU supports Enhanced Speedstep, but is not recognized.
    p4tcc6: <cpu frequency="" thermal="" control="">on cpu6
    est: CPU supports Enhanced Speedstep, but is not recognized.
    p4tcc7: <cpu frequency="" thermal="" control="">on cpu7
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #6 Launched!
    SMP: AP CPU #7 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #4 Launched!
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #5 Launched!

    Seems like netisr_init is overriding my preferences at boot time…  Anyone know a way around this?  It seems like netisr_init thinks I have one CPU, if I am reading this right...

    I can't set net.isr.maxthreads manually either:

    [2.0-RELEASE][admin@server]/root(10): sysctl net.isr.maxthreads=3
    sysctl: oid 'net.isr.maxthreads' is read only</cpu></cpu></cpu></cpu></cpu></cpu></cpu></cpu></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi>



  • I suggest that you also post in FreeBSD-net mailing list (http://lists.freebsd.org/mailman/listinfo/freebsd-net)



  • Did you set these parameters using loader.conf or by system tuanbles (GUI) ?



  • I set the following tunables via loader.conf.local:

    inet.tcp.tcbhashsize=4096
    net.inet.tcp.syncache.hashsize=1024
    net.inet.tcp.syncache.bucketlimit=100
    net.isr.bindthreads=0
    net.isr.direct=1
    net.isr.direct_force=1
    net.isr.maxthreads=3
    vm.kmem_size_max=12G
    vm.kmem_size=10G

    They all worked on a reboot except net.isr.maxthreads=3.  I contacted the FreeBSD folks about that one, and they said that there is a limitation right now in the kernel where you cannot set net.isr.maxthreads above 1 if DEVICE_POLLING support is compiled into the kernel.  Which it seems to be with pfSense 2.0-RELEASE.  They are working on fixing that limitation, but as far as pfSense goes, it's not an option, unless pfSense 3.0 has it.  ;)

    It's a shame though, because it seems like it would help parallel interrupt handling and as such increase the max bandwidth of the box.  And I'm not about to rebuild a kernel for pfSense, somehow I think I'd end up trashing more stuff than I was "fixing".  So for now I'm just playing with my buffer sizes for various tunables via sysctl (ala "System Tunables"), now that I've learned how to increase the kernel memory space.



  • Try 'top -SH' in a shell to see detailed CPU usage. My experience with the Intel 'em' driver is that it threads well, so throughput should benefit from multiple cores, at least with that driver.

    It has been said on this forum that you should turn hyperthreading off, although last time I looked into that I couldn't really see a good reason why.


Locked