Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    10Gb/s connected pfSense firewalls

    Scheduled Pinned Locked Moved
    Firewalling
    4
    13
    14.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      bubble1975
      last edited by

      Hi Y'all,

      I'm exploring the 10Gb/s possibilities on several firewalls I have.  I'm seeing some performance issues that seem to point to the need for some kernel tuning on the box.  I see this website:

      http://fasterdata.es.net/fasterdata/host-tuning/freebsd/

      Which suggests, for 10Gb/s connected hosts:

      set to at least 16MB for 10GE hosts

      kern.ipc.maxsockbuf=16777216
        # set autotuning maximum to at least 16MB too
        net.inet.tcp.sendbuf_max=16777216 
        net.inet.tcp.recvbuf_max=16777216
        # enable send/recv autotuning
        net.inet.tcp.sendbuf_auto=1
        net.inet.tcp.recvbuf_auto=1
        # increase autotuning step size
        net.inet.tcp.sendbuf_inc=16384
        net.inet.tcp.recvbuf_inc=524288
        # turn off inflight limitting
        net.inet.tcp.inflight.enable=0
        # set this on test/measurement hosts
        net.inet.tcp.hostcache.expire=1

      I looked at a 2.0-RELEASE pfSense firewall (amd64) and saw:

      kern.ipc.maxsockbuf: 262144
      net.inet.tcp.sendbuf_max: 262144
      net.inet.tcp.recvbuf_max: 262144
      net.inet.tcp.sendbuf_auto: 1
      net.inet.tcp.recvbuf_auto: 1
      net.inet.tcp.sendbuf_inc: 8192
      net.inet.tcp.recvbuf_inc: 16384
      net.inet.tcp.inflight.enable: 1

      As you can see, some of those numbers are a lot smaller than the recommended (according to that website, at least).  And tcp.inflight.enable is enabled, when they say disable it.

      The million dollar question:  Can anyone think of any reason why I shouldn't modify sysctl.conf to reflect the "recommended" items listed above?  Seems like I could get a big performance boost from it, maybe…

      1 Reply Last reply Reply Quote 0
      • B
        bubble1975
        last edited by

        Or rather, modify the corresponding "System Tunables" via the web configurator?  Probably the better method…

        1 Reply Last reply Reply Quote 0
        • N
          Nachtfalke
          last edited by

          Just try with system tunables. If you get a boost it is ok if not than the "problem" is something else.
          System tunables are a good way to try that.

          1 Reply Last reply Reply Quote 0
          • B
            bubble1975
            last edited by

            I dropped in my tunables and it didn't seem to help my problem.  I'll leave the tunables in there anyway just because it seems like they should be there for 10Gb/s connectivity.

            I seem to be hitting a ceiling of about 2Gb/s transfer rates incoming.  I can't see any dropped packets and I can't see any buffer problems on the firewall or my transfer servers.  Looking at the live traffic graph, I see a sawtooth patten that is usually indicative of TCP scaling up and down.  My understanding is that this is because of dropped packets somewhere.

            If I have 4 cores in my firewall server, and I see 10% CPU utilization on my pfSense dashboard, does that mean 10% of total possible CPU power on the box?  In other words, 10% means like 40% of one core?  Or does 10% just mean 10% of one core, and not 10% of the total power of the 4 cores?

            1 Reply Last reply Reply Quote 0
            • N
              Nachtfalke
              last edited by

              As far as I know system tunables need a reboot. Not sure about that.

              If dashboard shows 10% that means that one CPU uses 40% or 2 CPUs use 2x20%.

              I changed that - but not sure if this improved performance or not:

              kern.ipc.somaxconn

The kern.ipc.somaxconn sysctl variable limits the size of the listen queue for accepting new TCP connections. The default value of 128 is typically too low for robust handling of new connections in a heavily loaded web server environment. For such environments, it is recommended to increase this value to 1024 or higher.

2048 

              kern.ipc.nmbclusters

The NMBCLUSTERS kernel configuration option dictates the amount of network Mbufs available to the system. A heavily-trafficked server with a low number of Mbufs will hinder FreeBSD's ability. Each cluster represents approximately 2 K of memory, so a value of 1024 represents 2 megabytes of kernel memory reserved for network buffers. A simple calculation can be done to figure out how many are needed. If you have a web server which maxes out at 1000 simultaneous connections, and each connection eats a 16 K receive and 16 K send buffer, you need approximately 32 MB worth of network buffers to cover the web server. A good rule of thumb is to multiply by 2, so 2x32 MB / 2 KB = 64 MB / 2 kB = 32768\. We recommend values between 4096 and 32768 for machines with greater amounts of memory.

65536

              net.inet.tcp.sendbuf_max 

16777216

              net.inet.tcp.recvbuf_max 

16777216 

              kern.maxfilesperproc

Set maximum files allowed open per process 

32768 

              kern.maxfiles

Set maximum files allowed open 

262144 

              net.inet.ip.intr_queue_maxlen 

Maximum size of the IP input queue 

3000
              1 Reply Last reply Reply Quote 0
              • B
                bubble1975
                last edited by

                Ah, OK…  So, if the packet filtering process is single threaded and uses one core, then I am CPU bound if my dashboard reads 25% (25% being 1 core of 4 being fully pegged)?  But, even if I were CPU bound, wouldn't I begin to see dropped packets on the interfaces?

                Thanks!

                1 Reply Last reply Reply Quote 0
                • B
                  bubble1975
                  last edited by

                  OK, here's some interesting stuff on tuning FreeBSD firewalls:

                  https://calomel.org/network_performance.html

                  They say to do this in loader.conf:

                  /boot/loader.conf

                  autoboot_delay="3"                    # reduce boot menu delay from 10 to 3 seconds
                  inet.tcp.tcbhashsize=4096            # tcb hash size
                  loader_logo="beastie"                # old FreeBSD logo menu
                  net.inet.tcp.syncache.hashsize=1024  # syncache hash size
                  net.inet.tcp.syncache.bucketlimit=100 # syncache bucket limit
                  net.isr.bindthreads=0                # do not bind threads to CPUs
                  net.isr.direct=1                      # interrupt handling via multiple CPU
                  net.isr.direct_force=1                # "
                  net.isr.maxthreads=3                  # Max number of threads for NIC IRQ balancing (4 cores in box)
                  vm.kmem_size=1G                      # physical memory available for kernel (320Mb by default)

                  It looks like those tunables can't be set by sysctl (and hence not by the "System Tunables" in pfSense).  I'm hesitant to change loader.conf under pfSense (and then reboot, I assume) - but the last four items look like they might help 10G performance a lot, possibly.  Anyone heard of this stuff?

                  1 Reply Last reply Reply Quote 0
                  • N
                    Nachtfalke
                    last edited by

                    I checked which are the default settings on my system:

                    sysctl net.isr.direct
net.isr.direct: 1


                    sysctl vm.kmem_size
vm.kmem_size: 435544320


                    sysctl net.isr.maxthreads
net.isr.maxthreads: 1


                    sysctl net.isr.direct_force
net.isr.direct_force: 1


                    sysctl net.isr.bindthreads
net.isr.bindthreads: 0

                    You have to be careful on which version of BSD this tuning guides are based. pfsense 2.0 is using freebsd 8.1.

                    In the past I found much tuning tip which based on old versions.

                    Instead of putting the tuning parameters in loader.conf put the im loader.conf.local (perhaps you need to create it). This file will not be overwritten after a firmware update. This was nice to know when I used the beta and RC versions of pfsense with daily updates.

                    But I think you can create new system tunables in GUI because the tuning parameters above can all bet set by sysctl. I think this is the same as system tunables are doing. But a reboot will be neccessary I think.

                    1 Reply Last reply Reply Quote 0
                    • B
                      bubble1975
                      last edited by

                      I was able to set most the the loader.conf stuff, but net.isr.maxthreads seems to always be set to "1" even if I try to set it higher.  I did some digging and it seems like it is set on boot and is based on the number of CPU cores you have available.  Which is weird, because I see 8 cores:

                      [2.0-RELEASE][admin@server]/root(8): dmesg | grep maxthreads
                      netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling

                      [2.0-RELEASE][admin@server]/root(9): dmesg | grep CPU
                      CPU: Intel(R) Xeon(R) CPU          X5677  @ 3.47GHz (3458.02-MHz K8-class CPU)
                      FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
                      cpu0: <acpi cpu="">on acpi0
                      cpu1: <acpi cpu="">on acpi0
                      cpu2: <acpi cpu="">on acpi0
                      cpu3: <acpi cpu="">on acpi0
                      cpu4: <acpi cpu="">on acpi0
                      cpu5: <acpi cpu="">on acpi0
                      cpu6: <acpi cpu="">on acpi0
                      cpu7: <acpi cpu="">on acpi0
                      est: CPU supports Enhanced Speedstep, but is not recognized.
                      p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
                      est: CPU supports Enhanced Speedstep, but is not recognized.
                      p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
                      est: CPU supports Enhanced Speedstep, but is not recognized.
                      p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
                      est: CPU supports Enhanced Speedstep, but is not recognized.
                      p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
                      est: CPU supports Enhanced Speedstep, but is not recognized.
                      p4tcc4: <cpu frequency="" thermal="" control="">on cpu4
                      est: CPU supports Enhanced Speedstep, but is not recognized.
                      p4tcc5: <cpu frequency="" thermal="" control="">on cpu5
                      est: CPU supports Enhanced Speedstep, but is not recognized.
                      p4tcc6: <cpu frequency="" thermal="" control="">on cpu6
                      est: CPU supports Enhanced Speedstep, but is not recognized.
                      p4tcc7: <cpu frequency="" thermal="" control="">on cpu7
                      SMP: AP CPU #1 Launched!
                      SMP: AP CPU #6 Launched!
                      SMP: AP CPU #7 Launched!
                      SMP: AP CPU #2 Launched!
                      SMP: AP CPU #4 Launched!
                      SMP: AP CPU #3 Launched!
                      SMP: AP CPU #5 Launched!

                      Seems like netisr_init is overriding my preferences at boot time…  Anyone know a way around this?  It seems like netisr_init thinks I have one CPU, if I am reading this right...

                      I can't set net.isr.maxthreads manually either:

                      [2.0-RELEASE][admin@server]/root(10): sysctl net.isr.maxthreads=3
                      sysctl: oid 'net.isr.maxthreads' is read only</cpu></cpu></cpu></cpu></cpu></cpu></cpu></cpu></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi>

                      1 Reply Last reply Reply Quote 0
                      • D
                        dhatz
                        last edited by

                        I suggest that you also post in FreeBSD-net mailing list (http://lists.freebsd.org/mailman/listinfo/freebsd-net)

                        1 Reply Last reply Reply Quote 0
                        • N
                          Nachtfalke
                          last edited by

                          Did you set these parameters using loader.conf or by system tuanbles (GUI) ?

                          1 Reply Last reply Reply Quote 0
                          • B
                            bubble1975
                            last edited by

                            I set the following tunables via loader.conf.local:

                            inet.tcp.tcbhashsize=4096
                            net.inet.tcp.syncache.hashsize=1024
                            net.inet.tcp.syncache.bucketlimit=100
                            net.isr.bindthreads=0
                            net.isr.direct=1
                            net.isr.direct_force=1
                            net.isr.maxthreads=3
                            vm.kmem_size_max=12G
                            vm.kmem_size=10G

                            They all worked on a reboot except net.isr.maxthreads=3.  I contacted the FreeBSD folks about that one, and they said that there is a limitation right now in the kernel where you cannot set net.isr.maxthreads above 1 if DEVICE_POLLING support is compiled into the kernel.  Which it seems to be with pfSense 2.0-RELEASE.  They are working on fixing that limitation, but as far as pfSense goes, it's not an option, unless pfSense 3.0 has it.  ;)

                            It's a shame though, because it seems like it would help parallel interrupt handling and as such increase the max bandwidth of the box.  And I'm not about to rebuild a kernel for pfSense, somehow I think I'd end up trashing more stuff than I was "fixing".  So for now I'm just playing with my buffer sizes for various tunables via sysctl (ala "System Tunables"), now that I've learned how to increase the kernel memory space.

                            1 Reply Last reply Reply Quote 0
                            • C
                              clarknova
                              last edited by

                              Try 'top -SH' in a shell to see detailed CPU usage. My experience with the Intel 'em' driver is that it threads well, so throughput should benefit from multiple cores, at least with that driver.

                              It has been said on this forum that you should turn hyperthreading off, although last time I looked into that I couldn't really see a good reason why.

                              db

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post