Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    10Gb/s connected pfSense firewalls

    Scheduled Pinned Locked Moved Firewalling
    13 Posts 4 Posters 15.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      bubble1975
      last edited by

      I dropped in my tunables and it didn't seem to help my problem.  I'll leave the tunables in there anyway just because it seems like they should be there for 10Gb/s connectivity.

      I seem to be hitting a ceiling of about 2Gb/s transfer rates incoming.  I can't see any dropped packets and I can't see any buffer problems on the firewall or my transfer servers.  Looking at the live traffic graph, I see a sawtooth patten that is usually indicative of TCP scaling up and down.  My understanding is that this is because of dropped packets somewhere.

      If I have 4 cores in my firewall server, and I see 10% CPU utilization on my pfSense dashboard, does that mean 10% of total possible CPU power on the box?  In other words, 10% means like 40% of one core?  Or does 10% just mean 10% of one core, and not 10% of the total power of the 4 cores?

      1 Reply Last reply Reply Quote 0
      • N
        Nachtfalke
        last edited by

        As far as I know system tunables need a reboot. Not sure about that.

        If dashboard shows 10% that means that one CPU uses 40% or 2 CPUs use 2x20%.

        I changed that - but not sure if this improved performance or not:

        kern.ipc.somaxconn

The kern.ipc.somaxconn sysctl variable limits the size of the listen queue for accepting new TCP connections. The default value of 128 is typically too low for robust handling of new connections in a heavily loaded web server environment. For such environments, it is recommended to increase this value to 1024 or higher.

2048 

        kern.ipc.nmbclusters

The NMBCLUSTERS kernel configuration option dictates the amount of network Mbufs available to the system. A heavily-trafficked server with a low number of Mbufs will hinder FreeBSD's ability. Each cluster represents approximately 2 K of memory, so a value of 1024 represents 2 megabytes of kernel memory reserved for network buffers. A simple calculation can be done to figure out how many are needed. If you have a web server which maxes out at 1000 simultaneous connections, and each connection eats a 16 K receive and 16 K send buffer, you need approximately 32 MB worth of network buffers to cover the web server. A good rule of thumb is to multiply by 2, so 2x32 MB / 2 KB = 64 MB / 2 kB = 32768\. We recommend values between 4096 and 32768 for machines with greater amounts of memory.

65536

        net.inet.tcp.sendbuf_max 

16777216

        net.inet.tcp.recvbuf_max 

16777216 

        kern.maxfilesperproc

Set maximum files allowed open per process 

32768 

        kern.maxfiles

Set maximum files allowed open 

262144 

        net.inet.ip.intr_queue_maxlen 

Maximum size of the IP input queue 

3000
        1 Reply Last reply Reply Quote 0
        • B
          bubble1975
          last edited by

          Ah, OK…  So, if the packet filtering process is single threaded and uses one core, then I am CPU bound if my dashboard reads 25% (25% being 1 core of 4 being fully pegged)?  But, even if I were CPU bound, wouldn't I begin to see dropped packets on the interfaces?

          Thanks!

          1 Reply Last reply Reply Quote 0
          • B
            bubble1975
            last edited by

            OK, here's some interesting stuff on tuning FreeBSD firewalls:

            https://calomel.org/network_performance.html

            They say to do this in loader.conf:

            /boot/loader.conf

            autoboot_delay="3"                    # reduce boot menu delay from 10 to 3 seconds
            inet.tcp.tcbhashsize=4096            # tcb hash size
            loader_logo="beastie"                # old FreeBSD logo menu
            net.inet.tcp.syncache.hashsize=1024  # syncache hash size
            net.inet.tcp.syncache.bucketlimit=100 # syncache bucket limit
            net.isr.bindthreads=0                # do not bind threads to CPUs
            net.isr.direct=1                      # interrupt handling via multiple CPU
            net.isr.direct_force=1                # "
            net.isr.maxthreads=3                  # Max number of threads for NIC IRQ balancing (4 cores in box)
            vm.kmem_size=1G                      # physical memory available for kernel (320Mb by default)

            It looks like those tunables can't be set by sysctl (and hence not by the "System Tunables" in pfSense).  I'm hesitant to change loader.conf under pfSense (and then reboot, I assume) - but the last four items look like they might help 10G performance a lot, possibly.  Anyone heard of this stuff?

            1 Reply Last reply Reply Quote 0
            • N
              Nachtfalke
              last edited by

              I checked which are the default settings on my system:

              sysctl net.isr.direct
net.isr.direct: 1


              sysctl vm.kmem_size
vm.kmem_size: 435544320


              sysctl net.isr.maxthreads
net.isr.maxthreads: 1


              sysctl net.isr.direct_force
net.isr.direct_force: 1


              sysctl net.isr.bindthreads
net.isr.bindthreads: 0

              You have to be careful on which version of BSD this tuning guides are based. pfsense 2.0 is using freebsd 8.1.

              In the past I found much tuning tip which based on old versions.

              Instead of putting the tuning parameters in loader.conf put the im loader.conf.local (perhaps you need to create it). This file will not be overwritten after a firmware update. This was nice to know when I used the beta and RC versions of pfsense with daily updates.

              But I think you can create new system tunables in GUI because the tuning parameters above can all bet set by sysctl. I think this is the same as system tunables are doing. But a reboot will be neccessary I think.

              1 Reply Last reply Reply Quote 0
              • B
                bubble1975
                last edited by

                I was able to set most the the loader.conf stuff, but net.isr.maxthreads seems to always be set to "1" even if I try to set it higher.  I did some digging and it seems like it is set on boot and is based on the number of CPU cores you have available.  Which is weird, because I see 8 cores:

                [2.0-RELEASE][admin@server]/root(8): dmesg | grep maxthreads
                netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling

                [2.0-RELEASE][admin@server]/root(9): dmesg | grep CPU
                CPU: Intel(R) Xeon(R) CPU          X5677  @ 3.47GHz (3458.02-MHz K8-class CPU)
                FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
                cpu0: <acpi cpu="">on acpi0
                cpu1: <acpi cpu="">on acpi0
                cpu2: <acpi cpu="">on acpi0
                cpu3: <acpi cpu="">on acpi0
                cpu4: <acpi cpu="">on acpi0
                cpu5: <acpi cpu="">on acpi0
                cpu6: <acpi cpu="">on acpi0
                cpu7: <acpi cpu="">on acpi0
                est: CPU supports Enhanced Speedstep, but is not recognized.
                p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
                est: CPU supports Enhanced Speedstep, but is not recognized.
                p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
                est: CPU supports Enhanced Speedstep, but is not recognized.
                p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
                est: CPU supports Enhanced Speedstep, but is not recognized.
                p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
                est: CPU supports Enhanced Speedstep, but is not recognized.
                p4tcc4: <cpu frequency="" thermal="" control="">on cpu4
                est: CPU supports Enhanced Speedstep, but is not recognized.
                p4tcc5: <cpu frequency="" thermal="" control="">on cpu5
                est: CPU supports Enhanced Speedstep, but is not recognized.
                p4tcc6: <cpu frequency="" thermal="" control="">on cpu6
                est: CPU supports Enhanced Speedstep, but is not recognized.
                p4tcc7: <cpu frequency="" thermal="" control="">on cpu7
                SMP: AP CPU #1 Launched!
                SMP: AP CPU #6 Launched!
                SMP: AP CPU #7 Launched!
                SMP: AP CPU #2 Launched!
                SMP: AP CPU #4 Launched!
                SMP: AP CPU #3 Launched!
                SMP: AP CPU #5 Launched!

                Seems like netisr_init is overriding my preferences at boot time…  Anyone know a way around this?  It seems like netisr_init thinks I have one CPU, if I am reading this right...

                I can't set net.isr.maxthreads manually either:

                [2.0-RELEASE][admin@server]/root(10): sysctl net.isr.maxthreads=3
                sysctl: oid 'net.isr.maxthreads' is read only</cpu></cpu></cpu></cpu></cpu></cpu></cpu></cpu></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi>

                1 Reply Last reply Reply Quote 0
                • D
                  dhatz
                  last edited by

                  I suggest that you also post in FreeBSD-net mailing list (http://lists.freebsd.org/mailman/listinfo/freebsd-net)

                  1 Reply Last reply Reply Quote 0
                  • N
                    Nachtfalke
                    last edited by

                    Did you set these parameters using loader.conf or by system tuanbles (GUI) ?

                    1 Reply Last reply Reply Quote 0
                    • B
                      bubble1975
                      last edited by

                      I set the following tunables via loader.conf.local:

                      inet.tcp.tcbhashsize=4096
                      net.inet.tcp.syncache.hashsize=1024
                      net.inet.tcp.syncache.bucketlimit=100
                      net.isr.bindthreads=0
                      net.isr.direct=1
                      net.isr.direct_force=1
                      net.isr.maxthreads=3
                      vm.kmem_size_max=12G
                      vm.kmem_size=10G

                      They all worked on a reboot except net.isr.maxthreads=3.  I contacted the FreeBSD folks about that one, and they said that there is a limitation right now in the kernel where you cannot set net.isr.maxthreads above 1 if DEVICE_POLLING support is compiled into the kernel.  Which it seems to be with pfSense 2.0-RELEASE.  They are working on fixing that limitation, but as far as pfSense goes, it's not an option, unless pfSense 3.0 has it.  ;)

                      It's a shame though, because it seems like it would help parallel interrupt handling and as such increase the max bandwidth of the box.  And I'm not about to rebuild a kernel for pfSense, somehow I think I'd end up trashing more stuff than I was "fixing".  So for now I'm just playing with my buffer sizes for various tunables via sysctl (ala "System Tunables"), now that I've learned how to increase the kernel memory space.

                      1 Reply Last reply Reply Quote 0
                      • C
                        clarknova
                        last edited by

                        Try 'top -SH' in a shell to see detailed CPU usage. My experience with the Intel 'em' driver is that it threads well, so throughput should benefit from multiple cores, at least with that driver.

                        It has been said on this forum that you should turn hyperthreading off, although last time I looked into that I couldn't really see a good reason why.

                        db

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.