Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    10Gb/s connected pfSense firewalls

    Scheduled Pinned Locked Moved
    Firewalling
    4
    13
    14.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      bubble1975
      last edited by

      Or rather, modify the corresponding "System Tunables" via the web configurator?  Probably the better method…

      1 Reply Last reply Reply Quote 0
      • N
        Nachtfalke
        last edited by

        Just try with system tunables. If you get a boost it is ok if not than the "problem" is something else.
        System tunables are a good way to try that.

        1 Reply Last reply Reply Quote 0
        • B
          bubble1975
          last edited by

          I dropped in my tunables and it didn't seem to help my problem.  I'll leave the tunables in there anyway just because it seems like they should be there for 10Gb/s connectivity.

          I seem to be hitting a ceiling of about 2Gb/s transfer rates incoming.  I can't see any dropped packets and I can't see any buffer problems on the firewall or my transfer servers.  Looking at the live traffic graph, I see a sawtooth patten that is usually indicative of TCP scaling up and down.  My understanding is that this is because of dropped packets somewhere.

          If I have 4 cores in my firewall server, and I see 10% CPU utilization on my pfSense dashboard, does that mean 10% of total possible CPU power on the box?  In other words, 10% means like 40% of one core?  Or does 10% just mean 10% of one core, and not 10% of the total power of the 4 cores?

          1 Reply Last reply Reply Quote 0
          • N
            Nachtfalke
            last edited by

            As far as I know system tunables need a reboot. Not sure about that.

            If dashboard shows 10% that means that one CPU uses 40% or 2 CPUs use 2x20%.

            I changed that - but not sure if this improved performance or not:

            kern.ipc.somaxconn

The kern.ipc.somaxconn sysctl variable limits the size of the listen queue for accepting new TCP connections. The default value of 128 is typically too low for robust handling of new connections in a heavily loaded web server environment. For such environments, it is recommended to increase this value to 1024 or higher.

2048 

            kern.ipc.nmbclusters

The NMBCLUSTERS kernel configuration option dictates the amount of network Mbufs available to the system. A heavily-trafficked server with a low number of Mbufs will hinder FreeBSD's ability. Each cluster represents approximately 2 K of memory, so a value of 1024 represents 2 megabytes of kernel memory reserved for network buffers. A simple calculation can be done to figure out how many are needed. If you have a web server which maxes out at 1000 simultaneous connections, and each connection eats a 16 K receive and 16 K send buffer, you need approximately 32 MB worth of network buffers to cover the web server. A good rule of thumb is to multiply by 2, so 2x32 MB / 2 KB = 64 MB / 2 kB = 32768\. We recommend values between 4096 and 32768 for machines with greater amounts of memory.

65536

            net.inet.tcp.sendbuf_max 

16777216

            net.inet.tcp.recvbuf_max 

16777216 

            kern.maxfilesperproc

Set maximum files allowed open per process 

32768 

            kern.maxfiles

Set maximum files allowed open 

262144 

            net.inet.ip.intr_queue_maxlen 

Maximum size of the IP input queue 

3000
            1 Reply Last reply Reply Quote 0
            • B
              bubble1975
              last edited by

              Ah, OK…  So, if the packet filtering process is single threaded and uses one core, then I am CPU bound if my dashboard reads 25% (25% being 1 core of 4 being fully pegged)?  But, even if I were CPU bound, wouldn't I begin to see dropped packets on the interfaces?

              Thanks!

              1 Reply Last reply Reply Quote 0
              • B
                bubble1975
                last edited by

                OK, here's some interesting stuff on tuning FreeBSD firewalls:

                https://calomel.org/network_performance.html

                They say to do this in loader.conf:

                /boot/loader.conf

                autoboot_delay="3"                    # reduce boot menu delay from 10 to 3 seconds
                inet.tcp.tcbhashsize=4096            # tcb hash size
                loader_logo="beastie"                # old FreeBSD logo menu
                net.inet.tcp.syncache.hashsize=1024  # syncache hash size
                net.inet.tcp.syncache.bucketlimit=100 # syncache bucket limit
                net.isr.bindthreads=0                # do not bind threads to CPUs
                net.isr.direct=1                      # interrupt handling via multiple CPU
                net.isr.direct_force=1                # "
                net.isr.maxthreads=3                  # Max number of threads for NIC IRQ balancing (4 cores in box)
                vm.kmem_size=1G                      # physical memory available for kernel (320Mb by default)

                It looks like those tunables can't be set by sysctl (and hence not by the "System Tunables" in pfSense).  I'm hesitant to change loader.conf under pfSense (and then reboot, I assume) - but the last four items look like they might help 10G performance a lot, possibly.  Anyone heard of this stuff?

                1 Reply Last reply Reply Quote 0
                • N
                  Nachtfalke
                  last edited by

                  I checked which are the default settings on my system:

                  sysctl net.isr.direct
net.isr.direct: 1


                  sysctl vm.kmem_size
vm.kmem_size: 435544320


                  sysctl net.isr.maxthreads
net.isr.maxthreads: 1


                  sysctl net.isr.direct_force
net.isr.direct_force: 1


                  sysctl net.isr.bindthreads
net.isr.bindthreads: 0

                  You have to be careful on which version of BSD this tuning guides are based. pfsense 2.0 is using freebsd 8.1.

                  In the past I found much tuning tip which based on old versions.

                  Instead of putting the tuning parameters in loader.conf put the im loader.conf.local (perhaps you need to create it). This file will not be overwritten after a firmware update. This was nice to know when I used the beta and RC versions of pfsense with daily updates.

                  But I think you can create new system tunables in GUI because the tuning parameters above can all bet set by sysctl. I think this is the same as system tunables are doing. But a reboot will be neccessary I think.

                  1 Reply Last reply Reply Quote 0
                  • B
                    bubble1975
                    last edited by

                    I was able to set most the the loader.conf stuff, but net.isr.maxthreads seems to always be set to "1" even if I try to set it higher.  I did some digging and it seems like it is set on boot and is based on the number of CPU cores you have available.  Which is weird, because I see 8 cores:

                    [2.0-RELEASE][admin@server]/root(8): dmesg | grep maxthreads
                    netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling

                    [2.0-RELEASE][admin@server]/root(9): dmesg | grep CPU
                    CPU: Intel(R) Xeon(R) CPU          X5677  @ 3.47GHz (3458.02-MHz K8-class CPU)
                    FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
                    cpu0: <acpi cpu="">on acpi0
                    cpu1: <acpi cpu="">on acpi0
                    cpu2: <acpi cpu="">on acpi0
                    cpu3: <acpi cpu="">on acpi0
                    cpu4: <acpi cpu="">on acpi0
                    cpu5: <acpi cpu="">on acpi0
                    cpu6: <acpi cpu="">on acpi0
                    cpu7: <acpi cpu="">on acpi0
                    est: CPU supports Enhanced Speedstep, but is not recognized.
                    p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
                    est: CPU supports Enhanced Speedstep, but is not recognized.
                    p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
                    est: CPU supports Enhanced Speedstep, but is not recognized.
                    p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
                    est: CPU supports Enhanced Speedstep, but is not recognized.
                    p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
                    est: CPU supports Enhanced Speedstep, but is not recognized.
                    p4tcc4: <cpu frequency="" thermal="" control="">on cpu4
                    est: CPU supports Enhanced Speedstep, but is not recognized.
                    p4tcc5: <cpu frequency="" thermal="" control="">on cpu5
                    est: CPU supports Enhanced Speedstep, but is not recognized.
                    p4tcc6: <cpu frequency="" thermal="" control="">on cpu6
                    est: CPU supports Enhanced Speedstep, but is not recognized.
                    p4tcc7: <cpu frequency="" thermal="" control="">on cpu7
                    SMP: AP CPU #1 Launched!
                    SMP: AP CPU #6 Launched!
                    SMP: AP CPU #7 Launched!
                    SMP: AP CPU #2 Launched!
                    SMP: AP CPU #4 Launched!
                    SMP: AP CPU #3 Launched!
                    SMP: AP CPU #5 Launched!

                    Seems like netisr_init is overriding my preferences at boot time…  Anyone know a way around this?  It seems like netisr_init thinks I have one CPU, if I am reading this right...

                    I can't set net.isr.maxthreads manually either:

                    [2.0-RELEASE][admin@server]/root(10): sysctl net.isr.maxthreads=3
                    sysctl: oid 'net.isr.maxthreads' is read only</cpu></cpu></cpu></cpu></cpu></cpu></cpu></cpu></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi>

                    1 Reply Last reply Reply Quote 0
                    • D
                      dhatz
                      last edited by

                      I suggest that you also post in FreeBSD-net mailing list (http://lists.freebsd.org/mailman/listinfo/freebsd-net)

                      1 Reply Last reply Reply Quote 0
                      • N
                        Nachtfalke
                        last edited by

                        Did you set these parameters using loader.conf or by system tuanbles (GUI) ?

                        1 Reply Last reply Reply Quote 0
                        • B
                          bubble1975
                          last edited by

                          I set the following tunables via loader.conf.local:

                          inet.tcp.tcbhashsize=4096
                          net.inet.tcp.syncache.hashsize=1024
                          net.inet.tcp.syncache.bucketlimit=100
                          net.isr.bindthreads=0
                          net.isr.direct=1
                          net.isr.direct_force=1
                          net.isr.maxthreads=3
                          vm.kmem_size_max=12G
                          vm.kmem_size=10G

                          They all worked on a reboot except net.isr.maxthreads=3.  I contacted the FreeBSD folks about that one, and they said that there is a limitation right now in the kernel where you cannot set net.isr.maxthreads above 1 if DEVICE_POLLING support is compiled into the kernel.  Which it seems to be with pfSense 2.0-RELEASE.  They are working on fixing that limitation, but as far as pfSense goes, it's not an option, unless pfSense 3.0 has it.  ;)

                          It's a shame though, because it seems like it would help parallel interrupt handling and as such increase the max bandwidth of the box.  And I'm not about to rebuild a kernel for pfSense, somehow I think I'd end up trashing more stuff than I was "fixing".  So for now I'm just playing with my buffer sizes for various tunables via sysctl (ala "System Tunables"), now that I've learned how to increase the kernel memory space.

                          1 Reply Last reply Reply Quote 0
                          • C
                            clarknova
                            last edited by

                            Try 'top -SH' in a shell to see detailed CPU usage. My experience with the Intel 'em' driver is that it threads well, so throughput should benefit from multiple cores, at least with that driver.

                            It has been said on this forum that you should turn hyperthreading off, although last time I looked into that I couldn't really see a good reason why.

                            db

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post