Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Understanding BufferBloat and LAGG

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 7 Posters 1.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • mircolinoM
      mircolino @stephenw10
      last edited by mircolino

      @stephenw10, I'll re-run the tests tonight when nobody is using the Internet (right now my wife is on zoom with 16 other coworkers).

      But overall the numbers, whether it's the switch doing the WAN LAGG or the pfSense appliance, are similar. Always in the 1.2Gb/s range ยฑ 20Mb/s. It's only the bufferbloat that's higher when pfSense is handling the aggregation.

      Is it because pfsense LAGG is too fast and the rest of the firewall can't keep up?

      I'm running all the tests from Chrome on a Windows Server 2019.

      Windows Server (Intel X520-DA1) โ†” SFP+ Twinax DAC โ†” Switch port 20 (setup as a vlan trunk)

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        I sure hope you installed pfSense yourself on that.....

        mircolinoM Cool_CoronaC 2 Replies Last reply Reply Quote 1
        • mircolinoM
          mircolino @stephenw10
          last edited by mircolino

          @stephenw10 said in Understanding BufferBloat and LAGG:

          I sure hope you installed pfSense yourself on that.....

          I did. Why?

          UPDATE: it actually came with Ubuntu preinstalled.

          1 Reply Last reply Reply Quote 0
          • Cool_CoronaC
            Cool_Corona @stephenw10
            last edited by

            @stephenw10

            Why is that?? Isnt PFsense OS? and free to use?

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Several reasons. But for me the biggest is; if you buy a firewall direct from China you have no idea what's actually installed on it. Even if it came with pfSense installed (which it shouldn't because that's commercial redistribution) you should format it and reinstall.

              Steve

              1 Reply Last reply Reply Quote 0
              • W
                winger46146
                last edited by

                Check out this video https://www.youtube.com/watch?v=iXqExAALzR8
                I went from an F to an A+ on bufferbloat.

                mircolinoM 1 Reply Last reply Reply Quote 0
                • mircolinoM
                  mircolino @winger46146
                  last edited by mircolino

                  @winger46146, thank you for the link. Yes, I was going to setup limiters next.

                  I obviously rather have pfSense handle the WAN directly, instead of going through the switch first.
                  Just don't understand why, by having pfSense do the WAN LAGG, the overall performance is slightly degrading. I'd expect to be the opposite.

                  1 Reply Last reply Reply Quote 0
                  • chpalmerC
                    chpalmer
                    last edited by

                    I did not see any difference with bufferbloat on that test going from non-LAGG to LAGG on my MB8600 to pfsense (on my XTM5 box). Did you see a difference?

                    Im on an M400 box now so could try that test with it but its kind of one of those buzzwords that DSLR seems to have brought into the picture and made everyone worry..

                    Do you get your full speed from your ISP? When you max out your connection while on Zoom,VOIP ect.. does your jitter increase to the point where the call suffers?

                    I can not say I see any issue from my "D" grade on my bufferbloat as reported by DSLR.. Im not sure the effort is worth the payback.. But that said.. I am curious. :)

                    Triggering snowflakes one by one..
                    Intel(R) Core(TM) i5-4590T CPU @ 2.00GHz on an M400 WG box.

                    1 Reply Last reply Reply Quote 0
                    • mircolinoM
                      mircolino
                      last edited by mircolino

                      I really wanted to "remove" WAN traffic from the switch and let pfSense handle it directly.
                      But after reading here that LAGG interfaces don't support limiters, while VLANs do, I basically had no choice but let the switch handle the Motorola MB8600 LAGG.
                      Not my preferred choice, but after adding limiters to the WAN interface, following the link posted by @winger46146, these is the outcome:

                      alt text

                      I'll take a 50Mb/s speed penalty for all straight As ๐Ÿ™‚

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by stephenw10

                        LAGG interfaces can use Limiters no problem. They can't use ALTQ based traffic shaping.

                        It would be interesting to test without the switch in play at all if you can. So modem - pfSense - test client directly.

                        And, yeah, fixing buffer bloat can make a big difference to some things if you have it bad, like 'F'!

                        Steve

                        mircolinoM 1 Reply Last reply Reply Quote 1
                        • mircolinoM
                          mircolino @stephenw10
                          last edited by

                          @stephenw10 said in Understanding BufferBloat and LAGG:

                          LAGG interfaces can use Limiters no problem. They can't use ALTQ based traffic shaping.

                          Didn't know that ๐Ÿ˜ž.

                          It would be interesting to test without the switch in play at all if you can. So modem - pfSense - test client directly.

                          That's pretty easy to try. Tonight when again nobody's using the Internet I'll run another set of tests and post the results.

                          In the meantime, this is the slightly redacted "netstat -i" output with the switch doing the WAN LAGG:

                          ix0: LAN
                          ix0.2: WAN
                          ix0.3: DMZ
                          ix0.4: IOT
                          ix0.5: GUEST

                          Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
                          ix0    1500 <Link#1>      00:f0:xx:xx:xx:44 34496952     0     0 34515981     0     0
                          ix0       - 172.xx.8.0/24 edge                  5747     -     -     4307     -     -
                          ix0       - fe80::%ix0/64 fe80::1:1%ix0          634     -     -     5006     -     -
                          ix0       - 2601:646:8302 edge                 13558     -     -    15001     -     -
                          ix1*   1500 <Link#2>      00:f0:xx:xx:xx:45        0     0     0        0     0     0
                          igb0*  1500 <Link#3>      00:f0:xx:xx:xx:b5        0     0     0        0     0     0
                          igb1*  1500 <Link#4>      00:f0:xx:xx:xx:b6        0     0     0        0     0     0
                          ix2*   1500 <Link#5>      00:f0:xx:xx:xx:46        0     0     0        0     0     0
                          ix3*   1500 <Link#6>      00:f0:xx:xx:xx:47        0     0     0        0     0     0
                          lo0   16384 <Link#7>      lo0                     80     0     0       80     0     0
                          lo0       - localhost     localhost                0     -     -        0     -     -
                          lo0       - fe80::%lo0/64 fe80::1%lo0              0     -     -        0     -     -
                          lo0       - your-net      localhost               80     -     -       80     -     -
                          enc0*  1536 <Link#8>      enc0                     0     0     0        0     0     0
                          pfsyn  1500 <Link#9>      pfsync0                  0     0     0        0     0     0
                          pflog 33160 <Link#10>     pflog0                   0     0     0     5441     0     0
                          ix0.3  1500 <Link#11>     00:f0:xx:xx:xx:44    69106     0     0    42185     0     0
                          ix0.3     - 172.xx.9.0/24 edge-dmz                 8     -     -        8     -     -
                          ix0.3     - fe80::%ix0.3/ fe80::1:1%ix0.3        270     -     -     4147     -     -
                          ix0.3     - 2601:646:8302 edge-dmz               292     -     -      151     -     -
                          ix0.4  1500 <Link#12>     00:f0:xx:xx:xx:44  4123549     0     0  2684756     0     0
                          ix0.4     - 172.xx.10.0/2 edge-iot              2110     -     -     1798     -     -
                          ix0.4     - fe80::%ix0.4/ fe80::1:1%ix0.4        927     -     -     5309     -     -
                          ix0.4     - 2601:646:8302 edge-iot              1217     -     -      622     -     -
                          ix0.5  1500 <Link#13>     00:f0:xx:xx:xx:44     1861     0     0     3738     0     0
                          ix0.5     - 172.xx.11.0/2 edge-guest               0     -     -        0     -     -
                          ix0.5     - fe80::%ix0.5/ fe80::1:1%ix0.5          0     -     -     3732     -     -
                          ix0.5     - 2601:646:8302 edge-guest               0     -     -        0     -     -
                          ix0.2  1500 <Link#14>     00:f0:xx:xx:xx:44 25567476     0     0  9123326     0     0
                          ix0.2     - fe80::%ix0.2/ fe80::xxx:xxxx:fe    46506     -     -    46538     -     -
                          ix0.2     - 73.xxx.xx.0/2 c-73-xxx-xx-189.h   106079     -     -    46471     -     -
                          ix0.2     - 2001:558:6045 2001:558:6045:xx:    58138     -     -       12     -     -
                          
                          1 Reply Last reply Reply Quote 0
                          • mircolinoM
                            mircolino
                            last edited by

                            OK. Reconfigured the WAN with pfSense doing the LAGG and connected the Windows Server directly to the appliance (nothing else connected).

                            Speed test without limiters:

                            alt text

                            Speed test with limiters (CoDel 1200Mbs down, 50Mbs up, queue lenght left empty, both IPv4 and IPv6 floating rules):

                            alt text

                            Pretty impressive I have to say ๐Ÿ™‚
                            I can probably gain a bit more by playing with up/down speeds and queue length, but for now I'll leave it alone.

                            The following is "netstat -i" output:

                            Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
                            ix0    1500 <Link#1>      00:f0:xx:xx:xx:44  4033957     0     0  6836137     0     0
                            ix0       - 172.xx.8.0/24 edge                   371     -     -      563     -     -
                            ix0       - fe80::%ix0/64 fe80::1:1%ix0           31     -     -      134     -     -
                            ix0       - 2601:646:8302 edge                   254     -     -      300     -     -
                            ix1*   1500 <Link#2>      00:f0:xx:xx:xx:45        0     0     0        0     0     0
                            igb0   1500 <Link#3>      00:f0:xx:xx:xx:b5  2440366     0     0  1306587     0     0
                            igb1   1500 <Link#4>      00:f0:xx:xx:xx:b5  4545738     0     0  2941815     0     0
                            ix2*   1500 <Link#5>      00:f0:xx:xx:xx:46        0     0     0        0     0     0
                            ix3*   1500 <Link#6>      00:f0:xx:xx:xx:47        0     0     0        0     0     0
                            lo0   16384 <Link#7>      lo0                     77     0     0       77     0     0
                            lo0       - localhost     localhost                0     -     -        0     -     -
                            lo0       - fe80::%lo0/64 fe80::1%lo0              0     -     -        0     -     -
                            lo0       - your-net      localhost               77     -     -       77     -     -
                            enc0*  1536 <Link#8>      enc0                     0     0     0        0     0     0
                            pfsyn  1500 <Link#9>      pfsync0                  0     0     0        0     0     0
                            pflog 33160 <Link#10>     pflog0                   0     0     0     5607     0     0
                            lagg0  1500 <Link#11>     00:f0:xx:xx:xx:b5  6986138     0     0  4248402     5     0
                            lagg0     - fe80::%lagg0/ fe80::xxx:xxxx:fe    32878     -     -    32915     -     -
                            lagg0     - 73.xxx.xx.0/2 c-73-xxx-xx-178.h    91007     -     -        4     -     -
                            lagg0     - 2001:558:6045 2001:558:6045:xx:     2153     -     -        0     -     -
                            ix0.3  1500 <Link#12>     00:f0:xx:xx:xx:44    64950     0     0    25768     0     0
                            ix0.3     - 172.xx.9.0/24 edge-dmz                 0     -     -        0     -     -
                            ix0.3     - fe80::%ix0.3/ fe80::1:1%ix0.3          0     -     -      153     -     -
                            ix0.3     - 2601:646:8302 edge-dmz                 0     -     -        0     -     -
                            ix0.4  1500 <Link#13>     00:f0:xx:xx:xx:44  3090700     0     0  1940401     0     0
                            ix0.4     - 172.xx.10.0/2 edge-iot               384     -     -      323     -     -
                            ix0.4     - fe80::%ix0.4/ fe80::1:1%ix0.4         11     -     -      107     -     -
                            ix0.4     - 2601:646:8302 edge-iot                25     -     -       21     -     -
                            ix0.5  1500 <Link#14>     00:f0:xx:xx:xx:44     1342     0     0     2721     0     0
                            ix0.5     - 172.xx.11.0/2 edge-guest               0     -     -        0     -     -
                            ix0.5     - fe80::%ix0.5/ fe80::1:1%ix0.5          0     -     -       90     -     -
                            ix0.5     - 2601:646:8302 edge-guest               0     -     -        0     -     -
                            

                            I cannot prove it, but i still have the feeling that the switch is ever so slightly better at doing the LAGG.
                            However the convenience of having WAN traffic out of the way, easily outweigh that.

                            1 Reply Last reply Reply Quote 0
                            • DerelictD
                              Derelict LAYER 8 Netgate
                              last edited by

                              I would not be at all surprised that a switch is better at a Layer 2 protocol like LACP than FreeBSD.

                              Chattanooga, Tennessee, USA
                              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                              1 Reply Last reply Reply Quote 1
                              • mircolinoM
                                mircolino
                                last edited by

                                With the CoDel limiters now in place, I noticed a new warning in the log, every time the system boots up:

                                config_aqm Unable to configure flowset, flowset busy!
                                

                                I read somewhere else on this forum that this message can be ignored.
                                Is it true? Anyway to prevent it?

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  If it only appears at boot then, yes, it probably can be ignored.

                                  It looks like it's also associated with setting the QMA to CoDel which is not usually necessary. Leaving it as Taildrop with FQ-CoDel as the Scheduler should get the same results.

                                  Steve

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.