Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Downstream drops when upstream is saturated

    Scheduled Pinned Locked Moved General pfSense Questions
    20 Posts 3 Posters 943 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E Offline
      eng3 @stephenw10
      last edited by

      @stephenw10 As I said, this isnt the only software that may saturate the upstream. It also may not always be from the same device. Can we just focus on my actual question? Why would the downstream bandwidth drop 10X if upstream is saturated while latency being ok.

      1 Reply Last reply Reply Quote 0
      • stephenw10S Offline
        stephenw10 Netgate Administrator
        last edited by

        Well I imagine it's because it's dropping TCP ACK packets in the upload queue causing the TCP window to close down to something very small.

        Can we see how you have those Limiters applied?

        In the limiter Info above you have 12 packets total shown on the download pipe which seems wrong.

        Steve

        1 Reply Last reply Reply Quote 0
        • E Offline
          eng3
          last edited by eng3

          @stephenw10 Shouldn't it be dropping more without the limiter? Without the limiter, speeds are high. I just followed the manual exactly https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html

          Limiter and Queue has Queue Management Algorithm set to "Tail Drop"
          Limiter has Scheduler set to "FQ_CODEL"
          Everything else was left at default.

          Then I played around with the bandwidth settings

          I assume 12 packets on the down pipe was because I wasn't downloading anything at the time.
          If I do a speed test, then it goes up a little. Ofcourse with the downstream being so limited, there isnt much traffic. On the speed test, I notice "single stream" will only go at around 2M were "multi stream" will get up to 50M. Without the limiter its like 200M

          Limiters:
00001: 160.000 Mbit/s    0 ms burst 0 
q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:   6.000 Mbit/s    0 ms burst 0 
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001: 160.000 Mbit/s    0 ms burst 0 
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0      251   330328 46 69000   0
00002:   6.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     474062 593056906 265 298902 66543


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
          1 Reply Last reply Reply Quote 0
          • stephenw10S Offline
            stephenw10 Netgate Administrator
            last edited by

            Hmm, the totals on the queues there are ~600MB upload and ~300KB download.

            So either the download traffic is not correctly using the limiter. Or your firewall rule is applying the limiters reversed, which might explain the very low speeds.

            Did you note the warning that the queues are reversed for an outbound floating rule?

            E 1 Reply Last reply Reply Quote 0
            • E Offline
              eng3 @stephenw10
              last edited by eng3

              @stephenw10 I have IN set to the Up queue and OUT set to the Down queue.
              Up limiter is set to 6M Down limiter is set to 160M

              The issue only occurs with the limiters active AND upstream saturated.

              With no traffic and speed test running during download phase:

              Limiters:
00001: 160.000 Mbit/s    0 ms burst 0 
q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:   6.000 Mbit/s    0 ms burst 0 
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001: 160.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     201056 301086289 114 171000 1325
00002:   6.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0        3      144  0    0   0


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail

              Now with upstream saturated, speed test during download phase:

              Limiters:
00001: 160.000 Mbit/s    0 ms burst 0 
q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:   6.000 Mbit/s    0 ms burst 0 
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001: 160.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       76    82028  0    0   0
00002:   6.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     91307 121523662 170 216086 15670


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
              1 Reply Last reply Reply Quote 0
              • stephenw10S Offline
                stephenw10 Netgate Administrator
                last edited by

                That's what I would expect. As a test try setting the download limit to, say, 100Mbps and make sure it's actually catching that.

                E 1 Reply Last reply Reply Quote 0
                • E Offline
                  eng3 @stephenw10
                  last edited by

                  @stephenw10 What do you mean by "catching it"? what do I look for? Also, I just edited my post

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S Offline
                    stephenw10 Netgate Administrator
                    last edited by

                    I mean if the downstream limiter is set to 100Mbps and the rule it applied to is correctly matching all outbound connections you should not be able see more than 100Mbps in a speedtest.

                    E 2 Replies Last reply Reply Quote 0
                    • E Offline
                      eng3 @stephenw10
                      last edited by

                      @stephenw10 Yes I was able to verify this

                      1 Reply Last reply Reply Quote 0
                      • E Offline
                        eng3 @stephenw10
                        last edited by

                        @stephenw10 The other thing I noticed is that with the limiter active, if I go on another computer and trying to ping a random site (google.com), every 10-20th ping will just time out. I tried to apply the limiter only to the IP of the computer currently uploading (saturating) and now every ping works.
                        Overall performance seems a little better, however certain sites (ie united.com, tripadvisor.com, capitalone.com, maybe sites with alot of dynamically loading content) are very slow to load and some parts just fail causing me to have to click reload

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S Offline
                          stephenw10 Netgate Administrator
                          last edited by

                          Just all the time or only when the upload is saturated still?

                          Are you still using FQ_CoDel with the limiter only applied to the uploading host?

                          I would start with something basic here. Just set a limiter with default config to only the uploading host so it cannot saturate the upload bandwidth and go from there.

                          Steve

                          E 1 Reply Last reply Reply Quote 0
                          • E Offline
                            eng3 @stephenw10
                            last edited by

                            @stephenw10 Only when saturated. Yes for now, I figured I'd give it some time to see if the issue persists or was a coincidence.

                            ok. If this works, its a decent temporary fix, but in the future I may have multiple IPs that could saturate the upstream.

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.