Downstream drops when upstream is saturated

eng3

@stephenw10 As I said, this isnt the only software that may saturate the upstream. It also may not always be from the same device. Can we just focus on my actual question? Why would the downstream bandwidth drop 10X if upstream is saturated while latency being ok.

stephenw10

Well I imagine it's because it's dropping TCP ACK packets in the upload queue causing the TCP window to close down to something very small.

Can we see how you have those Limiters applied?

In the limiter Info above you have 12 packets total shown on the download pipe which seems wrong.

Steve

eng3

@stephenw10 Shouldn't it be dropping more without the limiter? Without the limiter, speeds are high. I just followed the manual exactly https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html

Limiter and Queue has Queue Management Algorithm set to "Tail Drop"
Limiter has Scheduler set to "FQ_CODEL"
Everything else was left at default.

Then I played around with the bandwidth settings

I assume 12 packets on the down pipe was because I wasn't downloading anything at the time.
If I do a speed test, then it goes up a little. Ofcourse with the downstream being so limited, there isnt much traffic. On the speed test, I notice "single stream" will only go at around 2M were "multi stream" will get up to 50M. Without the limiter its like 200M

Limiters:
00001: 160.000 Mbit/s    0 ms burst 0 
q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:   6.000 Mbit/s    0 ms burst 0 
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001: 160.000 Mbit/s    0 ms burst 0 
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0      251   330328 46 69000   0
00002:   6.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     474062 593056906 265 298902 66543


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail

stephenw10

Hmm, the totals on the queues there are ~600MB upload and ~300KB download.

So either the download traffic is not correctly using the limiter. Or your firewall rule is applying the limiters reversed, which might explain the very low speeds.

Did you note the warning that the queues are reversed for an outbound floating rule?

eng3

@stephenw10 I have IN set to the Up queue and OUT set to the Down queue.
Up limiter is set to 6M Down limiter is set to 160M

The issue only occurs with the limiters active AND upstream saturated.

With no traffic and speed test running during download phase:

Limiters:
00001: 160.000 Mbit/s    0 ms burst 0 
q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:   6.000 Mbit/s    0 ms burst 0 
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001: 160.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     201056 301086289 114 171000 1325
00002:   6.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0        3      144  0    0   0


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail

Now with upstream saturated, speed test during download phase:

Limiters:
00001: 160.000 Mbit/s    0 ms burst 0 
q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:   6.000 Mbit/s    0 ms burst 0 
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001: 160.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       76    82028  0    0   0
00002:   6.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     91307 121523662 170 216086 15670


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail

stephenw10

That's what I would expect. As a test try setting the download limit to, say, 100Mbps and make sure it's actually catching that.

eng3

@stephenw10 What do you mean by "catching it"? what do I look for? Also, I just edited my post

stephenw10

I mean if the downstream limiter is set to 100Mbps and the rule it applied to is correctly matching all outbound connections you should not be able see more than 100Mbps in a speedtest.

eng3

@stephenw10 Yes I was able to verify this

eng3

@stephenw10 The other thing I noticed is that with the limiter active, if I go on another computer and trying to ping a random site (google.com), every 10-20th ping will just time out. I tried to apply the limiter only to the IP of the computer currently uploading (saturating) and now every ping works.
Overall performance seems a little better, however certain sites (ie united.com, tripadvisor.com, capitalone.com, maybe sites with alot of dynamically loading content) are very slow to load and some parts just fail causing me to have to click reload

stephenw10

Just all the time or only when the upload is saturated still?

Are you still using FQ_CoDel with the limiter only applied to the uploading host?

I would start with something basic here. Just set a limiter with default config to only the uploading host so it cannot saturate the upload bandwidth and go from there.

Steve

eng3

@stephenw10 Only when saturated. Yes for now, I figured I'd give it some time to see if the issue persists or was a coincidence.

ok. If this works, its a decent temporary fix, but in the future I may have multiple IPs that could saturate the upstream.