Windows 10 updates bandwith limit
-
A common reason for Window's Updates decimating a connection is because of low latency connections with little bandwidth, causing TCP to not honor congestion signals. There's not much you can do about this because of an unnatural relation between provisioned bandwidth and latency.
You could potentially artificially delay these packets using dummynet, assuming this is the issue. If the issue is just normal bandwidth saturation with lots of connections and high bufferbloat, then just enable CoDel.
-
@w0w:
Just update to CU and do proper settings in Update and Security (deferring, branch, active hours, "how updates are delivered" or even pause the updates).
Also setting metering connection can help you.Unfortunately I can't really avoid doing updates. Getting these computers updated is necessary as part of the setup as we need to ensure that there are no problems with updates before returning them to clients. If there's a Windows update that breaks audio for example, we need to know about it. Mostly residential work so I can't use WSUS sadly.
@Harvy66:A common reason for Window's Updates decimating a connection is because of low latency connections with little bandwidth, causing TCP to not honor congestion signals. There's not much you can do about this because of an unnatural relation between provisioned bandwidth and latency.
You could potentially artificially delay these packets using dummynet, assuming this is the issue. If the issue is just normal bandwidth saturation with lots of connections and high bufferbloat, then just enable CoDel.
Hmm. Is there any way to identify Windows Update traffic and dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance. Also, codel, whilst nice and simple, won't cut the mustard either since I need HSFC so that my VOIP is optimised for low latency (which doesn't work due to the bufferbloat effects of Windows Update)
-
@w0w:
Just update to CU and do proper settings in Update and Security (deferring, branch, active hours, "how updates are delivered" or even pause the updates).
Also setting metering connection can help you.Unfortunately I can't really avoid doing updates. Getting these computers updated is necessary as part of the setup as we need to ensure that there are no problems with updates before returning them to clients. If there's a Windows update that breaks audio for example, we need to know about it. Mostly residential work so I can't use WSUS sadly.
@Harvy66:A common reason for Window's Updates decimating a connection is because of low latency connections with little bandwidth, causing TCP to not honor congestion signals. There's not much you can do about this because of an unnatural relation between provisioned bandwidth and latency.
You could potentially artificially delay these packets using dummynet, assuming this is the issue. If the issue is just normal bandwidth saturation with lots of connections and high bufferbloat, then just enable CoDel.
Hmm. Is there any way to identify Windows Update traffic and dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance. Also, codel, whilst nice and simple, won't cut the mustard either since I need HSFC so that my VOIP is optimised for low latency (which doesn't work due to the bufferbloat effects of Windows Update)
You can use codel within HFSC by ticking the "Codel Active Queue" toggle on whichever queue(s) you want.
-
dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance.
If the issue is low latency, reducing the bandwidth will not help. If the CDN is trying to send 20Mb/s at a 10Mb/s connection and refuses to slow down, no amount of shaping will make that any better. If this is the cause, you need to increase the latency.
The minimum bandwidth TCP will consume is (2*MTU)/RTT. Assuming a 12,000bit(1500byte) MTU and 10ms ping, the slowest rate TCP will send, assuming data to be sent, is 1.2Mbit/s. 5ms would be 2.4Mbit/s. This is per connection.
If you had a 1Mbit/s connection with a latency of 10ms, the sender would flood you with 1.2Mbit/s of data even though 17% of the packets are being lost.
-
dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance.
If the issue is low latency, reducing the bandwidth will not help. If the CDN is trying to send 20Mb/s at a 10Mb/s connection and refuses to slow down, no amount of shaping will make that any better. If this is the cause, you need to increase the latency.
The minimum bandwidth TCP will consume is (2*MTU)/RTT. Assuming a 12,000bit(1500byte) MTU and 10ms ping, the slowest rate TCP will send, assuming data to be sent, is 1.2Mbit/s. 5ms would be 2.4Mbit/s. This is per connection.
If you had a 1Mbit/s connection with a latency of 10ms, the sender would flood you with 1.2Mbit/s of data even though 17% of the packets are being lost.
I think that's incorrect. All TCP algos slow down when congestion is encountered. AFAIK, it's part of the spec that TCP algos must slow when congestion is detected. There is no "refuses to slow down" with TCP.
-
Interesting. Didn't know about Codel within HFSC and the mechanics of TCP. Either way, this issue is partially solved for me - I found a regedit that will disable the TCP flooding madness and instead revert to a more traditional form of fetching Windows Updates which is far more controllable: https://forums.whirlpool.net.au/forum-replies.cfm?t=2530363&r=53760513#r53760513
That said, silly question - do you enable codel on the just the qInternet parent queue (not the interface) or do you enable it on each desired subqueue as well..? or only on queues that have a "max bandwidth" limit defined for the queue?
-
dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance.
If the issue is low latency, reducing the bandwidth will not help. If the CDN is trying to send 20Mb/s at a 10Mb/s connection and refuses to slow down, no amount of shaping will make that any better. If this is the cause, you need to increase the latency.
The minimum bandwidth TCP will consume is (2*MTU)/RTT. Assuming a 12,000bit(1500byte) MTU and 10ms ping, the slowest rate TCP will send, assuming data to be sent, is 1.2Mbit/s. 5ms would be 2.4Mbit/s. This is per connection.
If you had a 1Mbit/s connection with a latency of 10ms, the sender would flood you with 1.2Mbit/s of data even though 17% of the packets are being lost.
I think that's incorrect. All TCP algos slow down when congestion is encountered. AFAIK, it's part of the spec that TCP algos must slow when congestion is detected. There is no "refuses to slow down" with TCP.
TCP defines "bandwidth" as the window size. Current implementations of TCP can't slow down to less than 2 segments in flight. That is where "(2*MTU)/RTT" comes from. It's possible some paced versions of TCP don't have this issue.
-
Windows update is on port 80 or 443
Pfsense can't tell the difference from that and a browser
I'd suggest setting up fairq shaper with codel check box enabled.
Set it on both lan and wan at 95% of your link speed.
This will prevent Windows updates from totally dominating your connection
-
I've done that. Even when going through the wizard I'll get somewhere between 80-200ms ping on my VOIP PBX when the link is saturated, which means voice calls can be very patchy.
-
Are you sure that your shaper works? The wizard creates standard rules and most of cases you must manually assign ports and queues in desired rules. Do you see any drops on Status - Queues page under heavy traffic?
Also I did not recommend you to avoid updates, but suggested to try change some settings, anyway it wont help you in this situation, sorry.
The best solution to your problem is to have some really good connection, something over 300Mbps down. If you will shape the updates traffic it is going to slow down and your productivity also goes down. If you don't care about delivery time and can wait some more hours then you must change the shaper model and make default queue with lowest priority and send all other traffic (VoIP, whatever else) to others queues with highest priorities, but it will work only when all parameters are set correctly, like bandwidth. You should see drops in your low-priority default queue, when everything is set correctly. Also, make sure that your pfSense hardware is powerful enough to do shaping.
Remember also that WU uses HTTP(S) and therefore anything else using HTTP(S) will be also laggy. There is nothing to do untill you have the complete microsoft server list.
The solution on windows side (may be complicated or not applicable to some windows versions):
https://answers.microsoft.com/en-us/windows/forum/windows_7-networking/windows-update-takes-all-my-bandwidth/e2edb296-5501-4843-ae33-eaac11e1adb1?page=2Setting the bandwidth allocation for Background Intelligent Transfer Service (BITS) using the group policy setting below would be the most appropriate way of throttling the Windows Update Service.
LCP > Computer Configuration > Administrative Templates > Network > Background Intelligent Transfer Service (BITS)> Limit the maximum network bandwidth for BITS background transfers
I’d suggest the following;
Limit background transfer rate (kbps) to : 10
From : 8AM
To : 6PM
At all other times : Use all available unused bandwidth
LCP > = just run gpedit.msc
and also look for "Limit the maximum network bandwidth for peer caching" -
Honestly, I've been fairly unimpressed with the shaping functionality in PFSense. I have floating rules that match devices and destination SIP addresses to the VOIP queue, but I'm really not too sure what is going on and whether my queues are effective or not. My pings on my workstation are much higher than my VOIP PBX I might add, and I tried using ntopng to analyse what is happening but it's not really telling me in a clear fashion what exactly is happening, whilst absolutely destroying my J1800 CPU (load average of around 2.5 with ntopng enabled - yes, I leave it disabled now). I know it is mostly to do with not knowing exactly what I'm doing, but as someone who has been working with networks for quite some time (10+ years), I don't think it should be this hard. Coming from Gargoyle, it's pretty overwhelming, and whilst I see the appeal in fine-tuning, even the wizard isn't getting me out of trouble.
I just switched back to Gargoyle on my old TP-Link 740N for the meantime on my ADSL link and shaping now works as expected. It also allows me to shape streams that exceed a certain session length (therefore allowing me to prioritise web browsing over HTTP downloads). I don't know why that particular functionality seems difficult or impossible in PFSense. I'll probably switch back once we have a better connection and more time to learn PFSense shaping wizardry (only have around 10/1 at the moment).
-
I thought pfSense's traffic shaping is easy to use and I had zero experience with networks when I first set it up. By "easy", I mean easy to use relative to the power it gives. I spent several days reading about shaping, but all of the tutorials were confusing. Eventually I just figured it out on my own after a few hours of playing.
-
dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance.
If the issue is low latency, reducing the bandwidth will not help. If the CDN is trying to send 20Mb/s at a 10Mb/s connection and refuses to slow down, no amount of shaping will make that any better. If this is the cause, you need to increase the latency.
The minimum bandwidth TCP will consume is (2*MTU)/RTT. Assuming a 12,000bit(1500byte) MTU and 10ms ping, the slowest rate TCP will send, assuming data to be sent, is 1.2Mbit/s. 5ms would be 2.4Mbit/s. This is per connection.
If you had a 1Mbit/s connection with a latency of 10ms, the sender would flood you with 1.2Mbit/s of data even though 17% of the packets are being lost.
How TCP works, if packets start being dropped, and ack starts being missed, the sender is informed to slow down. Strictly speaking, they're not obligated to listen, but microsoft will
-
Honestly, I've been fairly unimpressed with the shaping functionality in PFSense. I have floating rules that match devices and destination SIP addresses to the VOIP queue, but I'm really not too sure what is going on and whether my queues are effective or not. My pings on my workstation are much higher than my VOIP PBX I might add, and I tried using ntopng to analyse what is happening but it's not really telling me in a clear fashion what exactly is happening, whilst absolutely destroying my J1800 CPU (load average of around 2.5 with ntopng enabled - yes, I leave it disabled now). I know it is mostly to do with not knowing exactly what I'm doing, but as someone who has been working with networks for quite some time (10+ years), I don't think it should be this hard. Coming from Gargoyle, it's pretty overwhelming, and whilst I see the appeal in fine-tuning, even the wizard isn't getting me out of trouble.
I just switched back to Gargoyle on my old TP-Link 740N for the meantime on my ADSL link and shaping now works as expected. It also allows me to shape streams that exceed a certain session length (therefore allowing me to prioritise web browsing over HTTP downloads). I don't know why that particular functionality seems difficult or impossible in PFSense. I'll probably switch back once we have a better connection and more time to learn PFSense shaping wizardry (only have around 10/1 at the moment).
The wizard is garbage. Pretend it doesn't exist.
My suggestion: First thing to try:
Just setup a wan and lan queue, with sch_fairq with proper bandwidth limitations, and a single childqueue in each with codel enabled
It's amazing how much better your internet will feel
-
dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance.
If the issue is low latency, reducing the bandwidth will not help. If the CDN is trying to send 20Mb/s at a 10Mb/s connection and refuses to slow down, no amount of shaping will make that any better. If this is the cause, you need to increase the latency.
The minimum bandwidth TCP will consume is (2*MTU)/RTT. Assuming a 12,000bit(1500byte) MTU and 10ms ping, the slowest rate TCP will send, assuming data to be sent, is 1.2Mbit/s. 5ms would be 2.4Mbit/s. This is per connection.
If you had a 1Mbit/s connection with a latency of 10ms, the sender would flood you with 1.2Mbit/s of data even though 17% of the packets are being lost.
How TCP works, if packets start being dropped, and ack starts being missed, the sender is informed to slow down. Strictly speaking, they're not obligated to listen, but microsoft will
AFAIK, they (the sender) are obligated to listen.
According to my limited understanding of RFC 5681 "TCP Congestion Control" , the sender must slow down when congestion is recognized:
3. Congestion Control Algorithms
This section defines the four congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery, developed in [Jac88] and [Jac90]. In some situations, it may be beneficial for a TCP sender to be more conservative than the algorithms allow; however, a TCP MUST NOT be more aggressive than the following algorithms allow (that is, MUST NOT send data when the value of cwnd computed by the following algorithms would not allow the data to be sent).
-
dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance.
If the issue is low latency, reducing the bandwidth will not help. If the CDN is trying to send 20Mb/s at a 10Mb/s connection and refuses to slow down, no amount of shaping will make that any better. If this is the cause, you need to increase the latency.
The minimum bandwidth TCP will consume is (2*MTU)/RTT. Assuming a 12,000bit(1500byte) MTU and 10ms ping, the slowest rate TCP will send, assuming data to be sent, is 1.2Mbit/s. 5ms would be 2.4Mbit/s. This is per connection.
If you had a 1Mbit/s connection with a latency of 10ms, the sender would flood you with 1.2Mbit/s of data even though 17% of the packets are being lost.
How TCP works, if packets start being dropped, and ack starts being missed, the sender is informed to slow down. Strictly speaking, they're not obligated to listen, but microsoft will
AFAIK, they (the sender) are obligated to listen.
According to my limited understanding of RFC 5681 "TCP Congestion Control" , the sender must slow down when congestion is recognized:
3. Congestion Control Algorithms
This section defines the four congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery, developed in [Jac88] and [Jac90]. In some situations, it may be beneficial for a TCP sender to be more conservative than the algorithms allow; however, a TCP MUST NOT be more aggressive than the following algorithms allow (that is, MUST NOT send data when the value of cwnd computed by the following algorithms would not allow the data to be sent).
Due to algorithmic limitations, TCP cannot slow down below this limit. In order for TCP to do its thing, it cannot make the window any smaller than 2 MTUs. The minimum bandwidth cannot be lower than (MTU*2)/Latency. Even if 2 wasn't the smallest, 1 would be. Unless we somehow figure out how to send zero bytes.
These limitations may not apply to paced TCP implementations because they can delay sending packets. Most stacks have no notion of delaying, they just send a packet as soon as the data is ACK'd. If the data is ACK'd too quickly, the connection will get flooded. Of course TCP reacts to congestion by reducing the window, but the minimum window is 2. See the problem?
The issue has to do with an unnaturally low serialization delay to bandwidth relationship. If you have true 1Mb sync connection, it will take 0.024sec to serialize two 1500byte packets, resulting in a theoretical minimum RTT of 0.048. (1500*2)/0.048sec = 0.5Mbit/s as TCP's minimum bandwidth. But now days we have 1Gb networks being rate limited to 1Mb/s making the RTT go from 0.048sec to 0.000048sec, making TCP's minimum bandwidth 1000x greater, which breaks things.
I'm using an extreme example. In the real world, people are using DSL or DOCSIS with much higher framing delays and scheduling delays. But when someone gets provisions some absurdly low amount of bandwidth relative to the actual line rate, and a client opens up many TCP connections, bad things can still happen. Sometimes you hear about some ISP rolling out fiber, which is pretty much like 1Gb Ethernet, then they provision something like 5Mb/s of bandwidth. These situations really make things bad when some local CDN in the ISP has crazy low RTTs to the client.
-
dump it all in a queue of some sort that is only allowed a maximum of 5mbit/0.5mbit? I don't care if it takes a while to update, but killing the connection and causing high bufferbloat is a real nuisance.
If the issue is low latency, reducing the bandwidth will not help. If the CDN is trying to send 20Mb/s at a 10Mb/s connection and refuses to slow down, no amount of shaping will make that any better. If this is the cause, you need to increase the latency.
The minimum bandwidth TCP will consume is (2*MTU)/RTT. Assuming a 12,000bit(1500byte) MTU and 10ms ping, the slowest rate TCP will send, assuming data to be sent, is 1.2Mbit/s. 5ms would be 2.4Mbit/s. This is per connection.
If you had a 1Mbit/s connection with a latency of 10ms, the sender would flood you with 1.2Mbit/s of data even though 17% of the packets are being lost.
How TCP works, if packets start being dropped, and ack starts being missed, the sender is informed to slow down. Strictly speaking, they're not obligated to listen, but microsoft will
AFAIK, they (the sender) are obligated to listen.
According to my limited understanding of RFC 5681 "TCP Congestion Control" , the sender must slow down when congestion is recognized:
3. Congestion Control Algorithms
This section defines the four congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery, developed in [Jac88] and [Jac90]. In some situations, it may be beneficial for a TCP sender to be more conservative than the algorithms allow; however, a TCP MUST NOT be more aggressive than the following algorithms allow (that is, MUST NOT send data when the value of cwnd computed by the following algorithms would not allow the data to be sent).
Due to algorithmic limitations, TCP cannot slow down below this limit. In order for TCP to do its thing, it cannot make the window any smaller than 2 MTUs. The minimum bandwidth cannot be lower than (MTU*2)/Latency. Even if 2 wasn't the smallest, 1 would be. Unless we somehow figure out how to send zero bytes.
These limitations may not apply to paced TCP implementations because they can delay sending packets. Most stacks have no notion of delaying, they just send a packet as soon as the data is ACK'd. If the data is ACK'd too quickly, the connection will get flooded. Of course TCP reacts to congestion by reducing the window, but the minimum window is 2. See the problem?
The issue has to do with an unnaturally low serialization delay to bandwidth relationship. If you have true 1Mb sync connection, it will take 0.024sec to serialize two 1500byte packets, resulting in a theoretical minimum RTT of 0.048. (1500*2)/0.048sec = 0.5Mbit/s as TCP's minimum bandwidth. But now days we have 1Gb networks being rate limited to 1Mb/s making the RTT go from 0.048sec to 0.000048sec, making TCP's minimum bandwidth 1000x greater, which breaks things.
I'm using an extreme example. In the real world, people are using DSL or DOCSIS with much higher framing delays and scheduling delays. But when someone gets provisions some absurdly low amount of bandwidth relative to the actual line rate, and a client opens up many TCP connections, bad things can still happen. Sometimes you hear about some ISP rolling out fiber, which is pretty much like 1Gb Ethernet, then they provision something like 5Mb/s of bandwidth. These situations really make things bad when some local CDN in the ISP has crazy low RTTs to the client.
There is no minimum bitrate because a receiver will delay an ACK until the TCP stack fully receives a packet that's "in flight", which can be artificially delayed by traffic-shaping. Go run a test with a 1kbit incoming limit between 2 GbE LAN hosts and you will see no insane flooding or horribly inaccurate average bitrates at either host.
-
One thought: you could use dscp markings for Windows update, and then flag them as low priority in a shaper
-
Has anyone tried the gpo way?
By "gpo way" I mean tweaking Windows Delivery Optimization settings, as mentioned here(https://forums.whirlpool.net.au/forum-replies.cfm?t=2530363&p=9#r180), and here (https://docs.microsoft.com/en-us/windows/deployment/update/waas-delivery-optimization) (the latest, a more in-depth and "official" explanation)
Maybe this is an off-topic comment, but maybe it can shed light into another direction (the right one?) regarding a feature that doesn't seem to be well understood by the majority (me included) and, at the same time, is causing undeniable issues for many (again, me included).
I'm really not sure if I should try to address this aggressive Windows update mode in pfSense or try to shape it in Windows itself by tweaking its settings via gpo.
