Limiter bandwidth drops if I add delay
I need to simulate a bad connection and I have no problems using Limiters to limit bandwidth to almost any value I need for upload and download (I have a 300Mbit /up/down connection and can limit to any value I want. e.g. 5Mb down / 2Mb up without issues). However, when I throw delay (latency) or packet loss in the mix, the bandwidth drops heavily.
For example: if I set 200 Mb download and 200 Mb upload and not delay or packet loss, the speeds I see are 200 Mb +/- a few Mb. If I add 100ms delay to upload, then what I see is 60 Mb download and 50 Mb upload. If I add 200ms delay to upload, then I see 30 Mb download and 40 Mb upload.
If instead of a 200 Mb up/down limit I use a 100Mb limit, then with with 200ms delay I can only get a bandwidth of max 25 Mb. If I also add packet loss to the mix, then bandwidth further drops to less than 1 Mb for upload.
Is this correct?
Is there any way of simulating a connection with a set bandwidth (e.g. 100 Mb) while also having delay and packet loss?
Could be your TCP stacks or negotiation. Depending on how you're measuring the bandwidth, a TCP connection with a 64KiB window and 200ms of latency would have a max bandwidth of ~2.6Mb/s. If you test opened 10 TCP connections, you would see about 26Mb/s.
Most TCP stacks support large windows beyond 64KiB. If you're using something like SYN proxy, then this will be broken and limited to 64KiB.
I`m using several bandwidth monitoring tools like Speedtest.net (including their Windows Store app) and they all show pretty much the same results. There no other special configuration done to the network - pfSense is the router (I have not configured anything regarding TCP stack/negotiation/window) and behind it are the devices that need to be limited.
Would configuring the Advanced Options for limiters (Queue size (slots) & Bucket size (slots)) help with this in any way? Or is there anything else I can configure or a better way to test the bandwidth and delay / packet loss limitations?
The thing is that I`d like to simulate for example a connection with 10 Mb upload/download, 200 ms delay and 10% packet loss, but anyway I configure that I do not get more than 1Mb.
However, when I throw delay (latency) or packet loss in the mix, the bandwidth drops heavily.
That's how TCP works.
I use this on occasion to see if what is being seen is anywhere close to what is theoretically possible:
OK, thank you very much, that is informative and a good way to check if the connection is reaching the maximum potential.
However, then I understand that there is nothing I can do from pfSense itself to increase bandwidth when there is a delay and packet loss set, since pfSense will not mess with the TCP buffer size for connections passing through it - is that correct?
Right. The buffer size is negotiated between the endpoints.
You can clamp MSS but that is more about working around a packet size limitation somewhere in the path.
OK, got it, thank you for your help!
You might also be served well by testing locally using iperf3 or maybe even TRex so you know you are not looking at internet inconsistencies.
Thanks, I`ll definitely give at least iperf a try.
Just for clarification, lower bandwidth because of higher latency is not fundamental to TCP, but is a very common issue with many TCP stacks. In theory, the TCP large windows are supposed to prevent this, but in practice, I also notice upper "limits" with large RTTs. SYN proxy breaks TCP large window negotiations because the proxy can't know if the client will support it and the SYN proxy happens before interacting with the client.
Newer TCP implementations that are being rolled out in datacenters, but not quite ready for primetime for a worldwide rollout, are virtually unphased by higher terrestrial RTTs and mild loss. One such example is from Youtube and Google is called BRR, and BRRv2 is right around the corner.
I am not sure of Microsoft has plans for when they'll update their TCP, but I would hope that pretty much everyone will be using a "modern" TCP stack in the next 10 years.
There are theoretical TCP throughput limits based on RTT and window size.
That calculator does a pretty good job at estimating what can be an expected maximum.