Traffic shaper limit throughput not stable

FlangeMonkey

Hi Guys,

I've been trying to setup some basic limiters for testing before I move onto more advanced QoS. I am using pfSense in a VM on ESXi.

I have simple limiters of 200Mbit and 10Mbit. When I do a download test without limiters its 100% 230Mbit stable throughput. When I turn on the limiter of 200Mbit down, I get approx. 100 to 170 Mbit. Additionally, the download spikes up and down sometimes below 70Mbit.

My processor or memory isn't constrained. Does anyone have any idea why the limiter isn't limiting to a stable/constant 200Mbit? I've tried all kinds of things except physical hardware and I think this could be a virtual/FreeBSD thing.

Thanks,

FlangeMonkey

Hi Guys,

Is it my understanding of what the limter are? I'm not expecting any schedulers or algorithms on the pipe, so should I be expecting a hard limit or a scheduler of some sort to help with bufferbloat and therefore not 100% limit.

I notice when I run 'ipfw pipe list', the pipe is FIFO, but I am seeing a sched using WF2Q+, when I execute 'ipfw sched list'.

Thanks,

Harvy66

Is there a queue depth set? I don't know about limiters, but shapers default to 50 packets, which is not enough and causes dropped packets at high rates which reduces throughput.

FlangeMonkey

@Harvy66:

Is there a queue depth set? I don't know about limiters, but shapers default to 50 packets, which is not enough and causes dropped packets at high rates which reduces throughput.

Thank you Harvy66, this appears to have resolved the issue. I do however need to set this to 400 to get constant throughput, what would you expect this to be and can it be bad having too high a number?

Edit: so basically bufferbloat if the number is too high. So I need to understand the settings at this point and possibly move onto Codel or something else.

Thanks,

Harvy66

If you're worried about bufferbloat, you could look into setting up fq_Codel on your limiters. Otherwise, a good rule of thumb is to aim for about 10ms-50ms of buffer. How to calculate this. The average backbone packet size is about 600bytes. 600 bytes time 8 = 4800bits.

(Bandwidth*10ms)/4800bits = queue size

(200Mbit*10ms)/4800bits = 416 packets.

I guess your 400 packets is about right.

Keep an eye on this if you don't want to manually setup fq_Codel. https://redmine.pfsense.org/issues/6620 They're planning on adding it to the UI at some point. I assume it may be as easy as a check box on limiters.

lawrencedol

Check out this link:

https://www.reddit.com/r/PFSENSE/comments/3e67dk/flexible_vs_fixed_limiters_troubleshooting_with/

It worked perfectly for me, including giving me top-notch VOIP while bandwidth is fully saturated with upload or download traffic, and perfectly dividing the bandwidth between multiple users.

The most salient sections from that post follow.

Fixed Limiters

These are the more commonly discussed limits from what I've seen. Fixed limits are used when a network operator wants to permit only a very specific upper bound of bandwidth to be used by an individual device, no matter what. Use cases might include public WiFi scenarios, where a network operator wants to discourage people from relying on it being a top quality connection to avoid attracting people who camp out on their network consuming maximum bandwidth all day.

Example goal: 256kbps upload limit, 1mbps download limit (enforced per device)

The configuration in Firewall > Traffic Shaper > Limiter:

Create a new Pipe
Name: Upload
Bandwidth: 256kbps
Schedule: None
Mask: Source addresses (no need to type a number into either of the numeric field boxes in this section)
Create a new Pipe
Name: Download
Bandwidth: 1mbps
Schedule: None
Mask: Destination addresses (no need to type a number into either of the numeric field boxes in this section)

The configuration in the applicable LAN-side firewall rule:

Advanced > In/Out: Upload / Download

Flexible Limits

These are less common, and I didn't realize it was actually possible to do this with pfSense until I got Steve's feedback (forum discussions allude to it, but I haven't seen a correct config fully described anywhere yet). The purpose of flexible limits is to allow pfSense to enforce a total cap on user traffic and to dynamically manage the connections based on real network conditions – allocating more bandwidth per device when the network is quiet and less bandwidth per device when many clients are chatting at the same time. In my case, I've seen users report pleasantly usable network conditions consistently even while the network link was 100% saturated -- this is a very good tool to have in your kit for overloaded Internet uplinks (in one case, I've got a large download capacity but a very small upload capacity, and the users would completely overload the upload, resulting in poor conditions for everyone until I implemented this flexible limiter).

Example goal: Provide a high quality user experience for hundreds/thousands of devices sharing a business-class cable connection with 300mbps download and 20mbps upload capacity.

The configuration in Firewall > Traffic Shaper > Limiter:

Create a new Pipe

Name: Upload
Bandwidth: 18mbps (put the total amount of bandwidth available here; remember to save a small amount of bandwidth for remote management, downloading packages, etc -- in this example, we're allowing 18mbps for users on a 20mbps line)
Schedule: None
Mask: None
Create a new Queue under Upload
Name: UploadQueue
Mask: Source addresses
Create a new Pipe
Name: Download
Bandwidth: 290mbps (in this example, we're allowing 290mbps for users on a 300mbps line)
Schedule: None
Mask: None
Create a new Queue under Download
Name: DownloadQueue
Mask: Destination addresses

The configuration in the applicable LAN-side firewall rule:

Advanced > In/Out: UploadQueue / DownloadQueue