Traffic Shaping Upload per IP

maxx8888

Hi all,

I'm absolute newbie on PFSense, just installed 2 days ago and very satisfied already… Works even better that my loved Bintec tr200aw :)

However, I've read through hundrets of Articles related to pfSense to be prepared already when it finally arrives, but still I don't really understand how-to create the according Firewall Rule correctly.
I've one internal PC with specific IP, which should have absolute priority above all other Traffic. Even when the ADSL Line is completly Full Up-/Download of this specific IP shouldn't be affected at all.

Maybe somebody can give me an Hint where to find this Information on the Web or how-to setup correctly.

For now I've just ran the Wizard with HFSC Scheduler.
Afterwards created 2 additional Queues on Wan and Lan Interface and setup realtime Traffic within it. I think so far everything OK.

Afterwards I've created 1st floating Firewall Rule:
If: Wan&Lan IPv4/TCP SourceIP: 192.168.1.x * * * * qAck/qHighest

This Rule is affecting the LanQueue (Download), no Idea why... When going to Wan -> Lan the Local IP should be Destination not Source, right?

However, afterwards created another Rule:
If: Wan&Lan IPv4/TCP SourceIP: * DestIP: 192.168.1.x * * qAck/qHighest

This Rule seems to not affect any Queue.

Is there any Firewall Setup for Really big Dummies Description somewhere?

Sorry for asking stupid Questions, I just don't have any Ideas anymore.

Thanks a lot,
Markus

KOM

There is no such Traffic Shaping for Dummies guide. Traffic shaping is probably the hardest part of pfSense to understand. First off, I would get rid of the HFSC shaper and try PRIQ as it's much simpler to configure since you don't need to play around with bandwidth values/% for your queues. Make sure your floating rules that direct traffic into the correct queues are set to MATCH, not PASS. Then it's just a matter of reading, experimenting and question-asking.

Harvy66

If all you care about is buffer bloat, don't even doing shaping, just enable fairq or codel on the interface, set the bandwidth to slightly less than your actual bandwidth, and make sure your check "enable".

maxx8888

Hi!

Thanks a lot for your hints Harvy and KOM!

Finally I found the Issue. It was "just" a Typo in the Queue Name. One queue name had a Captial Letter (qABCde and qABcde), which caused pfSense to show 2 different Queues on the Rules. I've just selected the wrong one.

In regards to CoDel, where do i need to enable? Only in the Root Queue, or on all Root and Child Queues?

What i've expirienced with CoDel enabled on some Queues (not sure if it was kicking in):

1.) Important PC Traffic is sent within ~21ms when the Line is empty.
2.) Afterwards i was trying to fill the line, by just starting a Speedtest at speedtest.net.
3.) The Latency of the Important PC is jumping up to around 100ms (which might cause issues already. Even when playing online games 100ms would kick me of some Battlefield4 Servers :-))
4.) It stays at 100ms for around 2sec and then the Shaper kicks in and is slowing down the Speedtest.
5.) The Latency is going down again to around 40ms (which should be OK again).
6.) After the Speedtest is finished Latency is back at 21ms

Is there a way to "react" faster on the Line load change? Why is the Shaper allowing the Line to fill completly and then starts slowing down again.

I'm asking this questions because my old Router had a "Priority Queue". All Traffic handled by this Priority Queue was always around 21ms, no matter what the load on the line was.

Thanks for helping!!
Markus

maxx8888

Hi again,

Seems like i found it out myself already. It seems my ADSL Line has Issues with sending ACK's and Upload Files at the same time.
In order to get this fixed i limited the Upload Speed to 60%, this seems to work fine now.

Thanks,
Markus

Harvy66

Try this setup

Set the scheduler to fairq
Create a single queue
Set that single queue to codel and something like 512 for length
Set that queue to default

Use this speed test, it checks for bufferbloat, which is the issue you're having with gaming.
http://www.dslreports.com/speedtest

maxx8888

Hi Harvy,

Not sure if this is really Buffer bloating. Maybe on ISP side, yes…

If, i carry out like you described, the download looks normally. Bufferbloat indicator goes up to around 50-60ms and stabilize there
But during uploading this indicator stays at around 8ms and jumps to 2000 for just half a second then back to 8. This happens several times during testing.

So, if i connect my Laptop directly to ADSL Line, no Router, my Upload is exactly 910kbit and very stable there.
This is the Value i've entered in pfSense Shaper WAN overall Speed.
On The Child of it (Internet) i've entered to only use 95% maximum.

So, when i do just uploading a file, or a Speedtest, this works flawless. pfSense limits correctly, ping stays low. Upload around 820kbit... (Cannot tell exactly since others in Lan are active also...)

However, as soon as there is a download running in parrallel (lot of Ack (Speed limited to 30%)) Packets, the same Speedtest or Upload is not working anymore.
I need to reduce the Upload Speed to around 40-50%. If the Ack's are limited at 30% (and they don't hit the limit), where is the Rest of 20-30% of Upload Speed gone?

The Quality Tab on RRD Graphs is showing huge Package Loss Values during that.
Does this mean the ISP has dropped Packets because of Overload/shaping my traffic?

So this Download running is really just Bulk very low Importance Traffic.
Is there a way, to shape the Download in case there is Upload needed?

For the Download this is working perfect, I'm shaping the bulk traffic to 1kbyte in case download is needed somewhere else.
In case Upload is needed, this doesn't work at the moment since the Bulk traffic is just causing a lot of Ack's in the Upload which are high important currently.

How about creating a second Ack Queue which can be reduced in case of Upload needed?
This would cause a lot of resending of Packets, right?

So, better would be a way to shape the download to 1kbyte again, in case Upload is needed...

Nullity

@maxx8888:

Hi Harvy,

Not sure if this is really Buffer bloating. Maybe on ISP side, yes…

If, i carry out like you described, the download looks normally. Bufferbloat indicator goes up to around 50-60ms and stabilize there
But during uploading this indicator stays at around 8ms and jumps to 2000 for just half a second then back to 8. This happens several times during testing.

So, if i connect my Laptop directly to ADSL Line, no Router, my Upload is exactly 910kbit and very stable there.
This is the Value i've entered in pfSense Shaper WAN overall Speed.
On The Child of it (Internet) i've entered to only use 95% maximum.

So, when i do just uploading a file, or a Speedtest, this works flawless. pfSense limits correctly, ping stays low. Upload around 820kbit... (Cannot tell exactly since others in Lan are active also...)

However, as soon as there is a download running in parrallel (lot of Ack (Speed limited to 30%)) Packets, the same Speedtest or Upload is not working anymore.
I need to reduce the Upload Speed to around 40-50%. If the Ack's are limited at 30% (and they don't hit the limit), where is the Rest of 20-30% of Upload Speed gone?

Some ineffeciency is unavoidable. Unless you see a clear and obvious problem, I would leave it until you have a better understanding of precisely what is causing your problem, if it is even a problem.
You may need to adjust the rate-limit below 95%. This QoS tutorial is awesome, since the author not only runs the internet for multiple residential buildings, he also uses his own customized Tomato firmware.

The Quality Tab on RRD Graphs is showing huge Package Loss Values during that.
Does this mean the ISP has dropped Packets because of Overload/shaping my traffic?

You would not see a "drop" if your ISP dropped a packet, you would only see the void where a packet should be. If you see drops, then you are most likely the one dropping them. Like if I sent you mail and the letter got lost, the only person who could immediately know the letter was lost was the dude who lost it. We would just be waiting a couple weeks, wondering.

So this Download running is really just Bulk very low Importance Traffic.
Is there a way, to shape the Download in case there is Upload needed?

For the Download this is working perfect, I'm shaping the bulk traffic to 1kbyte in case download is needed somewhere else.
In case Upload is needed, this doesn't work at the moment since the Bulk traffic is just causing a lot of Ack's in the Upload which are high important currently.

How about creating a second Ack Queue which can be reduced in case of Upload needed?
This would cause a lot of resending of Packets, right?

So, better would be a way to shape the download to 1kbyte again, in case Upload is needed…

This all gets very complicated quite quickly.

2 ACK queues are completely possible, in theory, but I there are better ways to accomplish your goal, in my opinion. As long as you are keeping ingress streams below your purchased bitrate, by artificially limiting the incoming TCP streams at the LAN, then downloads should be free of buffer bloat.

You can only control, with any respectable latency, your upload/egress traffic. Download traffic is "shaped" by dropping packets (only works on TCP), and the missing packets cause the sending to throttile back, but the amount of throttiling and the time it takes for the sender to react are all unknowns. The QoS tutorial I meantioned says limiting your download to ~60% is just about the only way to keep it from becoming congested.

maxx8888

Hi Nullity,

Thanks for your detailed Explanation. I've limited the download for Bulk Traffic to 50% now. Not really a perfect Solution, but works very well.

The Tutorial is perfect! Need to read through it again…

Regards,
Markus

Harvy66

That QoS tutorial is pretty good, but I did find at least one thing that I did not agree with. It could be that his router did not support HFSC and is a limitation of lesser traffic shapers, but this is not an issue for a properly configure HFSC PFSense box.

Suppose our hero P2P user gets some good seeds, or you have several users all determined to download the Lord of the Rings in record time. Nobody else using the router, so our upload bandwidth rises to close to maximum. Now, let's suppose you wake up, and try to access the morning newspaper online while swigging your coffee. OOPS - you've got no bandwidth :mad:

Tomato has to do something. It will try to increase the bandwidth for your HTTP class, but it has to wait until some becomes free. There's a significant delay while tomato shuffles things around. In other words, an application in a lower class, if allowed to take a significant part of your upload bandwidth, CAN indeed slow down response time of higher priority classes. So don't set your rate TOO low….

He then goes on to mention

At the same time, incoming bandwidth due to the P2P is full or near full speed. There's probably a queue of P2P waiting at the ISP.

If you had your traffic shaper to be below your actual ISP bandwidth, this would not be an issue.

Nullity

Probably related to the fact that you cannot technically shape ingress traffic. He has to preemptively allocate enough throughput to consistently avoid ISP buffering, even when 30 simultaneously people click on the latest cat video Tweet blast.

Honestly, when I read his tutorial early in my traffic-shaping research I thought he had quite a few things wrong. Over time I have come to realize that he really knows his stuff. I suppose experience works that way. :)

Harvy66

You can't shape ingress traffic, but most traffic is not a DOS and follow rules. UDP traffic is typically fixed bandwidth and will not attempt to fill up your pipe, while TCP will attempt to fill up the pipe, but backs off on packet-loss.

In my case, prior to my ISP having an AQM and had a hard cut-off for bandwidth by using the rate limiting built into my ONT which was very strict, setting my LAN interface to about 95% of my bandwidth pretty much kept ping spikes out, which means no buffering on my ISP's side. I could have reduced my bandwidth further and tightened the ping spikes, but way too much diminishing returns. I was already down near 10ms. While 98% link speed resulted in packet-loss and some major ping spikes. That 3% different was pretty big.

My point is TCP is pretty good at responding to congestion. Latency is a big issue. My tests were primarily against busty traffic like speedtests or youtube, which I had between 10ms and 20ms. If the sender is further away, like 200ms, it will take that much longer for the packet-loss signal to reach them.

It really depends on your typical use cases.