You won't know about your priorities working or not working unless you set up a test where one of the higher priority queues is running enough traffic to shove the lower priority queue down, adequately continuously and for a long enough period that you can see what is or is not happening.

It may be that "borrow must be on" for any of the children to borrow, rather than "it's trying to share your qlink." If qLink is not borrow=on, I think the borrow=on setting on qInternet does not provide it with any access to your 1GB LAN/qLink. You could explicitly set qLink to borrow=off and see what happens, but I think that is the default - but I'm also still very much in the place where the depths of what I don't know about the shaper exceed what I do know by a huge margin.

You could also try setting qAck to borrow=off and see if you lose about 40% of your download on qLow, I guess. But that may not be the way borrow actually works. It would be a simpler experiment to set up, though.

I would assume so.

Hi Georgeman,

I'm really interested in the solution you're using, we have to connect 4 remote sites and bring phones trafic to central site and priorise VOip in the VPN tunnel.

Could you post the TS configuration you've done?

Thanks,

Thomas

I just set my receive queues to 2.5k. It's pretty much an issue just limited to traffic that can burst in quickly. Because interactive streams, like games, are on their own separate queue with reserved bandwidth, it seems to not affect anything except their own queues. Because the burst is being rate limited to fit into 48Mb/s in a much smoother fashion via PFSense, than Cisco, my machine cannot ACK data that it has not received yet, so the other side backs down. It seems to be that it's not so much the burst causing issues, but that my machine would normally ACK all of the data in that burst as quickly as it came in, indicating to the other side that I'm ready to receive more, when it really needs to back off before Cisco clamps down hard.

This is mostly me just theorizing, but I am seeing much better results.

I did find that I need to limit my P2P's queue size. During the ramp-up of a heavily seeded torrent, like Fedora, the hundreds of sending end-points would still peak over 50Mb/s on my WAN interface before leveling off, even though PFSense was making sure that I was only getting 48Mb/s. So while a large queue to soak the burst from a single sender works fine, a large queue for many senders that are all ramping up at the same time can cause issues.

P2P also has a lot less burst than Google services. I don't really have the issue of 1gb micro-bursting from Torrents. If I remember correctly, Google uses a custom TCP setup where they purposefully burst the first X bytes at or near full line rate, to make better use of available bandwidth. They let network buffers worry about the bursts. The "problem" is that between my ISP and Google is Level 3, and no congestion. It just lets that 1gb burst right on through 8 hops and 250 miles.

@KOM:

From what I have also read, blank LAN bandwidth equates to "100%".  We might get more detail with the output of the pfctl command.

Since bandwidth is just LinkShare, based on what I've read, the m2 in LinkShare should always override what's in bandwidth. Having said that, I went back and removed all of my link shares and just set my values in bandwidth instead and I set Real to the same as Bandwidth.

It works! I may not have it set optimally, but good enough. I'm just a home user.

It's easy for me to test my download, but it's hard to test my upload. Anyway, here's the results. You'l notice that my quality graph shows my ping going down. My ISP's gateway seems to respond to pings more quickly under load… Probably a thread scheduling thing, since ping responses are handled by the host CPU and not the ASIC.

This has solved my random packet-loss during high utilization. I normally get 0%, but some times when hammering my connection, it will get into the 0.04% range.

Thanks everyone!

P.S. If someone has a better way to loadtest my connection than Torrenting Linux ISOs while running SpeedTest.Net, I'm all ears.

Again, I have little experience and am just learning myself. You could set the default queue to have virtually no bandwidth, then create other queues for stuff like games and web. So 80/443 would get web, and you could add a list of known common games and add their ports.

But if you have multi-LAN and limit each LAN to it's proportional share of the WAN, then are you not essentially setting an upper limit for each LAN?  If you have a 40 Mb link and 4 LAN queues and giving each LAN queue 10 Mb, then if you have a busy LAN and 3 quiet ones, you are limiting the busy LAN to 10 Mb.  This all depends on how the WAN/LAN speed settings affect everything.  If it's just a value used in calculations and the queue will absorb whatever bandwidth is available, then fine.  If it also acts as a hard cap then that's a problem.

How will I setup port forward if I have 2x WANS? with LAGG!

@Ecnerwal:

I do not concur that "no documentation is a good thing."

FAIRQ is not an option in the Wizard setup (at least on 2.1.3, where my applicable system sits for now, but I bet the same is true in 2.1.5) though is IS an option for scheduler type on the shaper, non-wizardly. My experience of the "wizard" is not all that happy anyway. I suppose you could try PRIQ in the wizard and then change it to FAIRQ. Either way it's the moral equivalent of wiggling a screwdriver blindly in a high-voltage box in the hopes that it makes the right connection, with the documentation where it sits now.

It appears to me that most of what's mentioned in this retired topic from 2010/11 still applies to the shaper today. It certainly feels all too familiar and current. The fact that the shaper documentation (as linked from 2.1) still starts off with 1.2.x and then has 2.0 (work in progress) [but the work has never progressed] is rather depressing.

https://forum.pfsense.org/index.php?topic=26782.msg139435#msg139435

I don't see a question in there…  ???

There are (many) more places to find pfSense documentation than the official wiki.
I gained a bit of insight by reading source-code for FAIRQ, which was initially introduced in DragonflyBSD. Maybe that will help you too? I'm no C coder, but there are useful comments in the source-code.

For a super simple FAIRQ setup, you simply select your outbound interface in the Traffic Shaper, select FAIRQ and click Add queue. I think it is best to explicitly direct traffic into the queue because I think the maintainer of FAIRQ recently fixed a bug that was causing problems when traffic was defaulted into FAIRQ, and I'm not sure if that bugfix has been merged with pfSense's FAIRQ yet. (Disclaimer: I may not know wtf I am talking about.)

P.S. - I never said "no documentation is a good thing". Documentation is out there, you just need to find it. I did, and I'm a newb. :)

Hi, did it work pierre_rs?

Ok… thank you for your confirmation.

@Derelict:

When you set the destination slots there's really no reason to set burst.  They get the full download speed unless someone else is in contention for it anyway.  Unless there's a reason you don't want someone to run free if the capacity is available, I'd just ditch it.

Actually, looking at it, it's really not going to do much for you, because the "Limit" that's going to be applied after the burst is exceeded will be the same as if no burst was applied at all.

Thank you once again for the explanation - I see your point. I need to study up more on this. If bandwidth is available, I'd certainly want to allow any host to run free. However, when it's peak hours, I'd like to split evenly among hosts, which is what it does now and works great. The last think I'd love to have happen is to actually have it not quite split evenly between hosts when certain hosts have been downloading steadily while others are just trying to pull up a website. I thought that was something that burst would help with but perhaps I need to configure it differently.

I think you're going to have a hell of a time trying to get a TDM circuit transported over a 300ms RTT VPN link.  No amount of traffic shaping is going to make that any better.  SIP trunking would probably be better, but 300ms is a lot for that too.

Do you want to set, say, a hard limit on VLAN100 of 10Mbps down or do you want the whole 45Mbps available if nobody else is using it? (The latter is harder and I'm not sure if it's even possible using the limiter when multiple interfaces/bandwidth goals are present.)

If you switch to HFSC you will run into the multi-LAN interface config issues.  See the first part of this post for an brief explanation: https://forum.pfsense.org/index.php?topic=79589.msg434856#msg434856

Well, there's this, https://forum.pfsense.org/index.php?topic=62188.0 but I can't say I had the best of luck with it. Then there's this which I haven't tried yet: https://forum.pfsense.org/index.php?topic=66537.msg366615#msg366615

In theory if you can separate "traffic coming from the pfSense box to the LAN" and "traffic coming from the internet to the LAN" it "should be easy." Due (in my frustrated opinion) to the dismal, out of date, and incomplete documentation, nothing on the shaper is ever easy.  IF I grok the sense of the "short form approach for pfsense 2.1" in the second linked message (not the link to an older thing the linked message is replying to) I believe it's trying to do exactly that. But I have no idea if it actually accomplishes the desired effect or not.

I did successfully get shaping to work, but it shaped cache hits. I don't know who would want that behavior, but it's not me.

I strongly suggest making a backup of the configuration before you start working on the shaper, as every time I have tried to get it to shape without shaping cache hits, it's blown up in my face and needed to be reset to a working configuration from before that point.

I'm just about ready to try tilting at this particular windmill again, which is why I'm here reading your unanswered post. I wish I could offer you a more hopeful answer, but I can't, as it's all poking at things with clear as mud directions and holes you can drive a truck through in those.

Another approach suggested from the "olden days" is to just put squid on a separate box, i.e.
Internet <–>pfSense (and shaper)<-->Squid (all by itself)<--> users.
Annoying, but might be easier than trying to get this to work.

https://doc.pfsense.org/index.php/Traffic_Shaping_Guide#Limiter

@sideout:

1.)  The config files you provided.  Shaper config appears to be the shaper queues, filter config appears to be the supporting firewall rules, but what is aliases?

Seriously you don't know what aliases are after reading the tab in PFSense?

2.)  When setting up HFSC you need to tell it how much bandwidth you have up and down to make sure that prioritization occurrs locally, rather than remotely.  I can't seem to find where this setting is in your PRIQ example.  Is it not required for PRIQ?  Only one of your queues has a "Queue limit" of 500, and it is qLink, which doesn't appear to be assigned to anything in rules.

Multiple forums post on this - HFSC does not use the priority setting but the wizard puts it in there.  Also if you look at all the check marks on qLink you would see it is the default queue on the LAN interface so you would know that typically if there is not a rule allowing or disallowing something then it goes to the default rule.

3.) I'm gathering from your rules that traffic rules should be floating rules?  What is a floating rule?

Again - you dont know what a floating rule is after reading the tab in PFSense?  Plus if you went here https://doc.pfsense.org/index.php/Category:Firewall_Rules then you will see the very same question you asked answered already.

4.) Some of your queue's are assigned to WAN and some to LAN.  Does this correspond to incoming and outgoing traffic?  Which is which?  If I had to wager a guess upstream would be on the LAN side and downstream on the WAN side.  Is this correct?

All the queues on the floating rules tab should be assigned to the WAN interface only.  There are specific rules that get assigned to the LAN for things like the limiter.
5.) Clicking through all of your queues, I can't seem to find where I tell the queue if it is HFSC or PRIQ?  How do I define this?

Again you can only have HFSC or PRIQ not both.  That is defined on the interface so if you go under Traffic Shaping and read what the drop down box says , you know what you have set.
6.) Do you recommend starting with the wizard and modifying the queues as needed from there, or creating them manually?

I recommend creating them manually unless you dont know what you are doing then start with the wizard and choose a very basic simple setup and modify it from there.

7.) I can see how I can assign hosts to each queue using rules.  How do I tell the system to send all other clients that have not been manually assigned to a "Default client" queue?  Is it just like other firewall rules, where I create an ALL rule at the bottom, that assigns everything that hasn't been otherwise specified to my "default" queue?

https://doc.pfsense.org/index.php/Firewall_Rule_Basics

8.) In your example, you have specified UDP or TCP for all of your rules.  Is there any reason I can't just tell it to apply to all protocols for the specific host?

In my experience I have found that using a combo rule for TCP/UDP with HFSC shaping does not work that well in high packet situations.  I prefer to separate them as when using floating rules with TCP you need to define qACK but with UDP you do not need qACK.

9.) It would seem all of your rules are associated with the WAN interface.  Some specify the source and some the destination.  I'd imagine that this is to create rules for upstream and downstream for each.  Is that accurate?  I would have expected based on the observation in #4 above, that downstream would need to be assigned to WAN, and upstream to LAN.  Is this not the case?

https://doc.pfsense.org/index.php/Firewall_Rule_Basics

Thank you.  I do appreciate you taking the time, and having a little patience with me.

I think part of my problem is a terminology gap.  Been doing a lot of googling and browsing around the pfsense documentation, but obviously not for the right terms!

This - hopefully - should point me to the right reading to understand all of this.

Thanks!

Final Graphs.

nexuwanrddfinal.jpg_thumb
nexuwanrddfinal.jpg
nexuslanwanrddfinal.jpg_thumb
nexuslanwanrddfinal.jpg
nexuslanrddfinal.jpg_thumb
nexuslanrddfinal.jpg
NexulanPrtgfinal.jpg_thumb
NexulanPrtgfinal.jpg

The greatest difficulty lies in properly identifying p2p traffic, by which I assume you mean bittorrent. It will just move to any allowed port.

The fact you have it on only one PC does make shaping/prioritizing possible though.

The two ways you can go about it are use the Traffic Shaper Wizard and create a PRIQ shaper, then make a floating rule to place all traffic for that p2p PC's IP in the p2p or low priority queue. If it is the only PC using the internet it would be using all your bandwidth. As other PC's started surfing etc. their traffic would get priority over the p2p PC.

Alternatively setting up a limiter to evenly share the bandwidth among each PC actively using the internet. Only one PC on? It gets all of it. Two? Each gets 1/2 etc. Follow this thread to set it up:
https://forum.pfsense.org/index.php?topic=63531.msg364520#msg364520

You can also combine both methods as well which would grant even more control and likely a better experience for those surfing when a p2p download was occurring.