Ok So I an use a priority queue to guantee the uplink on a single WAN which is good as that is more limited, and dedicate a fixed bandwith to the VOIP on the downlink, which means there is some wasted when no calls are happening, but isn't too bad. If I had 2 boxes, the first with just 2 interfaces, then I could queue both in and out based on destination quiet happily. What about some clever configuration where by all traffic coming in on the wan got routed out of an interfaces with a queue, which just came back in on another spare interface to be then processed as normal. Would that work / have any disadvantages? Clearly would need 2 spare interfaces to do it.

You can't shape ingress traffic, but most traffic is not a DOS and follow rules. UDP traffic is typically fixed bandwidth and will not attempt to fill up your pipe, while TCP will attempt to fill up the pipe, but backs off on packet-loss. In my case, prior to my ISP having an AQM and had a hard cut-off for bandwidth by using the rate limiting built into my ONT which was very strict, setting my LAN interface to about 95% of my bandwidth pretty much kept ping spikes out, which means no buffering on my ISP's side. I could have reduced my bandwidth further and tightened the ping spikes, but way too much diminishing returns. I was already down near 10ms. While 98% link speed resulted in packet-loss and some major ping spikes. That 3% different was pretty big. My point is TCP is pretty good at responding to congestion. Latency is a big issue. My tests were primarily against busty traffic like speedtests or youtube, which I had between 10ms and 20ms. If the sender is further away, like 200ms, it will take that much longer for the packet-loss signal to reach them. It really depends on your typical use cases.

I have never used the L7 stuff, but just wanted to point out the bug in your XML.

sorry about the confusion. if i set the limiter to 2Mbit/s upload or anything else I get 0.09 Mbit/s Upload but if i take off the limiter in firewall rules i get my full 4 Mbit/s Upload. I suspect something weird is going on here.

I have not been able to get limiters working at all, since 2.1.5 or earlier. Could just be me though… but I would expect SOME life to show in the limiters when most of my other configs work as I assumed they would. Is there a standard practice for enabling debugging or verbose logging? I think I remember something at boot about verbose logging. Is there a debug toggle for ipfw/pf/altq?

No, they are ignored unless you craft your own traffic shaping rules to prioritize the traffic.

Thanks Derelict, will set the limiters and report results.

On your WAN Scheduler Type: fairq Bandwidth: 95% of your maximum. If you have really stable bandwidth, then possibly 98%. If you have very unstable bandwidth, then closer to 80%. Create a default queue, set the length to 4096, check codel. Results may vary. It should keep latency low.

I'll we glad when we can have our Cake and delete it too.

Okay. I will have to test those settings as well.  I saw the other post about Codel with UDP and dropping packets. Maybe that was some of my issue I was having. Will have to test with putting UDP only queues under some other queueing and then using Codel for TCP only queues. There were some complaints of packet loss in some of the games using UDP solely

Limiters + NAT -> broken. Search the fine bugtracker.

With all of the extremes this person is trying to do, it may be better to ask if there is any traffic they want getting through the firewall. I might assume that they only care about HTTP/HTTPS, but even HTTPS may be an issue for them because of filtering. Maybe they should start off with blocking all ports except 80/443.

https://forum.pfsense.org/index.php?topic=63531.0

thanks, i try this.

A major discrepancy of Fair Queueing is per-byte fairness vs per-packet fairness. According to DragonflyBSD's FAIRQ source-code, FAIRQ is per-byte fair… I think. Per-packet can be deceptively unfair, depending on expectations.

Limiting and shaping are two different things.  With a limiter, it is a hard cap regardless of maximum bandwidth.  With a shaper, low-priority queues will get to use all bandwidth until something more important comes along.  This is a simplified explanation of how it really works.

Thanks Deric. Your "1:79, 81:65535" suggestion is what I was looking for.

It's basically the same problem as in here… https://forum.pfsense.org/index.php?topic=91299.0 Sam

My next venture is to learn more about debugging ALTQ so I can see more stats. I am assuming I will need to resort to a full FreeBSD to get more control of the system though. Learning more about FreeBSD is probably a good idea. I bet that the bufferbloat/Codel mailing-list has some great info about how to simulate traffic for testing. I have only played with netperf casually. The few worthwhile tests I ran I did with either multiple sessions of iperf, or multiple sessions of netcat/nc. Netcat is insanely powerful. Every few months since I learned of netcat I am amazed by how powerful the simple tool can be; "netcat host2 port < /dev/zero" to a listening netcat and bam, single stream TCP/UDP traffic. or send a file with the same method. or create a remote shell. That is just the standard uses. There is gns3, but it is Cisco emulation primarily. Good userbase and polished interface. There must be a good, simple simulator, right? Call DARPA maybe? ;)