https://forum.pfsense.org/index.php?topic=126222.0

Figured it out to satisfaction. I will generalize steps below.

Traffic Shaper->Create WAN shaper type CBQ with ~95% of WAN upload bandwidth
Create WAN_OUT queue, priority 1, set as default queue and allow borrowing from other queues
Create DMZNET_OUT queue, priority 2, set as required, allow borrowing from other queues.

On the DMZNET out rule (for IPSec tunnel) edit the rule, go to advanced and set DMZNET_OUT as the queue.

Reset states.

Test by doing various iperf3 tests and watching queue status

I think one way to solve this problem might be to use limiters with multiple queues and weights on the queues to limit the amount of bandwidth one machine can consume.

For example, assuming you had two machines, you could use limiters and create two queues (let's call them queueA and queueB) under your upload and download limiter and assign each a weight of 50.  Then create the necessary firewall rules to pass traffic from machine 1 through queueA and and traffic from machine 2 through queueB.  This should ensure that each machine will get a least 50% of the bandwidth and more if the other machine is just idle.  I've got a similar setup to this, but it's done by subnet rather than by machine/host (i.e. to ensure each individual subnet gets at least a certain % of bandwidth if the connection is loaded down).

A further option would be to use multiple limiters with multiple queues.  For instance you could create an upload and download limiter for machine A and an upload and download limiter for machine B and limit the bandwidth of each set of limiters to 50% of your connection speed.  Then underneath those limiters you could create queues to prioritize your traffic.  While this allows for easier prioritization of traffic in multiple queues, the downside is that machine A and B will see no more than 50% of your bandwidth vs. the full bandwidth if the other machine is idle.  That would be the tradeoff.  Note that you can still use weights on your queues in this approach as well if you wanted to guarantee bandwidth on certain types of traffic on a given host (e.g. P2P vs. HTTP, etc.)

I realize there are limitations to these approaches, and if you have many machines then it's probably not practical.  There might be more elegant solutions out there, but unfortunately I'm not aware of an easier way to share bandwidth equally between hosts whose traffic all goes through the same queue (vs. setting up multiple queues).

Hope this helps.

Assuming an infinitely long running TCP connection, the only way managing ACKs affects the sender's rate is to delay the ACK to artificially create a larger RTT, which is a horrible idea, mark the ACK via ECN, or drop an incoming packet causing the sender to re-send and backoff.

In practice and especially with more modern TCP stacks, dropping ACKs does not affect peak bandwidth, only the rate at which the bandwidth grows.

Dropping or ECN marking data packets is the official way to signal the sender to back-off.

Like one that matches your NAS traffic? I would guess a floating rule at the end of your rules for outgoing where the source IP if your NAS and the destination port is 443?

I should elaborate, the topology is:

Nextiva cloud service
Charter consumer grade modem
Netgate SG-2440 with 2.4.1
Apple airport for wifi (bridged, naturally, with the SG-2440)
[6-12 computers and smartphones, all via wifi]
TGP-500 Panasonic SIP phone connected directly to the Apple airport as if it were a switch
[TEC wireless for the SIP phones]

The SIP phones are performing fine, and have for a year.
It is the smart phone on the Win 10 laptop, via wifi to the airport, bridged to the SG-2440, that has the crappy outbound audio

PRIQ does not require bandwidth settings since it is purely priority-based.  Under a PRIQ scheme, there shouldn't be any inter-LAN slowness unless you have major congestion & contention going on.  From the pfSense Book:

Priority Queuing (PRIQ)

PRIQ is one of the easiest disciplines to configure and understand. The queues are all directly under the root queue,
there is no structure to have queues under other queues with PRIQ as there is with HFSC and CBQ. It does not care
about bandwidth on interfaces, only the priority of the queues. The values for priority go from 0 to 15, and the higher
the priority number, the more likely the queue is to have its packets processed.

PRIQ can be harsh to lesser queues, starving them when the higher priority queues need the bandwidth. In extreme
cases, it is possible for a lower priority queue to have little or no packets handled if the higher priority queues are
consuming all available resources.

On my setup where I use PRIQ for our voip phones, I have LAN bandwidth set to 850 Mbit/s, WAN set to 85 Mbit/s because the wizard makes you input those even if they're not used by your chosen shaper.  My shaping rules float on WAN.

Try limiters with fq_codel in 2.4

https://forum.pfsense.org/index.php?topic=126637.120

Do as Johnpoz did in post 121, it will give the full bandwidth if the network is quiet but will split when needed + reducing bufferbloat

I googled around the forums and there are some similar projects people have tackled around billing and usage reporting.

I'm not sure your specific case has been addressed but it's definitely doable with a combination of a custom script or two, cron, and one of the addon packages.

I ended up switching to a PRIQ setup instead.  I limit the upload on my WAN slightly so I don't saturate my uplink.  I then set my LAN bandwidth to 980 Mb/s and squid then flows at almost full interface speed.  It would be nice to be able to depriortize squid but for now at least it works.

It was removed because it is broken in dummynet and does not operate as expected. It never did work, as evidenced by all the complaints in this thread.

It isn't viable, so unless it gets changed upstream in FreeBSD, it won't be coming back.

Bounty suspended as belt9 has been circumventing a ban. Those interested may submit a new bounty. Considering his attitude we cannot take the responsibility if he will deliver the funds to the solution.

No.  I do have 3 VLANs setup on it.

you would do that in squid..

Changing scan mode to hyperscan also fixed suricata with traffic shaper