Multi Wan 95% percentile bandwidth limiter
-
Let us know if it actually works. Have fun!
-
I figured I could try shaping the downlink, as Harvy suggested, even tho I'm not currently load-balancing the Wan's. Ran into a couple snags. I added a Lan queue with two child queues for Wan1 & Wan2, see attached. Then I addded a couple of floating Wan match out rules each to one of the Lan queues.
Issues;
PfSense warns one of the child queues should be default. If not the Lan queue becomes default by default. Every time a rule is modified it keeps warning me of this which is annoying. If a default is not selected the Status, Queues doesn't show info for the Lan adapter. If a child queue is set to default then the Status, Queues works again but all matches are with the default queue and nothing on the other child queue.Did I miss something?
-
Has any of you guys ever managed to get this setup to work ?
I have a test setup with 3x WAN ADSL 18M/1M.
I've setup a CODEL queue on each WAN to limit upload. Works !
I've setup a HFSC queue on LAN with 3 sub queues to limit download.
pfSense then complains that there is no default queue and sets the parent queue as default.
As soon as I setup the floating rules per WAN, direction out, with the sub queues, LAN machines no longer have internet.
i've retried this multiple times without success.Btw, why must the floating rules on WAN be direction out when the queues are supposed to limit download (in) speed ?
I'd be glad for any advice.
-
The floating rules match a state-pair. If you can at least match the state on the outgoing, then you also get the incoming. At least this has been my experience.
If you're not sure which WAN interface your traffic is going to go out AT THE TIME THE LAN STATE IS CREATED, then you need to use a floating rule to capture at the time the WAN state is created.
While you may have rules that will make sure traffic always get assigned to a queue, FreeBSD does not care. It wants a default queue no matter what. Keeps things safe and simple. Especially seeing that 99% of all issues are caused by the end user, you can't trust the end user to do anything right. Defensive programming.
Not sure about losing the Internet. Can we see your floating rules, and your LAN queues?
-
Thanks Harvy66, I actually messed up my fist tests by using "pass" instead of "match" floating rules.
I've done two full working test setups, with 2 fiber WANs, and with 3 ADSL WANs, including high priority ICMP queues to keep dpinger happy even on high latency lines.
I'll probably write a howto shortly about this.
Anyway, your idea is brilliant and yet simple.Thanks a lot.
-
Thanks Harvy66, I actually messed up my fist tests by using "pass" instead of "match" floating rules.
I've done two full working test setups, with 2 fiber WANs, and with 3 ADSL WANs, including high priority ICMP queues to keep dpinger happy even on high latency lines.
I'll probably write a howto shortly about this.
Anyway, your idea is brilliant and yet simple.Thanks a lot.
Why prioritize pings?
Usually, pings should be prioritized as standard traffic so that it's latency measurements correspond with regular traffic. If you prioritize pings the measured latency may go unchanged during link-saturation, limiting dpinger's diagnostic usefulness. -
No need to add ICMP priority on SDSL lines, but those ADSL lines I have must be pretty bad and dpinger sat them offline like every 5 minutes, even when I increased alarm threshold.
I actually have a probably worse problem:
The 3 ADSL links I have a are 18Mb/1Mb links. I've limited them to 16Mb/850Kb using traffic shaper.
When a computer is directly connected on those ADSL lines, I get 19Mb/900Kb on speedtest.net.
LAN traffic is about 10Mb/600Kb, but still, I get really bad packet loss / high latency when keeping ICMP as standard traffic.I'm wondering what's the problem, but in the meantime, I've setup high ICMP priority to be able to keep the gateway online and be able to remotely connect.
-
No need to add ICMP priority on SDSL lines, but those ADSL lines I have must be pretty bad and dpinger sat them offline like every 5 minutes, even when I increased alarm threshold.
I actually have a probably worse problem:
The 3 ADSL links I have a are 18Mb/1Mb links. I've limited them to 16Mb/850Kb using traffic shaper.
When a computer is directly connected on those ADSL lines, I get 19Mb/900Kb on speedtest.net.
LAN traffic is about 10Mb/600Kb, but still, I get really bad packet loss / high latency when keeping ICMP as standard traffic.I'm wondering what's the problem, but in the meantime, I've setup high ICMP priority to be able to keep the gateway online and be able to remotely connect.
When you say you have packet loss, are you referring to actual packet loss or dpinger packet loss?
Do these lines show packet loss at all times or only during link saturation?
-
Talking about dpinger packet loss. Gateway monitors IP are set to google DNS and opendns.
Happens without link being saturated (max is about 8Mb/600Kb on 18Mb/1Mb links). -
Talking about dpinger packet loss. Gateway monitors IP are set to google DNS and opendns.
Happens without link being saturated (max is about 8Mb/600Kb on 18Mb/1Mb links).Hmm… I would not expect prioritization to change anything if packets are randomly dropping regardless of link saturation.
-
Well it does ! Having ICMP traffic with high priority, I don't get packet loss on dpinger even if I probably have a lot of packet loss on the link itself (at least the link does not get shot every minutes or so by dpinger, unless it's completly down).
Btw, lines themselves don't suffer packet loss.It's kind of a mystery for me here. I think of two solutions:
1. Too much concurrent connections -> line quality drops ?
2. pfSense related misconfig…I'll check this next days again, brain is off for today.
Thanks anyway. -
Multi-WAN single LAN HFSC names queue thing seems to be working? It would be cool if there was some way to get some basic statistics. Like all 3 links loaded(download and upload) and maintaining a relatively stable ping.
-
Well it does ! Having ICMP traffic with high priority, I don't get packet loss on dpinger even if I probably have a lot of packet loss on the link itself (at least the link does not get shot every minutes or so by dpinger, unless it's completly down).
Btw, lines themselves don't suffer packet loss.It's kind of a mystery for me here. I think of two solutions:
1. Too much concurrent connections -> line quality drops ?
2. pfSense related misconfig…I'll check this next days again, brain is off for today.
Thanks anyway.So you have practically no ATM frame loss but you are constantly dropping IP packets regardless of link utilization?
I doubt the line cares about the number of connections. Maybe packets per second is your problem?
-
As I am trying to digest this solution, does anyone have a set of screen shots to guide the setup?
Also, does this solution achieve any improved load balancing across the available WANs or it is simply a method to maximize, with limits, the use of each WAN?
Thanks, -
I've wrote a quick tutorial from my multi WAN traffic shaper experience here: https://forum.pfsense.org/index.php?topic=120380
Any improvements are welcome !And hey, thank you Harvy66 for your solution !
@Nullity: There's still some serious packet loss going on. You thought of maybe too much packets. Is there a rule of thumb for the packet number / bandwidth ?