fq_codel for a Dual WAN
I followed this video to get fq_codel working. Unfortunately, this causes my second WAN to stop working. What rule do i need to specify for the WAN2 to keep working?
You might want to look into gateway groups.
My WANS are of different speeds (one is faster than the other). Hence I have one LAN feeding through one WAN and the GUEST LAN (on different vlan) through WAN2. I was hoping to have two different floating rules.
Anybody tried codel in a Dual WAN setup?
Make two sets of limiters one for each WAN with child queues as shown in the video, make two LAN rules, one for each subnet(network), and create a tag within the rule. You can now make two floating rules as in the video and apply the corresponding gateway, queues and tag in the tagged field and you should be set.
I have two set of limiters defined. Can you elaborate on "make two LAN rules, one for each subnet(network), and create a tag within the rule". How will this look like?
At the moment i am tagging my VPN traffic with a "No_WAN_Egress" following this. I have a floating rule to block this traffic.
Can you have more than 1 tag in a rule? Also why is it necessary to use tags?
@trumee well you might actually only need to make one additional LAN rule, let's say that is for your Guest LAN, place it below the "anti lockout rule" choose "Network" in source define the subnet used for your Guest LAN i.e. 192.168.10.0 /24 and finally set a tag in advanced options. The tag is important because you would otherwise not be able to discern traffic from your two LAN subnet when you place your floating rules on the WAN side and it is only possible to make one tag per rule. If you are following that video and making pass floating rules then your new floating rule for WAN2 needs to be above the one for WAN1 and remember to set your Guest LAN tag in the tagged field.
The tag is important because you would otherwise not be able to discern traffic from your two LAN subnet when you place your floating rules on the WAN side
ok, the pfSense book does mention this,
"Firewall rules are processed after NAT rules, so rules in the outbound direction on a WAN can never match a local/private IP address source if outbound NAT is active on that interface."
"Using the Tag and Tagged fields, a connection can be marked by an interface tab rule and then matched in the outbound direction on a floating rule. This is a useful way to act on WAN outbound traffic from one specific internal host that could not otherwise be matched due to NAT masking the source. It can also be used similarly for applying shaping outbound on WAN from traffic specifically tagged on the way into the firewall."
What if I dont care about the traffic source, whether it comes from LAN1 or LAN2. All i want is that everything which is exiting WAN1 should use Wan1Q , and likewise exiting WAN2 should use Wan2Q.
I took hints from this post and setup 2 floating rules per each WAN. One rule is for direction 'in' and the other is for direction 'out'. There things which i found different from the youtube video:
- Using Match instead of Pass rule
- Not using Quick rule
This is what my floating rule looks like,
The limiters were defined such that WAN1 had a upload/dowload speed of 285mbps and WAN2 had a upload/download speed of 90mbps.
Limiters: 00001: 285.000 Mbit/s 0 ms burst 0 q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: 285.000 Mbit/s 0 ms burst 0 q131074 50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail sched 65538 type FIFO flags 0x0 0 buckets 0 active 00003: 90.000 Mbit/s 0 ms burst 0 q131075 50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail sched 65539 type FIFO flags 0x0 0 buckets 0 active 00004: 90.000 Mbit/s 0 ms burst 0 q131076 50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail sched 65540 type FIFO flags 0x0 0 buckets 0 active Queues: q00001 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail q00002 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail q00003 50 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail q00004 50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
I did some testing with and without traffic shaping. I found things were as expected with WAN1 however WAN2 had a decrease in upload speed. WAN1 is a DHCP connection while WAN2 is a PPPoe connection.
This is what i get for WAN1
WAN1 Without Shaping
WAN1 With Shaping
WAN2 Without Shaping
WAN2 With Shaping
There is a massive decrease of upload speed from 109mbps to 52mbps in WAN2. Any idea why is that?
@trumee if you do not care about which WAN your LAN is utilizing were back to Gateway groups. How to setup shaping on individual lines in a gateway group i am not sure if it is as simple as setting Gateway in your floating rules to your Gateway group, remember to remove tagging if you try it out. For your speed problem can you try and set "Queue Management Algorithm" to CoDel and "Scheduler" to fq-CoDel in your limiters and keep tail-drop in your queues.
How to setup shaping on individual lines in a gateway group i am not sure if it is as simple as setting Gateway in your floating rules to your Gateway group
I dont think i can use this since WAN1 and WAN2 have different link speeds.
For your speed problem can you try and set "Queue Management Algorithm" to CoDel and "Scheduler" to fq-CoDel in your limiters and keep tail-drop in your queues.
I changed to these setting for my PPPOE based WAN2. It made no difference.
pipe 3 config bw 90Mb codel target 5ms interval 100ms ecn sched 3 config pipe 3 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ecn queue 3 config pipe 3 codel target 5ms interval 100ms noecn pipe 4 config bw 90Mb codel target 5ms interval 100ms ecn sched 4 config pipe 4 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ecn queue 4 config pipe 4 codel target 5ms interval 100ms noecn
I suspect limiter doesnt work on PPPOE based WANs?
@trumee you should be able to choose the interface for WAN1 and WAN2 within your floating rule however if that still apply if you create a gateway group i am unsure of.
I don't think that PPPoE should be a problem at those speeds, can you try setting the limit at 200 Mbit for the upload. What hardware are you using and which version of Pfsense?
With 200mbps, I am now getting the full line speed as shown by speedtest.net (DSL reports shows a decrease in speed though). I dont understand why i need to specify a higher upload speed in Codel though? Also, is Codel actually working now with this artificial upload speed?
Maybe i dont need Codel on WAN2.
@trumee since you are seeing the same problem with tail drop it is highly unlikely to be a CoDel problem. If you go to Diagnostics/Limiter Info you can see if traffic is going to the queues when you run a speedtest. Can you try a speedtest with the limiter set to 150 Mbps. Have you done any tweaking on Pfsense?
I'm experiencing the same problem. Two locations. One is SG-3100, fiber as WAN1, cable as WAN2. Second location SG-1100, cable as WAN1, DSL as WAN2. WAN1 is never affected, it does work as expected. No matter what I do WAN2 speed is affected as soon as I enable limiter/rule. I was not able to find any reason why and no workaround. Both systems are production systems, but now being at home more I will try to replicate that with my Intel Atom platform and see if I can find out why.
For sure something is wrong. Not using any gateway groups, I removed that, just simple policy based routing. If both WANs are up one of the WANS will not work properly. If one is down the other one is OK in that case. I tried to reorder rules, no changes. I switched limiters between WAN1 and WAN2 with no changes. If I bring WAN1 down then WAN2 works fine. It looks like a bug and very easy to replicate.
@crotechnologies just out of curiosity are any of your WAN's PPPoE?
No. I don't think that cable is, fiber is DIA from ATT, and DSL (UVerse) is IPoE. All interfaces are configured with either static IP or DHCP. I might do iperf test today with local server if I have time.
@crotechnologies how does your rules look like?
I tried so many different combinations and nothing worked. I just disabled rules for WAN2. I give up. I tried iperf on WAN2 and that works just fine. As soon as WAN2 is internet connection it doesn't work. Maybe I'll try if I have a chance to go to one of the sites and try to switch WAN1 and WAN2. I'll make cable WAN1 and DIA Fiber WAN2.
@crotechnologies it is impossible for anyone to help if you don't provide any details of what you have done. How does your Rules > LAN and NAT > Outbound looks like when it works and don't?
I have a similar problem. I have 1 500/100 Mbit/s Fiber thru vlan connection and a floating rule for limiters with interface wan and wan gateway, match, out.. I'm using this wan gateway in every out connection rule except one where I use a vpn gateway. As soon as I connect the VPN, my WAN only does 50 ish mbit/s upload. If I disable the floating rule, it returns to normal. If I disable the VPN, with floating ruleon, it returns to normal.
built on Wed May 15 18:53:44 EDT 2019
Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
4 CPUs: 1 package(s) x 4 core(s)
EDIT: Forgot to say that I solved my problem adding the queues in the rules and not using the floating rule.