fq_codel for a Dual WAN



  • Hello,

    I followed this video to get fq_codel working. Unfortunately, this causes my second WAN to stop working. What rule do i need to specify for the WAN2 to keep working?

    Thanks



  • You might want to look into gateway groups.



  • My WANS are of different speeds (one is faster than the other). Hence I have one LAN feeding through one WAN and the GUEST LAN (on different vlan) through WAN2. I was hoping to have two different floating rules.

    Anybody tried codel in a Dual WAN setup?



  • Make two sets of limiters one for each WAN with child queues as shown in the video, make two LAN rules, one for each subnet(network), and create a tag within the rule. You can now make two floating rules as in the video and apply the corresponding gateway, queues and tag in the tagged field and you should be set.



  • I have two set of limiters defined. Can you elaborate on "make two LAN rules, one for each subnet(network), and create a tag within the rule". How will this look like?

    At the moment i am tagging my VPN traffic with a "No_WAN_Egress" following this. I have a floating rule to block this traffic.

    Can you have more than 1 tag in a rule? Also why is it necessary to use tags?



  • @trumee well you might actually only need to make one additional LAN rule, let's say that is for your Guest LAN, place it below the "anti lockout rule" choose "Network" in source define the subnet used for your Guest LAN i.e. 192.168.10.0 /24 and finally set a tag in advanced options. The tag is important because you would otherwise not be able to discern traffic from your two LAN subnet when you place your floating rules on the WAN side and it is only possible to make one tag per rule. If you are following that video and making pass floating rules then your new floating rule for WAN2 needs to be above the one for WAN1 and remember to set your Guest LAN tag in the tagged field.



  • @bobbenheim said in fq_codel for a Dual WAN:

    The tag is important because you would otherwise not be able to discern traffic from your two LAN subnet when you place your floating rules on the WAN side

    ok, the pfSense book does mention this,

    "Firewall rules are processed after NAT rules, so rules in the outbound direction on a WAN can never match a local/private IP address source if outbound NAT is active on that interface."

    "Using the Tag and Tagged fields, a connection can be marked by an interface tab rule and then matched in the outbound direction on a floating rule. This is a useful way to act on WAN outbound traffic from one specific internal host that could not otherwise be matched due to NAT masking the source. It can also be used similarly for applying shaping outbound on WAN from traffic specifically tagged on the way into the firewall."

    What if I dont care about the traffic source, whether it comes from LAN1 or LAN2. All i want is that everything which is exiting WAN1 should use Wan1Q , and likewise exiting WAN2 should use Wan2Q.

    I took hints from this post and setup 2 floating rules per each WAN. One rule is for direction 'in' and the other is for direction 'out'. There things which i found different from the youtube video:

    • Using Match instead of Pass rule
    • Not using Quick rule

    This is what my floating rule looks like,

    fbd4196e-2522-4794-a836-27a46f9b7c16-image.png

    The limiters were defined such that WAN1 had a upload/dowload speed of 285mbps and WAN2 had a upload/download speed of 90mbps.

    Limiters:
    00001: 285.000 Mbit/s    0 ms burst 0 
    q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
     sched 65537 type FIFO flags 0x0 0 buckets 0 active
    00002: 285.000 Mbit/s    0 ms burst 0 
    q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
     sched 65538 type FIFO flags 0x0 0 buckets 0 active
    00003:  90.000 Mbit/s    0 ms burst 0 
    q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
     sched 65539 type FIFO flags 0x0 0 buckets 0 active
    00004:  90.000 Mbit/s    0 ms burst 0 
    q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
     sched 65540 type FIFO flags 0x0 0 buckets 0 active
    
    
    Queues:
    q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
    q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
    q00003  50 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
    q00004  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    

    I did some testing with and without traffic shaping. I found things were as expected with WAN1 however WAN2 had a decrease in upload speed. WAN1 is a DHCP connection while WAN2 is a PPPoe connection.

    This is what i get for WAN1

    WAN1 Without Shaping
    bd827537-c08c-4710-afa2-87007f9ca586-image.png

    WAN1 With Shaping
    00e4c336-e07b-4d33-b7da-baed8bef2d42-image.png

    WAN2 Without Shaping
    ba9357eb-9b99-4d71-acc1-332ce9398d18-image.png

    WAN2 With Shaping
    e94ce559-b7a8-4df8-8f40-4519887f9522-image.png

    There is a massive decrease of upload speed from 109mbps to 52mbps in WAN2. Any idea why is that?



  • @trumee if you do not care about which WAN your LAN is utilizing were back to Gateway groups. How to setup shaping on individual lines in a gateway group i am not sure if it is as simple as setting Gateway in your floating rules to your Gateway group, remember to remove tagging if you try it out. For your speed problem can you try and set "Queue Management Algorithm" to CoDel and "Scheduler" to fq-CoDel in your limiters and keep tail-drop in your queues.



  • @bobbenheim said in fq_codel for a Dual WAN:

    How to setup shaping on individual lines in a gateway group i am not sure if it is as simple as setting Gateway in your floating rules to your Gateway group

    I dont think i can use this since WAN1 and WAN2 have different link speeds.

    @bobbenheim said in fq_codel for a Dual WAN:

    For your speed problem can you try and set "Queue Management Algorithm" to CoDel and "Scheduler" to fq-CoDel in your limiters and keep tail-drop in your queues.

    I changed to these setting for my PPPOE based WAN2. It made no difference.

    /tmp/rules.limiter

    pipe 3 config  bw 90Mb codel target 5ms interval 100ms ecn
    sched 3 config pipe 3 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ecn
    queue 3 config pipe 3 codel target 5ms interval 100ms noecn
     
    
    pipe 4 config  bw 90Mb codel target 5ms interval 100ms ecn
    sched 4 config pipe 4 type fq_codel target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ecn
    queue 4 config pipe 4 codel target 5ms interval 100ms noecn
    

    29735972-8a1b-424f-ba4f-dd9d76160d7f-image.png

    I suspect limiter doesnt work on PPPOE based WANs?



  • @trumee you should be able to choose the interface for WAN1 and WAN2 within your floating rule however if that still apply if you create a gateway group i am unsure of.
    I don't think that PPPoE should be a problem at those speeds, can you try setting the limit at 200 Mbit for the upload. What hardware are you using and which version of Pfsense?



  • @bobbenheim I am using Supermicro C2758 motherboard with newly released pfSense 2.4.5

    With 200mbps, I am now getting the full line speed as shown by speedtest.net (DSL reports shows a decrease in speed though). I dont understand why i need to specify a higher upload speed in Codel though? Also, is Codel actually working now with this artificial upload speed?

    On WAN2

    Without Codel
    ca2c5ead-77f0-4477-8461-466a988b7b4f-image.png

    With Codel
    d95b17bd-10c8-466a-bb58-cda013c41262-image.png
    521588cb-c498-4653-b2c0-001ae9047af8-image.png file:///tmp/with_codel.png

    Maybe i dont need Codel on WAN2.



  • @trumee since you are seeing the same problem with tail drop it is highly unlikely to be a CoDel problem. If you go to Diagnostics/Limiter Info you can see if traffic is going to the queues when you run a speedtest. Can you try a speedtest with the limiter set to 150 Mbps. Have you done any tweaking on Pfsense?



  • I'm experiencing the same problem. Two locations. One is SG-3100, fiber as WAN1, cable as WAN2. Second location SG-1100, cable as WAN1, DSL as WAN2. WAN1 is never affected, it does work as expected. No matter what I do WAN2 speed is affected as soon as I enable limiter/rule. I was not able to find any reason why and no workaround. Both systems are production systems, but now being at home more I will try to replicate that with my Intel Atom platform and see if I can find out why.



  • For sure something is wrong. Not using any gateway groups, I removed that, just simple policy based routing. If both WANs are up one of the WANS will not work properly. If one is down the other one is OK in that case. I tried to reorder rules, no changes. I switched limiters between WAN1 and WAN2 with no changes. If I bring WAN1 down then WAN2 works fine. It looks like a bug and very easy to replicate.



  • @crotechnologies just out of curiosity are any of your WAN's PPPoE?



  • No. I don't think that cable is, fiber is DIA from ATT, and DSL (UVerse) is IPoE. All interfaces are configured with either static IP or DHCP. I might do iperf test today with local server if I have time.



  • @crotechnologies how does your rules look like?



  • I tried so many different combinations and nothing worked. I just disabled rules for WAN2. I give up. I tried iperf on WAN2 and that works just fine. As soon as WAN2 is internet connection it doesn't work. Maybe I'll try if I have a chance to go to one of the sites and try to switch WAN1 and WAN2. I'll make cable WAN1 and DIA Fiber WAN2.



  • @crotechnologies it is impossible for anyone to help if you don't provide any details of what you have done. How does your Rules > LAN and NAT > Outbound looks like when it works and don't?


Log in to reply