problems with flexible limiters set using floating rules



  • Hi,

    I'm observing serious problems with flexible limiters set using floating rules. Let me start from the beginning:
    In case the flexible limiter is configured/attached to the LAN-side interface rules, everything works perfectly.
    This, however, can't be used in case you have a multiWAN environment and the gateway for the LAN-side rule is a gateway group (assuming your two WANs have different speeds). In this sencario the same flexible limiter would used in case your main WAN works and in case your backupWAN takes over. Assuming your WANs have different speeds, you either have to set your limiter to the values of your slower WAN (awful solution!) or to the values of your faster WAN (which makes the limiter useless when the slower WAN is the only one available).
    A logical solution would be set the limiter in a floating rule (type = match, direction = out, IF=WAN, GW=WAN_GW) and a second floating rule for the other WAN. The problem is if you set it up like this the download limiter works fine but the upload limiter does not - it blocks the traffic almost completely.
    For example: a 3/5 Mbit limiter gives a 0,2/5Mbit internet access....
    I'm on the 2.4.4_2 so basically latest and greatest. I think I tried everything, including all the possible reconfigurations that make sense as well as all those that do not :) Still I can't make this work...

    This definitely seems like a bug. Does anyone have limiters working in case of a mutliWAN scenario?

    Brg,
    Adam



  • I found a workaround, i.e. I've set up floating rules (direction = in; attached to LAN interfaces; GW = GW1 for one rule and GW2 for the other rule) for those flexible limiters and this works fine.
    However, this doesn't change the fact that in case of 'out' floating rules on WAN interfaces, the 'up' limiter does not work - still looks like a bug to me.



  • Would you mind sharing your configuration forwhat is working ? I cannot seem to get limiters working on Floating rules while they work fine on rules attached to the interface (LAN).



  • Sure. There are two floating rules to match the traffic:
    0_1550553841096_58ce497d-8cb1-45bf-be9b-b461cef39a66-image.png

    They both have:

    • type = match
    • direction = in
    • interfce = all the LAN side interfaces
    • source = any
    • destination = not private networks

    The only difference is that one rule has GW1 and the second one has GW2 and of course one has in/out pipes based on ISP1 speeds and the other one has limiters based on ISP2 speed.



  • Hello All,

    Sorry but this configuration does not work . Traffic will be always limited by the limiter defined in tha last rule. Which is normal as in Floating rules , PF will begin by the last rules !! I Have the same config as you .

    So when a gateway Tier 1 fails the traffic is correctly switch to an another gateway but still pipe with the same limiter

    a08d2c0f-8810-4717-a5c6-aded0ab6d932-image.png

    An idea ?
    Thanks



  • @manu77 I'm afraid you are right. I have tested this a couple of times since February, I also had a change in the speeds of my WAN links and I can confirm that this setup is indeed no workaround....
    If anyone has any idea how to properly configure flexible limiters in a multiWAN environment, I'm all ears....



  • Hi Adam ,
    thanks for your confirmation ... i was going crazy ! :)

    I think our pb referred to this :
    df92a90a-dbdd-43cf-a24f-6ed8a76db730-image.png

    A post of it : https://forum.netgate.com/topic/145454/how-do-i-configure-limiters-on-a-multi-wan-setup/3

    But for me the gateway in the rule is never omitted or if sticked the rules is never recreated proprely!!!

    My test is easy :
    link 1 (firewall 1 ) - 1M
    link 2 ( firewall 2) - 5M
    gateway group with :
    link1 is Tier1
    link2 is tier 2

    So when the link 1 is down I should have a bandwith of 5M. But I 'm still have 1M

    I have tested in floating rules here

    e3a3905d-e55a-4777-a18a-a86063180b3a-image.png

    here l have still to 1M after link1 is down

    So i supposed floating weren't concerned by this I put directly the setup in the lan Rules

    0dfd901a-6c67-4cc6-9610-fdea0df46024-image.png

    Here the failover doesn't work at all ! as the firewall still put packets in the first rule it meet without omitting the rules as it should because my setup for it is :

    55ca8baf-a168-45a1-9b2e-ce29b903e617-image.png

    nothing works in fact !

    Big frustation !



  • @manu77 A couple of things here:

    1. when you do your LAN rules, have you tried using gateway group? Your setup seems fine and it is indeed very strange that the packets are still being pushed to GW1 when it's down (you have the 'skip rules when gw down' checked!) but maybe gateway group would do some difference? If not than this 'skip rules (...)' setting is absolutely and totally useless.....
    2. the goal, at least for me, is not to have a mainGW/backupGW solution but two gateways running and both having a flexible limiter. I'm directing some traffic to GW1 (with GW2 as backup) and some other traffic to GW2 (with GW1 as backup)
    3. you're on the latest&greatest version, right?


  • 1 . when you do your LAN rules, have you tried using gateway group? Your setup seems fine and it is indeed very strange that the packets are still being pushed to GW1 when it's down (you have the 'skip rules when gw down' checked!) but maybe gateway group would do some difference? If not than this 'skip rules (...)' setting is absolutely and totally useless.....*

    doing this config return to use floating rules without any gateway fixed in LAN rules because I fixed the default gateway in routing config here
    4fe8d3a6-6989-4ba6-b01b-7ed945c4eb6a-image.png

    Moreover, 'skip rules when gw down' is not efficient on a gateway group adn it makes sense in fact ! ( see this it is very interessant https://forum.netgate.com/topic/133099/required-number-of-gateway-groups/2

    for your point2 , you can acheive this with 2 groups with gateway tier inverted. But after, you have still the pb to affect the limiter ... like me :

    1. My version is the last as I know !
      07700730-576e-40bd-8206-42ea2742c221-image.png


  • @manu77 That is exactly my point. I absolutely have to use gateway groups and 'skip rules (...)' does not apply to them so this 'skip rules (...)' option will definitely never help me achieve a mulitWAN+flexible_limiter setup.
    It also doesn't help you achieve a mainWAN/backupWAN+flexible_limiter setup either ;)
    So basically we're back to nothing.... I still think floating rules for 'out' direction would be best but as I wrote on 14th of February, this doesn't work and I still stand by the opinion that it's a bug.



  • Adam,
    Sorry to break your dreams but catching packet with Floating on Wan Out is not possible !
    It is not a bug .
    You cannot get a packet only on outgoing direction ( except for a packet send by PF itself) because the packet is always handled by the ingoing interface and never by the Out interface. When the packet arrives , PF can decide what it will do with it . It is the unique moment for a router to take a decision and action regarding a packet.
    When the packet leaves the int IN to its destination, it will pass through the int OUT for instance , but at this step it is too late for treatment.



  • @manu77 If that is the case, if there is no treatment on the egress interface (no buckets there, no option to drop selectively based on pre-NATed source IP etc.) then this would expain why limiters on out_floating_rules behave like they are drunk (although they should not work at all instead of leaving 0,2Mbit/s like in my tests) ... Either way, if you ever find out how to make a proper QOS/limiters in a multiWAN PFS environment, please post it here - to me this is one of the two things that I lack in PFS (the other being parallel IPSEC tunnels to the same remote network range) but for sure it's the most important one....



  • I have this problem as well, with my multi-wan setup, but only with one of the wan's. WanUp and WanDown limiters works perfectly, Wan2 limiters have the problem described here - download works fine, but upload is limited to about 0.2 Mb. If the above description is accurate, how come the first Wan limiters works fine? I even think the Wan2 limiters worked at first, but stopped working when I was playing with it...but I might be wrong about that. Maybe those never worked.

    33ea40fc-08b2-417f-aff8-3f8ad8f43186-image.png



  • @eriknuds I can't say for sure if the first WAN limiter worked fine in my setup. I just don't remember... You have clearly encountered the exact same issue because you ran into the 0,2Mbit/s problem.


  • LAYER 8 Netgate

    @AdamL said in problems with flexible limiters set using floating rules:

    I found a workaround, i.e. I've set up floating rules (direction = in; attached to LAN interfaces; GW = GW1 for one rule and GW2 for the other rule) for those flexible limiters and this works fine.
    However, this doesn't change the fact that in case of 'out' floating rules on WAN interfaces, the 'up' limiter does not work - still looks like a bug to me.

    You do know that the limiter up/down direction reverses on an out floating rule right?


  • LAYER 8 Netgate

    You can also mark the connections made by hosts on LAN based on their inside address and match that mark as it leaves WAN using an outbound floating rule that sets a limiter there.



  • @Derelict said in problems with flexible limiters set using floating rules:

    @AdamL said in problems with flexible limiters set using floating rules:

    I found a workaround, i.e. I've set up floating rules (direction = in; attached to LAN interfaces; GW = GW1 for one rule and GW2 for the other rule) for those flexible limiters and this works fine.
    However, this doesn't change the fact that in case of 'out' floating rules on WAN interfaces, the 'up' limiter does not work - still looks like a bug to me.

    You do know that the limiter in/out direction reverses on an out floating rule right?

    Sure. When I say up or down I mean the actual direction from the 'user's perspective'.



  • @Derelict said in problems with flexible limiters set using floating rules:

    You can also mark the connections made by hosts on LAN based on their inside address and match that mark as it leaves WAN using an outbound floating rule that sets a limiter there.

    That is an interesting thought. So you mean staying with the concept of outbound floating rules with limiters but matching them not only to the WAN interface but also to some 'marks' set by LAN rules?


  • LAYER 8 Netgate

    If that will solve the problem.



  • @Derelict I will definitely test this out. Thanks!



  • I now tested with PIE and FQ_PIE, and I tested with limits above what the line can normally do (it's a WISP conection - both are actually...) and it doesn't happen anymore. I have the same firewall rules. Yes I know to reverse the queues for in/out wan rules, and I have the same setup for the other wan rules which worked all the time and I haven't changed the rules now that they work with PIE/FQ_PIE...really weird because yesterday I tried other settings than PIE and codel and none worked...I also did a state reset between the tests, not just making new connections, but it didn't fix the issue yesterday.



  • @eriknuds And what about flexible limiter? Honestly Qos(queuing) is not so important to me. Flexible limiter is...



  • Yes, it's set up as flixible now with masks on the queues and not the limiter, and it seems to work fine...



  • @eriknuds
    Hello

    Very interessant . But I don't see exactly your configuration.
    Could you please send us screenshots for :
    1 -Rules in LAN ( list view) and marking options in Rules you chose -> I m curious to see how you say to PF to mark the packet properply with two different possible gateway
    2 - Rules in Floating ( list view) and options in Rule for matching traffic -> I'm also curious to know how you match packet with 'out' direction on this step
    3 - Options chosen at this step bellow

    4a7c96b7-826f-4787-a468-43951d64a030-image.png

    thanks a lot



  • @manu77

    FW Rules:
    d439db35-9b1d-4963-a5fb-c3d2719161c4-image.png

    The gateway is the gateway group (Loadbalance) in all the rules. To test each wan connection separately I just select another Tier in the gateway group so only one gateway is used.

    I only have the 4 floating match rules related to Limiters/queues. None for the LAN interface.

    5984d182-e2bb-4718-ba9d-49f7c5c76666-image.png

    Not sure if I have done everything right, but it seems to isolate the traffic and not disturb other hosts even though I exhaust the line with speed checking...and the isolation is really all I need. AQM etc is not a requirement. My wan connections are pretty symmetric, though not very high bandwith, WISP connections. But I would really like triple isolation like in CAKE. It really sucks that OpenWRT have had CAKE functionality for so long and pfsense seem to be no closer to getting it.



  • @eriknuds
    Thanks to take time for showing us your conf. I will test it and tell you .
    I've a lab here with 6 firewalls to emulate multiwan. so we will see.



  • @manu77

    Great stuff, let me know if you need more details:-)

    Regards,
    Erik Knudsen



  • Hello All,

    I confirm this configuration works and works well. Each time the gateway changes, the Pipe is well affected too with 10 secondes of floating bandwitdth ( no traffic )
    Now I must go further to see how to add specific traffic in a specific queue and described from WAN ! because the floating rules for this test are set up as you post , I mean from * to *

    nice day



  • @manu77 ,

    I have selected the appropriate wan interface in each rule (in-rule and out-rule for each wan interface) - in the WanIn/Out rules I have selected only the wan interface, and in the corresponding wan2 rules I have selected only the wan2 interface.

    Good luck with any further testing:-)


Log in to reply