Multi WAN - Multi VLAN - LoadBalancer



  • Oh my god…

    I've got a quite nice burden here. Now I've finished to setup all 17 VLAN's, I noticed that it's not that easy to create Firewale Rules to route them over a LoadBalancer. We've got two WAN's which we use in combination in LoadBalancer mode. I'd like to send the traffic directly to the LoadBalancer group. Can someone help me out with defining a proper rule, especially with the destination I need to enter?

    Now I have the problem, that the traffic is going to any VLAN, which shall not be.

    Cheers,
    Szop


  • Netgate Administrator

    Your question is confusing.  :-\

    Your screenshot looks correct. For example traffic from your LAN interface with destination anywhere, port 80 will be sent to the loadbalanced gateway.

    With this set of rules clients on the LAN interface trying to access an internal webserver will not be able to do so. Is that the problem you are referring to?

    Edit: I was pretty sure this was true but it now doesn't appear so.
    Traffic routed to a specific gateway or gateway group can only use that WAN connection. Traffic routed between internal subnets need to use the system routing table (gateway = *).
    Here is what I have done to resolve this problem.

    Create an Alias containing all your local subnets.
    Create a new rule after the 'lan to OPT2 block' but before any load balancing rules:
    Allow traffic with destination 'local subnets alias' via gateway *.

    Steve


  • Netgate Administrator

    As an alternative you could set the destination in the loadbalancing rules as NOT 'local subnet alias'.
    Might be cleaner. You may need a catch-all rule at the end to allow traffic that isn't load balanced.

    See my screenshot. I am using a configuration as in my first reply here.

    Steve




  • Hey,

    thank you for your replies and trying to help me out and make things understandable. I hope I can clear those things a little bit up.

    1. We have a VLAN 103, which is the initial VLAN where we started from (It was once a single LAN and we switched it to VLAN 103). We start to put our clients/devices into different VLAN's step by step from here.

    2. Currently we have 17 VLAN's configured, starting from 192.168.2.0 - 192.168.17.0. Now we are trying to fill the different VLAN's by taking machines out from VLAN 103 and put them into the proper VLAN they belong (for example VLAN 6 for Development deparment)

    3. A Management VLAN 99 for Switches at 192.168.99.0. All switches that we have are in the VLAN 99

    4. Since we isolated the management of the Switches to VLAN 99 at 192.168.99.0, nobody should be able to connect to them from VLAN 103 before a rule is created (allow somthing to VLAN 99 on VLAN 103), but I'm able to connect to the switch from VLAN 103

    5. I assume, that since the destination is *, it allows to go to all VLAN's and therefor reaches the VLAN 99 from VLAN 103.

    6. Maybe I'm having a wrong point of view, but I guess I need to changes the rules to "just allow VLAN 103 traffic on VLAN 103" and so on. I'm getting very confused, sorry

    Cheers,
    Szop


  • Netgate Administrator

    Ah, that's much clearer thanks.  :)
    So you are able to access VLAN99 from VLAN103 and shouldn't be able to.
    I assume VLAN103 is the LAN interface and it's firewall rules are shown in your original post above?

    In pfSense, by default, all traffic is blocked on every interface unless you specifically allow it. The only exception to that is the LAN interface which has the antilockout rule (which cannot be removed) and the 'default LAN to any' rule which you appear to have removed.

    Edit: This seems to be wrong!
    Looking at your rules above you should not be able to access any other interface from LAN. You haven't said what you have named the VLAN99 interface but it doesn't matter. Thus either traffic is somehow being routed via some other device, like a mis-configured switch, or more likely there are firewall states still in the the state table that are still allowing access.
    You can clear the state table in the webgui Diagnostics: States: Reset States:
    Bare in mind this will end all current sessions so you may incur the wrath of your users!  ;)

    Looking again at your rules you may have a problem since you have not allowed access to the pfSense box for local DNS queries to the DNS forwarder if you are using it. I had to specifically allow that for my policy based routing on a wifi interface for example. See screenshot.
    Hmm, I'm unsure as to whether specifying a gateway will block this or not.  :-
    Edit: It doesn't.

    Steve



  • Netgate Administrator

    OK, forget everything I just said!  :-[

    Having run some tests it looks like specifying a gateway does NOT prevent routing to local subnets.
    Hmm, I'm sure that used to be the case, perhaps something changed in 2.0.1.  :-\

    Anyway back to your original question then.
    You need to specify a destination in those firewall rules in order to prevent access to other subnets on local interfaces.

    I still recommend creating an alias of all your local subnets as that can be useful in creating rules.

    Steve



  • afaik specifying a gateway does prevent access to other subnets … atleast it does in all my setups.


  • Netgate Administrator

    Hmm, that's what I thought. Having just disabled all my rules except that pointing the the loadbalacing gateway I am still able to access other local subnets, even after reseting the state table.
    Thanks for replying I felt like I was loosing my mind. I didn't just imagine that then?

    Steve



  • @stephenw10: So you are actually having the same result like me, right? That you are able to connect to other subnets, even when you shouldn't be able to?

    Cheers,
    Szop



  • i just checked at home (running 2.1 build from a couple of weeks ago)
    i have 1WAN but it's connect to once of my work sites over ovpn. (+- 10 remote subnets)

    When specifying a gateway and reset states, i'm unable to access the remote subnets.

    What i did notice:
    while running ping and hitting 'reset states' does not terminate the connection (ie ping continues)
    When stopping the ping, then hitting 'reset states' , i'm unable to restart the ping succesfully (ie dropped by pfsense)

    kind regards


  • Netgate Administrator

    @szop:

    @stephenw10: So you are actually having the same result like me, right? That you are able to connect to other subnets, even when you shouldn't be able to?

    It seems that way yes. I'm just checking my test method.
    I notice that you are using 2.1 while I have 2.0.1. Not sure what if any baring that may have here.

    Steve


  • Netgate Administrator

    I distinctly remember being caught out by this when I first moved to a multiwan setup.
    The problem was difficult to diagnose because it's a routing problem so it doesn't show up in the firewall logs.
    Was this something that changed between 2.0 and 2.0.1, perhaps:

    Fixed handling of routing with unmonitored gateways

    @heper what was the response to the pings with a specified gateway? No route to host?
    You have a slightly different situation because of the VPN addition. Presumably you have static routes in place to the remote subnets?

    Steve



  • no static routes … quagga handles routing.

    uhm i'm at the office and can now confirm your observations ...

    also while pinging to a subnet i shouldn't be able to ping to:

    @102 pass in log quick on em2_vlan30 from any to <negate_networks:11> flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"</negate_networks:11>
    

    i'm pretty sure this is the cause for what's happening.
    see relevant post: http://forum.pfsense.org/index.php/topic,48143.0/topicseen.html


  • Netgate Administrator

    Hmm.

    Heper I think you've nailed it there.

    @http://redmine.pfsense.org/issues/2367:

    The fact the negate policy routing rule isn't shown is bad as it has lead to unintended consequences (ends up passing traffic people don't realize is passed because it's hidden). They should be shown as a grayed out auto-added rule, similar to block private/bogon.

    So there is a hidden rule that passes traffic without telling you!  :o

    In 2.1 there is an option to disable that rule in System: Advanced:. Szop, since you are using 2.1 you can do that.

    I hope this makes it into 2.0.2 if it's released before 2.1. Time to rewrite my rules.  ;)

    Edit: Where/how are you seeing the Negate route rule logged?
    Steve



  • Thanks guys!

    I'll try this out as fast as I can!

    Cheers,
    Szop



  • @stephenw10

    i enabled logging on a rule at the bottom of the ruleset on one of the vlan tabs (any->any|gw:WAN1)

    then in logs they suddenly showed up as pass rules (clicking on the green arrow thingy showed the negate stuff)


  • Netgate Administrator

    Ah! So it's actually being caught by the existing rule but ignoring the gateway since the destination is in the negate_networks table.
    If you're at your 2.1 machine maybe you can confirm the option to disable negating.

    Steve



  • Hey guys,

    thanks for clearing things up and helping me out! I owe you one :)! I can confirm that after activating the "negate policy routing" disabled all access from VLAN's to VLAN's. This is what I was looking for. Now I'll need to modifiy my rules and set up the gateways :).

    Cheers,
    Szop


  • Netgate Administrator

    Ah good to know and thanks for confirming.  :)
    What would be useful would be to be able use some of the system "aliases" in firewall rules. For example use Private_networks or Negate_networks. As it is I have an alias I setup myself, LOCAL, but I have to remember to update it if I change anything and it doesn't include the WAN IP (though could it?). Negate_networks does that automagically.

    Steve


Log in to reply