Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    LAN with multiple subnets

    Scheduled Pinned Locked Moved Routing and Multi WAN
    17 Posts 3 Posters 5.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      Raiker
      last edited by

      I'm not sure if this is setup correctly or if i'm experiencing a bug:

      Setup

      pfSense 2.2.2
      VMWare ESXi 6.0, e1000 adapters

      1 LAN interface
      4 WAN interfaces

      Lan interface has 10.1.0.0/16 range, which i'm slowly migrating to various /24 subnets (10.2.1.0/24 10.2.2.0/24, etc) and for them to have internet routing i had to add both a routing entry for the subnet (adding a gateway for the LAN interface) and rules in the firewall.
      Now, the problem i have is that from time to time something gets screwed and it seems like pfSense makes LANGW the default gw! (attempting to ping internet results in ttl expired, a trace shows the packet bouncing between the switch, which is the inter-subnet-router and the pfsense box). Also during this time pfSense's GUI becomes unresponsive, ping works and local console also works.

      I really don't want to have to add an interface for each subnet nor have to add vlan interfaces on pfSense.

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        I really don't want to have to add an interface for each subnet

        What you say? A gateway for the LAN interface? An interface for each subnet?  What a novel idea!

        Draw a picture.

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • johnpozJ
          johnpoz LAYER 8 Global Moderator
          last edited by

          "I really don't want to have to add an interface for each subnet nor have to add vlan interfaces on pfSense."

          What??  How are you going to have different subnet without an interface be it physical/virtual or a vlan sitting on a interface?

          As to gateways??  You don't set gateways on pfsense lan interfaces..  It becomes a wan connection then, when you set a gateway on an interface your saying, hey if you want to get off your own network go to this IP.  that is not something a lan interface in pfsense does.  It is used for pfsense to talk to that segment, and to get traffic from that segment that it routes out its WAN interface(es) or its other lan interfaces if dest is directly connected to it.

          if you have a downstream router connected to a lan interface, then you create route(s) to that down stream router pfsense should use to get to those networks.  Not a gateway on the interface.

          Only thing that should have a "gateway" on your lan should be your devices - pointing to pfsense to get off that network, ie other lan segments or wan/internet via pfsense.

          If you don't want vlans or interfaces than just use your 10.1/16 – which come on dude really how many nodes to you have that you need a /16 JFC... 65K addresses, are you anywhere close to that??

          An intelligent man is sometimes forced to be drunk to spend time with his fools
          If you get confused: Listen to the Music Play
          Please don't Chat/PM me for help, unless mod related
          SG-4860 24.11 | Lab VMs 2.8, 24.11

          1 Reply Last reply Reply Quote 0
          • R
            Raiker
            last edited by

            I'm not using a lot of addresses. what i'm doing is having separate collision domains for each part of the company.
            We have several labs, some customer machines housed (which have their own set of subnets) and of course visitors get their own vlan where they can get to the internet but not the other subnets.

            I'm not sure a picture is needed, i have a Dell 5548 switch with 3 units stacked. that switch is the default gateway for each and every vlan. The switch has it's own default gateway set to the pfSense box.
            But if pfSense doesn't have routes loaded for every subnet it doesn't know how to reach them AND the only way i've found to add said routes is under:
            System->Routing->Routes, to add a route there i HAVE to create a gateway, it doesn't allow me to just set the next hop like it should, but forces me to create a GW.

            I understand that this should be as simple as adding a route for 10.2.1.0/24 via 10.1.255.254 but i haven't found it on the gui (or to be fair, it is there but it forces the creation of a GW)

            This part of the comment is exactly what i'm trying to do:
            "if you have a downstream router connected to a lan interface, then you create route(s) to that down stream router pfsense should use to get to those networks.  Not a gateway on the interface."

            1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate
              last edited by

              If you don't want to have a bunch of routes in pfSense it is up to you to route a supernet to the layer 3 switch and subnet that network to each segment.

              Make a small transit network between pfSense and the switch on its own VLAN.

              pfSense 172.24.56.1/24
              Switch 172.24.56.2/24
              Set the default gateway on the switch to 172.24.56.1

              Supernet: 10.2.0.0/16

              Make a gateway (ie SWITCH_GW)  for the switch in System > Routing fat 172.24.56.2

              Make a static route for 10.1.0.0/16 with a destination of SWITCH_GW

              You can then create layer 3 interfaces on each VLAN in the switch for:

              10.1.1.0/24
              10.1.2.0/24
              10.1.3.0/24
              10.1.4.0/24
              …

              Automatic NAT is pretty smart but you want to make sure there are outbound NAT rules for the entire 10.1.0.0/16.  You will also need to DHCP for your segments in the switch or with helpers to something other than pfSense (Like your DC).  It doesn't have the hooks for multi-scope DHCP on one interface.

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 0
              • R
                Raiker
                last edited by

                Thanks Derelict,

                I forgot to add. All of this is currently working, i have multi-scope DHCP from my AD controllers, the switch has an interface on each VLAN (that is how it routes after all), all of what you are saying is basically how i've got it set up right now. It's just the part where you say "Make a gateway (ie SWITCH_GW)…." that breaks my config. I've had cases where for some reason pfSense tries to use THAT GW as the default gw and breaks all outbound connections. I mean, it's working perfectly right now, but in the last month i had 3 separate occurences of pfSense trying to route back thru the switch, IF there was some way to have the routes loaded WITHOUT defining a gateway the multiwan module wouldn't try to do that, that's what i'm trying to find out.

                1 Reply Last reply Reply Quote 0
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  pfSense doesn't do that.  You have something wrong.  If it's not marked as the default gateway it will not use it as the default gateway.

                  Post your config.  Gateways, Routes, interface configurations.

                  Post a drawing.

                  Don't tell us what you think you have done.  Show us exactly what you HAVE done.

                  If you had configured it like that it would be working.  Always.

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  1 Reply Last reply Reply Quote 0
                  • R
                    Raiker
                    last edited by

                    It might have something to do with the finicky-ness of the internet links down here, at least one of the times this ocurred i had reconfigured the main switch therefore causing pfSense to mark all gateways as down.
                    Attached are the VM config, where each WAN has a vlan and it's interface, the interface tab and the route tab.
                    Network config is:
                    VMCluster -> bladecenter stacked switch - > main switch (in trunk mode, ESXi deals with untagging the vlans)
                    WAN1-4 -> main switch (with the ports correctly set as general, untagged and the corresponding vlan id)

                    Like i said, it works. it just so happens that for some reason it suddenly wants to route thru the LANGW. nothing in the logs saying why or how it does that (at least not after a reboot)

                    ![vm config.png](/public/imported_attachments/1/vm config.png)
                    ![vm config.png_thumb](/public/imported_attachments/1/vm config.png_thumb)
                    vm_ifs.png
                    vm_ifs.png_thumb
                    vm_routes.png
                    vm_routes.png_thumb
                    vm_if_if.png
                    vm_if_if.png_thumb

                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      What is the interface address and subnet mask configured on pfSense LAN?

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • R
                        Raiker
                        last edited by

                        10.222.254.251 /16, in the old non-subnetted range.

                        lan_iface.png
                        lan_iface.png_thumb

                        1 Reply Last reply Reply Quote 0
                        • DerelictD
                          Derelict LAYER 8 Netgate
                          last edited by

                          What else do you have anywhere that starts with 10.222 ?? besides the switch at 10.222.255.254

                          No idea why you have that on a /16 but it's your network.

                          Chattanooga, Tennessee, USA
                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • R
                            Raiker
                            last edited by

                            It's a /16 because i was young and stupid at the time (might be stupid still…) and didn't know anything about networking, vlans, subnets and didn't have hardware that supported it. The separate ranges are for each lab and it's people to be able to configure PXE, BOOTP, DHCP, and all the other multicast stuff they might want without crashing the rest of the network.

                            The stuff still on the 10.222.0.0/16 are some of the labs that i haven't been able to move to VLANs yet and one of the DCs that is awaiting new licenses for a migration.

                            1 Reply Last reply Reply Quote 0
                            • DerelictD
                              Derelict LAYER 8 Netgate
                              last edited by

                              So everything's working now and the issue is why, as you say, pfSense uses LANGW as the default gateway right?

                              Did you do something silly like make a failover gateway group with WAN and LANGW as members?

                              Chattanooga, Tennessee, USA
                              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                              1 Reply Last reply Reply Quote 0
                              • R
                                Raiker
                                last edited by

                                No, only the real WAN GWs are included in the gateway groups. i have 3 with different tiers for different needs (adsl for the visitors, cablemodem for users, best one for the servers, etc)

                                1 Reply Last reply Reply Quote 0
                                • DerelictD
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  Well, pfSense doesn't just arbitrarily route out LAN so I guess I don't know what to tell you.

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • R
                                    Raiker
                                    last edited by

                                    Damm, i hoped i had something misconfigured. Well, i'll try to delve deeper and see if it happens again.
                                    Thanks!

                                    1 Reply Last reply Reply Quote 0
                                    • DerelictD
                                      Derelict LAYER 8 Netgate
                                      last edited by

                                      You might but I have no idea where to tell you to look next.  Grep WANGW in /tmp/rules.debug and see if it's somewhere it shouldn't be.  Take a good, long look at what's really happening next time it blows up.  At least get a copy of Diagnostics > Routes.

                                      Chattanooga, Tennessee, USA
                                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.