Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Currently router on a stick, want to go hybrid

    Scheduled Pinned Locked Moved
    L2/Switching/VLANs
    2
    13
    1.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      realityman_
      last edited by

      It was a hybrid, which was likely what was causing some of my problems. I've sort of moved past that half baked, wish it worked, approach.

      I've read quite a few of your posts to help me in my understanding so let me ask you this. With my current topology, and not having one set of storage for my VMs and another set of storage for personal data what is the best way to setup this network?

      I was thinking about getting rid of the router-on-a-stick all together this weekend and go full-on inter-vlan routing. I would setup a new VLAN for transport on a /30, setup new gateways and a route to that /16, and use the sg300-10 as my core switch.

      The only thing that was stopping me was with that approach the firewall loses visibility into inter-vlan traffic. With that approach I'd just have a "LAN" interface and the "transport" interface in pfSense, so I'd basically have to set DMZ type rules on all traffic that was entering\leaving the network which isn't as ideal. Also, in the event my network was compromised I'd lose seeing that traffic between the VLANs as it would be handled at the switch level. I know being a home-gamer my exposure is low, but my skillset is also lower than people securing fortune 500s.

      I'm really not sure which approach to the network I should take having an esxi host that has internet viewable services, but uses back-end storage that is on a different VLAN. Everything else segregates pretty nicely.

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        I am not sure what you mean by "hybrid."

        You can route what you want on a L3 switch via a transit interface and use the switch as layer 2 and put the traffic through the firewall. It's all dependent on how you design it.

        0_1552094554065_pfSense-Layer-3-Switch.png

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • R
          realityman_
          last edited by

          What I mean is have pfSense be the interface/router for VLANs that aren’t 20 or 70. Then have my core switch do the inter-vlan routing for VLAN 20 and 70. It’s not all router on a stick and not all L3 switch routing. The downside of taking 20 and 70 down to the switch level from my perspective is having to apply the same IPS rules to both sets of traffic since they would hit pfsense on the same interface. I like having my DMZ traffic separate so I can put more rules around it and less around my trusted VLAN 20 network.

          I can’t think of a configuration that would allow L3 switching of vlan 20 and 30 and allow me to Apple snort only to the VLAN 70 level at the router\firewall.

          1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate
            last edited by Derelict

            pfSense will not know about VLAN 20 or 30. The layer 3 switch will.

            pfSense will have to know about the LAYER 3 networks that are being routed to it over the transit network so it can pass the traffic from them, perform outbound NAT for them, etc.

            VLAN 70 would be a pfSense interface tagged with 70. That VLAN would be tagged to the L3 switch but there would NOT be a LAYER 3 interface on the switch, just VLAN 70 with a tagged port to pfSense (this could be the same interface that is carrying the transit interface) and untagged ports to the VLAN 70 devices.

            That diagram outlines all of this. Be sure to read the comments there.

            The L3 switch will not route between VLAN 20/30 and VLAN 70 because there is no L3 interface on the switch for VLAN 70. the switch will send VLAN 70 traffic across the transit interface where pfSense can firewall it. VLAN 70 traffic for VLAN 20/30 (and everything else) will also be sent to pfSense.

            pfSense will not see traffic between VLAN 20 and 30 as that will all be routed by the switch.

            In my diagram, VLAN 20 would be 172.28.1.0/24, VLAN 30 172.28.2.0/24 and VLAN 70 172.29.1.0/24

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • R
              realityman_
              last edited by realityman_

              Right, sorry, I mistyped and meant VLAN 20 or 70. If I had VLAN 70 go up and be routed at layer 2 and have an interface for it, then it's traffic to the NAS on VLAN 20 would still be routed up through the pfSense box as well, which is what is causing problem I'm having now. Chatty esxi from VLAN 70 starting a lot of I\O at gigabit speeds giving snort strokes when it is routed from VLAN 70 to VLAN 20 via the interfaces in pfSense.

              I would think I would need to add VLAN 20 and 70 as interfaces at the core switch, setup the gateway for them in pfSense as well as a static route for their range. That would make pfSense unaware of any 20 and 70 traffic. Traffic that went to 10, 30, 40, 50 or 60 would still be routed via pfSense. VLAN 20 and 70 traffic that wasn't in those subnets would be routed up through the default route (transit route) and tagged with that VLAN, for which I could make an interface an pfSense. That would segregate the VLAN 20 and 70 traffic from one another, but understandably pfSense wouldn't be able to segregate it, nor would snort. I'd have to apply the same snort rules to everything on that transit interface. That sound right?

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by Derelict

                I don't know. man. VLAN 20, VLAN 30, VLAN 70 the principles are the same. It's either routed by the switch and routed up the transit interface or it's tagged to the switch and is just a Layer 2 VLAN.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                R 1 Reply Last reply Reply Quote 0
                • R
                  realityman_ @Derelict
                  last edited by realityman_

                  Right, so I can keep the VLANs that don't need L3 routing at the switch level in pfSense and have them L2 routed via VLAN tagging. The rub is to get inter-vlan routing at the switch level for VLAN 20 and 70 they have to be taken out of their pfSense interfaces. That means I wont be able to segregate them at the pfSense level like I can now so I can only apply more rigorous snort rules for VLAN 70 traffic. I will have to run snort on the transit interface which would include VLAN 20 and 70 traffic in it.

                  Here you can see what happens with snort on and pfSense doing the routing. pfSense can't route between 70 and 20 with gig speed traffic without causing massive latency.

                  https://i.imgur.com/CoBhiCa.png

                  Here is snort off with pfSense doing the routing. There are a few 100 ms increase blips, but not near as bad as with snort. The 2758 just doesn't have the horsepower to do gig with more than a few streams going through it it appears.

                  https://i.imgur.com/CoBhiCa.png

                  I think I've resolved myself to the fact that I'll have to L3 route VLAN 20 and 70. I just wish there was a way I could somehow tell Snort to limit the rulesets it runs on VLAN 20 if that makes sense. The only thing else I could do is get dedicated storage for the VLAN 70 boxes, then routing on a stick would continue to work as it'd all be contained within it's own subnet.

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    If you want pfSense to route it, pfSense has to route it.

                    If you don't, you can do it in the switch.

                    You can't have it both ways.

                    I'm probably failing to see the proper question here.

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      You should be able to tell snort to bypass certain layer3 source networks.

                      Not sure about how.

                      That would be a question for the IDS/IPS category as it is completely unrelated to L2/Switching/VLANs.

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      R 1 Reply Last reply Reply Quote 0
                      • R
                        realityman_ @Derelict
                        last edited by realityman_

                        Yea, I've come to the conclusion short of getting new storage and putting it squarely in each VLAN that I need to do L3 routing at the switch for VLAN 20 and 70. I guess I was just more or less asking if there was any other architecture you can think of that may have worked better, but it really seems like it is that straight forward.

                        So to summarize my changes:

                        • Disable Snort on VLAN 20 and 70

                        • Create 172. something /30 subnet for transit

                        • Create new VLAN tag 172 in pfSense

                        • Create new interface tied to this VLAN

                        • At both switches I will add a new VLAN for the transit network, and set that as the default route to 172.something.1

                        • Add VLAN 20 and VLAN 30 at the core switch (sg300-10). I'll put in ACLs to block everything between the VLANs except the IP\Port combos I currently have in my
                          firewall relating to those two subnets.

                        • Disable VLAN 20 and VLAN 70 interfaces in pfSense

                        • Create new gateway with 172.something.1 as gateway

                        • Create new static routes for 10.37.70.0/24 and 10.37.70.0/24 via the gateway created above

                        • Enable Snort rules on new Transit interface.

                        • Verify any needed VLAN 172 firewall rules that are needed (shouldn't be any as this will only be used for outbound requests, correct?)

                        Sound about right?

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.