Installing pfSense with a layer 3 switch



  • Well I made progress.  I went under the WAN interface General Config  selected IPv6 configuration type and selected none.

    My DSLReport's speedtest now is running 345 megabit.  IPv6 is having a drastic effect on NIC speed.

    Have you seen this before?


  • LAYER 8 Global Moderator

    that doesn't disable ipv6 on pfsense, that just sets your wan interface to not get a ipv6 from your provider.. I doubt ipv6 has anything to do with your speed other than you were prob using ipv6 for your testing and your isp ipv6 network is slower than their ipv4, or the connection to where you were testing via ipv6 is slower, etc..

    So your now at the speed you are paying for..

    Do you have any other devices on this 192.168.10 network?  If so then its not a transit and those devices going to have issues talking to stuff on your L3 switch.

    Also you don't need a /24 as a transit, you could just use a /30 - if you made it say 172.16.0/30 you could then just use a simple summary route 192.168/16 route to your networks on your L3 switch.  Then no matter what 192.168 vlan you add to that switch you never have to touch your routes again.



  • The idea with VLAN10 was to create a router VLAN that would be totally isolated from all other devices on the local network including things as slow wireless devices, broadcasts, Windows elections, and all chatty local traffic.

    Sometimes I need to add a short term device to the router VLAN because of changes and things like configuring pfSense.
    So I did not use /30 mask.

    I could arrange my networks so I could super scope them in a class B mask.  I have my layer 3 switch setup posted on SmallNetBuilder forums so I tried to keep it simple, no tricky masks.  The only thing a little hard is I have ACLs setup so I can share certain IPs to the guest network using a /29 mask.



  • Here are some setting I have changed today under System->Advanced. I think my NICs must be supported as I have been reading pfSense all day and tunning.

    1. I changed kern.ipc.nmbclusters="1000000" to increase mbufs. I had to add this entry to system tunables.

    2. The settings for Hardware TCP Segmentation Offload (TSO) and Hardware Large Receive Offload (LRO) under System > Advanced on the Networking tab default to checked (disabled) for good reason. Nearly all hardware/drivers have issues with these settings, and they can lead to throughput issues. Ensure the options are checked. Sometimes disabling via sysctl is also necessary. I enabled these by unchecking them.

    My system seems smoother now. What do you guys think? Any ideas about more tunning?



  • I have just added CoDel under traffic shaping without bw parameters.  It is working quite well.  Since I have a 300 megabit connection for home and I don't think I will saturate it this should be a prefect fit.  I have gained 3ms over my standard configuration using DSLReports speedtest.  My whole network seems to be flowing better.

    I should add now my system does not show any more resources being used.




  • @johnpoz:

    Do you have any other devices on this 192.168.10 network?  If so then its not a transit and those devices going to have issues talking to stuff on your L3 switch.

    Also you don't need a /24 as a transit, you could just use a /30 - if you made it say 172.16.0/30 you could then just use a simple summary route 192.168/16 route to your networks on your L3 switch.  Then no matter what 192.168 vlan you add to that switch you never have to touch your routes again.

    I have been thinking about your commit having devices on the 192.168.10.0 network.  I am thinking maybe you are having problems because you are using the pfsense box as the default gateway for workstations in the same network as pfsense.  If you use the L3 switch as the default gateway for workstations on the same segment, network as pfsense there will not be problems accessing devices on the L3 switch.  All nonlocal IPs for the workstation will flow out the default route which points to pfsense on the L3 switch so it should all work well.

    So when a PC on the same segment as pfsense requests an internet IP address the IP request hits the L3 switch and bounces back to the pfsense box.


  • LAYER 8 Global Moderator

    Who is having problems?  I don't have any problems.. And sure if you want to point devices to your L3 switch svi as their gateway still doesn't fix the asynchronous routing - you just now changed the direction of where the issue will happen.

    Here is the jux of asynchronous routing issues when you don't use a transit, which with stateful firewalls can cause all kinds of problems, let alone the performance issues with hairpinning connections.

    So when your taking to the network that is connected to your gateway your using your fine.. If your talking to the internet and you use pfsense as gateway.  Traffic flows to and from the same point.  If you were using your switch l3 svi in that segment as your gateway and talking to boxes on networks behind that l3 switch again your good.

    The problem is when your using a gateway in wrong direction - see 2nd pic, you point to your L3 switch, just to go back to pfsense to get to the internet.  But the return traffic will not go back to your L3 switch.. That packet to pfsense is going to a 192.168.10 address which is directly connected to pfsense.. So it just sends the packet directly to the client = asynchronous routing.

    Now you could fix this problem with host routing on your client in your 192.168.10 network and his default to get to internet is pfsense, while he has specific routes to your other networks.  Or you use a transit to connect pfsense to your L3 switch and don't have any clients on that network..








  • @johnpoz:

    Who is having problems?  I don't have any problems.. And sure if you want to point devices to your L3 switch svi as their gateway still doesn't fix the asynchronous routing - you just now changed the direction of where the issue will happen.

    Here is the jux of asynchronous routing issues when you don't use a transit, which with stateful firewalls can cause all kinds of problems, let alone the performance issues with hairpinning connections.

    So when your taking to the network that is connected to your gateway your using your fine.. If your talking to the internet and you use pfsense as gateway.  Traffic flows to and from the same point.  If you were using your switch l3 svi in that segment as your gateway and talking to boxes on networks behind that l3 switch again your good.

    The problem is when your using a gateway in wrong direction - see 2nd pic, you point to your L3 switch, just to go back to pfsense to get to the internet.  But the return traffic will not go back to your L3 switch.. That packet to pfsense is going to a 192.168.10 address which is directly connected to pfsense.. So it just sends the packet directly to the client = asynchronous routing.

    Now you could fix this problem with host routing on your client in your 192.168.10 network and his default to get to internet is pfsense, while he has specific routes to your other networks.  Or you use a transit to connect pfsense to your L3 switch and don't have any clients on that network..

    I have not noticed a problem with running the setup as stated above. But I have only run it a couple of days. I have a solution that I would like to run by you for pfsense and a layer 3 switch.

    192.168.10.X  X>9
            pfsense–----------------------workstation-----------------------------layer 3 switch
              192.168.10.1/30                default gateway192.168.10.9/29      L3 switch default route  192.168.10.1/30
                                                                                                            L3 switch VLAN10  192.168.10.2/24

    Will pfsense like this better?  If you like it I will make the changes.  My only thoughts are my local networks are all going to be contained in my alias definition for snort and everything.  I don't know what kind impact on pfsense this will have.


  • LAYER 8 Netgate

    This has nothing to do with pfSense. This is basic IP routing. What is 192.168.10.9?



  • @Derelict:

    This has nothing to do with pfSense. This is basic IP routing. What is 192.168.10.9?

    You are right I made a mistake.

    192.168.10.X  x>6
            pfsense–----------------------workstation-----------------------------layer 3 switch
              192.168.10.1/30                default gateway192.168.10.2/29      L3 switch default route  192.168.10.1/30
                                                                                                            L3 switch VLAN10  192.168.10.2/24


  • LAYER 8 Netgate

    But you indicated it was a default gateway of the workstation on your transit network (which is your entire problem - get your nodes off your transit network - either behind the L3 switch or on another pfSense interface.)



  • @Derelict:

    But you indicated it was a default gateway of the workstation on your transit network (which is your entire problem - get your nodes off your transit network - either behind the L3 switch or on another pfSense interface.)

    Never mind. I now see the problem you pointed out.

    My initial setup does work pointing to the switch.


  • LAYER 8 Global Moderator

    "My initial setup does work pointing to the switch."

    And that is a BROKE setup plain and simple..  You have a asynchronous routing, as in my 2nd pic.  While it may work depending on your traffic flows and firewall rules and or setting to be less strict its BAD practice!!

    You should always avoid this sort of setup, along with avoiding any sort of hair pins where ever possible..  Sometimes you have to live with a hairpin when you allow traffic between vlans that are on the same physical nic.  But all vlans and or subinterface always remove overall performance of that nic..



  • Well I have tried /30 mask.  pfsense does not seem to talk to the switch.  I can not ping either way.

    pfsense–----------------------------------VLAN10----------------layer 3 switch
      192.168.10.1/30                                192.168.10.2/30

    The other networks and VLANs are on the switch untouched.  The only thing I did was change the LAN interface IP 192.168.10.1/30 and the switch VLAN 10 to 192.168.10.2/30.  Have you tried this with your SG300 switch?

    If I use the /24 mask all works. It even works with no other devices in VLAN10 other than pfsense and the layer 3 switch.

    I reinstalled pfsense and have the same problem.

    Now I have a problem on the new install with the resolver.  I think it was working on the first install.  The new install seems to have issues with my routed networks and DNS.  I turned off resolver and turned on the forwarder and all works.  I am going to keep looking.


  • LAYER 8 Global Moderator

    you could use a /8 for your transit network if you wanted.  As long as no end use devices are on the network then its a "transit" network.  /30 is just common since its only got 2 IPs.. A network that connects 2 or more routers is a transit network..  When you put devices on such a network and have different paths for how traffic goes to and from that device is when you have asynchronous routing. Which is BAD thing, especially when it comes to stateful firewalls like pfsense.  Clients don't always like the going to mac A, and coming back from mac B either..  Like I said its BAD and Broken setup..

    /29 is also common since you have 6 IPs and allow for hsrp on both sides of the transit with HA pairs with your routers with your physical IPs and VIPs on both sides..  You can use any sized network you want/need as your transit.

    As to have I done this with my sg300.. I do not have my sg300 currently in L3 mode - I have no reason for downstream networks in my home setup, while I have thought of it for performance since my pfsense is VM on some aging hardware. But yes I have done it when I was playing with the switch when I first got it an had it in L3 mode.  But this is something that that is done every single day in any network anywhere on the planet.  As stated this is basic IP routing.

    You clearly show a workstation on your 192.168.10 network, pointing to to your switch on that network, while the gateway off to the internet is pfsense on that same 192.168.10 network - that is NOT a transit network..  And if you just tried changing the mask to /30 you would only have 2 address in that network… So you could NOT have any workstations on it..



  • For home use I find separating  devices from my server and workstations works better.  I also have a music, LAN and guest network. I have 3 wireless APs with 2 common SSIDs one for LAN and one for guest to allow roaming and they support my network by putting the user in the guest VLAN or the LAN VLAN based on logon.

    How all this started is a friend brought over a broken laptop for me to fix and it infected my music server which I had spent weeks putting CDs on.  I now have a music VLAN separate from all others.  This prompted me to build a guest VLAN.  I still needed to share printers and certain video stuff which friends bring over.  So I needed ACLs to share these devices to my guest network and my multi VLAN network was born using IP networks.


  • LAYER 8 Global Moderator

    I have multiple vlans and physical networks - agreed this is a good setup even for a home.  But I just let pfsense be my firewall/router between all my segments.  I have no need for a L3 switch in my home network.. Pfsense is more than capable or routing the traffic..  If I really needed the full gig between segments I would update my hardware vs running L3 downstream and loosing my firewall between the segments.

    My pfsense vm running on a old HP N40L can do 400-500mbps between the segments..  Which is good enough for my wifi that is for sure.. My workstation on the same segment as the stuff I work do..  The stuff on the other segments would never need to make sure they have full 1gig routed/firewalled… Internet is only 80mbps for gosh sake...

    My guest wifi is completely isolated, if you want to get on my normal wifi, that is still isolated from my other networks you need eap-tls setup..  I completely get the use and commend proper network segmentation and firewalls in a home setup..

    But if I was going to use a downstream router, I sure and the hell would not set up asynchronous routing - nor would I hairpin connections ;)


  • LAYER 8 Netgate

    I wouldn't use a layer 3 switch for that. Or at least for the segments you really want to lock down like the guest network.

    This is basic IP routing, bro. Only you can decide how you want your network topology laid out. You can have some networks on the layer 3 switch, relying on whatever its packet filtering capabilities are and some networks on pfSense using its full stateful firewall capabilities. You can have some VLANs with SVIs on the switch and some without SVIs for which pfSense provides all the Layer 3 services. It's really up to you.

    But you really can't put hosts on the same network that connects the two routers unless you want to maintain routing tables on those hosts.

    Well I have tried /30 mask.  pfsense does not seem to talk to the switch.  I can not ping either way.

    Then you did it wrong, plain and simple. Post details of what you have actually done, not what you think you've done because it's not what you think you've done or it would be working.


  • LAYER 8 Global Moderator

    you really can't put hosts on the same network that connects the two routers unless you want to maintain routing tables on those hosts.

    Exactly!!!! Very cleanly stated…


  • LAYER 8 Netgate

    I've been meaning to do this one for a while:




  • I use pfsense for a fast router firewall NAT device.  I like the idea of snort.

    I want to keep all my VLAN networks on the switch.  When I create VLANs I always assign an IP network to each VLAN.  ACLs provide enough control for me and my network.  I  have 3 Cisco WAP321 which dumps users into a guess network or a LAN network.  This works great for me. I am able to feed multiple VLANs to one router VLAN on the layer3 switch to feed the router. The SG300-28 has a 17 GIG backplane so it can run faster than any internet connection I would have.  I only have a 300megabit connection.  I just want the fastest firewall I can run on the front door to the internet.  pfSense is on the table now.

    I am not saying pfsense has a problem with a /30 mask. It could be the SG300-28 switch.  It is why I asked johpoz whether he had tried it as he owns an Cisco SG300 switch also.  I have posted a question on the Cisco small business web site to ask if there is an issue.

    I took a working pfsense router 192.168.10.1/24 mask plugged into a router VLAN on the SG200-28 switch defined with a /24 mask. It was working and is my current config I am using.  I changed the VLAN IP to 192.168.10.2/30 on the switch and I changed the default route to 192.168.10.2/30 also. I then ran 2 on the console for pfsense to change the LAN IP to 192.168.10.1/30.  I could not ping from the switch using ping on the switch to pfsense.  I could not ping from pfsense to the switch.  The switch port used is an access port defined to the router VLAN. It is the same port used for the /24 mask which works.  This sounds basic so I was looking for confirmation from john since he recommended using a /30 mask on this thread.


  • LAYER 8 Netgate

    Neither the Cisco SG nor pfSense has any problems with a /30 netmask. Guaran-effing-teed. You did something wrong.

    And it doesn't matter one bit what netmask you use. All that matters is that there are no hosts on the transit network that you expect to route properly unless you maintain a proper routing table on that host.



  • The net mask does make a difference.  It works with a /24 mask and does not work with /30 mask.  I do not run any clients in my router VLAN.  The idea of the router VLAN is to isolate from the broadcast domain, default VLAN, Windows chatter, etc. I do not want the router waiting for any local traffic not destined for the internet.


  • LAYER 8 Netgate

    Dude. You did something wrong. No way the Cisco or pfSense has a problem with /30. If it was /31 maybe. /30 no way. Look at your config again.



  • When I have another free evening.  I spent about 5 hours going over everything.  I even reinstalled pfsense.


  • LAYER 8 Netgate

    Chasing ghosts. And that was after changing the netmask on both ends from /29 to /30 - no reboots just change and apply.

    ![Screen Shot 2016-01-31 at 4.25.48 PM.png](/public/imported_attachments/1/Screen Shot 2016-01-31 at 4.25.48 PM.png)
    ![Screen Shot 2016-01-31 at 4.25.48 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2016-01-31 at 4.25.48 PM.png_thumb)
    ![Screen Shot 2016-01-31 at 4.25.38 PM.png](/public/imported_attachments/1/Screen Shot 2016-01-31 at 4.25.38 PM.png)
    ![Screen Shot 2016-01-31 at 4.25.38 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2016-01-31 at 4.25.38 PM.png_thumb)
    ![Screen Shot 2016-01-31 at 4.27.39 PM.png](/public/imported_attachments/1/Screen Shot 2016-01-31 at 4.27.39 PM.png)
    ![Screen Shot 2016-01-31 at 4.27.39 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2016-01-31 at 4.27.39 PM.png_thumb)



  • You need to be changing the LAN side em1 not em0.  This means all your local networks are routed on the LAN side. And you need static routes for all local traffic. I guess it could be an alias.


  • LAYER 8 Netgate

    em0 is my LAN side. Neither device has a problem with /30, regardless.



  • Sounds good.  I don't know why they will not link up with /30 mask for me.  It works with a /24 mask so there is not a rush.


  • LAYER 8 Global Moderator

    You can use whatever mask you want for the transit - the point is there can be no clients on the transit or you will end up with problems unless you create host routes on them for the networks in 2 different directions.

    The fact that you think a /30 is a problem for these devices is beyond nonsense.. As derelict said you did something wrong..



  • @johnpoz:

    You can use whatever mask you want for the transit - the point is there can be no clients on the transit or you will end up with problems unless you create host routes on them for the networks in 2 different directions.

    The fact that you think a /30 is a problem for these devices is beyond nonsense.. As derelict said you did something wrong..

    The /24 mask is more convenient so if you need to change the gateway to the static routes you can plug in a machine and make the change.  With a /30 mask there basically is no access to pfsense if something happens to your routing other than console from what I see.

    The question about the /30 mask can be answered by you since you have one of these SG300 switches.  Just set it up.  Please post the config on the SG300 and I will compare it to mine.


  • LAYER 8 Netgate

    Pertinent parts.

    vlan database
    vlan 1000,2000

    ip dhcp server
    ip dhcp pool network TEST_LAYER3
    address low 172.28.1.65 high 172.28.1.254 255.255.255.0
    default-router 172.28.1.1
    dns-server 192.168.223.1
    exit

    interface vlan 1000
    name TRANSIT
    ip address 192.168.230.2 255.255.255.252
    !

    interface vlan 2000
    name TEST_LAYER3
    ip address 172.28.1.1 255.255.255.0
    !

    interface gigabitethernet46
    description ROUTER_LAN
    switchport mode general
    switchport general allowed vlan add 1000 tagged
    switchport general acceptable-frame-type tagged-only
    !

    Not a lot to it. On pfSense I just created interface TRANSIT on vlan1000 as 192.168.230.1/30, a gateway for 192.168.230.2, and a static route for 172.28.0.0/16 to the gateway.

    Then I passed ICMP any source any dest TRANSIT address on the TRANSIT interface. Could ping across in both directions and from a host on vlan 2000.


  • LAYER 8 Netgate

    @coxhaus:

    The /24 mask is more convenient so if you need to change the gateway to the static routes you can plug in a machine and make the change.  With a /30 mask there basically is no access to pfsense if something happens to your routing other than console from what I see.

    That's what management VLANs are for.



  • @Derelict:

    Pertinent parts.

    vlan database
    vlan 1000,2000

    ip dhcp server
    ip dhcp pool network TEST_LAYER3
    address low 172.28.1.65 high 172.28.1.254 255.255.255.0
    default-router 172.28.1.1
    dns-server 192.168.223.1
    exit

    interface vlan 1000
    name TRANSIT
    ip address 192.168.230.2 255.255.255.252
    !

    interface vlan 2000
    name TEST_LAYER3
    ip address 172.28.1.1 255.255.255.0
    !

    interface gigabitethernet46
    description ROUTER_LAN
    switchport mode general
    switchport general allowed vlan add 1000 tagged
    switchport general acceptable-frame-type tagged-only
    !

    Not a lot to it. On pfSense I just created interface TRANSIT on vlan1000 as 192.168.230.1/30, a gateway for 192.168.230.2, and a static route for 172.28.0.0/16 to the gateway.

    Then I passed ICMP any source any dest TRANSIT address on the TRANSIT interface. Could ping across in both directions and from a host on vlan 2000.

    I think I have found at least one difference my switch port is defined as an access port. The idea was to keep all the routing on the L3 switch.  The port adds the tags as data comes in and strips tags as data flows out.


  • LAYER 8 Netgate

    If you're only doing one VLAN between pfSense and the switch an access port is fine but you can't define the VLANs on pfSense - it has to just be emX, not vlan X on emX.

    When talking to a managed switch I always tag it even if it's only one so you can add a vlan without screwing up what's already working.



  • @Derelict:

    Pertinent parts.

    vlan database
    vlan 1000,2000

    ip dhcp server
    ip dhcp pool network TEST_LAYER3
    address low 172.28.1.65 high 172.28.1.254 255.255.255.0
    default-router 172.28.1.1
    dns-server 192.168.223.1
    exit

    interface vlan 1000
    name TRANSIT
    ip address 192.168.230.2 255.255.255.252
    !

    interface vlan 2000
    name TEST_LAYER3
    ip address 172.28.1.1 255.255.255.0
    !

    interface gigabitethernet46
    description ROUTER_LAN
    switchport mode general
    switchport general allowed vlan add 1000 tagged
    switchport general acceptable-frame-type tagged-only
    !

    Not a lot to it. On pfSense I just created interface TRANSIT on vlan1000 as 192.168.230.1/30, a gateway for 192.168.230.2, and a static route for 172.28.0.0/16 to the gateway.

    Then I passed ICMP any source any dest TRANSIT address on the TRANSIT interface. Could ping across in both directions and from a host on vlan 2000.

    I´ve followed this discussion with great interest since I have a similar SG300 layer 3 setup as Coxhaus and have just started to look into replacing my Linksys LRT224 router with a pfSense firewall.

    When using a transit network for the routing between the switch and the firewall as in your nice example, I guess there must be a separate management interface/IP address used for logging into the pfSense Web GUI?



  • @Derelict:

    @coxhaus:

    The /24 mask is more convenient so if you need to change the gateway to the static routes you can plug in a machine and make the change.  With a /30 mask there basically is no access to pfsense if something happens to your routing other than console from what I see.

    That's what management VLANs are for.

    I´m totally new to pfSense and am collecting information on how to set it up and administer it. Is the default management VLAN in pfSense VLAN 1 and is the IP address configured for the LAN interface the address of the management VLAN?


  • LAYER 8 Netgate

    All interfaces in pfSense are untagged by default, with WAN configured to obtain an address via DHCP and LAN as 192.168.1.1 with an active DHCP server starting at .100. You should really start another thread with a specific question. Doesn't sound like this has much to do with this one.



  • @Derelict:

    Pertinent parts.

    vlan database
    vlan 1000,2000

    ip dhcp server
    ip dhcp pool network TEST_LAYER3
    address low 172.28.1.65 high 172.28.1.254 255.255.255.0
    default-router 172.28.1.1
    dns-server 192.168.223.1
    exit

    interface vlan 1000
    name TRANSIT
    ip address 192.168.230.2 255.255.255.252
    !

    interface vlan 2000
    name TEST_LAYER3
    ip address 172.28.1.1 255.255.255.0
    !

    interface gigabitethernet46
    description ROUTER_LAN
    switchport mode general
    switchport general allowed vlan add 1000 tagged
    switchport general acceptable-frame-type tagged-only
    !

    Not a lot to it. On pfSense I just created interface TRANSIT on vlan1000 as 192.168.230.1/30, a gateway for 192.168.230.2, and a static route for 172.28.0.0/16 to the gateway.

    Then I passed ICMP any source any dest TRANSIT address on the TRANSIT interface. Could ping across in both directions and from a host on vlan 2000.

    Did you define a default route on the L3 switch pointing to 192.168.230.1, or isn´t this necessary?


  • LAYER 8 Netgate

    Yes if you want all traffic without a better route to go to 192.168.230.1.