Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Typical setup and the same issue over and over! L3 switch + Pfsense, Need help!

    Scheduled Pinned Locked Moved Routing and Multi WAN
    20 Posts 4 Posters 2.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DerelictD
      Derelict LAYER 8 Netgate
      last edited by

      What are the rules on LAN?

      And the subnets matter. You're probably better off posting accurate details if you want accurate assistance.

      Chattanooga, Tennessee, USA
      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
      Do Not Chat For Help! NO_WAN_EGRESS(TM)

      1 Reply Last reply Reply Quote 0
      • johnpozJ
        johnpoz LAYER 8 Global Moderator
        last edited by

        Quite possible your /30 could be overlapping your vlans.. Since you could not be bother to actually post the exact info..

        "so I do set the LAN interface with a gateway of 10.218.1.89"

        NO you do not set a gateway on a lan interface..  Did you read what it says
        "If this interface is an Internet connection, select an existing Gateway from the list or add a new one using the link above. On local LANs the upstream gateway should be "none"."

        Pretty sure they default to lan nat as the source, which your downstream networks would not match.

        An intelligent man is sometimes forced to be drunk to spend time with his fools
        If you get confused: Listen to the Music Play
        Please don't Chat/PM me for help, unless mod related
        SG-4860 24.11 | Lab VMs 2.8, 24.11

        1 Reply Last reply Reply Quote 0
        • I
          IrixOS
          last edited by

          Well,

          As requested:

          VLAN2: 10.218.1.0 - 10.218.1.63 /26  GW: 10.218.1.1
          VLAN3: 10.218.1.64 - 10.218.1.79 /28  GW: 10.218.1.65
          VLAN6: 10.218.1.80 - 10.218.1.87 /29 GW: 10.218.1.81

          DEFAULT ROUTE: 10.218.1.88 - 10.218.1.91 /30

          PFSENSE Interface IP: 10.218.1.90 /30
          L3 SWITCH Interface IP: 10.218.1.89 /30

          No Derelict, I have not set any rule,…

          What do you have in mind exactly?

          JohnPOZ,

          I have no idea what you are talking about,...

          Thank you for having a first look into the issue,

          I am standing by for further instructions,...

          1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate
            last edited by

            What does this mean?

            DEFAULT ROUTE: 10.218.1.88 - 10.218.1.91 /30

            What are the rules on this pfSense interface

            PFSENSE Interface IP: 10.218.1.90 /30

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • I
              IrixOS
              last edited by

              Hmm,

              I already knew you would react to that… :-[

              I do not know, I call it that way, e.g ip route 0.0.0.0 0.0.0.0 10.218.1.90 :-X

              It is the subnet between the L3 switch routerport and the PFsense NIC.

              ....

              I include  images of the pfsense rules, Nat, Gateway/Routing section.
              That might bring up some enlightment,...Other than a few changed fields, I did not touched them and are as is. The route/gateways section is a complete farce when you look at the ip address.
              Frankly I cannot see the forest through the trees no more.

              Thank you for replying.

              NAT.jpg
              NAT.jpg_thumb
              Gateways.jpg
              Gateways.jpg_thumb
              Routes.jpg
              Routes.jpg_thumb
              Rules.jpg
              Rules.jpg_thumb

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                set the default route in the switch to 10.218.1.90

                create a gateway on pfSense for 10.218.1.89 - call it L3_SWITCH

                Create static routes for your /26, /28, and /29 with L3_SWITCH as the gateway.

                Make sure the firewall rules on the pfSense interface will pass traffic for the entire /24, not just the /29 (Called LAN net).

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • I
                  IrixOS
                  last edited by

                  Hmm, Derelict,

                  I performed your steps,

                  You can see on the image that I tried to ping from a host in VLAN3 to the gateway of another VLAN, VLAN2 and did not succeed, I got a TTL error. I know the routing on the switch works.

                  Why is the host using the pfsense routers gateway to communicate with another VLAN host? That is very odd.

                  Also on the image regarding the modem settings, you can see the 10.218.1.88 network is set for outbound, I used that to temporarely access the internet I believe when I configured my new ADSL modem which is in bridged mode.

                  I do not know about the Rules list either if they are correct after reading your steps,…

                  Ping.jpg
                  Ping.jpg_thumb
                  ![Rules Update.jpg](/public/imported_attachments/1/Rules Update.jpg)
                  ![Rules Update.jpg_thumb](/public/imported_attachments/1/Rules Update.jpg_thumb)
                  Rules_2.jpg
                  Rules_2.jpg_thumb

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    TTL means you have a routing loop and the traffic is never reaching its destination but is bouncing back and forth usually.

                    Traffic between VLAN2 and VLAN3 will be handled entirely by your layer 3 switch. pfSense has nothing to do with it. Did you create SVIs for all your VLANs?

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • I
                      IrixOS
                      last edited by

                      Hmm,…

                      I knew you would mention that. I do not know what happened here but I now have a ping between host on all vlans. I powered the trunked switch, telnetted from that switch to the routing switch to check the vlans and now it works. That is odd!

                      Derelict, did you see my images?

                      I successfully pinged the PFsense box from a host on VLAN 3 and I can access the webserver right now.
                      However I cannot still access the internet from the host, I have set the DNS ip of the host to 10.218.1.90, thus the ip of Pfsense,...

                      Did you see my images?

                      Can you see what is obsolete/wrong in the Rules list?
                      What regarding the screenshot about the bridged modem on the NAT rules outbound?

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        You need to resolve names, route to pfSense, route out WAN and probably NAT in order to get internet from the switch VLANs. Check all that.

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • I
                          IrixOS
                          last edited by

                          Hmm,

                          I tried everything you said,

                          Now my hosts can reach the firewall but cannot access the internet.

                          I tried to logically play with the rules based on the already existing one, the ones created automatically for the 10.218.1.88 /30 subnet, so when I give my PC an ip of 10.218.1.89 it is natting fine.
                          When I add the vlan subnets to the rules or in NAT, the hosts can not reach the internet. There must be something I overlooked,…but It very difficult to figger out what.

                          Like I said before, the configuration of pfsense is too complex for this kind of simple and stupid setup.

                          I added 2 more images,...If you are willing to analyze them,...

                          Thank you

                          Update.jpg
                          Update.jpg_thumb
                          Update_2.jpg
                          Update_2.jpg_thumb

                          1 Reply Last reply Reply Quote 0
                          • C
                            coxhaus
                            last edited by

                            I have a question in the second picture you have the rules set to manual configuration but on the right side they say auto created? I am wondering because I am running a L3 switch behind pfsense.  Mine is still set to auto.

                            fpsense1.PNG
                            fpsense1.PNG_thumb

                            1 Reply Last reply Reply Quote 0
                            • I
                              IrixOS
                              last edited by

                              Well,

                              I followed

                              https://doc.pfsense.org/index.php/Accessing_modem_from_inside_firewall

                              I have a modem with a router in bridged mode. This is what I had to do for internet access.

                              1 Reply Last reply Reply Quote 0
                              • C
                                coxhaus
                                last edited by

                                Here is my first setup one evening.  I don't know if is right but it works.  I never got a /30 mask to work for me as I probably did something wrong.

                                I am running a modem in bridge mode also and I have skipped the IP network on the modem so I can pull up the stats. It is found using the default gateway.

                                https://forum.pfsense.org/index.php?topic=105825.msg591728#msg591728

                                PS
                                I just tried accessing my modem and it does not work.  I guess it worked when I was using a router.  I just noticed there is a firewall rule to block all private IP addresses.  I may have to add one to allow the modem's IP out pfsense.  The ISP is going to block it as private IP addresses are not routable.  So I don't have a tested answer for accessing the modem.

                                PSS
                                Added the rule to allow 192.168.1.100 to pass the WAN interface.  The problem I have now is I cannot escalate the rule higher than the general rule blocking all private IP addresses.

                                1 Reply Last reply Reply Quote 0
                                • DerelictD
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  Like I said before, the configuration of pfsense is too complex for this kind of simple and stupid setup.

                                  A two-router set up is anything but simple for those not used to it.

                                  It is actually quite straightforward.

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • DerelictD
                                    Derelict LAYER 8 Netgate
                                    last edited by

                                    I have a question in the second picture you have the rules set to manual configuration but on the right side they say auto created? I am wondering because I am running a L3 switch behind pfsense.  Mine is still set to auto.

                                    I am not sure which version added it but some time ago pfSense got smarter about adding static routes to automatic outbound NAT in addition to LAN interface networks.

                                    Chattanooga, Tennessee, USA
                                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                    1 Reply Last reply Reply Quote 0
                                    • DerelictD
                                      Derelict LAYER 8 Netgate
                                      last edited by

                                      I added 2 more images,…If you are willing to analyze them,...

                                      That looks good.  You are going to have to provide more information than "can not reach the internet." It's impossible to tell from that if you have a routing problem, NAT problem, DNS problem, DHCP problem, Layer 2 problem, etc.

                                      Can the hosts on the L3 switch ping 8.8.8.8? How about 10.218.1.90? How about whatever DHCP is giving them as a default gateway which should be the VIF on the L3 switch for their segment???

                                      Chattanooga, Tennessee, USA
                                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                      1 Reply Last reply Reply Quote 0
                                      • C
                                        coxhaus
                                        last edited by

                                        @Derelict:

                                        I have a question in the second picture you have the rules set to manual configuration but on the right side they say auto created? I am wondering because I am running a L3 switch behind pfsense.  Mine is still set to auto.

                                        I am not sure which version added it but some time ago pfSense got smarter about adding static routes to automatic outbound NAT in addition to LAN interface networks.

                                        Yes my outbound NAT routes were added automatically with pfsense v 2.2.6.

                                        1 Reply Last reply Reply Quote 0
                                        • I
                                          IrixOS
                                          last edited by

                                          Derelict,  >:(

                                          I think we have got winner!!! ;D

                                          I was about to throw the Tyan GS10 through the window along withe the firewall :P

                                          You mentioned to check other factors that might block web access.

                                          I was thinking about a misconfigured NAT or rule, you said it could be anything, well it turned out to be the resolver ::)

                                          I should have known that as I was a former IPCop user.

                                          Derelict I would say BRAVO, I have learned alot about pfsense in a short time and will continue expoiting it to some degree at least.

                                          I will move forward to the access lists, and fine tuning

                                          Many thanks

                                          Sincerely,

                                          IRIXos
                                          I HATE FreeBSD desktops

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.