Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    There is definitely a rare, screwy bug with static routes

    Routing and Multi WAN
    2
    5
    578
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Z
      zippydan last edited by

      I already had a screwy situation with static routes before, which I was not able to solve without reinstalling pfsense: https://forum.pfsense.org/index.php?topic=116122.msg644275#msg644275

      Now, I have just setup a new instance of pfsense at a new site, running version 2.3.2p1.  It has been running fine for about 3 weeks.

      This site has several static routes, as it connects to several remote sites via a VPN gateway (a separate box).

      Today, I setup a point-to-point antenna to two sites that are nearby.  So I changed the static route for the first site from the VPN gateway to the antenna gateway.  Everything worked great!

      Then I changed the second static route for the second site from the VPN gateway to the antenna gateway and…

      My entire local network died.  Everything went offline, and I could barely communicate with other devices on the local network, much less on the internet or at the remote sites.  It was as if my entire network was being internally DDOSed.

      I experienced a similar symptom when messing with static routes in the same post I referenced above.

      But the worst thing about this one is: even after completely deleting the two static routes in question, and rebooting pfsense, the internal flood continued.

      The log shows a bunch of dpinger errors with code 55 to the antenna gateway in question, but I've tested with pfsense off, and the antenna gateway is fine.  And also all my other devices come back online when the pfsense box is off.

      This is a really terrible bug because it seems it could strike at any time without warning, and bring down an entire network.  It would be catastrophic if this occurred while I was making changes to pfsense from a remote location.

      Looks like I'm going to have to reinstall pfSense from scratch, again.

      1 Reply Last reply Reply Quote 0
      • Derelict
        Derelict LAYER 8 Netgate last edited by

        Sounds like you somehow created a loop.

        Probably not a bug. Probably a mistake.

        Without more details it is impossible to know.

        Please create a comprehensive network diagram detailing the starting state of the network, desired ending state of the network, and the exact steps you took.

        Chattanooga, Tennessee, USA
        The pfSense Book is free of charge!
        DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • Z
          zippydan last edited by

          It's not a loop.

          Site 1:
          pfsense router interfaces:
          internet gateway -> internet connection
          vpn gateway -> vpn router
          antenna gateway -> antenna to Site 2

          VPN router -> VPN router at Site 2

          Site 2:
          pfsense router interfaces:
          internet gateway -> internet connection
          vpn gateway -> vpn router
          antenna gateway 1 -> antenna to Site 1
          antenna gateway 2 -> antenna to Site 3

          VPN router -> VPN router at Site 1

          Site 3:
          pfsense router interfaces:
          antenna gateway -> antenna to Site 2

          Static Routes:
          Site 1: to Site 2 via vpn gateway, to Site 3 via vpn gateway
          Site 2: to Site 1 via vpn gateway, to Site 3 via antenna gateway 2
          Site 3: to Site 1 via antenna gateway, to Site 2 via antenna gateway

          I changed Site 1 Static Routes to:
          to Site 2 via antenna gateway <- worked great!
          to Site 3 via antenna gateway <- network death

          Did you even check my link in my original post? I also experienced a network storm with a perfectly valid static route config at a different site also using pfsense.  I had to reinstall pfsense completely to get it working again, with the exact same config.

          Also, everything is working fine now.  Guess what I did?

          Well, I left the machine on all night, network storm and all.  Didn't change any config on the pfsense box, or anywhere else on the network for that matter.

          I added some NICs to the virtual machine (my intent was to switch the existing VMware E1000 NICs to VMXNET 3 NICs, as I've read on the forums that E1000 can be problematic).

          So I shut down the machine, added the VMXNET NICs to the VM, restarted and … before I could even change anything in the pfsense config, everything was working fine this morning.

          Again, the configuration as is works just fine as long as pfsense decides to behave.  And this is the second time I have experienced a network storm when using a straight-forward routing table

          So, I'm sure there is no loop.  I was able to ping just fine from Site 1 to both Site 2 and Site 3 using the same configuration that borked my network last night.  I don't think it is specifically a configuration problem. I think it is a pfsense bug, somewhere deep in the network code.

          I have since changed all the interaces from E1000 to VMXNET 3, despite the fact that everything was working, just in case.  Note again that I did not make any other configuration changes and yet it all works fine.  We'll see if any problems return after the change of adapter type.

          1 Reply Last reply Reply Quote 0
          • Derelict
            Derelict LAYER 8 Netgate last edited by

            Layer 3 routing loops generally do not bring down networks thanks to the TTL.

            Layer 2 loops do.

            When you keep getting the same bad results nobody else is getting it's probably something unique you are doing.

            Do it again and take a packet capture.

            What you posted is not a diagram. What is in my sig is a diagram.

            Chattanooga, Tennessee, USA
            The pfSense Book is free of charge!
            DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • Z
              zippydan last edited by

              I think, possibly, the problem was arising from an error in my VLAN configuration.  Thank you for encouraging me to examine my network in more detail.  In the end, I don't think it was a problem with the routes themselves, but rather with an error on my switch with the VLANs.

              I'm not 100% sure yet, but everything is working for now and I will see how it goes.

              1 Reply Last reply Reply Quote 0
              • First post
                Last post