Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Simple Policy Route results in routing loop (TTL Expired in Transit)

    Scheduled Pinned Locked Moved Routing and Multi WAN
    5 Posts 3 Posters 7.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      alexsim2004
      last edited by

      Hi All, this simple routing scenario has got me a bit stumped and I can't see why it doesn't quite work as expected. Assume this simple Scenario:
      pfSense Interfaces:
      em0 (WAN, Default Gateway, Public IP)
      em1 (LAN): 192.168.0.1/24
      em2 (Cisco L3 Switch ): 192.168.3.1/24

      pfSense Gateways
      [Public IP] (WAN Gateway)
      192.168.0.1 (LAN Gateway)
      192.168.3.2 (Cisco L3 Switch)

      Cisco L3 switch:
      gi1/1: 192.168.3.2 (connected directly to em2 on pfSense)
      Default gateway: 192.168.3.1
      No NAT configured at all on the switch.

      Policy Based Routing rule on pfSense:
      Interface: LAN em1
      Protocol: any
      Source: 192.168.0.2 (my PC)
      Destination: any
      Gateway: Cisco L3 Switch (192.168.3.2)

      I also added an 'allow all' rule on the em2 Cisco-facing interface to ensure that traffic is allowed from Cisco Switch > Anywhere.

      I basically want to route my PC's (192.168.0.2) traffic via Policy Based Routing to the Cisco L3 switch, which should then route it straight back to pfSense via the same gi1/1 interface via the default gateway of 192.168.3.1 and out to the Internet (almost like router-on-a-stick).
      However I'm currently getting 'TTL expired in transit' messages when trying to ping a remote IP address such as 8.8.8.8. Suggesting that there may be a routing loop occuring somewhere, or it's matching the PBR firewall rule twice or something.

      A traceroute to 8.8.8.8 results in the Cisco's IP address repeating for every hop like:
      192.168.3.2
      192.168.3.2
      192.168.3.2
      …

      Of course everything works fine without the Policy rule in place.

      Any ideas on where I'm going wrong?  :)
      Thanks in Advance.

      1 Reply Last reply Reply Quote 0
      • H
        heper
        last edited by

        you could start by removing '192.168.0.1 (LAN Gateway)'

        you should "never' have to create a gateway for directly connected networks

        1 Reply Last reply Reply Quote 0
        • johnpozJ
          johnpoz LAYER 8 Global Moderator
          last edited by

          "which should then route it straight back to pfSense via the same gi1/1 interface via the default gateway of 192.168.3.1 and out to the Internet (almost like router-on-a-stick)."

          For what possible reason?  Is your L3 switch also going to nat this traffic to its 192.168.3 address?  If not you have asymmetrical routing..

          So you send traffic to L3 just to get sent back to pfsense to go to internet, return traffic from your nat 192.168.0/24 so where does pfsense send traffic for that network that is directly attached.. Why would it send it back to your L3?

          Not to forget this hairpin for no reason either.

          path.jpg
          path.jpg_thumb

          An intelligent man is sometimes forced to be drunk to spend time with his fools
          If you get confused: Listen to the Music Play
          Please don't Chat/PM me for help, unless mod related
          SG-4860 24.11 | Lab VMs 2.8, 24.11

          1 Reply Last reply Reply Quote 0
          • A
            alexsim2004
            last edited by

            @heper:

            you could start by removing '192.168.0.1 (LAN Gateway)'

            you should "never' have to create a gateway for directly connected networks

            Thanks, I've now removed that gateway, I think this was from when I was first setting up pfSense as a slight newbie :)

            @johnpoz:

            For what possible reason?  Is your L3 switch also going to nat this traffic to its 192.168.3 address?  If not you have asymmetrical routing..

            I agree it doesn't quite make sense - I had simplified the requirement when I made the original post. The backstory is that I want a Squid proxy server connected to the Cisco L3 switch and then use the WCCP protocol to redirect HTTP traffic to the Squid proxy.
            This works fine using Policy based Routing. But when the Squid proxy is offline, I want the Cisco L3 switch to route traffic back to pfSense (so users can browse the Internet as normal).

            Your diagram shows my intended layout correctly - thank you. Yes I did wonder about Asymmetric routing, today I tried creating a new firewall rule on the WAN interface to try and catch the return traffic (Source: Any, Destination: 192.168.0.2, Gateway: Cisco) but I think the routing loop problem remains.

            1 Reply Last reply Reply Quote 0
            • johnpozJ
              johnpoz LAYER 8 Global Moderator
              last edited by

              Why would your proxy be down??  If your proxy is down then internet should be down.  I assume your proxy does filtering, etc.  So if you just send traffic out the internet with no filtering you have no protection users surfing porn vs working, etc. etc..  Its madhouse I tell you a madhouse ;)

              If you don't want internet to go down then setup your proxies in HA, etc.

              An intelligent man is sometimes forced to be drunk to spend time with his fools
              If you get confused: Listen to the Music Play
              Please don't Chat/PM me for help, unless mod related
              SG-4860 24.11 | Lab VMs 2.8, 24.11

              1 Reply Last reply Reply Quote 0
              • First post
                Last post
              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.