Simple Policy Route results in routing loop (TTL Expired in Transit)

[alexsim2004]

Hi All, this simple routing scenario has got me a bit stumped and I can't see why it doesn't quite work as expected. Assume this simple Scenario:
pfSense Interfaces:
em0 (WAN, Default Gateway, Public IP)
em1 (LAN): 192.168.0.1/24
em2 (Cisco L3 Switch ): 192.168.3.1/24

pfSense Gateways
[Public IP] (WAN Gateway)
192.168.0.1 (LAN Gateway)
192.168.3.2 (Cisco L3 Switch)

Cisco L3 switch:
gi1/1: 192.168.3.2 (connected directly to em2 on pfSense)
Default gateway: 192.168.3.1
No NAT configured at all on the switch.

Policy Based Routing rule on pfSense:
Interface: LAN em1
Protocol: any
Source: 192.168.0.2 (my PC)
Destination: any
Gateway: Cisco L3 Switch (192.168.3.2)

I also added an 'allow all' rule on the em2 Cisco-facing interface to ensure that traffic is allowed from Cisco Switch > Anywhere.

I basically want to route my PC's (192.168.0.2) traffic via Policy Based Routing to the Cisco L3 switch, which should then route it straight back to pfSense via the same gi1/1 interface via the default gateway of 192.168.3.1 and out to the Internet (almost like router-on-a-stick).
However I'm currently getting 'TTL expired in transit' messages when trying to ping a remote IP address such as 8.8.8.8. Suggesting that there may be a routing loop occuring somewhere, or it's matching the PBR firewall rule twice or something.

A traceroute to 8.8.8.8 results in the Cisco's IP address repeating for every hop like:
192.168.3.2
192.168.3.2
192.168.3.2
…

Of course everything works fine without the Policy rule in place.

Any ideas on where I'm going wrong?  :)
Thanks in Advance.

[heper]

you could start by removing '192.168.0.1 (LAN Gateway)'

you should "never' have to create a gateway for directly connected networks

[johnpoz]

"which should then route it straight back to pfSense via the same gi1/1 interface via the default gateway of 192.168.3.1 and out to the Internet (almost like router-on-a-stick)."

For what possible reason?  Is your L3 switch also going to nat this traffic to its 192.168.3 address?  If not you have asymmetrical routing..

So you send traffic to L3 just to get sent back to pfsense to go to internet, return traffic from your nat 192.168.0/24 so where does pfsense send traffic for that network that is directly attached.. Why would it send it back to your L3?

Not to forget this hairpin for no reason either.

[alexsim2004]

@heper:

you could start by removing '192.168.0.1 (LAN Gateway)'

you should "never' have to create a gateway for directly connected networks

Thanks, I've now removed that gateway, I think this was from when I was first setting up pfSense as a slight newbie :)

@johnpoz:

For what possible reason?  Is your L3 switch also going to nat this traffic to its 192.168.3 address?  If not you have asymmetrical routing..

I agree it doesn't quite make sense - I had simplified the requirement when I made the original post. The backstory is that I want a Squid proxy server connected to the Cisco L3 switch and then use the WCCP protocol to redirect HTTP traffic to the Squid proxy.
This works fine using Policy based Routing. But when the Squid proxy is offline, I want the Cisco L3 switch to route traffic back to pfSense (so users can browse the Internet as normal).

Your diagram shows my intended layout correctly - thank you. Yes I did wonder about Asymmetric routing, today I tried creating a new firewall rule on the WAN interface to try and catch the return traffic (Source: Any, Destination: 192.168.0.2, Gateway: Cisco) but I think the routing loop problem remains.

[johnpoz]

Why would your proxy be down??  If your proxy is down then internet should be down.  I assume your proxy does filtering, etc.  So if you just send traffic out the internet with no filtering you have no protection users surfing porn vs working, etc. etc..  Its madhouse I tell you a madhouse ;)

If you don't want internet to go down then setup your proxies in HA, etc.