GRE Tunnel Can't Reach Site 3 in one direction only

helloadam

Hello,

I am having a strange issue that I can not figure out and I am hoping someone on the forums can help me.

I have three sites running 2.6.0 of pfSense CE and I can not get vm-02 to communicate with vm-03. However vm-02 can communicate with vm-03. Communication only works in one direction!

All firewalls have allow any rules configured and they all have the same route tables. Everything is configured with OSPF for route distributions.

See attached picture of the network setup:

If we do a traceroute from vm-02 to vm-03, packets fails when it reaches fwl-01 which would be the remote site of the GRE tunnel from fwl-02.

However vm-01 and vm-02 can communicate with each other just fine in both directions over the GRE tunnel! vm-01 and vm-03 can also communicate without issue in both directions. It is only when we create a third site that communication only works in one direction, always failing when fwl-02 starts the communication.

Should I be using a GIF tunnel instead of GRE? Is there a kernel setting I need to enable to allow routing for GRE tunnels? Anything else I should do for debugging?

# from vm-02 to vmd-03
# traceroute 10.183.30.10
traceroute to 10.183.30.10 (10.183.30.10), 30 hops max, 60 byte packets
 1  10.83.50.1 (10.83.50.1)  0.359 ms  0.340 ms  0.325 ms
 2  10.8.255.1 (10.8.255.1)  0.299 ms  0.283 ms  0.264 ms
 3  * * *

# from vm-03 to vmd-02
# traceroute 10.83.50.44
traceroute to 10.83.50.44 (10.83.50.44), 30 hops max, 60 byte packets
 1  10.183.30.1 (10.183.30.1)  0.395 ms  0.370 ms  0.350 ms
 2  10.183.255.2 (10.183.255.2)  145.531 ms  145.524 ms  145.497 ms
 3  10.8.255.2 (10.8.255.2)  145.693 ms  145.683 ms  145.667 ms
 4  10.83.50.44 (10.83.50.44)  145.736 ms  145.725 ms  145.707 ms

helloadam

Update:
Switching to a GIF tunnel vs GRE makes no difference. I get the same issues as before - vm-02 can not ping/traceroute to vm-03 however vm-03 can ping/traceroute to vm-02.

The route tables on fwl-01, fwl-02 and fwl-03 are all the same in that they have FRR routes via OSPF pointing to the correct gateway. I event added static routes on all three firewalls but still have the same issue.

I suspect fwl-01 does needs some sort of setting adjusted (maybe system tunable) to allow packets to be routed? But that wouldn't explain the one-way communication that is currently happening.

Anyone have any suggestions on how I can debug this further?

helloadam

Fixed the issue: I had to enable Filter IPsec VTI and Transport on assigned interfaces, block all tunnel mode traffic under VPN -> IPsec -> Advanced Settings on fwl-01 and fwl-03

Documentation here:
https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/advanced.html#advanced-ipsec-settings