Traffic exiting IPSEC not routed or dropped

Derelict

Sounds like the other side doesn't have the proper routes back to you.

Hmm...

Running a bunch of tcpdumps, I see an ICMP request come in from my LAN, traverse the ipsec interface, then I see an ESP out and an ESP in and an ICMP reply packet exiting the ipsec interface.

But it never exits the LAN interface.

Post a packet capture of said traffic. Take a good look at it and see if you can see where it is going. Are you SURE the reply never leaves the WAN? It's going someplace...

jeremy_mordkoff

IPSec IPs are 172.17.49.1 (far end) and 172.17.49.2 (local end)
local LAN is 10.64.0.0/16
far LAN is 10.103.0.0/16

my trace program:

#!/bin/sh 

trap 'pkill tcpdump' 2
tcpdump -i ipsec8000 -nn &
tcpdump -i igb0 host 60.248.18.200 -nn & 
tcpdump -i igb1 -nn net 10.103.0.0/16  or net 172.17.49.0/24

first ... ping far end's IPSEC IP -- LAN in, IPSEC out, ESP out, ESP in, IPSEC in, LAN out :

09:11:26.520815 IP 10.64.10.77 > 172.17.49.1: ICMP echo request, id 1812, seq 6, length 64
09:11:26.520833 IP 10.64.10.77 > 172.17.49.1: ICMP echo request, id 1812, seq 6, length 64
09:11:26.520860 IP 74.118.73.228 > 60.248.18.200: ESP(spi=0xc38716e3,seq=0x224), length 132
09:11:26.721719 IP 60.248.18.200 > 74.118.73.228: ESP(spi=0xc2898c55,seq=0x2b7), length 132
09:11:26.721747 IP 172.17.49.1 > 10.64.10.77: ICMP echo reply, id 1812, seq 6, length 64
09:11:26.721778 IP 172.17.49.1 > 10.64.10.77: ICMP echo reply, id 1812, seq 6, length 64

ping a host on the far end LAN -- exactly the same except the LAM out is missing ... seems like something is doing src IP filtering but I cannot find it

09:11:51.813801 IP 10.64.10.77 > 10.103.24.14: ICMP echo request, id 2583, seq 8, length 64
09:11:51.813771 IP 10.64.10.77 > 10.103.24.14: ICMP echo request, id 2583, seq 8, length 64
09:11:51.813818 IP 74.118.73.228 > 60.248.18.200: ESP(spi=0xc38716e3,seq=0x23b), length 132
09:11:52.014746 IP 60.248.18.200 > 74.118.73.228: ESP(spi=0xc2898c55,seq=0x2ce), length 132
09:11:52.014782 IP 10.103.24.14 > 10.64.10.77: ICMP echo reply, id 2583, seq 8, length 64

jeremy_mordkoff

I am stuck. I have spent many days on this now and I'm ready to string a wire to Taiwan myself as I think that will be faster

My set up:

HOST 10.64.10.77 <--> FW 172.17.49.2 <= VTI tunnel=> 172.17.49.1 FW2 <---> HOST 10.103.24.14

172.17.49 is the ipsec network

These ALL work
FW1 -> Fw2:: FW1$ ping -S 172.17.49.2 172.17.49.1
FW1 -> Far host:: FW1$ ping -S 172.17.49.2 10.103.24.14
local host to far FW:: HOST1$ ping 172.17.49.1

But host to host fails
HOST1$ ping 10.103.24.14

I DO see replies coming out of ipsec8000:

12:49:02.895722 IP (tos 0x0, ttl 64, id 20800, offset 0, flags [DF], proto ICMP (1), length 84)
10.64.10.77 > 10.103.24.14: ICMP echo request, id 63818, seq 1, length 64
12:49:03.096792 IP (tos 0x0, ttl 63, id 41223, offset 0, flags [none], proto ICMP (1), length 84)
10.103.24.14 > 10.64.10.77: ICMP echo reply, id 63818, seq 1, length 64

But they are not sent out my LAN interface, in fact I can't tell where they go

I looked at the rules, and ipsec8000 seems wrong. There are no pass rules --

[2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: pfctl -sr | egrep 'enc0|ipsec|Default deny rule IPv4'
scrub on ipsec8000 all fragment reassemble
anchor "ipsec/*" all
block drop in inet all label "Default deny rule IPv4"
block drop out inet all label "Default deny rule IPv4"
block drop in on ! ipsec8000 inet from 172.17.49.0/30 to any
block drop in on ipsec8000 inet6 from fe80::21e:67ff:fe4d:bd98 to any
pass out on enc0 all flags S/SA keep state label "IPsec internal host to host"
pass in quick on enc0 inet all flags S/SA keep state label "USER_RULE"
pass in quick on enc0 inet proto icmp all keep state label "USER_RULE"
pass in log quick on enc0 inet all flags S/SA keep state label "USER_RULE"
pass in log quick on enc0 inet6 all flags S/SA keep state label "USER_RULE"
pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE"

easyrule seems to be incapable of adding rules to ipsec8000 ... it adds them to enc0 instead. Somewhere I thought I read that I needed to add rules to both interfaces but I'll be damned if I can figure out how to add them to ipsec8000

Also, I get no outbound traffic on ipsec8000 unless I add a policy route on my LAN interface pointing to the far FW's IPSEC address even though I also have a static route pointing to it. Why do I need both?

This is a production router with 3 active IPSEC tunnels and it is an openvpn server. This is the first time I've tried VTI tunnels.

PLEASE any ideas what to do next? Do I start hacking up easyrule to add rules to ipsec8000?

Derelict

Even if you get a rule on ipsec8000 it will not be processed. That you cannot add rules to the VTI interface is intentional.

https://redmine.pfsense.org/issues/8686

You do not need a rule to process reply traffic. You only need a rule if the connection is initiated from an IPsec endpoint into the local firewall.

Is the firewall logging any blocks for source 10.103.24.14?

Anything special about this node? Is it in AWS, Azure, VM, etc? (probably not with an igb interface).

How are these set?

System > Advanced, Networking:

Hardware Checksum Offloading
Hardware TCP Segmentation Offloading
Hardware Large Receive Offloading

jeremy_mordkoff

Thank you so much for taking the time to reply

There are no blocks logged

This is running on bare metal and has been stable for a year with 3 IPSEC site to site tunnels (one phase 2 entry each) and an openvpn server. I did upgrade recently, but it was a small step. Maybe I should delete the entire VTI tunnel and start over?

Hardware Checksum Offloading NOT checked
Hardware TCP Segmentation Offloading CHECKED (i.e. disabled)
Hardware Large Receive Offloading CHECKED (i.e. disabled)

Any thoughts on why a static route was insufficient and a policy route on the LAN interface (igb1) was the only way to make this work?

pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE"

Without this rule I do get a drop message even though I have a route:

[2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: netstat -nr | grep 172.17
10.103.0.0/16      172.17.49.1        UGS    ipsec800
172.17.49.1        link#23            UH     ipsec800
172.17.49.2        link#23            UHS         lo0

jeremy_mordkoff

Okay, I just realized something

one of my existing IPSEC phase 2 tunnels has a dest 10.96.0.0/11 which overlaps this subnet (10.103.0.0/16)

Any ideas how I can prove this theory or bypass this problem? I guess I could replace that tunnel's phase2 entries with some that don't include 10.103

Derelict

An IPsec policy will get in the way every time. You should not have a policy that matches traffic you don't want to match. But in order for the traffic to match the policy the source would also need to match the policy. That would certainly explain the reply traffic "vanishing."

Static routes, policy routes. They both generally do the same in this case.

You are kind of bouncing back and forth between several different scenarios and solutions and I am having a hard time following exactly what is working and what is not.

Derelict

In fact, if the policy matches a static route would not work. Policy routing would be needed to bypass the IPsec policy and send the traffic in the desired direction. Policy routing would not be in place on the reply traffic, though.

jeremy_mordkoff

That was it

I replaced my one phase 2 with a remote net of 10.96/11 with (4) /16's and now 10.103 via the new tunnel works and I no longer need the policy route

Sorry for wasting your time, and many thanks for taking the time to read and reply

I guess I assumed the most specific route would apply

Derelict

IPsec policies match before the routing table.

Policy routing happens before IPsec policies and the routing table. But only when the connection is initially established into that interface.

Glad you found it.