Traffic exiting IPSEC not routed or dropped
-
Version 2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:17 EDT 2020
FreeBSD 11.3-STABLEI have a VTI tunnel routing a single subnet to a single subnet.
I have both a policy-route on my LAN interface and a static route, both pointing the ipsec IP on the far end.
From the FW I can ping the far end firewall or a device behind it if I set my source-IP to my own ipsec IP
But I cannot ping the far end OR the device behind it from the LAN
Running a bunch of tcpdumps, I see an ICMP request come in from my LAN, traverse the ipsec interface, then I see an ESP out and an ESP in and an ICMP reply packet exiting the ipsec interface.
But it never exits the LAN interface. Since everyday traffic routes fine between my WAN and my non-VTI IPSEC tunnels, I don't think it can be a routing issue. My next thought was a firewall issue, but the rules in the GUI are wide open. But when I view the rules using pfctl -sa | grep ipsec or pfctl -sa | grep <subnet> I do not see my rules:
[2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: pfctl -sa | grep 10.103 pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE" igb1 icmp 10.103.24.14:1217 <- 10.64.10.77:1217 0:0 ipsec8000 icmp 10.64.10.77:1217 -> 10.103.24.14:1217 0:0 [2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: pfctl -sa | grep ipsec scrub on ipsec8000 all fragment reassemble anchor "ipsec/*" all block drop in on ! ipsec8000 inet from 172.17.49.0/30 to any block drop in on ipsec8000 inet6 from fe80::21e:67ff:fe4d:bd98 to any pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE" ipsec8000 icmp 10.64.10.77:1217 -> 10.103.24.14:1217 0:0Do these rules look correct?
Also easyrule fails to update ipsec8000
-
Sounds like the other side doesn't have the proper routes back to you.
Hmm...
Running a bunch of tcpdumps, I see an ICMP request come in from my LAN, traverse the ipsec interface, then I see an ESP out and an ESP in and an ICMP reply packet exiting the ipsec interface.
But it never exits the LAN interface.
Post a packet capture of said traffic. Take a good look at it and see if you can see where it is going. Are you SURE the reply never leaves the WAN? It's going someplace...
-
IPSec IPs are 172.17.49.1 (far end) and 172.17.49.2 (local end)
local LAN is 10.64.0.0/16
far LAN is 10.103.0.0/16my trace program:
#!/bin/sh trap 'pkill tcpdump' 2 tcpdump -i ipsec8000 -nn & tcpdump -i igb0 host 60.248.18.200 -nn & tcpdump -i igb1 -nn net 10.103.0.0/16 or net 172.17.49.0/24first ... ping far end's IPSEC IP -- LAN in, IPSEC out, ESP out, ESP in, IPSEC in, LAN out :
09:11:26.520815 IP 10.64.10.77 > 172.17.49.1: ICMP echo request, id 1812, seq 6, length 64
09:11:26.520833 IP 10.64.10.77 > 172.17.49.1: ICMP echo request, id 1812, seq 6, length 64
09:11:26.520860 IP 74.118.73.228 > 60.248.18.200: ESP(spi=0xc38716e3,seq=0x224), length 132
09:11:26.721719 IP 60.248.18.200 > 74.118.73.228: ESP(spi=0xc2898c55,seq=0x2b7), length 132
09:11:26.721747 IP 172.17.49.1 > 10.64.10.77: ICMP echo reply, id 1812, seq 6, length 64
09:11:26.721778 IP 172.17.49.1 > 10.64.10.77: ICMP echo reply, id 1812, seq 6, length 64ping a host on the far end LAN -- exactly the same except the LAM out is missing ... seems like something is doing src IP filtering but I cannot find it
09:11:51.813801 IP 10.64.10.77 > 10.103.24.14: ICMP echo request, id 2583, seq 8, length 64
09:11:51.813771 IP 10.64.10.77 > 10.103.24.14: ICMP echo request, id 2583, seq 8, length 64
09:11:51.813818 IP 74.118.73.228 > 60.248.18.200: ESP(spi=0xc38716e3,seq=0x23b), length 132
09:11:52.014746 IP 60.248.18.200 > 74.118.73.228: ESP(spi=0xc2898c55,seq=0x2ce), length 132
09:11:52.014782 IP 10.103.24.14 > 10.64.10.77: ICMP echo reply, id 2583, seq 8, length 64 -
I am stuck. I have spent many days on this now and I'm ready to string a wire to Taiwan myself as I think that will be faster
My set up:
HOST 10.64.10.77 <--> FW 172.17.49.2 <= VTI tunnel=> 172.17.49.1 FW2 <---> HOST 10.103.24.14
172.17.49 is the ipsec network
These ALL work
FW1 -> Fw2:: FW1$ ping -S 172.17.49.2 172.17.49.1
FW1 -> Far host:: FW1$ ping -S 172.17.49.2 10.103.24.14
local host to far FW:: HOST1$ ping 172.17.49.1But host to host fails
HOST1$ ping 10.103.24.14I DO see replies coming out of ipsec8000:
12:49:02.895722 IP (tos 0x0, ttl 64, id 20800, offset 0, flags [DF], proto ICMP (1), length 84)
10.64.10.77 > 10.103.24.14: ICMP echo request, id 63818, seq 1, length 64
12:49:03.096792 IP (tos 0x0, ttl 63, id 41223, offset 0, flags [none], proto ICMP (1), length 84)
10.103.24.14 > 10.64.10.77: ICMP echo reply, id 63818, seq 1, length 64But they are not sent out my LAN interface, in fact I can't tell where they go
I looked at the rules, and ipsec8000 seems wrong. There are no pass rules --
[2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: pfctl -sr | egrep 'enc0|ipsec|Default deny rule IPv4'
scrub on ipsec8000 all fragment reassemble
anchor "ipsec/*" all
block drop in inet all label "Default deny rule IPv4"
block drop out inet all label "Default deny rule IPv4"
block drop in on ! ipsec8000 inet from 172.17.49.0/30 to any
block drop in on ipsec8000 inet6 from fe80::21e:67ff:fe4d:bd98 to any
pass out on enc0 all flags S/SA keep state label "IPsec internal host to host"
pass in quick on enc0 inet all flags S/SA keep state label "USER_RULE"
pass in quick on enc0 inet proto icmp all keep state label "USER_RULE"
pass in log quick on enc0 inet all flags S/SA keep state label "USER_RULE"
pass in log quick on enc0 inet6 all flags S/SA keep state label "USER_RULE"
pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE"easyrule seems to be incapable of adding rules to ipsec8000 ... it adds them to enc0 instead. Somewhere I thought I read that I needed to add rules to both interfaces but I'll be damned if I can figure out how to add them to ipsec8000
Also, I get no outbound traffic on ipsec8000 unless I add a policy route on my LAN interface pointing to the far FW's IPSEC address even though I also have a static route pointing to it. Why do I need both?
This is a production router with 3 active IPSEC tunnels and it is an openvpn server. This is the first time I've tried VTI tunnels.
PLEASE any ideas what to do next? Do I start hacking up easyrule to add rules to ipsec8000?
-
Even if you get a rule on ipsec8000 it will not be processed. That you cannot add rules to the VTI interface is intentional.
https://redmine.pfsense.org/issues/8686
You do not need a rule to process reply traffic. You only need a rule if the connection is initiated from an IPsec endpoint into the local firewall.
Is the firewall logging any blocks for source 10.103.24.14?
Anything special about this node? Is it in AWS, Azure, VM, etc? (probably not with an igb interface).
How are these set?
System > Advanced, Networking:
Hardware Checksum Offloading
Hardware TCP Segmentation Offloading
Hardware Large Receive Offloading -
Thank you so much for taking the time to reply
There are no blocks logged
This is running on bare metal and has been stable for a year with 3 IPSEC site to site tunnels (one phase 2 entry each) and an openvpn server. I did upgrade recently, but it was a small step. Maybe I should delete the entire VTI tunnel and start over?
Hardware Checksum Offloading NOT checked
Hardware TCP Segmentation Offloading CHECKED (i.e. disabled)
Hardware Large Receive Offloading CHECKED (i.e. disabled)Any thoughts on why a static route was insufficient and a policy route on the LAN interface (igb1) was the only way to make this work?
pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE"
Without this rule I do get a drop message even though I have a route:
[2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: netstat -nr | grep 172.17 10.103.0.0/16 172.17.49.1 UGS ipsec800 172.17.49.1 link#23 UH ipsec800 172.17.49.2 link#23 UHS lo0 -
Okay, I just realized something
one of my existing IPSEC phase 2 tunnels has a dest 10.96.0.0/11 which overlaps this subnet (10.103.0.0/16)
Any ideas how I can prove this theory or bypass this problem? I guess I could replace that tunnel's phase2 entries with some that don't include 10.103
-
An IPsec policy will get in the way every time. You should not have a policy that matches traffic you don't want to match. But in order for the traffic to match the policy the source would also need to match the policy. That would certainly explain the reply traffic "vanishing."
Static routes, policy routes. They both generally do the same in this case.
You are kind of bouncing back and forth between several different scenarios and solutions and I am having a hard time following exactly what is working and what is not.
-
In fact, if the policy matches a static route would not work. Policy routing would be needed to bypass the IPsec policy and send the traffic in the desired direction. Policy routing would not be in place on the reply traffic, though.
-
That was it
I replaced my one phase 2 with a remote net of 10.96/11 with (4) /16's and now 10.103 via the new tunnel works and I no longer need the policy route
Sorry for wasting your time, and many thanks for taking the time to read and reply
I guess I assumed the most specific route would apply
-
IPsec policies match before the routing table.
Policy routing happens before IPsec policies and the routing table. But only when the connection is initially established into that interface.
Glad you found it.
