Traffic exiting IPSEC not routed or dropped



  • Version 2.4.5-RELEASE-p1 (amd64)
    built on Tue Jun 02 17:51:17 EDT 2020
    FreeBSD 11.3-STABLE

    I have a VTI tunnel routing a single subnet to a single subnet.

    I have both a policy-route on my LAN interface and a static route, both pointing the ipsec IP on the far end.

    From the FW I can ping the far end firewall or a device behind it if I set my source-IP to my own ipsec IP

    But I cannot ping the far end OR the device behind it from the LAN

    Running a bunch of tcpdumps, I see an ICMP request come in from my LAN, traverse the ipsec interface, then I see an ESP out and an ESP in and an ICMP reply packet exiting the ipsec interface.

    But it never exits the LAN interface. Since everyday traffic routes fine between my WAN and my non-VTI IPSEC tunnels, I don't think it can be a routing issue. My next thought was a firewall issue, but the rules in the GUI are wide open. But when I view the rules using pfctl -sa | grep ipsec or pfctl -sa | grep <subnet> I do not see my rules:

    [2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: pfctl -sa | grep 10.103
    pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE"
    igb1 icmp 10.103.24.14:1217 <- 10.64.10.77:1217       0:0
    ipsec8000 icmp 10.64.10.77:1217 -> 10.103.24.14:1217       0:0
    
    [2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: pfctl -sa | grep ipsec 
    scrub on ipsec8000 all fragment reassemble
    anchor "ipsec/*" all
    block drop in on ! ipsec8000 inet from 172.17.49.0/30 to any
    block drop in on ipsec8000 inet6 from fe80::21e:67ff:fe4d:bd98 to any
    pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE"
    ipsec8000 icmp 10.64.10.77:1217 -> 10.103.24.14:1217       0:0
    
    

    Do these rules look correct?

    Also easyrule fails to update ipsec8000


  • LAYER 8 Netgate

    Sounds like the other side doesn't have the proper routes back to you.

    Hmm...

    Running a bunch of tcpdumps, I see an ICMP request come in from my LAN, traverse the ipsec interface, then I see an ESP out and an ESP in and an ICMP reply packet exiting the ipsec interface.

    But it never exits the LAN interface.

    Post a packet capture of said traffic. Take a good look at it and see if you can see where it is going. Are you SURE the reply never leaves the WAN? It's going someplace...



  • IPSec IPs are 172.17.49.1 (far end) and 172.17.49.2 (local end)
    local LAN is 10.64.0.0/16
    far LAN is 10.103.0.0/16

    my trace program:

    #!/bin/sh 
    
    trap 'pkill tcpdump' 2
    tcpdump -i ipsec8000 -nn &
    tcpdump -i igb0 host 60.248.18.200 -nn & 
    tcpdump -i igb1 -nn net 10.103.0.0/16  or net 172.17.49.0/24
    

    first ... ping far end's IPSEC IP -- LAN in, IPSEC out, ESP out, ESP in, IPSEC in, LAN out :

    09:11:26.520815 IP 10.64.10.77 > 172.17.49.1: ICMP echo request, id 1812, seq 6, length 64
    09:11:26.520833 IP 10.64.10.77 > 172.17.49.1: ICMP echo request, id 1812, seq 6, length 64
    09:11:26.520860 IP 74.118.73.228 > 60.248.18.200: ESP(spi=0xc38716e3,seq=0x224), length 132
    09:11:26.721719 IP 60.248.18.200 > 74.118.73.228: ESP(spi=0xc2898c55,seq=0x2b7), length 132
    09:11:26.721747 IP 172.17.49.1 > 10.64.10.77: ICMP echo reply, id 1812, seq 6, length 64
    09:11:26.721778 IP 172.17.49.1 > 10.64.10.77: ICMP echo reply, id 1812, seq 6, length 64

    ping a host on the far end LAN -- exactly the same except the LAM out is missing ... seems like something is doing src IP filtering but I cannot find it

    09:11:51.813801 IP 10.64.10.77 > 10.103.24.14: ICMP echo request, id 2583, seq 8, length 64
    09:11:51.813771 IP 10.64.10.77 > 10.103.24.14: ICMP echo request, id 2583, seq 8, length 64
    09:11:51.813818 IP 74.118.73.228 > 60.248.18.200: ESP(spi=0xc38716e3,seq=0x23b), length 132
    09:11:52.014746 IP 60.248.18.200 > 74.118.73.228: ESP(spi=0xc2898c55,seq=0x2ce), length 132
    09:11:52.014782 IP 10.103.24.14 > 10.64.10.77: ICMP echo reply, id 2583, seq 8, length 64



  • I am stuck. I have spent many days on this now and I'm ready to string a wire to Taiwan myself as I think that will be faster

    My set up:

    HOST 10.64.10.77 <--> FW 172.17.49.2 <= VTI tunnel=> 172.17.49.1 FW2 <---> HOST 10.103.24.14

    172.17.49 is the ipsec network

    These ALL work
    FW1 -> Fw2:: FW1$ ping -S 172.17.49.2 172.17.49.1
    FW1 -> Far host:: FW1$ ping -S 172.17.49.2 10.103.24.14
    local host to far FW:: HOST1$ ping 172.17.49.1

    But host to host fails
    HOST1$ ping 10.103.24.14

    I DO see replies coming out of ipsec8000:

    12:49:02.895722 IP (tos 0x0, ttl 64, id 20800, offset 0, flags [DF], proto ICMP (1), length 84)
    10.64.10.77 > 10.103.24.14: ICMP echo request, id 63818, seq 1, length 64
    12:49:03.096792 IP (tos 0x0, ttl 63, id 41223, offset 0, flags [none], proto ICMP (1), length 84)
    10.103.24.14 > 10.64.10.77: ICMP echo reply, id 63818, seq 1, length 64

    But they are not sent out my LAN interface, in fact I can't tell where they go

    I looked at the rules, and ipsec8000 seems wrong. There are no pass rules --

    [2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: pfctl -sr | egrep 'enc0|ipsec|Default deny rule IPv4'
    scrub on ipsec8000 all fragment reassemble
    anchor "ipsec/*" all
    block drop in inet all label "Default deny rule IPv4"
    block drop out inet all label "Default deny rule IPv4"
    block drop in on ! ipsec8000 inet from 172.17.49.0/30 to any
    block drop in on ipsec8000 inet6 from fe80::21e:67ff:fe4d:bd98 to any
    pass out on enc0 all flags S/SA keep state label "IPsec internal host to host"
    pass in quick on enc0 inet all flags S/SA keep state label "USER_RULE"
    pass in quick on enc0 inet proto icmp all keep state label "USER_RULE"
    pass in log quick on enc0 inet all flags S/SA keep state label "USER_RULE"
    pass in log quick on enc0 inet6 all flags S/SA keep state label "USER_RULE"
    pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE"

    easyrule seems to be incapable of adding rules to ipsec8000 ... it adds them to enc0 instead. Somewhere I thought I read that I needed to add rules to both interfaces but I'll be damned if I can figure out how to add them to ipsec8000

    Also, I get no outbound traffic on ipsec8000 unless I add a policy route on my LAN interface pointing to the far FW's IPSEC address even though I also have a static route pointing to it. Why do I need both?

    This is a production router with 3 active IPSEC tunnels and it is an openvpn server. This is the first time I've tried VTI tunnels.

    PLEASE any ideas what to do next? Do I start hacking up easyrule to add rules to ipsec8000?


  • LAYER 8 Netgate

    Even if you get a rule on ipsec8000 it will not be processed. That you cannot add rules to the VTI interface is intentional.

    https://redmine.pfsense.org/issues/8686

    You do not need a rule to process reply traffic. You only need a rule if the connection is initiated from an IPsec endpoint into the local firewall.

    Is the firewall logging any blocks for source 10.103.24.14?

    Anything special about this node? Is it in AWS, Azure, VM, etc? (probably not with an igb interface).

    How are these set?

    System > Advanced, Networking:

    Hardware Checksum Offloading
    Hardware TCP Segmentation Offloading
    Hardware Large Receive Offloading



  • Thank you so much for taking the time to reply

    There are no blocks logged

    This is running on bare metal and has been stable for a year with 3 IPSEC site to site tunnels (one phase 2 entry each) and an openvpn server. I did upgrade recently, but it was a small step. Maybe I should delete the entire VTI tunnel and start over?

    Hardware Checksum Offloading NOT checked
    Hardware TCP Segmentation Offloading CHECKED (i.e. disabled)
    Hardware Large Receive Offloading CHECKED (i.e. disabled)

    Any thoughts on why a static route was insufficient and a policy route on the LAN interface (igb1) was the only way to make this work?

    pass in quick on igb1 route-to (ipsec8000 172.17.49.1) inet from any to 10.103.0.0/16 flags S/SA keep state label "USER_RULE"

    Without this rule I do get a drop message even though I have a route:

    [2.4.5-RELEASE][admin@fw1.eng.riftio.com]/root: netstat -nr | grep 172.17
    10.103.0.0/16      172.17.49.1        UGS    ipsec800
    172.17.49.1        link#23            UH     ipsec800
    172.17.49.2        link#23            UHS         lo0
    


  • Okay, I just realized something

    one of my existing IPSEC phase 2 tunnels has a dest 10.96.0.0/11 which overlaps this subnet (10.103.0.0/16)

    Any ideas how I can prove this theory or bypass this problem? I guess I could replace that tunnel's phase2 entries with some that don't include 10.103


  • LAYER 8 Netgate

    An IPsec policy will get in the way every time. You should not have a policy that matches traffic you don't want to match. But in order for the traffic to match the policy the source would also need to match the policy. That would certainly explain the reply traffic "vanishing."

    Static routes, policy routes. They both generally do the same in this case.

    You are kind of bouncing back and forth between several different scenarios and solutions and I am having a hard time following exactly what is working and what is not.


  • LAYER 8 Netgate

    In fact, if the policy matches a static route would not work. Policy routing would be needed to bypass the IPsec policy and send the traffic in the desired direction. Policy routing would not be in place on the reply traffic, though.



  • That was it

    I replaced my one phase 2 with a remote net of 10.96/11 with (4) /16's and now 10.103 via the new tunnel works and I no longer need the policy route

    Sorry for wasting your time, and many thanks for taking the time to read and reply

    I guess I assumed the most specific route would apply


  • LAYER 8 Netgate

    IPsec policies match before the routing table.

    Policy routing happens before IPsec policies and the routing table. But only when the connection is initially established into that interface.

    Glad you found it.


Log in to reply