Routed IPSEC not working

[janets]

I can't seem to get routed IPSEC working. I have IPSEC1000 interfaces on both sides, and have routes added. When I try to ping the remote LAN, I can see the traffic leave that router's IPSEC interface and I can see the inbound traffic on enc0 on the remote box. But I see absolutely nothing on the IPSEC1000 interface on the other machine, and my pings die. What am I doing wrong?

Side A:

Side B:

[Konstanti]

@janets Hi
Show or check the rules on the IPSEC interface

[janets]

#Konstanti IPv4 and !Pv6 any/any permits are on both side's IPSEC tab.

[Konstanti]

@janets
192.168.42.1 can ping 172.16.0.2 ?
10.4.96.XXX can ping 172.16.0.1 ??

[janets]

@Konstanti No, nothing can ping anything. If I try to ping 192.168.42.1 from the other side while running a tcpdump, I can see the packets go out the IPSEC interface. On the other side, I never see the packets on the IPSEC interface, however I do see them on the enc0 interface, so they are getting over there.

I just tried the same configs in the cloud on a couple of VMs and it worked. The firewall that doesn't work has 6 physical interfaces, while the VMs only had 1 + the IPSEC interface. That's really the only major difference I can see.

[Konstanti]

@janets

what I see is strange - 2 SA almost identical on both sides of the tunnel (only in one case there is no PFS key group , and in the other there is)

There should only be one SA
Try to restart the IPSEC

[janets]

@Konstanti Actually, the config in the cloud also didn't work. I pinged the wrong IP - dur, sorry, its 5:00 a.m. and I've been up all night. ;-)

I've rebooted the firewalls on both sides, and have the same problem.

[janets]

@Konstanti Got it to work. If Phase 1's internet protocol is set to IPv4, it breaks. If its set to Dual Stack, it works. Nothing else changed.

[janets]

It also works in just IPv6 mode. There is something about IPv4 and that config that breaks. It feels like a bug.

[janets]

Here's what a working vs. a non-working ifconfig looks like.

Working:

ipsec1000: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
tunnel inet6 2600:3c01::f03c:91ff:fe8c:6af1 --> 2001:19f0:8001:688:5400:2ff:fe2c:6deb
inet6 fe80::84b8:2eb3:a617:de8a%ipsec1000 prefixlen 64 scopeid 0x6
inet 172.16.0.1 --> 172.16.0.2 netmask 0xfffffffc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
reqid: 1000
groups: ipsec

Not working, configured for ipv4, no "tunnel" line in ifconfig.

    ipsec1000: flags=8011<UP,POINTOPOINT,MULTICAST> metric 0 mtu 1500
    inet6 fe80::2e0:67ff:fe13:6324%ipsec1000 prefixlen 64 tentative scopeid 0xb
    inet 10.50.50.1 --> 10.50.50.2 netmask 0xfffffffc
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    reqid: 1000
    groups: ipsec

[janets]

Now I can't get it working on IPv6 or IPv4. My working config stopped working. For grins, I tried 2.5.0-devel and have the same problem.

[janets]

@Konstanti

I got it working, but had to do a manual step on each side. Reading the FreeBSD docs, I figured I was missing the tunnel on my ipsec1000 interface:

ipsec1000: flags=8011<UP,POINTOPOINT,MULTICAST> metric 0 mtu 1500
inet6 fe80::84b8:2eb3:a617:de8a%ipsec1000 prefixlen 64 tentative scopeid 0x6
inet 10.20.30.1 --> 10.20.30.2 netmask 0xfffffffc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
reqid: 1000
groups: ipsec

So I ssh'd in and did this:
ifconfig ipsec1000 inet tunnel 96.126.96.153 73.140.16.217 up

Then on the other side I ran this:
ifconfig ipsec1000 inet tunnel 73.140.16.217 96.126.96.153 up

Everything worked then, and my ifconfig now shows the tunnel:

ipsec1000: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
tunnel inet 96.126.96.153 --> 73.140.16.217
inet6 fe80::84b8:2eb3:a617:de8a%ipsec1000 prefixlen 64 tentative scopeid 0x6
inet 10.20.30.1 --> 10.20.30.2 netmask 0xfffffffc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
reqid: 1000
groups: ipsec

This seems to be a bug.

[janets]

The tunnel also didn't route IPv6 over itself, even though I had IPv4 & IPv6 P2s defined. Again, from the commandline I did this on one side:

ifconfig ipsec1000 inet6 2600:3c01:e000:31e::2 prefixlen 112

and this on the other:

ifconfig ipsec1000 inet6 2600:3c01:e000:31e::1 prefixlen 112

Giving me this:

ipsec1000: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
tunnel inet 96.126.96.153 --> 73.140.16.217
inet6 fe80::84b8:2eb3:a617:de8a%ipsec1000 prefixlen 64 scopeid 0x6
inet6 2600:3c01:e000:31e::2 prefixlen 112
inet 10.20.30.1 --> 10.20.30.2 netmask 0xfffffffc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
reqid: 1000
groups: ipsec

And IPv6 worked.