IPsec upgrade to 2.5

Ofloo

Seems more like the problem is with FRR then it does with IPsec, ..

bgp_process_packet: BGP OPEN receipt failed for peer:

tunnel:

ipsec1000: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
	tunnel inet 158.x.x.x --> 81.x.x.x
	inet6 fe80::x:x:x:x%ipsec1000 prefixlen 64 scopeid 0x9
	inet 10.128.x.x --> 10.128.x.x netmask 0xfffffffc
	groups: ipsec
	reqid: 1000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

rte11.ofloo.net (10.128.x.x) -> 10.128.x.x                                   2021-02-21T09:58:41+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                Packets               Pings
 Host                                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.128.x.x                                               0.0%     4   93.0  95.6  93.0  97.5   1.9

Tunnel works just fine, when I manually add routing it works.

jimp

To ensure you have all of the current known and fixed IPsec issues corrected, You can install the System Patches package and then create entries for the following commit IDs to apply the fixes:

ead6515637a34ce6e170e2d2b0802e4fa1e63a00 #11435
57beb9ad8ca11703778fc483c7cba0f6770657ac #11435
10eb04259fd139c62e08df8de877b71fdd0eedc8 #11442
ded7970ba57a99767e08243103e55d8a58edfc35 #11486
afffe759c4fd19fe6b8311196f4b6d5e288ea4fb #11487
2fe5cc52bd881ed26723a81e0eed848fd505fba6 #11488

Also check for an install the latest FRR updates from today.

Reboot and test again after that.

Ofloo

@jimp Thank you for your responds, this package is really awesome, however unfortunately it doesn't make any difference.

Ofloo

I get a lot of these errors in routing log:

bgpd[80722]: %NOTIFICATION: sent to neighbor 10.128.x.x 6/7 (Cease/Connection collision resolution) 0 bytes 
bgpd[80722]: [EC 33554451] bgp_process_packet: BGP OPEN receipt failed for peer: 10.128.x.x
bgpd[80722]: %ADJCHANGE: neighbor 10.128.x.x(rtexx.xxxxx.net) in vrf default Down Peer closed the session
zebra[79149]: [EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
zebra[79149]: [EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
zebra[79149]: [EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
bgpd[80722]: %ADJCHANGE: neighbor 10.128.x.x(rtexx.x.net) in vrf default Down Interface down
zebra[79149]: [EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
zebra[79149]: Can't lookup mtu by ioctl(SIOCGIFMTU)
bgpd[80722]: [EC 100663301] INTERFACE_STATE: Cannot find IF ipsec2000 in VRF 0
zebra[79149]: warning: connected_add_ipv6 called for interface ipsec2000 with peer flag set, but no peer address supplied
bgpd[80722]: %ADJCHANGE: neighbor 10.128.x.x(rtexx.xxxx.net) in vrf default Up
bgpd[80722]: %NOTIFICATION: received from neighbor 10.128.x.x 6/7 (Cease/Connection collision resolution) 0 bytes 
zebra[79149]: [EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured

jimp

@ofloo said in IPsec upgrade to 2.5:

The log also shows

rc.bootup: The command '/sbin/ifconfig 'ipsec3000' inet tunnel '' '2001:xxx:xxxx:xxx::1' up' returned exit code '1', the output was 'ifconfig: error in parsing address string: Name does not resolve' 
route: writing to routing socket: Network is unreachable

I missed this one before. Somehow it's trying to configure that IPv6 address as an IPv4 address, so the interface itself is probably missing or not configured properly.

What are the exact settings you have on your VTI P2 entries for that tunnel?

Ofloo

@jimp

        con3000 {
                fragmentation = yes
                unique = replace
                version = 2
                proposals = aes256gcm128-sha512-prfsha512-modp4096
                dpd_delay = 10s
                dpd_timeout = 60s
                rekey_time = 25920s
                reauth_time = 0s
                over_time = 2880s
                rand_time = 2880s
                encap = no
                mobike = yes
                local_addrs = 2001:xx:b112:xxxx::1
                remote_addrs = 2001:xx:c9dc:xxxx::1
                local {
                        id = 2001:xx:b112:xxxx::1
                        auth = psk
                }
                remote {
                        id = 2001:xx:c9dc:xxxx::1
                        auth = psk
                }
                children {
                        con300000 {
                                dpd_action = restart
                                policies = no
                                life_time = 3600s
                                rekey_time = 3240s
                                rand_time = 360s
                                start_action = start
                                local_ts = fdxx:xx::2/126,0.0.0.0/0
                                remote_ts = fdxx:xx::1,0.0.0.0/0
                                reqid = 3000
                                esp_proposals = aes256gcm128-modp4096
                        }
                }
        }

Ofloo

@ofloo

what about this error

zebra[71429]: Can't lookup mtu by ioctl(SIOCGIFMTU)

while:

jimp

That error also implies the interface doesn't exist or can't be queried.

The swanctl config is nice but can you give a screenshot of the GUI settings for that P2? Or at least the config.xml -- I need more about how it is getting to that point, not the end result.

Ofloo

@jimp I've removed it just a min ago so no, but even when removed it makes no difference.
i've removed all IPv6 just to be sure but still no difference.

Ofloo

@jimp is it important that you see it? I still can rollback the vm. If it helps..

jimp

It may be enough to know that the external addresses were IPv6. I checked in my lab and I don't have any VTI that are setup using IPv6 on the outside like that. I can't recall the last time I tested it.

jimp

I opened an issue to track it at https://redmine.pfsense.org/issues/11537, hopefully it won't be too difficult for one of us to reproduce and solve.

Ofloo

@jimp

Just rebooted once more and I also noticed a lot of these errors?

14[CFG] trap not found, unable to acquire reqid 2000

 00[CFG] loaded PKCS#11 v2.20 library 'opensc' (/usr/local/lib/opensc-pkcs11.so) 
00[CFG] PKCS11 module '<name>' lacks library path 
00[DMN] Starting IKE charon daemon (strongSwan 5.9.1, FreeBSD 12.2-STABLE, amd64)

<con5000|3718> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found 
15[KNL] <con5000|3718> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found 
 15[KNL] <con5000|3718> querying policy fdxx:xxxx:xxx::2/128|/0 === fdxx:xxxx:xxx::2/128|/0 in failed, not found

Feb 26 08:05:19 	charon 	50795 	15[CFG] <con5000|3720> selecting traffic selectors for us:
Feb 26 08:05:19 	charon 	50795 	15[CFG] <con5000|3720> selected proposal: ESP:AES_GCM_16_256/MODP_4096/NO_EXT_SEQ
Feb 26 08:05:19 	charon 	50795 	15[CFG] <con5000|3720> configured proposals: ESP:AES_GCM_16_256/MODP_4096/NO_EXT_SEQ
Feb 26 08:05:19 	charon 	50795 	15[CFG] <con5000|3720> received proposals: ESP:AES_CBC_256/HMAC_SHA2_512_256/MODP_4096/NO_EXT_SEQ, ESP:AES_GCM_16_256/MODP_4096/NO_EXT_SEQ
Feb 26 08:05:19 	charon 	50795 	15[CFG] <con5000|3720> proposal matches
Feb 26 08:05:19 	charon 	50795 	15[CFG] <con5000|3720> selecting proposal:
Feb 26 08:05:19 	charon 	50795 	15[CFG] <con5000|3720> no acceptable ENCRYPTION_ALGORITHM found

On that last error no acceptable encryption proposals found, .. is strange cause both are configured with same encryption scheme/configuration.

Ofloo

@ofloo NVM bad patch applied.

EDIT:

@jimp NVM bad patch applied.

These appear though:

<con5000|3> querying policy fdxx:xxxx:xxxx::2/128|/0 === fdxx:xxxx:xxxx::2/128|/0 in failed, not found
<con5000|3> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found
<con5000|3> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found
<con1000|2> querying policy fdxx:xxxx:44xx::1/128|/0 === fdxx:xxxx:44xx::2/128|/0 in failed, not found
<con1000|2> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found
<con1000|2> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found

pasting logs is consider spam?

Ofloo

@ofloo An other problem I've noticed. It was a bug before but i was able to make it work.

In relase 2.4.5p1 both when you wanted a dual stack VTI you could set it to IPv4 and it would just work. You then could add both P2 IPv4 and IPv6 this worked.

However now in 2.5 this configuration doesn't seem to work anymore. When it's set to IPv4 vti only IPv4 works as it should i guess but when set to dual stack nothing works as it did before. But now you can't make dual stack work anymore.

Ofloo

@ofloo

I think the problem still lies with FRR, maybe it's a configuration thing at least for the IPv4 part.

tcpdump -ni ipsec5000 not icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ipsec5000, link-type NULL (BSD loopback), capture size 262144 bytes
08:35:56.952614 IP 10.128.x.9.52117 > 10.128.x.10.179: Flags [FP.], seq 1342714925:1342714944, ack 1126756536, win 131, options [nop,nop,TS val 2158382576 ecr 2930881738], length 19: BGP
08:35:57.049220 IP 10.128.x.10.179 > 10.128.x.9.52117: Flags [R], seq 1126756536, win 0, length 0
08:36:02.013888 IP 10.128.x.10.179 > 10.128.x.9.46094: Flags [R.], seq 3726290407, ack 2899991084, win 512, options [nop,nop,TS val 4145417838 ecr 2305700040], length 0
08:36:07.589692 IP 10.128.x.9.179 > 10.128.x.10.46063: Flags [FP.], seq 255473205:255473226, ack 100809217, win 128, options [nop,nop,TS val 1581327985 ecr 96898160], length 21: BGP
08:36:11.152170 IP 10.128.x.9.2199 > 10.128.x.10.179: Flags [P.], seq 485006693:485006712, ack 3890827107, win 131, options [nop,nop,TS val 2143866376 ecr 2627774982], length 19: BGP
08:36:11.213583 IP 10.128.x.10.179 > 10.128.x.9.2199: Flags [P.], seq 1:20, ack 0, win 512, options [nop,nop,TS val 2627833734 ecr 2143807680], length 19: BGP
08:36:11.213623 IP 10.128.x.9.2199 > 10.128.x.10.179: Flags [.], ack 20, win 131, options [nop,nop,TS val 2143866438 ecr 2627833734], length 0
08:36:11.249057 IP 10.128.x.10.179 > 10.128.x.9.2199: Flags [.], ack 19, win 511, options [nop,nop,TS val 2627833769 ecr 2143866376], length 0
08:36:21.343545 IP 10.128.x.10.179 > 10.128.x.9.2199: Flags [P.], seq 20:336, ack 19, win 512, options [nop,nop,TS val 2627843862 ecr 2143866438], length 316: BGP
08:36:21.343608 IP 10.128.x.9.2199 > 10.128.x.10.179: Flags [.], ack 336, win 131, options [nop,nop,TS val 2143876562 ecr 2627843862], length 0
08:36:48.363641 IP 10.128.x.9.52117 > 10.128.x.10.179: Flags [FP.], seq 0:19, ack 1, win 131, options [nop,nop,TS val 2158433987 ecr 2930881738], length 19: BGP
08:36:48.461330 IP 10.128.x.10.179 > 10.128.x.9.52117: Flags [R], seq 1126756536, win 0, length 0
08:37:04.116239 IP 10.128.x.9.179 > 10.128.x.10.46063: Flags [FP.], seq 0:21, ack 1, win 128, options [nop,nop,TS val 1581384512 ecr 96898160], length 21: BGP
08:37:11.163081 IP 10.128.x.9.2199 > 10.128.x.10.179: Flags [P.], seq 19:38, ack 336, win 131, options [nop,nop,TS val 2143926387 ecr 2627843862], length 19: BGP
08:37:11.263206 IP 10.128.x.10.179 > 10.128.x.9.2199: Flags [.], ack 38, win 511, options [nop,nop,TS val 2627893780 ecr 2143926387], length 0
08:37:11.263233 IP 10.128.x.10.179 > 10.128.x.9.2199: Flags [P.], seq 336:355, ack 38, win 512, options [nop,nop,TS val 2627893780 ecr 2143926387], length 19: BGP
08:37:11.263252 IP 10.128.x.9.2199 > 10.128.x.10.179: Flags [.], ack 355, win 131, options [nop,nop,TS val 2143926487 ecr 2627893780], length 0
08:37:39.763627 IP 10.128.x.9.52117 > 10.128.x.10.179: Flags [FP.], seq 0:19, ack 1, win 131, options [nop,nop,TS val 2158485387 ecr 2930881738], length 19: BGP
08:37:39.859222 IP 10.128.x.10.179 > 10.128.x.9.52117: Flags [R], seq 1126756536, win 0, length 0
08:38:00.732652 IP 10.128.x.9.179 > 10.128.x.10.46063: Flags [FP.], seq 0:21, ack 1, win 128, options [nop,nop,TS val 1581441128 ecr 96898160], length 21: BGP
08:38:11.262922 IP 10.128.x.9.2199 > 10.128.x.10.179: Flags [P.], seq 38:57, ack 355, win 131, options [nop,nop,TS val 2143986487 ecr 2627893780], length 19: BGP
08:38:11.287734 IP 10.128.x.10.179 > 10.128.x.9.2199: Flags [P.], seq 355:374, ack 38, win 512, options [nop,nop,TS val 2627953810 ecr 2143926487], length 19: BGP
08:38:11.287759 IP 10.128.x.9.2199 > 10.128.x.10.179: Flags [.], ack 374, win 131, options [nop,nop,TS val 2143986512 ecr 2627953810], length 0
08:38:11.358617 IP 10.128.x.10.179 > 10.128.x.9.2199: Flags [.], ack 57, win 511, options [nop,nop,TS val 2627953880 ecr 2143986487], length 0
08:38:31.163620 IP 10.128.x.9.52117 > 10.128.x.10.179: Flags [R.], seq 20, ack 1, win 0, options [nop,nop,TS val 2158536787 ecr 2930881738], length 0

The routes are just not distributing.

Ofloo

@ofloo I did a backup of a router installed a cloud version of it (linode) 2.5 configuration, added a wireguard tunnel as well now both ipsec vti I can ping both IPv4 and IPv6, i can ping through wireguard.

However frr bgp* still doesn't distribute routes not even through wireguard, added allow IP 0.0.0.0/0 and ::1/0

danjeman

@jimp Can we simply update to 2.5.1 or 21.02.2 over the top of these system patches or should they be removed before or after?

jimp

You can just update, the patches are a part of 21.02.2/2.1.5.

Alternately, you can remove the patch entries (Do NOT revert, just delete them) either before or after upgrade and leave the patches package in place.

The only possible action you might need to take is to make sure none of them are set to auto-apply. In most cases that wouldn't hurt anything since it would just fail to apply, but certain diffs may end up adding themselves multiple times that way.