A brief update on the issue:
I did a several tcpdumps to see how traffic traveled through the machine.
pings from the Raspberry Pi 10.0.1.253 to 172.17.20.130 are sent to the ipsec interface and receive replies:
tcpdump -i lagg0.100 icmp
14:22:27.875175 IP 10.0.1.253 > 172.17.20.130: ICMP echo request, id 8761, seq 14, length 64
14:22:27.899031 IP 172.17.20.130 > 10.0.1.253: ICMP echo reply, id 8761, seq 14, length 64
These in turn are encapsulated:
tcpdump -i enc0 icmp
14:23:23.969153 (authentic,confidential): SPI 0x82bee771: IP 10.0.1.253 > 172.17.20.130: ICMP echo request, id 8761, seq 70, length 64
14:23:23.992999 (authentic,confidential): SPI 0xc030c198: IP 172.17.20.130 > 10.0.1.254: ICMP echo reply, id 21640, seq 70, length 64
We see the requests going out as 10.0.1.253 and replies coming in for 10.0.1.254, which are then translated back to finally be delivered to the Pi.
Now, when repeating this check with an OpenVPN client, we see different behavior:
tcpdump -i ovpns7 icmp
14:27:13.375762 IP 192.168.248.2 > 172.17.20.130: ICMP echo request, id 13598, seq 1, length 64
14:27:14.383797 IP 192.168.248.2 > 172.17.20.130: ICMP echo request, id 13598, seq 2, length 64
Requests go out, but nothing returns. Let's see where the requests go:
tcpdump -i enc0 icmp yields nothing, so let's try WAN:
tcpdump -i lagg0.4090 icmp
14:30:10.318176 IP 192.168.248.2 > 172.17.20.130: ICMP echo request, id 30820, seq 3, length 64
14:30:11.326380 IP 192.168.248.2 > 172.17.20.130: ICMP echo request, id 30820, seq 4, length 64
Now it's getting interesting: the pings from OpenVPN clients are sent to the default route (WAN). If I change the NAT rule's interface from IPsec to WAN, the translation also shows up in the tcpdump output:
address family: ipv4
NAT address: INT_IP (10.0.1.254)
Pool options: default
Let's try the tcpdump again:
tcpdump -i lagg0.4090 icmp
14:33:12.248168 IP 10.0.1.254 > 172.17.20.130: ICMP echo request, id 15552, seq 1, length 64
14:33:13.251668 IP 10.0.1.254 > 172.17.20.130: ICMP echo request, id 15552, seq 2, length 64
So the firewall is correctly performing outbound NAT translation for 192.168.248.0/24->172.17.0.0/16 as 10.0.1.254, but the traffic is routed from the OpenVPN interface to the default gateway (WAN interface / lagg0.4090) rather than to the IPsec tunnel for encapsulation.
The whole point of wanting to use outbound NAT for traffic to 172.17.0.0/16 is so either side only needs to define one P2 entry, but we can set up multiple OpenVPN tunnels (with each a different subnet) that can all reach that remote network.
Would anybody know a trick / way to force traffic from 192.168.248.0 to 172.17.0.0/16 to go via the IPsec interface (enc0) rather than the default gateway (WAN/lagg0.4090)? I've tried creating a bogus gateway under System/Routing/Gateways and creating a static route under System/Routing/Static Routes for 172.17.0.0/16 via this bogus gateway, but this only resulted in all traffic from ovpns7 meant for this subnet to get put through a loop until the TTL expired.