Problem 2 fixed by adding route to 192.168.5.0/24 on Mikrotik side

@cmcdonald thank you for the explanation. indeed the problem was my frr configuration, all is working fine now.

As an update, I have done some more troubleshooting on the issue:

Switching to static routes over the VTI tunnel works. Using regular tunnel IPv4 also works Its only when we use FRR via OSPF (have not tested BGP) that traffic does not flow between hub and spoke.

Topology is 1 Hub (virtual) with 3 spokes (2 virtual, 1 physical pfSense). Its the physical pfSense spoke that is having issue

Enable IPsec MSS Clamping with different values, 1400, 1350, 1200, etc. on both hub and spoke and no issue. Also adjusted the VTI MTU value as well with no luck

Both sides are using AES-NI CPU Crypto. Enable/Disabling this has no effect

Both sides are using IPsec Asynchronous Cryptography. Enable/Disabling this has no effect

Tried different P2 encryption options but no luck. Currently using

P1: AES128-GCM (128 bits) AES-XCBC via 14 (2048) DH Group P2: ESP AES128-GCM (128 bits) PFS Group: 14 (2048). NO Hash algorithms

It appears another user on Reddit is facing similar issues: https://www.reddit.com/r/PFSENSE/comments/mzab6v/251_and_ipsec_vti/

Any ideas why FRR and OSPF is not sending traffic over the network? What troubleshooting steps can I take to debug this further?

First of all, you need to clarify if the pritunl VPN users (while connected) will be "going" out with their 192.168.22.x IP address , or with the IP address of the Pritunl network interface (192.168.226.1).

Also, I assume that you have created a Server in the pritunl that assigns the 192.168.226.x IP addresses. In that server, you will have to add a route towards the 172.17.172.x network (see below)
b7fc52a1-f8e5-4555-8671-6d04a35c5b5b-image.png

After you do the above, then you can start pinging from a VPN user towards your Servers. In order to see if the Pritunl VPN user is going out with its assigned IP addres (192.168.2226.2) and not with the Pritunl server IP (192.168.226.1), go to Packet Capture in pfsense and check the traffic on the pfsense interface that belongs to 172.17.172.x network.

*I would create an alias for these VPN users and name it "OpenVPN_Users" (Alias type is network with an IP address 192.168.226.0/24).

Then I would go to the firewall rules and I would add a rule to allow the OpenVPN_Users network towards the 102.17.172.0 network. Not sure if you have to configure the Advanced Settings on that rule, but if you still cannot ping the servers, you may have to go and change the TCP flags to "Any" and the State Type to "sloppy" (see below)

4e012871-d683-4bee-a1e1-8e3c38a6307e-image.png

Also, I assume these VPN users will be having internet access via your pfsense, which means that they will be going to the outside world via the WAN interface. If so, maybe you would have to add a NAT rule, but check first if it works without any NAT rule.