IPSEC VTI Iperf3 and UDP troubleshooting

ddbnj

Here's my setup:

Site to Site IPSEC-VTI

Iperf is running server on remote Pfsense firewall:

remote#: iperf3 -s -V

Iperf is running client on local Pfsense firewall:

local: iperf3 -c 172.20.1.1 -u -V

remote (server) output:

Time: Sat, 12 Sep 2020 05:09:14 UTC
Accepted connection from 10.8.222.1, port 33288
     Cookie: 2zy5jawhztvpkyw7ziwmsfi62vecdk66klnd
     Target Bitrate: 1048576
[  9] local 10.8.222.2 port 5201 connected to 10.8.222.1 port 62609
Starting Test: protocol: UDP, 1 streams, 1360 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  9]   0.00-1.01   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
[  9]   1.01-2.00   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
[  9]   2.00-3.03   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
[  9]   3.03-4.00   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
[  9]   4.00-5.03   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
[  9]   5.03-6.02   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)
...
iperf3: the client has unexpectedly closed the connection
iperf 3.7
FreeBSD pfSense.server 11.3-STABLE FreeBSD 11.3-STABLE #239 885b1ed26b6(factory-RELENG_2_4_5): Tue Jun  2 17:47:00 EDT 2020     root@buildbot3-nyi.netgate.com:/build/factory-crossbuild-245-aarch64/obj/aarch64/Og10eFss/build/factory-crossbuild-245-aarch64/sources/FreeBSD-src/sys/pfSense arm64

local (client) output:

iperf 3.7
FreeBSD protectli.client 11.3-STABLE FreeBSD 11.3-STABLE #243 abf8cba50ce(RELENG_2_4_5): Tue Jun  2 17:53:37 EDT 2020     root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-245/obj/amd64/YNx4Qq3j/build/ce-crossbuild-245/sources/FreeBSD-src/sys/pfSense amd64
Control connection MSS 1360
Setting UDP block size to 1360
Time: Sat, 12 Sep 2020 05:09:14 UTC
Connecting to host 172.20.1.1, port 5201
      Cookie: 2zy5jawhztvpkyw7ziwmsfi62vecdk66klnd
      Target Bitrate: 1048576
iperf3: error - unable to read from stream socket: Resource temporarily unavailable

There is nothing in my firewall logs. Ipsec rules on both sides pass everything to everywhere. Iperf3 via TCP works fine.

Thanks for any assistance,

Devan