I am working on troubleshooting through iperf and here is what I have going public IP to public IP (read not through IPSec Tunnel) and private IP to private IP (read through the IPSec Tunnel)
Public IP to Public IP - Site B to Site A
iperf 3.17.1
FreeBSD ngr2-rtr01.mynextgenrx.com 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:08:46 UTC 2024 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/amd64/AKWlAIiM/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBS amd64
Control connection MSS 1460
Time: Mon, 03 Mar 2025 16:13:38 UTC
Connecting to host 207.162.137.152, port 5201
Cookie: g3oi22k7ksrx6tkmxub52glxdgoc7l2xocks
TCP MSS: 1460 (default)
[ 5] local 75.61.85.194 port 15753 connected to 207.162.137.152 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.9 MBytes 90.8 Mbits/sec 0 399 KBytes
[ 5] 1.00-2.06 sec 12.9 MBytes 102 Mbits/sec 0 399 KBytes
[ 5] 2.06-3.06 sec 12.2 MBytes 103 Mbits/sec 0 399 KBytes
[ 5] 3.06-4.00 sec 11.5 MBytes 102 Mbits/sec 0 399 KBytes
[ 5] 4.00-5.01 sec 12.2 MBytes 102 Mbits/sec 0 399 KBytes
[ 5] 5.01-6.03 sec 12.9 MBytes 106 Mbits/sec 0 444 KBytes
[ 5] 6.03-7.06 sec 15.5 MBytes 126 Mbits/sec 0 522 KBytes
[ 5] 7.06-8.05 sec 15.4 MBytes 131 Mbits/sec 42 181 KBytes
[ 5] 8.05-9.00 sec 5.75 MBytes 50.4 Mbits/sec 0 213 KBytes
[ 5] 9.00-10.00 sec 6.88 MBytes 57.7 Mbits/sec 0 233 KBytes
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 116 MBytes 97.4 Mbits/sec 42 sender
[ 5] 0.00-10.04 sec 115 MBytes 96.2 Mbits/sec receiver
CPU Utilization: local/sender 12.4% (0.0%u/12.4%s), remote/receiver 9.3% (2.0%u/7.3%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
Private IP to Private IP THROUGH IPSec - Site B to Site A
Connecting to host 10.1.0.2, port 5201
[ 5] local 10.2.0.2 port 31524 connected to 10.1.0.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 16.0 MBytes 134 Mbits/sec 0 1.07 MBytes
[ 5] 1.00-2.01 sec 16.0 MBytes 133 Mbits/sec 411 218 KBytes
[ 5] 2.01-3.00 sec 7.12 MBytes 60.2 Mbits/sec 0 257 KBytes
[ 5] 3.00-4.04 sec 8.50 MBytes 68.9 Mbits/sec 0 281 KBytes
[ 5] 4.04-5.00 sec 8.50 MBytes 73.8 Mbits/sec 0 293 KBytes
[ 5] 5.00-6.06 sec 9.50 MBytes 75.0 Mbits/sec 0 306 KBytes
[ 5] 6.06-7.00 sec 9.12 MBytes 81.6 Mbits/sec 0 327 KBytes
[ 5] 7.00-8.00 sec 10.4 MBytes 86.9 Mbits/sec 0 351 KBytes
[ 5] 8.00-9.00 sec 10.9 MBytes 91.1 Mbits/sec 0 374 KBytes
[ 5] 9.00-10.05 sec 12.4 MBytes 99.3 Mbits/sec 0 398 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.05 sec 108 MBytes 90.5 Mbits/sec 411 sender
[ 5] 0.00-10.08 sec 107 MBytes 89.4 Mbits/sec receiver
iperf Done.
Public IP to Public IP - Site A to Site B
iperf 3.17.1
FreeBSD ngr1-rtr01.mynextgenrx.com 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:08:46 UTC 2024 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/amd64/AKWlAIiM/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBS amd64
Control connection MSS 1460
Time: Mon, 03 Mar 2025 16:53:33 UTC
Connecting to host 75.61.85.194, port 5201
Cookie: fediws4o47ceehawmnp3zegzg4zch3oa2wn3
TCP MSS: 1460 (default)
[ 5] local 207.162.137.152 port 33788 connected to 75.61.85.194 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.04 sec 11.8 MBytes 94.8 Mbits/sec 299 367 KBytes
[ 5] 1.04-2.06 sec 12.1 MBytes 99.6 Mbits/sec 0 412 KBytes
[ 5] 2.06-3.05 sec 12.9 MBytes 109 Mbits/sec 0 441 KBytes
[ 5] 3.05-4.06 sec 13.8 MBytes 114 Mbits/sec 0 458 KBytes
[ 5] 4.06-5.06 sec 14.0 MBytes 118 Mbits/sec 0 466 KBytes
[ 5] 5.06-6.05 sec 14.0 MBytes 118 Mbits/sec 0 468 KBytes
[ 5] 6.05-7.00 sec 13.5 MBytes 119 Mbits/sec 0 469 KBytes
[ 5] 7.00-8.01 sec 14.2 MBytes 119 Mbits/sec 0 482 KBytes
[ 5] 8.01-9.01 sec 15.0 MBytes 126 Mbits/sec 0 503 KBytes
[ 5] 9.01-10.00 sec 15.1 MBytes 128 Mbits/sec 0 523 KBytes
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 136 MBytes 114 Mbits/sec 299 sender
[ 5] 0.00-10.03 sec 135 MBytes 113 Mbits/sec receiver
CPU Utilization: local/sender 11.2% (0.0%u/11.1%s), remote/receiver 17.8% (1.2%u/16.7%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
Private IP to Private IP THROUGH IPSec - Site A to Site B
iperf 3.17.1
FreeBSD ngr1-rtr01.mynextgenrx.com 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:08:46 UTC 2024 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/amd64/AKWlAIiM/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBS amd64
Control connection MSS 1460
Time: Mon, 03 Mar 2025 16:54:25 UTC
Connecting to host 10.2.0.2, port 5201
Cookie: aji4w5n3rzsobdygb2zxvqydl7mrciv3gum7
TCP MSS: 1460 (default)
[ 5] local 10.1.0.2 port 30411 connected to 10.2.0.2 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.06 sec 11.1 MBytes 87.8 Mbits/sec 210 362 KBytes
[ 5] 1.06-2.06 sec 11.6 MBytes 97.9 Mbits/sec 0 406 KBytes
[ 5] 2.06-3.02 sec 12.2 MBytes 107 Mbits/sec 0 434 KBytes
[ 5] 3.02-4.06 sec 14.1 MBytes 113 Mbits/sec 0 453 KBytes
[ 5] 4.06-5.01 sec 13.0 MBytes 115 Mbits/sec 0 462 KBytes
[ 5] 5.01-6.03 sec 14.4 MBytes 118 Mbits/sec 0 463 KBytes
[ 5] 6.03-7.06 sec 14.4 MBytes 117 Mbits/sec 0 464 KBytes
[ 5] 7.06-8.02 sec 13.6 MBytes 120 Mbits/sec 0 464 KBytes
[ 5] 8.02-9.06 sec 14.9 MBytes 119 Mbits/sec 0 482 KBytes
[ 5] 9.06-10.01 sec 14.1 MBytes 126 Mbits/sec 0 499 KBytes
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 134 MBytes 112 Mbits/sec 210 sender
[ 5] 0.00-10.04 sec 133 MBytes 111 Mbits/sec receiver
CPU Utilization: local/sender 63.1% (0.1%u/63.0%s), remote/receiver 40.2% (2.2%u/38.0%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic
iperf Done.
Speeds seem better using iperf thought they are still somewhat slower than it should be.