PFsense as OpenVPN Client - Networks can't be reached

[Orwi]

Hi,
I tried all I can, but don't get any further, so here I am.
Any help regards a solution or better analysis is welcome, I am EndOfWisdom.

The Scenario
A Site-To-Site OpenVPN connection, where my main firewall is the VPN Client.

The Problem
I can't reach any other client IP other than the PFSense OpenVPN ClientIP from server-side. Otherwise I am able to contact the other networks of the server from the clients without any headache.

General settings
You can safely assume, that every firewall tab has some allow all - as I am debugging this (unsuccessfully) quite a time.

Configs -VPN
Server - Ingress (OPNsense)
Network Settings
HADES (ovpns1) -> v4: 192.168.250.1/24
WAN (vtnet0) -> v4: publicIP/24
testLAN (lo1) -> v4: 10.1.1.1/24

OpenVPN Settings
The VPN connection in general works 'flawless', except of the clients subnets routing problems

Server - ingress (OPNSense)
General Information
Server Mode: Remote Access (SSL/TLS)
Protocol: UDP4
Device Mode: tun
Interface: WAN

Tunnel Settings
IPv4 Tunnel Network: 192.168.250.0/24
IPv4 Local Network: 10.1.1.0/24
IPv4 Remote Network: 10.10.10.10/32,10.10.100.0/22
Concurrent connections: 1
Disable IPv6: check

Client Settings
Dynamic IP: check
Address Pool: check
Topology: check

Advanced configuration
tun-mtu 1500
mssfix 1500

Client - Hades (PFsense)
OpenVPN Settings
The VPN connection in general works 'flawless', except of the clients subnets routing problems

General Information
Server Mode: Remote Access (SSL/TLS)
Protocol: UDP4
Device Mode: tun
Interface: WAN

Tunnel Settings
IPv4 Tunnel Network: 192.168.250.0/24
IPv4 Remote Network: 10.1.1.0/24
IPv4 Local Network !! NOT available in options !! Would be: 10.10.10.10/32,10.10.100.0/22
Topology: Subnet -- One IP address per client in a common subnet

Advanced configuration
tun-mtu 1500
mssfix 1500

Routes Server
root@ingress:~ # netstat -r
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default pupIP UGS vtnet0
ingress link#6 UH lo1
10.10.10.5/32 192.168.250.2 UGS ovpns1
10.10.10.7/32 192.168.250.2 UGS ovpns1
10.10.10.10/32 192.168.250.2 UGS ovpns1
10.10.100.0/22 192.168.250.2 UGS ovpns1
localhost link#3 UH lo0
192.168.250.0/24 192.168.250.2 UGS ovpns1
ingress link#7 UHS lo0
192.168.250.2 link#7 UH ovpns1
publicIP/24 link#1 U vtnet0
ingress link#1 UHS lo0

Routes Client

Destination Gateway Flags Use Mtu Netif Expire
default pupIP UGS 950434 1500 igb0
10.1.1.0/24 192.168.250.1 UGS 0 1500 ovpnc5
10.10.10.1 link#2 UHS 0 16384 lo0
10.10.10.1/32 link#2 U 0 1500 igb1
10.10.10.5 link#10 UHS 0 16384 lo0
10.10.10.5/32 link#10 U 0 1500 igb2.100
10.10.10.7 link#10 UHS 342897 16384 lo0
10.10.10.7/32 link#10 U 0 1500 igb2.100
10.10.10.8 link#11 UHS 0 16384 lo0
10.10.10.8/32 link#11 U 0 1500 igb2.200
10.10.10.9 link#11 UHS 0 16384 lo0
10.10.10.9/32 link#11 U 0 1500 igb2.200
10.10.10.10 link#10 UHS 0 16384 lo0
10.10.10.10/32 link#10 U 0 1500 igb2.100
10.10.10.11 link#9 UHS 0 16384 lo0
10.10.10.11/32 link#9 U 0 1500 igb2.10
10.10.100.0/22 link#12 U 4278819 1500 igb2.1000
192.168.250.1 link#17 UH 7795 1500 ovpnc5
192.168.250.2 link#17 UHS 0 16384 lo0

TCPdump serverside WAN
The server sents to the correct route, but.. retransmission, doesn't get anything. Tested with port forward on 444

1 0.000000 anotherPublicDialUpIP ingressPermIP TCP 74 58777 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417955 TSecr=0 WS=512
2 0.151688 anotherPublicDialUpIP ingressPermIP TCP 74 37745 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417980 TSecr=0 WS=512
3 0.921882 anotherPublicDialUpIP ingressPermIP TCP 74 [TCP Retransmission] 58777 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418056 TSecr=0 WS=512
4 1.160898 anotherPublicDialUpIP ingressPermIP TCP 74 [TCP Retransmission] 37745 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418080 TSecr=0 WS=512

TCPdump serverside VPN
1 0.000000 anotherPublicDialUpIP 10.10.10.10 TCP 64 58777 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417955 TSecr=0 WS=512
2 0.151733 anotherPublicDialUpIP 10.10.10.10 TCP 64 37745 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417980 TSecr=0 WS=512
3 0.921826 anotherPublicDialUpIP 10.10.10.10 TCP 64 [TCP Retransmission] 58777 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418056 TSecr=0 WS=512
4 1.160824 anotherPublicDialUpIP 10.10.10.10 TCP 64 [TCP Retransmission] 37745 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418080 TSecr=0 WS=512

TCPdump hades clientside
nothing

Rough network diagram

possible solutions and workaround
The goal of this is, to forward incoming requests to the ingressPermIP to servers in the segments of hades (vpn client)
For trivial web, I could go with 2 HAProxies so, the webrequest goes from ingress->hadesVpnIP->server via proxy protocol. Done and tested.

But sadly this is not for every and all services a solution. And it doesn't solve the problem in general.

some random observations
traceroute is nearly useless for me. Don't know why, but even working path end up just with stars until given up after 30 hops.
Currently I am trying to rebuild the TTL error, I was able to produce instead just a timeout with ping.

ecosystem
the pfsense is baremetal on an pcengine
the opnsense is a virtualized "cloud node", I have no further access or information to.

Did I forgot something? Just ask and I am eager to give the information. This is really burning my time. :(

[Orwi]

It may be worth to mention, that I am routing some 'LAN' device through the VPN to use the permIP as gateway - mainly for testing. (policy-routing via src of device)
THIS works great, ever since.
And also setups with the pfsense as server have no known problems. To be specific the "testdevice" is dialed in via vpn in hades and then routed through the vpn where hades is the client and exits ingress.

[Orwi]

I made a simpler setup pfsense instance with to the minmal reduced base config and the same behavior.
It has nearly no additional packages, except HAProxy and ACME, only 1 virtual IP on LAN side and the same behavior.

Next step: a clean instance of PFsense. Will report back. And also test this with an opnsense instance on local side.