PFsense as OpenVPN Client - Networks can't be reached



  • Hi,
    I tried all I can, but don't get any further, so here I am.
    Any help regards a solution or better analysis is welcome, I am EndOfWisdom.

    The Scenario
    A Site-To-Site OpenVPN connection, where my main firewall is the VPN Client.

    The Problem
    I can't reach any other client IP other than the PFSense OpenVPN ClientIP from server-side. Otherwise I am able to contact the other networks of the server from the clients without any headache.

    General settings
    You can safely assume, that every firewall tab has some allow all - as I am debugging this (unsuccessfully) quite a time.

    Configs -VPN
    Server - Ingress (OPNsense)
    Network Settings
    HADES (ovpns1) -> v4: 192.168.250.1/24
    WAN (vtnet0) -> v4: publicIP/24
    testLAN (lo1) -> v4: 10.1.1.1/24

    OpenVPN Settings
    The VPN connection in general works 'flawless', except of the clients subnets routing problems

    Server - ingress (OPNSense)
    General Information
    Server Mode: Remote Access (SSL/TLS)
    Protocol: UDP4
    Device Mode: tun
    Interface: WAN

    Tunnel Settings
    IPv4 Tunnel Network: 192.168.250.0/24
    IPv4 Local Network: 10.1.1.0/24
    IPv4 Remote Network: 10.10.10.10/32,10.10.100.0/22
    Concurrent connections: 1
    Disable IPv6: check

    Client Settings
    Dynamic IP: check
    Address Pool: check
    Topology: check

    Advanced configuration
    tun-mtu 1500
    mssfix 1500

    Client - Hades (PFsense)
    OpenVPN Settings
    The VPN connection in general works 'flawless', except of the clients subnets routing problems

    General Information
    Server Mode: Remote Access (SSL/TLS)
    Protocol: UDP4
    Device Mode: tun
    Interface: WAN

    Tunnel Settings
    IPv4 Tunnel Network: 192.168.250.0/24
    IPv4 Remote Network: 10.1.1.0/24
    IPv4 Local Network !! NOT available in options !! Would be: 10.10.10.10/32,10.10.100.0/22
    Topology: Subnet -- One IP address per client in a common subnet

    Advanced configuration
    tun-mtu 1500
    mssfix 1500

    Routes Server
    root@ingress:~ # netstat -r
    Routing tables

    Internet:
    Destination Gateway Flags Netif Expire
    default pupIP UGS vtnet0
    ingress link#6 UH lo1
    10.10.10.5/32 192.168.250.2 UGS ovpns1
    10.10.10.7/32 192.168.250.2 UGS ovpns1
    10.10.10.10/32 192.168.250.2 UGS ovpns1
    10.10.100.0/22 192.168.250.2 UGS ovpns1
    localhost link#3 UH lo0
    192.168.250.0/24 192.168.250.2 UGS ovpns1
    ingress link#7 UHS lo0
    192.168.250.2 link#7 UH ovpns1
    publicIP/24 link#1 U vtnet0
    ingress link#1 UHS lo0

    Routes Client

    Destination Gateway Flags Use Mtu Netif Expire
    default pupIP UGS 950434 1500 igb0
    10.1.1.0/24 192.168.250.1 UGS 0 1500 ovpnc5
    10.10.10.1 link#2 UHS 0 16384 lo0
    10.10.10.1/32 link#2 U 0 1500 igb1
    10.10.10.5 link#10 UHS 0 16384 lo0
    10.10.10.5/32 link#10 U 0 1500 igb2.100
    10.10.10.7 link#10 UHS 342897 16384 lo0
    10.10.10.7/32 link#10 U 0 1500 igb2.100
    10.10.10.8 link#11 UHS 0 16384 lo0
    10.10.10.8/32 link#11 U 0 1500 igb2.200
    10.10.10.9 link#11 UHS 0 16384 lo0
    10.10.10.9/32 link#11 U 0 1500 igb2.200
    10.10.10.10 link#10 UHS 0 16384 lo0
    10.10.10.10/32 link#10 U 0 1500 igb2.100
    10.10.10.11 link#9 UHS 0 16384 lo0
    10.10.10.11/32 link#9 U 0 1500 igb2.10
    10.10.100.0/22 link#12 U 4278819 1500 igb2.1000
    192.168.250.1 link#17 UH 7795 1500 ovpnc5
    192.168.250.2 link#17 UHS 0 16384 lo0

    TCPdump serverside WAN
    The server sents to the correct route, but.. retransmission, doesn't get anything. Tested with port forward on 444

    1 0.000000 anotherPublicDialUpIP ingressPermIP TCP 74 58777 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417955 TSecr=0 WS=512
    2 0.151688 anotherPublicDialUpIP ingressPermIP TCP 74 37745 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417980 TSecr=0 WS=512
    3 0.921882 anotherPublicDialUpIP ingressPermIP TCP 74 [TCP Retransmission] 58777 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418056 TSecr=0 WS=512
    4 1.160898 anotherPublicDialUpIP ingressPermIP TCP 74 [TCP Retransmission] 37745 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418080 TSecr=0 WS=512

    TCPdump serverside VPN
    1 0.000000 anotherPublicDialUpIP 10.10.10.10 TCP 64 58777 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417955 TSecr=0 WS=512
    2 0.151733 anotherPublicDialUpIP 10.10.10.10 TCP 64 37745 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417980 TSecr=0 WS=512
    3 0.921826 anotherPublicDialUpIP 10.10.10.10 TCP 64 [TCP Retransmission] 58777 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418056 TSecr=0 WS=512
    4 1.160824 anotherPublicDialUpIP 10.10.10.10 TCP 64 [TCP Retransmission] 37745 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418080 TSecr=0 WS=512

    TCPdump hades clientside
    nothing

    Rough network diagram
    550e5683-b49b-4f01-ad29-fcf538850402-image.png

    possible solutions and workaround
    The goal of this is, to forward incoming requests to the ingressPermIP to servers in the segments of hades (vpn client)
    For trivial web, I could go with 2 HAProxies so, the webrequest goes from ingress->hadesVpnIP->server via proxy protocol. Done and tested.

    But sadly this is not for every and all services a solution. And it doesn't solve the problem in general.

    some random observations
    traceroute is nearly useless for me. Don't know why, but even working path end up just with stars until given up after 30 hops.
    Currently I am trying to rebuild the TTL error, I was able to produce instead just a timeout with ping.

    ecosystem
    the pfsense is baremetal on an pcengine
    the opnsense is a virtualized "cloud node", I have no further access or information to.

    Did I forgot something? Just ask and I am eager to give the information. This is really burning my time. :(



  • It may be worth to mention, that I am routing some 'LAN' device through the VPN to use the permIP as gateway - mainly for testing. (policy-routing via src of device)
    THIS works great, ever since.
    And also setups with the pfsense as server have no known problems. To be specific the "testdevice" is dialed in via vpn in hades and then routed through the vpn where hades is the client and exits ingress.



  • I made a simpler setup pfsense instance with to the minmal reduced base config and the same behavior.
    It has nearly no additional packages, except HAProxy and ACME, only 1 virtual IP on LAN side and the same behavior.

    Next step: a clean instance of PFsense. Will report back. And also test this with an opnsense instance on local side.


Log in to reply