PFSense not communicating with remote subnet


  • I have a Peer-to-Peer OpenVPN tunnel established between PFsense in our datacenter and a VyOS VM running in an AWS VPC. PFSense can ping the vpn ip of VyOS and vice versa. However, the subnet on the VyOS side is unreachable.

    PFSense source IP: 192.168.13.1 -> VyOS VPN IP 192.168.13.2 -> remote test ip: 10.200.16.165

    In the PFSense routing table I see
    10.200.16.0/24 via 192.168.13.2 UGS

    A traceroute from PFSense shows no first hops. Pinigng from the other end however shows packets flowing through VyOS and through the tunnel, however I do not receive any traffic back. I have a firewall rule on PFSense to track packets from the remote subnet but nothing ever reaches it. There are no other rules ahead of this one. For testing purposes I even created a temporary allow all rule and this still did not log any packets from the remote subnet. If I create a source nat rule on the VyOS side the pings get through.

    The OpenVPN settings have the local and remote subnets set and I am assuming this is where the static route comes from since I do not have a route manually configured.

    So in short it appears PFSense simply ignores packets from the remote subnet over the tunnel, and only listens for tunnel IPs. Any ideas as to what to look for next?


  • I would need to see the configs from both sides to assist any further, but the fact that pings go through when you source NAT on the VyOS side suggests there's no return route for your LAN subnet on the remote end (VyOS).


  • I agree with you, however
    PFSense route table entry
    10.200.16.0/24  192.168.13.2      UGS      150        1500      ovpns2

    Works with Source NAT configured on VyOS

    [ VM in AWS 10.200.16.165 ] -> [ 10.200.16.123 VyOS in AWS 192.168.13.2 ] -> [192.168.13.1 PFSense in datacenter ]

    Below shows what VyOS sees with the "test3" showing the four return packets from when I used source NAT.

    vyos@VyOS-AMI:~$ show firewall
    
    -----------------------------
    Rulesets Information
    -----------------------------
    --------------------------------------------------------------------------------
    IPv4 Firewall "test":
    
    Active on (eth0,IN)
    
    rule  action   proto     packets  bytes
    ----  ------   -----     -------  -----
    100   accept   all       8        480
      condition - saddr 10.200.16.0/24 daddr 0.0.0.0/0
    
    101   accept   all       0        0
      condition - saddr 10.200.12.0/22 daddr 0.0.0.0/0
    
    10000 accept   all       0        0
      condition - saddr 0.0.0.0/0 daddr 0.0.0.0/0
    
    --------------------------------------------------------------------------------
    IPv4 Firewall "test2":
    
    Active on (vtun0,OUT)
    
    rule  action   proto     packets  bytes
    ----  ------   -----     -------  -----
    100   accept   all       8        480
      condition - saddr 10.200.16.0/24 daddr 0.0.0.0/0
    
    10000 accept   all       0        0
      condition - saddr 0.0.0.0/0 daddr 0.0.0.0/0
    
    --------------------------------------------------------------------------------
    IPv4 Firewall "test3":
    
    Active on (vtun0,IN)
    
    rule  action   proto     packets  bytes
    ----  ------   -----     -------  -----
    100   accept   all       4        240
      condition - saddr 0.0.0.0/0 daddr 10.200.16.165
    
    10000 accept   all       0        0
      condition - saddr 0.0.0.0/0 daddr 0.0.0.0/0
    

    What specifically do you want to see from the PFSense config?


  • What specifically do you want to see from the PFSense config?

    I actually would like to see the openvpn config from both sides.  On PFsense, those files are in "/var/etc/openvpn".  Not sure where they are on VyOS.


  • No problem

    [2.3.4-RELEASE][admin@####]/root: cat /var/etc/openvpn/server2.conf
    dev ovpns2
    verb 6
    dev-type tun
    tun-ipv6
    dev-node /dev/tun2
    writepid /var/run/openvpn_server2.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto tcp-server
    cipher AES-256-CBC
    auth SHA256
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local 65.###.###.###
    tls-server
    server 192.168.13.0 255.255.255.0
    client-config-dir /var/etc/openvpn-csc/server2
    ifconfig 192.168.13.1 192.168.13.2
    tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'vpn.#######.com' 1"
    lport 81
    management /var/etc/openvpn/server2.sock unix
    push "route 10.200.2.0 255.255.254.0"
    route 10.200.16.0 255.255.255.0
    ca /var/etc/openvpn/server2.ca 
    cert /var/etc/openvpn/server2.cert 
    key /var/etc/openvpn/server2.key 
    dh /etc/dh-parameters.4096
    crl-verify /var/etc/openvpn/server2.crl-verify 
    comp-lzo adaptive
    topology subnet
    
    
    vyos@VyOS-AMI:~$ configure
    [edit]
    vyos@VyOS-AMI# show interfaces openvpn
    openvpn vtun0 {
         encryption aes256
         firewall {
             in {
                 name test3
             }
             out {
                 name test2
             }
         }
         hash sha256
         mode client
         openvpn-option "comp-lzo adaptive"
         persistent-tunnel
         protocol tcp-active
         remote-host vpn.#####.com
         remote-port 81
         tls {
             ca-cert-file /config/auth/ca.crt
             cert-file /config/auth/client.crt
             key-file /config/auth/client.key
         }
    }
    [edit]
    vyos@VyOS-AMI# exit
    exit
    
    vyos@VyOS-AMI:~$ show ip route
    Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
           I - ISIS, B - BGP, > - selected route, * - FIB route
    
    S>* 0.0.0.0/0 [210/0] via 10.200.16.1, eth0
    K>* 10.200.2.0/23 via 192.168.13.1, vtun0
    C>* 10.200.16.0/24 is directly connected, eth0
    C>* 127.0.0.0/8 is directly connected, lo
    C>* 192.168.13.0/24 is directly connected, vtun0
    
    vyos@VyOS-AMI:~$ ip a
    1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN group default
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 06:bb:14:31:96:de brd ff:ff:ff:ff:ff:ff
        inet 10.200.16.123/24 brd 10.200.16.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 2600:###::/64 scope global
           valid_lft forever preferred_lft forever
        inet6 fe80::4bb:14ff:fe31:96de/64 scope link
           valid_lft forever preferred_lft forever
    3: vtun0: <pointopoint,multicast,noarp,up,lower_up> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
        link/none
        inet 192.168.13.2/24 brd 192.168.13.255 scope global vtun0
           valid_lft forever preferred_lft forever</pointopoint,multicast,noarp,up,lower_up></broadcast,multicast,up,lower_up></loopback,up,lower_up>
    

  • I have not had a chance to go back into this, but will be jumping in again before long. Ideally I want to have a working tunnel, otherwise I need to create a tunnel between the servers. There are other systems that need replication so this could get messy with a bunch of tunnels  8)

    Any ideas as to what to look for in PFSense routing?


  • I ended up switching to using ipsec, which I wanted to use anyway but didn't feel like working out the NAT translation. Ended up working out well. Though for some reason PFSense reports that VyOS didn't send the MODP for the ESP group even though I have it configured. Not the worst thing since the key is runs PFS, but still not ideal.

    Still would love to figure out why PFSense is having this routing issue. I suppose if it gets real bad I will have to convince my boss to pay for official support. Considering the thousands that we saved from not using Cisco….