PFSense not communicating with remote subnet
-
I have a Peer-to-Peer OpenVPN tunnel established between PFsense in our datacenter and a VyOS VM running in an AWS VPC. PFSense can ping the vpn ip of VyOS and vice versa. However, the subnet on the VyOS side is unreachable.
PFSense source IP: 192.168.13.1 -> VyOS VPN IP 192.168.13.2 -> remote test ip: 10.200.16.165
In the PFSense routing table I see
10.200.16.0/24 via 192.168.13.2 UGSA traceroute from PFSense shows no first hops. Pinigng from the other end however shows packets flowing through VyOS and through the tunnel, however I do not receive any traffic back. I have a firewall rule on PFSense to track packets from the remote subnet but nothing ever reaches it. There are no other rules ahead of this one. For testing purposes I even created a temporary allow all rule and this still did not log any packets from the remote subnet. If I create a source nat rule on the VyOS side the pings get through.
The OpenVPN settings have the local and remote subnets set and I am assuming this is where the static route comes from since I do not have a route manually configured.
So in short it appears PFSense simply ignores packets from the remote subnet over the tunnel, and only listens for tunnel IPs. Any ideas as to what to look for next?
-
I would need to see the configs from both sides to assist any further, but the fact that pings go through when you source NAT on the VyOS side suggests there's no return route for your LAN subnet on the remote end (VyOS).
-
I agree with you, however
PFSense route table entry
10.200.16.0/24 192.168.13.2 UGS 150 1500 ovpns2Works with Source NAT configured on VyOS
[ VM in AWS 10.200.16.165 ] -> [ 10.200.16.123 VyOS in AWS 192.168.13.2 ] -> [192.168.13.1 PFSense in datacenter ]
Below shows what VyOS sees with the "test3" showing the four return packets from when I used source NAT.
vyos@VyOS-AMI:~$ show firewall ----------------------------- Rulesets Information ----------------------------- -------------------------------------------------------------------------------- IPv4 Firewall "test": Active on (eth0,IN) rule action proto packets bytes ---- ------ ----- ------- ----- 100 accept all 8 480 condition - saddr 10.200.16.0/24 daddr 0.0.0.0/0 101 accept all 0 0 condition - saddr 10.200.12.0/22 daddr 0.0.0.0/0 10000 accept all 0 0 condition - saddr 0.0.0.0/0 daddr 0.0.0.0/0 -------------------------------------------------------------------------------- IPv4 Firewall "test2": Active on (vtun0,OUT) rule action proto packets bytes ---- ------ ----- ------- ----- 100 accept all 8 480 condition - saddr 10.200.16.0/24 daddr 0.0.0.0/0 10000 accept all 0 0 condition - saddr 0.0.0.0/0 daddr 0.0.0.0/0 -------------------------------------------------------------------------------- IPv4 Firewall "test3": Active on (vtun0,IN) rule action proto packets bytes ---- ------ ----- ------- ----- 100 accept all 4 240 condition - saddr 0.0.0.0/0 daddr 10.200.16.165 10000 accept all 0 0 condition - saddr 0.0.0.0/0 daddr 0.0.0.0/0What specifically do you want to see from the PFSense config?
-
What specifically do you want to see from the PFSense config?
I actually would like to see the openvpn config from both sides. On PFsense, those files are in "/var/etc/openvpn". Not sure where they are on VyOS.
-
No problem
[2.3.4-RELEASE][admin@####]/root: cat /var/etc/openvpn/server2.conf dev ovpns2 verb 6 dev-type tun tun-ipv6 dev-node /dev/tun2 writepid /var/run/openvpn_server2.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto tcp-server cipher AES-256-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 65.###.###.### tls-server server 192.168.13.0 255.255.255.0 client-config-dir /var/etc/openvpn-csc/server2 ifconfig 192.168.13.1 192.168.13.2 tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'vpn.#######.com' 1" lport 81 management /var/etc/openvpn/server2.sock unix push "route 10.200.2.0 255.255.254.0" route 10.200.16.0 255.255.255.0 ca /var/etc/openvpn/server2.ca cert /var/etc/openvpn/server2.cert key /var/etc/openvpn/server2.key dh /etc/dh-parameters.4096 crl-verify /var/etc/openvpn/server2.crl-verify comp-lzo adaptive topology subnetvyos@VyOS-AMI:~$ configure [edit] vyos@VyOS-AMI# show interfaces openvpn openvpn vtun0 { encryption aes256 firewall { in { name test3 } out { name test2 } } hash sha256 mode client openvpn-option "comp-lzo adaptive" persistent-tunnel protocol tcp-active remote-host vpn.#####.com remote-port 81 tls { ca-cert-file /config/auth/ca.crt cert-file /config/auth/client.crt key-file /config/auth/client.key } } [edit] vyos@VyOS-AMI# exit exit vyos@VyOS-AMI:~$ show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - ISIS, B - BGP, > - selected route, * - FIB route S>* 0.0.0.0/0 [210/0] via 10.200.16.1, eth0 K>* 10.200.2.0/23 via 192.168.13.1, vtun0 C>* 10.200.16.0/24 is directly connected, eth0 C>* 127.0.0.0/8 is directly connected, lo C>* 192.168.13.0/24 is directly connected, vtun0 vyos@VyOS-AMI:~$ ip a 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 06:bb:14:31:96:de brd ff:ff:ff:ff:ff:ff inet 10.200.16.123/24 brd 10.200.16.255 scope global eth0 valid_lft forever preferred_lft forever inet6 2600:###::/64 scope global valid_lft forever preferred_lft forever inet6 fe80::4bb:14ff:fe31:96de/64 scope link valid_lft forever preferred_lft forever 3: vtun0: <pointopoint,multicast,noarp,up,lower_up> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100 link/none inet 192.168.13.2/24 brd 192.168.13.255 scope global vtun0 valid_lft forever preferred_lft forever</pointopoint,multicast,noarp,up,lower_up></broadcast,multicast,up,lower_up></loopback,up,lower_up> -
I have not had a chance to go back into this, but will be jumping in again before long. Ideally I want to have a working tunnel, otherwise I need to create a tunnel between the servers. There are other systems that need replication so this could get messy with a bunch of tunnels 8)
Any ideas as to what to look for in PFSense routing?
-
I ended up switching to using ipsec, which I wanted to use anyway but didn't feel like working out the NAT translation. Ended up working out well. Though for some reason PFSense reports that VyOS didn't send the MODP for the ESP group even though I have it configured. Not the worst thing since the key is runs PFS, but still not ideal.
Still would love to figure out why PFSense is having this routing issue. I suppose if it gets real bad I will have to convince my boss to pay for official support. Considering the thousands that we saved from not using Cisco….