OpenVPN peer to peer - connects but won't pass traffic

mclaborn · Nov 13, 2017, 5:02 PM

Site A - OpenVPN server
pfSense 2.4.1
Open VPN server running on port 1195
Has 3 WAN interfaces. I'm trying to get the VPN running on WAN1.

Site B - OpenVPN client
pfSense 2.3.4

I've tripled checked the OpenVPN settings between client and server and they match. The status page indicates that the VPN is connected, but I am unable from a computer on the B network to ping anything on the A network. A computer on the B network can ping both sides of the private VPN network (10.0.27.1 and 10.0.27.2). I can ping from the pfSense B to various addresses on the A network.

B has an allow all firewall rule on the Open VPN tab.

A has an allow all firewall rule on the Open VPN tab.
A has an allow all firewall rule for port 1195 on the WAN1 tab. I've tried the destination as both "any" and "this firewall" with the same results.
A has an outbound NAT rule -
Interface:WAN1
Source: B's network/24
Source Port: *
Destination: *
Destination Port: *
NAT Address: WAN1 address
NAT Port: *

There are other Outbound NAT rules for B's network/24 but they are for specific addresses (both pfSense boxes). These allow me to connect to the pfSense boxes when using IPSec VPN.

Any ideas on what I'm doing wrong? I suspect the NAT rule - mostly because I've never been quite able to wrap my mind around how those work.

viragomann · Nov 13, 2017, 9:45 PM

@mclaborn:

but I am unable from a computer on the B network to ping anything on the A network. A computer on the B network can ping both sides of the private VPN network (10.0.27.1 and 10.0.27.2). I can ping from the pfSense B to various addresses on the A network.

That seems that the site B pfSense isn't the default gateway in the site B network. If it isn't you need a static route for the site A network on each device at site B you want to access or you do NAT on the VPN.

@mclaborn:

A has an outbound NAT rule -
Interface:WAN1
Source: B's network/24
Source Port: *
Destination: *
Destination Port: *
NAT Address: WAN1 address
NAT Port: *

That rule only makes sense for accessing the internet from site B over the VPN. Don't know if that's what you want.

mclaborn · Nov 14, 2017, 3:47 PM

Here is the routing table for my computer on the B network.

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.5.1     0.0.0.0         UG    100    0        0 eno1
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eno1
172.16.52.0     0.0.0.0         255.255.255.0   U     0      0        0 vmnet1
172.16.150.0    0.0.0.0         255.255.255.0   U     0      0        0 vmnet8
192.168.5.0     0.0.0.0         255.255.255.0   U     100    0        0 eno1

The B network is 192.168.5.0/24 with the gateway as .1.

As far as I can tell, the routes on the B pfSense are correct:

Destination	Gateway	Flags	Use	Mtu	Netif	Expire
default	47.186.30.1	UGS	6287086	1500	em0	
10.0.27.1	link#8	UH	5	1500	ovpnc2	
10.0.27.2	link#8	UHS	0	16384	lo0	
10.1.16.0/24	10.1.16.2	UGS	0	1500	ovpns1	
10.1.16.1	link#7	UHS	0	16384	lo0	
10.1.16.2	link#7	UH	1544773	1500	ovpns1	
47.186.30.0/24	link#1	U	1356533	1500	em0	
47.186.30.3	link#1	UHS	0	16384	lo0	
67.232.254.142	47.186.30.1	UGHS	126810	1500	em0	
127.0.0.1	link#6	UH	197111	16384	lo0	
172.16.0.0/24	10.0.27.1	UGS	181240	1500	ovpnc2	
192.168.5.0/24	link#2	U	4555834	1500	em1	
192.168.5.1	link#2	UHS	0	16384	lo0	
207.177.38.109	47.186.30.1	UGHS	162037	1500	em0

The A network is 172.16.0.0/24.

mclaborn · Nov 14, 2017, 3:59 PM

@viragomann:

That rule only makes sense for accessing the internet from site B over the VPN. Don't know if that's what you want.

I don't really need to access the outside world over the VPN. I disabled all the outbound NAT rules for B's network, but it didn't help.

mclaborn · Nov 14, 2017, 4:03 PM

Traceroute from my computer on the B network below. Looks like the routing is correct, but something is not passing the traffic?

traceroute -n 172.16.0.212
traceroute to 172.16.0.212 (172.16.0.212), 30 hops max, 60 byte packets
 1  192.168.5.1  0.453 ms  0.440 ms  0.560 ms
 2  10.0.27.1  30.880 ms  30.886 ms  30.879 ms
 3  * * *
 4  * * *

viragomann · Nov 14, 2017, 4:28 PM

Do you run multiple OpenVPN instances on one site, either server or client?

mclaborn · Nov 14, 2017, 4:41 PM

@viragomann:

Do you run multiple OpenVPN instances on one site, either server or client?

Yes, the server side (A) has another Open VPN instance on port 1194 for single users rather than peer to peer. I've been using that one to connect to network A since 2.4 messed up our IPSec VPNs.

The client pfSense (B) also runs an Open VPN instance for single users on port 1194.

viragomann · Nov 14, 2017, 5:01 PM

So you should assign an interface to each OpenVPN instance. Have you done this?

mclaborn · Nov 15, 2017, 10:36 PM

@viragomann:

So you should assign an interface to each OpenVPN instance. Have you done this?

I'm sure my ignorance will show, but I'm not quite sure what you mean. Are you saying that each OpenVPN server instance needs to be assigned to a different interface on the pfSense, even though they are listening on different ports?

viragomann · Nov 15, 2017, 11:37 PM

No, you miss-understood. An interface has to be assigned to the VPN instance.
Interfaces > assign.

At "available network ports" select the vpn instance (ovpnc1, ovpns1) and hit add, then click the new interface, enable it and set an appropriate name, no further settings to be made.
After that you should move the needed firewall rules to the new interfaces. "OpenVPN" is handled as interface group containing all OpenVPN instances. Rules on that tab effects all OpenVPN instances.

mclaborn · Nov 20, 2017, 3:52 PM

I must still be missing something.

I assigned the interfaces as you suggested, on both the server and client sides. On the server side, ovpns1 is assigned to OPT5 for the original server that is used by various individuals. ovpns2 is assigned to OPT6 for the new peer-to-peer that is not yet working.

On the client side, ovpns1 is assigned to OPT1 for the server running there - I use this from my computer when traveling. ovpnc2 is assigned to OPT2 - the other side of the peer-to-peer that is not yet working.

Both client and server have an allow all rule for IPV4 on the OpenVPN tab.

The VPN shows connected on both sides.

I can still ping from the client pfSense to addresses on the server network if I use "Open VPN Client (perr-to-peer)" as the source address. I think this means that traffic can pass from the client pfSense over the VPN tunnel.

The routes on the client pfSense appear to me to be correct, as do the routes on the computers on the client network.

viragomann · Nov 20, 2017, 5:50 PM

@mclaborn:

I can still ping from the client pfSense to addresses on the server network if I use "Open VPN Client (perr-to-peer)" as the source address. I think this means that traffic can pass from the client pfSense over the VPN tunnel.

And if you select the LAN address here it's not working?

mclaborn · Nov 21, 2017, 2:15 AM

Correct. I can ping (from pfSense) addresses on the server network from "Open VPN Client" but cannot ping from LAN.

The routes look correct as far as I can tell (see attached). There is a an allow all rule from the LAN network to any.

viragomann · Nov 21, 2017, 11:06 AM

The routes on site A pfSense may be essential here. Check if there is a route for 192.168.5.0/24 pointing to the VPN clients IP.

mclaborn · Nov 21, 2017, 3:31 PM

Yes, there is a route on site A to B's internal network. See attached.
10.0.27.2 is the client end of the VPN tunnel network.
I can also ping from pfSense A to addresses on B's internal network if I use the OPT6 (VPN) source address. It fails using the LAN address.

I forgot to mention earlier - the LAN, WAN1 and WAN2 interfaces on site A are CARP. The OpenVPN server is not running on the CARP virtual IP address but the real address of WAN1 on one of the pfSense boxes in site A.

viragomann · Nov 21, 2017, 3:51 PM

So it seems the LAN computer does not response if the source address is one from LAN B network.
Hardly to believe, since it does though if the source address is out of the vpn tunnel.
Maybe the response goes up to another gateway?

To see what's going on here, run a packet capture on site A LAN while you try to ping a LAN computer from site B LAN. Set to filter to ICMP to avoid too much noise.
Also run a capture while pinging the same computer from the site B pfSense with the default source.

mclaborn · Nov 21, 2017, 4:51 PM

I am now more confused than ever.

Trying from a computer on the B network, a packet capture on site A LAN does not capture anything. A capture on the site A OpenVPN interface captures echo and reply messages between the 2 sides of the tunnel.

10:42:40.190119 AF IPv4 (2), length 32: (tos 0x0, ttl 64, id 3651, offset 0, flags [none], proto ICMP (1), length 28)
    10.0.27.1 > 10.0.27.2: ICMP echo request, id 41407, seq 928, length 8
10:42:40.228986 AF IPv4 (2), length 32: (tos 0x0, ttl 64, id 29671, offset 0, flags [none], proto ICMP (1), length 28)
    10.0.27.2 > 10.0.27.1: ICMP echo reply, id 41407, seq 928, length 8
10:42:40.595307 AF IPv4 (2), length 32: (tos 0x0, ttl 64, id 26687, offset 0, flags [none], proto ICMP (1), length 28)
    10.0.27.2 > 10.0.27.1: ICMP echo request, id 39100, seq 931, length 8
10:42:40.595328 AF IPv4 (2), length 32: (tos 0x0, ttl 64, id 50986, offset 0, flags [none], proto ICMP (1), length 28)
    10.0.27.1 > 10.0.27.2: ICMP echo reply, id 39100, seq 931, length 8

From pfSense B default source, capturing on the A site Lan:

10:47:36.624118 0c:c4:7a:17:17:57 > ec:f4:bb:cf:2b:60, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 53205, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.27.2 > 172.16.0.212: ICMP echo request, id 39056, seq 0, length 64
10:47:36.624306 ec:f4:bb:cf:2b:60 > 00:00:5e:00:01:03, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 27802, offset 0, flags [none], proto ICMP (1), length 84)
    172.16.0.212 > 10.0.27.2: ICMP echo reply, id 39056, seq 0, length 64

From pfSense B default source, capturing on site A OpenVPN:

10:49:52.181762 AF IPv4 (2), length 32: (tos 0x0, ttl 64, id 4772, offset 0, flags [none], proto ICMP (1), length 28)
    10.0.27.2 > 10.0.27.1: ICMP echo request, id 39100, seq 1771, length 8
10:49:52.181786 AF IPv4 (2), length 32: (tos 0x0, ttl 64, id 10633, offset 0, flags [none], proto ICMP (1), length 28)
    10.0.27.1 > 10.0.27.2: ICMP echo reply, id 39100, seq 1771, length 8
10:49:52.278123 AF IPv4 (2), length 32: (tos 0x0, ttl 64, id 37252, offset 0, flags [none], proto ICMP (1), length 28)
    10.0.27.1 > 10.0.27.2: ICMP echo request, id 41407, seq 1761, length 8
10:49:52.314909 AF IPv4 (2), length 32: (tos 0x0, ttl 64, id 19078, offset 0, flags [none], proto ICMP (1), length 28)
    10.0.27.2 > 10.0.27.1: ICMP echo reply, id 41407, seq 1761, length 8

viragomann · Nov 21, 2017, 5:23 PM

The captures on the OpenVPN interface only show ICMP packets between the OpenVPN client an server, not any packet destined to the LAN computer.
These seem to be the pings from dpinger on both sites. For Testing it will be better to turn off dpinger on the OpenVPN interfaces.

Maybe packets from the site B LAN are never routed into the tunnel.

mclaborn · Nov 22, 2017, 4:57 PM

The packets from site B LAN are making it to and through the tunnel. Here is a packet trace on the Open VPN interface on site A (server):

10:47:34.720334 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 60640, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.5.5 > 172.16.0.212: ICMP echo request, id 51458, seq 0, length 64
10:47:35.724967 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 55095, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.5.5 > 172.16.0.212: ICMP echo request, id 51458, seq 1, length 64

The same trace on the LAN interface of A does not see those packets. I suppose that means that either pfSense A does not know how to route those packets, or they are being blocked. pfSense A can ping an internal address from both the LAN and OpenVPN interfaces.

There are 2 OpenVPN tabs in the firewall -> rules page and both of them have an allow all rule.

Does this setup need an outbout NAT rule? I've tried both with and without one - see attached.

viragomann · Nov 22, 2017, 6:28 PM

The outbound NAT should not be necessary if the routing work.
Beyond that, the rule source in the route would have to be the site B LAN network.

What's your OpenVPN tunnel settings? At server it should be a /30 subnet, on client the tunnel field it should be empty.