[solved] cannot route traffic between site to site OpenVPNs

SpaceBass

It's probably true that 90% of bug reports are, in fact, user error. And I'm fairly sure that's the case here too. In fact, this post will elucidate the shortcomings of my networking chops. But nonetheless, I'm racking my brain on this one and could use a 2nd or 102nd set of eyes.

Lately, I've been having problem setting up site-to-site VPNs with OpenVPN between PF boxes. Maybe -almost certainly -it's because I bragged about being good at it one day not too long ago. I should know better. It's like the time I thought I'd nailed OS X's kerberos implementation and then everything broke…that was 8 years ago...

FIRST a preamble. This isn't a problem I'm experiencing in isolation. I first encountered it when I was doing a very similar setup in a different environment. That setup has a lot in common with my current problem. Those things in common could be the problem, or they could be validating.

This is documented in: NSnet VPN Diagram.001.jpg

On one end is a NetGate PF Appliance. On the other, is a hosted environment. Specifically, it's ProxMox 4.x running on a SoYouStart server. PFSense is a virtual guest. It's WAN NIC has the MAC of the failover IP virtual switch from SYS, and it's assigned the IP block as as /28 with the appropriate upstream gateway. Traffic from the virtual PF install and the other virtual hosts on the lan (a different vbridge on the proxmox box) works perfectly.

If I set up the Netgate, it's noted as 192.168.1.1 in the image below, as server, then traffic passes in one way, to a limited range of hosts. Traffic goes from the client PF box to the server. That's it. Nothing from either LANs make it. If I reverse the tunnel, the pattern holds: client to server, no lan routing.

I've run tcpdump on both sides, and in both cases I can see traffic going into the tunnel on the side from which it originates, but it doesn't come out the other end (with the exception of the line above - client PF box to server PF box).

In my LAN and OpenVPN rules I have any/any rules with any protocol.

Realtime filtering logs in the console don't show anything.

how I fixed it: when to IPsec :(

Onto the main show
My other environment is, admittedly, more complicated. There are multiple PF boxen involved. I share that, not to articulate the complexity, but to highlight that one of the three set-ups works perfectly.

There are three PF boxes in the setup (more actually, but only 3 relevant to this discussion… I think)

1. netgate appliance
2. PF running as a VM on a SYS server running Proxmox 4.x
3. BRAND NEW PF running as a VM on a SYS server running Proxmox 4.x

This is documented in: NSnet VPN Diagram.002.jpg

Everything works perfectly between #1 and #2
On the image those are:
#1: 10.15.1.0/24
#2: 10.50.1.0/24

Between #1 and #3 AND between #2 and #3 I have the exact same problem as noted in my preamble. At best, traffic works in one, marginal connection.

I've run tcpdump, Ive watched realtime filter logs, and as the images hint, I've put 'log' flags on every firewall rule I can think of, even those that will never be triggered because an upstream rule is the first match. Nothing in the logs.

I've read and re-read every howto including the docs on the PF site. I'm sure I'm missing something. I'm sure it's basic and obvious and it's there between 50.x.x and 15.x.x but I just can't find it.

I've tried pushing routes … I've tried making my OpenVPN interface a defined interface in PF (and added any/any rules).

Nothing changes.

So - what do you see? What other troubleshooting tips would you suggest?

![NSnet VPN Diagram.001.jpg](/public/imported_attachments/1/NSnet VPN Diagram.001.jpg)
![NSnet VPN Diagram.001.jpg_thumb](/public/imported_attachments/1/NSnet VPN Diagram.001.jpg_thumb)
![NSnet VPN Diagram.002.jpg](/public/imported_attachments/1/NSnet VPN Diagram.002.jpg)
![NSnet VPN Diagram.002.jpg_thumb](/public/imported_attachments/1/NSnet VPN Diagram.002.jpg_thumb)
![NSnet VPN Diagram.003.jpg](/public/imported_attachments/1/NSnet VPN Diagram.003.jpg)
![NSnet VPN Diagram.003.jpg_thumb](/public/imported_attachments/1/NSnet VPN Diagram.003.jpg_thumb)

10_75_1_1_Tunnel_Settings.jpg_thumb

10_50_1_1_to_10_75_1_1_tunnel_settings.jpg_thumb

10_50_1_1_to_10_15_1_1_tunnel.jpg_thumb

10_75_1_1_LAN_Rules.jpg_thumb

10_75_1_1_OpenVPN_Rules.jpg_thumb

10_50_1_1_LAN_Rules.jpg_thumb

10_50_1_1_OpenVPN_rules.jpg_thumb

10_15_1_1_to_10_75_1_1_tunnel.jpg_thumb

10_15_1_1_LAN_Rules.jpg_thumb

10_15_1_1_OpenVPN_rules.jpg_thumb

SpaceBass

offered as evidence that the mistake is probably user error: my images above :)
Noticed that I have a few of the tunnels labeled incorrectly in the diagrams. The screenshots of the PF screens is canonical.
EG: tunnel between .75 and .15 is:
172.168.11.1 <–--> 172.168.11.2
confirmed (again my diagram is wrong)

Derelict

Traffic going out a certain way requires a route.

Traffic coming in a certain way requires a rule.

You can check the routes in into OpenVPN Diagnostics > Routes on each node.

It looks like the OpenVPN and LAN rules are wide open. Don't forget whatever routes and rules might need to be in place on the hosts at each end.

Don't put too much weight into pinging OpenVPN interfaces. That doesn't always work. Instead, ping from one side selecting the local LAN interface (The one in the Remote Networks on the other side) as the source address and the remote LAN interface as the target/hostname to ping. That's what you're really concerned with anyway. When that is verified to work you know routing and the basic rules are there. Move the ping outward to a host on the remote network. If that doesn't work look at the routing/rules on THAT HOST.

Shared-key vs SSL/TLS also matters. You might need iroutes and Client-Specific overrides. Status > OpenVPN will have a place to show the iRouting table if applicable. The tunnel network size also matters. I see one is a /30 and the others are /24. That will preclude the need for iroutes on the /30 and might well require them on the /24s.

ETA: In fact, just change those tunnel networks from /24 to /30 and see if it all doesn't just fall into place. Be sure to restart both OpenVPN services on every node.

SpaceBass

@Derelict:

Traffic going out a certain way requires a route.

Traffic coming in a certain way requires a rule.

I see one is a /30 and the others are /24. That will preclude the need for iroutes on the /30 and might well require them on the /24s.

ETA: In fact, just change those tunnel networks from /24 to /30 and see if it all doesn't just fall into place. Be sure to restart both OpenVPN services on every node.

Thanks!

Just curious where you are seeing the /30? I dont have a /30 for any of my tunnels (all are /24s)

Here are the routes:
My belief is that these are defined by the OpenVPN GUI when I enter the local and remote LANs.
I do not have any routes or iroutes defined in advanced options.

10.50.1.1

IPv4 Routes
Destination	Gateway	Flags	Use	Mtu	Netif	Expire
default	144.217.108.254	UGS	173106822	1500	em0	
10.5.1.0/24	192.168.11.2	UGS	23484	1500	ovpns1	
10.15.1.0/24	192.168.43.2	UGS	10254437	1500	ovpns4	
10.50.1.0/24	link#2	U	68421290	1500	em1	
10.50.1.1	link#2	UHS	0	16384	lo0	
10.75.1.0/24	172.16.25.2	UGS	4	1500	ovpns5	
10.99.1.0/24	link#3	U	62566644	1500	em2	
10.99.1.1	link#3	UHS	0	16384	lo0	
127.0.0.1	link#7	UH	827	16384	lo0	
144.217.108.176/29	link#1	U	0	1500	em0	
144.217.108.177	link#1	UHS	0	16384	lo0	
144.217.108.178	link#1	UHS	0	16384	lo0	
144.217.108.178/32	link#1	U	0	1500	em0	
144.217.108.179	link#1	UHS	0	16384	lo0	
144.217.108.179/32	link#1	U	0	1500	em0	
144.217.108.180	link#1	UHS	0	16384	lo0	
144.217.108.180/32	link#1	U	0	1500	em0	
144.217.108.181	link#1	UHS	4	16384	lo0	
144.217.108.181/32	link#1	U	0	1500	em0	
144.217.108.254	02:00:00:7f:a9:a3	UHS	20311517	1500	em0	
172.16.25.0/24	172.16.25.1	UGS	0	1500	ovpns5	
172.16.25.1	link#12	UHS	0	16384	lo0	
172.16.25.2	link#12	UH	206038	1500	ovpns5	
178.62.0.0/18	192.168.38.2	UGS	2699	1500	ovpns8

10.15.1.1

IPv4 Routes
Destination	Gateway	Flags	Use	Mtu	Netif	Expire
0.0.0.0/8	link#1	U	0	1500	re0	
default	108.31.103.1	UGS	105025592	1500	re1	
10.5.1.0/24	192.168.94.2	UGS	28953	1500	ovpns7	
10.15.1.0/24	link#3	U	443790691	1500	re2	
10.15.1.1	link#3	UHS	0	16384	lo0	
10.15.8.0/24	link#9	U	4757900	1500	re2_vlan3	
10.15.8.1	link#9	UHS	0	16384	lo0	
10.15.9.0/24	link#8	U	17473563	1500	re2_vlan2	
10.15.9.1	link#8	UHS	0	16384	lo0	
10.50.1.0/24	192.168.43.1	UGS	6976418	1500	ovpnc6	
10.75.1.0/24	172.16.11.1	UGS	7789	1500	ovpnc8	
10.99.1.0/24	192.168.43.1	UGS	6842646	1500	ovpnc6	
68.237.161.12	00:0d:b9:34:b2:95	UHS	25	1500	re1	
71.252.0.12	00:0d:b9:34:b2:95	UHS	25	1500	re1	
108.31.103.0/24	link#2	U	8727504	1500	re1	
108.31.103.106	link#2	UHS	0	16384	lo0	
127.0.0.1	link#7	UH	150460	16384	lo0	
172.16.11.0/24	172.16.11.1	UGS	0	1500	ovpnc8	
172.16.11.1	link#18	UH	0	1500	ovpnc8	
172.16.11.2	link#18	UHS	0	16384	lo0	
192.76.33.0/24	192.168.43.1	UGS	0	1500	ovpnc6	
192.168.27.0/24	192.168.27.2	UGS	120231	1500	ovpns2	
192.168.27.1	link#11	UHS	0	16384	lo0	
192.168.27.2	link#11	UH	1680989	1500	ovpns2	
192.168.42.0/24	192.168.43.1	UGS	13806	1500	ovpnc6	
192.168.43.1	link#14	UH	9752	1500	ovpnc6	
192.168.43.2	link#14	UHS	0	16384	lo0	
192.168.83.1	link#10	UHS	0	16384	lo0	
192.168.83.2	link#10	UH	33962754	1500	ovpns4	
192.168.85.0/24	link#12	U	0	1500	ovpns5	
192.168.85.1	link#12	UHS	0	16384	lo0	
192.168.94.0/24	192.168.94.2	UGS	0	1500	ovpns7	
192.168.94.1	link#15	UHS	0	16384	lo0	
192.168.94.2	link#15	UH	0	1500	ovpns7	
198.27.66.0/24	192.168.83.2	UGS	53113087	1500	ovpns4

10.75.1.1

IPv4 Routes
Destination	Gateway	Flags	Use	Mtu	Netif	Expire
default	144.217.191.62	UGS	238900	1500	em0	
10.5.1.0/24	172.16.25.1	UGS	0	1500	ovpnc1	
10.50.1.0/24	172.16.25.1	UGS	8821	1500	ovpnc1	
10.75.1.0/24	link#2	U	7217	1500	em1	
10.75.1.1	link#2	UHS	0	16384	lo0	
10.99.1.0/24	172.16.25.1	UGS	0	1500	ovpnc1	
127.0.0.1	link#6	UH	3689	16384	lo0	
144.217.191.48/28	link#1	U	0	1500	em0	
144.217.191.49	link#1	UHS	0	16384	lo0	
144.217.191.50	link#1	UHS	0	16384	lo0	
144.217.191.50/32	link#1	U	0	1500	em0	
144.217.191.62	02:00:00:cc:7d:0e	UHS	104260	1500	em0	
172.16.25.0/24	172.16.25.1	UGS	0	1500	ovpnc1	
172.16.25.1	link#7	UH	206623	1500	ovpnc1	
172.16.25.2	link#7	UHS	0	16384	lo0	
192.76.33.0/24	172.16.25.1	UGS	0	1500	ovpnc1	
192.168.42.0/24	172.16.25.1	UGS	0	1500	ovpnc1

NOTE between 10.15.1.1 <–-> 10.75.1.1 I've moved to IPsec... I just needed to get traffic flowing. But I'd really like to go back to OpenVPN once we can suss out what's happening.

Derelict

/30: https://forum.pfsense.org/index.php?action=dlattach;topic=129155.0;attach=98690

When you run Peer to Peer SSL/TLS with larger than a /30 subnet on the tunnel network it is considered a PTMP server and you need routes into OpenVPN (Remote Networks in the server) and iroutes to each client (even if there is only one) using Remote Networks in Client-Specific Overrides.

When the tunnel network is a /30 it is considered a PTP connection and the iroutes are not necessary.

SpaceBass

@Derelict:

/30: https://forum.pfsense.org/index.php?action=dlattach;topic=129155.0;attach=98690

When you run Peer to Peer SSL/TLS with larger than a /30 subnet on the tunnel network it is considered a PTMP server and you need routes into OpenVPN (Remote Networks in the server) and iroutes to each client (even if there is only one) using Remote Networks in Client-Specific Overrides.

When the tunnel network is a /30 it is considered a PTP connection and the iroutes are not necessary.

Ah! Interesting!
HOLY COW! That was it, that's the key!
Thank you - that fixed everything!

I saw this:

For site-to-site shared key, only a /30 is used, not a /24, even if /24 is specified.

but didn't interpret it as clearly as how you explained it above.

Thank you!

Back to OpenVPN for everything!