OpenVPN Routing Issue? (FreeBSD route add command failed)

[Scampicfx]

Dear folks,

I went through a lot of websites, videos and tutorials in order to get a simple OpenVPN test-connection between two sites up and running.

Mainly I focussed on:

• https://doc.pfsense.org/index.php/OpenVPN_Site-to-Site_PKI_(SSL)
• https://forum.pfsense.org/index.php?topic=53950.0 (posting from jimp)
  I configured the Certificates, OpenVPN Server and OpenVPN Client. I skipped the Client Specific Overrides since for testing purpose I would like to establish only a connection between two sites. More sites should be added a later point. When I get it correctly, Client Specific Overrides are only necessary for sites larger than 2?

In my scenario I have one main site and one remote site. I attached a brief overview about the network (see below).

After configuring, it looks like the OpenVPN connection gets established. When looking at Status -> OpenVPN of the router of the remote site, it shows:
Client Instance Statistics: Status up

However, when trying to send a ping command from Router A to Router B (or vice versa) I dont' receive any response.

I had a look to System Logs -> Open VPN of the Main Site and discovered:

ERROR: FreeBSD route add command failed: external program exited with error status: 1 

I'm not quite sure, what this means? I guess it has something to do with Routing?

Since the connections gets established (Status "Up"), I assume that the firewall rules are working and there is only a problem in routing?

Here's a brief overview about my Open VPN tunnel settings:

Main Site Open VPN Server configuration:
IPv4 Tunnel Network: 10.0.121.0/24
IPv4 Local networks: 10.2.100.0/24
IPv4 Remote networks: 10.1.100.0/24

Remote Site Open VPN Client configuration:
IPv4 Tunnel Network: 10.0.121.0/24
IPv4 Remote network: 10.2.100.0/24

Could there be an error in the firewall config, although the client shows VPN Status "Up" and although it displays Connected Since "Thu Jan 5 23:17:00 2017"?

Since I'm completely new to pfSense, I'm a bit unsure where to start searching for the problem? I'm not sure if it's just a routing problem? At some tutorials I saw people inserting some route-push commands into the advanced configuration of OpenVPN Server. However, in the tutorial https://doc.pfsense.org/index.php/OpenVPN_Site-to-Site_PKI_(SSL) I can't find anything about it…

Do I have to insert certain commands in the Advanced Configuration Textarea of the OpenVPN Server?
I just inserted

push "route 10.2.100.0 255.0.0.0"

in the server config, however it is still the same problem :/

EDIT: fixed the subnets…

[Scampicfx]

Just to be sure the firewall is configured correctly:

Server:

(Note: The Client has the same configuration as picture Unbenannt-1.jpg; However, the client does not have the additional OpenVPN rule of Unbenannt-2.jpg).

[heper]

IPv4 Local networks: 10.2.100.0/8
IPv4 Remote networks: 10.1.100.0/8

those subnets overlap. fix those ridiculous lan subnets on both ends & your problem will more then likely be solved

[Scampicfx]

Hey,

thanks for your quick reply.

I changed the local subnets to:

LAN Main Site: 10.2.100.0**/16**
LAN Remote Site: 10.1.100.0**/16**

I reconfigured and attached two screenshots of the OpenVPN Tunnel configuration of the OpenVPN Server and the OpenVPN Client.

However, I still receive the error message on the OpenVPN pfSense Client.

ERROR: FreeBSD route add command failed: external program exited with error status: 1 

I tried a configuration as seen on the two screenshots attached below. But as mentioned, it results in the same error.

One addition to the screenshots:

I also tried ….

OpenVPN Server:
IPv4 Local Networks: 10.2**.0.0/16
IPv4 Remote Networks: 10.1.0.**0/16

OpenVPN Client:
IPv4 Remote Networks: 10.2**.0.**0/16

… however, it results in the same error: VPN Status shows "up", however, still the same error code (ERROR: FreeBSD route add command failed: external program exited with error status: 1 ) is displayed at Status -> System Logs -> OpenVPN.

[Scampicfx]

Hey,

I had a look to Diagnostics -> Routes.

As attachments, you can see the Routes displayed by Server and displayed by Client.

To sum up the configuration once again:

OpenVPN Server Site:
pfSense LAN IP address: 10.2.100.1/16
Server mode: Peer to Peer (SSL / TLS)
Protocol: UDP
Device mode: tun
Interface: WAN
Local port: 1195

IPv4 Tunnel Network: 10.0.121.0/24
IPv4 Local network(s): 10.2.0.0/16
IPv4 Remote network(s): 10.1.0.0/16

Dynamic IP checkmark
Address Pool checkmark
Topology: Subnet

No custom options!
No Client Specific Overrides

Firewall Config:
Firewall Rules -> WAN:
IPv4 UDP * * WAN address 1195 * none

Firewall Rules -> OpenVPN:
IPv4* * * * * * none

OpenVPN Remote Site (Client):
pfSense LAN IP address: 10.1.100.1/16
Server mode: Peer to Peer (SSL / TLS)
Protocol: UDP
Device mode: tun
Interface: WAN

Local port: blank
Server host or address: [HOSTNAME OF MAIN SITE]
Server port: 1195
Server hostname resolution: checkmark

IPv4 Tunnel Network: 10.0.121.0/24
IPv4 Remote Networks: 10.2.0.0/16

No custom options
No Client Specific Overrides

Firewall Config
Firewall -> WAN:
IPv4 UDP * * WAN address 1194 (OpenVPN) * none (Remark: For Road Warrior, has nothing to do with Site-to-Site Connection)

Firewall -> OpenVPN:
IPv4* * * * * * none

PS: On the Remote Site, there is one OpenVPN Server configured for Road Warrior Access. Does this affect the OpenVPN service for Site-to-Site connection?

In case anyone has an idea, where to start searching for the problem, I would be really thankful! Client side shows VPN Status "up", however, the pfSense Routers can't ping each other… Since I'm completely new to networking, I'm not sure what the Diagnostics -> Routes should show to me! I attached the screenshots…

[heper]

the tunnel network should be identical on the server & client

[Scampicfx]

Ahh sorry, that's just a typo. I corrected it. Tunnel network is identical…

[imaginary_number]

When you were doing pings, were you using the tunnel IP addresses?  In a site-to-site tunnel scenario like this they are probably 10.0.121.1 and 10.0.121.2 for the server and client respectively.  Both devices should be able to ping both addresses.

If it's being dumb you can try formatting the ping command like this (assuming from the console, server to client version):
ping -S 10.0.121.1 10.0.121.2

and respectively on the client:
ping -S 10.0.121.2 10.0.121.1

Should be able to do that much without routing being involved, maybe rule out a problem with traffic on the tunnel itself…

[Scampicfx]

Hey cool, thank you so much for this hint!

I just discovered that it is possible to ping from Client Site (10.1.100.1) to Main Site (10.2.100.1).

From Client Site, I tried to ping the Main Site Router itself (10.2.100.1) as well as one computer in it's Network (10.2.100.10). The ping was returned! Cool! :)

However, that's all what I achieved! Accessing for example the web configuration of the Main Site Router (from the Client Site) is not working. Hmm… I don't get this... How can it be, that a ping works but accessing the webpanel of the Main Site Router doesn't?

I attached screenshots from my current settings...

Remark: On the Client Site, there is a VPN Road Warrior Server running. I hope this doesn't interfere with Site-to-Site Connection. The Road Warrior is using the default tunnel 10.0.0.8 (as can be seen on the screenshot).

EDIT:
I just tried a packet capture.
When pinging from Mainsite pfsense shell (10.2.100.1) to Remote Site pfSense (10.1.100.1) I receive the following packet capture:

10:53:18.833129 IP 10.0.121.1 > 10.1.100.1: ICMP echo request, id 56155, seq 0, length 64
10:53:19.842638 IP 10.0.121.1 > 10.1.100.1: ICMP echo request, id 56155, seq 1, length 64
10:53:20.852518 IP 10.0.121.1 > 10.1.100.1: ICMP echo request, id 56155, seq 2, length 64

When doing a ping from Remote-Site pfsense (10.1.100.1) to Main Site pfSense (10.2.100.1), packet capture shows the following:

10:56:56.971346 IP 10.0.121.2 > 10.2.100.1: ICMP echo request, id 32190, seq 0, length 64
10:56:56.981851 IP 10.2.100.1 > 10.0.121.2: ICMP echo reply, id 32190, seq 0, length 64
10:56:57.972220 IP 10.0.121.2 > 10.2.100.1: ICMP echo request, id 32190, seq 1, length 64
10:56:57.982162 IP 10.2.100.1 > 10.0.121.2: ICMP echo reply, id 32190, seq 1, length 64
10:56:59.035098 IP 10.0.121.2 > 10.2.100.1: ICMP echo request, id 32190, seq 2, length 64
10:56:59.065843 IP 10.2.100.1 > 10.0.121.2: ICMP echo reply, id 32190, seq 2, length 64

Furthermore, I just entered …

route 10.1.100.0 255.255.255.0;
push "route 10.2.100.0 255.255.255.0"

… into the Advanced Configuration of the OpenVPN Server on the Main Site... However, in exactly that moment the error...

ERROR: FreeBSD route add command failed: external program exited with error status: 1 

… gets triggered on the Main Site. So I deleted again - regarding pinging from the Remote Site, this makes no difference!

[Scampicfx]

Okay I got it working now…

For anyone who might get into the same trouble as me, I did two things:

• check, if there are VPN Connections referring to the same subnet as your Site-to-Site connection! => I deleted my Road Warrior Setup on the Client Site! It was referring to the same subnet (10.1.100.1) as the Site-to-Site Connection between Main Site and Remote Site.
• I switched from PKI to shared key Site-to-Site

and it's running :D However, one problem occured:

When switching from PKI to shared Key, on the Server, I am missing the option "Supply DNS servers to the client"... So, Hostname Resolution of the Clients of the remote VPN network does not work. Is it possible to provide hostname resolution also with shared key OpenVPN?

[hobbes80]

I have been having the exact same problem that you have had and I fixed it the exact same way, by switching from PKI over Shared-Key.

I would REALLY rather use PKI as my planned design is hub and spoke, as well as the ease of configuration for many clients. If anyone has any ideas as to what caused it to suddenly break free and start working when it was changed over to Shared Key, I'd love to hear those ideas.

[Scampicfx]

Hi hobbes,

I haven't tried it yet, but I think hub-and-spoke should also work with shared key, shouldn't it? (-> Client Specific Overrides?)

I like shared key, because every openvpn server is running in it's own instance / process :)

[hobbes80]

Hey Scampicfx–

I fixed my problem yesterday. For me, the issue was a couple fold:

1. I was assuming that each client override had to specify their individual /30 range. The reality is that the client override just uses the same as the primary, it just needs to be big enough for the number of clients. So a /24 would break out into about 64 client networks. This is in the documentation I somehow just didn't understand it the first 8 times I read it.

2. I was assuming that by specifying the individual client networks in the override, it would insert all needed routes. This was a wrong assumption. I needed to insert all my individual client networks in the server config (separated by commas), and then use the client config to specify which client got which route... so instead of the override saying "use this instead" it was saying "of the ones you know, this client = this network"

3. My client and the server were not negotiating compression with each other. When I disabled compression on both sides after fixing bullets 1 and 2, I suddenly started seeing pings.

4. Patience. All of the routes took anywhere from 30 seconds to 2 mins to fully propagate after the VPN was "up". So instead of waiting that time, I was pinging, seeing no response, then making a change and restarting the wait time. It is likely I had the correct configuration a few times before I realized it was working because of this lack of patience.

I didn't like shared key because it meant I had to manage a whole bunch of server processes and a whole slew of open ports. By using SSL I could have just one instance, just one port, and then I can deploy a standard configuration to all my sites with the exception to the client certificate.... but to each their own.

[Scampicfx]

Dear hobbes80,

thank you for your posting.

I appreciate it when people share their knowledge! It might help other users out there!

I haven't tried yet the client specific override, but when I got it correctly from your posting, a hub-and-spoke setup does only work when doing a VPN setup like you did?
I just had a look to the client specific override page and at the beginning I'm requested to enter the client's X.509 common name. However, this name doesn't exist when doing a shared key setup, does it?

I'm wondering if it is possible to setup hub and spoke when using shared key method as well…?

[hobbes80]

Hub and spoke from the perspective of one running OpenVPN server and a bunch of clients only works with SSL.

Hub and spoke from the perspective of many external places connecting back to one datacenter can be configured with shared, but you'd need to set it a different OpenVPN server for every client. Which is why I didn't want to go down that path.

The client override only applies to certificates that exist in the certificate manager, whether imported from somewhere else or created internally.