Routing Problem in Test Network

Alejandro.Carbonara

I've been playing with a small vagrant test network, trying to get a VPN set up.
The network looks like:

Client: 192.168.57.2/24 (WAN)
pfSense: 192.168.57.3/24 (WAN)
192.168.58.2/24 (LAN)
Internal: 192.168.58.3/24 (LAN)

I've set up pfSense's openvpn using the wizard, mostly sticking to the defaults.

IPv4 Tunnel Network : 192.168.80.0/24
IPv4 Local network(s): 192.168.58.0/24
Custom Options: push "route 192.168.58.0 255.255.255.0"

Afterwards, I set up an account, exported the configuration file, and connected to pfsense using the client.

openvpn –config configFile.ovpn --auth-user-pass auth.txt
Wed Oct 11 16:26:52 2017 WARNING: file 'auth.txt' is group or others accessible
Wed Oct 11 16:26:52 2017 OpenVPN 2.4.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jun 30 2017
Wed Oct 11 16:26:52 2017 library versions: OpenSSL 1.0.2l 25 May 2017, LZO 2.08
Wed Oct 11 16:26:52 2017 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.57.3:1194
Wed Oct 11 16:26:52 2017 UDP link local (bound): [AF_INET][undef]:1194
Wed Oct 11 16:26:52 2017 UDP link remote: [AF_INET]192.168.57.3:1194
Wed Oct 11 16:26:52 2017 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
Wed Oct 11 16:26:52 2017 [server] Peer Connection Initiated with [AF_INET]192.168.57.3:1194
Wed Oct 11 16:26:53 2017 TUN/TAP device tun0 opened
Wed Oct 11 16:26:53 2017 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Wed Oct 11 16:26:53 2017 /sbin/ip link set dev tun0 up mtu 1500
Wed Oct 11 16:26:53 2017 /sbin/ip addr add dev tun0 192.168.80.2/24 broadcast 192.168.80.255
RTNETLINK answers: File exists
Wed Oct 11 16:26:53 2017 ERROR: Linux route add command failed: external program exited with error status: 2
Wed Oct 11 16:26:53 2017 Initialization Sequence Completed

In another tab, I attempted to connect to the internal machine. What I found was that I could reach the LAN interface of the pfsense machine from the Client machine, but not the internal machine. Looking into the logs, I found that this happened.

Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 UDPv4 READ [133] from [AF_INET]192.168.57.2:1194: P_DATA_V1 kid=0 DATA 8f1629dd d9f14fff a811ffc6 22fe80eb dc68cf72 9e0fd397 5d4a6621 c8a7a3e[more…]
Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 TLS: tls_pre_decrypt, key_id=0, IP=[AF_INET]192.168.57.2:1194
Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 DECRYPT IV: 9e0fd397 5d4a6621 c8a7a3e5 e7aebda1
Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 DECRYPT TO: 00000012 45000054 84ae4000 4001aaa4 c0a85002 c0a83a03 080061de 4313000[more…]
Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 PID_TEST [0] [SSL-0] [1233456789>>>>>EE] 0:17 0:18 t=1507753641[0] r=[-2,64,15,0,1] sl=[47,17,64,528]
Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 GET INST BY VIRT: 192.168.80.2 -> client1/192.168.57.2:1194 via 192.168.80.2
Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 GET INST BY VIRT: 192.168.58.3 [failed]

It appears that the machine is having problems routing from the pfsense machine to the internal machine.
However, I checked that both the pfsense machine and the internal machine could ping each other within the network.
I'm at a loss of how to fix this problem. Can anyone help?

viragomann

Adding the route fails on the client as the log shows:

Wed Oct 11 16:26:53 2017 ERROR: Linux route add command failed: external program exited with error status: 2

The "push route" command in the custom options is not needed, since this is already set by the entry in "Local Network(s)". So delete it from the custom options field. Maybe this solve your issue.

Alejandro.Carbonara

@viragomann:

The "push route" command in the custom options is not needed, since this is already set by the entry in "Local Network(s)". So delete it from the custom options field. Maybe this solve your issue.

I made the change you suggested. It removed the add route error, thanks!

Wed Oct 11 17:21:53 2017 WARNING: file 'auth.txt' is group or others accessible
Wed Oct 11 17:21:53 2017 OpenVPN 2.4.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jun 30 2017
Wed Oct 11 17:21:53 2017 library versions: OpenSSL 1.0.2l 25 May 2017, LZO 2.08
Wed Oct 11 17:21:53 2017 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.57.3:1194
Wed Oct 11 17:21:53 2017 UDP link local (bound): [AF_INET][undef]:1194
Wed Oct 11 17:21:53 2017 UDP link remote: [AF_INET]192.168.57.3:1194
Wed Oct 11 17:21:53 2017 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
Wed Oct 11 17:21:53 2017 [server] Peer Connection Initiated with [AF_INET]192.168.57.3:1194
Wed Oct 11 17:22:04 2017 TUN/TAP device tun0 opened
Wed Oct 11 17:22:04 2017 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Wed Oct 11 17:22:04 2017 /sbin/ip link set dev tun0 up mtu 1500
Wed Oct 11 17:22:04 2017 /sbin/ip addr add dev tun0 192.168.80.2/24 broadcast 192.168.80.255
Wed Oct 11 17:22:04 2017 Initialization Sequence Completed

However, I am still failing to reach the internal machine, and the pfsense logs are still showing the similar GET INST BY VIRT [failed] error.

viragomann

Now you're missing the "route add" command on the client.
Please post your server and client config.

Alejandro.Carbonara

Okay, so taking from my config.xml:

_<openvpn><openvpn-server><vpnid>1</vpnid>
<mode>server_tls_user</mode>
<authmode>Local Database</authmode>
<protocol>UDP</protocol>
<dev_mode>tun</dev_mode>
<ipaddr></ipaddr>
<interface>wan</interface>
<local_port>1194</local_port>

<tls>-removed-

<local_network>192.168.58.0/24</local_network>
<local_networkv6></local_networkv6>
<maxclients>10</maxclients>

<passtos></passtos>
<client2client>yes</client2client>
<dynamic_ip>yes</dynamic_ip>
<pool_enable>yes</pool_enable>
<topology>subnet</topology>

<serverbridge_interface>none</serverbridge_interface>
<serverbridge_dhcp_start></serverbridge_dhcp_start>

<netbios_enable></netbios_enable>
<netbios_ntype>0</netbios_ntype>

<verbosity_level>10</verbosity_level></tls></openvpn-server></openvpn>
<dnshaper></dnshaper>
<cert><refid>52db1097e7e40</refid>

<crt>-removed-</crt></cert>
<cert><refid>59de7040dec5d</refid>

<type>server</type>
<caref>59de7040c414a</caref>
<crt>-removed-</crt>
<prv>-removed-</prv></cert>
<cert><refid>59de707095d08</refid>

<type>user</type>
<caref>59de7040c414a</caref>
<crt>-removed-</crt>
<prv>-removed-</prv></cert>

<gateways></gateways>
<ovpnserver><step1><type>local</type></step1>
<step6><certca>Cert-CA</certca>
<keylength>2048</keylength>
<lifetime>3650</lifetime>
<country>US</country>
<state>-removed-</state>
<city>-removed-</city>
<organization></organization>
<email>-removed-</email>
<uselist>on</uselist></step6>
<step9><certname>server</certname>
<keylength>2048</keylength>
<lifetime>3650</lifetime>
<country>US</country>
<state>-removed-</state>
<city>-removed-</city>
<organization></organization>
<email>-removed-</email>
<uselist>on</uselist></step9>
<step10><interface>wan</interface>
<protocol>UDP</protocol>
<localport>1194</localport>

<tlsauth>on</tlsauth>
<gentlskey>on</gentlskey>
<dhkey>2048</dhkey>
<crypto>AES-256-CBC</crypto>
<digest>SHA1</digest>
<engine>none</engine>
<tunnelnet>192.168.80.0/24</tunnelnet>
<localnet>192.168.58.0/24</localnet>
<concurrentcon>10</concurrentcon>
<interclient>on</interclient>
<dynip>on</dynip>
<addrpool>on</addrpool>
<topology>subnet</topology>
<nbttype>0</nbttype>
<advanced>push "route 192.168.58.0 255.255.255.0"</advanced></step10>
<step11><ovpnrule>on</ovpnrule>
<ovpnallow>on</ovpnallow></step11></ovpnserver>
<ca><refid>59de7040c414a</refid>

<crt>-removed-</crt>
<prv>-removed-</prv>
<serial>2</serial></ca>_

Taking from configFile.ovpn:

_dev tun
persist-tun
persist-key
cipher AES-256-CBC
auth SHA1
tls-client
client
resolv-retry infinite
remote 192.168.57.3 1194 udp
verify-x509-name "server" name
auth-user-pass
remote-cert-tls server
<ca>–---BEGIN CERTIFICATE-----
-removed-
-----END CERTIFICATE-----</ca>
<cert>-----BEGIN CERTIFICATE-----
-removed-
-----END CERTIFICATE-----</cert>
<key>-----BEGIN PRIVATE KEY-----
-removed-
-----END PRIVATE KEY-----</key>
<tls-auth>#

2048 bit OpenVPN static key

-----BEGIN OpenVPN Static key V1-----
-removed-
-----END OpenVPN Static key V1-----</tls-auth>
key-direction 1_

viragomann

XML-File :o
A screenshot from the server setting page is much easier to read. The whole config-file can be found in /var/etc/openvpn.

However, I cannot find any mistake in the config. Though the client doesn't set the routes. Maybe you're missing permissions for changing routes. However, this should be mentioned in the client log.

Alejandro.Carbonara

@viragomann:

Though the client doesn't set the routes. Maybe you're missing permissions for changing routes. However, this should be mentioned in the client log.

What do you mean the client doesn't set the routes? Is there a line missing?

When running the client VPN I am running as a user with passwordless sudo, using the ovpn generated by the openvpn exporter.

viragomann

If you have entered a network in the "Local Networks" field in the server setting, a route for this network should be pushed to the client.
After the client had connected you should see an entry for adding routes in the client log like

/sbin/ip route add 192.168.58.0/24 via 192.168.80.1

That line is missing in your log.

You may set the route manually to see if you have the necessary permissions.

Alejandro.Carbonara

I checked and was able to run that command, but I was getting "RTNETLINK answers: File exists" as a response.
To check, I looked at the IP routing tables:

Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default 10.0.2.2 0.0.0.0 UG 0 0 0 eth0
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.57.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.58.0 192.168.80.1 255.255.255.0 UG 0 0 0 tun0
192.168.80.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0

(Note that due to this being on vagrant, the first gateway is used to provision the machine.)

viragomann

So the necessary route is set fine as the routing table shows.
Has it been already set before you were running that command?

Alejandro.Carbonara

The route was set before running the command.

I've started and stopped the openVPN client several times on the machine, so it may have been set earlier.

johnpoz

"192.168.57.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1"

That route is not going down your tunnel.. So how would you expect to get their through the tunnel.. Seems your client is trying to go out eth1 to get there. With no gateway so it thinks that network is on its eth1 interface.

viragomann

I think, that's just the way how his OS prints networks connected directly to an interface. The line has only an U-flag, no G for a gateway.
It the same as
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

viragomann

Now, as the route is set as it should be, why do you think, you have a routing problem?

Try to ping pfSense internal address and see if you get a response.

Alejandro.Carbonara

As of now, I can ping the internal ip of pfsense and get a positive response.

Attempting to ping the internal machine gets me no response whatsoever.

viragomann

Check the firewall of the internal machine.

Alejandro.Carbonara

The target machine is an ubuntu server VM.

sudo ufw status
Status: inactive

viragomann

And its default gateway is set to 192.168.58.2 (the pfSense internal address)?

johnpoz

Oh my bad.. I was reading that you wanted to get to 192.168.57… ooops..

If you can ping the internal IP address of pfsense then your tunnel is up and routing that network down the tunnel. Not being able to get to the machine on the network behind pfsense points to problem on the machine. As viragomann pointed out host firewall and or wrong gateway on the host not pointing back to pfsense are 2 very common problems.

Just because ufw is not running does not mean for example iptables is not running.. I run iptables on my ubuntu vms not ufw..

Alejandro.Carbonara

The routing table on the internal machine looks like:

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.0.2.2 0.0.0.0 UG 0 0 0 eth0
10.0.2.0 * 255.255.255.0 U 0 0 0 eth0
192.168.58.0 * 255.255.255.0 U 0 0 0 eth1

Also, I checked and iptables is all accept rules.