Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Routing Problem in Test Network

    Scheduled Pinned Locked Moved OpenVPN
    22 Posts 3 Posters 2.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      Alejandro.Carbonara
      last edited by

      I've been playing with a small vagrant test network, trying to get a VPN set up.
      The network looks like:

      Client:    192.168.57.2/24 (WAN)
      pfSense:  192.168.57.3/24 (WAN)
                    192.168.58.2/24 (LAN)
      Internal:  192.168.58.3/24 (LAN)

      I've set up pfSense's openvpn using the wizard, mostly sticking to the defaults.

      IPv4 Tunnel Network : 192.168.80.0/24
      IPv4 Local network(s): 192.168.58.0/24
      Custom Options: push "route 192.168.58.0 255.255.255.0"

      Afterwards, I set up an account, exported the configuration file, and connected to pfsense using the client.

      openvpn –config configFile.ovpn --auth-user-pass auth.txt
      Wed Oct 11 16:26:52 2017 WARNING: file 'auth.txt' is group or others accessible
      Wed Oct 11 16:26:52 2017 OpenVPN 2.4.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jun 30 2017
      Wed Oct 11 16:26:52 2017 library versions: OpenSSL 1.0.2l  25 May 2017, LZO 2.08
      Wed Oct 11 16:26:52 2017 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.57.3:1194
      Wed Oct 11 16:26:52 2017 UDP link local (bound): [AF_INET][undef]:1194
      Wed Oct 11 16:26:52 2017 UDP link remote: [AF_INET]192.168.57.3:1194
      Wed Oct 11 16:26:52 2017 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
      Wed Oct 11 16:26:52 2017 [server] Peer Connection Initiated with [AF_INET]192.168.57.3:1194
      Wed Oct 11 16:26:53 2017 TUN/TAP device tun0 opened
      Wed Oct 11 16:26:53 2017 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
      Wed Oct 11 16:26:53 2017 /sbin/ip link set dev tun0 up mtu 1500
      Wed Oct 11 16:26:53 2017 /sbin/ip addr add dev tun0 192.168.80.2/24 broadcast 192.168.80.255
      RTNETLINK answers: File exists
      Wed Oct 11 16:26:53 2017 ERROR: Linux route add command failed: external program exited with error status: 2
      Wed Oct 11 16:26:53 2017 Initialization Sequence Completed

      In another tab, I attempted to connect to the internal machine. What I found was that I could reach the LAN interface of the pfsense machine from the Client machine,  but not the internal machine. Looking into the logs, I found that this happened.

      Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 UDPv4 READ [133] from [AF_INET]192.168.57.2:1194: P_DATA_V1 kid=0 DATA 8f1629dd d9f14fff a811ffc6 22fe80eb dc68cf72 9e0fd397 5d4a6621 c8a7a3e[more…]
      Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 TLS: tls_pre_decrypt, key_id=0, IP=[AF_INET]192.168.57.2:1194
      Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 DECRYPT IV: 9e0fd397 5d4a6621 c8a7a3e5 e7aebda1
      Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 DECRYPT TO: 00000012 45000054 84ae4000 4001aaa4 c0a85002 c0a83a03 080061de 4313000[more…]
      Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 PID_TEST [0] [SSL-0] [1233456789>>>>>EE] 0:17 0:18 t=1507753641[0] r=[-2,64,15,0,1] sl=[47,17,64,528]
      Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 GET INST BY VIRT: 192.168.80.2 -> client1/192.168.57.2:1194 via 192.168.80.2
      Oct 11 20:27:21 openvpn 57629 client1/192.168.57.2:1194 GET INST BY VIRT: 192.168.58.3 [failed]

      It appears that the machine is having problems routing from the pfsense machine to the internal machine.
      However, I checked that both the pfsense machine and the internal machine could ping each other within the network.
      I'm at a loss of how to fix this problem. Can anyone help?

      1 Reply Last reply Reply Quote 0
      • V
        viragomann
        last edited by

        Adding the route fails on the client as the log shows:

        Wed Oct 11 16:26:53 2017 ERROR: Linux route add command failed: external program exited with error status: 2
        

        The "push route" command in the custom options is not needed, since this is already set by the entry in "Local Network(s)". So delete it from the custom options field. Maybe this solve your issue.

        1 Reply Last reply Reply Quote 0
        • A
          Alejandro.Carbonara
          last edited by

          @viragomann:

          The "push route" command in the custom options is not needed, since this is already set by the entry in "Local Network(s)". So delete it from the custom options field. Maybe this solve your issue.

          I made the change you suggested. It removed the add route error, thanks!

          Wed Oct 11 17:21:53 2017 WARNING: file 'auth.txt' is group or others accessible
          Wed Oct 11 17:21:53 2017 OpenVPN 2.4.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jun 30 2017
          Wed Oct 11 17:21:53 2017 library versions: OpenSSL 1.0.2l  25 May 2017, LZO 2.08
          Wed Oct 11 17:21:53 2017 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.57.3:1194
          Wed Oct 11 17:21:53 2017 UDP link local (bound): [AF_INET][undef]:1194
          Wed Oct 11 17:21:53 2017 UDP link remote: [AF_INET]192.168.57.3:1194
          Wed Oct 11 17:21:53 2017 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
          Wed Oct 11 17:21:53 2017 [server] Peer Connection Initiated with [AF_INET]192.168.57.3:1194
          Wed Oct 11 17:22:04 2017 TUN/TAP device tun0 opened
          Wed Oct 11 17:22:04 2017 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
          Wed Oct 11 17:22:04 2017 /sbin/ip link set dev tun0 up mtu 1500
          Wed Oct 11 17:22:04 2017 /sbin/ip addr add dev tun0 192.168.80.2/24 broadcast 192.168.80.255
          Wed Oct 11 17:22:04 2017 Initialization Sequence Completed

          However, I am still failing to reach the internal machine, and the pfsense logs are still showing the similar GET INST BY VIRT [failed] error.

          1 Reply Last reply Reply Quote 0
          • V
            viragomann
            last edited by

            Now you're missing the "route add" command on the client.
            Please post your server and client config.

            1 Reply Last reply Reply Quote 0
            • A
              Alejandro.Carbonara
              last edited by

              Okay, so taking from my config.xml:

              _<openvpn><openvpn-server><vpnid>1</vpnid>
              <mode>server_tls_user</mode>
              <authmode>Local Database</authmode>
              <protocol>UDP</protocol>
              <dev_mode>tun</dev_mode>
              <ipaddr></ipaddr>
              <interface>wan</interface>
              <local_port>1194</local_port>

              <tls>-removed-

              <certref>59de7040dec5d</certref>
              <dh_length>2048</dh_length>
              <cert_depth>1</cert_depth>

              <crypto>AES-256-CBC</crypto>
              <digest>SHA1</digest>
              <engine>none</engine>
              <tunnel_network>192.168.80.0/24</tunnel_network>

              <local_network>192.168.58.0/24</local_network>
              <local_networkv6></local_networkv6>
              <maxclients>10</maxclients>

              <passtos></passtos>
              <client2client>yes</client2client>
              <dynamic_ip>yes</dynamic_ip>
              <pool_enable>yes</pool_enable>
              <topology>subnet</topology>

              <serverbridge_interface>none</serverbridge_interface>
              <serverbridge_dhcp_start></serverbridge_dhcp_start>

              <netbios_enable></netbios_enable>
              <netbios_ntype>0</netbios_ntype>

              <verbosity_level>10</verbosity_level></tls></openvpn-server></openvpn>
              <dnshaper></dnshaper>
              <cert><refid>52db1097e7e40</refid>

              <crt>-removed-</crt></cert>
              <cert><refid>59de7040dec5d</refid>

              <type>server</type>
              <caref>59de7040c414a</caref>
              <crt>-removed-</crt>
              <prv>-removed-</prv></cert>
              <cert><refid>59de707095d08</refid>

              <type>user</type>
              <caref>59de7040c414a</caref>
              <crt>-removed-</crt>
              <prv>-removed-</prv></cert>

              <gateways></gateways>
              <ovpnserver><step1><type>local</type></step1>
              <step6><certca>Cert-CA</certca>
              <keylength>2048</keylength>
              <lifetime>3650</lifetime>
              <country>US</country>
              <state>-removed-</state>
              <city>-removed-</city>
              <organization></organization>
              <email>-removed-</email>
              <uselist>on</uselist></step6>
              <step9><certname>server</certname>
              <keylength>2048</keylength>
              <lifetime>3650</lifetime>
              <country>US</country>
              <state>-removed-</state>
              <city>-removed-</city>
              <organization></organization>
              <email>-removed-</email>
              <uselist>on</uselist></step9>
              <step10><interface>wan</interface>
              <protocol>UDP</protocol>
              <localport>1194</localport>

              <tlsauth>on</tlsauth>
              <gentlskey>on</gentlskey>
              <dhkey>2048</dhkey>
              <crypto>AES-256-CBC</crypto>
              <digest>SHA1</digest>
              <engine>none</engine>
              <tunnelnet>192.168.80.0/24</tunnelnet>
              <localnet>192.168.58.0/24</localnet>
              <concurrentcon>10</concurrentcon>
              <interclient>on</interclient>
              <dynip>on</dynip>
              <addrpool>on</addrpool>
              <topology>subnet</topology>
              <nbttype>0</nbttype>
              <advanced>push "route 192.168.58.0 255.255.255.0"</advanced></step10>
              <step11><ovpnrule>on</ovpnrule>
              <ovpnallow>on</ovpnallow></step11></ovpnserver>
              <ca><refid>59de7040c414a</refid>

              <crt>-removed-</crt>
              <prv>-removed-</prv>
              <serial>2</serial></ca>_

              Taking from configFile.ovpn:

              _dev tun
              persist-tun
              persist-key
              cipher AES-256-CBC
              auth SHA1
              tls-client
              client
              resolv-retry infinite
              remote 192.168.57.3 1194 udp
              verify-x509-name "server" name
              auth-user-pass
              remote-cert-tls server
              <ca>–---BEGIN CERTIFICATE-----
              -removed-
              -----END CERTIFICATE-----</ca>
              <cert>-----BEGIN CERTIFICATE-----
              -removed-
              -----END CERTIFICATE-----</cert>
              <key>-----BEGIN PRIVATE KEY-----
              -removed-
              -----END PRIVATE KEY-----</key>
              <tls-auth>#

              2048 bit OpenVPN static key

              -----BEGIN OpenVPN Static key V1-----
              -removed-
              -----END OpenVPN Static key V1-----</tls-auth>
              key-direction 1_

              1 Reply Last reply Reply Quote 0
              • V
                viragomann
                last edited by

                XML-File  :o
                A screenshot from the server setting page is much easier to read. The whole config-file can be found in /var/etc/openvpn.

                However, I cannot find any mistake in the config. Though the client doesn't set the routes. Maybe you're missing permissions for changing routes. However, this should be mentioned in the client log.

                1 Reply Last reply Reply Quote 0
                • A
                  Alejandro.Carbonara
                  last edited by

                  @viragomann:

                  Though the client doesn't set the routes. Maybe you're missing permissions for changing routes. However, this should be mentioned in the client log.

                  What do you mean the client doesn't set the routes? Is there a line missing?

                  When running the client VPN I am running as a user with passwordless sudo, using the ovpn generated by the openvpn exporter.

                  1 Reply Last reply Reply Quote 0
                  • V
                    viragomann
                    last edited by

                    If you have entered a network in the "Local Networks" field in the server setting, a route for this network should be pushed to the client.
                    After the client had connected you should see an entry for adding routes in the client log like

                    /sbin/ip route add 192.168.58.0/24 via 192.168.80.1
                    

                    That line is missing in your log.

                    You may set the route manually to see if you have the necessary permissions.

                    1 Reply Last reply Reply Quote 0
                    • A
                      Alejandro.Carbonara
                      last edited by

                      I checked and was able to run that command, but I was getting "RTNETLINK answers: File exists" as a response.
                      To check, I looked at the IP routing tables:

                      Kernel IP routing table
                      Destination      Gateway          Genmask        Flags    MSS Window  irtt Iface
                      default            10.0.2.2            0.0.0.0            UG      0    0          0 eth0
                      10.0.2.0          0.0.0.0            255.255.255.0  U        0    0          0 eth0
                      192.168.57.0    0.0.0.0            255.255.255.0  U        0    0          0 eth1
                      192.168.58.0    192.168.80.1    255.255.255.0  UG      0    0          0 tun0
                      192.168.80.0    0.0.0.0            255.255.255.0  U        0    0          0 tun0

                      (Note that due to this being on vagrant, the first gateway is used to provision the machine.)

                      1 Reply Last reply Reply Quote 0
                      • V
                        viragomann
                        last edited by

                        So the necessary route is set fine as the routing table shows.
                        Has it been already set before you were running that command?

                        1 Reply Last reply Reply Quote 0
                        • A
                          Alejandro.Carbonara
                          last edited by

                          The route was set before running the command.

                          I've started and stopped the openVPN client several times on the machine, so it may have been set earlier.

                          1 Reply Last reply Reply Quote 0
                          • johnpozJ
                            johnpoz LAYER 8 Global Moderator
                            last edited by

                            "192.168.57.0    0.0.0.0            255.255.255.0  U        0    0          0 eth1"

                            That route is not going down your tunnel.. So how would you expect to get their through the tunnel.. Seems your client is trying to go out eth1 to get there.  With no gateway so it thinks that network is on its eth1 interface.

                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                            If you get confused: Listen to the Music Play
                            Please don't Chat/PM me for help, unless mod related
                            SG-4860 24.11 | Lab VMs 2.8, 24.11

                            1 Reply Last reply Reply Quote 0
                            • V
                              viragomann
                              last edited by

                              I think, that's just the way how his OS prints networks connected directly to an interface. The line has only an U-flag, no G for a gateway.
                              It the same as
                              10.0.2.0          0.0.0.0            255.255.255.0  U        0    0          0 eth0

                              1 Reply Last reply Reply Quote 0
                              • V
                                viragomann
                                last edited by

                                Now, as the route is set as it should be, why do you think, you have a routing problem?

                                Try to ping pfSense internal address and see if you get a response.

                                1 Reply Last reply Reply Quote 0
                                • A
                                  Alejandro.Carbonara
                                  last edited by

                                  As of now, I can ping the internal ip of pfsense and get a positive response.

                                  Attempting to ping the internal machine gets me no response whatsoever.

                                  1 Reply Last reply Reply Quote 0
                                  • V
                                    viragomann
                                    last edited by

                                    Check the firewall of the internal machine.

                                    1 Reply Last reply Reply Quote 0
                                    • A
                                      Alejandro.Carbonara
                                      last edited by

                                      The target machine is an ubuntu server VM.

                                      sudo ufw status
                                      Status: inactive

                                      1 Reply Last reply Reply Quote 0
                                      • V
                                        viragomann
                                        last edited by

                                        And its default gateway is set to 192.168.58.2 (the pfSense internal address)?

                                        1 Reply Last reply Reply Quote 0
                                        • johnpozJ
                                          johnpoz LAYER 8 Global Moderator
                                          last edited by

                                          Oh my bad.. I was reading that you wanted to get to 192.168.57… ooops..

                                          If you can ping the internal IP address of pfsense then your tunnel is up and routing that network down the tunnel.  Not being able to get to the machine on the network behind pfsense points to problem on the machine.  As viragomann pointed out host firewall and or wrong gateway on the host not pointing back to pfsense are 2 very common problems.

                                          Just because ufw is not running does not mean for example iptables is not running.. I run iptables on my ubuntu vms not ufw..

                                          An intelligent man is sometimes forced to be drunk to spend time with his fools
                                          If you get confused: Listen to the Music Play
                                          Please don't Chat/PM me for help, unless mod related
                                          SG-4860 24.11 | Lab VMs 2.8, 24.11

                                          1 Reply Last reply Reply Quote 0
                                          • A
                                            Alejandro.Carbonara
                                            last edited by

                                            The routing table on the internal machine looks like:

                                            Kernel IP routing table
                                            Destination    Gateway        Genmask        Flags Metric Ref    Use Iface
                                            default        10.0.2.2        0.0.0.0        UG    0      0        0 eth0
                                            10.0.2.0        *              255.255.255.0  U    0      0        0 eth0
                                            192.168.58.0    *              255.255.255.0  U    0      0        0 eth1

                                            Also, I checked and iptables is all accept rules.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.