Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Strange logs -> losing OpenVPN connection every 20 - 120 seconds

    Scheduled Pinned Locked Moved
    2.1.1 Snapshot Feedback and Problems - RETIRED
    6
    18
    13.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      phil.davis
      last edited by

      The OpenVPN clients are restarted if a WAN seems to go down/up even if the new WAN IP turns out to be the same as the old one. I believe this is because there were (are?) issues with OpenVPN continuing on after such a WAN transition. Restarting them makes sure they re-establish happily.

      The DNS translations of Aliases that contain FQDNs are re-checked every 5 minutes. But that happens on my (and many people's) systems without triggering a WAN transition and all the resultant processing.

      What is in cron? (Install the Cron package to look at that easily) Any jobs that specify "3-59/5" that would make it start at 00:03 and every 5 minutes thereafter?

      Or is there a 5-minute egg timer somewhere on a connected device or cable that glitches it every 5 minutes  ;)

      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

      1 Reply Last reply Reply Quote 0
      • D
        digdug3
        last edited by

        I have the same problem, the problem is less since the 2.1.1-PRERELEASE (i386) built on Fri Feb 7 11:46:37 EST 2014
        I had to change the "DOWN" time in gateway monitoring to 60 seconds.

        BUT:

        OpenVPN does restart all services, even if there was no change to the WAN-ip:

        Feb 10 12:46:24 	php: rc.start_packages: Restarting/Starting all packages.
Feb 10 12:46:22 	php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(opt1).
Feb 10 12:46:22 	php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan).
Feb 10 12:46:22 	check_reload_status: Starting packages
Feb 10 12:46:22 	php: rc.newwanip: pfSense package system has detected an ip change -> 192.168.8.1 ... Restarting packages.
Feb 10 12:46:22 	php: rc.newwanip: rc.newwanip: on (IP address: 192.168.8.1) (interface: ) (real interface: ovpns1).
Feb 10 12:46:22 	php: rc.newwanip: rc.newwanip: Informational is starting ovpns1.
Feb 10 12:46:19 	check_reload_status: rc.newwanip starting ovpns1
Feb 10 12:46:19 	php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(opt1).
Feb 10 12:46:19 	kernel: ovpns1: link state changed to UP
Feb 10 12:46:19 	php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan).
Feb 10 12:46:19 	check_reload_status: Reloading filter
Feb 10 12:46:19 	kernel: ovpns1: link state changed to DOWN
Feb 10 12:46:19 	php: rc.openvpn: OpenVPN: Resync server1 OpenVPN WAN
Feb 10 12:46:19 	php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use GW_WAN.
Feb 10 12:46:16 	check_reload_status: Reloading filter
Feb 10 12:46:16 	check_reload_status: Restarting OpenVPN tunnels/interfaces
Feb 10 12:46:16 	check_reload_status: Restarting ipsec tunnels
Feb 10 12:46:16 	check_reload_status: updating dyndns GW_WAN
Feb 10 12:45:02 	php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(opt1).
Feb 10 12:45:01 	php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan).
Feb 10 12:30:39 	php: rc.update_urltables: /etc/rc.update_urltables: pfBlockerBadIPList does not need updated.
Feb 10 12:30:39 	php: rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates
Feb 10 12:30:02 	php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(opt1).
Feb 10 12:30:01 	php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(wan).
Feb 10 12:30:01 	php: rc.update_urltables: /etc/rc.update_urltables: Sleeping for 37 seconds.
Feb 10 12:30:01 	php: rc.update_urltables: /etc/rc.update_urltables: Starting up.

        Also I keep getting```
        php: rc.filter_configure_sync: Could not find IPv6 gateway for interface(opt1).

        1 Reply Last reply Reply Quote 0
        • O
          Oliver
          last edited by

          @phil.davis:

          The OpenVPN clients are restarted if a WAN seems to go down/up even if the new WAN IP turns out to be the same as the old one. I believe this is because there were (are?) issues with OpenVPN continuing on after such a WAN transition. Restarting them makes sure they re-establish happily.

          Wouldn't be such an event logged under "Gateways"? If so, that's not the case.

          @phil.davis:

          The DNS translations of Aliases that contain FQDNs are re-checked every 5 minutes. But that happens on my (and many people's) systems without triggering a WAN transition and all the resultant processing.

          Just for the lols, how would one disable this?

          @phil.davis:

          What is in cron? (Install the Cron package to look at that easily) Any jobs that specify "3-59/5" that would make it start at 00:03 and every 5 minutes thereafter?

          Nope.```
          minute hour mday month wday who command
          1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a
          1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh
          */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
          1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update
          */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
          30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables

          
@phil.davis:

> Or is there a 5-minute egg timer somewhere on a connected device or cable that glitches it every 5 minutes  ;)

I hope not, but it seems so…  :'(

Edit: I just tested this scenario using Mikrotik RouterOS. My OpenVPN tunnel is up and running fine for the last two hours or so. No hiccups at all. I really don't want to switch back to RouterOS tho. I'm already starting to miss pfSense's amazing WebUI. So, please continue help me to get this fixed. :-[
          1 Reply Last reply Reply Quote 0
          • E
            eri--
            last edited by

            I think you have assigned your ovpn interface.
            That activates the monitoring and probably causing a lopping of sorts there.

            Can you check if you can ping the monitoring ip by default or disable it entirely?
            One of those i think will fix your issues.

            1 Reply Last reply Reply Quote 0
            • O
              Oliver
              last edited by

              @ermal:

              Can you check if you can ping the monitoring ip by default or disable it entirely?
              One of those i think will fix your issues.

              @Oliver:

              • I turned off gateway monitoring altogether. This didn't solve the problem either.

              No, it did not. :-\ And I disabled monitoring on all the gateways. They always appear as online, which is why I don't understand most of the logs.

              Btw. I had to assign it, because I wanted to transparently route all traffic of one specific workstation.

              1 Reply Last reply Reply Quote 0
              • E
                eri--
                last edited by

                So find a machine to ping on the other side then :)

                Though disabling monitoring should have avoided this unless you have other problems in the tunnel.

                1 Reply Last reply Reply Quote 0
                • O
                  Oliver
                  last edited by

                  I reverted back to

                  2.1-RELEASE (amd64)
                  built on Wed Sep 11 18:17:48 EDT 2013

                  and this oddity is gone for good. Guess I'm going to stick to that for a some more time!

                  1 Reply Last reply Reply Quote 0
                  • C
                    cmb
                    last edited by

                    The system log isn't all that telling on its own, what does the OpenVPN log show?

                    1 Reply Last reply Reply Quote 0
                    • D
                      doktornotor Banned
                      last edited by

                      Still happening even with latest snapshot. The WAN IP is in fact static, it never changes. These newwanip detections are a pile of BS.

                      
rc.newwanip: pfSense package system has detected an ip change

                      Same thing happens with IPsec:

                      
php: rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.

                      As for OpenVPN log, nothing useful there either:

                      
Mar 6 12:25:37	openvpn[47673]: /sbin/ifconfig ovpns1 10.0.8.1 10.0.8.1 mtu 1500 netmask 255.255.255.0 up
Mar 6 12:25:37	openvpn[47673]: do_ifconfig, tt->ipv6=1, tt->did_ifconfig_ipv6_setup=1
Mar 6 12:25:37	openvpn[47673]: TUN/TAP device /dev/tun1 opened
Mar 6 12:25:37	openvpn[47673]: TUN/TAP device ovpns1 exists previously, keep at program end
Mar 6 12:25:37	openvpn[47673]: Control Channel Authentication: using '/var/etc/openvpn/server1.tls-auth' as a OpenVPN static key file
Mar 6 12:25:36	openvpn[47673]: Initializing OpenSSL support for engine 'cryptodev'
Mar 6 12:25:36	openvpn[47673]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Mar 6 12:25:36	openvpn[47673]: OpenVPN 2.3.2 i386-portbld-freebsd8.3 [SSL (OpenSSL)] [LZO] [eurephia] [MH] [IPv6] built on Jan 23 2014
Mar 6 12:25:36	openvpn[97470]: SIGTERM[hard,] received, process exiting
Mar 6 12:25:36	openvpn[97470]: /usr/local/sbin/ovpn-linkdown ovpns1 1500 1558 10.0.8.1 255.255.255.0 init
Mar 6 12:25:36	openvpn[97470]: event_wait : Interrupted system call (code=4)
                      1 Reply Last reply Reply Quote 0
                      • E
                        eri--
                        last edited by

                        Every time a connection/reconnection occurs you see the logs for newwanip.

                        Do you have anything similar to below in your logs?

                        
log_error("DEVD Ethernet attached event for {$iface}");
log_error("HOTPLUG: Configuring interface {$iface}");

                        Give this a try again as well:

                        
diff --git a/etc/rc.linkup b/etc/rc.linkup
index 1994336..43607b1 100755
--- a/etc/rc.linkup
+++ b/etc/rc.linkup
@@ -100,7 +100,7 @@ if (!file_exists("{$g['varrun_path']}/booting") && empty($g['booting'])) {
                break;
        }
        $interface = convert_real_interface_to_friendly_interface_name($argv[2]);
-       if (!empty($interface))
+       if (!empty($interface) && substr($argv[2], 0, 4) != "ovpn")
                handle_argument_group($interface, $action);
 }
                        1 Reply Last reply Reply Quote 0
                        • D
                          doktornotor Banned
                          last edited by

                          No, no hotplug/devd events in the log. As for the patch, well yes I can try that (should obviously stop triggering the OVPN restart), but it's not really like it'd happen for any good reason at all. It seems like it just decides the WAN IP "changed" a couple of times a day, randomly.

                          1 Reply Last reply Reply Quote 0
                          • E
                            eri--
                            last edited by

                            Well if your Dyndns or any other monitoring for hostname changes triggers the event you have to find which is doing that.
                            Also find out why its triggering the event.

                            To me from the logs it looked like devd was doing that since it was not clear on dyndns/hostnames on vpns being used from the logs.

                            Maybe the config.xml can help to validate the options here.

                            1 Reply Last reply Reply Quote 0
                            • D
                              doktornotor Banned
                              last edited by

                              @ermal:

                              Well if your Dyndns or any other monitoring for hostname changes triggers the event you have to find which is doing that.
                              Also find out why its triggering the event.

                              To me from the logs it looked like devd was doing that since it was not clear on dyndns/hostnames on vpns being used from the logs.

                              Now that you mention dyndns…

                              
Mar 7 08:22:53	check_reload_status: Restarting OpenVPN tunnels/interfaces
Mar 7 08:22:53	check_reload_status: updating dyndns HEIPV6_TUNNELV6
Mar 7 08:22:53	check_reload_status: Restarting OpenVPN tunnels/interfaces
Mar 7 08:22:53	check_reload_status: updating dyndns WAN_DHCP
Mar 7 08:22:52	check_reload_status: Reloading filter
Mar 7 08:22:52	check_reload_status: Restarting OpenVPN tunnels/interfaces
Mar 7 08:22:52	check_reload_status: Restarting ipsec tunnels
Mar 7 08:22:52	check_reload_status: updating dyndns WAN_DHCP
Mar 7 08:22:52	check_reload_status: Reloading filter
Mar 7 08:22:52	check_reload_status: Restarting OpenVPN tunnels/interfaces
Mar 7 08:22:52	check_reload_status: Restarting ipsec tunnels
Mar 7 08:22:52	check_reload_status: updating dyndns HEIPV6_TUNNELV6

                              Obviously, this again makes no sense since there is nothing dynamic - except for the gateway being statically "dynamic". The GW IP never changes, it is configured as static IP on the GIF interface. What's "updating" there? Again, the WAN_DHCP is a static DHCP lease, never changes.

                              1 Reply Last reply Reply Quote 0
                              • E
                                eri--
                                last edited by

                                Can you check the relevant files for this to see why it triggers?

                                1 Reply Last reply Reply Quote 0
                                • D
                                  doktornotor Banned
                                  last edited by

                                  @ermal:

                                  Can you check the relevant files for this to see why it triggers?

                                  Maybe… if only I knew the relevant files. :D

                                  1 Reply Last reply Reply Quote 0
                                  • E
                                    eri--
                                    last edited by

                                    /conf/dyndns*.cache

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post