NTPD crashes after starting OpenVPN service
-
When I stop OpenVPN everything is fine. As soon as I start OpenVPN the ntpd service shows as stopped. I rebooted and the same result. If I stop and then start OpenVPN the ntpd service changes to stopped on the service status page.
2.2-ALPHA (i386)
built on Sat Jun 14 01:00:55 CDT 2014
FreeBSD 10.0-STABLE -
ntp logs when I start OpenVPN service (after it is stopped)… Time in reverse order... newest at the top
Jun 15 04:23:42 ntpd[58012]: ntpd exiting on signal 15 (Terminated) Jun 15 04:23:40 ntpd[58012]: new interface(s) found: waking up resolver Jun 15 04:23:40 ntpd[58012]: Listen normally on 12 ovpns1 x.x.x.1:123 Jun 15 04:23:40 ntpd[58012]: failed to init interface for address fe80::260:e0ff:fe04:5cfd%9 Jun 15 04:23:40 ntpd[58012]: unable to create socket on ovpns1 (11) for fe80::260:e0ff:fe04:5cfd%9#123 Jun 15 04:23:40 ntpd[58012]: bind(29) AF_INET6 fe80::260:e0ff:fe04:5cfd%9%9#123 flags 0x13 failed: Can't assign requested address Jun 15 04:23:19 ntpd[58012]: Deleting interface #10 ovpns1, x.x.x.1#123, interface stats: received=0, sent=0, dropped=0, active_time=68 secs Jun 15 04:23:19 ntpd[58012]: Deleting interface #9 ovpns1, fe80::260:e0ff:fe04:5cfd%9#123, interface stats: received=0, sent=0, dropped=0, active_time=68 secs
OpenVPN logs when I stopped and started the OpenVPN service (again in reverse time order)
Jun 15 04:23:41 openvpn[3292]: Initialization Sequence Completed Jun 15 04:23:41 openvpn[3292]: UDPv4 link remote: [undef] Jun 15 04:23:41 openvpn[3292]: UDPv4 link local (bound): [AF_INET]y.y.y.103:1194 Jun 15 04:23:38 openvpn[445]: /usr/local/sbin/ovpn-linkup ovpns1 1500 1557 x.x.x.1 x.x.x.2 init Jun 15 04:23:38 openvpn[445]: /sbin/ifconfig ovpns1 x.x.x.1 x.x.x.2 mtu 1500 netmask 255.255.255.255 up Jun 15 04:23:38 openvpn[445]: do_ifconfig, tt->ipv6=1, tt->did_ifconfig_ipv6_setup=0 Jun 15 04:23:38 openvpn[445]: TUN/TAP device /dev/tun1 opened Jun 15 04:23:38 openvpn[445]: TUN/TAP device ovpns1 exists previously, keep at program end Jun 15 04:23:38 openvpn[445]: Control Channel Authentication: using '/var/etc/openvpn/server1.tls-auth' as a OpenVPN static key file Jun 15 04:23:38 openvpn[445]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts Jun 15 04:23:38 openvpn[445]: library versions: OpenSSL 1.0.1g-freebsd 7 Apr 2014, LZO 2.06 Jun 15 04:23:38 openvpn[445]: OpenVPN 2.3.4 i386-portbld-freebsd10.0 [SSL (OpenSSL)] [LZO] [MH] [IPv6] built on May 29 2014 Jun 15 04:23:19 openvpn[38943]: SIGTERM[hard,] received, process exiting Jun 15 04:23:18 openvpn[38943]: /usr/local/sbin/ovpn-linkdown ovpns1 1500 1557 x.x.x.1 x.x.x.2 init Jun 15 04:23:18 openvpn[38943]: event_wait : Interrupted system call (code=4)
-
In addition. If I manually start ntpd though the services page after it stops it will start up without issues it seems.
-
Added version info to first post.
2.2-ALPHA (i386)
built on Sat Jun 14 01:00:55 CDT 2014
FreeBSD 10.0-STABLE -
Jun 15 04:24:01 pfsense kernel: pid 39066 (ntpd), uid 0: exited on signal 11 (core dumped)
That is what happens when I start OpenVPN.
-
I updated to the 27th snapshot a few days ago and I am still getting the ntpd crash when I start openvpn (I stop openvpn first without issue). This with the 27th snapshot. I uninstalled Suricata so that no packages are installed (except the OpenVPN export package) and I still get the ntpd crash.
Jun 30 14:58:01 kernel: pid 34069 (ntpd), uid 0: exited on signal 11 (core dumped)
2.2-ALPHA (i386)
built on Fri Jun 27 00:02:52 CDT 2014
FreeBSD 10.0-STABLE -
Can you post the contents of /var/etc/ntpd.conf on this system?
-
It looks pretty basic.
# # # pfSense ntp configuration file # tinker panic 0 # Orphan mode stratum tos orphan 12 # GPS Setup server 127.127.20.0 mode 0 minpoll 4 maxpoll 4 prefer fudge 127.127.20.0 flag1 0 flag3 0 # Upstream Servers server 0.pfsense.pool.ntp.org iburst maxpoll 9 disable monitor statsdir /var/log/ntp logconfig =syncall +clockall driftfile /var/db/ntpd.drift restrict default kod limited nomodify nopeer notrap restrict -6 default kod limited nomodify nopeer notrap
-
Can you add the two lines at the end:
interface ignore all
interface listen <interfaces you="" want="" ntpd="" on="">For my case, I listen only on my LAN interface. I believe you can do this in the web gui (services - ntp) by selecting your interface(s) and hitting 'save'. If you don't, it defaults to all interfaces. Note that the interface choices you make don't seem to be carried over and highlighted next time you come back to the services - NTP page.</interfaces> -
Thanks for the info. I tried that a few hours ago by using the gui thinking that maybe it was because openvpn was on the external interface and it didn't make a difference. It still crashes when I start openvpn. I just tried it again too. OpenVPN is configured to use the wan btw. I only have 1 openvpn instance.
# # pfSense ntp configuration file # tinker panic 0 # Orphan mode stratum tos orphan 12 # GPS Setup server 127.127.20.0 mode 0 minpoll 4 maxpoll 4 prefer fudge 127.127.20.0 flag1 0 flag3 0 # Upstream Servers server 0.pfsense.pool.ntp.org iburst maxpoll 9 disable monitor statsdir /var/log/ntp logconfig =syncall +clockall driftfile /var/db/ntpd.drift restrict default kod limited nomodify nopeer notrap restrict -6 default kod limited nomodify nopeer notrap interface ignore all interface listen rl1
I changed it back to listen on all since it didn't make a difference.
-
Still getting this on the current snapshot…
2.2-ALPHA (i386)
built on Wed Jul 09 01:33:54 CDT 2014
FreeBSD 10.0-STABLE -
Still happening for me with 2.2-beta. If I stop openvpn everything looks fine. Refresh services status and it shows openvpn as Stopped. I start openvpn and now openvpn shows as Running but now ntpd shows as Stopped. If I start ntpd service it will start just fine.
2.2-BETA (i386)
built on Thu Sep 18 17:43:56 CDT 2014
FreeBSD 10.1-PRERELEASEFor logs below…
L.L.L.# is the lan subnet
o.o.o.# is the openvpn subnet
e.e.e.# is external subnetSTOP OPENVPN:
status->services shows openvpn stopped and ntpd runningsystem general log:
Sep 19 02:16:48 kernel: ovpns1: link state changed to DOWN Sep 19 02:16:48 check_reload_status: Reloading filter Sep 19 02:17:25 check_reload_status: Syncing firewall
openvpn log:
Sep 19 02:16:48 openvpn[69610]: event_wait : Interrupted system call (code=4) Sep 19 02:16:48 openvpn[69610]: /usr/local/sbin/ovpn-linkdown ovpns1 1500 1557 o.o.o.1 o.o.o.2 init Sep 19 02:16:49 openvpn[69610]: SIGTERM[hard,] received, process exiting
ntpd log:
Sep 19 02:16:50 ntpd[26038]: Deleting interface #9 ovpns1, fe80::260:e0ff:fe04:5cfd%9#123, interface stats: received=0, sent=0, dropped=0, active_time=197 secs Sep 19 02:16:50 ntpd[26038]: Deleting interface #10 ovpns1, o.o.o.1#123, interface stats: received=0, sent=0, dropped=0, active_time=197 secs
START OPENVPN:
status->services shows openvpn started and ntpd stoppedsystem general log:
Sep 19 02:21:32 check_reload_status: Reloading filter Sep 19 02:21:32 kernel: ovpns1: link state changed to UP Sep 19 02:21:33 check_reload_status: rc.newwanip starting ovpns1 Sep 19 02:21:35 php-fpm[258]: /rc.newwanip: rc.newwanip: Informational is starting ovpns1. Sep 19 02:21:35 php-fpm[258]: /rc.newwanip: rc.newwanip: on (IP address: o.o.o.1) (interface: []) (real interface: ovpns1). Sep 19 02:21:35 php-fpm[258]: /rc.newwanip: pfSense package system has detected an ip change -> o.o.o.1 ... Restarting packages. Sep 19 02:21:35 check_reload_status: Reloading filter Sep 19 02:21:35 check_reload_status: Starting packages Sep 19 02:21:37 php-fpm[257]: /rc.start_packages: Restarting/Starting all packages. Sep 19 02:21:53 kernel: pid 82838 (ntpd), uid 0: exited on signal 11 (core dumped)
openvpn log:
Sep 19 02:21:32 openvpn[68759]: OpenVPN 2.3.4 i386-portbld-freebsd10.0 [SSL (OpenSSL)] [LZO] [MH] [IPv6] built on Jul 10 2014 Sep 19 02:21:32 openvpn[68759]: library versions: OpenSSL 1.0.1i-freebsd 6 Aug 2014, LZO 2.06 Sep 19 02:21:32 openvpn[68759]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts Sep 19 02:21:32 openvpn[68759]: Control Channel Authentication: using '/var/etc/openvpn/server1.tls-auth' as a OpenVPN static key file Sep 19 02:21:32 openvpn[68759]: TUN/TAP device ovpns1 exists previously, keep at program end Sep 19 02:21:32 openvpn[68759]: TUN/TAP device /dev/tun1 opened Sep 19 02:21:32 openvpn[68759]: do_ifconfig, tt->ipv6=1, tt->did_ifconfig_ipv6_setup=0 Sep 19 02:21:32 openvpn[68759]: /sbin/ifconfig ovpns1 o.o.o.1 o.o.o.2 mtu 1500 netmask 255.255.255.255 up Sep 19 02:21:32 openvpn[68759]: /usr/local/sbin/ovpn-linkup ovpns1 1500 1557 o.o.o.1 o.o.o.2 init Sep 19 02:21:33 openvpn[70272]: UDPv4 link local (bound): [AF_INET]e.e.e.54:1194 Sep 19 02:21:33 openvpn[70272]: UDPv4 link remote: [undef] Sep 19 02:21:33 openvpn[70272]: Initialization Sequence Completed
ntp log:
Sep 19 02:21:35 ntpd[26038]: ntpd exiting on signal 15 (Terminated) Sep 19 02:21:53 ntpdate[75961]: adjust time server 50.7.64.4 offset 0.001330 sec Sep 19 02:21:53 ntp: Successfully synced time after 1 attempts. Sep 19 02:21:53 ntp: Starting NTP Daemon. Sep 19 02:21:53 ntpd[82655]: ntpd 4.2.7p475@1.2483-o Wed Sep 17 19:27:05 UTC 2014 (1): Starting Sep 19 02:21:53 ntpd[82655]: Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid Sep 19 02:21:53 ntpd[82838]: proto: precision = 1.011 usec (-20) Sep 19 02:21:53 ntpd[82838]: Listen and drop on 0 v6wildcard [::]:123 Sep 19 02:21:53 ntpd[82838]: Listen and drop on 1 v4wildcard 0.0.0.0:123 Sep 19 02:21:53 ntpd[82838]: Listen normally on 2 rl0 [fe80::260:e0ff:fe04:5cfd%1]:123 Sep 19 02:21:53 ntpd[82838]: setsockopt IPV6_MULTICAST_IF 0 for fe80::260:e0ff:fe04:5cfd%1 fails: Can't assign requested address Sep 19 02:21:53 ntpd[82838]: Listen normally on 3 rl0 e.e.e.54:123 Sep 19 02:21:53 ntpd[82838]: Listen normally on 4 rl1 L.L.L.1:123 Sep 19 02:21:53 ntpd[82838]: Listen normally on 5 rl1 [fe80::260:e0ff:fe04:5cfc%2]:123 Sep 19 02:21:53 ntpd[82838]: setsockopt IPV6_MULTICAST_IF 0 for fe80::260:e0ff:fe04:5cfc%2 fails: Can't assign requested address Sep 19 02:21:53 ntpd[82838]: Listen normally on 6 lo0 127.0.0.1:123 Sep 19 02:21:53 ntpd[82838]: Listen normally on 7 lo0 [::1]:123 Sep 19 02:21:53 ntpd[82838]: Listen normally on 8 lo0 [fe80::1%7]:123 Sep 19 02:21:53 ntpd[82838]: setsockopt IPV6_MULTICAST_IF 0 for fe80::1%7 fails: Can't assign requested address Sep 19 02:21:53 ntpd[82838]: Listen normally on 9 ovpns1 [fe80::260:e0ff:fe04:5cfd%9]:123 Sep 19 02:21:53 ntpd[82838]: setsockopt IPV6_MULTICAST_IF 0 for fe80::260:e0ff:fe04:5cfd%9 fails: Can't assign requested address Sep 19 02:21:53 ntpd[82838]: Listen normally on 10 ovpns1 o.o.o.1:123 Sep 19 02:21:53 ntpd[82838]: Listening on routing socket on fd #31 for interface updates Sep 19 02:21:53 ntpd[82838]: refclock_open /dev/gps0: Is a directory Sep 19 02:21:53 ntpd[82838]: 127.127.20.0 local addr 127.0.0.1 ->
-
Sounds remarkably like this long-running problem that was eventually reported solved in 2.1.1:
https://redmine.pfsense.org/issues/3045 -
Seems that it may be related to the dynamic interface updating in ntpd. Can you try adding "-U 0" to the command line to disable the dynamic interface updating?
For testing, you can do it like this from the command line:```
[2.2-BETA][root@pfsense.localdomain]/etc/inc(25): ps axfw | grep ntpd
82886 - Ss 0:00.09 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
83198 0 S+ 0:00.00 grep ntpd
[2.2-BETA][root@pfsense.localdomain]/etc/inc(26): killall ntpd
[2.2-BETA][root@pfsense.localdomain]/etc/inc(27): /usr/local/sbin/ntpd -g -U 0 -c /var/etc/ntpd.conf -p /var/run/ntpd.pidThis may not be enough: The "-U 0" may end interval triggered scanning for interface changes, but ntpd is still listening for changes: > Sep 19 11:24:58 pfsense ntpd[5817]: Command line: /usr/local/sbin/ntpd -g -U 0 -c /var/etc/ntpd.conf -p /var/run/ntpd.pid > Sep 19 11:24:58 pfsense ntpd[6063]: proto: precision = 0.330 usec (-21) > Sep 19 11:24:58 pfsense ntpd[6063]: Listen and drop on 0 v6wildcard [::]:123 > Sep 19 11:24:58 pfsense ntpd[6063]: Listen and drop on 1 v4wildcard 0.0.0.0:123 > Sep 19 11:24:58 pfsense ntpd[6063]: Listen normally on 2 bge0 192.168.2.128:123 > Sep 19 11:24:58 pfsense ntpd[6063]: Listen normally on 3 bge0 [fe80::2e0:66ff:fe6e:260%1]:123 > Sep 19 11:24:58 pfsense ntpd[6063]: setsockopt IPV6_MULTICAST_IF 0 for fe80::2e0:66ff:fe6e:260%1 fails: Can't assign requested address > Sep 19 11:24:58 pfsense ntpd[6063]: Listen normally on 4 lo0 127.0.0.1:123 > Sep 19 11:24:58 pfsense ntpd[6063]: Listen normally on 5 lo0 [::1]:123 > Sep 19 11:24:58 pfsense ntpd[6063]: Listening on routing socket on fd #26 for interface updates <==== Here; is '-U 0' not working? > Sep 19 11:24:58 pfsense ntpd[6063]: refclock_atom: /dev/pps0: Is a directory > Sep 19 11:24:58 pfsense ntpd[6063]: 127.127.22.0 local addr 127.0.0.1 -> <null>Sep 19 11:24:58 pfsense ntpd[6063]: GPS_NMEA(0) serial /dev/gps0 open at 9600 bps > Sep 19 11:24:58 pfsense ntpd[6063]: 0.0.0.0 c01d 0d kern kernel time sync enabled > Sep 19 11:24:58 pfsense ntpd[6063]: 0.0.0.0 c012 02 freq_set kernel -31.400 PPM > Sep 19 11:24:58 pfsense ntpd[6063]: 0.0.0.0 c016 06 restart > Sep 19 11:24:58 pfsense ntpd[6063]: DNS 0.pfsense.pool.ntp.org -> 50.116.55.65 > Sep 19 11:24:58 pfsense ntpd[6063]: DNS 1.fedora.pool.ntp.org -> 66.228.35.252 > Sep 19 11:24:59 pfsense ntpd[6063]: 0.0.0.0 c615 05 clock_sync > Sep 19 11:25:00 pfsense ntpd[6063]: 0.0.0.0 041d 0d kern PPS enabled</null> Looking through ntpd source (ntpd.c), it seems that the variable disable_dynamic_updates is only set if [ (!HAVE_LINUX_CAPABILITIES) and (!HAVE_SOLARIS_PRIVS) and (we are not root user) ]
-
I modified the two scripts below to add the -U 0 option and now it works. Note that ntpd does show down as soon as you start openvpn just like it did before (because ntpdate_sync_once.sh gets run which forces an ntpdate to sync the time and then it starts ntpd again) but if you refresh the service status page ntpd does eventually start up after a few seconds. Before it just stayed as Stopped status no matter how many times I refreshed the services status page. I don't see the crashes by ntpd anymore now either in the system/general log.
/etc/inc/system.inc
/usr/local/sbin/ntpdate_sync_once.shIt looks like it is working properly now adding -U 0 to both the system.inc and ntpdate_sync_once.sh script. Interestingly I don't see the -U 0 in the freebsd man page doing a google search.
[2.2-BETA][root@pfsense]/etc/inc(89): ps ax | grep ntp 80956 - S 0:00.02 /bin/sh /usr/local/sbin/ntpdate_sync_once.sh 88622 - S 0:00.03 /usr/local/sbin/ntpdate -s -t 5 0.pfsense.pool.ntp.org 1226 0 S+ 0:00.01 grep ntp [2.2-BETA][root@pfsense]/etc/inc(90): ps ax | grep ntp 2125 - Ss 0:00.11 /usr/local/sbin/ntpd -U 0 -g -c /var/etc/ntpd.conf -p 5647 0 S+ 0:00.01 grep ntp
-
It looks like it is working properly now adding -U 0 to both the system.inc and ntpdate_sync_once.sh script.
Great, I'm glad it works! Since the forum where this was first discussed is closed (2.1 beta feedback, https://forum.pfsense.org/index.php?topic=62099.msg341908#msg341908), can you post a short note to redmine: https://redmine.pfsense.org/issues/3045 This is a crutch rather than a proper fix, but it should be a big clue to the devs who can do a real fix. That's above my pay grade though :)
Interestingly I don't see the -U 0 in the freebsd man page doing a google search.
Dynamic interface scanning has been in ntpd for a while, since about 2006: http://bugs.ntp.org/show_bug.cgi?id=622