pfSense NTP server is very unstable.
-
As per
Diagnostics -> Backup & Restore -> Config History, the last time I updated the NTP settings was1/23/22 05:51:12(my timezone is UTC+8).The last time I updated the NTP setting, I changed the server to IP address
203.107.6.88(IP ofntp.aliyun.com), because some Google results indicate that using IP address than domain name may be more stable for pfSense):
Current NTP status:
It means pfSense can't get time from the external server, but as my test in a PC, this external server just works fine. And as I remembered, I checked the pfSense NTP status in WebUI just after updating settings last time, and it showed the status was good.
The logs from the last time I updated NTP settings (
Status -> System Logs -> NTP):Jan 23 05:51:12 ntpd 3974 ntpd exiting on signal 15 (Terminated) Jan 23 05:51:12 ntpd 3974 0.0.0.0 8812 82 demobilize assoc 27640 Jan 23 05:51:12 ntpd 3974 17.253.84.253 1612 82 demobilize assoc 27641 Jan 23 05:51:12 ntpd 3974 17.253.84.253 local addr 60.25.138.110 -> <null> Jan 23 05:51:12 ntpd 3974 17.253.116.125 1012 82 demobilize assoc 27642 Jan 23 05:51:12 ntpd 3974 17.253.116.125 local addr 60.25.138.110 -> <null> Jan 23 05:51:12 ntpd 3974 17.253.114.125 1012 82 demobilize assoc 27643 Jan 23 05:51:12 ntpd 3974 17.253.114.125 local addr 60.25.138.110 -> <null> Jan 23 05:51:12 ntpd 3974 17.253.114.253 1012 82 demobilize assoc 27644 Jan 23 05:51:12 ntpd 3974 17.253.114.253 local addr 60.25.138.110 -> <null> Jan 23 05:51:12 ntpd 3974 17.253.116.253 1012 82 demobilize assoc 27645 Jan 23 05:51:12 ntpd 3974 17.253.116.253 local addr 60.25.138.110 -> <null> Jan 23 05:51:12 ntpd 3974 0.0.0.0 061d 0d kern kernel time sync disabled Jan 23 05:51:12 ntpd 8270 ntpd 4.2.8p15@1.3728-o Thu Jun 24 21:53:38 UTC 2021 (1): Starting Jan 23 05:51:12 ntpd 8270 Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid Jan 23 05:51:12 ntpd 8270 ---------------------------------------------------- Jan 23 05:51:12 ntpd 8270 ntp-4 is maintained by Network Time Foundation, Jan 23 05:51:12 ntpd 8270 Inc. (NTF), a non-profit 501(c)(3) public-benefit Jan 23 05:51:12 ntpd 8270 corporation. Support and training for ntp-4 are Jan 23 05:51:12 ntpd 8270 available at https://www.nwtime.org/support Jan 23 05:51:12 ntpd 8270 ---------------------------------------------------- Jan 23 05:51:12 ntpd 8520 proto: precision = 7.590 usec (-17) Jan 23 05:51:12 ntpd 8520 basedate set to 2021-06-12 Jan 23 05:51:12 ntpd 8520 gps base set to 2021-06-13 (week 2162) Jan 23 05:51:12 ntpd 8520 Listen and drop on 0 v6wildcard [::]:123 Jan 23 05:51:12 ntpd 8520 Listen and drop on 1 v4wildcard 0.0.0.0:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 2 vtnet0 [fe80::5054:ff:fe00:1%1]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 3 vtnet1 [fe80::5054:ff:fe00:2%2]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 4 em0 [fe80::5054:ff:fe00:99%3]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 5 igb0 [fe80::2e53:4aff:fe07:7ee0%4]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 6 igb1 [fe80::2e53:4aff:fe07:7ee1%5]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 7 igb2 [fe80::2e53:4aff:fe07:7ee2%6]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 8 igb3 [fe80::2e53:4aff:fe07:7ee3%7]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 9 lo0 [::1]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 10 lo0 [fe80::1%9]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 11 lo0 127.0.0.1:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 12 pppoe0 60.25.138.110:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 13 pppoe0 [fe80::2e53:4aff:fe07:7ee0%12]:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 14 bridge0 10.1.1.2:123 Jan 23 05:51:12 ntpd 8520 Listen normally on 15 bridge1 10.1.2.2:123 Jan 23 05:51:12 ntpd 8520 Listening on routing socket on fd #36 for interface updates Jan 23 05:51:12 ntpd 8520 203.107.6.88 8011 81 mobilize assoc 9119 Jan 23 05:51:12 ntpd 8520 kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized Jan 23 05:51:12 ntpd 8520 0.0.0.0 c01d 0d kern kernel time sync enabled Jan 23 05:51:12 ntpd 8520 kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized Jan 23 05:51:12 ntpd 8520 0.0.0.0 c012 02 freq_set kernel -8.436 PPM Jan 23 05:51:12 ntpd 8520 0.0.0.0 c016 06 restart Jan 23 05:51:13 ntpd 8520 203.107.6.88 8014 84 reachable Jan 23 05:51:19 ntpd 8520 203.107.6.88 901a 8a sys_peer Jan 23 05:51:19 ntpd 8520 0.0.0.0 c615 05 clock_sync Jan 23 05:51:23 ntpd 8520 0.0.0.0 0618 08 no_sys_peer Jan 23 14:13:33 ntpd 32039 ntpd 4.2.8p15@1.3728-o Thu Jun 24 21:53:38 UTC 2021 (1): Starting Jan 23 14:13:33 ntpd 32039 Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid Jan 23 14:13:33 ntpd 32039 ---------------------------------------------------- Jan 23 14:13:33 ntpd 32039 ntp-4 is maintained by Network Time Foundation, Jan 23 14:13:33 ntpd 32039 Inc. (NTF), a non-profit 501(c)(3) public-benefit Jan 23 14:13:33 ntpd 32039 corporation. Support and training for ntp-4 are Jan 23 14:13:33 ntpd 32039 available at https://www.nwtime.org/support Jan 23 14:13:33 ntpd 32039 ---------------------------------------------------- Jan 23 14:13:33 ntpd 32129 proto: precision = 17.640 usec (-16) Jan 23 14:13:33 ntpd 32129 basedate set to 2021-06-12 Jan 23 14:13:33 ntpd 32129 gps base set to 2021-06-13 (week 2162) Jan 23 14:13:33 ntpd 32129 Listen and drop on 0 v6wildcard [::]:123 Jan 23 14:13:33 ntpd 32129 Listen and drop on 1 v4wildcard 0.0.0.0:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 2 vtnet0 [fe80::5054:ff:fe00:1%1]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 3 vtnet1 [fe80::5054:ff:fe00:2%2]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 4 em0 [fe80::5054:ff:fe00:99%3]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 5 igb0 [fe80::2e53:4aff:fe07:7ee0%4]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 6 igb1 [fe80::2e53:4aff:fe07:7ee1%5]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 7 igb2 [fe80::2e53:4aff:fe07:7ee2%6]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 8 igb3 [fe80::2e53:4aff:fe07:7ee3%7]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 9 lo0 [::1]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 10 lo0 [fe80::1%9]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 11 lo0 127.0.0.1:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 12 pppoe0 116.130.78.65:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 13 pppoe0 [fe80::2e53:4aff:fe07:7ee0%12]:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 14 bridge0 10.1.1.2:123 Jan 23 14:13:33 ntpd 32129 Listen normally on 15 bridge1 10.1.2.2:123 Jan 23 14:13:33 ntpd 32129 Listening on routing socket on fd #36 for interface updates Jan 23 14:13:33 ntpd 32129 203.107.6.88 8011 81 mobilize assoc 41585 Jan 23 14:13:33 ntpd 32129 kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Jan 23 14:13:33 ntpd 32129 0.0.0.0 c01d 0d kern kernel time sync enabled Jan 23 14:13:33 ntpd 32129 kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Jan 23 14:13:33 ntpd 32129 0.0.0.0 c012 02 freq_set kernel -8.436 PPM Jan 23 14:13:33 ntpd 32129 0.0.0.0 c016 06 restart Jan 23 14:13:35 ntpd 32129 203.107.6.88 8014 84 reachable Jan 23 14:13:41 ntpd 32129 203.107.6.88 901a 8a sys_peer Jan 23 14:13:41 ntpd 32129 0.0.0.0 c61c 0c clock_step +14.575908 s Jan 23 14:13:55 ntpd 32129 0.0.0.0 c615 05 clock_sync Jan 23 14:15:03 ntpd 32129 0.0.0.0 c618 08 no_sys_peer Jan 23 14:15:03 ntpd 32129 203.107.6.88 8014 84 reachable Jan 23 14:15:09 ntpd 32129 203.107.6.88 901a 8a sys_peer Jan 23 14:15:13 ntpd 32129 0.0.0.0 0628 08 no_sys_peer Jan 24 02:39:41 ntpd 33147 ntpd 4.2.8p15@1.3728-o Thu Jun 24 21:53:38 UTC 2021 (1): Starting Jan 24 02:39:41 ntpd 33147 Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid Jan 24 02:39:41 ntpd 33147 ---------------------------------------------------- Jan 24 02:39:41 ntpd 33147 ntp-4 is maintained by Network Time Foundation, Jan 24 02:39:41 ntpd 33147 Inc. (NTF), a non-profit 501(c)(3) public-benefit Jan 24 02:39:41 ntpd 33147 corporation. Support and training for ntp-4 are Jan 24 02:39:41 ntpd 33147 available at https://www.nwtime.org/support Jan 24 02:39:41 ntpd 33147 ---------------------------------------------------- Jan 24 02:39:41 ntpd 33227 proto: precision = 4.560 usec (-18) Jan 24 02:39:41 ntpd 33227 basedate set to 2021-06-12 Jan 24 02:39:41 ntpd 33227 gps base set to 2021-06-13 (week 2162) Jan 24 02:39:41 ntpd 33227 Listen and drop on 0 v6wildcard [::]:123 Jan 24 02:39:41 ntpd 33227 Listen and drop on 1 v4wildcard 0.0.0.0:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 2 vtnet0 [fe80::5054:ff:fe00:1%1]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 3 vtnet1 [fe80::5054:ff:fe00:2%2]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 4 em0 [fe80::5054:ff:fe00:99%3]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 5 igb0 [fe80::2e53:4aff:fe07:7ee0%4]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 6 igb1 [fe80::2e53:4aff:fe07:7ee1%5]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 7 igb2 [fe80::2e53:4aff:fe07:7ee2%6]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 8 igb3 [fe80::2e53:4aff:fe07:7ee3%7]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 9 lo0 [::1]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 10 lo0 [fe80::1%9]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 11 lo0 127.0.0.1:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 12 pppoe0 117.11.135.78:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 13 pppoe0 [fe80::2e53:4aff:fe07:7ee0%12]:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 14 bridge0 10.1.1.2:123 Jan 24 02:39:41 ntpd 33227 Listen normally on 15 bridge1 10.1.2.2:123 Jan 24 02:39:41 ntpd 33227 Listening on routing socket on fd #36 for interface updates Jan 24 02:39:41 ntpd 33227 203.107.6.88 8011 81 mobilize assoc 39013 Jan 24 02:39:41 ntpd 33227 kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Jan 24 02:39:41 ntpd 33227 0.0.0.0 c01d 0d kern kernel time sync enabled Jan 24 02:39:41 ntpd 33227 kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Jan 24 02:39:41 ntpd 33227 0.0.0.0 c012 02 freq_set kernel -8.436 PPM Jan 24 02:39:41 ntpd 33227 0.0.0.0 c016 06 restart Jan 24 02:39:42 ntpd 33227 203.107.6.88 8014 84 reachable Jan 24 02:39:48 ntpd 33227 203.107.6.88 901a 8a sys_peer Jan 24 02:39:48 ntpd 33227 0.0.0.0 c61c 0c clock_step +1.006254 s Jan 24 02:39:49 ntpd 33227 0.0.0.0 c615 05 clock_sync Jan 24 02:40:58 ntpd 33227 0.0.0.0 c618 08 no_sys_peer Jan 24 02:40:58 ntpd 33227 203.107.6.88 8014 84 reachable Jan 24 02:41:04 ntpd 33227 203.107.6.88 901a 8a sys_peer Jan 24 02:41:08 ntpd 33227 0.0.0.0 0628 08 no_sys_peerMoreover, my pfSense instance is running in a virtual machine. The host OS is running Ubunutu + KVM + QEMU + libvirt in an Intel server.
-
@einsdisp said in pfSense NTP server is very unstable.:
0.0.0.0 061d 0d kern kernel time sync disabled
well that not good.. Huge time difference could be the cause of that.
If this is a vm, you prob want to make sure the VM isn't doing sync with the host, etc if you want it to sync time with ntp.
-
Mmm, it's unlikely to sync to a single server that is showing a 46s offset.
If you add a pool there so it can see multiple servers and they are all showing close to the same offset I would expect it to sync.
Why is the offset so large initially anyway?
Steve
-
I already tried NTP pool (rather than a single NTP server) before. No matter what I set, pfSense NTP always works fine only for the first several hours, then no longer working.
I just adjusted the system clock of the host OS and restarted pfSense VM. This time I set the external server to
ntp.aliyun.com. Everything is fine now:
Jan 25 01:52:47 ntpd 34952 ntpd 4.2.8p15@1.3728-o Thu Jun 24 21:53:38 UTC 2021 (1): Starting Jan 25 01:52:47 ntpd 34952 Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid Jan 25 01:52:47 ntpd 34952 ---------------------------------------------------- Jan 25 01:52:47 ntpd 34952 ntp-4 is maintained by Network Time Foundation, Jan 25 01:52:47 ntpd 34952 Inc. (NTF), a non-profit 501(c)(3) public-benefit Jan 25 01:52:47 ntpd 34952 corporation. Support and training for ntp-4 are Jan 25 01:52:47 ntpd 34952 available at https://www.nwtime.org/support Jan 25 01:52:47 ntpd 34952 ---------------------------------------------------- Jan 25 01:52:47 ntpd 35232 proto: precision = 17.470 usec (-16) Jan 25 01:52:47 ntpd 35232 basedate set to 2021-06-12 Jan 25 01:52:47 ntpd 35232 gps base set to 2021-06-13 (week 2162) Jan 25 01:52:47 ntpd 35232 Listen normally on 0 lo0 [::1]:123 Jan 25 01:52:47 ntpd 35232 Listen normally on 1 lo0 127.0.0.1:123 Jan 25 01:52:47 ntpd 35232 Listen normally on 2 bridge0 10.1.1.2:123 Jan 25 01:52:47 ntpd 35232 Listening on routing socket on fd #23 for interface updates Jan 25 01:52:47 ntpd 35232 kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Jan 25 01:52:47 ntpd 35232 0.0.0.0 c01d 0d kern kernel time sync enabled Jan 25 01:52:47 ntpd 35232 kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Jan 25 01:52:47 ntpd 35232 0.0.0.0 c012 02 freq_set kernel -8.436 PPM Jan 25 01:52:47 ntpd 35232 0.0.0.0 c016 06 restart Jan 25 01:52:53 ntpd 35232 DNS ntp.aliyun.com -> 203.107.6.88 Jan 25 01:52:53 ntpd 35232 203.107.6.88 8011 81 mobilize assoc 57546 Jan 25 01:52:54 ntpd 35232 203.107.6.88 8014 84 reachable Jan 25 01:53:00 ntpd 35232 203.107.6.88 901a 8a sys_peer Jan 25 01:53:00 ntpd 35232 0.0.0.0 c61c 0c clock_step +0.236668 s Jan 25 01:53:00 ntpd 35232 0.0.0.0 c615 05 clock_sync Jan 25 01:54:08 ntpd 35232 0.0.0.0 c618 08 no_sys_peer Jan 25 01:54:08 ntpd 35232 203.107.6.88 8014 84 reachable Jan 25 01:54:14 ntpd 35232 203.107.6.88 901a 8a sys_peerBut as expected, pfSense NTP will very likely to fail some hours later. I will track its status periodically.
To figure out whether it is due to VM, I run an OpenWrt instance in the same host OS, using the same KVM/libvirt config, and enable NTP server in OpenWrt. It turns out OpenWrt NTP works fine currently. I will check its status as well.
-
@einsdisp said in pfSense NTP server is very unstable.:
NTP server in OpenWrt.
While that is a good test, better test would be a freebsd vm..
Have you make sure to disable ntp sync with the host on the vm? I take it your running the openvm tools package.. Been quite sometime since have used that - but more than likely you want to disable its time sync function..
I could fire up the vm I have running under my nas virtual machine stuff, but I have never left it running for any length of time, and never even installed the vm tools package.
Edit: Seems I did have the openvm package installed.. So I have turned on graphing for ntp and will let this vm run for a day or so and see what it shows.
I just booted, and here is current status
edit2: here is current ntp graph
-
-
My host OS is Linux, and it was set to sync time to pfSense VM before. As you suggested, I disabled NTP in host OS just now:
sudo timedatectl set-ntp false. -
My virtualization software is KVM+QEMU+libvirt. "openvm-tools" is VMware staff. The KVM equivalent is
qemu-guest-agent, which does the host-VM time sync job. But pfSense VM does not have such staff apparently. There is no such "tools" running in pfSense VM which syncs VM time to host. -
My current NTP graph:
-
-
@einsdisp well if you got it turned off in the host, lets see if that has any effect on the issue you were seeing.
the openvm tools for doesnt really have a setting either that I can see in the gui, so guessing it might just disable that by default..
But from this - I take it the qemu package is available
https://forum.netgate.com/post/995504
that is if running 2.6 or 22.01 I take it.. I am looking forward to it myself for my VMs - since my nas virtual machine is qemu based.. I not sure why I had those openvm tools installed - might of been habit from when I ran esxi ;) I have now removed it. And think might update that vm to 2.6 to try out the qemu tools - now maybe my dashboard will show the IP of pfsense, and will be able to shutdown vs having to halt the system from inside the vm.
edit:
Well I updated to 2.6, and installed the package and then ran it and now I see IPs on that vm
On my VM dashboard on my nas..
edit: just to update its been hours and hours now and working as it should..
-
As I tested more these days, I finally figured out the cause of this issue: The host OS should not be set to sync time with pfSense VM. As my test, if I stopped the NTP of host OS, all runs fine. If I enabled host OS NTP, the host RTC will advance 3 more seconds compare to the real world clock, in every 5 minutes. The accumulative error is about 10 minutes per day.
I am not a KVM expert, but I guess it is due to that, by default, (or in my VM config), KVM may adjust VM RTC clock ticking rate, when host time is changed. So if host OS NTP sever is set to pfSense VM, it may ended up in a "dead loop".
My original VM config related to clocking:
<clock offset="utc"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="yes"/> </clock>I guess (but haven't tested), changing
<timer name="rtc" tickpolicy="catchup"/>to<timer name="rtc" tickpolicy="delay" track="guest"/>may direct KVM to handle VM RTC clock as normal, when host time changes, as if host time is not changed, thus resolving the "dead loop".But a more simple solution is disabling host OS NTP or set host OS NTP server to an external one, rather than pfSense VM.
-
My final question regarding pfSense itself:
it's unlikely to sync to a single server that is showing a 46s offset.
In my test, if pfSense VM RTC clock differs from remote NTP server by a large amount, pfSense refuses to sync time.
How to force pfSense to believe remote time of a single server, in case the offset is very large? I already checked "prefer" checkbox in the NTP server settings, but it is no use.
If there is no way for a single remote server, then how many servers is needed at least?
-
https://www.ntp.org/ntpfaq/NTP-s-trbl-general.htm#AEN5162
NTP will reject a peer that is #roughtly 20 or more minutes off.http://www.ntp.org/ntpfaq/NTP-s-algo.htm
And it will consider a 128ms diff enough to be "unsync'ed"@einsdisp said in pfSense NTP server is very unstable.:
How to force pfSense to believe remote time of a single server, in case the offset is very large?
ntpdate will "step the time" ,but requires the ntp daemon to have released it's binding to the UDP 123 port ... AKA "usually" not running.
/Bingo
-
This post is deleted!
