NTP server running on pfsense is rejecting some peers (NTP clients)

mauro.tridici

Dear Users,

I recently noticed that some of our switches is out of NTP sync and I started investigating about this issue with the dedicated support staff.

We have gone through the logs, the NTP is out of sync due to intermittent reject from the NTP server.

NTP sync is flapping and, when it doesn't work as expected, I see that the peer is rejected.

cumulus@MNGL07AX:mgmt:~$ ntpq -p
remote refid st t when poll reach delay offset jitter
172.31.0.1 80.88.90.14 3 u 38 64 1 0.343 44.614 8.652

cumulus@MNGL07AX:mgmt:~$ ntpq -c as
ind assid status conf reach auth condition last_event cnt
1 6726 9014 yes yes none reject reachable 1

Since the problem has been seen only on a particular group of switches, I don't think it is related to our internal NTP server (that is pfSense server).

Anyway, support staff asked me to collect the NTP server logs to understand why the peer is rejected.

Could you please help me to understand how I can collect/download these logs capturing the reason of rejection?

Thank you in advance.
Mauro

KOM

@mauro-tridici Status - System Logs - NTP

mauro.tridici

@kom thank you for your reply.

Unfortunately, the NTP logs listed in Status - System Logs - NTP don't help so much because I only can see that the NTP server is correctly listening on the active interfaces.

I'm not able to see the reason of "reject".

Do you have some other idea?
Thank you,
Mauro

mauro.tridici

Good morning @stephenw10 ,

sorry if I'm disturbing you, but, if it is possible, i would like to know your opinion/suggestions about my issue.

Could you please take a look at my case?

Thank you in advance,
Mauro

Gertjan

@mauro-tridici

ntpq -c as and ntpq -p :
You only have one peer ?

Why not a list of them ? :

[23.01-RELEASE][admin@pfSense.near.by]/root: ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 fr.pool.ntp.org .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+cp01.webhd.nl   27.124.125.251   3 u  185  512  377   20.799   +0.997   0.152
*saturne.obs-bes .LTFB.           1 u  104  512  377   26.789   +1.250  29.841
+82-64-32-33.sub 82.64.45.50      2 u  245  512  375   19.930   +1.484   0.493
-37.59.63.125    193.190.230.65   2 u  329  512  377   20.966   +1.927   0.473
[23.01-RELEASE][admin@pfSense.near.by]/root: ntpq -c as
ind assid status  conf reach auth condition  last_event cnt
===========================================================
  1 53095  8811   yes  none  none    reject    mobilize  1
  2 53096  141a    no   yes  none candidate    sys_peer  1
  3 53097  162a    no   yes  none  sys.peer    sys_peer  2
  4 53098  1414    no   yes  none candidate   reachable  1
  5 53099  132a    no   yes  none   outlier    sys_peer  2

What happens when you set up a public NTP source, instead of pfSense on the switch ?

You can limit the IP of the switch to a selected number of trusted NTP sources, if you don't want your switch going outside.

Or : Diagnostics > Packet Capture and do some packet capturing.
IPs and ports involved, as the protocol UDP, are known.

I can image that, if the switches are 'flooding' the pfSense NTP server with requests, it says 'shut up' after a while.

When I have a device that handles time, like PCs, printers, NAS, DVR, etc etc, I give them '192.168.1.1' and never come back.

mauro.tridici

Good morning @gertjan
Thanks for your reply.

Below you can find my answers:

yes, on pfsense NTP service, I set only one NTP server.

In the pfsense NTP service page I read that: " If only one server is configured, it will be believed, and if 2 servers are configured and they disagree, neither will be believed. ". So, in order to troubleshoot my issue, I removed the 0.it.ntp.pool.org pool. (please, correct me if I'm wrong)

I tried to set a public NTP source, but it didn't fix the problem. Only a particular subset of my switches has this issue. All the other hosts, switches and devices are correctly synchronised. The bad switches belong to the same hardware model family/type.
I just captured the pcap on the NTP client, but I'm not an expert. Do you want to take a look at it?

Many thanks for your help and suggestions.
Mauro

stephenw10

In the NTP server settings enable Log peer messages and probably also Log system messages. That will likely show you the rejection.

KOM

@mauro-tridici Accessing an NTP server is free so I'm not sure why you limit to just one. I use:

0.ca.pool.ntp.org
1.ca.pool.ntp.org
2.ca.pool.ntp.org
3.ca.pool.ntp.org

The more, the merrier.

johnpoz

@mauro-tridici said in NTP server running on pfsense is rejecting some peers (NTP clients):

Do you want to take a look at it?

hard to look at if you don't post it.

Dobby_

@mauro-tridici
Bumb! Some news here? Have you solved out that problem?

What version of pfSense you are using?

mauro.tridici

Hello @dobby_ ,

thank you for your reply.
I was able to fix the NTP sync problem detected on some particular "NTP client" devices.
It was an issue related to the NTP client software. The devices vendor support suggested to uninstall ntp client and install chrony.

Now, everything is working as expected.
Anyway, I wasn't able to increase the verbosity of NTP server logs on pfSense 2.6 and I wasn't able to detect the reason of "reject" issue.

Have a great day,
Mauro