NTP Status Broken?

[dweimer]

My pfSense firewalls NTP GUI status is broken. I believe the problem is that the NTPv4 localhost ntpq is getting blocked but I can't figure out why. I can connect to the console and use ntpq to query peers unless I put the -4 switch. I then get this.

ntpq: write to localhost failed: Can't assign requested address

I have done nothing to block IPv4 in fact I don't have any IPv6 addresses besides the localhost configured as my local ISP doesn't have support for IPv6. I have tried adding custom ACLs but nothing seems to help. netstat -an shows its listening on 127.0.0.1:123 and I have compared NTP settings to other pfSense setups that are working. Does anyone have any ideas as to what could be causing this or how to fix it?

[Gertjan]

@dweimer

Check here : https://forum.netgate.com/topic/76535/status-ntp-never-displays-any-peers ?

If the issue related to IPv6, and your ISP doesn't support it, you can allow IPv6 here : System > Advanced > Networking.

[dweimer]

@gertjan I did find that post earlier, but in my case IPv6 is working, its IPv4 that is not.

[2.5.2-RELEASE][admin@pfSense.dweimer.me]/root: ntpq -n ::1
ntpq> lpeers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.pfsense.pool. .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+65.100.46.164   .SOCK.           1 u  176  128  376   59.108   -1.459   2.249
*44.190.6.254    .PPS.            1 u   31  128  377   49.185   -0.467   0.294
-159.89.86.140   128.59.0.245     2 u   72  128  377   41.247   -1.087   0.165
+192.210.199.110 217.180.209.214  2 u   52  128  377   25.762   -0.352   0.214
ntpq> quit
[2.5.2-RELEASE][admin@pfSense.dweimer.me]/root: ntpq -n 127.0.0.1
ntpq> lpeers
ntpq: write to 127.0.0.1 failed: Can't assign requested address
ntpq> quit

Its listening on all IPv4 and IPv6 UDP ports and correctly servicing clients behind it.

[2.5.2-RELEASE][admin@pfSense.dweimer.me]/root: netstat -an | grep 123
udp4       0      0 127.0.0.1.123          *.*
udp6       0      0 fe80::1%lo0.123        *.*
udp6       0      0 ::1.123                *.*
udp4       0      0 192.168.190.1.123      *.*
udp6       0      0 fe80::4262:31ff:.123   *.*
udp4       0      0 10.9.5.1.123           *.*
udp6       0      0 fe80::4262:31ff:.123   *.*

But, the GUI status displays no peers.

I checked the Loopback rules from the message you posted above.

[2.5.2-RELEASE][admin@pfSense.dweimer.me]/root: grep loop /tmp/rules.debug
loopback = "{ lo0 }"
# Allow IPv6 on loopback
pass in  quick on $loopback inet6 all tracker 1000000001 label "pass IPv6 loopback"
pass out  quick on $loopback inet6 all tracker 1000000002 label "pass IPv6 loopback"
# loopback
pass in  on $loopback inet all tracker 1000004761 label "pass IPv4 loopback"
pass out  on $loopback inet all tracker 1000004762 label "pass IPv4 loopback"

Looks like its allowed, also IPv4 works on the LAN port as well, its just not on the loopback.

[2.5.2-RELEASE][admin@pfSense.dweimer.me]/root: ntpq -n 10.9.5.1
ntpq> lpeers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.pfsense.pool. .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+65.100.46.164   .SOCK.           1 u   18  128  367   59.224   -1.256   1.181
*44.190.6.254    .PPS.            1 u    5  128  377   49.185   -0.467   0.291
-159.89.86.140   128.59.0.245     2 u   49  128  377   41.247   -1.087   0.106
+192.210.199.110 217.180.209.214  2 u   27  128  377   25.739   -0.386   0.162

I did also try a reinstall and restore configuration it didn't clear it. So whatever is causing it is something in my configuration but I am at a lost as to what it could be. Is there a way to reset only the NTP configuration?

[johnpoz]

@dweimer can we see your acl page, and your actual ntp.conf file

example here mine.

[dweimer]

@johnpoz

[2.5.2-RELEASE][admin@pfSense.dweimer.me]/root: cat /var/etc/ntpd.conf
#
# pfSense ntp configuration file
#

tinker panic 0

# Orphan mode stratum and Maximum candidate NTP peers
tos orphan 12 maxclock 5


# Upstream Servers
pool 0.pfsense.pool.ntp.org iburst maxpoll 9


enable stats
statistics peerstats
statsdir /var/log/ntp
logconfig =syncall +clockall +sysall
driftfile /var/db/ntpd.drift
restrict default kod limited nomodify nopeer notrap
restrict -6 default kod limited nomodify nopeer notrap
restrict source kod limited nomodify notrap
interface ignore all
interface ignore wildcard
interface listen igb1
interface listen igb2
interface listen lo0

[dweimer]

I am thinking its not an NTP configuration issue, its something else. I just tried doing a telnet to 127.0.0.1 using port 123. That would be on TCP instead of UDP but I get the same error can't write to requested address. So something on the system is blocking attempts to connect outbound from 127.0.0.1.

[2.5.2-RELEASE][admin@pfSense.dweimer.me]/root: telnet 127.0.0.1 123
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Can't assign requested address
telnet: Unable to connect to remote host

[johnpoz]

@dweimer said in NTP Status Broken?:

restrict source kod limited nomodify notrap

Off the top I don't know what "source" is in that statement?

also I don't have this

interface listen lo0

And I can query on loopback just fine.. That is for the ntp come up as server on loopback.. Which is really kind of pointless ;)

Do you have vip or something setup there?

[21.05.2-RELEASE][admin@sg4860.local.lan]/: ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[21.05.2-RELEASE][admin@sg4860.local.lan]/: 

I don't have loopback selected, but still show 123 listen on loopback

[Gertjan]

@johnpoz said in NTP Status Broken?:

And I can query on loopback just fine.. That is for the ntp come up as server on loopback.. Which is really kind of pointless ;)

A local 'pFsense' process could address 127.0.0.1:123 for some time info?!

If you query loopback or localhost or 127.0.0.1 (:123) then a ntp server process should listen on that address:port.

sockstat -4l | grep 'ntp'

confirms it does so for me.

We me, "udp4 *:123" overlaps even "udp4 127.0.0.1:123".

root     ntpd       51142 21 udp4   *:123                 *:*
.....
root     ntpd       51142 32 udp4   127.0.0.1:123         *:*
.....

when I de select 'Localhost' I still see :

root     ntpd       68960 21 udp4   192.168.10.3:123      *:*
root     ntpd       68960 23 udp4   192.168.1.1:123       *:*
root     ntpd       68960 26 udp4   192.168.2.1:123       *:*
root     ntpd       68960 29 udp4   127.0.0.1:123         *:*
root     ntpd       68960 31 udp4   192.168.3.1:123       *:*

Nice .... !?!

Btw : IPv6 : the same thing : "udp6 ::1:123 :" stays in place.

[dweimer]

@johnpoz
I removed the loopback interface from the listen on list, no change in behavior still shows its listening on 127.0.0.1:123 as well as ::1:123 but still can't connect. to 127.0.0.1 I did test, DNS is listening on 127.0.0.1:53 and I can't connect to it either.
Something is preventing the system from binding to lo0 with IPv4 for querying itself.

[johnpoz]

@dweimer yeah I don't know.. But I don't have that selected and I can query 127 for ntp..

What would be doing a query on pfsense to ntp though? To set its time, ntp sets the system time, etc.

But to validate that am actually connecting to 127.0.0.1 when do ntpq, I tried then an IP that clearly is not on pfsense or anywhere on my network.. and get permission denied error

Can you output the ipconfig for lo0

[dweimer]

@johnpoz I simply want to look at the dashboard and quickly see that my NTP is in sync. Instead of it telling me that I have no peers and to check service.

[2.5.2-RELEASE][admin@pfSense.dweimer.me]/: ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I would expect that ntpq couldn't connect to a nonexistent address, but it should bind to the adapter and attempt a query. the permission denied message above is different than the can't assign requested address I receive connecting to 127.0.0.1.

[johnpoz]

@dweimer I was just wanting to show that it was actually taking in the IP put in and not connecting to something else.

edit: still curious to what this is and does
restrict source kod limited nomodify notrap

Off the top of my head source seems like a placeholder, but not sure what it is.. I sure do not have that.. Do your other instances that do work have that?

[dweimer]

@johnpoz yes the other instances have that, I don't know what is putting it in there I do know if I manually remove it and restart the service it comes back.

[johnpoz]

@dweimer do you have some local pps or gps or something that is suppose to be setting time? Under the Serial GPS or PPS tabs?

[dweimer]

@johnpoz Nope, only configured to use external NTP servers, currently just set to 0.pfsense.pool.ntp.org.

[dweimer]

Well, I somehow resolved it...
Sort of, I downloaded the configuration manually edited the XML file, removing the <ntpd>...</ntpd> section. Did a restore of full configuration, after the reboot it works, checked the NTP configuration, all looks the same. Even Debug output is all the same except now both IPv4 127.0.0.1 and IPv6 ::1 query through ntpq work.
Only thing I can figure is that there is a hidden or corrupted character in old ntp configuration section.