fails to get WAN IPv6 address ( was wrong date and time)

lifespeed · May 20, 2020, 3:07 AM

I see the wrong date and time everywhere, it's behind three days and the time error isn't due to UMT time. I have the NTP servers set to:

pool.ntp.org 0.pfsense.pool.ntp.org

What is going on?

Edit: the title of this post is updated to reflect the root of the problem, failed IPv6 WAN address, and internet connectivity. Please skip to this post for logs showing failure and success.

May 16 16:08:21 	php-fpm 	363 	/services_dhcp.php: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid ixl2' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.3.6-P1 Copyright 2004-2018 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ /etc/dhcpdv6.conf line 23: partial base64 value left over: 7. secret MCEW4oPUvz7wPqLNRGn6H3; ^ Configuration file errors encountered -- exiting If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug. These pages explain the proper process and the information we find helpful for debugging. exiting.'
May 16 16:08:19 	dhcpleases 		Could not deliver signal HUP to process because its pidfile (/var/run/dnsmasq.pid) does not exist, No such process.
May 16 16:08:19 	dhcpleases 		/etc/hosts changed size from original!
May 16 16:08:18 	check_reload_status 		Syncing firewall
May 16 16:05:55 	check_reload_status 		Reloading filter
May 16 16:05:55 	php-fpm 	363 	/system.php: NTPD is starting up.

netblues · May 20, 2020, 5:27 AM

@lifespeed try ntpq -p on cli.. what does it say?

lifespeed · May 21, 2020, 3:55 AM

Does this show pfSense not reaching the two NTP servers listed?

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 pool.ntp.org    .POOL.          16 p    -   64    0    0.000    0.000   0.000
 0.pfsense.pool. .POOL.          16 p    -   64    0    0.000    0.000   0.000
 ntp18.doctor.co 50.205.244.28    2 u   13   64  377   32.772  3119953 233177.
 173.0.48.220 (1 43.77.130.254    2 u   13   64  177   15.977  3119953 242990.
 LAX.CALTICK.NET 98.152.165.38    2 u    6   64  377    9.058  3120007 233509.
 www-ntp-org.nss 163.237.218.19   2 u   11   64  177   48.878  3119969 216499.
 tick.srs1.ntfo. 206.55.64.77     3 u    4   64  177    6.201  3120023 217997.
 ntp1.wiktel.com .PPS.            1 u   12   64   77   37.059  3119961 197102.
 ntp2.wiktel.com .GPS.            1 u    6   64   77   37.150  3120007 200672.
 time.cloudflare 10.4.0.166       3 u   11   64   77   10.648  3119968 196423.
 puppet.kenyonra .PHC0.           1 u    8   64   37    9.086  3119992 167410.
 66.85.78.80     172.16.23.153    2 u    4   64   17   43.065  3120023 134910.
 time.airgapped. 216.218.254.202  2 u    6   64   17   44.081  3120007 136793.
 69.10.161.7     195.205.216.85   3 u    5   64    7   13.168  3120014 95259.6
 vps5.ctyme.com  216.218.254.202  2 u    6   64    7    5.124  3120007 94638.9
 12.167.151.1    66.85.78.80      3 u    6   64    7   44.555  3120007 93908.8
 clock.nyc.he.ne .CDMA.           1 u    2   64    3   40.558  3120038 49260.9
 216.126.233.109 128.227.205.3    2 u    3   64    3   53.616  3120030 46951.5

netblues · May 21, 2020, 4:38 AM

Well it means it contacts the pool, gets the various ntp servers but it seems that no traffic on port 123 occurs. There should be one star and a few + and - in front of the names. Certainly you don't need to create any rules on your wan interface.
Looks like a connectivity issue.
Most probably something blocks ntp traffic upstream.

lifespeed · May 21, 2020, 5:11 AM

@netblues said in wrong date and time:

Most probably something blocks ntp traffic upstream.

If that were the case wouldn't my devices on the network have problems with NTP as well? I just checked the time on my Windows 10 PC, and the time is exactly the same as my mobile phone.

lifespeed · May 21, 2020, 5:39 AM

Is the "clock unsynchronized" error below the symptom, not the problem? ixl2 is LAN, igb0 is WAN.


May 17 07:02:11 	ntpd 	42225 	Soliciting pool server 173.255.215.209
May 17 07:02:10 	ntpd 	42225 	Soliciting pool server 64.22.253.155
May 17 07:02:10 	ntpd 	42225 	Soliciting pool server 138.236.128.112
May 17 07:02:09 	ntpd 	42225 	Soliciting pool server 207.192.69.118
May 17 07:02:09 	ntpd 	42225 	Soliciting pool server 162.159.200.1
May 17 07:02:08 	ntpd 	42225 	Soliciting pool server 199.102.46.78
May 17 07:02:08 	ntpd 	42225 	Soliciting pool server 45.33.84.208
May 17 07:02:07 	ntpd 	42225 	Soliciting pool server 50.116.52.97
May 17 07:02:06 	ntpd 	42225 	kernel reports TIME_ERROR: 0x4041: Clock Unsynchronized
May 17 07:02:06 	ntpd 	42225 	kernel reports TIME_ERROR: 0x4041: Clock Unsynchronized
May 17 07:02:06 	ntpd 	42225 	Listening on routing socket on fd #31 for interface updates
May 17 07:02:06 	ntpd 	42225 	Listen normally on 10 lo0 127.0.0.1:123
May 17 07:02:06 	ntpd 	42225 	Listen normally on 9 lo0 [fe80::1%10]:123
May 17 07:02:06 	ntpd 	42225 	Listen normally on 8 lo0 [::1]:123
May 17 07:02:06 	ntpd 	42225 	Listen normally on 7 ixl2 [fe80::1:1%7]:123
May 17 07:02:06 	ntpd 	42225 	Listen normally on 6 ixl2 [2601:646:xxxx:xxxx:225:90ff:febb:xxxx]:123
May 17 07:02:06 	ntpd 	42225 	Listen normally on 5 ixl2 192.168.1.1:123
May 17 07:02:06 	ntpd 	42225 	Listen normally on 4 igb0 [2001:xxx:6045:45:xxxx:6b5c:xxxx:xxx]:123
May 17 07:02:06 	ntpd 	42225 	Listen normally on 3 igb0 76.126.xxx.xxx:123
May 17 07:02:06 	ntpd 	42225 	Listen normally on 2 igb0 [fe80::225:90ff:feba:5364%1]:123
May 17 07:02:06 	ntpd 	42225 	Listen and drop on 1 v4wildcard 0.0.0.0:123
May 17 07:02:06 	ntpd 	42225 	Listen and drop on 0 v6wildcard [::]:123
May 17 07:02:06 	ntpd 	42225 	gps base set to 2019-04-28 (week 2051)
May 17 07:02:06 	ntpd 	42225 	basedate set to 2019-04-28
May 17 07:02:06 	ntpd 	42225 	proto: precision = 0.047 usec (-24)
May 17 07:02:06 	ntpd 	42060 	Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
May 17 07:02:06 	ntpd 	42060 	ntpd 4.2.8p13@1.3847-o Fri May 10 20:05:13 UTC 2019 (1): Starting
May 17 07:02:06 	ntpd 	35185 	64.22.253.155 local addr 76.1xx.xxx.72 -> <null>

netblues · May 21, 2020, 5:26 AM

@lifespeed said in wrong date and time:

@netblues said in wrong date and time:

Most probably something blocks ntp traffic upstream.

If that were the case wouldn't my devices on the network have problems with NTP as well? I just checked the time on my Windows 10 PC, and the time is exactly the same as my mobile phone.

Seems so..
Have a look at states, filter on port 123 and see if you get anything
Sync doesnt occur all the time
You should also see ntp traffic from your lan.

This is the wan port syncing pf

lifespeed · May 21, 2020, 5:37 AM

@netblues said in wrong date and time:

Have a look at states, filter on port 123 and see if you get anything

Is this what you mean, firewall logs filtered by port? My external IP is 76.126.xxx.xxx. Do I need to allow inbound from WAN on port 123 to my external IP for pFsense? I'm using dual stack v4 and v6.

 	May 17 07:03:47 	WAN 	Default deny rule IPv4 (1000000103) 	156.96.155.246:57873		76.126.xxx.xxx:123		UDP
	May 17 07:00:54 	WAN 	Default deny rule IPv4 (1000000103) 	51.159.59.122:55133		76.126.xxx.xxx:123		UDP
	May 17 07:00:02 	WAN 	Default deny rule IPv4 (1000000103) 	45.13.93.82:44178		76.126.xxx.xxx::8123		TCP:S
	May 17 06:54:04 	WAN 	Default deny rule IPv4 (1000000103) 	5.182.210.95:59074		76.126.xxx.xxx:123		UDP
	May 17 06:53:53 	WAN 	Default deny rule IPv4 (1000000103) 	194.180.224.123:33839		76.126.xxx.xxx:123		UDP
	May 17 06:38:00 	WAN 	Default deny rule IPv4 (1000000103) 	80.82.65.190:47382		76.126.xxx.xxx:123		TCP:S
	May 17 06:26:19 	WAN 	Default deny rule IPv4 (1000000103) 	167.172.49.247:48318		76.126.xxx.xxx:12336		TCP:S
	May 17 06:06:38 	WAN 	Default deny rule IPv4 (1000000103) 	5.182.210.95:39736		76.126.xxx.xxx:123		UDP

netblues · May 21, 2020, 5:48 AM

@lifespeed Well, logs have much more noise, but could do.
States is realtime

However you have to refresh a bit to spot it. Sync doesn't occur all the time.
Now judging from the logs seems that something blocks port 123.
Having said that, on a simple pf installation, with only ping allowed on wan and everything allowed on the lan, ntp doesn't need anything special to function.

As a test, try allowing incoming on wan on 123 and see if it works.
If it does, then the problem has to be investigated on firewall rules.

lifespeed · May 21, 2020, 5:57 AM

I don't know what you mean by "filter on states". I allowed WAN with a destination of port 123, no change.

At one time I had VLANs setup, but gave up as they seemed beyond my limited abilities. Is it possible I have somehow assigned NTP to a VLAN that no longer exists? I see the NTP server has both WAN and LAN listed, but is that relevant? I'm not trying to run an NTP server on my network.

netblues · May 21, 2020, 6:16 AM

@lifespeed go to web interface, diagnostics states, and use 123 as a filter and see what you get. See my picture attached. :)

Well, as it says, if you have nothing selected on ntp, it will listen on all interfaces.
And a server and a client is essentially the same thing here.

I doubt vlans have anything to do with that.
Please verify that you are able to ping from cli most if not all of the ip's that appear on ntpq -p too.

If still all is well and no ntp works, a packet capture on the wan , on udp 123 is the next thing (diagnostics, packet capture)

lifespeed · May 21, 2020, 6:36 AM

I ran a packet capture on the WAN, port 123, not promiscuous, and got nothing after I stopped the capture and pressed "view capture". I ran ntpq -p from CLI during the packet capture.

Thanks for all your help this far. I can't help but think I have deleted or changed a default setting that has borked NTP. But it would seem the default allow all outgoing would work.

Here are my LAN rules, outgoing is allowed.

lifespeed · May 21, 2020, 6:45 AM

I can ping the NTP servers from CLI.

netblues · May 21, 2020, 6:48 AM

Lan rules is ok, certainly not an issue.
How about outbound nat?
(if you can ping, most probably no issues there too)
Do you still getting logs with 123 denied? With an allow rule on wan you must be able to capture that traffic no matter what.

This is what capture looks

lifespeed · May 23, 2020, 3:17 AM

I enabled WAN port 123 again, and packet captured while running

ntpq -p

from the command line. Even with WAN rule set there doesn't seem to be any action on packet capture.

Here's outbound NAT, it may show an automatic rule vestige of my failed VLAN attempt. 192.168.2.0 I think is a leftover VLAN, while my LAN is 192.168.1.0.

I am not seeing any port 123 blocks in the firewall logs. States shows port 123 activity.

netblues · May 23, 2020, 3:47 AM

Well, since you see states, with 76 bytes on your wan, then you are getting ntp packets..
So no firewall is blocking anything. You should be able to capture them too.
Try it via the web interface, diagnostics packet capture.

What is status/ntp say and what is in status/system logs/ntp ?
Also enable all logging options in services/ntp
I guess you have tried rebooting, as a last resort.

lifespeed · May 23, 2020, 5:21 AM

@netblues

it worked this time, packet capture that is

10:06:52.522715 IP 76.126.xxx.xxx.123 > 4.53.160.75.123: UDP, length 48
10:06:52.554326 IP 4.53.160.75.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:53.523242 IP 76.126.xxx.xxx.123 > 185.215.224.214.123: UDP, length 48
10:06:53.523260 IP 76.126.xxx.xxx.123 > 192.81.135.252.123: UDP, length 48
10:06:53.532481 IP 192.81.135.252.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:53.534081 IP 185.215.224.214.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:54.523411 IP 76.126.xxx.xxx.123 > 50.212.98.107.123: UDP, length 48
10:06:54.580536 IP 50.212.98.107.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:55.523635 IP 76.126.xxx.xxx.123 > 192.81.135.252.123: UDP, length 48
10:06:55.533074 IP 192.81.135.252.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:56.527876 IP 76.126.xxx.xxx.123 > 4.53.160.75.123: UDP, length 48
10:06:56.527882 IP 76.126.xxx.xxx.123 > 209.50.63.74.123: UDP, length 48
10:06:56.527887 IP 76.126.xxx.xxx.123 > 45.63.54.13.123: UDP, length 48
10:06:56.533229 IP 209.50.63.74.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:56.539392 IP 45.63.54.13.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:56.556456 IP 4.53.160.75.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:57.526188 IP 76.126.xxx.xxx.123 > 192.81.135.252.123: UDP, length 48
10:06:57.526195 IP 76.126.xxx.xxx.123 > 44.190.6.254.123: UDP, length 48
10:06:57.531438 IP 44.190.6.254.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:57.531588 IP 192.81.135.252.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:58.524430 IP 76.126.xxx.xxx.123 > 4.53.160.75.123: UDP, length 48
10:06:58.556095 IP 4.53.160.75.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:59.522822 IP 76.126.xxx.xxx.123 > 192.81.135.252.123: UDP, length 48
10:06:59.522833 IP 76.126.xxx.xxx.123 > 159.203.158.197.123: UDP, length 48
10:06:59.529147 IP 192.81.135.252.123 > 76.126.xxx.xxx.123: UDP, length 48
10:06:59.571784 IP 159.203.158.197.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:00.522475 IP 76.126.xxx.xxx.123 > 4.53.160.75.123: UDP, length 48
10:07:00.551285 IP 4.53.160.75.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:01.524139 IP 76.126.xxx.xxx.123 > 192.81.135.252.123: UDP, length 48
10:07:01.524151 IP 76.126.xxx.xxx.123 > 162.159.200.1.123: UDP, length 48
10:07:01.533563 IP 162.159.200.1.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:01.533960 IP 192.81.135.252.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:02.527698 IP 76.126.xxx.xxx.123 > 4.53.160.75.123: UDP, length 48
10:07:02.556506 IP 4.53.160.75.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:03.530844 IP 76.126.xxx.xxx.123 > 192.81.135.252.123: UDP, length 48
10:07:03.536242 IP 192.81.135.252.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:04.528000 IP 76.126.xxx.xxx.123 > 4.53.160.75.123: UDP, length 48
10:07:04.528013 IP 76.126.xxx.xxx.123 > 72.87.88.203.123: UDP, length 48
10:07:04.556810 IP 4.53.160.75.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:04.574781 IP 72.87.88.203.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:05.530521 IP 76.126.xxx.xxx.123 > 192.81.135.252.123: UDP, length 48
10:07:05.536805 IP 192.81.135.252.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:06.523536 IP 76.126.xxx.xxx.123 > 4.53.160.75.123: UDP, length 48
10:07:06.552973 IP 4.53.160.75.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:07.523388 IP 76.126.xxx.xxx.123 > 66.228.58.20.123: UDP, length 48
10:07:07.563108 IP 66.228.58.20.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:08.525179 IP 76.126.xxx.xxx.123 > 45.76.244.193.123: UDP, length 48
10:07:08.525183 IP 76.126.xxx.xxx.123 > 104.194.8.227.123: UDP, length 48
10:07:08.538954 IP 104.194.8.227.123 > 76.126.xxx.xxx.123: UDP, length 48
10:07:08.542720 IP 45.76.244.193.123 > 76.126.xxx.xxx.123: UDP, length 48

NTP status is still bad

Network Time Protocol Status
Pool Placeholder 	pool.ntp.org 	.POOL. 	16 	p 	- 	64 	0 	0.000 	0.000 	0.000
Pool Placeholder 	0.pfsense.pool. 	.POOL. 	16 	p 	- 	64 	0 	0.000 	0.000 	0.000
Unreach/Pending 	45.76.244.193 	216.239.35.0 	2 	u 	8 	64 	377 	18.508 	3894306 	232734.
Unreach/Pending 	198.60.22.240 	.XMIS. 	1 	u 	6 	64 	377 	14.489 	3894322 	232993.
Unreach/Pending 	66.228.58.20 	198.72.72.10 	3 	u 	12 	64 	377 	33.938 	3894276 	230345.
Unreach/Pending 	50.212.98.107 	209.50.63.74 	3 	u 	22 	64 	377 	58.577 	3894199 	228598.
Unreach/Pending 	159.203.158.197 	128.59.0.245 	2 	u 	20 	64 	377 	50.835 	3894214 	228293.
Unreach/Pending 	208.67.72.43 	152.2.133.55 	2 	u 	4 	64 	377 	47.912 	3894337 	229180.
Unreach/Pending 	104.236.116.147 	128.59.0.245 	2 	u 	3 	64 	377 	45.172 	3894345 	231712.
Unreach/Pending 	68.233.45.146 	132.163.96.1 	2 	u 	5 	64 	271 	46.219 	3894330 	303620.
Unreach/Pending 	162.159.200.1 	10.4.0.166 	3 	u 	17 	64 	77 	8.466 	3894237 	196053.
Unreach/Pending 	45.63.54.13 	98.152.165.38 	2 	u 	20 	64 	37 	11.729 	3894214 	170106.
Unreach/Pending 	185.215.224.214 	164.67.62.194 	2 	u 	20 	64 	37 	12.150 	3894214 	171169.
Unreach/Pending 	44.190.6.254 	127.67.113.92 	2 	u 	23 	64 	37 	5.146 	3894191 	166467.
Unreach/Pending 	209.50.63.74 	198.60.22.240 	2 	u 	22 	64 	37 	6.787 	3894198 	168655.
Unreach/Pending 	4.53.160.75 	142.66.101.13 	2 	u 	22 	64 	17 	29.953 	3894199 	133965.
Unreach/Pending 	192.81.135.252 	.PHC0. 	1 	u 	20 	64 	17 	8.279 	3894214 	136273.
Unreach/Pending 	163.237.218.19 	.GPS. 	1 	u 	23 	64 	7 	31.696 	3894191 	92463.5
Unreach/Pending 	72.5.72.15 	216.218.254.202 	2 	u 	37 	64 	3 	14.127 	3894083 	49288.6
Unreach/Pending 	69.10.161.7 	195.205.216.85 	3 	u 	28 	64 	1 	13.803 	3894152 	5098.69

lifespeed · May 23, 2020, 5:22 AM

Isn't the outbound NAT table auto rules a bit suspicious with the extra subnet?

lifespeed · May 23, 2020, 5:32 AM

@netblues said in wrong date and time:

Well, since you see states, with 76 bytes on your wan, then you are getting ntp packets..

Doesn't states show I'm sending NTP, not receiving?

netblues · May 23, 2020, 5:39 AM

@lifespeed 76/76 means 76 send 76 received... so you are receiving
No, nat is not suspicius.. Move the tab to manual, delete everything and rehit auto.
(i suppose you don't have any manual entries)
it should create new ones without the stale vlan.

And in your capture there are packets traveling both ways too.

Post the ntp log from system logs, and restart ntp service