NTP Issue
-
@stephenw10 Should I have opened the PR against a branch for 2.7 when I first opened it instead of master? As of now, it is in the correct place though right?
-
Nope master is almost always the correct place because that's where all the development happens. It's much easier to merge and test stuff there.
If it all works well, and there is cause to do so, it can be back ported to 2.7.X.
-
I'm still running 2.7.2 as everything else seems to be just fine, but things have changed on my network a while ago with yet another stratum 1 NTP server.
I expect I won't be totally happy until I have four, but that will be risking death from my girlfriend, if she ever finds out.
The number of NTP based clocks has also increased from one to four.
I can recommend the "When" iottimer wifi clock from Aliexpress actually. Just follow the instructions about refreshing your browser window once you try changing it to English, or it won't appear to change.Anyway, the second NTP server has two interfaces which can be independently configured, so today I decided to fire up another firewall interface and allocate it a class B address and directly connect it to the second interface on the new NTP server.
So now I have three local stratum 1 NTP sources configured. All NTP servers show the connections from the firewall on all three interfaces when it polls them every minute.
As far as the firewall is concerned, they're pending/unreachable.
For extra fun, the interface connected solely to the NTP second interface has an explicit permit any NTP rule.I even tried setting the second NTP server interface to broadcast mode. Zilch.
It's disappointing as prior to 2.7.2, my firewall was sub millisecond synchronised.
-
Do you see replies back to the fire wall in the states? Or in a pcap?
-
@stephenw10
Nope.
Only the external NTP servers are seen and used. -
@stephenw10
Might have helped to post this:09:22:49.170320 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 28353, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x8f85!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.006912, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801704.158000500 (2024-07-24T09:21:44Z)
Receive Timestamp: 3930801704.161610624 (2024-07-24T09:21:44Z)
Transmit Timestamp: 3930801769.170296701 (2024-07-24T09:22:49Z)
Originator - Receive Timestamp: +0.003610124
Originator - Transmit Timestamp: +65.012296201
09:22:49.170381 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10031, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801769.000000000 (2024-07-24T09:22:49Z)
Originator Timestamp: 3930801769.170296701 (2024-07-24T09:22:49Z)
Receive Timestamp: 3930801769.167000500 (2024-07-24T09:22:49Z)
Transmit Timestamp: 3930801769.167000500 (2024-07-24T09:22:49Z)
Originator - Receive Timestamp: -0.003296201
Originator - Transmit Timestamp: -0.003296201
09:22:57.187023 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 45902, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x5082!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.007034, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801769.167000500 (2024-07-24T09:22:49Z)
Receive Timestamp: 3930801769.170386584 (2024-07-24T09:22:49Z)
Transmit Timestamp: 3930801777.187008800 (2024-07-24T09:22:57Z)
Originator - Receive Timestamp: +0.003386084
Originator - Transmit Timestamp: +8.020008299
09:22:57.187083 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10032, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801777.000000000 (2024-07-24T09:22:57Z)
Originator Timestamp: 3930801777.187008800 (2024-07-24T09:22:57Z)
Receive Timestamp: 3930801777.183000500 (2024-07-24T09:22:57Z)
Transmit Timestamp: 3930801777.183000500 (2024-07-24T09:22:57Z)
Originator - Receive Timestamp: -0.004008300
Originator - Transmit Timestamp: -0.004008300
09:23:05.179910 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 43251, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x5129!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.007156, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801777.183000500 (2024-07-24T09:22:57Z)
Receive Timestamp: 3930801777.187088760 (2024-07-24T09:22:57Z)
Transmit Timestamp: 3930801785.179879918 (2024-07-24T09:23:05Z)
Originator - Receive Timestamp: +0.004088260
Originator - Transmit Timestamp: +7.996879418
09:23:05.179988 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10033, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801785.000000000 (2024-07-24T09:23:05Z)
Originator Timestamp: 3930801785.179879918 (2024-07-24T09:23:05Z)
Receive Timestamp: 3930801785.176000500 (2024-07-24T09:23:05Z)
Transmit Timestamp: 3930801785.176000500 (2024-07-24T09:23:05Z)
Originator - Receive Timestamp: -0.003879418
Originator - Transmit Timestamp: -0.003879418
09:24:11.223769 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 17146, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x2cfc!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.008148, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801785.176000500 (2024-07-24T09:23:05Z)
Receive Timestamp: 3930801785.180030896 (2024-07-24T09:23:05Z)
Transmit Timestamp: 3930801851.223755123 (2024-07-24T09:24:11Z)
Originator - Receive Timestamp: +0.004030396
Originator - Transmit Timestamp: +66.047754622
09:24:11.223843 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10034, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801851.000000000 (2024-07-24T09:24:11Z)
Originator Timestamp: 3930801851.223755123 (2024-07-24T09:24:11Z)
Receive Timestamp: 3930801851.220000500 (2024-07-24T09:24:11Z)
Transmit Timestamp: 3930801851.220000500 (2024-07-24T09:24:11Z)
Originator - Receive Timestamp: -0.003754622
Originator - Transmit Timestamp: -0.003754622
09:24:19.195033 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 50821, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0xae46!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.008270, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801851.220000500 (2024-07-24T09:24:11Z)
Receive Timestamp: 3930801851.223882897 (2024-07-24T09:24:11Z)
Transmit Timestamp: 3930801859.195007028 (2024-07-24T09:24:19Z)
Originator - Receive Timestamp: +0.003882397
Originator - Transmit Timestamp: +7.975006527
09:24:19.195121 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10035, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801859.000000000 (2024-07-24T09:24:19Z)
Originator Timestamp: 3930801859.195007028 (2024-07-24T09:24:19Z)
Receive Timestamp: 3930801859.192000500 (2024-07-24T09:24:19Z)
Transmit Timestamp: 3930801859.192000500 (2024-07-24T09:24:19Z)
Originator - Receive Timestamp: -0.003006528
Originator - Transmit Timestamp: -0.003006528
09:24:27.176376 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 48593, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x37e1!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.008392, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801859.192000500 (2024-07-24T09:24:19Z)
Receive Timestamp: 3930801859.195166604 (2024-07-24T09:24:19Z)
Transmit Timestamp: 3930801867.176368362 (2024-07-24T09:24:27Z)
Originator - Receive Timestamp: +0.003166103
Originator - Transmit Timestamp: +7.984367861
09:24:27.176494 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10036, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801867.000000000 (2024-07-24T09:24:27Z)
Originator Timestamp: 3930801867.176368362 (2024-07-24T09:24:27Z)
Receive Timestamp: 3930801867.173000500 (2024-07-24T09:24:27Z)
Transmit Timestamp: 3930801867.173000500 (2024-07-24T09:24:27Z)
Originator - Receive Timestamp: -0.003367861
Originator - Transmit Timestamp: -0.003367861
09:25:34.172849 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 10788, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x34ff!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.009384, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801867.173000500 (2024-07-24T09:24:27Z)
Receive Timestamp: 3930801867.176498219 (2024-07-24T09:24:27Z)
Transmit Timestamp: 3930801934.172838781 (2024-07-24T09:25:34Z)
Originator - Receive Timestamp: +0.003497719
Originator - Transmit Timestamp: +66.999838281
09:25:34.172908 a6:4c:5e:80:2e:fb > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.16.1.1 tell 172.16.1.2, length 46
09:25:34.172917 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 172.16.1.1 is-at 60:be:b4:07:c6:17, length 28
09:25:34.172956 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10037, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801934.000000000 (2024-07-24T09:25:34Z)
Originator Timestamp: 3930801934.172838781 (2024-07-24T09:25:34Z)
Receive Timestamp: 3930801934.169000500 (2024-07-24T09:25:34Z)
Transmit Timestamp: 3930801934.169000500 (2024-07-24T09:25:34Z)
Originator - Receive Timestamp: -0.003838281
Originator - Transmit Timestamp: -0.003838281 -
-
@johnpoz
No checksum offload tick. -
-
@stephenw10
Yeah, bad explanation on my part. It's not disabled.
I even cycled it on and off a few times with a reboot to see if it made any difference.No change.
I'm a little disappointed.
-
If it's actually disabled you can see that directly using ifconfig at the CLI, for example:
[24.08-DEVELOPMENT][admin@4200.stevew.lan]/root: ifconfig -m igc1 igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=48020b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,HWSTATS,MEXTPG> capabilities=4f43fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>Where tx and rx checksums are shown in capabilities but not options. Though you see vlan checkums are still enabled.
-
Well, it looks like it's not disabled, as all previous indications have said that it isn't.
igc2: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN
options=4e020bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f43fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 60:be:b4:07:c6:16
inet 192.168.25.254 netmask 0xffffff00 broadcast 192.168.25.255
inet6 fe80::62be:b4ff:fe07:c616%igc2 prefixlen 64 scopeid 0x3
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
media 2500Base-T
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> -
Ok then the checksum failure in a pcap is expected. That isn't really a problem normally though.
-
Well, just for more fun, a couple of weeks ago, I bought an SG-1100 as I needed a backup firewall in case mine self toasts somehow.
I reset it to defaults, stuck a rough config on and as a quick test, pointed it at both my local NTP stratum one time servers and also time.apple.com.The apple timeserver responded instantly and the 1100 totally ignored the two locals.
I updated it to the latest Pfsense Plus and the same thing happened.
Today I was given a VK-T40E2 by a colleague who'd upgraded his home firewall setup.
I reset it to defaults and configured it, then pointed it at the same three timeservers as the 1100.
It instantly latched onto one of the locals while still seeing the other two.
It was still on the original version, so I updated that to the latest legacy 2.3.5 (I think). Going further will require a complete reinstall and I don't have a null modem cable to hand these days.
It still saw all the local and internet NTP servers that I cared to list.So of course, I can't wait to get it to 2.4 so that it can't see the two local stratum ones...
At least that's what I'm betting will happen. -
The apple timeserver responded instantly and the 1100 totally ignored the two locals.
It will put the other two as a 'hot spare'.
Check for yourself (example) :
Btw : look at that : NTP went IPv6 only also ...
I updated it to the latest Pfsense Plus
24.11.
the same three timeservers
Normally, you shouldn't use 3 'off something' (DNS, NTP, whatever) as it created problems.
See it like this : you have to buy a new car, but you want to think about it, so your interrogations @home and discuss it with your wive.
What would happen if you had 3 wives ?
I can tell you what will happen. You won't drive any car any soon (you couldn't afford one anyway ^^)So :
NTP : pick a 'pool'.
DNS : stick to 'root servers' (a pool actually).
etc etc.latest legacy 2.3.5
An APU ? 32 bits ??
it to 2.4 so
2.4 ?
-
- Go and read the entire thread, start to finish. Pay particular attention to what I have already posted.
- Go and read how NTP actually works.
Not SNTP.
The REAL complete NTP and learn EXACTLY WHY you put in multiple NTP sources.
Then maybe go and read why Industry Best Practice requires four.
-
And sadly, the issue still exists with 2.8
There's just no way that I can find to get pfsense back to taking notice of any NTP servers on the LAN.
-
Ok reading back a few things are unclear here.
The configures internal ntp server(s) just show as unreaching in pfSense?
A pcap on the ntp server shows requests from pfSense and responses from the server?
But the state table on the firewall only shows one way traffic?
Does a pcap on pfSense show the replies from server?
Does it do the same thing if there are no other servers configured?
-
It still works fine for me here from 2.8 and 25.03:
[2.8.0-RELEASE][admin@cedev-2.stevew.lan]/root: ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== *172.21.16.1 .GPS. 1 u 43 64 377 0.378 -0.347 2.602Now the jitter is crap on that server because it's a USB connected GPS. My real GPS device failed a while back
So in a list it doesn't get selected as the active peer. But it still polls it just fine:
[2.8.0-RELEASE][admin@t70.stevew.lan]/root: ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== -172.21.16.1 .GPS. 1 u 43 64 377 0.404 -2.999 0.919 2.pfsense.pool. .POOL. 16 p - 64 0 0.000 +0.000 0.000 +217.154.60.177 139.143.5.31 2 u 26 64 377 8.644 +1.450 0.256 *85.199.214.99 .GPS. 1 u 22 64 377 6.458 +1.507 0.204 +185.83.169.27 .GPS. 1 u 22 64 377 10.610 +1.119 0.366That also includes whether or not the server is WAN or LAN side, as long as the ACLs allow it.
-
A brief recap of the situation.
I run two LAN based GPS ntp servers.
They used to run fine before I updated to 2.7.
They would instantly get selected as peer and next best within a very short time.
Other ntp servers on the internet were also selected so as to show internet conditions within the state, country and various parts of the world.
As soon as the 2.7 update happened, the LAN based NTP servers became unreachable.
There have been at least three other people in the world who had this exact same issue.The NTP servers each remain available to each of the 72 devices currently on the home network.
My particular box does not like USB GPS devices and the serial port is an RJ45 port, which never responded to any combination of mapping the PPS, so that option is out of the window.So with your real GPS device failed, how can you be sure that your install still works with a LAN based NTP server?
