Redirect all NTP traffic to internal IP

demux

Trying to redirect all NTP traffic destined to external servers to an internal ntp time source.

Did exactly as described in the post below (redirect dns and changed dns to ntp):

Re: Redirect ALL NTP requests to local NTP sever
https://www.netgate.com/docs/pfsense/dns/redirecting-all-dns-requests-to-pfsense.html

Solution is similar to this: https://www.linuxincluded.com/ntp-server-ip-blacklisted-nat-redirection-ftw/

I see the packets being redirected to the internal ntp server and answered there. I see them travelling back and arrive at the querying host. Fine.
But ntpdate responds ("-d") with "receive: server not found".
Same with Windows w32tm.
I understand this. ntpdate asked some servers (eg pool.ntp.org) but got the answer from a different one (internal) and refuses to take that. If I were an ntp client I would do the same ;-)

Did anyone manage to get around this?
I believe I need to fake the sender's internal address to be the external address which was queried originally. But I have no clue how to do that.

Background: We need this because we have some devices that use their own hardcoded ntp servers (in China) and these servers are sometimes simply wrong or sometimes not reachable. On the other hand we have a highly precise ntp server locally that we want to use.

Thank you very much.

johnpoz

I run my own local ntp, and I do redirect some outside ntp fqdn that clients use to this... Let me check with windows client and see what is happening. To be honest I never followed through to validate the iot devices that ask for external ntp were setting correct time.

But then again none of iot devices have complained about no ntp and have not noticed time being off, etc..

I will try and duplicate your windows client test - since I have a windows 10 client I can play with ;)

BBL..

edit: Ok back not having any issues. So I turned off the ntp client I normally run.. And enabled w32tm again had to do a /register, etc.

So set it to sync to a source that in the list time.nist.gov, I don't override that one.

Then changed it and did an update to one I do override to my local 192.168.3.32, and it shows it sync's just fine

All I am doing here is a simple local dns override pointing the fqdn to my local ntp server. You sure your client can actually get there? You could try simple /monitor test.

Here I did a /stripchart, and then added packetinfo to see exactly what is happening

C:\WINDOWS\system32>w32tm /stripchart /computer:time.nist.com /dataonly
Tracking time.nist.com [192.168.3.32:123].
The current time is 2/12/2019 10:44:22 AM.
10:44:22, -00.0075295s
10:44:24, -00.0075167s
10:44:26, -00.0075633s
10:44:28, -00.0075856s
10:44:30, -00.0076041s
10:44:32, -00.0076541s
^C
C:\WINDOWS\system32>w32tm /stripchart /computer:time.nist.com /dataonly /packetinfo
Tracking time.nist.com [192.168.3.32:123].
The current time is 2/12/2019 10:44:38 AM.
10:44:38, -00.0077075s
[NTP Packet]
Leap Indicator: 0(no warning)
Version Number: 1
Mode: 4 (Server)
Stratum: 1 (primary reference - syncd by radio clock)
Poll Interval: 0 (unspecified)
Precision: -20 (953.674ns per tick)
Root Delay: 0x0000.0000 (unspecified)
Root Dispersion: 0x0000.02B8 (0.0106201s)
ReferenceId: 0x50505300 (source name:  "PPS")
Reference Timestamp: 0xE00D766D47D55BA9 (152713 16:44:29.2805993s - 2/12/2019 10:44:29 AM)
Originate Timestamp: 0xE00D7676E134BD76 (152713 16:44:38.8797110s - 2/12/2019 10:44:38 AM)
Receive Timestamp: 0xE00D7676DF4EDED0 (152713 16:44:38.8722972s - 2/12/2019 10:44:38 AM)
Transmit Timestamp: 0xE00D7676DF5327D5 (152713 16:44:38.8723626s - 2/12/2019 10:44:38 AM)
[non-NTP Packet]
Destination Timestamp: Roundtrip Delay: 587400 (+00.0005874s)
Local Clock Offset: -7707500 (-00.0077075s)

10:44:40, -00.0077406s
[NTP Packet]
Leap Indicator: 0(no warning)
Version Number: 1
Mode: 4 (Server)
Stratum: 1 (primary reference - syncd by radio clock)
Poll Interval: 0 (unspecified)
Precision: -20 (953.674ns per tick)
Root Delay: 0x0000.0000 (unspecified)
Root Dispersion: 0x0000.02BA (0.0106506s)
ReferenceId: 0x50505300 (source name:  "PPS")
Reference Timestamp: 0xE00D766D47D55BA9 (152713 16:44:29.2805993s - 2/12/2019 10:44:29 AM)
Originate Timestamp: 0xE00D7678E54C0D1E (152713 16:44:40.8956917s - 2/12/2019 10:44:40 AM)
Receive Timestamp: 0xE00D7678E365C67A (152713 16:44:40.8882717s - 2/12/2019 10:44:40 AM)
Transmit Timestamp: 0xE00D7678E36A3E6D (152713 16:44:40.8883399s - 2/12/2019 10:44:40 AM)
[non-NTP Packet]
Destination Timestamp: Roundtrip Delay: 641300 (+00.0006413s)
Local Clock Offset: -7740600 (-00.0077406s)

10:44:42, -00.0077852s
[NTP Packet]
Leap Indicator: 0(no warning)
Version Number: 1
Mode: 4 (Server)
Stratum: 1 (primary reference - syncd by radio clock)
Poll Interval: 0 (unspecified)
Precision: -20 (953.674ns per tick)
Root Delay: 0x0000.0000 (unspecified)
Root Dispersion: 0x0000.02BC (0.0106812s)
ReferenceId: 0x50505300 (source name:  "PPS")
Reference Timestamp: 0xE00D766D47D55BA9 (152713 16:44:29.2805993s - 2/12/2019 10:44:29 AM)
Originate Timestamp: 0xE00D767AE95DA420 (152713 16:44:42.9115851s - 2/12/2019 10:44:42 AM)
Receive Timestamp: 0xE00D767AE776846F (152713 16:44:42.9041522s - 2/12/2019 10:44:42 AM)
Transmit Timestamp: 0xE00D767AE782BA28 (152713 16:44:42.9043385s - 2/12/2019 10:44:42 AM)
[non-NTP Packet]
Destination Timestamp: Roundtrip Delay: 704700 (+00.0007047s)
Local Clock Offset: -7785200 (-00.0077852s)

10:44:44, -00.0078187s
[NTP Packet]
Leap Indicator: 0(no warning)
Version Number: 1
Mode: 4 (Server)
Stratum: 1 (primary reference - syncd by radio clock)
Poll Interval: 0 (unspecified)
Precision: -20 (953.674ns per tick)
Root Delay: 0x0000.0000 (unspecified)
Root Dispersion: 0x0000.02BE (0.0107117s)
ReferenceId: 0x50505300 (source name:  "PPS")
Reference Timestamp: 0xE00D766D47D55BA9 (152713 16:44:29.2805993s - 2/12/2019 10:44:29 AM)
Originate Timestamp: 0xE00D767CED3528D6 (152713 16:44:44.9265924s - 2/12/2019 10:44:44 AM)
Receive Timestamp: 0xE00D767CEB4CC3CA (152713 16:44:44.9191401s - 2/12/2019 10:44:44 AM)
Transmit Timestamp: 0xE00D767CEB518FAF (152713 16:44:44.9192133s - 2/12/2019 10:44:44 AM)
[non-NTP Packet]
Destination Timestamp: Roundtrip Delay: 732800 (+00.0007328s)
Local Clock Offset: -7818700 (-00.0078187s)

10:44:46, -00.0077907s
[NTP Packet]
Leap Indicator: 0(no warning)
Version Number: 1
Mode: 4 (Server)
Stratum: 1 (primary reference - syncd by radio clock)
Poll Interval: 0 (unspecified)
Precision: -20 (953.674ns per tick)
Root Delay: 0x0000.0000 (unspecified)
Root Dispersion: 0x0000.02C0 (0.0107422s)
ReferenceId: 0x50505300 (source name:  "PPS")
Reference Timestamp: 0xE00D766D47D55BA9 (152713 16:44:29.2805993s - 2/12/2019 10:44:29 AM)
Originate Timestamp: 0xE00D767EF4354CE8 (152713 16:44:46.9539383s - 2/12/2019 10:44:46 AM)
Receive Timestamp: 0xE00D767EF24B1B80 (152713 16:44:46.9464585s - 2/12/2019 10:44:46 AM)
Transmit Timestamp: 0xE00D767EF250AFF5 (152713 16:44:46.9465437s - 2/12/2019 10:44:46 AM)
[non-NTP Packet]
Destination Timestamp: Roundtrip Delay: 621800 (+00.0006218s)
Local Clock Offset: -7790700 (-00.0077907s)

10:44:48, -00.0078519s
[NTP Packet]
Leap Indicator: 0(no warning)
Version Number: 1
Mode: 4 (Server)
Stratum: 1 (primary reference - syncd by radio clock)
Poll Interval: 0 (unspecified)
Precision: -20 (953.674ns per tick)
Root Delay: 0x0000.0000 (unspecified)
Root Dispersion: 0x0000.02C2 (0.0107727s)
ReferenceId: 0x50505300 (source name:  "PPS")
Reference Timestamp: 0xE00D766D47D55BA9 (152713 16:44:29.2805993s - 2/12/2019 10:44:29 AM)
Originate Timestamp: 0xE00D7680F8548C49 (152713 16:44:48.9700401s - 2/12/2019 10:44:48 AM)
Receive Timestamp: 0xE00D7680F669EE43 (152713 16:44:48.9625539s - 2/12/2019 10:44:48 AM)
Transmit Timestamp: 0xE00D7680F66E7FFF (152713 16:44:48.9626236s - 2/12/2019 10:44:48 AM)
[non-NTP Packet]
Destination Timestamp: Roundtrip Delay: 731500 (+00.0007315s)
Local Clock Offset: -7851900 (-00.0078519s)

10:44:50, -00.0078960s
[NTP Packet]
Leap Indicator: 0(no warning)
Version Number: 1
Mode: 4 (Server)
Stratum: 1 (primary reference - syncd by radio clock)
Poll Interval: 0 (unspecified)
Precision: -20 (953.674ns per tick)
Root Delay: 0x0000.0000 (unspecified)
Root Dispersion: 0x0000.02C3 (0.0107880s)
ReferenceId: 0x50505300 (source name:  "PPS")
Reference Timestamp: 0xE00D766D47D55BA9 (152713 16:44:29.2805993s - 2/12/2019 10:44:29 AM)
Originate Timestamp: 0xE00D7682FECA57A7 (152713 16:44:50.9952750s - 2/12/2019 10:44:50 AM)
Receive Timestamp: 0xE00D7682FCD9BDF7 (152713 16:44:50.9876975s - 2/12/2019 10:44:50 AM)
Transmit Timestamp: 0xE00D7682FCDF577D (152713 16:44:50.9877829s - 2/12/2019 10:44:50 AM)
[non-NTP Packet]
Destination Timestamp: Roundtrip Delay: 637000 (+00.0006370s)
Local Clock Offset: -7896000 (-00.0078960s)

^C
C:\WINDOWS\system32>

So sure looks like working to me with simple dns override, atleast with the built in windows time service.. Now back to actual ntp client ;)

edit: Well I am back to official ntp client.. But was thinking maybe its your windows firewall.. I don't run the firewall when locally connected.. My local network is secure ;) And controlled by me.. The only thing on the network or has access to my PC is devices controlled and managed by me, etc. So I have no use for host firewalls..

But its possible that could be your problem - since you state via sniff you saw the return.. Guess I could try enable local firewall and test again... But just disabled local windows time in favor of my official ntp client.. So before I do that try just disable your local firewall or security software if running something 3rd party - for simple test.

demux

I see the same in Linux and Windows. The Linux machine does not have any firewall or something else that could change things.
The difference seems to be that you do it via DNS redirect.
We want to do it via NAT/port redirect to be as generic as possible. Some of the devices query NTP servers via IP address.
It works. I can see it. But the source address of the returning packet/answer does not match the address the client asked for. It seems strange to me that there are some references to detailed descriptions on how to do it, but they don't work.
You deliver the "wrong" IP address from the beginning - the DNS' answer. Your clients don't have any reason to be in doubt. They accept what they get.
For us, it is different. The client gets the real IP address from DNS or asks via IP address and gets the NTP answer from a completely different address.

johnpoz

So you want to just intercept all NTP queries no matter if to fqdn or IP and just send them to your ntp...

Let me test that...

Ok that works just fine as well - client thinks it heard back from who it asked..

On my lan interface (which is 192.168.9/24 network) I setup a port forward for my ntp on 192.168.3.32

Did a time update from windows, and clearly it got answer back from my ntp server... See the response of .0006 seconds.. No freaking way that went out to the internet ;)

sniff from client doing ntp update

Ah!!!! I know what your prob doing that is wrong - your trying to do a port forward back out the same interface right???

So you have lan port redirecting to something on LAN... In that case then yeah your client would get the answer from ntp direct vs from pfsense interface.

Put your NTP on different vlan/network than your clients are on if you want to do port forward interception of the ntp... That would be my guess because it works just fine when I do a port forward from my lan to my dmz... Client thinks answer came back from who they asked IP.

hmmm: If you can not move your ntp to their own segment.. Then redirect it to ntp services running on pfsense.. I had to restart ntp on pfsense because I did not have it listening on loopback... Give it a minute and will validate it works... That is how your suppose to do the dns redirect as well.

edit: Ok back - yeah it works if you redirect the ntp to pfsense.. I just redirect it to pfsense on IP on dmz 192.168.3.253... And bam it answers when client ask for some other ntp server.

Here is sniff on pfsense.. No freaking way outside ntp could answer that fast ;)

14:28:56.294607 IP 192.168.3.31.40120 > 129.6.15.30.123: UDP, length 48
14:28:56.294744 IP 129.6.15.30.123 > 192.168.3.31.40120: UDP, length 48
14:28:58.292753 IP 192.168.3.31.40120 > 129.6.15.30.123: UDP, length 48
14:28:58.292863 IP 129.6.15.30.123 > 192.168.3.31.40120: UDP, length 48
14:29:00.292756 IP 192.168.3.31.40120 > 129.6.15.30.123: UDP, length 48
14:29:00.292911 IP 129.6.15.30.123 > 192.168.3.31.40120: UDP, length 48
14:29:02.292749 IP 192.168.3.31.40120 > 129.6.15.30.123: UDP, length 48
14:29:02.292858 IP 129.6.15.30.123 > 192.168.3.31.40120: UDP, length 48

here is my port forward.

So it works if you port forward to ntp server on different network locally than your client, and or if you redirect to ntp running on pfsense itself... Which in turn could be syncing from your local ntp on your lan, etc.

demux

Right! It comes from the LAN and gets redirected to the LAN.
I will try tomorrow what you suggested. Thank you!

demux

After many tries I wanted to restore a backup. Now I have a problem. See other thread. I have to fix them first...

tman222

Just wanted to chime in and confirm that I got this work as well after reading this discussion. All I did was add port forwarding (NAT Redirection) rules to forward NTP to the pfSense interface (for a given network segment) if the NTP request was bound for any other server (IP) that wasn't the pfSense interface (i.e. essentially like the DNS redirect instructions, but using NTP instead DNS and the interface IP instead of localhost). Did a quick test in a Linux VM and sure enough even though I specified an outside NTP server, traffic was directed to the internal NTP server running on the pfSense interface.

This is also neat way to help get all devices on the network perfectly in sync, i.e. especially mobile devices (phones, tablets, IoT) where it may not be easy (or even impossible) to separately configure the NTP (time) server.

johnpoz

I am not exactly sure why it doesn't work if you try and redirect to loopback for ntp.. Have not spent any time troubleshooting this.. Its just as easy to direct the port forward to the interface IP vs loopback, etc.

Guess we could dig into it a bit deeper, developers or admins might have insight to this little oddness. If you set ntp to listen on all interfaces, it clearly binds to loopback

root     ntpd       44443 36 udp4   127.0.0.1:123         *:*

So why would it be that dns works but ntp does not? Not sure to be honest.

But yeah if you just portforward to the IP of the interface on that network/vlan it works great.

demux

Got it.
I selected the LAN interface only in ntp setting and it listens on localhost, too.
In NAT I redirected all ntp traffic to the localhost IP. That's it.

There is one problem left that I cannot find the reason for.
We have two ntp servers. They are used internally and externally and are running pretty fine.
From pfsense I have a constant reach of 377 to external servers, but our internal servers make problems that we do not have from other hosts: They are very, very often unreachable for a long time. And later on they recover magically, but this may take hours.
On the ntp servers I see that they sent answer packets but these packets do never arrive at the pfsense machine. One of them is also part of ntp pool and makes no problems. Both are havily used inside wothout problems. On the pfsense machine it looks like this:

Did someone else see this before? It is seen from pfsense only.

Thanks!

Now it looks like this:

johnpoz

Nope never seen that.. So you sniff on pfsense and see it sending query to your internal ntp, and you see the return but reach doesn't go up?

Look at the NTP packet you get back, if its not valid for some reason ntp would not count it as a reach..

demux

@johnpoz exactly like this.

a few minutes later...

funny, if it were not my job to correct this ;-)

johnpoz

they it only takes like 8 good queries to reach 377, if your polling every 64s out of the gate than yeah only a few minutes.

Maybe your ntp server looses its sync and its answers are not valid?

demux

All switches, machines, devices whatever use the combination of these two local ntp servers. And in the meantime while I see these broken reach bits, the other machines/devices talk to the ntp servers without any problems. Even ntppool goes up again. I believe there is something (deep) inside pfsense.
Until yesterday I had version 2.4.3 running, do you know what ntpd version was used with 2.4.3? Because I cannot remember having seen this before with 2.4.3 and we already use this pfsense ntp server setup for about 2 years. The only difference now is the new pfsense version and NAT, and when I disable NAT things don't get better.

demux

In a first step I let run ntpdate for about 12h every 5 secs from pfsense to the ntp server. I recorded a tcpdump on both ends and I had not one missed ntp packet. Meanwhile ntpd had the usual problems. Really strange. In the logs of doing the ntpq -p against the pfsense machine (every 7 seconds) I saw that not only my local servers seen from pfsense do experience timeouts, the external servers also have timeouts but less often. Strange...

johnpoz

@demux said in Redirect all NTP traffic to internal IP:

I had not one missed ntp packet.

But what did the data say, you can't just look for a reply but what was in the reply.

And what do you mean by nat - there should be no nat to anything local

demux

@johnpoz I meant NAT / port forward.
I did not dig into the packets or assemble them. This is why I used ntpdate and ntpq -p.
I assume ntpdate and ntpq will tell me if there is something unusual inside the packets.
If ntpdate reports increasing times by 5 secs if called every 5 secs, then I assume it's ok.
If I see packets travelling from left to right and back and a resuling answer by ntpdate, I assume it is ok. And I had run ntpq -p against pfsense to see what the pfsense ntp server currently does.
So I see an uninterrupted data flow and data that makes sense to ntpdate (and to me).
Do you know what version of ntpd was included in version 2.4.3?

johnpoz

Again what would port forwarding have to do with pfsense talking to your local NTP server? There is no nat from pfsense IP to some device on one of pfsense networks.

No I do not... Look in the release notes.. 2.4.4p2 is current
ntpq 4.2.8p12@1.3728-o Wed Sep 5 02:13:06 UTC 2018 (1)

this version came out well after 2.4.3 so yeah it was a slightly older version.

demux

I did a packet trace. What I see is - I believe - strange:
11:22:12.477255 IP (tos 0xb8, ttl 64, id 11111, offset 0, flags [none], proto UDP (17), length 76)
10.200.100.pfSense.123 > 10.200.100.ntp_server.123: [udp sum ok] NTPv4, length 48
Client, Leap indicator: (0), Stratum 2 (secondary reference), poll 5 (32s), precision -22
Root Delay: 0.025558, Root dispersion: 0.500915, Reference-ID: 130.149.17.21
Reference Timestamp: 3759301270.497451600 (2019/02/16 11:21:10)
Originator Timestamp: 3759301236.436497210 (2019/02/16 11:20:36)
Receive Timestamp: 3759301236.437431054 (2019/02/16 11:20:36)
Transmit Timestamp: 3759301332.477197054 (2019/02/16 11:22:12)
Originator - Receive Timestamp: +0.000933844
Originator - Transmit Timestamp: +96.040699843
11:22:12.477621 IP (tos 0xb8, ttl 64, id 20383, offset 0, flags [DF], proto UDP (17), length 76)
10.200.100.ntp_server.123 > 10.200.100.pfSense.123: [udp sum ok] NTPv4, length 48
Server, Leap indicator: (0), Stratum 1 (primary reference), poll 5 (32s), precision -23
Root Delay: 0.000000, Root dispersion: 0.550033, Reference-ID: DCFb
Reference Timestamp: 3759301329.370467552 (2019/02/16 11:22:09)
Originator Timestamp: 3759301332.477197054 (2019/02/16 11:22:12)
Receive Timestamp: 3759301332.475587226 (2019/02/16 11:22:12)
Transmit Timestamp: 3759301332.475714633 (2019/02/16 11:22:12)
Originator - Receive Timestamp: -0.001609827
Originator - Transmit Timestamp: -0.001482420

As far as I understood, the first packet leaving the client should have (nearly) Originator=Receive=Transmit. The server uses sets Originator=Originator client, Receive=receive time at the server, Transmit=transmit time at the server.
What does the client (pfSense ntp server) do between Receive and Transmit times?
This is whay I see the correct and fast flow of packets; it seems to hang around in pfSense's ntp server before being sent out.
???

4o4rh

Thanks fellas, i got this to work with the above info.
Had to forward to the interface, instead of 127.0.0.1.

Works if set to time.nist.gov option in windows 10, but not time.windows.com
Any ideas why

johnpoz

What do you mean doesn't work is set to time.windows? If your doing redirection shouldn't matter what the client asks for.. As long as it would resolve.