ntp only connecting to some time servers

stephenw10

The ISAKMP rules are for clients behind pfSense using IPSec. Some servers require a fixed source port (usually also 500) so that rule does not change the source port for port 500 connections.

Let's see your routing table. Though if it's looping there I'd expect something very obviously wrong. And also other traffic to fail.

Steve

nback

@stephenw10
ipv4 and ipv6 routing tables. Hopefully I've obscured all sensitive IP addresses
netstat-ipv4.txt
netstat-ipv6.txt

stephenw10

Nothing glaringly obvious there. Either it is looping or it's sending with a stupidly low TTL value.

Try a traceroute to one of those failing IPs from pfSense, you would see a loop there.

Check the state table or a packet capture, is it actually sending from one of the WANs?

On a side note you're lucky to have different gateways with two PPP connections to the same ISP. I use plusnet and have not been so lucky.

Steve

nback

@stephenw10

On pfsense

traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 85.199.214.102
traceroute: findsaddr: failed to connect to peer for src addr selection
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 212.71.248.69
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 138.68.183.179
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 212.71.248.69
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 178.62.250.107
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 217.114.59.66
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 138.68.183.179
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 212.71.248.69
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 178.62.250.107
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 217.114.59.66
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 138.68.183.179
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 212.71.248.69
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 178.62.250.107
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 217.114.59.66
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 138.68.183.179
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 212.71.248.69
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 178.62.250.107
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 217.114.59.66
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 138.68.183.179
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 212.71.248.69
traceroute: findsaddr: failed to connect to peer for src addr selection.
[2.4.0-RELEASE][root@pfsense.local1]/root: traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 178.62.250.107
traceroute: findsaddr: failed to connect to peer for src addr selection.

I cleared the state table and tried again
traceroute 0.gentoo.pool.ntp.org
traceroute: Warning: 0.gentoo.pool.ntp.org has multiple addresses; using 134.0.16.1
traceroute: findsaddr: failed to connect to peer for src addr selection.

IP 134.0.16.1 does not appear in the state table. So I guess it's not sending.

From box on lan

traceroute to 0.gentoo.pool.ntp.org (129.250.35.250), 30 hops max, 60 byte packets
1 253.core.plus.net (195.166.130.253) 7.825 ms 7.805 ms 7.842 ms
2 84.93.253.107 (84.93.253.107) 8.455 ms 84.93.253.111 (84.93.253.111) 8.797 ms 8.741 ms
3 195.99.125.142 (195.99.125.142) 9.191 ms core1-BE1.southbank.ukcore.bt.net (195.99.125.130) 9.185 ms 195.99.125.138 (195.99.125.138) 8.722 ms
4 109.159.252.140 (109.159.252.140) 9.222 ms core4-hu0-8-0-0.faraday.ukcore.bt.net (195.99.127.52) 9.415 ms 9.953 ms
5 62.6.201.145 (62.6.201.145) 9.901 ms 9.440 ms 9.385 ms
6 166-49-209-194.eu.bt.net (166.49.209.194) 9.877 ms 8.754 ms 8.736 ms
7 * ae-1.r02.londen03.uk.bb.gin.ntt.net (129.250.4.148) 9.805 ms 9.530 ms
8 x.ns.gin.ntt.net (129.250.35.250) 8.553 ms !X 8.787 ms !X ae-1.r02.londen03.uk.bb.gin.ntt.net (129.250.4.148) 9.777 ms
traceroute 0.gentoo.pool.ntp.org
traceroute to 0.gentoo.pool.ntp.org (85.199.214.102), 30 hops max, 60 byte packets
1 253.core.plus.net (195.166.130.253) 7.847 ms 7.366 ms 7.360 ms
2 84.93.253.107 (84.93.253.107) 8.630 ms 84.93.253.111 (84.93.253.111) 8.301 ms 84.93.253.107 (84.93.253.107) 9.054 ms
3 195.99.125.142 (195.99.125.142) 8.607 ms core1-BE1.southbank.ukcore.bt.net (195.99.125.130) 9.040 ms 195.99.125.142 (195.99.125.142) 9.081 ms
4 peer7-et-3-1-6.telehouse.ukcore.bt.net (109.159.252.234) 9.279 ms peer7-et-3-1-5.telehouse.ukcore.bt.net (109.159.252.236) 9.119 ms peer7-et-4-1-5.telehouse.ukcore.bt.net (194.72.16.142) 9.983 ms
5 * * *
6 edge2.lon.uk.as61337.net (91.230.243.221) 9.282 ms 7.909 ms 7.897 ms
7 core1.blet.uk.as61337.net (91.230.243.226) 10.365 ms 9.857 ms 9.921 ms
8 core1.np.uk.as61337.net (91.230.243.42) 10.382 ms 10.434 ms 10.375 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

I hadn't noticed the different gateways. I think they change on occasionally.

Norman

nback

@nback said in ntp only connecting to some time servers:
If I clear the state table and ping 134.0.16.1 from pfsense I see:
Nothing passed to WAN (or any other interface!)

If I do the same from the box on the lan I see:

stephenw10

Hmm, what interfaces do you have set for ntp? Try just unselecting any interfaces so it uses all interfaces.

Steve

nback

@stephenw10
Was WAN,WAN2 and LAN. Originally just WAN amd WAN2 before I added the box on lan.
Unselecting all interface made no difference to the issue. Just seeing all interfaces listened to in the NTP log. Nothing additional in the NTP status.

nback

@nback
I meant to add that I then tried a reboot of pfsense but no change.

stephenw10

Hmm, why is it using localhost as source there....

Did you actually choose localhost to ping from? It should still work but does it work if you choose WAN as source?

Steve

nback

This post is deleted!

nback

ping-from wan-2.png

However from LAN or default does not work.

nback

@nback
Maybe this is a clue. All of the dns servers setup in pfsense ping OK using default source address in the Diagnosics Ping web interface.
127.0.0.1
74.82.42.42
208.67.220.220
2001:470:20::2
2620:0:ccc::2
8.8.8.8
8.8.4.4
2001:4860:4860::8888
2001:4860:4860::8844
2620:0:ccd::2
208.67.222.222
212.159.6.9
212.159.6.10

stephenw10

Ok, just reviewed everything here and this is the problem, I should have spotted it the first time:

Routing tables

Internet:
Destination        Gateway            Flags       Use    Mtu      Netif Expire
default            yyy-yyy-yyy-yyy.plus.com UGS  2425824   1492        lo0
dns.google         253.core.plus.net  UGHS     171106   1492     pppoe1
dns.google         252.core.plus.net  UGHS     171108   1492     pppoe0

Your default route is via lo0.

There are gateways for each of the PPPoE connections but there is an additional gateway there, whatever yyy-yyy-yyy-yyy.plus.com is. That is not expected. The default route should be one of the other two gateways shown there.

You don't appear to have any static routes via internal gateways so make sure you don't have any gateways defined on internal interface, such as LAN. The outbound NAT rules apply you have 5 interfaces with gateways set

Do you have an extra gateway shown in System > Routing? Something left over from the previous config perhaps?

What is the default v4 gateway set to there? Set it to one of the ppp gateways or a failover group of the two if it's still set to automatic.

Steve

nback

@stephenw10

You've found it!

It's a hangover in System/Routing/Gateways for multiwan. Originally set up in pfsense 1.2.x
Each pfsense upgrade seemed to adjust these setting so that multiwan still worked.
I've uncheck the 'Default Gateway' and now I have lots of new ntp servers and pings now works as expected.
I now need to review my System/Routing/Gateways and NTP settings.
Thanks for all your help.

Norman

stephenw10

Nice. I should have spotted that the first time I read the routing table. I saw it was a plusnet address but missed it was on lo0. Todays lesson....

Steve

nback

@stephenw10
OK, that seems to have sorted out ipv4 but I notice in the NTP log
'Soliciting pool server 2a01:7e00::f03c:91ff:fead:69a'
but no response.

So from the box on the lan:
ping -c3 -6 2a01:7e00::f03c:91ff:fead:69a
PING 2a01:7e00::f03c:91ff:fead:69a(2a01:7e00::f03c:91ff:fead:69a) 56 data bytes
64 bytes from 2a01:7e00::f03c:91ff:fead:69a: icmp_seq=1 ttl=58 time=11.3 ms
64 bytes from 2a01:7e00::f03c:91ff:fead:69a: icmp_seq=2 ttl=58 time=10.9 ms
64 bytes from 2a01:7e00::f03c:91ff:fead:69a: icmp_seq=3 ttl=58 time=10.4 ms

--- 2a01:7e00::f03c:91ff:fead:69a ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 10.397/10.849/11.274/0.378 ms

From pfsense
[2.4.0-RELEASE][root@pfsense.local1]/root: ping6 -c 3 2a01:7e00::f03c:91ff:fead:69a
ping6: UDP connect: No route to host

2.4.0-RELEASE][root@pfsense.local1]/root: ping6 -c 3 -S 2001:yyyy:yyyy:yyyy::2 2a01:7e00::f03c:91ff:fead:69a
PING6(56=40+8+8 bytes) 2001:yyyy:yyyy:yyyy::2 --> 2a01:7e00::f03c:91ff:fead:69a
ping6: sendmsg: No route to host
ping6: wrote 2a01:7e00::f03c:91ff:fead:69a 16 chars, ret=-1
ping6: sendmsg: No route to host
ping6: wrote 2a01:7e00::f03c:91ff:fead:69a 16 chars, ret=-1
ping6: sendmsg: No route to host
ping6: wrote 2a01:7e00::f03c:91ff:fead:69a 16 chars, ret=-1

--- 2a01:7e00::f03c:91ff:fead:69a ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

Where 2001:yyyy:yyyy:yyyy::2 is the pfsense interface address of an HE tunnel.

This seems like a similar problem but I can't see it, or should I raise a separate question?

nback

@nback
Fixed it! Set a default gateway for ipv6.

JKnott

@nback said in ntp only connecting to some time servers:

Fixed it! Set a default gateway for ipv6.

You shouldn't have to. That should happen automagically, through router advertisements.