Incoming NTP packets not reaching destination



  • Hi. Running 2.1 AMD64 everything is virtualized with ESX 4.1

    I'm trying to sync time in various Windows Server 2008 R2, all these servers are behind a pfSense.
    I have 2 WANs and 1 LAN. Some servers use WAN1, others WAN2.
    No matter what gateway WAN a server uses, it's unable to sync time against the most common publics NTP servers.

    I used Packet Capture and found out that the traffic from my servers to the NPT servers is going out ok (they are not beign blocked). So my guest is that there is some "rule" (or lack off) or something on pfSense that is not allowing the public NTP server to reply back to the server. I also checked the firewall logs but there no blocked UDP packages.

    I created a RULE on "LAN" for all IPv4 UDP traffic any port any destitation any gateway
    I also created a RULE on "WAN" for IPv4 UDP traffic any port any destitation any gateway

    There is no other rule involving UDP, except for a IPv4 all proto, source LAN, all port, all dest, all gateway.

    On the windows Server, when I try to sync, I get a "An error ocurred… The peer is unreachable"
    Needles to say I can ping the NTP Server from the windows server.

    It is worth mentioning that the included pfSense NTP server is always on "Unreach/Pending" state, but I already found on the forums that I'm not the onlly one facing that issue. I'm not using the built in NTP, just letting everyone know.

    Any help would be appreciated.
    Thanks.-



  • In the packet capture, you saw the traffic leaving, but nothing coming back? That sounds like your ISP is blocking NTP traffic, which could be a knee jerk reaction to the NTP DDoS attacks recently. If you don't see the reply coming back in the capture on WAN, it's not getting back to you, there isn't anything for the firewall to pass back in.



  • That will make total sense… since this was working until a couple of days ago!!!
    Thanks!


  • Netgate Administrator

    GPS FTW!  :D

    https://forum.pfsense.org/index.php/topic,67189.msg398272.html#msg398272

    Blocking NTP entirely seems a bit need-jerk indeed.

    Steve



  • The 2 WANs are WORLDNET (primary) and LIBERTY (backup). pfSense IP is 192.168.1.1
    Now that I know that WORLDNET has decided to block all incoming NTP traffic because of the recent DDoS attack, my only hope is to set up the pfSense internal NTP server and have all the Windows servers syncronize against 192.168.1.1.

    LIBERTY is allowing incoming NTP traffic, so it should be a simply matter, but it doesn't work.

    In this image the NTP service is configured only on LIBERTY interface (configuring it on WORLDNET would not work since WORLDNET is blocking incoming NTP traffic)

    And if I check the status everything is working, so great

    However, since I need pfSense NTP serving on the local LAN, I also need t include LAN on the NTP configuration, like this:

    But then the NTP server seems to get lost and don't know how to get to the public NTPs. This problem has already been described here https://forum.pfsense.org/index.php?topic=58796.0

    I already set up a rule on LAN to use LIBERTY for all UDP traffic, just to make sure the NTP service is not going trough WORLDNET

    In short, my question is… how can I have a working pfSense NTP service (like in the 2nd image) that also serves NTP request on the LAN?
    Thanks.-


  • Rebel Alliance Global Moderator

    To be clear you did your packet sniff on pfsense WAN and saw the packets leave,l but nothing come back.

    Have you tried different ntp server to query?  Its possible they are just offline or stop answering you because you queried them too much ;)  Or there is a network issue between you and them, etc.

    That lan rule does what?  To me its sending all UDP out your lib gatway - so how would clients even talk to your pfsense lan interface on udp?  Pfsense looks at inbound traffic to its lan interface and says – oh your udp, here go talk to the lib gateway.  that is not right if you want to be able to talk to local stuff on udp.  if your going to great rules with specific gateway, you need to have rules above it to allow traffic to your normally routed network.  Setting a gateway over rides all pfsense routing and just sends traffic to the gateway.

    Put a rule above that that allows udp to your local networks.  Then sure you can send other traffic not destined for your local network to go out the lib gateway vs your world one, etc. But I thought you said pfsense was your ntp server, why do other clients on your network need to use ntp to the public internet - shouldn't they just be using your pfsense ntpd?

    And how would ntp get confused - you don't have a gateway off your lan do you?  Only gateway should be out wan interfaces so why would ntp use your lan as a path to ntp servers on the public internet?  Where it send the traffic - there is no gateway on the lan - is there?


  • Netgate Administrator

    The NTP server setup you have screenshots of is NOT the NTP client. You should not have it serving incoming NTP requests from your liberty WAN. Select only LAN.

    Steve



  • @stephenw10 If I select only LAN the NTP status screen goes all "unreach/pending" as described on the first post. There also a link from a person (with much more knowledge than I on pfSense) explaining why that happens.


  • Netgate Administrator

    Hmm, I see. Odd behaviour.
    Can you switch the system default gateway to Liberty? (assuming it's currently Worldnet) Does that impact your other services too much?

    Steve



  • @sthepenw10 that will certainly help, but I cannot do that… the main ISP is WORLDNET, much bigger bandwidth... LIBERTY is only for backup / emergency uses.



  • Here's is with the UDP rule deleted, NTP service only using LAN

    And, as always, NTP service is going trough WORLDNET to query the public NTPs:


  • Netgate Administrator

    I'm not sure that linked thread still applies. It was written at the time of 2.0.2 so many fixes have gone in since then.
    What happens if you unselect all three interfaces so that it's listening on all three?
    I assume you haven't opened port 123 on WAN by the way which would then expose your NTP server to the world, which would be bad!

    Your UDP rule on LAN will not do anything unless your LAN servers are trying to connect to external NTP servers directly.

    Steve



  • @sthepen If I select no interface on the NTP service all goes "unreach/pending"
    If I select the 3, all also goes "unreach/pending"

    There's no NAT rule UDP Port 123. It's closed


  • Netgate Administrator

    Are those NTP servers you have set all public or are some of them at your ISP?

    You could try setting a static route to one of them via the Liberty gateway.

    I meant a firewall rule not a NAT rule. There would be no need for NAT as the NTP service is running on the WAN interfaces themselves.

    Steve



  • @johnpoz Sorry I didnt explain the situation in great detail, your suggested rule to allow UDP in the LAN wont do nothing.

    The Window servers I want to sync and the pfSense are in the same "physical" ESX… meaning, it's a virtual network and pfSense has no control over it.
    There reason there's a LAN interface on the pfSense is because ESX1 is connected to ESX2... that is the "LAN" pfSense controls.

    All the servers and inconming ISP connections are physically on the ESX1, so there should be no need at all to touch any LAN rule. The reason I created the rule in the first place was to make sure all NTP traffic to public servers was going trough LIBERTY and not WORLDNET.



  • Your UDP rule on LAN will not do anything unless your LAN servers are trying to connect to external NTP servers directly.

    and actually that rule will push all requests out the Liberty gateway. Even a LAN client trying to get NTP from pfSense LAN address will have the request pushed out Liberty gateway, which won't be able to route it anywhere useful.
    Put destination !LANaddress in that rule.



  • @sthepen here's the NTP list I'm using:
    0.pfsense.pool.ntp.org
    1.pfsense.pool.ntp.org
    2.pfsense.pool.ntp.org
    time-a.nist.gov
    time-b.nist.gov
    0.north-america.pool.ntp.org
    1.north-america.pool.ntp.org
    2.north-america.pool.ntp.org

    A static route is a great suggestion. I set 1 up to time-a.nist.gov (129.6.15.28):

    And now the NTP service, withc is only configured on LAN and would normally give all "unreach/pending" is showing:

    However, the static route seems to be working only by IP… is there a way to configure with FQDN?


  • Rebel Alliance Global Moderator

    and actually that rule will push all requests out the Liberty gateway

    Agreed and not only ntp but ANY udp traffic – like DNS for example..  That is really bad rule to put in place to be honest and not required.


  • Netgate Administrator

    No, I don't think so but you could just add more routes and use fixed NTP servers. You probably don't need 8 anyway.

    Take note of Phils excellent point above. That rule might be bypassing the pfSense NTP server altogether, though I believe the negate rules should allow access to it if you haven't disabled that.

    I'm confused about your ESX setup. You surely need a LAN interface for your servers behind pfSense which are on the same VM host?  :-\

    Steve



  • @johnpoz @shepen Yes I deleted the rule right away after reading John comment, thanks :)

    So I ended up using a fixed IP list on NTP server (utcnist2.colorado.edu time-a.nist.gov time-b.nist.gov time-c.nist.gov nist1.symmetricom.com)

    Thank you very much guys for you time and patience :)


  • Netgate Administrator

    Is it still necessary to have the NTP server listen on your WAN interfaces? I would certainly have it configured to only listen on LAN if not.

    Steve



  • @stephen Is only configured on the LAN port, thanks for asking!