PfSense is killing time!
-
Hi All
I am having huge issues with getting NTP functional on my network and I discovered after days of fidgeting that I can't get an NTP sync from the outside world on any device, Linux, Windows or otherwise; only syncs within my network are functioning. I tried enabling the built-in NTP server in pfSense and syncing from it without success and I also tried disabling the NTP service and syncing to the outside world (ntp.org, nist,gov, etc) and nothing is working. There are no corresponding entries in the firewall logs indicating that packets are getting blocked and I uninstalled all unnecessary packages on pfSense and triple-checked every setting. I also tried bypassing pfSense and jacking straight into the cable modem which worked for an ntpdate sync so the problem is definitely somewhere in pfSense. Any help would be greatly appreciated, I am out of ideas here!
-
I run a ntp server behind pfsense that is part of ntp.pool and lots of people sync off my box, etc. And don't have any issues maintaining sync with public ntp servers.
Is your lan rules default that allow all outbound? NTP runs on 123 UDP. You mention cable modem, are we sure its not a gateway.. So your pfsense gets an actual public IP address on its wan?
Can you post up your lan rules? Also I would suggest you do a sniff/packet capture on pfsense on both the lan and wan interfaces looking for udp 123 and see what you see.
-
Hi johnpoz
The cable modem is running as a gateway and the WAN port has a public IP address. Here is the outbound rules:
IPv4 * LAN net * * * * none
IPv4 * 192.168.1.0/24 * * * * noneI tested this on a similar setup with another client (pfsense + cable modem) and they don't have this problem. I think I may have an idea what the problem is; the router is set up with two LANs (using IP alias, not second adapter) and I had to mess with NAT to get it working. Here is my NAT setup:
WAN 192.168.0.0/24 * * 500 WAN address * YES Auto created rule for ISAKMP - LAN to WAN
WAN 192.168.0.0/24 * * * WAN address * NO Auto created rule for LAN to WAN
WAN 127.0.0.0/8 * * * WAN address 1024:65535 NO Auto created rule for localhost to WAN
WAN 192.168.1.0/24 * * * WAN address * NOI tried running pcap on UDP/123 and it showed no traffic whatsoever.
-
Cable modem as a gateway? You mean its bridging, gateway would indicate its got an IP that you use a gateway and would suggest its natting.
Well if your not seeing any udp 123 traffic - it would make sense that your ntp server never syncs time ;)
I would suggest you listen on the LAN port for traffic. Your ntp server should increase its poll time to a max of 1024 seconds. But when it first starts the poll time should be very short and you should see lots of traffic.
So your running just a vip on the lan, not a vlan? So your pretty much just running 2 different IP address space networks over the same physical wire, etc.
Is your ntp server or even pfsense having a hard time resolving fqdn of your ntp servers your pointing to? Have you tried using an IP vs fqdn?
If you not seeing traffic on pfsense from your ntp server, then how is it an issue with pfsense? Atleast for testing, I would make sure your ntp server is on your actual physical lan network not the vip. And I would then start your ntp server, can you post your ntp.conf file? And sniff for udp 123 from your ntp servers source IP.. You should be seeing quite a bit of queries trying to sync time.
If you not does this box your running ntp on resolve the fqdn your using for your time sources? Can you ping them? etc..
On a default setup of pfsense you should not have to do anything special to get ntp to work. All ports are allowed outbound and as long as dns is working you should not have any issues.
edit: So you mention when you take pfsense out of the path and use ntpdate it works? Now if I recall correctly when you do a ntpdate the source port would be random above 1024, but normal ntp source port would be source port 123 to dest 123.. If I recall correctly, is it possible your ISP is blocking that traffic when its inbound coming from 123?
You really need to sniff on pfsense and validate that your traffic goes out!!
-
You really need to sniff on pfsense and validate that your traffic goes out!!
I'll second this. I had problems with ntp until a thread on the forum suggested this and I found that ntp traffic was going out with an ip on a vlan not setup for outbound nat.
-
exactly!! So I had a ubuntu vm that I had not quite setup yet, did not even have ntp installed. So this is right after a apt-get install ntp
I fired up ntpq, took a look – as you can see its talking to whatever default ntp servers is setup in the package (ntp.pool.org servers). You can see that reach value is increasing, you can see on pfsense that packets are hitting the lan, and also see the responses, etc.
Now you can download the capture and look at the details if you want. But if your saying your not seeing any udp 123 traffic on pfsense lan that your ntp server is connected too - then you have something wrong other than pfsense blocking or doing something wrong with the traffic. If you see the packets hit your pfsense lan, then sniff on your wan you should see your queries go out - do you see responses? you will notice in the sniffs that source and dest ports are both 123.. Now if I do a ntpdate.. See the last screenshot where its a random source port.
-
Sorry, I may have muddied the water with my last response. The cable modem is a bridge, it isn't performing NAT under normal circumstances. The virtual IP setup isn't for this equipment, it's for something unrelated; the reason I mentioned it is because I had to change outbound NAT from automatic to manual and add an entry to get the second subnet working. I don't know if that is related or not, I just mentioned it because it is one of the only things that is different on this setup than the other pfSense installs I have (which don't have this problem).
I ran a package capture on the LAN set to only show traffic from one of the servers but didn't limit it to port/proto/etc, simultaneously ran tshark on the server with a filter set to UDP/123 and ran an ntpdate against time-c.nist.gov. I'll attach the outputs from both but in essence, the server shows sending the traffic but the router doesn't record anything on it's end other than the active ssh sessions I was using. All other traffic from this server passes to and from the Internet without problems and every other device on the network has this same specific problem with NTP. When I bypass the pfSense box and use the cable modem's built-in routing fuction, this problem doesn't show itself so I am pretty sure that whatever is going on is with pfSense. Is there any way that I can get the router to tell me what it is doing with that traffic other than pcap?
-
And do you have firewalls on these servers that could be blocking the traffic?
If pfsense does not see the traffic on its lan - how and the hell would it be expected to route it, etc.
You need to figure out why pfsense is not seeing this traffic, and you will solve your problem.
You didn't give enough info in your routercap, but in your servercap I see dest mac of 00:25:90:c6:1b:d7 for the packet, is this pfsense mac address?
Your server in this example is vm I see, so possible your blocking UDP in your vm setup?
Again - if pfsense is NOT seeing the traffic, its not pfsense. unless you were filtering your sniff to tcp. A capture would show you every single packet that interface saw.. Be it pfsense ignored it, blocked it, etc.. If your not seeing the packets at pfsense - pfsense has NOTHING to do with your issue.
-
Hi johnpoz
This particular server did have UFW enabled but it does the same thing with UFW disabled. This problem duplicates on all devices on the network, not just a couple of the servers. Here's the odd thing: I ran nmap -p 123 -sU -P0 192.168.0.1 from this particular server it comes back with
Starting Nmap 6.40 ( http://nmap.org ) at 2013-12-05 09:56 EST
Nmap scan report for 192.168.0.1
Host is up (0.00051s latency).
PORT STATE SERVICE
123/udp open ntp
MAC Address: 00:25:90:C6:1B:D7 (Super Micro Computer)Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds
On the firewall end a simultaneous package capture shows the following:
09:48:56.612866 IP 192.168.0.6.41950 > 192.168.0.1.123: UDP, length 48
09:48:56.613012 IP 192.168.0.1.123 > 192.168.0.6.41950: UDP, length 48Please excuse the timestamps not matching up, I can't get time synced without working NTP :-)
So UDP traffic on 123 is being sent and received without issue and when I do this same test against a public time server such as time-c.nist.gov, the results are the same. However, ntpdate commands against either of these hosts fails and a pcap on pfsense doesn't record any UDP/123 traffic. I suppose it could be something going on with the switch; I'll try doing a factory reset on in and see if that fixes this. Is there any settings/configuration on the router end that you know of which could cause this behavior?
-
"a pcap on pfsense doesn't record any UDP/123 traffic"
Again if pfsense is NOT seeing the traffic, how do you expect it to route it?
Is there some firewall on the server side that would block and application?
This is really simple network troubleshooting – if you DONT see the traffic on pfsense, pfsense can not do anything with the traffic.
As to settings on the router - this has nothing to do with pfsense until you can see packets on the lan side of pfsense that are your ntp queries..
so when you do a nmap to some IP outside pfsense - do you see this traffic with a sniff on pfsense?
-
I understand what you are saying but if this is the case than why can nmap connect to UDP/123 and also through the router on UDP/123 to the outside world?
-
That is the question now isnt it.. It quite simple - if pfsense does not see the packets, pfsense can not do anything to the packets. Why pfsense is not seeing the packets is the question. Firewall on client side?
These are vms right - what driver are you using? I know there was an issue, not sure if still is with vmxnet3 driver dropping small udp packets.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2019944
UDP packets are dropped from Linux systems using the VMXNET3 network adapter (2019944)What switches are between the client and pfsense - just had weird issue at work where a cisco switch stop forwarding vlan traffic, only in one direction. Unless you put a svi in that vlan, freakiest thing seen in a really long time. Took a bit to figure this one out.
https://tools.cisco.com/bugsearch/bug/CSCth74527
Symptom:
Due to a rare timing issue with diagnostics, a register used for data forwarding is getting corrupted. As a result, traffic between devices in the same vlan are getting black-holed.Conditions:
This issue has been observed in Catalyst 6500 with VS-S720-10G Sup engines
running 12.2(33)SXH2a, 12.2(33)SXI5 and 12.2(33)SXI9 releases.
AND
No SVI present for the vlan(s) in discussion.Was very strange – you could sniff the vlan and watch the packet come in and get forward, then you saw the reply but it did not get forwarded. You put a svi on the thing and bam started working. Take the svi away and stop working again... Now a reboot should of cleared the issue - but production and would of been real pain to reboot that switch. So currently we just created svi's on the 50+ vlans we had going through that switch until such time that we can update its os to never version. This thing ran for years without any issues, just all of sudden freaked out. Only reason we found the svi work around was putting it on the switch to troubleshoot more, these vlans were all layer 2 and switch had no reason for a svi, etc.
All of the traffic you have shown being seen on pfsense has come from random source port.. Maybe you got something on your network blocking source udp traffic from 123?
But what I can tell you for fact - is if pfsense is NOT seeing the traffic, pfsense can not DO anything with the traffic ;)
What is weirder is you say if you run pfsense ntp server it does not sync either?? Where are you pointing it for source? Is it having issues with name resolution? How exactly is your networked wired? Is pfsense VM as well? What you doing vlan tagging, any sort of QoS? A drawing of your network might help us discuss possible problems or where to sniff to follow the packets, etc.
-
Setup here is very straightforward, virtual IP notwithstanding; it's a /24 network with no QoS or VLANs, one static IPSEC tunnel and an IPSEC roadwarrior setup. The server that I posted the results from is a VM but this problem duplicates on all devices, physical or virtual, Linux or Windows. I am suspicious of the main switch; I'll give it a factory reset once business closes down today and see if that does anything. NTP doesn't work at all on the pfsense box, either as a client or a server. I tried pointing it various ntp.org servers along with a couple different NIST hosts with no luck. NTP updates do work over our internet connection if you bypass pfsense though. The pfsense router itself is a physical device and I am also going to try a factory reset on it and temporarily disable the VIP functionality in case there is something with the routing or VIP that caused this or a mistake I made when I set it up. Looks like I'm in for the long haul on this one! :-)
-
So question for you - you say ntp of pfsense does not work which should have nothing to do with not seeing traffic to your lan.
So again I am going to ask, because seem unwilling or unable to answer such a simple question. Are you using fqdn or IP addresses? Look on the WAN of your pfsense – do you see the ntp query go out?
This should take only a couple of minutes test.
So here is listing of public ntp servers.. http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers
I am going to use this one to test
http://support.ntp.org/bin/view/Servers/PublicTimeServer000919it has host names "access via time.steadfast.net or chi.time.steadfast.net" which both resolve to multiple IPs -- so if issue with name resolution validate you can resolve to IP and or try using the IP directly.. You should be able to use a nonsense IP you pull out of the air to validate pfsense is sending the query.
C:>dig time.steadfast.net +short
69.162.170.4
208.100.4.54
208.100.4.52
67.202.100.50
208.100.0.228
69.162.170.5C:>dig chi.time.steadfast.net +short
208.100.4.54
208.100.4.52
208.100.0.228
67.202.100.50Im going to use 208.100.0.228 directly for this simple test since it resolves to that, and that is the IP they also list directly.
So
So let me change pfsense to use this see attached, I then stop the ntp service.. I then open another browser window and setup packet capture. I then go to my other browser window and start the ntp service.. Do you see the IP you set for your server, does REACH ever go up? See the poll, hit refresh this should be going up and as you hit the number 64, it should go back to start counting again and your reach should start counting up.. If not counting up or even if even if it is -- after a couple of pools have gone through go back to your capture stop it do you see your queries go out, do you see returns? Download it and look with say wireshark.
This is really basic network troubleshooting 101 stuff here to be honest..
-
One of the first things I tried before posting to this forum was to try using IP addresses rather than FQDN for both internal and public NTP servers and it didn't make a difference. At any rate it's irrelevant; I did a factory reset on the main switch and it fixed all of the NTP problems.
