We've seen that now and then but haven't found a 100% fix for it. It seems to happen when rc.newwanip resets NTPD after an IP address change or interface up or down event.
maybe this will help:
-U number, --updateinterval=number interval in seconds between scans for new or dropped interfaces. This option takes an integer number as its argument. Give the time in seconds between two scans for new or dropped interfaces. For systems with routing socket support the scans will be performed shortly after the interface change has been detected by the system. Use 0 to disable scanning. 60 seconds is the minimum time between scans.