Runtime went backwards???


  • Banned

    An ybody else seeing this in 2.0.2??

    Running in VmWare and both are using the same NTP servers.



  • This looks a symptom of having ESXi and pfSense each running NTP while also having ESXi trying to adjust the pfSense VM's clock via VM Tools.

    As a matter of interest, what version of ESXi and what version of Open-VM-Tools do you have installed?

    Did you upgrade an existing 2.0.1 VM to 2.0.2?  I haven't been able to get the "stable" VM tools (8.7.0.3046) to work with a new 2.0.2 install on ESXi 5.1  Marcelloc's 8.8.1 version will work but throws up an error that was mentioned in another thread http://forum.pfsense.org/index.php/topic,48060.0.html.  That doesn't seem to cause any problems but I haven't run that test VM for very long.


  • Banned

    I am running the VM-tools package in 2.0.2 no modifications.

    ESXi is 4.1U3 Enterprise Plus if it matters.



  • Virtual Machine Specific info:

    Time issues that manifest in all sorts of different error messages are common on just about all virtual machine hosting environments due to how most OS's track time via processor ticks.  As a side effect of virtualization, a virtual machine simply doesn't get every processor tick (otherwise you'd have full cores or more devoted to each VM.)  So, you get time drift, which is handled usually by some sort of application/tool/service that runs within the virtual machine, in VMWare virtual machines you run the VMWare tools (or some equivalent, often open source, version for less supported OS's), that periodically pulls time back to the reality reported by the VM host (which, itself may be set and time maintained via NTP.)

    My theory on what's happening to you:

    When you have 2 or more time sources competing and re-setting time like that, you can easily get a situation where one may over-shoot the other as they'll rarely be in perfect sync (or some other latency may introduce some variations.)  I can't tell you what is best for your situation, but, assuming you're not otherwise relying on pfSense for some kind of time sync on your network, I would probably set ESXi via NTP and let the VM-tools keep the pfSense OS in sync, not use NTP in pfSense itself.  VMWare will set the Virtual BIOS time via it's local host time (which may be set via NTP) so that your pfSense OS will get time from BIOS on boot, then VMWare tools should keep the OS up to date from there.



  • I agree.  They are competing with each other.  Either:

    • don't run NTP and allow VM Tools to set the time on the VM or

    • run NTP on pfSense and stop VM Tools from setting the VM's time



  • @biggsy:

    I agree.  They are competing with each other.  Either:

    • don't run NTP and allow VM Tools to set the time on the VM or

    • run NTP on pfSense and stop VM Tools from setting the VM's time

    Yeah, I thought that's what you were getting at earlier, but I think it'd be more noticeable if the ESXi host wasn't getting time from NTP, you'd have a larger discrepancy, like he's seeing.  If I'm reading it right, he's seeing 10 or more minutes on some of the processes, which could also lead me to believe it could also be a time zone calculation problem (not an hour, since it's process run time, right?)  If VMWare tools is setting based on one time zone and NTP on another (or differing application of Daylight Savings Time.)

    But, yes, any way about it, it certainly looks like 2 competing time setting mechanisms.

    I did some quick looking around and the main consensus is to pick just one, of course, but there didn't seem to be a whole lot of overwhelming bias towards one or another time setting mechanism.  The down side of VM Tools is that it depends on the host to be right and the time service of a VM host isn't always perfectly stable (I've seen whole groups of hosts lose the NTP service before, but the time of the host itself is usually pretty stable even if NTP isn't working.)  On the other hand, a down side of using NTP on a VM is some OS's may fail updating time if it's too far off, but that usually only happens in rare situations where you've suspended a VM for a while and it didn't sync when restarting, or when the same problem strikes after reverting to a snapshot with snapshotted memory, but again, that is pretty rare since time should be re-synced during the restart/revert process even if VMWare tools isn't set to sync OS time.

    But, I'm probably digressing from the main point, don't run more than 1.



  • Some good detail in here: http://www.vmware.com/pdf/vmware_timekeeping.pdf

    … I thought that's what you were getting at earlier ...

    Yeah, but I did say it in a round about sort of way  :)

    If VMWare tools is setting based on one time zone and NTP on another…

    I see what you're getting at but NTP only deals with UTC.


  • Banned

    I am running the same timeserver dk.pool.ntp.org on both.

    Can I just delete the specified server in Pfsense? I cannot configure vm-tools in pfsense from the GUI.



  • @biggsy:

    If VMWare tools is setting based on one time zone and NTP on another…

    I see what you're getting at but NTP only deals with UTC.

    I wasn't sure how BSD deals with time zones in respect to UTC, if the OS "lives" on UTC and translates it out to the user environments or the system clock is set to local time with the offset from UTC as defined in time zones.  Same with VMWare tools in BSD.  If they're setting according to a different time zone, or one is applying a Daylight Savings Time offset and the other isn't, I could see it gaining and losing an hour often.  I also don't know what kinds of intervals NTP and the VMWare tools are checking/setting time at in BSD.

    I come from the Windows world in most respects, especially regarding VM's on ESX(i).  In Windows the VMWare Tools should be setting time based on the local Guest OS Time Zone setting, same as if you're polling a Time Server.  It can get slightly messy if it's on a domain since the Domain should be the authoritative time source.

    Of course, I'm tangenting, again.  I do that.  It just seems that there's quite a difference between the 2 time sources and/or how they're being applied.


  • Banned

    Yes, but if you use your AD based on VMtools in your guest OS, thats the right way to do it.

    Let VM handle the NTP and not your guest OS.

    Thats why I think it would be better if it could be turned of in PFsense if it detects vmtools package….



  • Can I just delete the specified server in Pfsense? I cannot configure vm-tools in pfsense from the GUI.

    I don't think there is a GUI way to "disable" NTP in pfSense.  http://forum.pfsense.org/index.php/topic,57041.0.html.  You can manually stop it but I guess that wouldn't survive a reboot.

    Configure VM Tools through the vShpere client: Right-click VM > Edit Settings > Options tab  (I think that's the same on Esxi 4.1)


  • Banned

    I know….but it DOESNT synchronize time in Vsphere GUI....

    So I dont get it. Thats why I wondered if there was anything in the tools package that did it without telling anybody??


  • Rebel Alliance Developer Netgate

    You can try to change the timecounter (search around for kern.timecounter.choice and kern.timecounter.hardware) to see if that helps.

    Certain newer versions of vmware can have the clock flake out if it's using HPET.



  • @jimp:

    You can try to change the timecounter (search around for kern.timecounter.choice and kern.timecounter.hardware) to see if that helps.

    Certain newer versions of vmware can have the clock flake out if it's using HPET.

    Thanks for the tip, I had the same issue with my pfsense vm, I tried the above commands:

    Current settings

    sysctl kern.timecounter.hardware

    Available options

    sysctl kern.timecounter.choice

    New settings based on available options

    sysctl -w kern.timecounter.hardware=ACPI-safe

    With the ACPI-safe option, the errors went away, the previous HPET option caused the problem for me, as noted.

    I understand the same can be achieved by switching off HPET in the bios, if its available.


  • Rebel Alliance Developer Netgate

    No need to run the sysctl manually, you can add that as a tunable under System > Advanced on the Tunables tab, it will be applied automatically once you add it, and at boot time.



  • Thanks


Log in to reply