It's possible that if you are using the serialpps.sys mentioned below withthe kernel-mode PPS handling, that when Windows is updated, either through aroutine monthly update or a more major version upgrade such as 8.0 to 8.1, thatyour careful changes to the system will be reverted back to a standard Windowsdriver. The symptom of this would be that the offset and jitter becomemuch worse, and that the 'o' tally code line in the output of ntpq-pn disappears. With serialpps.sys
Government Customs Records Notifications available for Meinberg Funkuhren Gmbh & Co. See past imports to Siemens Sociedad Anonima, an importer based in Colombia. Follow future shipping activity from Meinberg Funkuhren Gmbh & Co. My thanks to Martin Burnicki of Meinberg for making this available. Using a kernel-mode driver Following a request for testing from Dave Hart, I recently (2011 Mar 17) managed to get NTP working on a 64-bit Windows-7, SP1 system using Dave's modified serial driver even though that driver is unsigned and not normally allowed on 64-bit Windows.
Another way to check may to use the Device Manager to check the driver chainfor the COM port you are using, and see whether serialpps.sys is listed in thedriver chain. In File Explorer (a.k.a Windows Explorer), right-clickComputer, and select Manage. Click on the Device Manager, expand Ports,right-click, Properties on the COM port you are using. On the Driver tab,click Driver Details, and look for serialpps.sys in the list. Download leadtek sound cards & media devices driver. Here's whatI see on one Windows-7/64 system:
The cure is simple, re-install the serialpp.sys driver.
For non-standard COM ports, or if you encounter other problems, try the loopback-ppsapi-providerinstead.
Signed kernel-mode driver available
There is no longer any need to go through the magic hoop described below toget Windows-64 to accept unsigned drivers. You can download the packagefrom here: http://people.ntp.org/burnicki/windows/serialpps-20120321-signed.zip
My thanks to Martin Burnicki of Meinberg for making this available.
Using a kernel-mode driver
Following a request for testing from Dave Hart, I recently (2011 Mar 17)managed to get NTP working on a 64-bit Windows-7, SP1 system using Dave'smodified serial driver even though that driver is unsigned and not normallyallowed on 64-bit Windows.
- Obtain the DSEO13b.exe program from: http://www.ngohq.com/home.php?page=dseo
- Obtain the 64-bit serialpps.sys driver from http://people.ntp.org/burnicki/windows/serialpps-20120321-signed.zip
- In Admin mode, run the install from the Zip archive to set the registry variables and copy the driver.
- (I found that the driver did not copy automatically, and I had to copy it by hand).
- Run DSEO13b.exe to set Windows into TESTING mode.
- Run DSEO13b.exe to add a testing signature to the driver in Windowssystem32drivers
- Install the serialpps-ppsapi-provider.dll as instructed in Dave Hart's notes.
- Set the PPSAPI_DLLS system environment variable.
- Reboot and check with the Device Manager that the new serial driver is in use.
- Check that NTP is seeing the serial and PPS data from your GPS
- Try setting the system environment variable: NTPD_USE_INTERP_DANGEROUS=1 for even better performance under Windows-7. It's not needed for Windows-8 as that OS has a more precise time API.
As I am using a Sure Electronics GPS board (whichruns at 9600 baud and has $GPGGA as the first sentence), my ntp.conf file haslines like:
How well does it work?
Here are the rather dramatic results! First, a plot of offset as seenby MRTG. Variations of up to 2ms or more with the LAN sync, variations ofsomething less than a millisecond with the kernel-mode PPS sync after 07:30, andfinally offsets around 50 microseconds with the interpolation - looking almostlike noise on the MRTG plot, after 15:30.
I did wonder about the almost sinusoidal nature of the offset with the PPSand kernel-mode driver, and closer investigation showed that the offset had aperiod of about 1.62 minutes (about 97.2 seconds), and as MRTG samples onlyevery five minutes this as aliased into a near hourly period on the MRTG plot.
The jitter as reported by NTPD in the loopstats data shows an equallydramatic improvement. Using a basic LAN sync, but with a very tightcoupling between the PC and an excellent stratum-1 clock (minpoll 3 maxpoll 3,the lowest averaged jitter was round 1.7 milliseconds (and even with user-modePPS it only dropped to 1.55 milliseconds). Adding the kernel-mode driveralong reduced the jitter to just under 1 millisecond, about the best which couldbe expected just using the ~1KHz clock which Windows- provides. However,forcing NTPD to use the interpolation - normally disabled in Vista and Windows-7- produces an even more dramatic improvement, with the averages jitter droppingto less than 40 microseconds - this is over 40 times better than the jitterusing the LAN alone, and 25 times better than the kernel-mode driver alone.
August 2009 - LAN-synced system
With the RTM (release to manufacture) version of Windows-7,the performance on the LAN-synced system was much as I reported belowwith the Release Candidate 1 version. I need to run ntpd 4.2.4p6 to getacceptable performance, as I do on a LAN-synced Windows Vista SP2 system. I did try one small test to improve the performance by reducing the LAN pollinginterval from 64s to 16s. On my first try at this, NTP 4.2.4 rejected theLAN servers as 'minpoll was greater than maxpoll'. Dave Hartsuggested defining both poll settings in the ntp.conf, so I ended up with a filelike:
This resulted in the jitter being reduced (64s poll) from anaverage of around 1.5ms with a range from 1.0 - 5.0ms, to an average value (16spoll) of just under 1ms with a much reduced variation from 1.0 to perhaps1.2ms. Offset was visibly more tightly controlled, with most offsets beingin the range +/-0.5ms, with excursions to +/-1.3ms. With the previous 64spoll the offsets had been typically within +/-1ms, with excursions to about+/-3ms. In the graphs below, the period of 16spolling is from about 08:30 to 15:00, in the centre-left portion of the graphs.
I don't have strictly comparable figures for Windows XPworking, as during April there may have been more temperaturevariation. However, the data I do have suggests an offset variation of upto +/-1.7ms, but with a much smoother variation, and a jitter trending towardsan averaged value of 40µs, with peaks up to 100-150µs.
Windows-7 RTM - August 2009
Windows XP - April 2009
August 2009 - stratum-1 serial GPS system
Immediately after a fresh install of Windows-7 (RTM) on PCStamsund - a PC running a serial GPS reference clock - I noticed that theperformance was actually much better than I had seen with WindowsXP. At least, it was until I installed a new network card (at about themiddle of the second graph below, around 11:00 UTC).
PC Stamsund under Windows XP
PC Stamsund under Windows 7
Ah, you say, poor drivers or something faulty with the newnetwork card. Possible, but not so in this case. Look carefully, andyou will see that the 11:00 reboot didn't cause the visible jitter to increase,bit only a subsequent restart of NTP at around 13:30 UTC. As you canimagine, installing the network card required powering the PC down andrebooting. Of course there were then some minor software changes to bemade which didn't require reboots, but did require a restart of NTP. As ithappened, the beta version of NTPD I was using [ntpd 4.2.5p181-o Jun 0615:21:08.65 (UTC-00:00) 2009 (1)] happens to have a 'defect' wherebythe 1KHz clock which may be used by Windows Vista and XP is not detectedcorrectly on a reboot, so the normal interpolation code was used. On therestart of NTP, the faster clock was detected, and the interpolation codedisabled. So for Windows-7 do we need a version of NTP which uses theinterpolation code even though the 1KHz clock is in use?
Here are some event log entries:
Warm restart of NTP with the high jitter:
Cold restart of NTP with the low jitter:
Following e-mail exchanges with Dave Hart, I tried setting thesystem environment variable:
and restarted NTPD (Sun Aug 23 at 17:40 UTC). Thisproduced:
Using these settings, I am now seeing a better offsetperformance, but a poorer jitter performance, from this PC under Windows-7 thanI did with Windows XP SP3. The high offsets and low jitter of the WindowsXP system have been replaced with low offsets at the expense of higher jitter.
|Windows-7 RTM||Typically within +/-25µs, occasional peaks to +/-500µs. No noticeable daily drift.||Trending to an average of under 20µs, but with occasional peaks to 500µs.|
|Windows XP SP3||Noticeable daily drift. Typically within +/-700µs.||Trending to an average of under 3µs, but with occasional peaks to 12-20µs.|
Performance with Windows-7 RTM
Performance under Windows XP SP3
On 2009 June 7 Dave Hart announced a new version of NTP, nearto a release candidate. However, it was version 4.2.5 which I have seendoes not behave very well with Windows Vista or Windows-7 RC 1. Forinterest, I changed from 'ntpd [email protected] Mar 15 21:30:20.47 (UTC)2009 (239)' to 'ntpd 4.2.5p181-o Jun 06 15:21:08.65 (UTC-00:00) 2009(1)' however, as before, this version was much worse, as can be seen fromthe Offset, relative frequency, and jitter graphs below.
I don't know whether it might be a clue, but the frequencyerror estimate seems to have gone from a relatively stable value (within about0.1ppm) with the (239) software to a value varying by about 2.5ppm (over twentytimes less stable), and the jitter has increased from about 1.4ms (average) tosomething in the order of 10ms. This needs to be addressed, as version(239) shows that this system is capable of much better performance than what isbeing proposed as the new release.
On Sunday, 2009 May 03, I installed the release candidate 1 ofWindows 7 on a test PC, Hydra. This only has an AMD 3200+, so it's notdual-core and quite low powered, but it does have 2GB of memory. In thepast, it has been a good timekeeper when running Windows XP SP3, but as with myother Vista PCs, timekeeping under Windows-7 proved to be rather adisappointment. You can see the effect quite well on the 30-minute averagegraph below, where the right-hand seven days is with Windows-7.
The configuration for this was two LAN-based stratum-1 clockswith a maxpoll of 64s, and some Internet servers with a minpoll of 1024s. The version of NTP was: ntpd [email protected] Mar 10 15:23:14.36 (UTC) 2009 (230)
As part of the tests on PC Hydra, I wanted to check theSkyStar DVB PCI card which had been used in that PC before. In my earlyWindows-7 tests the software for this card failed to install, but I discoveredon May 10 the correct sequence of installation (which required the drivers to beinstalled first using the Device Manager and then the application software, asopposed to the usual click-and-install under previous Windows). As rebootswere involved, I also took the opportunity to install some Windows-7 updates,and also to test my favourite defragmenting software - mst Defrag. Nochanges were made to NTP, but there was a very considerable difference in thetimekeeping after the change.
I left the weekly graph because although it's similar to theone above, it shows the three regimes quite clearly - XP from Friday to earlySunday on the left, Windows-7 with the USB/DVB box for the majority of the plot,and the Windows-7 PCI/DVB configuration on the right-hand side from late Sundayand Monday.
Was this change due to a Windows-7 update, due to adding thedefragmenter (sounds unlikely!), or due to changing from the USB satellite datasource and DVBWorld software to the SkyStar PCI card and TechniSatSoftware? The tests continue, and suggestions are welcome! One measurementwhich we can make is the DPC Latency, with a freetest program.
|Configuration||PC With no cards plugged in|
|With internal SkyStar PCI card||With external DVBWorld USB box|
|No antenna connected|
|Receiver software alone|
|Receiver and TelliCast software|
|All software running normally|
|All software after several hours|
Some further casual tests on the effect of maxpoll on jitterwere carried out during March 2009. They showed that with a large maxpoll,and hence a longer time between corrections, the observed variation in offsetwas greater.
In the graph above, the PC started off with maxpoll=6, so thatNTP would make its calculations and corrections every 64 seconds. Despitetheir being mostly a flat line, there are still some excursions of up to 60ms,although these are often short-lived. At time A,the maxpoll was reset to 10, typical of using the PC with just Internetservers. At time B the maxpoll wasexperimentally set to just 4 (16 seconds), and the minpoll also had to be set to4 to override its normal minimum value of 6 (64s). Because of the loadthis would create, just one local PC was selected as a server. Althoughthe jitter is much reduced, there are still some transients visible. Youcan download a Zip file with all the loopstats for PC Gemini on request, and there is a program to display them here.
On March 10 and 11, I noticed that there was a glitch intimekeeping at about 0200 on both days. This seems to coincide with anautomated system restore, download of Windows Defender definitions, and a run ofWindows Defender. That whole sequence seems to start two minutes early -at 0158. Whilst I haven't been aware of this before, I would like to knowwhich of those three elements may be the cause of the problem, so I shiftedWindows Defender to do the same operation at 0400 instead of 0200 and we'll seeif the glitch shifts. This was with Gemini syncing to one stratum 1 andtwo stratum 2 servers just on my LAN, with a maxpoll of 6 (64 seconds).
I happened to notice that there were regular timekeeping stepsat 0200 every morning, when Pixie was running with maxpoll=6 using just LANservers. Looking at the Application and System event logs suggested thatWindows Defender was being run at that time, following a definitionsdownload. As that PC isn't used a lot for interactive use, I decided tochange the daily schedule for the program to being just Wednesdays. Notthat I have anything against Wednesdays, of course! I hadn't seen thisbehaviour before, though. Thispage says more about the program.
With the new version of the Windows NTP port by Dave Hart, wehave come much closer to Vista offering an acceptable performance for NTP, andwith his suggestions and expertise we have been able to achieve some very goodresults. My thanks to Dave for his time and ideas.
In late January 2009 I received an interesting e-mail fromDave Hart, offering an improved NTPfor Vista. We worked together testing the changes which Dave had made,and the result as of mid-February 2009 was as shown below.
The amplitude of the oscillations has been reduced, and thepeaks seem smoother and more rounded to me. Quite why the oscillationsstill exist is a mystery to be, as are two other effects seen on Vista - theability to run for a period with no oscillations (such as in the Puffin graphbelow), and the apparent ability for a system to be more stable after restartingNTP. Yes, I did check that the drift file exists, has a reasonable value,and is being updated.
We have continued to experiment with this, and by using /just/the local stratum 1 GPS/PPS server, and with a reduced, non-standard poll interval of 64s(maxpoll 6), the jitter could be brought down to a very low value, with atypical offset of -0.6ms. With maxpoll 8, the typical offset was -7.5mswith transient offset peaks up to +0.+2.5ms with a couple up to +17.18ms (i.e.an excursion of 7.5.10ms and 22.214.171.124ms). With maxpoll=9, the jittershows peaks up to 20-30ms, many of which are positive.
As of March 06, I know that Dave Hart is still playing withsome algorithm tweaks, but I am uncertain whether these will affect the Vistaperformance shown above, of whether they are just for use with a localserial-input reference clock.
Clarification from Dave Hart: The vista interpolation disabling has hardly blinked since I firstwrote it. Operationally, there hasn't been a difference with any of my released versions' Vista-specific code. However, there was a period of as much as a week (I forget exactly) where my binariesfailed to try to raise the priority unless -N was given on the command line, and that (running at normal instead of high orreal-time) definitely dragged down performance on Vista as well as other Windows releases.
6 January 2009
I tried an experiment with Cecilia's new PC, adding NTP towhat is a fairly empty Windows Vista Home Premium SP1 system. I wanted tosee whether the poor performance seen in September 2008 was typical of Vista, orjust typical of the system I was then using. Initially, the results werepromising, as, after the initial transient (as NTP had never been run on that PCbefore, times B to C on the graphs below), the offset seemed to be just a noisier-than-expected straightline. See the PC Puffin on the graph below. However, after about 17hours, time D, there was a negative spike in the offset, followed by a period of greaterinstability of around 50ms peak-to-peak.
I therefore tried the main Vista system with none of theapplications I normally run started up (Dexatek DVB RX, TelliCast, MSG DataManager, AVHRR Manager, Metop Manager, SBS-1 BaseStation, GAS populate, PlanePlotter and Flight Display). I rebooted at about 07:00 - time A on the graphsbelow. However, the oscillations started after aboutone hour, so I restarted all the normal apps at 08:48. The resulting instability on the PC Gemini is perhaps greater than on PC Puffin,being over 100ms peak to peak.
Questions which remain:
- Why the PC Puffin ran well for the first 17 hours, and then developed instability?
- Why was the jitter during those first 17 hours a lot greater than on my Windows XP or Windows 2000 systems?
- What causes the gross variations seen on PC Gemini and, to a lesser extent, on PC Puffin?
- Why are the variations of a different amplitude on PC Puffin to those on PC Gemini?
For interest, I tried logging the user out of PC Puffin at14:27, to see if that makes any difference or not. Unfortunately, with theparticular wireless network USB stick I have for that PC, logging out stops thenetwork connection, so I now have an hour of zero results! The network wasreconnected at about 15:30, and instability similar to the other Vista PCcontinued.
Meinberg Funkuhren Driver Updater
December 2008 - a small discovery about Vista
This link http://17slon.com/blogs/gabr/2007_11_01_archive.htmlmentions that the timing resolution on Vista is improved to 1 millisecond, andthere is a link to a note from Microsoft (http://support.microsoft.com/kb/274323)which mentions that the 'Performance counter value may unexpectedly leapforward' with certain chipsets. I don't know whether either of thesewould affect NTP.
Following a suggestion from Mike Hughes, I tried setting CPUaffinity for the NTPD process on the Vista PC, Gemini, to just a single CPU. I happened to choose CPU 1. It seems to have helped Mike get bettertimekeeping. Mike may force CPU on the ntpd.exe with the ImgeCfg.exeprogram, if the improvement continues. However, it didn't seem to help mea lot, so I tried stopping ntp service, marking the image uni-processor only,and CPU-affinity 0x1 (i.e. CPU 0), and restarted. See: windowsitpro.comarticle and andbrinkster.com article about using imagecfg.exe.
On looking back, it may have helped reduce the noise as seenon the 30-minute average graph, looking at the data from mid-Thursday onwards itperhaps has less variation than that from before. However, looking at themonthly 2-hour average graph, the change is less visible, so it may be a randomeffect after all. The gross variations seen on a 5-minute averagecontinue, however.
Well, I thought that the NTP problem was resolved afterinstalling Vista SP1, because the initial trace after the first reboot with fullSP1 showed a stable period. The installation of SP1 completed at 0600 UTCyesterday - where the green line is shown:
However, as you can see the oscillations developed once again,and the timekeeping looks no better than before. The drift file doesappear to be updating correctly. There is nothing unusual in the NTP eventlog, just the usual start-up messages about syncing to a couple of nodes. The W32time service is disabled, and it does not show as having a PID inthe Task Manager.
Any clues about why NTP might be behaving like this wouldtruly be appreciated!
I recently installed the Windowsport of NTP on to Windows Vista. The initial results were quitedisappointing, with wild swings even though the average value was OK. Idid not have time to investigate this, but there is no power reduction, speedswitching enabled in the BIOS, nor any spread-spectrum functions. I leftthe PC from Friday last week and, as you can see by the Weekly graph below, itseems that sometime around 2200 UTC last night it suddenly snapped intosynchronisation, although there is some evident that is it starting to oscillateonce again. There appears to have been a gradual correction over Wednesdayto Saturday towards a smaller offset. The same software has been runningon that PC continuously. Current data is no longer available. You can see the behaviour of the same hardware (except for a changed hard disk)under Windows XP in the period up to near the end of week 45, when the WindowsXP HD failed and I decided to make that PC my Windows Vista test-bed.
Any ideas or suggestions as to what is going on, and as to howNTP could be configured better for that system? It's using UK poolservers, plus a local GPS NTP source which isgood to within 20 microseconds.
NTP Events logged
To help resolve this problem, here is a plot of recentperformance (ending 09:30 Dec 01) and a section of the Event Log, filtered forNTP events, since the system was rebooted in its final configuration (at about14:40 on 2007 Nov 29). There are no obvious logged events which wouldcause NTP problems, although the time step made by NTP does cause logging ofboth a Kernel and a Security event.
I'm not sure what to draw from this. I may have restartedNTP at 1448 on Nov 29, to be sure of a known starting condition, and it thensynced to the correct server. There are three times when the time wasstepped by NTP: 18:45 Nov 29, 01:01 Nov 30, and 07:03 Nov 30. All threeresets are positive, and followed by syncing to an Internet server or two,before syncing to the preferred server. The drift file is being updated(judging by both revision date and the contents).
I've tried this both with and without the -M parameter (to enablethe multimedia timer) and it doesn't seem to make a difference. TheW32Time service is stopped. It doesn't seem to be possible to boot thisAMD dual-core system with just one core.
There was a peculiar spell on December 01 which might providea clue - it was almost as if NTP was oscillating between two values and tryingto correct the median?
An aside - how NTP overcomes the limitation of Windows timer ticks
I extracted the following message from comp.protocols.time.ntpnewsgroup, as it sheds light why NTP normally works so well on Windows. Ihope Martin will give his permission for these paragraphs to stay. Martin Burnickiwrote:
'In most cases the OS system clock is counting timer ticks to keep the time.When an application reads the system time then a good OS returns thecurrent time with a higher resolution than the timer tick interval, i.e. itreturns a time based on the current timer tick count plus the current valueof the time counter register which gives an idea whether time is at thebeginning or the end of a timer tick interval, and thus is responsible forthe resolution of the system clock, depending on the clock rate of thatcounter.
'The details depend on which time counter register is used for this purpose.If there's just a single register, e.g. in the chipset, then it's prettyeasy to deal with it.
'If a CPU register is used to determine the time between 2 timer ticks andyou have a SMP or multicore system then you must take care that either thetime stamp is always taken from the same CPU/core, or the timers in allCPUs/cores are synchronized to each other so that it doesn't matter whichCPU is being read.
'The nanokernel code mentioned by Dave [Mills in theAlpha UNIX implementation] takes care about registers in multipleCPUs, but as its name implies it is some code which needs to run in thekernel to be able to access the registers and provide a proper API to theapplication.
'If the Microsoft developers would pick up Dave's code and integrate it into theWindows kernel then Windows would probably be a good time keeper as well.
However, Windows does not evaluate the current counter value at all, so thesystem time is just derived from the number of timer ticks, increases onlywhen a timer tick occurs, and thus the resolution of the clock is limitedto the timer tick interval (Vista seems to be slightly different, though).
'Since the Windows kernel source is not available, we can not apply Dave'scode to increase the resolution of the Windows system clock since we areunable to modify the timekeeping code in the Windows kernel.
'That's why NTP uses a hack to try to increase the resolution from userspace. Of course this is not as reliable and exact as a properimplementation in the kernel could be. However, it's better than just usethe resolution provided by Windows. Also, as-is, the interpolated systemtime is not available to other applications.
'The clock interpolation code for Windows can only use standard API calls,e.g. the PerformanceCounter API. How this API behaves in detail depends onhow it is implemented in the HAL. It can either use the CPU's TSCregisters, or use any other register which may be available with certainchipsets. This code does not try to synchronize the TSCs on different CPUs,it just tries to run always on the same CPU in order to avoid glitches dueto different TSC values on different CPUs.
Martin Burnicki, Meinberg Funkuhren,Bad Pyrmont,Germany'