Living with the Operating System
Background
The GPS clock delivers GMT/UTC time strings to the DSB card, which in
turn
distributes the signals to the DOR cards in each of two DOM Hubs.
The readgps log file entry
GPS 167:04:48:05 TQUAL(' '
exclnt.,<1us) DOR 00001cdae4b166ea dt=20000000 ticks
corresponds to the system time entry
Wed Jun 15 21:48:07 PDT 2005
so the GMT time is roughly 6 hours, 59 minutes, 58 seconds ahead of
local system time.
Actually, the system clock is GMT, adjusted to
local time for the convenience of the user.
The system
periodically corrects its clock to agree with a time server via the
NTPD daemon.
The Nature of the Change to cron Job Scheduling
System cron jobs are essential to the health of any *nix
system.
For more information on cron, see the man page, and the info pages.
The typical cron control file,
/etc/crontab, looks like this:
01 * * * * root
run-parts
/etc/cron.hourly
02 4 * * * root
run-parts
/etc/cron.daily
22 4 * * 0 root
run-parts
/etc/cron.weekly
42 4 1 * * root
run-parts
/etc/cron.monthly
In each /etc/cron.* directory there are scripts to run. Each will
be run at the noted nice level
This file tells us, for instance, that daily jobs run at 04:02 local
time every day of the year.
Any daily job that hogs all the CPU for
over 12 seconds, without allowing any other processs to have a
time
slice, causes the DOR card time string record FIFO to fill to
overflowing, causing the loss of data.
To overcome this problem, /etc/crontab has been modified, June 14,
09:12 in local time, julian date 165:16:12 in utc, to this:
01 * * * * root
nice -n 19
run-parts /etc/cron.hourly
02 4 * * * root
nice -n 19
run-parts /etc/cron.daily
22 4 * * 0 root
nice -n 19
run-parts /etc/cron.weekly
42 4 1 * * root
nice -n 19
run-parts /etc/cron.monthly
The priority reduction mechanism, nice, will, hopefully, permit
user processes doing data acquisition to keep up with the data.
Anomalies from log files spanning the change
For domhub51
Note that domhub51 was concurrently being used for debugging of
DOM main boards.
Furthermore, when severe communications
problems are encountered, the processing of data is apparently
disrupted.
DOR4, DOR5, and DOR7
were being used
during normal work-day hours
during the periods of time when severely troubled boards were being
loaded with software, or evaluated.
Beginning 157:22:48,
domhub51-DOR0--8ns157.txt:GPS
164:11:16:56 TQUAL(' ' exclnt.,<1us) DOR 000018908f4dc4bf
dt=60000000 ticks
domhub51-DOR1--8ns157.txt:GPS
164:11:16:56 TQUAL(' ' exclnt.,<1us) DOR 000018908ea6cbff
dt=60000000 ticks
domhub51-DOR2--8ns157.txt:GPS
159:11:02:40 TQUAL(' ' exclnt.,<1us) DOR 000010b0e96f1505
dt=160000000 ticks
domhub51-DOR2--8ns157.txt:GPS
164:11:16:56 TQUAL(' ' exclnt.,<1us) DOR 000018908dff0d05
dt=60000000 ticks
domhub51-DOR3--8ns157.txt:GPS
164:11:16:56 TQUAL(' ' exclnt.,<1us) DOR 000018908d5708d5
dt=60000000 ticks
domhub51-DOR4--8ns157.txt:GPS
159:11:02:40 TQUAL(' ' exclnt.,<1us) DOR 000010b0e81f6673
dt=160000000 ticks
domhub51-DOR4--8ns157.txt:GPS
164:11:16:56 TQUAL(' ' exclnt.,<1us) DOR 000018908caf5e73
dt=60000000 ticks
domhub51-DOR5--8ns157.txt:GPS
158:21:49:22 TQUAL(' ' exclnt.,<1us) DOR 000fd3423ac8f701
dt=4190067466 ticks
domhub51-DOR5--8ns157.txt:GPS
158:21:49:24 TQUAL(' ' exclnt.,<1us) DOR 00000fd3449d22f7
dt=164899830 ticks
domhub51-DOR5--8ns157.txt:GPS
158:21:53:24 TQUAL(' ' exclnt.,<1us) DOR 00000fd462b752f7
dt=40000000 ticks
domhub51-DOR5--8ns157.txt:GPS
158:22:01:48 TQUAL(' ' exclnt.,<1us) DOR 00000fd6bb87eaf7
dt=40000000 ticks
domhub51-DOR5--8ns157.txt:GPS
159:11:02:40 TQUAL(' ' exclnt.,<1us) DOR 000010b0e7779ef7
dt=160000000 ticks
domhub51-DOR5--8ns157.txt:GPS
161:22:29:45 TQUAL(' ' exclnt.,<1us) DOR 001495892633f732
dt=2654924859 ticks
domhub51-DOR5--8ns157.txt:GPS
161:22:29:46 TQUAL(' ' exclnt.,<1us) DOR 000014958a5760f7
dt=1680042437 ticks
domhub51-DOR5--8ns157.txt:GPS
161:23:25:24 TQUAL(' ' exclnt.,<1us) DOR 000014a5158c22f7
dt=40000000 ticks
domhub51-DOR5--8ns157.txt:GPS
162:14:33:16 TQUAL UNKNOWN! DOR 0015a2bd3b4af733 dt=2134956348 ticks
domhub51-DOR5--8ns157.txt:GPS
162:14:33:17 TQUAL(' ' exclnt.,<1us) DOR 000015a2be6c77f7
dt=2200010948 ticks
domhub51-DOR5--8ns157.txt:GPS
162:22:45:13 TQUAL(' ' exclnt.,<1us) DOR 0000162c303cd3f7
dt=40000000 ticks
domhub51-DOR5--8ns157.txt:GPS
164:11:16:56 TQUAL(' ' exclnt.,<1us) DOR 000018908c0796f7
dt=80000000 ticks
domhub51-DOR5--8ns157.txt:GPS
164:17:55:39 TQUAL(' ' exclnt.,<1us) DOR 000018fff277cdf7
dt=40000000 ticks
domhub51-DOR5--8ns157.txt:GPS
164:20:27:11 TQUAL(' ' exclnt.,<1us) DOR 0000192a48fa01f7
dt=40000000 ticks
domhub51-DOR5--8ns157.txt:GPS
167:00:08:19 TQUAL(' ' exclnt.,<1us) DOR 00001c8cbb5145f7
dt=40000000 ticks
domhub51-DOR5--8ns157.txt:GPS
167:00:55:48 TQUAL(' ' exclnt.,<1us) DOR 00001c99ff9712f7
dt=40000000 ticks
domhub51-DOR5--8ns157.txt:GPS
167:03:01:26 TQUAL(' ' exclnt.,<1us) DOR 00001cbd19961cf7
dt=40000000 ticks
domhub51-DOR6--8ns157.txt:GPS
164:11:16:56 TQUAL(' ' exclnt.,<1us) DOR 000018908b5fc431
dt=80000000 ticks
domhub51-DOR7--8ns157.txt:GPS
159:11:02:40 TQUAL(' ' exclnt.,<1us) DOR 000010b0e65f15ea
dt=160000000 ticks
domhub51-DOR7--8ns157.txt:GPS
164:11:16:56 TQUAL(' ' exclnt.,<1us) DOR 000018908aef0dea
dt=60000000 ticks
ending 167:04:30
The anomalies on DOR5 are caused by testing of faulty DOM main boards.
The anomalies at *:11:02 are due to the daily cron jobs.
I can not account for the anomaly at 164:11:16 occurring in DOR2, DOR4,
DOR5, and DOR7
For domhubjr.lbl.gov
Beginning 157:22:48,
domhubjr-DOR0--8ns157.txt:GPS
158:08:10:37 TQUAL(' ' exclnt.,<1us) DOR 00001b43592deb81
dt=20000002 ticks
domhubjr-DOR0--8ns157.txt:GPS
158:11:04:24 TQUAL(' ' exclnt.,<1us) DOR 00001b73e721ca81
dt=100000000 ticks
domhubjr-DOR0--8ns157.txt:GPS
159:23:21:43 TQUAL(' ' exclnt.,<1us) DOR 00001dd43cf4ad83
dt=20000002 ticks
domhubjr-DOR0--8ns157.txt:GPS
160:11:04:35 TQUAL(' ' exclnt.,<1us) DOR 00001e989de5b983
dt=80000000 ticks
domhubjr-DOR1--8ns157.txt:GPS
158:08:10:37 TQUAL(' ' exclnt.,<1us) DOR 00001b4358b244a7
dt=20000002 ticks
domhubjr-DOR1--8ns157.txt:GPS
158:11:04:24 TQUAL(' ' exclnt.,<1us) DOR 00001b73e6a623a7
dt=100000000 ticks
domhubjr-DOR1--8ns157.txt:GPS
159:23:21:43 TQUAL(' ' exclnt.,<1us) DOR 00001dd43c7906a9
dt=20000002 ticks
domhubjr-DOR1--8ns157.txt:GPS
160:11:04:35 TQUAL(' ' exclnt.,<1us) DOR 00001e989d6a12a9
dt=80000000 ticks
domhubjr-DOR2--8ns157.txt:GPS
158:08:10:37 TQUAL(' ' exclnt.,<1us) DOR 00001b4358026777
dt=20000002 ticks
domhubjr-DOR2--8ns157.txt:GPS
158:11:04:24 TQUAL(' ' exclnt.,<1us) DOR 00001b73e5f64677
dt=100000000 ticks
domhubjr-DOR2--8ns157.txt:GPS
159:23:21:43 TQUAL(' ' exclnt.,<1us) DOR 00001dd43bc92979
dt=20000002 ticks
domhubjr-DOR2--8ns157.txt:GPS
160:11:04:35 TQUAL(' ' exclnt.,<1us) DOR 00001e989cba3579
dt=80000000 ticks
domhubjr-DOR3--8ns157.txt:GPS
158:08:10:37 TQUAL(' ' exclnt.,<1us) DOR 00001b4357627a4d
dt=20000002 ticks
domhubjr-DOR3--8ns157.txt:GPS
158:11:04:24 TQUAL(' ' exclnt.,<1us) DOR 00001b73e556594d
dt=100000000 ticks
domhubjr-DOR3--8ns157.txt:GPS
159:23:21:43 TQUAL(' ' exclnt.,<1us) DOR 00001dd43b293c4f
dt=20000002 ticks
domhubjr-DOR3--8ns157.txt:GPS
160:11:04:35 TQUAL(' ' exclnt.,<1us) DOR 00001e989c1a484f
dt=80000000 ticks
domhubjr-DOR4--8ns157.txt:GPS
158:11:04:24 TQUAL(' ' exclnt.,<1us) DOR 00001b73e4ae95cb
dt=100000000 ticks
domhubjr-DOR4--8ns157.txt:GPS
160:11:04:35 TQUAL(' ' exclnt.,<1us) DOR 00001e989b7284cb
dt=80000000 ticks
domhubjr-DOR5--8ns157.txt:GPS
158:11:04:24 TQUAL(' ' exclnt.,<1us) DOR 00001b73e38e9e3d
dt=100000000 ticks
domhubjr-DOR5--8ns157.txt:GPS
160:11:04:35 TQUAL(' ' exclnt.,<1us) DOR 00001e989a528d3d
dt=80000000 ticks
domhubjr-DOR6--8ns157.txt:GPS
158:11:04:24 TQUAL(' ' exclnt.,<1us) DOR 00001b73e3129e2b
dt=100000000 ticks
domhubjr-DOR6--8ns157.txt:GPS
160:11:04:35 TQUAL(' ' exclnt.,<1us) DOR 00001e9899d68d2b
dt=80000000 ticks
domhubjr-DOR7--8ns157.txt:GPS
158:11:04:24 TQUAL(' ' exclnt.,<1us) DOR 00001b73e26acbd3
dt=100000000 ticks
domhubjr-DOR7--8ns157.txt:GPS
160:11:04:35 TQUAL(' ' exclnt.,<1us) DOR 00001e98992ebad3
dt=80000000 ticks
Ending 167:04:30
As above the anomalies at *11:04 are related to the daily cron jobs.
The anomalies with the extra two counts cannot be accounted for.
However, this flavor of anomaly don't appear in domhub51, driven by the
same GPS clock.
Evaluation
No anomalies since 165:16:12
A week or ten days without error would be much more convincing
than
just one day. Stay tuned...
Work Day Hours
08:00 local time, corresponds to 15:00 UTC
18:00 local time, corresponds to 01:00 UTC the next day.
The local time zone is PDT, as of the writing of this page.