2004 CDF E-Log -- Eve shift. Wed Mar 3, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Stephan Lammel Chris Marino Ian Vollrath Diego Cauz Mary Convery


Start of Shift Notes:  

Store 3271 (inst lumi 1.875E31) and run 179600 in progress
 Wed Mar 3 16:18:55 
 
 - run 179600, TEST_MET15_DECOUPLED[1,434,403], 35k events so far 
 - expect COT HV on tomorrow morning (Aseet)
 - Stephan
Wed Mar 3 16:22:32 Apparently the fine print says the COT cannot turn on until _3_ gas volume exchanges, which means tomorrow morning.

sketchy plan from RunCoordinator elog:
15:16:04- Since we are a bit low on pbars due to todays Recycler transfers, we will most likely run this store down to an average luminosity of 10E30. I ll generate a plan later this evening, but it looks like the store would be terminated late in the Thursday owl shift. We would go directly into shot setup afterwards. - JPM
 - convery

Wed Mar 3 16:43:44 icicle heartbeat alarm. icicle appears to be running fine. my colleague Mr. Schwarz had this also in the previous shift - ian
-- Wed Mar 3 17:31:35 comment by...ian --  
has been back to green for a while (i.e. no purple heart(beat) for icicle)
Wed Mar 3 16:53:30 Run 179600 Terminated at 2004.03.03 16:52:02 - RunControl
Wed Mar 3 16:53:31 Run 179600 TERMINATE: L2 WORK commencing - Christopher x2080
Wed Mar 3 16:57:43 Run 179601 Activated at 2004.03.03 16:56:31 - RunControl
Wed Mar 3 16:58:28 Run 179601 ACTIVATE: JET_ST3_DECOUPLED[1,433,403]
default for L2 work - Christopher x2080
Wed Mar 3 17:00:20 Run 179600 RUNSTATUS: 
Marked Bad, explanation:
COT  off due to gas flow change
 - cdfscico
Wed Mar 3 17:22:40 Run 179601 Terminated at 2004.03.03 17:22:18 - RunControl
Wed Mar 3 17:22:47 Run 179601 TERMINATE: end default run - Christopher x2080
Wed Mar 3 17:30:17 Run 179602 Activated at 2004.03.03 17:28:34 - RunControl
Wed Mar 3 17:30:19 Run 179602 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST II - Christopher x2080
Wed Mar 3 17:31:35 Run 179602 Terminated at 2004.03.03 17:30:59 - RunControl
Wed Mar 3 17:32:53 Run 179602 TERMINATE: L2 TEST II Complete - Christopher x2080
Wed Mar 3 17:37:36 Run 179602 TERMINATE: L2 Test II complete. XCES thresholds set to 0.  - Christopher x2080
Wed Mar 3 17:40:33 Run 179603 Activated at 2004.03.03 17:39:58 - RunControl
Wed Mar 3 17:40:34 Run 179603 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST III - Christopher x2080
Wed Mar 3 17:45:04 Run 179603 Terminated at 2004.03.03 17:43:46 - RunControl
Wed Mar 3 17:45:05 Run 179603 TERMINATE: NO SMXR threshold - Christopher x2080
Wed Mar 3 17:51:26 Run 179604 Activated at 2004.03.03 17:51:06 - RunControl
Wed Mar 3 17:53:03 Run 179604 Terminated at 2004.03.03 17:52:04 - RunControl
Wed Mar 3 17:53:04 Run 179604 TERMINATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST check for Vadim - Christopher x2080
Wed Mar 3 17:55:52 Run 179605 Activated at 2004.03.03 17:55:35 - RunControl
Wed Mar 3 17:57:01 Run 179605 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST IV
SMXR thresholds set to 0, l2 ces trig. bits all high - Christopher x2080
Wed Mar 3 17:59:09 Run 179605 Terminated at 2004.03.03 17:58:02 - RunControl
Wed Mar 3 17:59:10 Run 179605 TERMINATE: end of L2 test IV - Christopher x2080
Wed Mar 3 18:01:41 Run 179606 Activated at 2004.03.03 18:01:35 - RunControl
Wed Mar 3 18:02:24 Run 179606 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST V - Christopher x2080
Wed Mar 3 18:03:40 Run 179606 Terminated at 2004.03.03 18:03:24 - RunControl
Wed Mar 3 18:04:31 Run 179606 TERMINATE: l2 test v - Christopher x2080
Wed Mar 3 18:05:23 
asked to check out tof pc -- it is not frozen, i.e. is updating. there are no error messages
present. note that tof hv is currently off.
 - ian
Wed Mar 3 18:07:03 Run 179607 Activated at 2004.03.03 18:06:40 - RunControl
Wed Mar 3 18:08:24 Run 179607 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- short check run - Christopher x2080
Wed Mar 3 18:12:36 Run 179607 Terminated at 2004.03.03 18:12:29 - RunControl
Wed Mar 3 18:15:31 Run 179607 TERMINATE: check complete -no L2 decision timeouts - Christopher x2080
Wed Mar 3 18:21:51 Run 179608 Activated at 2004.03.03 18:21:40 - RunControl
Wed Mar 3 18:22:26 Run 179608 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST VI - Christopher x2080
Wed Mar 3 18:23:10 Run 179592 RUNSTATUS: 
Marked Bad, explanation:
TOF  HighVoltage off
 - cdfscico
Wed Mar 3 18:23:43 Run 179600 RUNSTATUS: 
Marked Bad, explanation:
TOF  HighVoltage off
 - cdfscico
Wed Mar 3 18:25:52 Run 179608 Terminated at 2004.03.03 18:25:01 - RunControl
Wed Mar 3 18:27:10 Run 179608 TERMINATE: smxr thresholds set to 0, splinter on E12 and E23
received high rate of L3 reformatter errors - Christopher x2080
Wed Mar 3 18:29:51 
tof hv turned on at 18:28.
 - ian
Wed Mar 3 18:32:34 Run 179609 Activated at 2004.03.03 18:32:15 - RunControl
Wed Mar 3 18:32:48 Run 179609 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST VII - Christopher x2080
Wed Mar 3 18:33:51
 - plots (15:45-18:00) --ian
Wed Mar 3 18:36:18 Run 179609 Terminated at 2004.03.03 18:35:56 - RunControl
Wed Mar 3 18:38:04 Run 179609 TERMINATE: end run -- more L3 reformatter errors (duplicate FIB id) - Christopher x2080
-- Wed Mar 3 19:11:00 comment by...SCN --  By the way, the troubled wedge is SB3W3 - looks like something screwed up in the Fib header for this wedge, according to the reformatter log. It could either be
  1. the Fib (but the other wedge in this fib looks ok,)
  2. the Glink Transmitters between the FIB and VRB
  3. The VTM or VRB
SVT should be sensitive to 1 and 2 but not 3, one might askt hem what they think. Also, a COLDSTART might fix any/all of the above, popssibly even a HRR might do it.
Wed Mar 3 18:39:19 Dropping silicon from L2 test runs - Christopher
-- Wed Mar 3 18:46:59 comment by...Rainer --  

by standing procedures, silicon is not available for trigger tests unless oked by the silicon main pager carrier or the SPL

asked the shift crew to drop the silicon crates immediately.

  • asked the shift crew to drop the silicon crates immediately
  • page silicon once the L2 tests are over. See also this link

    • -- Wed Mar 3 18:59:33 comment by...convery --  This was an oversight on my part... I approved the fiber splitting tests thinking that they would be basically transparent for this jet trigger table that we had been running all day.
    Wed Mar 3 18:43:16 Run 179610 Activated at 2004.03.03 18:42:46 - RunControl
    Wed Mar 3 18:43:17 Run 179610 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST  - Christopher x2080
    Wed Mar 3 18:57:45 Run 179610 Terminated at 2004.03.03 18:57:16 - RunControl
    Wed Mar 3 18:57:46 Run 179610 TERMINATE: end L2 test ruun - Christopher x2080
    Wed Mar 3 19:01:03 Run 179610 TERMINATE: L2 test - Christopher x2080
    Wed Mar 3 19:05:14 Run 179611 Activated at 2004.03.03 19:04:11 - RunControl
    Wed Mar 3 19:05:15 Run 179611 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST  - Christopher x2080
    Wed Mar 3 19:14:52
     - plots (18:00-19:00) --ian
    -- Wed Mar 3 19:18:09 comment by...ian --  
    note B0PAGC is on the rise and is now above 15kHz. we are not getting any tevmon warnings ...
have the limits been changed? as far as i know limits are: 


warning: >15kHz
danger (alarm): >20kHz

    -- Wed Mar 3 20:30:22 comment by...convery --  The 20kHz limit where the run is halted and silicon brought to standby has not changed. However, the warning at 15kHz has been replaced with a message at 18kHz to the effect that the shift crew should keep a close watch on the abort gap losses.
    Wed Mar 3 19:21:50 Run 179601 RUNSTATUS: 
    Marked Bad, explanation:
L2T  muon trigger corrupted, fibre split work
COT  off due to gas flow change
TOF  HighVoltage off
     - cdfscico
    Wed Mar 3 19:25:26 
    silicon brought standby at rainer's request.
     - ian
    Wed Mar 3 19:26:13 Run 179610 RUNSTATUS: 
    Marked Bad, explanation:
L2T  muon trigger information corrupted, fibre slit work
COT  off due to gas flow change
SVX  \
ISL   |  Silicon out
L00  /
     - cdfscico
    Wed Mar 3 19:39:02
    jump in current around time si set to standby.
     - ian
    Wed Mar 3 19:54:43 Run 179611 Terminated at 2004.03.03 19:54:04 - RunControl
    Wed Mar 3 19:56:15 Run 179611 TERMINATE: continuing with L2 tests
    this run was halted, and adjustements were made before recover/run many times - Christopher x2080
    Wed Mar 3 19:59:05 Run 179612 Activated at 2004.03.03 19:58:24 - RunControl
    Wed Mar 3 19:59:06 Run 179612 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 TEST  - Christopher x2080
    Wed Mar 3 20:19:51 Run 179612 Terminated at 2004.03.03 20:19:39 - RunControl
    Wed Mar 3 20:21:20 Run 179612 TERMINATE: L2 run - Christopher x2080
    Wed Mar 3 20:30:19 Run 179613 Activated at 2004.03.03 20:27:49 - RunControl
    Wed Mar 3 20:30:20 Run 179613 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 Test - Christopher x2080
    Wed Mar 3 20:31:42 Run 179613 Terminated at 2004.03.03 20:31:30 - RunControl
    Wed Mar 3 20:32:28 Run 179613 TERMINATE: L2 test - Christopher x2080
    Wed Mar 3 20:36:31 Run 179614 Activated at 2004.03.03 20:35:34 - RunControl
    Wed Mar 3 20:36:32 Run 179614 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 Test - Christopher x2080
    Wed Mar 3 20:39:33 Run 179614 Terminated at 2004.03.03 20:39:05 - RunControl
    Wed Mar 3 20:39:34 Run 179614 TERMINATE: l2 test - Christopher x2080
    Wed Mar 3 20:47:02 Run 179611 RUNSTATUS: 
    Marked Bad, explanation:
L2T  muon trigger info corrupted, fibre split work
COT  off, gas flow change
SVX  \
ISL   |  Silicon out
L00  /
     - cdfscico
    Wed Mar 3 20:47:03 Run 179615 Activated at 2004.03.03 20:45:15 - RunControl
    Wed Mar 3 20:47:04 Run 179615 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 Test - Christopher x2080
    Wed Mar 3 21:00:40 Run 179615 Terminated at 2004.03.03 21:00:13 - RunControl
    Wed Mar 3 21:01:14 Run 179615 TERMINATE: L2 test - Christopher x2080
    Wed Mar 3 21:08:11 Run 179616 Activated at 2004.03.03 21:06:39 - RunControl
    Wed Mar 3 21:08:13 Run 179616 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 Test - Christopher x2080
    Wed Mar 3 21:36:20 Run 179616 Terminated at 2004.03.03 21:36:12 - RunControl
    Wed Mar 3 21:36:33 Run 179616 TERMINATE: L2 test - Christopher x2080
    Wed Mar 3 21:41:15 Run 179615 RUNSTATUS: 
    Marked Bad, explanation:
L2T  muon trigger info corrupted, fibre split work
COT  off, gas flow change
SVX  \
ISL   | Silicon out
L00  /
     - cdfscico
    Wed Mar 3 21:41:48
     - plots (19:00-21:30) --ian
    Wed Mar 3 21:42:51 Run 179616 RUNSTATUS: 
    Marked Bad, explanation:
L2T  muon level 2 trigger info corrupted, fibre split work
COT  off, gas flow change
SVX  \
ISL   | Silicon out
L00  /
     - cdfscico
    Wed Mar 3 21:44:50 Run 179617 Activated at 2004.03.03 21:44:35 - RunControl
    Wed Mar 3 21:45:05
     - Chris Neu
    -- Wed Mar 3 21:47:51 comment by...neu --  L2 upgrade fiber splitting tests were performed this evening. After testing was over, restored the fibers to original state. Run 179601 is before, run 179616 is after...low stats but no hot channels introduced.
    Wed Mar 3 21:46:19 Run 179617 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- L2 Test - Christopher x2080
    Wed Mar 3 21:49:10
     - Chris Neu
    -- Wed Mar 3 21:50:35 comment by...neu --  More validation of L2 fiber splitting tests. Here are occupancies of mismatched bits. Same runs as above, before and after splitting work. No high mismatch rates introduced.
    Wed Mar 3 21:53:12 Run 179617 Terminated at 2004.03.03 21:52:28 - RunControl
    Wed Mar 3 21:53:13 Run 179617 TERMINATE: End of L2 tests - Christopher x2080
    Wed Mar 3 21:53:30 
    bringing up silicon hv.
     - ian
    Wed Mar 3 22:01:08 Run 179618 Activated at 2004.03.03 22:00:34 - RunControl
    Wed Mar 3 22:01:38 Run 179618 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] -- Run for 30 minutes after bringing up Silicon HV
    Then take D-mode Calibrations - Christopher x2080
    Wed Mar 3 22:02:13
     - Burkard
    -- Wed Mar 3 22:06:25 comment by...Burkard --  L2 Muon Data when trying to split the Fiber 1 with the p. splitter plugged directly into the L2 Muon Pulsar board. All other fibers behave when split.
    -- Wed Mar 3 23:09:57 comment by...Burkard --  who dunnit? Tx, Fiber, Rx I swapped Fiber MB 2 with a spare fiber PM 12 downstairs at the MUON trigger crate. Resume running results in errors as it should, as fibers upstairs were not swapped. By swapping fibers upstairs aswell the error count does not increase anymore. I bring in the p. fiber spitter: massive bit errors on the splitted fiber. I think that excludes the fiber as culprit. I go back to default fiber setup (pre swapping), I left labels (BCR) on the fibers I touched.
    -- Wed Mar 3 23:31:09 comment by...Burkard --  Hint by Jonathan: There is one more thing we can do. Take a run with the fiber splitter in default position for fiber one. Swap the splitted fiber with the neighboring fiber. Do some offline analysis. That should give us some clue about the receiver side. Will take these runs tomorrow early morning when I am more able to think myself, aiming to do that test before 8am.
    Wed Mar 3 22:58:01
     - plots (21:30-22:30) --ian
    Wed Mar 3 23:01:24 Run 179618 Terminated at 2004.03.03 23:01:03 - RunControl
    Wed Mar 3 23:01:38 Run 179618 TERMINATE: take silicon d-mode calibrations - Christopher x2080
    Wed Mar 3 23:07:56 Run 179619 ACTIVATE: Silicon D-mode Calibration DPS-on - Christopher x2080
    Wed Mar 3 23:10:30 Run 179619 TERMINATE: Silicon D-mode Calibration DPS-on - Christopher x2080
    Wed Mar 3 23:16:24 Run 179620 ACTIVATE: Silicon D-mode Calibration DPS-off - Christopher x2080
    Wed Mar 3 23:20:18 Run 179620 TERMINATE: Silicon D-mode Calibration DPS-off - Christopher x2080
    Wed Mar 3 23:25:19 Run 179621 Activated at 2004.03.03 23:25:00 - RunControl
    Wed Mar 3 23:25:54 Run 179621 ACTIVATE: JET_ST3_DECOUPLED[1,433,403] - Christopher x2080
    -- Wed Mar 3 23:49:36 comment by...rainer --  at this luminosity, can run with prescale of 2 of ST3_.... of course, if another store comes in, default prescale of x10 is warranted - but page silicon SPL if that is going to be the case.
    Wed Mar 3 23:33:23 Run 179618 RUNSTATUS: 
    Marked Bad, explanation:
L2T  muon level 2 trigger errors
COT  off, gas flow change
     - cdfscico
    Wed Mar 3 23:36:43
     - plots (22:30-23:30) --ian
    Wed Mar 3 23:37:32 
    ISOLIST:  
Vadim got rid of the MOD time out.  
Saw next level of errors: etsum missmatches, eta & phi are fine. 
Jonathan has plenty of ideas how we could increase the rate of events which could help us to nail
this problem.
     - Burkard for Vadim
    -- Thu Mar 4 00:08:33 comment by...Tom --  We have observed this same problem (etsum errors but none in eta and phi) in the recent past, before you guys started messing around with the fibers. It very well may be unrelated to your work.
    Wed Mar 3 23:51:27 Rainer called to verify no more Reformatter errors in D-mode calibration and resumed "normal"(normal for this store) running. We then, based on rate of L1_JET3_PS1 trigger, adjusted the prescale down to 3 and then down to 2. Within 5 minutes I did receive a Reformatter error; therefore I have moved the prescale back to 3 and recommend this setting for owl shift. - Christopher :: (run 179621)
    -- Wed Mar 3 23:54:59 comment by...rainer --  this is very odd - keep an eye on it. if reformatter errors persist, page me. check for 'duplicate FIB' type problems - there is a very low rate one which is harmless.
    Wed Mar 3 23:55:33
    Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
    179592 x2BD88 BEAM JET_ST3_DECOUPLED [1,433,403] 14:33:46 16:00:58 01:26:49 48,303,433 476,297 476,297 100.863 0 1 1
    179600 x2BD90 BEAM TEST_MET15_DECOUPLED [1,434,403] 16:07:17 16:52:02 00:43:39 300,679 233,295 233,294 48.039 0 1 1
    179601 x2BD91 BEAM JET_ST3_DECOUPLED [1,433,403] 16:56:31 17:22:18 00:25:44 13,229,596 139,329 139,329 27.631 0 1 1
    179610 x2BD9A BEAM JET_ST3_DECOUPLED [1,433,403] 18:42:46 18:57:16 00:14:27 6,930,480 77,501 77,501 14.395 0 0 1
    179611 x2BD9B BEAM JET_ST3_DECOUPLED [1,433,403] 19:04:11 19:54:04 00:30:12 14,087,863 161,067 161,067 29.524 0 0 1
    179615 x2BD9F BEAM JET_ST3_DECOUPLED [1,433,403] 20:45:15 21:00:13 00:13:47 30,439,407 76,314 76,313 12.767 0 0 1
    179616 x2BDA0 BEAM JET_ST3_DECOUPLED [1,433,403] 21:06:39 21:36:12 00:24:22 52,865,952 133,666 133,666 22.264 0 0 1
    179618 x2BDA2 BEAM JET_ST3_DECOUPLED [1,433,403] 22:00:34 23:01:03 00:32:35 13,578,697 172,105 172,105 28.542 0 1 1
    179621 x2BDA5 BEAM JET_ST3_DECOUPLED [1,433,403] 23:25:00 00:28:22 16,272,685 149,864 148,761 23.485 1
    Totals 23:55:01 05:00:00 196,008,792 1,619,438 1,618,333 307.509
     - End of Shift Report
    Wed Mar 3 23:59:56 Shift Summary: 
    Store 3271 (inst lumi 1.39E31) and run 179621 in progress


   - took over 200k evts with TEST_MET15_DECOUPLED[1,434,403] 
   - level 2 muon fibre split work for about 5 1/2 hours 
   - silicon Dmode calibration done 
   - event display shows "permanent" COT hits, noisy TDCs? 
   - running trigger table JET_ST3_DECOUPLED[1,433,403] 

Plan is to continue with JET_ST3_DECOUPLED[1,433,403] 
   - remeber to set prescale after config 
   - page silicon 1 hour before end of store 
   - call Burkard 357-1530 1 hour before end of store 
   - between stores: - cal calibration, page cal front end 
                     - calibration and cosmics

    End of Shift Numbers
    CDF Run II 

    Runs                   179592, 179600, 179618, 179621
Delivered Luminosity   0.461 pb^-1  
Acquired Luminosity    0.262 pb^-1  
Efficiency             56.9
     - Stephan