Forward Muon System Hardware Problems History

Started after October 2001 shutdown, 11/29/2001, Run #139626

Last update 06/19/2008, updated up to Run #243093 (including)

Current editor: V.Malyshev

Problem begins

Run#

Date

Description

Problem ends

Run#

Date

2001

142145

12/22/01

Pixels front-end crate #70 NABET(octants North A 0&1 and North B 0&1) is OFF. Crate removed from readout.

142887

01/08/02

142198

12/23/01

L1 crates disabled by DAQ shifter.

142203

12/23/01

142203

12/23/01

Crate #70 NABET removed from L1.

142887

01/08/02

2002

144494

01/26/02

Pixels FE-crate #0x299 SCW ( channel 2b of the readout crate #0x32, South C 2,3,4,5 octants) removed from readout.

144549

01/27/02

145547

02/07/02

L1 trigger undetermined in South A Pixel oct 7.

145526

02/07/02

145527

02/07/02

Part of South A7 Pixel octant (low-eta half) removed from trigger L1.

145689

02/08/02

146485

02/15/02

South Readout MDT crate GS0x30 ( All South MDT Octants) removed from global run.

146492

02/15/02

146492

02/15/02

Channel 5B ( South MDT octants SB2 and SB3) of the Readout MDT crate GS0x30 disabled.

146496

02/16/02

148686

03/13/02

Sync problem of SABWT Pixel FE crate . Rejected about 70% South A&B Pixel octants 2&3 data.

148837

03/15/02

148990

03/18/02

2b channel of G0x32 Pixel Readout crate( South C 2,3,4,5 octants) removed from run. FE Crate Power Supply problem.

149162

03/20/02

149204

03/22/02

Pixel SABEB FE( octants South A&B 6&7) removed from run. Crate is not reseting by 1553 net.

149275

03/22/02

149920

04/01/02

Channel 11A of MDT Readout crate #0x30 ( South C 4&5 octants) removed from readout . MDSC3 FE-crate Power Supply problem.

150860

04/11/02

151893

04/21/02

All South Pixel octants removed from L1. LVPS of South A HV Crate fault (HV of South A FE-crates switch off).

151896

04/21/02

153342

05/04/02

Channel 11A of MDT Readout crate #0x30 (South C 4&5 octants) dismissed from readout crates configuration.

153343

05/04/02

157537

06/19/02

All South L1 triggers are masked off (due to bad card 11 in the MTCS crate).

157591

06/20/02

159702

07/15/02

Forward Scintillators Front-End crate SCW power supply failure.

159707

07/15/02

160141

07/19/02

MDT Front-End crate SCWB (0x30, 11A) power supply failure. The MRC is taken off from readout.

160229

07/21/02

160224

07/21/02

MDT ADB LVPS's fuse (octant NC7) blow-out.

160229

07/21/02

160538

07/25/02

FSC Front-End crate SCE (0x31, 2A) had incorrect set of working parameters.

160539

07/25/02

165973

10/11/02

MDT Front-End crate SCET (0x30, 9A, or crate 280) power supply failure. Thus the MRC is not in readout.

165986

10/12/02

166113

10/14/02

MDT octant NB1's HV turned off due to constant trips.

166116

10/14/02

166295

10/14/02

MDT octant NB1's plane #3 disconnected from HV because of a short-circuit or a similar.

172359

02/10/03

167190

10/31/02

Continuous E1 in RO cratex31(section9B). Problem - bad MDC #6 in FE crate MDNC2(MDNCWT).

167262

11/02/02

168023

11/13/02

Has broken down gas rack monitor MUO_RM_18-3 (SCW). Replaced.

168033

11/14/02

169002

12/09/02

MDT octant SB7's plane #3 disconnected from HV because of a short-circuit or a similar.

172359

02/10/03

2003

172398

02/11/03

Pixels octant NAET first 16 counters missed ( absent L1 from)

172400

02/11/03

174236

03/13/03

Pixels crate FE70 falls out from run

174236

03/13/03

174268

03/14/03

MDT octant NC5 30% of channels disappeared (LVPS fuse problem)

174288

03/14/03

174310

03/15/03

Pixels octants NA1&NB1 bad hitmaps (hardware problem)

174313

03/15/03

174338

03/16/03

Pixels octant NA1 lost 50% hits(hardware problem)

174341

03/16/03

174490

03/20/03

MDT octant SC7 30% of channels disappeared (LVPS fuse problem)

174506

03/21/03

174639

03/23/03

Pixels crate FE90 (SABET - octants SA0,SA1,SB0,SB1) hardware problem

174647

03/23/03

175344

04/07/03

L2 SLIC 90 and 96 disabled (South Pixels FE crates problem)

175364

04/08/03

175519

04/10/03

Pixels ch#0A of readout crate 0x32 disabled (SABEB FE problem)

175530

04/10/03

175629

04/11/03

Pixels crate FE92 (SABWT) dead. LVPS problem.

175664

04/12/03

175891

04/20/03

70% L1 muon eff (SFE problem of Pixel FE NABEB)

175893

04/20/03

176163

04/26/03

MDT MDNAWB (oct NA4&NA5) disabled (LVPS dead)

176199

04/26/03

176206

04/27/03

Pixel FE#92 (SABWT) disabled (LVPS dead)

176222

04/27/03

176531

05/03/03

Pixels ch#2A of readout crate 0x32 disabled. FE#292 (SCE) problem

176555

05/03/03

176555

05/03/03

MDNBWT MDC#2 trouble ( 192 MDT channels dismissed)

176601

05/06/03

177005

05/17/03

Pixel's crate FE#279 (NCW) dead. FE LVPS problem

177011

05/18/03

177763

06/04/03

Pixel's crate FE#90 (SABET) dead. FE LVPS problem

177764

06/04/03

177976

06/11/03

MDT FE MDNBET problem ( broken code?). Fixed by reseting.

177976

06/11/03

178232

06/21/03

Pixel's crate FE#72 (NABWT) disabled. L2 error problem.

178239

06/21/03

178556

07/02/03

Pixel's crate FE#92 (SABWT) disabled. SFE#6 error.

178559

07/02/03

178718

07/07/03

Pixel's crate FE#298 (SCE) disabled. SFE#8 error.

178730

07/08/03

179545

08/01/03

Pixel FE NABWB disabled. Master module problem.

179579

08/02/03

180209

08/17/03

MDT FE NCET Master module problem.

180209

08/17/03

during fall 2003 shutdown

10/07/03

MDT octant SB0's plane #2 disconnected from HV because of a short-circuit or a similar.

during fall 2003 shutdown

11/04/03

2004

188329

01/18/04

Pixel's crate FE#72 (NABWT) disabled. FE LVPS problem. Until run #188334 whole RO crate 0x33 was disabled

188336

01/18/04

188544

01/24/04

Dead HV channel #29 for octant 4 SC plane. During this period don't work 12 sc. counters.

188555

01/25/04

188566

01/25/04

HV of MDT octant NC7 turned off because a short-circuit by broken wire.

188570

01/26/04

189768

02/27/04

Persistent FC error on 1st SFE in FE0x96. No hits from 1st half of SA6 octant.

189787

02/29/04

190171

03/09/04

MDT FE SCWT was disabled because of LVPS failure.

190305

03/11/04

190318

03/11/04

Pixels FE 299 (SCW) was disabled because of LVPS failure.

190431

03/16/04

193081

05/18/04

Pixels FE 94 (SABWB) has bad amplitudes spectra. Also this crate generated a lot of L1 parity error. All plots (except amplitudes) are OK

193110

05/19/04

193105

05/18/04

6 noisy channels in the MDT NA1 (plane 0, ch. 163-168)

193110

05/19/04

193804

06/08/04

MDT octant SA1's HV turned off due to short circuit

193808

06/09/04

193809

06/09/04

MDT octant SA1's plane 4 has been disconnected from HV because of a short-circuit by broken wire

-

-

194257

06/17/04

Pixels FE 299 (SCW) has some L2 problems.

194343

06/21/04

194546

06/27/04

GS 0x33 is out of readout due to problems with 4 front-ends

194551

06/27/04

194549

06/27/04

MDT crate SBWT (SB2, SB3 octants) was disabled due to readout problems

194550

06/27/04

194553

06/27/04

4 pixels front-ends are disabled (NCW, NABEB, NABET, NABWB) due to various problems

194564

06/28/04

194740

07/03/04

Pixels crate 298 shows bad histograms (times, hitmaps, amplitudes).

194794

07/04/04

194975

07/08/04

MDT crate NAEB (octants NA6, NA7) is disabled because of VME LVPS failure.

194977

07/08/04

201537

12/07/04

Pixel crates 94, 278, 279 had ~ -6 -7 ns timing shift.

201584

12/08/04

2005

203014

01/17/05

Pixels B Layer HV off.

203014

01/17/05

204045

02/23/05

Pixels crate SABEB (Octants: SA 6,7 and SB 6,7) is dead due to LVPS damage.

204078

02/24/05

204809

03/17/05

SFE busy sometimes in octant SA5 (low Eta), Crate #94 (SABWB). SFE L1 buffer has been overwritten.

205003

03/30/05

205827

04/22/05

At the beginning of store 4108 channel PBN31 didn't go from Standby to Full HV. For next run #205828 HV - Full.

205827

04/22/05

205857

04/23/05

MDT octants NB2,3 , NC2,3 and Pixel octants NA2,3 , NB2,3 , NC2,3 had 0 efficienfy on Examine plots. All MDT & Pixel octant's hit plots look reasonable. Triggers have not been affected. After appropriate FE crates power cycle and RO crates reboot these efficiencies are restored to normal. Selective off-line test of runs from this set didn't recognize bad things.

205896

04/25/05

206331

05/06/05

The problem with FE crate SCE (298): address error. FE LVPS swittched off than on.

206409

05/10/05

206379

05/08/05

HV problem with NC plane. Processor "ctlol29" is replaced.

206379

05/09/05

207498

07/05/05

All MDT HV is at Standby due to bad conditions oh halo.

207498

07/05/05

208041

07/10/05

All MDT HV is at Standby due to bad conditions oh halo.

208041

07/10/05

207999

07/07/05

HV shortcut in MDT octant NA1 plane #4 (broken wire). This plane is disconnected from HV.

212198

11/23/05

208205

07/13/05

During data taking with Trigger Configuration file of versions 14.20 - 14.30 L3 was used Monte Carlo geomery instead of real geometry.

208826

07/30/05

209148

08/08/05

The L1 muon power supply problems.Two l1muon crates are off. They affected muon triggers in the central and south

209166

08/10/05

209297

08/12/05

All L1 muon triggers disabled

209297

08/12/05

209305

08/12/05

South L1 Muon triggers disabled

209357

08/12/05

209380

08/14/05

South L1 Muon triggers disabled

209381

08/14/05

209862

08/31/05

Pixel crate SCE (crate ID 298, octants SC0,SC1,SC6,SC7) removed from readout due to the control problem of power supply.

209868

08/31/05

211490

11/01/05

MDT FE SCWT LVPS failure between stores.

211512

11/01/05

211889

11/15/05

Pixels FE SCE LVPS failure between stores.

211890

11/15/05

213272

12/25/05

All L1 muon crates are out of readout due to LVPS failure in L1 muon 0x19 (MTM).

213283

12/26/05

213305

12/26/05

MDT readout crate 0x31 is out of readout due to LVPS failure in NAWT.

213305

12/26/05

213305

12/26/05

MDT FE NAWT LVPS failure.

213333

12/27/05

213335

12/27/05

Bad HV pod for pixels FE076 (NABEB). Number of hits in octant NA6, HV group 2, counters #32-47 reduced by 35-40%. Efficiency for single muon events reduced by ~ 0.5%. HV pod replaced.

213516

01/04/06

2006

214674

01/29/06

Motherboard fault of LVPS in SAEB FE crate. LVPS is replaced.

214813

01/30/06

222484

06/22/06

L1 muon input disabled for pixels NA6, NB6 (second SFE) due to bad L1 muon card. New card installed.

222607

06/27/06

222647

06/28/06

North 3 A, B and C layer pixels and MDT octants hit maps have dips due to bad chip in L1 muon trigger card for mu1pt3wtlx and mu1pt4wtlx triggers.

222999

07/09/06

222667

06/28/06

HV short cut in MDT NC7 octant, plane #3. Zero counting rate in MDT NC7 octant. Zero efficiency in MDT NB7, NC7 octants and in FSC NA7, NB7, NC7 octants.

222669

06/28/06

222810

07/03/06

Pixels FE NABWT LVPS failure between stores. LVPS is replaced.

222829

07/03/06

223529

07/25/06

MDT NCEB FE crate lost connection to L2 inputs (SLIC67). FE crate removed from readout due to bad cable between NCEB FE crate and MCH3. Bad cable replaced with spare.

223673

07/29/06

224113

08/10/06

Broken power supply of Digi Ports (North side) This failure not corrupted data taken. Power supply replaced.

224158

08/11/06

227571

11/11/06

Pixels FE NCW LVPS failure. LVPS is replaced.

227580

11/12/06

After power glitch. No physics run.

11/16/06

Due to power glitch lost MDT ADB low voltage power supply of NA0 octant. LVPS is replaced. Problem with MDT crate 31. It shows L1 Error right after starting readout. The location of the error is always FE 0x66 (MDT NAEB). Looks like MDC CPLD problem. This problem fixed by mean refreshing MDC firmware in NAEB crate.

No physics runs.

11/17/06

228251

12/04/06

MDT FE SBWT LVPS failure. LVPS is replaced.

228275

12/05/06

------

12/15/06

Latency change. First run #228854. First physics run #228930

------

12/15/06

229057

12/20/06

Pixels FE SABWT LVPS failure. LVPS is replaced. Also +5V fuse in the last SFE board of this crate was blown. SFE is replaced.

229071

12/21/06

2007

230711

02/19/07

Pixel's FE NABET LVPS failure. LVPS replaced.

230748

02/20/07

230807

02/23/07

MDT's FE SAWT LVPS failure. LVPS replaced.

230817

02/24/07

231389

03/16/07

Pixel's FE NABWT LVPS failure. LVPS replaced.

231440

03/17/07

231718

03/29/07

Pixel's FE NCW LVPS failure. LVPS replaced.
Pixel's FE NCE Master module failure. Module replaced.

231754

03/30/07

231805

04/02/07

MDT FE SBWT LVPS & Master module failure. Replaced.

231861

04/03/07

233174

05/21/07

Pixel FE NABWB (Crate 74) LVPS failure. Replaced by new one.

233193

05/21/07

233376

06/02/07

MDT FE SBWB LVPS failure. Replaced by new one.

233393

06/04/07

234594

07/24/07

MDT FE NAEB LVPS failure. Replaced by new one.

234594

07/24/07

237525

11/03/07

MDT FE NCET LVPS failure. Replaced by new one.

237543

11/04/07

2008

239674

01/29/08

Pixel FE NCW (0x279) LVPS failure. Replaced by new one.

239678

01/29/08

239918

02/08/08

MDT FE SBWT LVPS failure. Replaced by new one.

239919

02/08/08

240658

03/11/08

MDT FE SCEB LVPS failure. Replaced by new one.

240665

03/11/08

241126

03/28/08

MDT FE NAET LVPS failure. Replaced by new one. Also not clear problem with NCET FE (on MDT resource monitor it looks like power supply failure, but LVPS is OK). For the first step replaced master-module.

241131

03/29/08

241152

03/29/08

L1 serial link 2 (SL2) of MDC #8 of MDT FE SAWT(MDSA2) problem. MDC replaced.

241156

03/30/08

243040

06/17/08

Pixel FE NCW (0x279) LVPS failure. Replaced by new one.

243093

06/18/08

243245

06/23/08

Pixel FE SCW LVPS failure. Replaced by new one.

243268

06/23/08

243398

06/29/08

MDNB 4 (NBEB), MDC 4 (Octant 6), SL1 transmitter

243516

07/02/08

245311

09/04/08

MDT octant NB7 plane 2 has HV shortcut due to broken wire. Plane is disconnected from HV.
Run 245295 has 0 hits in this octant (all octant disconnected from HV).

XXXXXX

XX/XX/08

245741

09/21/08

SAWB FE crate (RO crate 30, section 3A) L1 & L2 errors. CPLD failure.
MDRC of SAWB is replaced. Run #245741 has fully disabled RO crate 30. All other - just section 3A.

245750

09/22/08

245975

10/01/08

SCEB VME Power Supply failure.

246009

10/01/08