In the first table you have the corresponding run, events that caused a problem. Here is alist of all of them, even those that were not included in the output and had to be removed. +-------------------+--------+---------+ | filename | run | trigger | +-------------------+--------+---------+ ar017983.0001phys 96643 0 ar0191a2.0016phys 102818 63437 ar0191b3.0010phys 102835 47455 ar0191b4.0001phys 102836 3741 ar0191b4.000aphys 102836 31138 ar019207.0007phys 102919 18162 ar019207.0023phys 102919 109555 ar0192ea.0022phys 103146 100686 ar0192f1.0007phys 103153 24872 ar0192f1.007cphys 103153 369131 ar0192f1.00a9phys 103153 504196 ar0192f1.00bbphys 103153 565227 ar0192f1.0133phys 103153 920619 ar019326.001cphys 103206 82298 ar01947d.004cphys 103549 230888 ar01947d.0052phys 103549 248812 ar01947f.0034phys 103551 159415 ar01947f.0037phys 103551 163840 ar01947f.0076phys 103551 357479 ar01947f.00a9phys 103551 507076 ar019481.0004phys 103553 15890 ar019488.0007phys 103560 23553 ar019488.0009phys 103560 29498 ar019488.000dphys 103560 41885 ar019488.000fphys 103560 47479 ar019488.0011phys 103560 53360 ar019488.0017phys 103560 71714 ar019488.0019phys 103560 77633 ar019488.001bphys 103560 82323 ar019488.001bphys 103560 83035 ar01948e.0074phys 103566 345680 ar019493.0001phys 103571 7648 ar019494.0001phys 103572 5942 ar019494.0007phys 103572 24337 ar019495.0001phys 103573 6353 ar019495.0007phys 103573 23959 ar019497.0001phys 103575 2403 ar019497.0001phys 103575 4940 ar019497.0004phys 103575 9078 ar019497.0004phys 103575 10685 ar019497.0004phys 103575 12705 ar019497.0004phys 103575 14084 ar019497.0004phys 103575 15674 ar019497.0007phys 103575 18688 ar019497.0007phys 103575 25540 ar019497.0007phys 103575 26646 ar019498.0001phys 103576 193 ar019498.0001phys 103576 296 ar019498.0001phys 103576 680 ar019498.0001phys 103576 857 ar019498.0001phys 103576 894 ar019498.0004phys 103576 9160 ar019498.0004phys 103576 9225 ar019498.0004phys 103576 9346 ar019498.0004phys 103576 9408 ar019498.0004phys 103576 9416 ar019498.0007phys 103576 18151 ar019498.0007phys 103576 18245 ar019498.0007phys 103576 18260 ar019498.0007phys 103576 18315 ar019498.0007phys 103576 18343 ar019498.000aphys 103576 27014 ar019498.000aphys 103576 27173 ar019498.000aphys 103576 27187 ar019498.000aphys 103576 27208 ar019499.0001phys 103577 25 ar019499.0001phys 103577 411 ar019499.0001phys 103577 633 ar019499.0001phys 103577 767 ar019499.0001phys 103577 812 ar019499.0001phys 103577 1218 ar019499.0001phys 103577 1359 ar019499.0001phys 103577 1442 ar019499.0001phys 103577 1460 ar019499.0004phys 103577 9071 ar019499.0004phys 103577 9158 ar019499.0004phys 103577 9376 ar019499.0004phys 103577 9479 ar019499.0004phys 103577 9487 ar019499.0004phys 103577 9817 ar019499.0004phys 103577 9876 ar019499.0004phys 103577 10028 ar019499.0004phys 103577 10065 ar019499.0004phys 103577 10208 ar019499.0004phys 103577 10365 ar01949a.0004phys 103578 16653 ar01949c.0001phys 103580 7179 ar01949c.0001phys 103580 8465 ar01949c.0004phys 103580 16328 ar01949d.0001phys 103581 134 ar01949d.0001phys 103581 380 ar01949d.0001phys 103581 422 ar01949d.0001phys 103581 526 ar01949d.0001phys 103581 568 ar01949d.0001phys 103581 750 ar01949d.0001phys 103581 964 ar01949d.0001phys 103581 1128 ar01949d.0001phys 103581 1144 ar01949d.0001phys 103581 1193 ar01949d.0001phys 103581 1255 ar01949d.0004phys 103581 8961 ar01949d.0004phys 103581 9126 ar01949d.0004phys 103581 9170 ar01949d.0004phys 103581 9227 ar01949d.0004phys 103581 9764 ar01949d.0004phys 103581 9806 ar01949d.0004phys 103581 9809 ar01949d.0004phys 103581 9811 ar01949d.0004phys 103581 9968 ar01949d.0004phys 103581 10472 ar01949d.0004phys 103581 10503 ar01949d.0007phys 103581 18148 ar01949d.0007phys 103581 18355 ar01949d.0007phys 103581 18384 ar01949d.0007phys 103581 18519 ar01949d.0007phys 103581 18650 ar01949d.0007phys 103581 18923 ar01949d.0007phys 103581 18980 ar01949d.0007phys 103581 19037 ar01949d.0007phys 103581 19135 ar01949d.0007phys 103581 19213 ar01949d.0007phys 103581 19234 ar01949e.0001phys 103582 407 ar01949e.0001phys 103582 413 ar01949e.0001phys 103582 535 ar01949e.0001phys 103582 546 ar01949e.0001phys 103582 601 ar01949e.0004phys 103582 8945 ar01949e.0004phys 103582 9164 ar01949e.0004phys 103582 9383 ar01949e.0004phys 103582 9407 ar01949e.0004phys 103582 9472 ar01949e.0004phys 103582 9476 ar01949e.0004phys 103582 9566 ar01949e.0004phys 103582 9597 ar01949e.0004phys 103582 9825 ar01949e.0004phys 103582 9867 ar01949e.0004phys 103582 9902 ar01949e.0007phys 103582 17903 ar01949e.0007phys 103582 18027 ar01949e.0007phys 103582 18075 ar01949e.0007phys 103582 18196 ar01949e.0007phys 103582 18245 ar01949e.000aphys 103582 27047 ar01949e.000aphys 103582 27056 ar01949e.000aphys 103582 27114 ar01949e.000aphys 103582 27179 ar01949e.000aphys 103582 27184 ar01949e.000aphys 103582 27200 ar01949e.000aphys 103582 27384 ar01949e.000aphys 103582 27397 ar01949e.000aphys 103582 27442 ar01949e.000aphys 103582 27512 ar01949e.000aphys 103582 27551 ar01949e.000dphys 103582 36022 ar01949e.000dphys 103582 36227 ar01949e.000dphys 103582 36251 ar01949e.000dphys 103582 36275 ar01949e.000dphys 103582 36391 ar01949e.0010phys 103582 45035 ar01949e.0010phys 103582 45123 ar01949e.0010phys 103582 45134 ar01949e.0010phys 103582 45270 ar01949e.0010phys 103582 45358 ar0194a0.0010phys 103584 51564 ar0194a0.0019phys 103584 79203 ar0194b4.0016phys 103604 67620 +-------------------+--------+---------+ 20 rows in set (0.00 sec) They are mostly infinite loops. In this table you'd be able to find some of the corresponding log file: +-------------------+--------+-------------+----------+ | fname | FBS_id | FBS_process | fileset | +-------------------+--------+-------------+----------+ | ar0192f1.0133phys | 5959 | 1 | CA3071.3 | | ar0192ea.0022phys | 5935 | 3 | CA3069.1 | | ar019207.0023phys | 5906 | 4 | CA3067.3 | | ar0192f1.00bbphys | 5918 | 2 | CA3070.4 | | ar0192f1.00a9phys | 5916 | 4 | CA3070.3 | | ar019207.0007phys | 5904 | 3 | CA3067.3 | | ar0192f1.007cphys | 5913 | 1 | CA3070.2 | | ar0191e5.0007phys | 5802 | 4 | CA3065.2 | | ar0191b2.001fphys | 5756 | 2 | CA3060.3 | | ar0191af.001cphys | 5756 | 1 | CA3062.0 | | ar0191b4.0001phys | 5750 | 4 | CA3060.4 | | ar0191b3.0010phys | 5749 | 1 | CA3060.4 | | ar0191b4.000aphys | 5751 | 3 | CA3060.4 | | ar0191a2.0016phys | 5753 | 1 | CA3059.0 | | ar0192f1.0007phys | 5940 | 4 | CA3069.3 | | ar01947f.0037phys | 5984 | 3 | CA3077.2 | | ar01947f.00a9phys | 5979 | 2 | CA3079.1 | | ar01947f.0034phys | 5984 | 2 | CA3077.2 | | ar01947f.0076phys | 5998 | 2 | CA3077.4 | | ar019326.001cphys | 6032 | 1 | CA3073.3 | | ar01948e.0074phys | 6055 | 2 | CA3080.2 | | ar019497.0001phys | 6087 | 3 | CA3082.0 | | ar017983.0001phys | 6305 | 1 | CA3000.0 | | ar019481.0004phys | 6277 | 2 | CA3078.0 | | ar019495.0007phys | 6278 | 1 | CA3081.4 | | ar019493.0001phys | 6277 | 4 | CA3081.4 | | ar0194b4.0016phys | 6295 | 2 | CA3084.0 | | ar019488.001bphys | 6316 | 1 | CA3078.3 | +-------------------+--------+-------------+----------+ I.e. when you choose a file in the first table, you know what is the run number and event number that you are about to look for. The second table shows you with the same filename what log file you should look for: i.e. in case of ar0192f1.0133phys you should look for a log file: rec.5959.1.log and so forth... Everything is available at fcdfsgi2: > pwd /cdf/data05/s5/farms/commissioning_run/pass2 > ls core hist logs In most cases those events caused infinite loop. There are a few that core dumped and there is a core file available in the core directory. Also, notice that the "histograms" from the current production already take about 10GB (I'm actually still transferring these files...)... -Enjoy. Miro