Storage Services Administration Open Tickets

Last updated 2009-02-26


Open Ticket Distribution

2/9/2009	33	129014	Assigned	Medium	NAYMOLA, STAN	FUESS, STUART	"I'd like to start the process of getting a stken dCache and enstore allocation for use by the JDEM prototyping activity. I suspect overall usage will be less that 100 GB, but details are yet to be determined."
2/7/2009	35	128951	Assigned	Medium	HICKS, STAN	RUBIN, HOWARD	"Here are the job failure times: F00041348_0003.0 2008-07 1 2009-02-06 20:02:11 fnpc344 F00041345_0012.0 2008-07 1 2009-02-06 20:02:13 fnpc233 F00033573_0000.0 2006-01 1 2009-02-06 20:02:40 fnpc209 F00041339_0001.0 2008-07 1 2009-02-06 20:03:06 fnpc379 F00037676_0020.0 2007-02 1 2009-02-06 20:09:52 fnpc181 F00037676_0022.0 2007-02 1 2009-02-06 20:09:52 fnpc173 F00033570_0007.0 2006-01 1 2009-02-06 20:18:04 fnpc198 Here is a sample output log, first stdout, then stderr: Beginning job analyze on fnpc198 on Fri Feb 06 at 18:43:09 CST MANPATH=/usr/local/grid/globus/man::/usr/local/grid/vdt/man:/usr/local/grid /perl /man:/usr/local/grid/expat/man:/usr/local/grid/logrotate/man:/usr/local/gri d/wge t/man:/usr/local/grid/jdk1.5/man:/usr/local/grid/curl/share/man:/usr/local/ grid/ glite/share/man:/usr/local/grid/lcg/man:/usr/local/grid/prima/man GRID3_DATA_DIR=/grid/data GRID3_TMP_WN_DIR=/local/stage1 PAC_ANCHOR=/usr/local/grid OSG_GLEXEC_LOCATION=/opt/glite/sbin/glexec VOMS_USERCONF=/usr/local/grid/glite/etc/vomses GLOBUS_PATH=/usr/local/grid/globus GLOBUS_LOCATION=/usr/local/grid/globus PERL5LIB=/usr/local/grid/vdt/lib:/usr/local/grid/perl/lib/5.8.0:/usr/local/ grid/ perl/lib/5.8.0/x86_64-linux-thread-multi:/usr/local/grid/perl/lib/site_perl /5.8. 0:/usr/local/grid/perl/lib/site_perl/5.8.0/x86_64-linux-thread-multi: _CONDOR_SCRATCH_DIR=/local/stage1/condor/execute/dir_24656 X509_CERT_DIR=/usr/local/grid/globus/TRUSTED_CA OSG_WN_TMP=/local/stage1 ANT_HOME=/usr/local/grid/ant GLITE_LOCATION_LOG=/usr/local/grid/glite/log OSG_JOB_CONTACT=fnpcfg2.fnal.gov/jobmanager-condor OSG_SQUID_LOCATION=squid.fnal.gov SRM_CONFIG=/usr/local/grid/srm-client-fermi/etc/config-2.xml OSG_SITE_WRITE=srm://fndca1.fnal.gov:8443//pnfs/fnal.gov/usr/fermigrid/vola tile GLOBUS_ERROR_VERBOSE=true LD_LIBRARY_PATH=/usr/local/grid/glexec-osg/lib64:/usr/local/grid/glexec-osg /lib: /usr/local/grid/prima/lib:/usr/local/grid/lcg/lib64:/usr/local/grid/lcg/lib :/usr /local/grid/glite/lib64:/usr/local/grid/glite/lib:/usr/local/grid/openldap/ lib:/ usr/local/grid/curl/lib:/usr/local/grid/jdk1.5/jre/lib/i386:/usr/local/grid /jdk1 5/jre/lib/i386/server:/usr/local/grid/jdk1.5/jre/lib/i386/client:/usr/loca l/gri d/globus/lib:/usr/local/grid/berkeley-db/lib:/usr/local/grid/expat/lib:/usr /loca l/vdt-1.10.1/apache/lib:/usr/local/vdt-1.10.1/MonaLisa/Service/VDTFarm/pgsq l/lib :/usr/local/vdt-1.10.1/glite/lib64:/usr/local/vdt-1.10.1/glite/lib:/usr/loc al/vd t-1.10.1/prima/lib:/usr/local/vdt-1.10.1/mysql/lib/mysql:/usr/local/vdt-1.1 0.1/j dk1.5/jre/lib/i386:/usr/local/vdt-1.10.1/jdk1.5/jre/lib/i386/server:/usr/lo cal/v dt-1.10.1/jdk1.5/jre/lib/i386/client:/usr/local/vdt-1.10.1/berkeley-db/lib: /usr/ local/vdt-1.10.1/expat/lib:/usr/local/vdt-1.10.1/globus/lib:/usr/local/vdt- 1.10. 1/apache/lib:/usr/local/vdt-1.10.1/MonaLisa/Service/VDTFarm/pgsql/lib:/usr/ local /vdt-1.10.1/glite/lib64:/usr/local/vdt-1.10.1/glite/lib:/usr/local/vdt-1.10 1/pr ima/lib:/usr/local/vdt-1.10.1/mysql/lib/mysql:/usr/local/vdt-1.10.1/jdk1.5/ jre/l ib/i386:/usr/local/vdt-1.10.1/jdk1.5/jre/lib/i386/server:/usr/local/vdt-1.1 0.1/j dk1.5/jre/lib/i386/client:/usr/local/vdt-1.10.1/berkeley-db/lib:/usr/local/ vdt-1 10.1/expat/lib: GPT_LOCATION=/usr/local/grid/gpt GLITE_LOCATION_TMP=/usr/local/grid/glite/tmp OSG_LOCATION=/usr/local/vdt-1.10.1 _CONDOR_ANCESTOR_1446=1447:1232038179:645145024 _CONDOR_HIGHPORT=65535 _CONDOR_ANCESTOR_1447=24656:1233967389:243619398 LIBPATH=/usr/local/grid/globus/lib:/usr/lib:/lib OSG_SITE_READ=dcap://fndca1.fnal.gov:24525//pnfs/fnal.gov/usr/fermigrid/vol atile GLOBUS_OPTIONS=-Xmx512M GRID_SECURITY_DIR=/usr/local/grid/globus/etc GLOBUS_GRAM_MYJOB_CONTACT=URLx-nexus://fermigridosg1.fnal.gov:40034/ X509_CADIR=/usr/local/grid/globus/TRUSTED_CA PATH=/usr/local/grid/prima/bin:/usr/local/grid/lcg/bin:/usr/local/grid/glit e/sbi n:/usr/local/grid/glite/bin:/usr/local/grid/dccp/bin:/usr/local/grid/srm-cl ient- lbnl/bin:/usr/local/grid/srm-client-fermi/sbin:/usr/local/grid/srm-client-f ermi/ bin:/usr/local/grid/curl/bin:/usr/local/grid/ant/bin:/usr/local/grid/jdk1.5 /bin: /usr/local/grid/gpt/sbin:/usr/local/grid/globus/bin:/usr/local/grid/globus/ sbin: /usr/local/grid/wget/bin:/usr/local/grid/logrotate/sbin:/usr/local/pacman-3 25/b in:/usr/local/grid/vdt/sbin:/usr/local/grid/vdt/bin:/usr/local/bin:/bin:/us r/bin :/bin:/usr/bin:/sbin:/usr/sbin:/grid/fermiapp/minos/minfarm/bin VDT_LOCATION=/usr/local/grid OSG_DATA=/grid/data OSG_APP=/grid/app PWD=/local/stage1/condor/execute/dir_24656 JAVA_HOME=/usr/local/grid/jdk1.5 GLOBUS_REMOTE_IO_URL=/grid/home/minospro/.globus/job/fermigridosg1.fnal.gov /1046 4.1233967256/remote_io_url VOMS_LOCATION=/usr/local/grid/glite GRID3_TMP_DIR=/grid/data _CONDOR_SLOT=4 CATALINA_OPTS=-Dorg.globus.wsrf.container.persistence.dir=/usr/local/grid/v dt-ap p-data/globus/persisted OSG_GRID=/usr/local/grid _CONDOR_LOWPORT=60000 GRID3_SITE_NAME=FNAL_GPGRID_3 GRID3_APP_DIR=/grid/app OSG_HOSTNAME=fnpcfg2.fnal.gov GLITE_LOCATION_VAR=/usr/local/grid/glite/var HOME=/grid/home/minospro SHLVL=2 OSG_STORAGE_ELEMENT=True X509_USER_PROXY=/grid/home/minospro/.globus/job/fnpcfg2.fnal.gov/15859.1233 96728 1/x509_up DYLD_LIBRARY_PATH=/usr/local/grid/globus/lib VDT_INSTALL_LOG=vdt-install.log OSG_DEFAULT_SE=FNAL_FERMIGRID_SE SCRATCH_DIRECTORY=/grid/home/minospro//gram_scratch_cjvZPQHmgS LOGNAME=minospro OSG_SITE_NAME=FNAL_GPGRID_3 SHLIB_PATH=/usr/local/grid/globus/lib VDT_POSTINSTALL_README=/usr/local/grid/post-install/README GLITE_LOCATION=/usr/local/grid/glite _CONDOR_ANCESTOR_24656=24657:1233967389:902637760 GRID3_BASE_DIR=/usr/local/vdt-1.10.1 GLOBUS_GRAM_JOB_CONTACT=https://fermigridosg1.fnal.gov:40023/10464/12339672 56/ PEGASUS_HOME=/usr/local/grid/pegasus PACMAN_LOCATION=/usr/local/pacman-3.25 _=/usr/bin/printenv OLDPWD=/grid/home/minospro/gram_scratch_lGBAWPrJei -rw-r--r-- 1 root root 1606 Oct 13 09:07 /usr/local/grid/globus/TRUSTED_CA/e1fc e4e9.0 subject : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy/CN=limited proxy issuer : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy identity : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy type : unknown strength : 512 bits path : /grid/home/minospro/.globus/job/fnpcfg2.fnal.gov/15859.1233967281/x5 09_up timeleft : 5464:13:44 === VO fermilab extension information === VO : fermilab subject : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284 issuer : /DC=org/DC=doegrids/OU=Services/CN=http/voms.fnal.gov attribute : /fermilab/minos/Role=Production/Capability=NULL attribute : /fermilab/Role=NULL/Capability=NULL attribute : /fermilab/minos/Role=NULL/Capability=NULL timeleft : 0:00:00 Requesting F00033570_0007.mdaq.root using srmcp at 2009-02-06 18:43:09 SRM Configuration: debug=true gsissl=true help=false pushmode=false userproxy=true buffer_size=131072 tcp_buffer_size=0 streams_num=1 config_file=/usr/local/grid/srm-client-fermi/etc/config-2.xml glue_mapfile=/usr/local/grid/srm-client-fermi/conf/SRMServerV1.map webservice_path=srm/managerv1 webservice_protocol=https gsiftpclinet=globus-url-copy protocols_list=http,gsiftp save_config_file=null srmcphome=/usr/local/grid/srm-client-fermi urlcopy=sbin/urlcopy.sh x509_user_cert= x509_user_key= x509_user_proxy=/grid/home/minospro/.globus/job/fnpcfg2.fnal.gov/15859.1 233967281/x509_up x509_user_trusted_certificates=/usr/local/grid/globus/TRUSTED_CA globus_tcp_port_range=null gss_expected_name=null storagetype=permanent retry_num=20 retry_timeout=10000 wsdl_url=null use_urlcopy_script=false connect_to_wsdl=false delegate=true full_delegation=true server_mode=active srm_protocol_version=1 request_lifetime=86400 access latency=null overwrite mode=null priority=0 from[0]=srm://fndca1:8443/pnfs/fnal.gov/usr/minos/fardet_data/2006-01/F0 0033570_0007.mdaq.root to=file:////local/stage1/condor/execute/dir_24656/F00033570_0007.mdaq.ro ot Fri Feb 06 18:43:11 CST 2009: starting SRMGetClient Fri Feb 06 18:43:11 CST 2009: In SRMClient ExpectedName: host Fri Feb 06 18:43:11 CST 2009: SRMClient(https,srm/managerv1,true) SRMClientV1 : user credentials are: /DC=org/DC=doegrids/OU=People/CN=Howard Rubi n 763284 SRMClientV1 : SRMClientV1 calling org.globus.axis.util.Util.registerTransport() SRMClientV1 : connecting to srm at httpg://stkendca2a.fnal.gov:8443/srm/managerv 1 Fri Feb 06 18:43:12 CST 2009: connected to server, obtaining proxy Fri Feb 06 18:43:12 CST 2009: got proxy of type class org.dcache.srm.client.SRMC lientV1 SRMClientV1 : get: surls[0]=""srm://fndca1:8443/pnfs/fnal.gov/usr/minos/fardet_ data/2006-01/F00033570_0007.mdaq.root"" SRMClientV1 : get: protocols[0]=""gsiftp"" SRMClientV1 : get: protocols[1]=""dcap"" SRMClientV1 : get: protocols[2]=""http"" copy_jobs is empty Fri Feb 06 18:43:15 CST 2009: srm returned requestId = -2144033882 Fri Feb 06 18:43:15 CST 2009: sleeping 4 seconds ... Fri Feb 06 18:43:19 CST 2009: sleeping 4 seconds ... Fri Feb 06 18:43:24 CST 2009: sleeping 4 seconds ... Fri Feb 06 18:43:28 CST 2009: sleeping 4 seconds ... Fri Feb 06 18:43:33 CST 2009: sleeping 4 seconds ... Fri Feb 06 18:43:37 CST 2009: sleeping 7 seconds ... Fri Feb 06 18:43:44 CST 2009: sleeping 7 seconds ... Fri Feb 06 18:43:52 CST 2009: sleeping 7 seconds ... Fri Feb 06 18:43:59 CST 2009: sleeping 7 seconds ... Fri Feb 06 18:44:06 CST 2009: sleeping 7 seconds ... Fri Feb 06 18:44:13 CST 2009: sleeping 10 seconds ... Fri Feb 06 18:44:24 CST 2009: sleeping 10 seconds ... Fri Feb 06 18:44:34 CST 2009: sleeping 10 seconds ... Fri Feb 06 18:44:45 CST 2009: sleeping 10 seconds ... Fri Feb 06 18:44:55 CST 2009: sleeping 10 seconds ... Fri Feb 06 18:45:05 CST 2009: sleeping 13 seconds ... Fri Feb 06 18:45:19 CST 2009: sleeping 13 seconds ... Fri Feb 06 18:45:32 CST 2009: sleeping 13 seconds ... Fri Feb 06 18:45:45 CST 2009: sleeping 13 seconds ... Fri Feb 06 18:45:59 CST 2009: sleeping 13 seconds ... Fri Feb 06 18:46:12 CST 2009: sleeping 16 seconds ... Fri Feb 06 18:46:28 CST 2009: sleeping 16 seconds ... Fri Feb 06 18:46:45 CST 2009: sleeping 16 seconds ... Fri Feb 06 18:47:01 CST 2009: sleeping 16 seconds ... Fri Feb 06 18:47:17 CST 2009: sleeping 16 seconds ... Fri Feb 06 18:47:34 CST 2009: sleeping 19 seconds ... Fri Feb 06 18:47:53 CST 2009: sleeping 19 seconds ... Fri Feb 06 18:48:12 CST 2009: sleeping 19 seconds ... Fri Feb 06 18:48:32 CST 2009: sleeping 19 seconds ... Fri Feb 06 18:48:51 CST 2009: sleeping 19 seconds ... Fri Feb 06 18:49:10 CST 2009: sleeping 22 seconds ... Fri Feb 06 18:49:33 CST 2009: sleeping 22 seconds ... Fri Feb 06 18:49:55 CST 2009: sleeping 22 seconds ... Fri Feb 06 18:50:18 CST 2009: sleeping 22 seconds ... Fri Feb 06 18:50:40 CST 2009: sleeping 22 seconds ... Fri Feb 06 18:51:02 CST 2009: sleeping 25 seconds ... Fri Feb 06 18:51:27 CST 2009: sleeping 25 seconds ... Fri Feb 06 18:51:53 CST 2009: sleeping 25 seconds ... Fri Feb 06 18:52:18 CST 2009: sleeping 25 seconds ... Fri Feb 06 18:52:44 CST 2009: sleeping 25 seconds ... Fri Feb 06 18:53:09 CST 2009: sleeping 28 seconds ... Fri Feb 06 18:53:37 CST 2009: sleeping 28 seconds ... Fri Feb 06 18:54:06 CST 2009: sleeping 28 seconds ... Fri Feb 06 18:54:34 CST 2009: sleeping 28 seconds ... Fri Feb 06 18:55:03 CST 2009: sleeping 28 seconds ... Fri Feb 06 18:55:31 CST 2009: sleeping 31 seconds ... Fri Feb 06 18:56:02 CST 2009: sleeping 31 seconds ... Fri Feb 06 18:56:34 CST 2009: sleeping 31 seconds ... Fri Feb 06 18:57:05 CST 2009: sleeping 31 seconds ... Fri Feb 06 18:57:36 CST 2009: sleeping 31 seconds ... Fri Feb 06 18:58:08 CST 2009: sleeping 34 seconds ... Fri Feb 06 18:58:42 CST 2009: sleeping 34 seconds ... Fri Feb 06 18:59:16 CST 2009: sleeping 34 seconds ... Fri Feb 06 18:59:51 CST 2009: sleeping 34 seconds ... Fri Feb 06 19:00:25 CST 2009: sleeping 34 seconds ... Fri Feb 06 19:00:59 CST 2009: sleeping 37 seconds ... Fri Feb 06 19:01:37 CST 2009: sleeping 37 seconds ... Fri Feb 06 19:02:14 CST 2009: sleeping 37 seconds ... Fri Feb 06 19:02:51 CST 2009: sleeping 37 seconds ... Fri Feb 06 19:03:29 CST 2009: sleeping 37 seconds ... Fri Feb 06 19:04:06 CST 2009: sleeping 40 seconds ... Fri Feb 06 19:04:46 CST 2009: sleeping 40 seconds ... Fri Feb 06 19:05:26 CST 2009: sleeping 40 seconds ... Fri Feb 06 19:06:07 CST 2009: sleeping 40 seconds ... Fri Feb 06 19:06:47 CST 2009: sleeping 40 seconds ... Fri Feb 06 19:07:27 CST 2009: sleeping 43 seconds ... Fri Feb 06 19:08:11 CST 2009: sleeping 43 seconds ... Fri Feb 06 19:08:54 CST 2009: sleeping 43 seconds ... Fri Feb 06 19:09:37 CST 2009: sleeping 43 seconds ... Fri Feb 06 19:10:21 CST 2009: sleeping 43 seconds ... Fri Feb 06 19:11:04 CST 2009: sleeping 46 seconds ... Fri Feb 06 19:11:50 CST 2009: sleeping 46 seconds ... Fri Feb 06 19:12:37 CST 2009: sleeping 46 seconds ... Fri Feb 06 19:13:23 CST 2009: sleeping 46 seconds ... Fri Feb 06 19:14:09 CST 2009: sleeping 46 seconds ... Fri Feb 06 19:14:56 CST 2009: sleeping 49 seconds ... Fri Feb 06 19:15:45 CST 2009: sleeping 49 seconds ... Fri Feb 06 19:16:34 CST 2009: sleeping 49 seconds ... Fri Feb 06 19:17:24 CST 2009: sleeping 49 seconds ... Fri Feb 06 19:18:13 CST 2009: sleeping 49 seconds ... Fri Feb 06 19:19:02 CST 2009: sleeping 52 seconds ... Fri Feb 06 19:19:55 CST 2009: sleeping 52 seconds ... Fri Feb 06 19:20:47 CST 2009: sleeping 52 seconds ... Fri Feb 06 19:21:39 CST 2009: sleeping 52 seconds ... Fri Feb 06 19:22:32 CST 2009: sleeping 52 seconds ... Fri Feb 06 19:23:24 CST 2009: sleeping 55 seconds ... Fri Feb 06 19:24:19 CST 2009: sleeping 55 seconds ... Fri Feb 06 19:25:15 CST 2009: sleeping 55 seconds ... Fri Feb 06 19:26:10 CST 2009: sleeping 55 seconds ... Fri Feb 06 19:27:05 CST 2009: sleeping 55 seconds ... Fri Feb 06 19:28:00 CST 2009: sleeping 58 seconds ... Fri Feb 06 19:28:59 CST 2009: sleeping 58 seconds ... Fri Feb 06 19:29:57 CST 2009: sleeping 58 seconds ... Fri Feb 06 19:30:55 CST 2009: sleeping 58 seconds ... Fri Feb 06 19:31:54 CST 2009: sleeping 58 seconds ... Fri Feb 06 19:32:52 CST 2009: sleeping 61 seconds ... Fri Feb 06 19:33:53 CST 2009: sleeping 61 seconds ... Fri Feb 06 19:34:55 CST 2009: sleeping 61 seconds ... Fri Feb 06 19:35:56 CST 2009: sleeping 61 seconds ... Fri Feb 06 19:36:57 CST 2009: sleeping 61 seconds ... Fri Feb 06 19:37:59 CST 2009: sleeping 64 seconds ... Fri Feb 06 19:39:03 CST 2009: sleeping 64 seconds ... Fri Feb 06 19:40:08 CST 2009: sleeping 64 seconds ... Fri Feb 06 19:41:12 CST 2009: sleeping 64 seconds ... Fri Feb 06 19:42:16 CST 2009: sleeping 64 seconds ... Fri Feb 06 19:43:21 CST 2009: sleeping 67 seconds ... Fri Feb 06 19:44:28 CST 2009: sleeping 67 seconds ... Fri Feb 06 19:45:35 CST 2009: sleeping 67 seconds ... Fri Feb 06 19:46:43 CST 2009: sleeping 67 seconds ... Fri Feb 06 19:47:50 CST 2009: sleeping 67 seconds ... Fri Feb 06 19:48:57 CST 2009: sleeping 70 seconds ... Fri Feb 06 19:50:08 CST 2009: sleeping 70 seconds ... Fri Feb 06 19:51:18 CST 2009: sleeping 70 seconds ... Fri Feb 06 19:52:28 CST 2009: sleeping 70 seconds ... Fri Feb 06 19:53:39 CST 2009: sleeping 70 seconds ... Fri Feb 06 19:54:49 CST 2009: sleeping 73 seconds ... Fri Feb 06 19:56:03 CST 2009: sleeping 73 seconds ... Fri Feb 06 19:57:16 CST 2009: sleeping 73 seconds ... Fri Feb 06 19:58:29 CST 2009: sleeping 73 seconds ... Fri Feb 06 19:59:43 CST 2009: sleeping 73 seconds ... Fri Feb 06 20:00:56 CST 2009: sleeping 76 seconds ... Fri Feb 06 20:02:12 CST 2009: sleeping 76 seconds ... Fri Feb 06 20:03:29 CST 2009: sleeping 76 seconds ... Fri Feb 06 20:04:45 CST 2009: sleeping 76 seconds ... Fri Feb 06 20:06:02 CST 2009: sleeping 76 seconds ... Fri Feb 06 20:07:18 CST 2009: sleeping 79 seconds ... Fri Feb 06 20:08:37 CST 2009: sleeping 79 seconds ... Fri Feb 06 20:09:57 CST 2009: sleeping 79 seconds ... Fri Feb 06 20:11:16 CST 2009: sleeping 79 seconds ... Fri Feb 06 20:12:36 CST 2009: sleeping 79 seconds ... Fri Feb 06 20:13:55 CST 2009: sleeping 82 seconds ... Fri Feb 06 20:15:18 CST 2009: sleeping 82 seconds ... Fri Feb 06 20:16:41 CST 2009: sleeping 82 seconds ... copy_jobs is empty stopping copier Fri Feb 06 20:18:03 CST 2009: setting all remaining file statuses to ""Done"" Fri Feb 06 20:18:03 CST 2009: setting file request -2144033881 status to Done Fri Feb 06 20:18:04 CST 2009: set all file statuses to ""Done"" >> Unable to get F00033570_0007.mdaq.root from srm://fndca1:8443/pnfs/fnal.gov/u sr/minos/fardet_data/2006-01 << Exiting with status 1 printenv ls -l $OSG_GRID/globus/TRUSTED_CA/e1fce4e9.0 voms-proxy-info -all WARNING: Unable to verify signature! Server certificate possibly not installed. Error: Cannot find certificate of AC issuer for vo fermilab set +v Storage Resource Manager (SRM) implementation version 2.0.3 Copyright (c) 2002-2008 Fermi National Accelerator Laboratory Specification Version 2.0 by SRM Working Group (http://sdm.lbl.gov/srm-wg) Fri Feb 06 20:18:03 CST 2009: rs.state = Failed rs.error = at Fri Feb 06 18:43: 15 CST 2009 state Pending : created RequestFileStatus#-2144033881 failed with error:[ at Fri Feb 06 20:17:05 CST 20 09 state Failed : Pinning failed] Fri Feb 06 20:18:03 CST 2009: ====> fileStatus state ==Failed java.io.IOException: rs.state = Failed rs.error = at Fri Feb 06 18:43:15 CST 20 09 state Pending : created RequestFileStatus#-2144033881 failed with error:[ at Fri Feb 06 20:17:05 CST 20 09 state Failed : Pinning failed] at gov.fnal.srm.util.SRMGetClientV1.start(SRMGetClientV1.java:298)"
1/30/2009	43	128525	Assigned	Medium	MESSER, TIM	SCHUMACHER, KENNETH	"I reported a problem to dCache-admin 14:00 on 1/14/2009. An LQCD user reported an odd error when one of his 'dccp' commands failed. I was able to reproduce the problem at that time. It appeared that if the user tried to use 'dccp' to copy a file that was not already in cache, he got the following error: > --(0)> dccp > /pnfs/theory/milc/l2064f21b679m020m050/gauge/Coulomb//l2064f21b679m020m050- Coul_000342 > /tmp > Command failed! > Server error message for [1]: ""902"" (errno 902). > Failed open file in the dCache. > Can't open source file : ""902"" > System error: Input/output error > --(root@kaon1)-(~)-- > --(255)> date > Wed Jan 14 13:47:15 CST 2009 I have several e-mails that were exchanged between myself and dCache-admin. The last e-mail (2/14 16:28) was from Vladimir Podstavkov where he said he would continue to look into the problem. That was 2 weeks ago. I am opening this helpdesk request so that the resolution of this error can be tracked."
1/16/2009	57	127805	Assigned	Medium	SZMUKSTA, GEORGE	KREYMER, ARTHUR	"dcache-admin : I do not see the new Minos DCache pools in the pool file listings at http://fndca3a.fnal.gov/dcache/files/ The pools are online, so it is important that we have these lists, especially now that we lack Layer 2 PNFS data. Some of the pools are also missing from the Cell Services list at http://fndca3a.fnal.gov:2288/cellInfo r-minos-stkendca25a-3 r-minos-stkendca26a-3 r-minos-stkendca27a-2 w-pub-minos-stkendca25a-3 w-raw-minos-stkendca26a-1"
1/15/2009	58	127654	Work In Progress	High	SZMUKSTA, GEORGE	MSS, MSS	"['1232028186.49', 1232028186.4890411, 'stkensrv2n.fnal.gov', 20260, 'enstore', 'I (1)', 'Enstore_Up_Down', 'ENSTORE BALL IS RED - Ticket Generated', 'RedBall', 'STK Enstore', None, {'text': {'Reason': ['SL8500G1.media_changer down']}, 'r_a': (('131.225.13"
1/14/2009	59	127573	Assigned	Medium	SZMUKSTA, GEORGE	SIMMONDS, EDWARD	"I need to get ""read-write"" access for six new servers to cdfensrv1:/fs Those servers are: fcdfdcache10 fcdfdcache11 fcdfdcache12 fcdfdcache13 fcdfdcache14 fcdfdcache15 Just copy the access already granted to fcdfdcache1. Thanks very much, Edward Simmonds"
1/14/2009	59	127564	Assigned	Medium	SZMUKSTA, GEORGE	KREYMER, ARTHUR	"dcache-admin,minos-data : I get an empty file when accessing a recent Minos raw data file in DCache. /pnfs/minos/fardet_data/2009-01/F00042685_0003.mdaq.root But the file should not be empty, see ls and Layer 4 metadata below : -rw-r--r-- 1 buckley e875 18677506 Jan 13 08:01 F00042685_0003.mdaq.root VO8699 0000_000000000_0003456 18677506 fardet_data /pnfs/fs/usr/minos/fardet_data/2009-01/F00042685_0003.mdaq.root 000F00000000000009016408 CDMS123185530300000 stkenmvr10a:/dev/rmt/tps2d0n:479002012194 According to today's pool listings, this file is in w-stkendca8a-2 I see no files listed in the DCache filemonitor page. But that may not mean much, as those listing seem dated 23-Jul-2008 http://www-stken.fnal.gov/enstore/dcache_monitor/ Please restore F00042685_0003.mdaq.root in DCache."
1/13/2009	60	127543	Assigned	Medium	SZMUKSTA, GEORGE	RUBIN, HOWARD	"All my jobs of a certain class are failing srm reads. Here is the complete transcript of the problem. If Glenn is reading this, this is exactly what I was talking about when you last auto-requested whether a problem was 'solved.' fnpcsrv1% gum n13037001_0001_L010185N_D06_nccohbkg.0.cedar_phy_bhcurv.log.gz Beginning job ana_mc on fcdfcaf995 on Tue Jan 13 at 14:54:52 CST _CONDOR_ANCESTOR_19245=19246:1231880091:2066836224 MANPATH=/usr/local/grid/globus/man::/usr/local/grid/vdt/man:/usr/local/grid /perl /man:/usr/local/grid/expat/man:/usr/local/grid/logrotate/man:/usr/local/gri d/wge t/man:/usr/local/grid/jdk1.5/man:/usr/local/grid/curl/share/man:/usr/local/ grid/ glite/share/man:/usr/local/grid/lcg/man:/usr/local/grid/prima/man GRID3_DATA_DIR=/grid/data GRID3_TMP_WN_DIR=/local/stage1 PAC_ANCHOR=/usr/local/grid OSG_GLEXEC_LOCATION=/opt/glite/sbin/glexec VOMS_USERCONF=/usr/local/grid/glite/etc/vomses GLOBUS_PATH=/usr/local/grid/globus GLOBUS_LOCATION=/usr/local/grid/globus PERL5LIB=/usr/local/grid/vdt/lib:/usr/local/grid/perl/lib/5.8.0:/usr/local/ grid/ perl/lib/5.8.0/x86_64-linux-thread-multi:/usr/local/grid/perl/lib/site_perl /5.8. 0:/usr/local/grid/perl/lib/site_perl/5.8.0/x86_64-linux-thread-multi: _CONDOR_SCRATCH_DIR=/local/stage1/condor/execute/dir_19245 X509_CERT_DIR=/usr/local/grid/globus/TRUSTED_CA OSG_WN_TMP=/local/stage1 ANT_HOME=/usr/local/grid/ant GLITE_LOCATION_LOG=/usr/local/grid/glite/log OSG_JOB_CONTACT=fcdfosg1.fnal.gov/jobmanager-condor OSG_SQUID_LOCATION=squid.fnal.gov SRM_CONFIG=/usr/local/grid/srm-client-fermi/etc/config-2.xml OSG_SITE_WRITE=srm://fndca1.fnal.gov:8443//pnfs/fnal.gov/usr/fermigrid/vola tile GLOBUS_ERROR_VERBOSE=true LD_LIBRARY_PATH=/usr/local/grid/glexec-osg/lib64:/usr/local/grid/glexec-osg /lib: /usr/local/grid/prima/lib:/usr/local/grid/lcg/lib64:/usr/local/grid/lcg/lib :/usr /local/grid/glite/lib64:/usr/local/grid/glite/lib:/usr/local/grid/openldap/ lib:/ usr/local/grid/curl/lib:/usr/local/grid/jdk1.5/jre/lib/i386:/usr/local/grid /jdk1 5/jre/lib/i386/server:/usr/local/grid/jdk1.5/jre/lib/i386/client:/usr/loca l/gri d/globus/lib:/usr/local/grid/berkeley-db/lib:/usr/local/grid/expat/lib:/usr /loca l/vdt-1.10.1/apache/lib:/usr/local/vdt-1.10.1/MonaLisa/Service/VDTFarm/pgsq l/lib :/usr/local/vdt-1.10.1/glite/lib64:/usr/local/vdt-1.10.1/glite/lib:/usr/loc al/vd t-1.10.1/prima/lib:/usr/local/vdt-1.10.1/mysql/lib/mysql:/usr/local/vdt-1.1 0.1/j dk1.5/jre/lib/i386:/usr/local/vdt-1.10.1/jdk1.5/jre/lib/i386/server:/usr/lo cal/v dt-1.10.1/jdk1.5/jre/lib/i386/client:/usr/local/vdt-1.10.1/berkeley-db/lib: /usr/ local/vdt-1.10.1/expat/lib:/usr/local/vdt-1.10.1/globus/lib:/usr/local/vdt- 1.10. 1/apache/lib:/usr/local/vdt-1.10.1/MonaLisa/Service/VDTFarm/pgsql/lib:/usr/ local /vdt-1.10.1/glite/lib64:/usr/local/vdt-1.10.1/glite/lib:/usr/local/vdt-1.10 1/pr ima/lib:/usr/local/vdt-1.10.1/mysql/lib/mysql:/usr/local/vdt-1.10.1/jdk1.5/ jre/l ib/i386:/usr/local/vdt-1.10.1/jdk1.5/jre/lib/i386/server:/usr/local/vdt-1.1 0.1/j dk1.5/jre/lib/i386/client:/usr/local/vdt-1.10.1/berkeley-db/lib:/usr/local/ vdt-1 10.1/expat/lib: GPT_LOCATION=/usr/local/grid/gpt GLITE_LOCATION_TMP=/usr/local/grid/glite/tmp OSG_LOCATION=/usr/local/vdt-1.10.1 _CONDOR_HIGHPORT=65535 LIBPATH=/usr/local/grid/globus/lib:/usr/lib:/lib OSG_SITE_READ=dcap://fndca1.fnal.gov:24525//pnfs/fnal.gov/usr/fermigrid/vol atile GLOBUS_OPTIONS=-Xmx512M GRID_SECURITY_DIR=/usr/local/grid/globus/etc _CONDOR_ANCESTOR_3627=19245:1231880091:4291538751 _CONDOR_ANCESTOR_3624=3627:1226439981:2081533312 GLOBUS_GRAM_MYJOB_CONTACT=URLx-nexus://fermigridosg1.fnal.gov:40045/ X509_CADIR=/usr/local/grid/globus/TRUSTED_CA PATH=/usr/local/grid/prima/bin:/usr/local/grid/lcg/bin:/usr/local/grid/glit e/sbi n:/usr/local/grid/glite/bin:/usr/local/grid/dccp/bin:/usr/local/grid/srm-cl ient- lbnl/bin:/usr/local/grid/srm-client-fermi/sbin:/usr/local/grid/srm-client-f ermi/ bin:/usr/local/grid/curl/bin:/usr/local/grid/ant/bin:/usr/local/grid/jdk1.5 /bin: /usr/local/grid/gpt/sbin:/usr/local/grid/globus/bin:/usr/local/grid/globus/ sbin: /usr/local/grid/wget/bin:/usr/local/grid/logrotate/sbin:/usr/local/pacman-3 25/b in:/usr/local/grid/vdt/sbin:/usr/local/grid/vdt/bin:/usr/local/bin:/bin:/us r/bin :/bin:/usr/bin:/sbin:/usr/sbin:/grid/fermiapp/minos/minfarm/bin VDT_LOCATION=/usr/local/grid OSG_DATA=/grid/data OSG_APP=/grid/app PWD=/local/stage1/condor/execute/dir_19245 JAVA_HOME=/usr/local/grid/jdk1.5 GLOBUS_REMOTE_IO_URL=/grid/home/minospro/.globus/job/fermigridosg1.fnal.gov /1440 5.1231880033/remote_io_url VOMS_LOCATION=/usr/local/grid/glite GRID3_TMP_DIR=/grid/data _CONDOR_SLOT=4 CATALINA_OPTS=-Dorg.globus.wsrf.container.persistence.dir=/usr/local/grid/v dt-ap p-data/globus/persisted OSG_GRID=/usr/local/grid _CONDOR_LOWPORT=60000 GRID3_SITE_NAME=FNAL_CDFOSG_1 GRID3_APP_DIR=/grid/app OSG_HOSTNAME=fcdfosg1.fnal.gov GLITE_LOCATION_VAR=/usr/local/grid/glite/var HOME=/grid/home/minospro SHLVL=2 OSG_STORAGE_ELEMENT=True X509_USER_PROXY=/grid/home/minospro/.globus/job/fcdfosg1.fnal.gov/26011.123 18800 57/x509_up DYLD_LIBRARY_PATH=/usr/local/grid/globus/lib VDT_INSTALL_LOG=vdt-install.log OSG_DEFAULT_SE=FNAL_FERMIGRID_SE SCRATCH_DIRECTORY=/grid/home/minospro//gram_scratch_Dq5Zl8jDio LOGNAME=minospro OSG_SITE_NAME=FNAL_CDFOSG_1 SHLIB_PATH=/usr/local/grid/globus/lib VDT_POSTINSTALL_README=/usr/local/grid/post-install/README GLITE_LOCATION=/usr/local/grid/glite GRID3_BASE_DIR=/usr/local/vdt-1.10.1 GLOBUS_GRAM_JOB_CONTACT=https://fermigridosg1.fnal.gov:40043/14405/12318800 33/ PEGASUS_HOME=/usr/local/grid/pegasus PACMAN_LOCATION=/usr/local/pacman-3.25 _=/usr/bin/printenv OLDPWD=/grid/home/minospro/gram_scratch_qC7hrXYCFH -rw-r--r-- 1 root root 1606 Oct 13 09:07 /usr/local/grid/globus/TRUSTED_CA/e1fc e4e9.0 subject : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy/CN=limited proxy issuer : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy identity : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy type : unknown strength : 512 bits path : /grid/home/minospro/.globus/job/fcdfosg1.fnal.gov/26011.1231880057/x 509_up timeleft : 6044:02:01 === VO fermilab extension information === VO : fermilab subject : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284 issuer : /DC=org/DC=doegrids/OU=Services/CN=http/voms.fnal.gov attribute : /fermilab/minos/Role=Production/Capability=NULL attribute : /fermilab/Role=NULL/Capability=NULL attribute : /fermilab/minos/Role=NULL/Capability=NULL timeleft : 0:00:00 Requesting n13037001_0001_L010185N_D06_nccohbkg.reroot.root from DCACHE using sr mcp at 2009-01-13 14:54:52 SRM Configuration: debug=true gsissl=true help=false pushmode=false userproxy=true buffer_size=131072 tcp_buffer_size=0 streams_num=1 config_file=/usr/local/grid/srm-client-fermi/etc/config-2.xml glue_mapfile=/usr/local/grid/srm-client-fermi/conf/SRMServerV1.map webservice_path=srm/managerv1 webservice_protocol=https gsiftpclinet=globus-url-copy protocols_list=http,gsiftp save_config_file=null srmcphome=/usr/local/grid/srm-client-fermi urlcopy=sbin/urlcopy.sh x509_user_cert= x509_user_key= x509_user_proxy=/grid/home/minospro/.globus/job/fcdfosg1.fnal.gov/26011. 1231880057/x509_up x509_user_trusted_certificates=/usr/local/grid/globus/TRUSTED_CA globus_tcp_port_range=null gss_expected_name=null storagetype=permanent retry_num=20 retry_timeout=10000 wsdl_url=null use_urlcopy_script=false connect_to_wsdl=false delegate=true full_delegation=true server_mode=active srm_protocol_version=1 request_lifetime=86400 access latency=null overwrite mode=null priority=0 from[0]=srm://fndca1:8443/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_ 06/L010185N_nccohbkg/700/n13037001_0001_L010185N_D06_nccohbkg.reroot.root to=file:////local/stage1/condor/execute/dir_19245/n13037001_0001_L010185 N_D06_nccohbkg.reroot.root Tue Jan 13 14:54:54 CST 2009: starting SRMGetClient Tue Jan 13 14:54:54 CST 2009: In SRMClient ExpectedName: host Tue Jan 13 14:54:54 CST 2009: SRMClient(https,srm/managerv1,true) SRMClientV1 : user credentials are: /DC=org/DC=doegrids/OU=People/CN=Howard Rubi n 763284 SRMClientV1 : SRMClientV1 calling org.globus.axis.util.Util.registerTransport() SRMClientV1 : connecting to srm at httpg://stkendca2a.fnal.gov:8443/srm/managerv 1 Tue Jan 13 14:54:56 CST 2009: connected to server, obtaining proxy Tue Jan 13 14:54:56 CST 2009: got proxy of type class org.dcache.srm.client.SRMC lientV1 SRMClientV1 : get: surls[0]=""srm://fndca1:8443/pnfs/fnal.gov/usr/minos/mcin_da ta/near/daikon_06/L010185N_nccohbkg/700/n13037001_0001_L010185N_D06_nccohbk g.rer oot.root"" SRMClientV1 : get: protocols[0]=""gsiftp"" SRMClientV1 : get: protocols[1]=""dcap"" SRMClientV1 : get: protocols[2]=""http"" copy_jobs is empty Tue Jan 13 14:54:59 CST 2009: srm returned requestId = -2144105543 Tue Jan 13 14:54:59 CST 2009: sleeping 4 seconds ... Tue Jan 13 14:55:04 CST 2009: sleeping 4 seconds ... Tue Jan 13 14:55:08 CST 2009: sleeping 4 seconds ... Tue Jan 13 14:55:12 CST 2009: sleeping 4 seconds ... Tue Jan 13 14:55:17 CST 2009: sleeping 4 seconds ... Tue Jan 13 14:55:21 CST 2009: sleeping 7 seconds ... Tue Jan 13 14:55:28 CST 2009: sleeping 7 seconds ... Tue Jan 13 14:55:35 CST 2009: sleeping 7 seconds ... Tue Jan 13 14:55:43 CST 2009: sleeping 7 seconds ... Tue Jan 13 14:55:50 CST 2009: sleeping 7 seconds ... Tue Jan 13 14:55:57 CST 2009: sleeping 10 seconds ... Tue Jan 13 14:56:08 CST 2009: sleeping 10 seconds ... Tue Jan 13 14:56:18 CST 2009: sleeping 10 seconds ... Tue Jan 13 14:56:28 CST 2009: sleeping 10 seconds ... Tue Jan 13 14:56:39 CST 2009: sleeping 10 seconds ... Tue Jan 13 14:56:49 CST 2009: sleeping 13 seconds ... Tue Jan 13 14:57:02 CST 2009: sleeping 13 seconds ... Tue Jan 13 14:57:15 CST 2009: sleeping 13 seconds ... Tue Jan 13 14:57:29 CST 2009: sleeping 13 seconds ... Tue Jan 13 14:57:42 CST 2009: sleeping 13 seconds ... Tue Jan 13 14:57:55 CST 2009: sleeping 16 seconds ... Tue Jan 13 14:58:12 CST 2009: sleeping 16 seconds ... Tue Jan 13 14:58:28 CST 2009: sleeping 16 seconds ... Tue Jan 13 14:58:44 CST 2009: sleeping 16 seconds ... Tue Jan 13 14:59:00 CST 2009: sleeping 16 seconds ... copy_jobs is empty stopping copier Tue Jan 13 14:59:17 CST 2009: setting all remaining file statuses to ""Done"" Tue Jan 13 14:59:17 CST 2009: setting file request -2144105542 status to Done Tue Jan 13 14:59:17 CST 2009: set all file statuses to ""Done"" >> Unable to get n13037001_0001_L010185N_D06_nccohbkg.reroot.root from srm://fnd ca1:8443/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_06/L010185N_nccohbkg /700 << Exiting with status 1 printenv ls -l $OSG_GRID/globus/TRUSTED_CA/e1fce4e9.0 voms-proxy-info -all WARNING: Unable to verify signature! Server certificate possibly not installed. Error: Cannot find certificate of AC issuer for vo fermilab set +v Storage Resource Manager (SRM) implementation version 2.0.3 Copyright (c) 2002-2008 Fermi National Accelerator Laboratory Specification Version 2.0 by SRM Working Group (http://sdm.lbl.gov/srm-wg) Tue Jan 13 14:59:17 CST 2009: rs.state = Failed rs.error = at Tue Jan 13 14:54: 59 CST 2009 state Pending : created RequestFileStatus#-2144105542 failed with error:[ at Tue Jan 13 14:59:02 CST 20 09 state Failed : Pinning failed] Tue Jan 13 14:59:17 CST 2009: ====> fileStatus state ==Failed java.io.IOException: rs.state = Failed rs.error = at Tue Jan 13 14:54:59 CST 20 09 state Pending : created RequestFileStatus#-2144105542 failed with error:[ at Tue Jan 13 14:59:02 CST 20 09 state Failed : Pinning failed] at gov.fnal.srm.util.SRMGetClientV1.start(SRMGetClientV1.java:298) at gov.fnal.srm.util.SRMDispatcher.work(SRMDispatcher.java:795) at gov.fnal.srm.util.SRMDispatcher.main(SRMDispatcher.java:374) srm copy of at least one file failed or not completed rm: cannot remove `/n13037001_0001_L010185N_D06_nccohbkg.*': No such file or dir ectory"
12/23/2008	81	126731	Assigned	Medium	HICKS, STAN	BELLAVANCE, ANGELA	"Hello, This ticket should be directed to the dCache admins. I've been noticing that when CDF nodes get rebooted recently they have not been coming up with the proper number of max movers. I am also not sure if the job timeout settings are correct. Could someone please go through the config files and check the values? For the timeout settings, it should be something like: Job Timeout Manager WAN (lastAccess=43200;total=432000) default (lastAccess=259200;total=432000) For the max movers, I have compiled a list below. Thanks, Angela CellName@DomainName Max Max Max Max Max Max Restore Stores P2P-Ser P2P-Cli WAN default r-fcdfcache01-1@r-fcdfcache01-1Domain 16 1 4 2 4 48 r-fcdfcache01-2@r-fcdfcache01-2Domain 16 1 4 2 4 48 r-fcdfcache01-3@r-fcdfcache01-3Domain 16 1 4 2 4 48 r-fcdfcache01-4@r-fcdfcache01-4Domain 16 1 4 2 4 48 r-fcdfcache02-1@r-fcdfcache02-1Domain 16 1 4 2 4 48 r-fcdfcache02-2@r-fcdfcache02-2Domain 16 1 4 2 4 48 r-fcdfcache02-3@r-fcdfcache02-3Domain 16 1 4 2 4 48 r-fcdfcache02-4@r-fcdfcache02-4Domain 16 1 4 2 4 48 r-fcdfcache03-1@r-fcdfcache03-1Domain 16 1 4 2 4 48 r-fcdfcache03-2@r-fcdfcache03-2Domain 16 1 4 2 4 48 r-fcdfcache03-3@r-fcdfcache03-3Domain 16 1 4 2 4 48 r-fcdfcache03-4@r-fcdfcache03-4Domain 16 1 4 2 4 48 r-fcdfcache04-1@r-fcdfcache04-1Domain 16 1 4 2 4 48 r-fcdfcache04-2@r-fcdfcache04-2Domain 16 1 4 2 4 48 r-fcdfcache04-3@r-fcdfcache04-3Domain 16 1 4 2 4 48 r-fcdfcache04-4@r-fcdfcache04-4Domain 16 1 4 2 4 48 r-fcdfcache05-1@r-fcdfcache05-1Domain 16 1 4 2 4 48 r-fcdfcache05-2@r-fcdfcache05-2Domain 16 1 4 2 4 48 r-fcdfcache05-3@r-fcdfcache05-3Domain 16 1 4 2 4 48 r-fcdfcache05-4@r-fcdfcache05-4Domain 16 1 4 2 4 48 r-fcdfcache06-1@r-fcdfcache06-1Domain 16 1 4 2 4 48 r-fcdfcache06-2@r-fcdfcache06-2Domain 16 1 4 2 4 48 r-fcdfcache06-3@r-fcdfcache06-3Domain 16 1 4 2 4 48 r-fcdfcache06-4@r-fcdfcache06-4Domain 16 1 4 2 4 48 r-fcdfcache07-1@r-fcdfcache07-1Domain 16 1 4 2 4 48 r-fcdfcache07-2@r-fcdfcache07-2Domain 16 1 4 2 4 48 r-fcdfcache07-3@r-fcdfcache07-3Domain 16 1 4 2 4 48 r-fcdfcache07-4@r-fcdfcache07-4Domain 16 1 4 2 4 48 r-fcdfcache08-1@r-fcdfcache08-1Domain 16 1 4 2 4 48 r-fcdfcache08-2@r-fcdfcache08-2Domain 16 1 4 2 4 48 r-fcdfcache08-3@r-fcdfcache08-3Domain 16 1 4 2 4 48 r-fcdfcache08-4@r-fcdfcache08-4Domain 16 1 4 2 4 48 r-fcdfcache09-1@r-fcdfcache09-1Domain 16 1 4 2 4 48 r-fcdfcache09-2@r-fcdfcache09-2Domain 16 1 4 2 4 48 r-fcdfcache09-3@r-fcdfcache09-3Domain 16 1 4 2 4 48 r-fcdfcache09-4@r-fcdfcache09-4Domain 16 1 4 2 4 48 r-fcdfdata096-1@r-fcdfdata096-1Domain 16 1 4 2 4 48 r-fcdfdata096-2@r-fcdfdata096-2Domain 16 1 4 2 4 48 r-fcdfdata096-3@r-fcdfdata096-3Domain 16 1 4 2 4 48 r-fcdfdata097-1@r-fcdfdata097-1Domain 16 1 4 2 4 48 r-fcdfdata097-2@r-fcdfdata097-2Domain 16 1 4 2 4 48 r-fcdfdata097-3@r-fcdfdata097-3Domain 16 1 4 2 4 48 r-fcdfdata098-1@r-fcdfdata098-1Domain 16 1 4 2 4 48 r-fcdfdata098-2@r-fcdfdata098-2Domain 16 1 4 2 4 48 r-fcdfdata098-3@r-fcdfdata098-3Domain 16 1 4 2 4 48 r-fcdfdata100-1@r-fcdfdata100-1Domain 16 1 4 2 4 48 r-fcdfdata100-2@r-fcdfdata100-2Domain 16 1 4 2 4 48 r-fcdfdata100-3@r-fcdfdata100-3Domain 16 1 4 2 4 48 r-fcdfdata105-1@r-fcdfdata105-1Domain 16 1 4 2 4 48 r-fcdfdata105-2@r-fcdfdata105-2Domain 16 1 4 2 4 48 r-fcdfdata105-3@r-fcdfdata105-3Domain 16 1 4 2 4 48 r-fcdfdata107-1@r-fcdfdata107-1Domain 16 1 4 2 4 48 r-fcdfdata107-2@r-fcdfdata107-2Domain 16 1 4 2 4 48 r-fcdfdata107-3@r-fcdfdata107-3Domain 16 1 4 2 4 48 r-fcdfdata108-1@r-fcdfdata108-1Domain 16 1 4 2 4 48 r-fcdfdata108-2@r-fcdfdata108-2Domain 16 1 4 2 4 48 r-fcdfdata108-3@r-fcdfdata108-3Domain 16 1 4 2 4 48 r-fcdfdata109-1@r-fcdfdata109-1Domain 16 1 4 2 4 48 r-fcdfdata109-2@r-fcdfdata109-2Domain 16 1 4 2 4 48 r-fcdfdata109-3@r-fcdfdata109-3Domain 16 1 4 2 4 48 r-fcdfdata110-1@r-fcdfdata110-1Domain 16 1 4 2 4 48 r-fcdfdata110-2@r-fcdfdata110-2Domain 16 1 4 2 4 48 r-fcdfdata110-3@r-fcdfdata110-3Domain 16 1 4 2 4 48 r-fcdfdata111-1@r-fcdfdata111-1Domain 16 1 4 2 4 48 r-fcdfdata111-2@r-fcdfdata111-2Domain 16 1 4 2 4 48 r-fcdfdata111-3@r-fcdfdata111-3Domain 16 1 4 2 4 48 r-fcdfdata132-1@r-fcdfdata132-1Domain 12 2 4 2 4 48 r-fcdfdata132-2@r-fcdfdata132-2Domain 12 2 4 2 4 48 r-fcdfdata132-3@r-fcdfdata132-3Domain 12 2 4 2 4 48 r-fcdfdata132-4@r-fcdfdata132-4Domain 12 2 4 2 4 48 r-fcdfdata135-1@r-fcdfdata135-1Domain 12 2 4 2 4 48 r-fcdfdata135-2@r-fcdfdata135-2Domain 12 2 4 2 4 48 r-fcdfdata135-3@r-fcdfdata135-3Domain 12 2 4 2 4 48 r-fcdfdata135-4@r-fcdfdata135-4Domain 12 2 4 2 4 48 r-fcdfdata136-1@r-fcdfdata136-1Domain 12 2 4 2 4 48 r-fcdfdata136-2@r-fcdfdata136-2Domain 12 2 4 2 4 48 r-fcdfdata136-3@r-fcdfdata136-3Domain 12 2 4 2 4 48 r-fcdfdata136-4@r-fcdfdata136-4Domain 12 2 4 2 4 48 r-fcdfdata137-1@r-fcdfdata137-1Domain 12 2 4 2 4 48 r-fcdfdata137-2@r-fcdfdata137-2Domain 12 2 4 2 4 48 r-fcdfdata137-3@r-fcdfdata137-3Domain 12 2 4 2 4 48 r-fcdfdata137-4@r-fcdfdata137-4Domain 12 2 4 2 4 48 r-fcdfdata138-1@r-fcdfdata138-1Domain 12 2 4 2 4 48 r-fcdfdata138-2@r-fcdfdata138-2Domain 12 2 4 2 4 48 r-fcdfdata138-3@r-fcdfdata138-3Domain 12 2 4 2 4 48 r-fcdfdata138-4@r-fcdfdata138-4Domain 12 2 4 2 4 48 r-fcdfdata139-1@r-fcdfdata139-1Domain 12 2 4 2 4 48 r-fcdfdata139-2@r-fcdfdata139-2Domain 12 2 4 2 4 48 r-fcdfdata139-3@r-fcdfdata139-3Domain 12 2 4 2 4 48 r-fcdfdata139-4@r-fcdfdata139-4Domain 12 2 4 2 4 48 r-fcdfdata144-1@r-fcdfdata144-1Domain 12 2 4 2 4 48 r-fcdfdata144-2@r-fcdfdata144-2Domain 12 2 4 2 4 48 r-fcdfdata144-3@r-fcdfdata144-3Domain 12 2 4 2 4 48 r-fcdfdata144-4@r-fcdfdata144-4Domain 12 2 4 2 4 48 r-fcdfdata145-1@r-fcdfdata145-1Domain 12 2 4 2 4 48 r-fcdfdata145-2@r-fcdfdata145-2Domain 12 2 4 2 4 48 r-fcdfdata145-3@r-fcdfdata145-3Domain 12 2 4 2 4 48 r-fcdfdata145-4@r-fcdfdata145-4Domain 12 2 4 2 4 48 r-fcdfdata146-1@r-fcdfdata146-1Domain 12 2 4 2 4 48 r-fcdfdata146-2@r-fcdfdata146-2Domain 12 2 4 2 4 48 r-fcdfdata146-3@r-fcdfdata146-3Domain 12 2 4 2 4 48 r-fcdfdata146-4@r-fcdfdata146-4Domain 12 2 4 2 4 48 r-fcdfdata147-1@r-fcdfdata147-1Domain 12 2 4 2 4 48 r-fcdfdata147-2@r-fcdfdata147-2Domain 12 2 4 2 4 48 r-fcdfdata147-3@r-fcdfdata147-3Domain 12 2 4 2 4 48 r-fcdfdata147-4@r-fcdfdata147-4Domain 12 2 4 2 4 48 r-fcdfdata148-1@r-fcdfdata148-1Domain 12 2 4 2 4 48 r-fcdfdata148-2@r-fcdfdata148-2Domain 12 2 4 2 4 48 r-fcdfdata148-3@r-fcdfdata148-3Domain 12 2 4 2 4 48 r-fcdfdata148-4@r-fcdfdata148-4Domain 12 2 4 2 4 48 r-fcdfdata149-1@r-fcdfdata149-1Domain 12 2 4 2 4 48 r-fcdfdata149-2@r-fcdfdata149-2Domain 12 2 4 2 4 48 r-fcdfdata149-3@r-fcdfdata149-3Domain 12 2 4 2 4 48 r-fcdfdata149-4@r-fcdfdata149-4Domain 12 2 4 2 4 48 r-fcdfdata154-1@r-fcdfdata154-1Domain 12 2 4 2 4 48 r-fcdfdata154-2@r-fcdfdata154-2Domain 12 2 4 2 4 48 r-fcdfdata154-3@r-fcdfdata154-3Domain 12 2 4 2 4 48 r-fcdfdata154-4@r-fcdfdata154-4Domain 12 2 4 2 4 48 r-fcdfdata155-1@r-fcdfdata155-1Domain 12 2 4 2 2 48 r-fcdfdata155-2@r-fcdfdata155-2Domain 12 2 4 2 2 48 r-fcdfdata155-3@r-fcdfdata155-3Domain 12 2 4 2 2 48 r-fcdfdata155-4@r-fcdfdata155-4Domain 12 2 4 2 2 48 r-fcdfdata316-1@r-fcdfdata316-1Domain 12 2 4 2 2 48 r-fcdfdata316-2@r-fcdfdata316-2Domain 12 2 4 2 2 48 r-fcdfdata316-3@r-fcdfdata316-3Domain 12 2 4 2 2 48 r-fcdfdata316-4@r-fcdfdata316-4Domain 12 2 4 2 2 48 r-fcdfdata317-1@r-fcdfdata317-1Domain 12 2 4 2 2 48 r-fcdfdata317-2@r-fcdfdata317-2Domain 12 2 4 2 2 48 r-fcdfdata317-3@r-fcdfdata317-3Domain 12 2 4 2 2 48 r-fcdfdata317-4@r-fcdfdata317-4Domain 12 2 4 2 2 48 r-fcdfdata318-1@r-fcdfdata318-1Domain 12 2 4 2 2 48 r-fcdfdata318-2@r-fcdfdata318-2Domain 12 2 4 2 2 48 r-fcdfdata318-3@r-fcdfdata318-3Domain 12 2 4 2 2 48 r-fcdfdata318-4@r-fcdfdata318-4Domain 12 2 4 2 2 48 r-fcdfdata320-1@r-fcdfdata320-1Domain 12 2 4 2 2 48 r-fcdfdata320-2@r-fcdfdata320-2Domain 12 2 4 2 2 48 r-fcdfdata320-3@r-fcdfdata320-3Domain 12 2 4 2 2 48 r-fcdfdata320-4@r-fcdfdata320-4Domain 12 2 4 2 2 48 r-fcdfdata323-1@r-fcdfdata323-1Domain 48 2 4 2 2 30 r-fcdfdata324-1@r-fcdfdata324-1Domain 48 2 4 2 2 30 r-fcdfdata325-1@r-fcdfdata325-1Domain 48 2 4 2 2 30 r-fcdfdata327-1@r-fcdfdata327-1Domain 48 2 4 2 2 30 w-fcdfdata111-4@w-fcdfdata111-4Domain 1 64 4 2 2 64 w-fcdfdata157-1@w-fcdfdata157-1Domain 1 64 4 2 2 64 w-fcdfdata157-2@w-fcdfdata157-2Domain 1 64 4 2 2 64 w-fcdfdata157-3@w-fcdfdata157-3Domain 1 64 4 2 2 64 -- Dr. Angela Bellavance Fermilab Computing Division Running Experiments Operations 630-840-2284 bellavan@fnal.gov"
12/19/2008	85	126634	Assigned	Medium	BEAGLE, GLENN	RUBIN, HOWARD	"Between 16:03 and 16:05 69 srm reads failed as below: Storage Resource Manager (SRM) implementation version 2.0.3 Copyright (c) 2002-2008 Fermi National Accelerator Laboratory Specification Version 2.0 by SRM Working Group (http://sdm.lbl.gov/srm-wg) Thu Dec 18 16:05:08 CST 2008: rs.state = Failed rs.error = at Thu Dec 18 13:00: 36 CST 2008 state Pending : created RequestFileStatus#-2144176028 failed with error:[ at Thu Dec 18 16:03:08 CST 20 08 state Failed : Pinning failed] Thu Dec 18 16:05:08 CST 2008: ====> fileStatus state ==Failed java.io.IOException: rs.state = Failed rs.error = at Thu Dec 18 13:00:36 CST 20 08 state Pending : created RequestFileStatus#-2144176028 failed with error:[ at Thu Dec 18 16:03:08 CST 20 08 state Failed : Pinning failed] at gov.fnal.srm.util.SRMGetClientV1.start(SRMGetClientV1.java:298) at gov.fnal.srm.util.SRMDispatcher.work(SRMDispatcher.java:795) at gov.fnal.srm.util.SRMDispatcher.main(SRMDispatcher.java:374) srm copy of at least one file failed or not completed A complete transcript of the job, including srm debug information, job environment, proxy information, etc. will be found at /minos/data2/minfarm/farmtest/mclogs/dogwoodtest4/near/daikon_04/L010185N/7 06/n13037064_0009_L010185N_D04.0.dogwoodtest4.log.gz"
12/13/2008	91	126305	Assigned	Medium	MESSER, TIM	KREYMER, ARTHUR	"Please authorize Kregg to write to fermigrid/volatile/minos , using /DC=org/DC=doegrids/OU=People/CN=Kregg Arms 875233 User/group mapping should probably be arms/e875 Kregg intends to write from Teragrid sites, probably with Grid FTP."
12/8/2008	96	126014	Assigned	Medium	MESSER, TIM	KREYMER, ARTHUR	"One of the Minos users has reported three files to be unavailable via dccp. Their metadata looks good, and the first two are in recent pool listings. I have verified that a 'dccp' of the first of these gets stuck indefinitely : Under /pnfs/fnal.gov/usr/minos/reco_far/cedar_phy_bhcurv/.bcnd_data/2005-08/ File Pool listing F00032507_0004.spill.bcnd.cedar_phy_bhcurv.0.root r-stkendca16a-6 F00032507_0005.spill.bcnd.cedar_phy_bhcurv.0.root r-stkendca9a-2 F00032507_0006.spill.bcnd.cedar_phy_bhcurv.0.root MINOS26 > ./dccptest F00032507_0004.spill.bcnd.cedar_phy_bhcurv.0.root Connected in 0.00s. [Mon Dec 8 09:17:36 2008] Going to open file dcap://fndca1.fnal.gov:24136/pnfs/fnal.gov/usr/minos/reco_far/cedar_phy_bhc urv/.bcnd_data/2005-08/F00032507_0004.spill.bcnd.cedar_phy_bhcurv.0.root in cache. ( still stuck as of 15:00 ) Odd, the entry in the Lazy Restore Queue for the first file is dated 11.28, and indicates a pool to pool transfer to a write queue ??? 000F0000000000000730EA60 0.0.0.0/0.0.0.0-*/* r-stkendca16a-6->w-stkendca10a-6 11.28 19:32:25 136 0 Pool2Pool 11.28 19:32:25 /pnfs/fnal.gov/usr/minos/reco_far/cedar_phy_bhcurv/.bcnd_data/2005-08/F0003 2507_0004.spill.bcnd.cedar_phy_bhcurv.0.root"
12/3/2008	101	125734	Work In Progress	Medium	JONES, TERRY	CAVANAUGH, STEVEN	"Hi, I am running grid jobs which dccp a number of files which have all been prestaged. The dccp seems to hang on certain files and not on others, and the behavior is reproducible on minos machines For example, on minos08 The following command copies immediately dccp /pnfs/minos//reco_far/cedar_phy_bhcurv/.bcnd_data/2005-04/F00030617_0001.spill.bcnd.cedar_phy_bhcurv.0.root . But the following just hangs (this behavior resulted in no response in excess of 4 hours before I killed the effected jobs) dccp /pnfs/minos//reco_far/cedar_phy_bhcurv/.bcnd_data/2005-04/F00030617_0002.spill.bcnd.cedar_phy_bhcurv.0.root . This issue is repeatable on a per file basis Both of these files have been prestaged, so I am unaware of any difference between them. This behavior was seen yesterday as well as today, but the files which are having difficulty have changed sometime over the night. Thanks, Steve "
11/24/2008	110	125372	Assigned	Medium	SZMUKSTA, GEORGE	GENSER, KRZYSZTOF	"Dear MSS Admins, Could you enable the siddet group (unix group siddet) to use enstore/dCache system? The group would like to use enstore/dCache from from ilcsim, ilcsim2 and fermigrid worker nodes. The amount of enstore space would be 10TB initially with a rough estimate of the anticipated growth rate of about 15TB/year. The required dCache space would be about 2TB with dCache writes enabled if possible. The contact people would be Adam Para ,Hans Wenzel, and myself. The initial list of users with the read/write access would be: para wenzel genser ngraf jhill adriutti Please let us know if there are any questions which we would need to answer at this time, Thank you, Krzysztof"
11/22/2008	112	125350	Assigned	Medium	SZMUKSTA, GEORGE	KREYMER, ARTHUR	"dcache-admin - It appears that no Minos raw data files have been written to tape from the RawDataWritePools group since Thursday 19 November. Looking at http://fndca3a.fnal.gov:2288/usageInfo, I see many precious files in write pools across the whole DCache system, over half of the capacity of some pools. Most of the writes I see active in Enstore are for one file family. minos.mcout_cedar_phy_linfix_near_daikon_00_cand.cpio_odc There are also problems with file restores from tape. There are hundreds of file restores pending for the readPools group, but all are directed to one pool, r-stkendca15a I will ask the Minos team to shut down production processing, to help take some of the load off the system until these problems are resolved."
11/20/2008	114	125243	Work In Progress	High	SZMUKSTA, GEORGE	NGOP, NGOP	"ServiceType: 8to17by5 ID: Ping.STKEN Description: Ping failed for multiple nodes!Type:nodes;Names:needfile;stkensrv7 are unpingable for at least 10 min"
11/19/2008	115	125167	Assigned	Medium	SZMUKSTA, GEORGE	LAMMEL, STEPHAN	"Could you send me the list of nodes that can mount stkensrv1:/cdf. Thanks, cheers, Stephan"
11/19/2008	115	125152	Assigned	Medium	SZMUKSTA, GEORGE	RUBIN, HOWARD	"Several hundred jobs (out of several thousand submitted over the past day with the same scripts) have failed with SRM errors. Below is the full stdout and stderr from one job. Immediately is an ls of the file in question indicating there's nothing wrong with it. fnpcsrv1% ls -l /pnfs/minos/mcin_data/near/daikon_00/L010185N/180/n13011802_0009_L010185N_D 00.reroot.root -rw-r--r-- 1 rhatcher numi 363852469 Jun 28 2007 /pnfs/minos/mcin_data/near/daikon_00/L010185N/180/n13011802_0009_L010185N_D 00.reroot.root Beginning job ana_mc on fnpc258 on Wed Nov 19 at 07:08:18 CST MANPATH=/usr/local/grid/globus/man::/usr/local/grid/vdt/man:/usr/local/grid /perl /man:/usr/local/grid/expat/man:/usr/local/grid/logrotate/man:/usr/local/gri d/wge t/man:/usr/local/grid/jdk1.5/man:/usr/local/grid/curl/share/man:/usr/local/ grid/ glite/share/man:/usr/local/grid/lcg/man:/usr/local/grid/prima/man GRID3_DATA_DIR=/grid/data GRID3_TMP_WN_DIR=/local/stage1 PAC_ANCHOR=/usr/local/grid OSG_GLEXEC_LOCATION=/opt/glite/sbin/glexec VOMS_USERCONF=/usr/local/grid/glite/etc/vomses GLOBUS_PATH=/usr/local/grid/globus GLOBUS_LOCATION=/usr/local/grid/globus PERL5LIB=/usr/local/grid/vdt/lib:/usr/local/grid/perl/lib/5.8.0:/usr/local/ grid/ perl/lib/5.8.0/x86_64-linux-thread-multi:/usr/local/grid/perl/lib/site_perl /5.8. 0:/usr/local/grid/perl/lib/site_perl/5.8.0/x86_64-linux-thread-multi: _CONDOR_SCRATCH_DIR=/local/stage1/condor/execute/dir_8602 X509_CERT_DIR=/usr/local/grid/globus/TRUSTED_CA OSG_WN_TMP=/local/stage1 ANT_HOME=/usr/local/grid/ant GLITE_LOCATION_LOG=/usr/local/grid/glite/log OSG_JOB_CONTACT=fnpcfg1.fnal.gov/jobmanager-condor OSG_SQUID_LOCATION=squid.fnal.gov SRM_CONFIG=/usr/local/grid/srm-client-fermi/etc/config-2.xml OSG_SITE_WRITE=srm://fndca1.fnal.gov:8443//pnfs/fnal.gov/usr/fermigrid/vola tile GLOBUS_ERROR_VERBOSE=true LD_LIBRARY_PATH=/usr/local/grid/glexec-osg/lib64:/usr/local/grid/glexec-osg /lib: /usr/local/grid/prima/lib:/usr/local/grid/lcg/lib64:/usr/local/grid/lcg/lib :/usr /local/grid/glite/lib64:/usr/local/grid/glite/lib:/usr/local/grid/openldap/ lib:/ usr/local/grid/curl/lib:/usr/local/grid/jdk1.5/jre/lib/i386:/usr/local/grid /jdk1 5/jre/lib/i386/server:/usr/local/grid/jdk1.5/jre/lib/i386/client:/usr/loca l/gri d/globus/lib:/usr/local/grid/berkeley-db/lib:/usr/local/grid/expat/lib:/usr /loca l/vdt-1.10.1/apache/lib:/usr/local/vdt-1.10.1/MonaLisa/Service/VDTFarm/pgsq l/lib :/usr/local/vdt-1.10.1/glite/lib64:/usr/local/vdt-1.10.1/glite/lib:/usr/loc al/vd t-1.10.1/prima/lib:/usr/local/vdt-1.10.1/mysql/lib/mysql:/usr/local/vdt-1.1 0.1/j dk1.5/jre/lib/i386:/usr/local/vdt-1.10.1/jdk1.5/jre/lib/i386/server:/usr/lo cal/v dt-1.10.1/jdk1.5/jre/lib/i386/client:/usr/local/vdt-1.10.1/berkeley-db/lib: /usr/ local/vdt-1.10.1/expat/lib:/usr/local/vdt-1.10.1/globus/lib:/usr/local/vdt- 1.10. 1/apache/lib:/usr/local/vdt-1.10.1/MonaLisa/Service/VDTFarm/pgsql/lib:/usr/ local /vdt-1.10.1/glite/lib64:/usr/local/vdt-1.10.1/glite/lib:/usr/local/vdt-1.10 1/pr ima/lib:/usr/local/vdt-1.10.1/mysql/lib/mysql:/usr/local/vdt-1.10.1/jdk1.5/ jre/l ib/i386:/usr/local/vdt-1.10.1/jdk1.5/jre/lib/i386/server:/usr/local/vdt-1.1 0.1/j dk1.5/jre/lib/i386/client:/usr/local/vdt-1.10.1/berkeley-db/lib:/usr/local/ vdt-1 10.1/expat/lib: GPT_LOCATION=/usr/local/grid/gpt GLITE_LOCATION_TMP=/usr/local/grid/glite/tmp OSG_LOCATION=/usr/local/vdt-1.10.1 _CONDOR_HIGHPORT=65535 LIBPATH=/usr/local/grid/globus/lib:/usr/lib:/lib OSG_SITE_READ=dcap://fndca1.fnal.gov:24525//pnfs/fnal.gov/usr/fermigrid/vol atile GLOBUS_OPTIONS=-Xmx512M GRID_SECURITY_DIR=/usr/local/grid/globus/etc GLOBUS_GRAM_MYJOB_CONTACT=URLx-nexus://fermigridosg1.fnal.gov:40044/ X509_CADIR=/usr/local/grid/globus/TRUSTED_CA PATH=/usr/local/grid/prima/bin:/usr/local/grid/lcg/bin:/usr/local/grid/glit e/sbi n:/usr/local/grid/glite/bin:/usr/local/grid/dccp/bin:/usr/local/grid/srm-cl ient- lbnl/bin:/usr/local/grid/srm-client-fermi/sbin:/usr/local/grid/srm-client-f ermi/ bin:/usr/local/grid/curl/bin:/usr/local/grid/ant/bin:/usr/local/grid/jdk1.5 /bin: /usr/local/grid/gpt/sbin:/usr/local/grid/globus/bin:/usr/local/grid/globus/ sbin: /usr/local/grid/wget/bin:/usr/local/grid/logrotate/sbin:/usr/local/pacman-3 25/b in:/usr/local/grid/vdt/sbin:/usr/local/grid/vdt/bin:/usr/local/bin:/bin:/us r/bin :/bin:/usr/bin:/sbin:/usr/sbin:/grid/fermiapp/minos/minfarm/bin _CONDOR_ANCESTOR_3872=3880:1226056784:2659204352 _CONDOR_ANCESTOR_8602=8603:1227100098:880900608 VDT_LOCATION=/usr/local/grid OSG_DATA=/grid/data OSG_APP=/grid/app PWD=/local/stage1/condor/execute/dir_8602 JAVA_HOME=/usr/local/grid/jdk1.5 GLOBUS_REMOTE_IO_URL=/grid/home/minospro/.globus/job/fermigridosg1.fnal.gov /2704 5.1227069458/remote_io_url VOMS_LOCATION=/usr/local/grid/glite GRID3_TMP_DIR=/grid/data _CONDOR_SLOT=2 CATALINA_OPTS=-Dorg.globus.wsrf.container.persistence.dir=/usr/local/grid/v dt-ap p-data/globus/persisted OSG_GRID=/usr/local/grid _CONDOR_LOWPORT=60000 GRID3_SITE_NAME=FNAL_GPGRID_2 GRID3_APP_DIR=/grid/app OSG_HOSTNAME=fnpcfg1.fnal.gov GLITE_LOCATION_VAR=/usr/local/grid/glite/var HOME=/grid/home/minospro SHLVL=2 OSG_STORAGE_ELEMENT=True X509_USER_PROXY=/grid/home/minospro/.globus/job/fnpcfg1.fnal.gov/2129.12271 00085 /x509_up DYLD_LIBRARY_PATH=/usr/local/grid/globus/lib VDT_INSTALL_LOG=vdt-install.log OSG_DEFAULT_SE=FNAL_FERMIGRID_SE SCRATCH_DIRECTORY=/grid/home/minospro//gram_scratch_ZEJxgrAgCd LOGNAME=minospro OSG_SITE_NAME=FNAL_GPGRID_2 SHLIB_PATH=/usr/local/grid/globus/lib VDT_POSTINSTALL_README=/usr/local/grid/post-install/README GLITE_LOCATION=/usr/local/grid/glite GRID3_BASE_DIR=/usr/local/vdt-1.10.1 GLOBUS_GRAM_JOB_CONTACT=https://fermigridosg1.fnal.gov:40021/27045/12270694 58/ PEGASUS_HOME=/usr/local/grid/pegasus PACMAN_LOCATION=/usr/local/pacman-3.25 _CONDOR_ANCESTOR_3880=8602:1227100098:302827716 _=/usr/bin/printenv OLDPWD=/grid/home/minospro/gram_scratch_SQpqboFWP2 -rw-r--r-- 1 root root 1606 Sep 29 12:54 /usr/local/grid/globus/TRUSTED_CA/e1fc e4e9.0 subject : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy/CN=limited proxy issuer : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy identity : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284/CN=proxy/CN=lim ited proxy type : unknown strength : 512 bits path : /grid/home/minospro/.globus/job/fnpcfg1.fnal.gov/2129.1227100085/x50 9_up timeleft : 7371:48:35 === VO fermilab extension information === VO : fermilab subject : /DC=org/DC=doegrids/OU=People/CN=Howard Rubin 763284 issuer : /DC=org/DC=doegrids/OU=Services/CN=http/voms.fnal.gov attribute : /fermilab/minos/Role=Production/Capability=NULL attribute : /fermilab/Role=NULL/Capability=NULL attribute : /fermilab/minos/Role=NULL/Capability=NULL timeleft : 0:00:00 Requesting n13011802_0009_L010185N_D00.reroot.root from DCACHE using srmcp at 20 08-11-19 07:08:18 SRM Configuration: debug=true gsissl=true help=false pushmode=false userproxy=true buffer_size=131072 tcp_buffer_size=0 streams_num=1 config_file=/usr/local/grid/srm-client-fermi/etc/config-2.xml glue_mapfile=/usr/local/grid/srm-client-fermi/conf/SRMServerV1.map webservice_path=srm/managerv1 webservice_protocol=https gsiftpclinet=globus-url-copy protocols_list=http,gsiftp save_config_file=null srmcphome=/usr/local/grid/srm-client-fermi urlcopy=sbin/urlcopy.sh x509_user_cert= x509_user_key= x509_user_proxy=/grid/home/minospro/.globus/job/fnpcfg1.fnal.gov/2129.12 27100085/x509_up x509_user_trusted_certificates=/usr/local/grid/globus/TRUSTED_CA globus_tcp_port_range=null gss_expected_name=null storagetype=permanent retry_num=20 retry_timeout=10000 wsdl_url=null use_urlcopy_script=false connect_to_wsdl=false delegate=true full_delegation=true server_mode=active srm_protocol_version=1 request_lifetime=86400 access latency=null overwrite mode=null priority=0 from[0]=srm://fndca1:8443/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_ 00/L010185N/180/n13011802_0009_L010185N_D00.reroot.root to=file:////local/stage1/condor/execute/dir_8602/n13011802_0009_L010185N _D00.reroot.root Wed Nov 19 07:08:18 CST 2008: starting SRMGetClient Wed Nov 19 07:08:18 CST 2008: In SRMClient ExpectedName: host Wed Nov 19 07:08:18 CST 2008: SRMClient(https,srm/managerv1,true) SRMClientV1 : user credentials are: /DC=org/DC=doegrids/OU=People/CN=Howard Rubi n 763284 SRMClientV1 : SRMClientV1 calling org.globus.axis.util.Util.registerTransport() SRMClientV1 : connecting to srm at httpg://stkendca2a.fnal.gov:8443/srm/managerv 1 Wed Nov 19 07:08:19 CST 2008: connected to server, obtaining proxy Wed Nov 19 07:08:19 CST 2008: got proxy of type class org.dcache.srm.client.SRMC lientV1 SRMClientV1 : get: surls[0]=""srm://fndca1:8443/pnfs/fnal.gov/usr/minos/mcin_da ta/near/daikon_00/L010185N/180/n13011802_0009_L010185N_D00.reroot.root"" SRMClientV1 : get: protocols[0]=""gsiftp"" SRMClientV1 : get: protocols[1]=""dcap"" SRMClientV1 : get: protocols[2]=""http"" copy_jobs is empty Wed Nov 19 07:08:26 CST 2008: srm returned requestId = -2144281129 Wed Nov 19 07:08:26 CST 2008: sleeping 4 seconds ... Wed Nov 19 07:08:43 CST 2008: sleeping 4 seconds ... Wed Nov 19 07:09:47 CST 2008: sleeping 4 seconds ... Wed Nov 19 07:10:00 CST 2008: sleeping 4 seconds ... Wed Nov 19 07:10:08 CST 2008: sleeping 4 seconds ... Wed Nov 19 07:10:21 CST 2008: sleeping 7 seconds ... Wed Nov 19 07:10:39 CST 2008: sleeping 7 seconds ... Wed Nov 19 07:10:51 CST 2008: sleeping 7 seconds ... Wed Nov 19 07:11:25 CST 2008: sleeping 7 seconds ... Wed Nov 19 07:11:36 CST 2008: sleeping 7 seconds ... Wed Nov 19 07:11:52 CST 2008: sleeping 10 seconds ... Wed Nov 19 07:12:09 CST 2008: sleeping 10 seconds ... Wed Nov 19 07:12:26 CST 2008: sleeping 10 seconds ... Wed Nov 19 07:12:43 CST 2008: sleeping 10 seconds ... Wed Nov 19 07:13:00 CST 2008: sleeping 10 seconds ... Wed Nov 19 07:13:13 CST 2008: sleeping 13 seconds ... Wed Nov 19 07:13:29 CST 2008: sleeping 13 seconds ... Wed Nov 19 07:13:43 CST 2008: sleeping 13 seconds ... Wed Nov 19 07:13:59 CST 2008: sleeping 13 seconds ... Wed Nov 19 07:14:15 CST 2008: sleeping 13 seconds ... Wed Nov 19 07:14:31 CST 2008: sleeping 16 seconds ... Wed Nov 19 07:14:50 CST 2008: sleeping 16 seconds ... Wed Nov 19 07:15:10 CST 2008: sleeping 16 seconds ... Wed Nov 19 07:15:32 CST 2008: sleeping 16 seconds ... Wed Nov 19 07:15:52 CST 2008: sleeping 16 seconds ... Wed Nov 19 07:16:09 CST 2008: sleeping 19 seconds ... Wed Nov 19 07:16:28 CST 2008: sleeping 19 seconds ... Wed Nov 19 07:16:49 CST 2008: sleeping 19 seconds ... Wed Nov 19 07:17:09 CST 2008: sleeping 19 seconds ... Wed Nov 19 07:17:29 CST 2008: sleeping 19 seconds ... Wed Nov 19 07:17:49 CST 2008: sleeping 22 seconds ... Wed Nov 19 07:18:11 CST 2008: sleeping 22 seconds ... Wed Nov 19 07:18:34 CST 2008: sleeping 22 seconds ... Wed Nov 19 07:18:57 CST 2008: sleeping 22 seconds ... Wed Nov 19 07:19:20 CST 2008: sleeping 22 seconds ... Wed Nov 19 07:19:42 CST 2008: sleeping 25 seconds ... Wed Nov 19 07:20:08 CST 2008: sleeping 25 seconds ... Wed Nov 19 07:20:33 CST 2008: sleeping 25 seconds ... Wed Nov 19 07:21:00 CST 2008: sleeping 25 seconds ... Wed Nov 19 07:21:26 CST 2008: sleeping 25 seconds ... Wed Nov 19 07:21:52 CST 2008: sleeping 28 seconds ... Wed Nov 19 07:22:20 CST 2008: sleeping 28 seconds ... Wed Nov 19 07:22:49 CST 2008: sleeping 28 seconds ... Wed Nov 19 07:23:19 CST 2008: sleeping 28 seconds ... Wed Nov 19 07:23:48 CST 2008: sleeping 28 seconds ... Wed Nov 19 07:24:18 CST 2008: sleeping 31 seconds ... Wed Nov 19 07:24:49 CST 2008: sleeping 31 seconds ... Wed Nov 19 07:25:24 CST 2008: sleeping 31 seconds ... Wed Nov 19 07:25:55 CST 2008: sleeping 31 seconds ... Wed Nov 19 07:26:26 CST 2008: sleeping 31 seconds ... Wed Nov 19 07:27:12 CST 2008: sleeping 34 seconds ... Wed Nov 19 07:27:50 CST 2008: sleeping 34 seconds ... Wed Nov 19 07:28:38 CST 2008: sleeping 34 seconds ... Wed Nov 19 07:29:19 CST 2008: sleeping 34 seconds ... Wed Nov 19 07:29:53 CST 2008: sleeping 34 seconds ... Wed Nov 19 07:30:35 CST 2008: sleeping 37 seconds ... Wed Nov 19 07:31:13 CST 2008: sleeping 37 seconds ... Wed Nov 19 07:31:52 CST 2008: sleeping 37 seconds ... Wed Nov 19 07:32:31 CST 2008: sleeping 37 seconds ... Wed Nov 19 07:33:13 CST 2008: sleeping 37 seconds ... Wed Nov 19 07:33:52 CST 2008: sleeping 40 seconds ... Wed Nov 19 07:34:35 CST 2008: sleeping 40 seconds ... Wed Nov 19 07:35:19 CST 2008: sleeping 40 seconds ... Wed Nov 19 07:36:01 CST 2008: sleeping 40 seconds ... Wed Nov 19 07:36:43 CST 2008: sleeping 40 seconds ... Wed Nov 19 07:37:24 CST 2008: sleeping 43 seconds ... Wed Nov 19 07:38:08 CST 2008: sleeping 43 seconds ... Wed Nov 19 07:38:51 CST 2008: sleeping 43 seconds ... Wed Nov 19 07:39:36 CST 2008: sleeping 43 seconds ... Wed Nov 19 07:40:19 CST 2008: sleeping 43 seconds ... Wed Nov 19 07:41:03 CST 2008: sleeping 46 seconds ... Wed Nov 19 07:41:50 CST 2008: sleeping 46 seconds ... Wed Nov 19 07:42:36 CST 2008: sleeping 46 seconds ... Wed Nov 19 07:43:23 CST 2008: sleeping 46 seconds ... Wed Nov 19 07:44:09 CST 2008: sleeping 46 seconds ... Wed Nov 19 07:44:57 CST 2008: sleeping 49 seconds ... Wed Nov 19 07:45:46 CST 2008: sleeping 49 seconds ... Wed Nov 19 07:46:36 CST 2008: sleeping 49 seconds ... Wed Nov 19 07:47:26 CST 2008: sleeping 49 seconds ... Wed Nov 19 07:48:15 CST 2008: sleeping 49 seconds ... Wed Nov 19 07:49:04 CST 2008: sleeping 52 seconds ... Wed Nov 19 07:49:57 CST 2008: sleeping 52 seconds ... Wed Nov 19 07:50:49 CST 2008: sleeping 52 seconds ... Wed Nov 19 07:51:42 CST 2008: sleeping 52 seconds ... Wed Nov 19 07:52:34 CST 2008: sleeping 52 seconds ... Wed Nov 19 07:53:26 CST 2008: sleeping 55 seconds ... Wed Nov 19 07:54:22 CST 2008: sleeping 55 seconds ... Wed Nov 19 07:55:17 CST 2008: sleeping 55 seconds ... Wed Nov 19 07:56:13 CST 2008: sleeping 55 seconds ... Wed Nov 19 07:57:08 CST 2008: sleeping 55 seconds ... Wed Nov 19 07:58:03 CST 2008: sleeping 58 seconds ... Wed Nov 19 07:59:03 CST 2008: sleeping 58 seconds ... Wed Nov 19 08:00:01 CST 2008: sleeping 58 seconds ... Wed Nov 19 08:00:59 CST 2008: sleeping 58 seconds ... Wed Nov 19 08:01:58 CST 2008: sleeping 58 seconds ... Wed Nov 19 08:02:56 CST 2008: sleeping 61 seconds ... Wed Nov 19 08:03:58 CST 2008: sleeping 61 seconds ... Wed Nov 19 08:05:00 CST 2008: sleeping 61 seconds ... Wed Nov 19 08:06:01 CST 2008: sleeping 61 seconds ... Wed Nov 19 08:07:02 CST 2008: sleeping 61 seconds ... Wed Nov 19 08:08:05 CST 2008: sleeping 64 seconds ... Wed Nov 19 08:09:09 CST 2008: sleeping 64 seconds ... Wed Nov 19 08:10:13 CST 2008: sleeping 64 seconds ... Wed Nov 19 08:11:18 CST 2008: sleeping 64 seconds ... Wed Nov 19 08:12:22 CST 2008: sleeping 64 seconds ... Wed Nov 19 08:13:26 CST 2008: sleeping 67 seconds ... Wed Nov 19 08:14:34 CST 2008: sleeping 67 seconds ... Wed Nov 19 08:15:41 CST 2008: sleeping 67 seconds ... Wed Nov 19 08:16:49 CST 2008: sleeping 67 seconds ... Wed Nov 19 08:17:56 CST 2008: sleeping 67 seconds ... Wed Nov 19 08:19:03 CST 2008: sleeping 70 seconds ... Wed Nov 19 08:20:14 CST 2008: sleeping 70 seconds ... Wed Nov 19 08:21:24 CST 2008: sleeping 70 seconds ... Wed Nov 19 08:22:35 CST 2008: sleeping 70 seconds ... Wed Nov 19 08:23:45 CST 2008: sleeping 70 seconds ... Wed Nov 19 08:24:55 CST 2008: sleeping 73 seconds ... Wed Nov 19 08:26:09 CST 2008: sleeping 73 seconds ... Wed Nov 19 08:27:22 CST 2008: sleeping 73 seconds ... Wed Nov 19 08:28:36 CST 2008: sleeping 73 seconds ... Wed Nov 19 08:29:49 CST 2008: sleeping 73 seconds ... Wed Nov 19 08:31:03 CST 2008: sleeping 76 seconds ... Wed Nov 19 08:32:19 CST 2008: sleeping 76 seconds ... Wed Nov 19 08:33:35 CST 2008: sleeping 76 seconds ... Wed Nov 19 08:34:52 CST 2008: sleeping 76 seconds ... Wed Nov 19 08:36:08 CST 2008: sleeping 76 seconds ... Wed Nov 19 08:37:25 CST 2008: sleeping 79 seconds ... Wed Nov 19 08:38:44 CST 2008: sleeping 79 seconds ... Wed Nov 19 08:40:04 CST 2008: sleeping 79 seconds ... Wed Nov 19 08:41:23 CST 2008: sleeping 79 seconds ... Wed Nov 19 08:42:42 CST 2008: sleeping 79 seconds ... Wed Nov 19 08:44:01 CST 2008: sleeping 82 seconds ... Wed Nov 19 08:45:24 CST 2008: sleeping 82 seconds ... Wed Nov 19 08:46:46 CST 2008: sleeping 82 seconds ... Wed Nov 19 08:48:09 CST 2008: sleeping 82 seconds ... Wed Nov 19 08:49:31 CST 2008: sleeping 82 seconds ... Wed Nov 19 08:50:53 CST 2008: sleeping 85 seconds ... Wed Nov 19 08:52:19 CST 2008: sleeping 85 seconds ... Wed Nov 19 08:53:44 CST 2008: sleeping 85 seconds ... Wed Nov 19 08:55:09 CST 2008: sleeping 85 seconds ... Wed Nov 19 08:56:35 CST 2008: sleeping 85 seconds ... Wed Nov 19 08:58:00 CST 2008: sleeping 88 seconds ... Wed Nov 19 08:59:29 CST 2008: sleeping 88 seconds ... Wed Nov 19 09:00:57 CST 2008: sleeping 88 seconds ... Wed Nov 19 09:02:25 CST 2008: sleeping 88 seconds ... Wed Nov 19 09:03:54 CST 2008: sleeping 88 seconds ... Wed Nov 19 09:05:22 CST 2008: sleeping 91 seconds ... Wed Nov 19 09:06:54 CST 2008: sleeping 91 seconds ... Wed Nov 19 09:08:25 CST 2008: sleeping 91 seconds ... Wed Nov 19 09:09:56 CST 2008: sleeping 91 seconds ... Wed Nov 19 09:11:28 CST 2008: sleeping 91 seconds ... copy_jobs is empty stopping copier Wed Nov 19 09:12:59 CST 2008: setting all remaining file statuses to ""Done"" Wed Nov 19 09:12:59 CST 2008: setting file request -2144281128 status to Done Wed Nov 19 09:12:59 CST 2008: set all file statuses to ""Done"" >> Unable to get n13011802_0009_L010185N_D00.reroot.root from srm://fndca1:8443/ pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_00/L010185N/180 << Exiting with status 1 printenv ls -l $OSG_GRID/globus/TRUSTED_CA/e1fce4e9.0 voms-proxy-info -all WARNING: Unable to verify signature! Server certificate possibly not installed. Error: Cannot find certificate of AC issuer for vo fermilab set +v Storage Resource Manager (SRM) implementation version 2.0.3 Copyright (c) 2002-2008 Fermi National Accelerator Laboratory Specification Version 2.0 by SRM Working Group (http://sdm.lbl.gov/srm-wg) Wed Nov 19 09:12:59 CST 2008: rs.state = Failed rs.error = at Wed Nov 19 07:08: 26 CST 2008 state Pending : created RequestFileStatus#-2144281128 failed with error:[ at Wed Nov 19 09:12:03 CST 20 08 state Failed : Pinning failed] Wed Nov 19 09:12:59 CST 2008: ====> fileStatus state ==Failed java.io.IOException: rs.state = Failed rs.error = at Wed Nov 19 07:08:26 CST 20 08 state Pending : created RequestFileStatus#-2144281128 failed with error:[ at Wed Nov 19 09:12:03 CST 20 08 state Failed : Pinning failed] at gov.fnal.srm.util.SRMGetClientV1.start(SRMGetClientV1.java:298) at gov.fnal.srm.util.SRMDispatcher.work(SRMDispatcher.java:795) at gov.fnal.srm.util.SRMDispatcher.main(SRMDispatcher.java:374) srm copy of at least one file failed or not completed"
11/17/2008	117	125001	Assigned	Medium	SZMUKSTA, GEORGE	KREYMER, ARTHUR	"According to MRTG data, stkendca2a went off the net around 08:30 this morning. All the stlendca2a services are offline, including dcap dcapK dcapG SRM GFTP0/1 KFTP WFTP Minos raw data archiving has stopped."
10/8/2008	158	122821	Assigned	Medium	SZMUKSTA, GEORGE	RUBIN, HOWARD	"Again -- 4 jobs this AM: fnpcsrv1% gum n13038022_0000_M100200N_D04_helium.0.cedar_phy_bhcurv.log.gz Beginning job ana_mc on fcdfcaf1094 on Wed Oct 08 at 09:02:30 CDT Requesting n13038022_0000_M100200N_D04_helium.reroot.root from DCACHE using srmc p at 2008-10-08 09:02:33 >> Unable to get n13038022_0000_M100200N_D04_helium.reroot.root from srm://fndca 1:8443/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_04/M100200N_helium/802 << Exiting with status 1 SRMClientV1 : java.io.EOFException SRMClientV1 : get : try # 0 failed with error SRMClientV1 : java.io.EOFException srm copy of at least one file failed or not completed rm: cannot remove `/n13038022_0000_M100200N_D04_helium.*': No such file or direc tory 000 (2160771.000.000) 10/08 08:59:29 Job submitted from host: <131.225.167.44:64 247> .. I have had no response to yesterday's ticket with about 100 such failures."
10/7/2008	159	122724	Assigned	Medium	SZMUKSTA, GEORGE	RUBIN, HOWARD	"As was the case for the last 2 weeks, many MINOS jobs (but not all this time) are failing with a java.IO exception in srmcp. Failures began last night at about 23:37 and went through about 00:48 today. Jobs failed on CDFGrid and GPGrid nodes. Here is the debug=true output for one of these jobs: fnpcsrv1% gum n13037083_0019_L010185N_D04_charm.0.cedar_phy_bhcurv.log.gz Beginning job ana_mc on fcdfcaf1670 on Mon Oct 06 at 23:36:22 CDT Requesting n13037083_0019_L010185N_D04_charm.reroot.root from DCACHE using srmcp at 2008-10-06 23:36:25 SRM Configuration: debug=true gsissl=true help=false pushmode=false userproxy=true buffer_size=131072 tcp_buffer_size=0 streams_num=1 config_file=/usr/local/grid/srm-client-fermi/etc/config-2.xml glue_mapfile=/usr/local/grid/srm-client-fermi/conf/SRMServerV1.map webservice_path=srm/managerv1 webservice_protocol=https gsiftpclinet=globus-url-copy protocols_list=http,gsiftp save_config_file=null srmcphome=/usr/local/grid/srm-client-fermi urlcopy=sbin/urlcopy.sh x509_user_cert= x509_user_key= x509_user_proxy=/grid/home/minospro/.globus/job/fcdfosg4.fnal.gov/14379. 1223353924/x509_up x509_user_trusted_certificates=/usr/local/grid/globus/TRUSTED_CA globus_tcp_port_range=null gss_expected_name=null storagetype=permanent retry_num=20 retry_timeout=10000 wsdl_url=null use_urlcopy_script=false connect_to_wsdl=false delegate=true full_delegation=true server_mode=active srm_protocol_version=1 request_lifetime=86400 access latency=null overwrite mode=null priority=0 from[0]=srm://fndca1:8443/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_ 04/L010185N_charm/708/n13037083_0019_L010185N_D04_charm.reroot.root to=file:////local/stage1/condor/execute/dir_18087/n13037083_0019_L010185 N_D04_charm.reroot.root Mon Oct 06 23:38:35 CDT 2008: starting SRMGetClient Mon Oct 06 23:38:37 CDT 2008: In SRMClient ExpectedName: host Mon Oct 06 23:38:37 CDT 2008: SRMClient(https,srm/managerv1,true) SRMClientV1 : user credentials are: /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=fnpc srv1.fnal.gov/CN=cron/CN=Howard A. Rubin/CN=UID:rubin SRMClientV1 : SRMClientV1 calling org.globus.axis.util.Util.registerTransport() SRMClientV1 : connecting to srm at httpg://stkendca2a.fnal.gov:8443/srm/managerv 1 Mon Oct 06 23:39:04 CDT 2008: connected to server, obtaining proxy Mon Oct 06 23:39:04 CDT 2008: got proxy of type class org.dcache.srm.client.SRMC lientV1 SRMClientV1 : get: surls[0]=""srm://fndca1:8443/pnfs/fnal.gov/usr/minos/mcin_da ta/near/daikon_04/L010185N_charm/708/n13037083_0019_L010185N_D04_charm.rero ot.ro ot"" SRMClientV1 : get: protocols[0]=""gsiftp"" SRMClientV1 : get: protocols[1]=""dcap"" SRMClientV1 : get: protocols[2]=""http"" copy_jobs is empty copy_jobs is empty stopping copier Mon Oct 06 23:40:38 CDT 2008: setting all remaining file statuses to ""Done"" Mon Oct 06 23:40:38 CDT 2008: set all file statuses to ""Done"" >> Unable to get n13037083_0019_L010185N_D04_charm.reroot.root from srm://fndca1 :8443/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_04/L010185N_charm/708 << Exiting with status 1 Storage Resource Manager (SRM) implementation version 2.0.3 Copyright (c) 2002-2008 Fermi National Accelerator Laboratory Specification Version 2.0 by SRM Working Group (http://sdm.lbl.gov/srm-wg) SRMClientV1 : java.io.EOFException SRMClientV1 : get : try # 0 failed with error SRMClientV1 : java.io.EOFException srm copy of at least one file failed or not completed rm: cannot remove `/n13037083_0019_L010185N_D04_charm.*': No such file or direct ory"
9/19/2008	177	121930	Work In Progress	High	MESSER, TIM	KREYMER, ARTHUR	"Several Minos raw data files were written to DCache write pools, but are not on tape, or in DCache now : N00014862_0013.mdaq.root Fri Sep 19 14:06:19 UTC 2008 F00041958_0003.mdaq.root Fri Sep 19 10:13:58 UTC 2008 B080918_080001.mbeam.root Fri Sep 19 10:18:07 UTC 2008 N080918_000003.mdcs.root Fri Sep 19 10:28:30 UTC 2008 The full paths to these files are : /pnfs/minos/neardet_data/2008-09/N00014862_0013.mdaq.root /pnfs/minos/fardet_data/2008-09/F00041958_0003.mdaq.root /pnfs/minos/beam_data/2008-09/B080918_080001.mbeam.root /pnfs/minos/near_dcs_data/2008-09/N080918_000003.mdcs.root There is also an unusual backlog of files which are present in DCache writePools pool group, but not yet on tape : There are such files in /pnfs/minos/reco_far/cedar_phy_bhcurv/cand_data/2008-01 F00040170_0022.spill.cand.cedar_phy_bhcurv.0.root F00040167_0018.all.cand.cedar_phy_bhcurv.0.root F00040148_0021.spill.cand.cedar_phy_bhcurv.0.root F00040176_0011.spill.cand.cedar_phy_bhcurv.0.root F00040148_0003.all.cand.cedar_phy_bhcurv.0.root F00040167_0005.spill.cand.cedar_phy_bhcurv.0.root F00040173_0019.spill.cand.cedar_phy_bhcurv.0.root /pnfs/minos/reco_far/cedar_phy_bhcurv/.bcnd_data/2008-01 F00040145_0023.spill.bcnd.cedar_phy_bhcurv.0.root /pnfs/minos/reco_far/cedar_phy_bhcurv/sntp_data/2008-01"
9/10/2008	186	121377	Assigned	Medium	JONES, TERRY	RUBIN, HOWARD	"This has been a long-standing annoyance. kerberized dccp always returns a spurious error message, apparently when it checks to see if the target already exists. If the target is pnfs, it checks to see if it exists, and if it doesn't it returns an error message saying it doesn't exist even though this is exactly what *should be* the case. I know you're aware of this, but why, after 6 months, is it not fixed?"
3/31/2008	349	113334	Assigned	Medium	SZMUKSTA, GEORGE	SODERBERG, MITCHELL	"I have been in contact with Gene Oleynik and Stan Naymola about setting up ~20TB of storage space for the T962 test beam project. I need instructions on how to proceed with this request. Thanks, Mitch x2034"
10/12/2007	520	105528	Work In Progress	Medium	SCHUMACHER, KEN	LEBRUN, PAUL	"The following was submitted by: Paul L. G. Lebrun (lebrun@fnal.gov) on Friday, October 12, 2007 at 10:30:04 --------------------------------------------------------------------------- email: lebrun@fnal.gov realname: Paul L. G. Lebrun Fermi ID: 06058N Lab Status: N Phone: 630 840 3947 Div/Sect: CD Department: Accel Detect. Simu. Experiment: Auger Affiliation: Description of Project: Archive raw Auger data on tape. Approx number of user accounts: 2 Term of Project in Months: 24 Number of Slots Requested: 6 SubmitButton: Submit Form"