Final Report Use of the Critical Incident Technique to Evaluate the Impact of MEDLINE September 30, 1989 Sandra R. Wilson, Ph.D., Project Director Norma Starr-Schneidkraut, Ph.D., Project Coordinator Michael D. Cooper, Ph.D., Consultant Submitted to Elliot R. Siegel, Ph.D., Project Officer Office of Planning and Evaluation National Library of Medicine 8600 Rockville Pike Bethesda, Maryland 20894 Contract No. N01-LM-8-3529 ----------------------------------------------------------------- ACKNOWLEDGMENTS The authors wish to acknowledge the important collaborative role of Elliot R. Siegel, Ph.D., Assistant Director for Planning and Evaluation/NLM, and Barbara A. Rapp, Ph.D., throughout this study. Dr. Siegel's official capacity was that of Project Officer, and Dr. Rapp was Alternate Project Officer. However, AIR's role as support contractor to NLM meant that, functionally, both were an integral part of the study team, with Dr. Rapp providing day-to-day project coordination on NLM's behalf. Their insight, guidance, and technical input at all points were key ingredients in the successful outcome of the study. Other persons at NLM who contributed to the study effort at various stages of planning and implementation were Dr. Michael Ackerman, Mr. Sean Donohue, Dr. Richard Hsieh, Mr. David Kenton, Dr. Henry Kissman, Mr. Charles Kalina, Ms. Becky Lyon-Hartmann, Mr. Robert Mehnert, Ms. Susan Slater, and NLM consultants Dr. Jay Goldman of the University of Alabama and Dr. Edithe Levit, past President of the National Board of Medical Examiners. Special acknowledgment is due NLM Director Dr. Donald A. B. Lindberg, who originally conceived the notion of using the CIT methodology to evaluate MEDLINE. Any residual deficiencies in the study execution or reporting, however, remain the ultimate responsibility of AIR and the authors. The authors also wish to express their appreciation to a number of current and former staff of the American Institutes for Research for their help in this project. Jeffrey McHenry, Ph.D., worked with Dr. Wilson in the development of AIR's approach to the study and proposal. Prior to the assumption of the role of Project Coordinator by Norma Starr-Schneidkraut, Ph.D., Ms. Susan Vintilla-Friedman served as Coordinator and conducted the interviewer training and initial respondent interviewing. Ms. Nancy Hull, AIR Palo Alto Office librarian, participated in respondent interviewing, taxonomy development, and preparation of the case study abstracts. Ms. Elise Mosse also wrote such abstracts. Special thanks are due to Mss. Emily Campbell, Carrie Davis, Linda Erickson, and Irene Yurash for their expert word processing of the many hundreds of incident reports and case study abstracts, as well as drafts of the technical reports. Throughout most of the study, administrative support was provided by Ms. Alice Cherveny. Appreciation is also due to Ms. Parisa Ashrafi for her careful monitoring of the progress of the recruitment, the interviewing, and the processing of the data, as well as for her coordination of report preparation and for her data processing support. Credit for the preparation of the dBase IV file goes to Bradley Hesse, Ph.D., and Frances Stancavage, M.A.; for development of the SAS data file, to Mr. Mitchell Yee and KatherineTroyer, Ph.D.; and for help in the mailing of letters of invitation, to Winnie Young, MLS, and Mss. Elaine Balek, Alison McGrady, and Roberta Caldwell of the AIR Washington office. Thanks also go to two students, Mr. Kwame Burroughs and Mr. Peter Hu, NIH Biomedical Research Support Grant summer interns, for their help in a variety of project tasks. Great appreciation is due the staff of the University of Texas Regional Medical Library who identified and conducted interviews with individuals requesting MEDLINE searches at that institution. These staff members include Ms. Jean Miller, Director, Health Science Center Library; Mr. Spencer Marsh, who coordinated the interviews; and Ms. Laura Wilder, Pennie Billings, Alan Carr, and Helen Mayo, who carried out the interviews. Thanks are also offered to Elaine Graham, Associate Director, Pacific Southwest Medical Library Service, and to the staff of the University of California at Los Angeles Regional Medical Library and the libraries of community hospitals in Region 7, who identified a sample of individuals requesting searches of these hospitals' librarians. The assistance of Muriel H. Bagshaw, M.D., is also gratefully acknowledged. Her careful review of the patient care incident reports and the case studies helped insure the accuracy of their medical content and their general readability. The participation of Laurence Way, M.D., and Gary M. Arsham, M.D., in the review of some of the initial incident reports is also acknowledged. Cristina Campbell, M.L.S., of the University of California School of Public Health Library, working with Michael D. Cooper, Ph.D., Professor, University of California School of Library and Information Studies, carried out the verification of the search parameters reported in the incidents and provided an analysis of the transaction logs of ineffective searches. Finally, great appreciation is due to the hundreds of physicians, researchers, and other health professionals who contributed their time to share their experiences in searching MEDLINE and their perspectives on the value of and potential improvements in MEDLINE and Grateful Med. TABLE OF CONTENTS EXECUTIVE SUMMARY INTRODUCTION METHODS OVERVIEW OF METHODOLOGY SAMPLE Sample selection Recruitment Response rates DATA COLLECTION PROCEDURES Protocol development Interviewer training Interviewing procedures Data processing DATA ANALYSIS Critical incident data Respondent and search characteristics Verification Analysis of reports and logs of ineffective searches RESULTS CHARACTERISTICS OF RESPONDENTS AND SEARCHES Overview of the data Respondent characteristics Searches RESPONDENT COMMENTS CONCERNING NLM SERVICES THE INCIDENT REPORTS Examples of incident reports Verification of parameters of the end user searches TAXONOMY: WHY INFORMATION WAS NEEDED FROM MEDLINE I. Patient care II. Research III. Teaching IV. Learning V. Administration VI. Other REASONS WHY THE INDIVIDUAL CHOSE TO DO A MEDLINE SEARCH I. Why the individual wanted to access the journal literature II. Why the individual chose to do an online search III. Why the searcher used Grateful Med TAXONOMY: IMPACT ON MEDICAL DECISION-MAKING I. Patient care II. Research III. Teaching IV. Learning V. Administration VI. Consultation TAXONOMY: IMPACT ON THE OUTCOMES OF PROFESSIONAL ACTIVITIES I. Patient end results II. Research end results III. Teaching end results IV. Administrative end results V. Legal end results RESULTS OF ANALYSIS OF INEFFECTIVE SEARCHES Analysis of user interviews Detailed analysis of searches Conclusions about searching and indexing CONCLUSIONS METHODOLOGY RESULTS Reasons for using MEDLINE to obtain information Reasons for needing information Impact on medical decision-making Impact on outcomes User problems IMPLICATIONS IMPROVING MEDLINE SERVICES INCREASING USE OF MEDLINE INCREASING SUPPORT FOR LIBRARY SERVICES FUTURE RESEARCH Improvements to Grateful Med Outreach efforts Research on continuing education and clinical reasoning Use of the research data base BIBLIOGRAPHY TABLES Table 1. Location of Interviewer and Type of MEDLINE User, by Source of Sample Table 2. Recency and Results of Searches Reported by End Users and Mediated Users FIGURES Figure 1. Examples of Reports of Effective MEDLINE Searches Figure 2. Example of Reports of Ineffective MEDLINE Searches Figure 3. Reasons Why the Individual Needed Information from MEDLINE Figure 4. Reasons Why Individual Chose to Do a MEDLINE Search (Rather than Use Other Means of Obtaining the Information) Figure 5. Impact of the Information Obtained from MEDLINE on Medical Decision-Making Figure 6. Impact of the Information Obtained from MEDLINE on the Outcomes of Professional Activities APPENDICES APPENDIX A. LETTERS OF INVITATION APPENDIX B. CRITICAL INCIDENT REPORT FORMS APPENDIX C. INCIDENTS AND TRANSCRIPTIONS APPENDIX D. CHARACTERISTICS OF RESPONDENTS AND SEARCHES APPENDIX E. EXAMPLES OF CASE STUDY ABSTRACTS APPENDIX F. REASONS WHY THE INDIVIDUAL NEEDED INFORMATION APPENDIX G. IMPACT OF THE INFORMATION OBTAINED FROM MEDLINE ON MEDICAL DECISION-MAKING APPENDIX H. IMPACT OF THE INFORMATION OBTAINED FROM MEDLINE ON THE OUTCOMES OF PROFESSIONAL ACTIVITIES APPENDIX I. SUMMARY OF ANALYSIS OF INEFFECTIVE SEARCHES