IRE Laboratory tests of manual systems chapter E. Michael Keen Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 154 Laboratory tests of manual Systems Our handling of the human subject in manual systems testing may well have been over-cautious in the care with which groups of different people have been used as indexers, searchers, relevance judges, and so on. The idea of using the researchers in some of these roles has been strenuously avoided, We have argued strongly against the attempt to compare different retrieval systems without the rigour of laboratory control, but the semi-operational off-shelf approach may be valid. Perhaps the greatest rigidity in thinking has been that one well-conducted experiment settles both the issues the experiment was designed to investigate and queries about methodology. Experiments do need to return even to the fundamental parameters of exhaustivity and specificity so that understanding may be deepened and non trivial design equations advanced. The `single experiment' mentality fails to demonstrate repeatability, and the effort of replication should not now be an option. Manual testing has achieved much[OCRerr]now is not the time to stop. References 1. CLFvFRr'oN, C. W Report on the First Stage of an Investigation into the Comparative Efficiency ofindexing Systems, First Aslib Cranfield Project, College of Aeronautics, Cranfield (1960) 2. CLFvFRr'oN, C. W. Report on the Testing and Analysis of an Investigation into the ComparaUve Efficiency of Indexing Systems, First Aslib Cranfield Project, College of Aeronautics, Cranfield (1962) 3. CLEVERDON, C. W., MILLS, j. and KEEN, F. M. Factors Determining the Performance oflndexing Systems, 2 Vols, Second Aslib Cranfield Project, College of Aeronautics, Cranfield (1966) 4. CLEvERDON, C. W. The Cranfield tests on index language devices, Aslib Proceedings 19, 173- 194(1967) 5. SWANSON, D. R. Interrogating a computer in natural language. In: Jnformation Processing 62, Proceedings of IFIP Congress 1962, (Ed. C. M. Popplewell), North-Holland, Amsterdam (1963) 6. KEEN, F.M. A retrieval comparison of six published indexes, UNESCO Bulletin for Libraries 30, 26-36 (1976) 7. CLEvERDON, C. W. Evaluation tests of information retrieval systems, JournalofDocumentation 26, 5567 (1970) 8. AITCHISON, j. and CLEVERDON, C. w. A Report on a Test ofthe Index ofMetallurgical Literature of Western Reserve University, First Aslib Cranfield Project, College of Aeronautics, Cranfield (1963) 9. KEEN, F.M. and DIGGER, J. A. Report of an Information Science Index Language Test, 2 VoIs, College of Librarianship Wales, Aberystwyth (1972) 10. KEEN, F.M. The Aberystwyth index languages test, Journal ofDocumentation 29,1-35 (1973) 11. SARACEvIC, T. et al. An Inquiry into Testing of Infrrmation Retrieval Systems, 3 VoIs1 Comparative Systems Laboratory, Centre for Documentation and Communication Research, Case Western Reserve University, (1968) 12. AITCEIISON, T. M. et al. Comparative Evaluation of Index Languages, Part I: Design; Part If; Results, Reports R70/1 and R70/2, INSPEC, Institution of Electrical Engineers, London (1969, 1970) 13. FARRADANE, J. et al. Research on Relational Indexing, Final Condensed Report to OSTI, City University, London (1968) 14. KEEN, F.M. On the generation and searching of entries in printed subject indexes, Journal of Documentation 33, 1545 (1977) 15. KEEN, F.M. On the processing of printed subject index entries during searching, Journal Of Documentation 33, 26[OCRerr]276 (1977) 16. KEEN, F.M. On the Performance ofNine Printed Subject Index Entry Types, A Selective Report of EPSILON, College of Librarianship Wales, Aberystwyth (1978) 17. BARRACLOUGH, F. D. et al. The Medusa Current Awareness Experiment, Computing Laboratory University of Newcastle upon Tyne (1975) 18. KEEN, F. M. An analysis of the documentation requests. In: Report ISR-13, Section X, Department of Computer Science, Cornell University (1967) 1[OCRerr] 1 31 in 1' I I I i I I