IRE Gedanken experimentation: An alternative to traditional system testing? chapter William S. Cooper Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. I' Gedanken experimentation: An alternative to traditional system [OCRerr][OCRerr]sting[OCRerr]* William S. Cooper Technological progress is generally brought about through a combination of intelligent theorizing, experimentation, and inspired tinkering. The techno- logy of literature searching is no exception, and elements of all three have contributed to recent progress in the information retrieval field. However, without disparaging in any way the research that has been carried out, it might fairly be observed that information retrieval is an area in which by the very nature of the subject matter the theory is thin and large-scale experimentation cumbersome and offen inconclusive. Perhaps, therefore, the time is ripe to start giving more attention to the third of the aforementioned alternatives, the scientifically disreputable but often surpris- ingly successful course of `inspired tinkering'. I have in mind especially a kind of `tinkering' with retrieval system parameters through educated guesswork rather than careful full-scale experimentation. It involves the making of estimates, or `guesstimates', based on very little data-gathering or even on nothing more than human intuition and experience combined with the few available shreds of theory. Since the process involves the vigorous use of the imagination, it might in the phraseology of the older sciences be called `thought experimentation' or `gedanken experimentation'. (Gedanken experiment. An experiment carried out by proposing a hypothesis in thought only[OCRerr]Webster.) Physicists, for instance, sometimes analyse at length what would happen if they were to carry out certain experiments in a freely falling elevator, but for some reason never seem to get around to the actual execution of the experiments. As it applies to document retrieval, gedanken experimentation amounts to thoughtful, theory-guided guesswork about what is likely to make a system work most effectively. The guesswork may concern any of various decision problems or parameter-setting tasks which arise in setting up, maintaining, and using a retrieval system. Making the guesses may be the responsibility of any of several agents including the system designer, the analyst in charge of implementing the system design, the indexer, or the end user. The unique contribution of the information scientists is to suggest ways of making the * This material is based upon work supported by the National Science Foundation under Grant IST-791 7566.