IRE The Cranfield tests chapter Karen Sparck Jones Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Criticisms of Cranfield 1 267 Electric test supported the Cranfield results in relation to success rates and reasons for failures, while the WRU test was notable first for searching without reference to wanted document numbers, and second, for the use of both recall and precision; the results for recall for the facet system were like those of the main test, while the WRU failures were attributable chiefly to poor searching. According to Lancaster and Mills, the Aslib Cranfield Project as a whole served to establish `reasonably reliable figures. . . for the degree of recall and relevance [i.e. precision] likely to be achieved by a good index employing fairly exhaustive indexing' (p.8), i.e. performance in terms of operating efficiency. Further, the broad implication is that `as one factor amongst several determining the operating efficiency of indexes, the indexing system, per se, is of less importance than has been assumed generally.' (p.9) However in their view `the first Cranfield investigation was perhaps less important for the light it shed on four actual indexes than for its contribution to the development of techniques for the testing of information retrieval systems and for exposing the basic parameters in the operation of an indexing system.... What was, to begin with, a rather blunt tool is being developed into a sharp analytical instrument.' (p.9) Lancaster and Mills conclude that the inverse relationship between recall and precision is inescapable, and that the recall/precision character of a system is determined by indexing exhaustivity and specificity, determining recall and precision respectively. Moreover, since search relaxation can improve recall, `we are left with the specificity of the index as its key characteristic. Hospitality to specific indexing, by which we mean the ability to describe precisely the concepts chosen as retrieval tags, is the most important single criterion of an index language' (p. 11); or `to summarise, for every index language, depending upon its hospitality for specific indexing, there is a maximum relevance ratio which cannot be exceeded. However it is always possible to improve the recall ratio along the fixed performance curve to its maximum by means of variations in search programmes. But the ability to do this is dependent upon the exhaustivity of the indexing.' (p. 11) 13.2 Criticisms of Cranfield 1 The aims and character of the Aslib-Cranfield Project meant that the Cranfield 1 results attracted much attention. Those with indexing language positions to defend tended to attack the results without offering any concrete i