IRE Simulation, and simulation experiments chapter Michael D. Heine Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 194 Simulation, and simulation experiments abstract `Systems perspective' should help the overcoming of artificial subject barriers such as this. The work of Blunt, Duquet and Luckie23 was concerned with the extent ol, the resources needed, and the response time, of an information retrieval service, and appears to be especially useful in determining the extent to which response time is affected by the competition by queries for resources, Hertz et al.21 and Fried25 examined the problem of simulating indexed document record files. Heine29 was concerned with using simulation to predict the effect on Recall-Precision performance of using document age as a component of the query in addition to the more conventional semantic attributes of documents. Lastly we discuss the two theses by M.D. Cooper28 and Griffiths30. Cooper was essentially concerned, in the simulation part of his study, with the extent to which different queries retrieved different numbers of items from a database. Pseudo-queries and pseudo-documents were defined, each as sets of document attributes. The similarity of a query with a document was expressed as the number of attributes in common between them, i.e. as co- ordination level (to use the Cranfield concept and terminology), and the distribution of the database over non-zero values of the latter was found for a wide variety of queries. Cooper's work appears to be notable for (a) the careful placing of the study in the context of retrieval system evaluation, and (b) the incorporation of term association (i.e. pairwise dependence between terms in some subset of the database, in this case the entire database) in the simulation. However, as stated by him the simulation is limited in its usefulness in that the notion of a partitioning of the database, and in particular a partitioning of the retrieved set, into relevant and non-relevant subsets, is not recognized in the model. The possibility of doing so was rejected by him on the ground that `not enough information is available to characterize the process' (p. 156). This apparently minor point is dwelt on here because in the writer's view it illustrates the occasional critical dependence of simulation upon experimental results (obtained in the laboratory or from operational systems) as well as upon the system description. Griffiths, like Cooper, was concerned with creating pseudo- documents and pseudo-queries in order to simulate the process of post- coordinate searching a database. Unlike Cooper, who chose not to model users' relevance judgements or retrieved sets, Griffiths partitioned the retrieved sets arising in the simulation (identified by a matching process + threshold) by using experimental data obtained from an INSPEC test on retrieval strategies carried out in 1974, and an EEC study of databases containing veterinary literature. The simulation procedure apparently labelled retrieved documents (attached to each co-ordination level) as either relevant or non-relevant on the basis of the value of a Bernoulli variable, but a detailed description of this step and justification of it are unfortunately not given. No attempt was made to model the relevance values of non-retrieved documents (i.e. to partition non-retrieved documents into relevant non retrieved and non-relevant non-retrieved) so that only Precision, not Recall, is modelled. Co-occurrence frequencies of terms are also not introduced into the model (unlike Cooper's model), presumably because empirical evidence was not available in support of this. Although the main objective of Griffiths was to simulate post-coordinate searching using data obtained from