5-1~


4.   Cutoff-Independent Performance Indices


     A.  Derivation


         Performance indices for document retrieval systems which are

based on a contingency table description (as introduced in section 2)

assume  hat a retrieval operation partitions the reference collection,

i.e. identifies a retrieved subset.  in the model system considered

in this thesis (vector indexing and cosine correlation 4uery-document

matching), and in other models of interest (see Chapter 4), the result

of a retrieval operation is more accurately described by the

distribution of the matching coefficient over the reference collection

or by the ordering induced on the document set from this distribution.

The use of partition based evaluation parameters for such systems

requires, then, that some decision function (or cutoff criterion) be

introduced into the retrieval process.  Operationally, the number of

retrieved documents a user will examine is likely to be dependent on

a number of subjective variables.  There is, therefore, considerable

difficulty in the a priori specification of a meaningful partitioning

algorithm.  For this reason, then, some performance measures are

derived here which are functionally dependent on the full ordering of

the reference collection produced by a retrieval operation.  Such

measures eliminate the need to introduce any notion of cutoff.

         Under the assumption that the ordering induced on the set of

reference documents by the search process M is the principal result

of a retrieval operation and that a set of relevant documents D  is
                                                               R
available corresponding to each request q, the objective of a