- 38 -

        Some examples of a strong match are: Chemical, Kinetic (Question 9); Viscous,
flat plate (Q. 82); and Slip (frequency of 19, Q. 87). Some examples of a weak match
are: High Speed (Q. 2); Aircraft (Q. 2); Hypersonic (Q. 9); and Structural (Q. 49). Further
examples can be seen by reference to Table 3.9.

        Out of the 287 documents examined against the 35 questions, 89 (31%) showed a
strong match; an additional 87 had a weak match, and the total of 176 represents
61.3% matching. 28 of the questions had one or more documents with a strong match
and 32 had one or more with a weak match.

         This shows that nearly one-third do have a strong question-title match, but
since the assessment of relevance has been done in four grades, we may expect that
those documents with a strong match will be graded as more relevant than those with
a weak match. Table 3.10 divides the results into the four relevance grades, and
shows that the probability of a relevance (1) document being strongly matched is more
than twice that of the relevance (2), {3) or (4) documents. That the relevance (2),(3)
and (4) documents show the same probability may be accounted for by the difficulty
of consistently doing such a refined grading of relevance, but the relevance (1) docu-
ments seem to indicate a strong trend.

         Whether it is taken that these figures show an unusual question-title match or
not, the presence of an unnatural question-document relationship cannot be proved
or disproved by this. One would have expected a certain strength of title match in
this subject, where titles are usually fairly long and a good indication of the subject
of the document. The documents    examined were the total of those relevant to
each question, and included both the original documents cited in the authors' base
document, and also the additional documents discovered in the collection. It is obvious
that these additional relevant documents, discovered by the students, examination of
the collection and by bibliographic coupling, were discovered and assessed as rele-
vant in a situation equivalent to a real life one, and therefore it would be quite
absurd to suggest that an unnatural or biased relationship could possibly exist in
their case. So a comparison of t}~e question-title match between the 'cited' relevant
documents and the 'additional' relevant documents will provide some evidence of any
unnatural differences in the question-document relationships.

        The 287 relevant documents comprised 149 cited and 138 additional, and the
matching scores were calculated for each group. Table 3.11 presents the results,
and it is shown that the additional relevant documents had a slightly stronger question-
title match than the cited ones, 32.6% to 29.5% for the strong matches, and 68.1% to
55.0% for the weak matches. Ten of the 35 questions had no additional documents at
all, and the cited document for these questions have been included in the results;
deleting these ten questions would reduce the matches for cited docut~ents to 27.6% and
51,4%.

        These results might alter over the whole set of questions, but there is no
reason to expect that they would change significantly. On the basis of the question-
title match anyway, no real difference exists between the cited and additional docu-
ments. We suggest that this indicates that there is no justification for any implication
that there is a biased or unnatural question-document relationship, and that the rele-
vance assessments and relevant documents found are not really different from that
which might happen in a real life situation. Further evidence can be obtained from
some of the test results themselves, where the retrieval performance in recall of
the cited documents can be compared with the additional documents.