Experiment Design and Results for the TREC-6 Interactive Track Pre-Experiment ============================================================================= 1. Pre-experiment as designed The TREC-6 Interactive Track experimental design was constructed to support the comparison of multiple experimental systems, each at a different site Each site runs its experimental system and a control system common to all sites. By subtracting the response on the control system from that of the experimental system, the design seeks to eliminate the site effect, allowing comparison of experimental systems at different sites without confounding of or contamination by site-to-site differences and without the high cost of running even a fraction of the experimental systems at all sites. An effective control system is one that captures the site effect, which includes the influence of at least the following on the response: - the physical surroundings in which the experiment takes place - the operational characteristics of the experimental system which are dependent on its operating environment - the experimenter's interaction directly and indirectly with the experimental participant The purpose of running the pre-experiment is to assess in advance of the main experiment whether the recommended control system adequately captures the site effect, i.e., the extent to which it is capable of eliminating site-to-site differences. In order to assess this we need a design which lets us compare two or more experimental systems in two ways: - directly (by running multiple experimental systems per site - this should be doable if the number of such systems is small compared to that of the main experiment) - indirectly (in the manner to be used in the main experiment, i.e., through use of the control running at each site) The following designs for 2 sites is based on latin square designs and are similar to the design proposed for the main experiment. In the pre-experiments, unlike in the main experiment, the experimental system from a given site is run at all other sites as well. This allows us to compare experimental systems directly for each site, i.e. without any confounding due to site, and so without the need for a control system. - Site i, for i = 1, ..., I sites - P(i,j) is participant j at site i, j = 1, ..., J - T(k) is topic k, k = 1, ..., K topics, common to all sites - En is experimental system n - Co is the control system - Order is order in which systems will be run The design for N experimental systems is a v x v latin square for EACH site, where v = N + 1. The number of sites must be at least two, but is otherwise unlimited." - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 2 sites, 2 systems: Order 1 2 3 _________________ | Site 1 P(1,1) | E2 E1 Co P(1,2) | Co E2 E1 P(1,3) | E1 Co E2 Site 2 P(2,1) | Co E1 E2 P(2,2) | E1 E2 Co P(2,3) | E2 Co E1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - How to run this experiment: A. Each site randomly assigns participants to rows and topics T(1), T(2), and T(3) to columns (I removed topic labels from the columns to avoid confusion here). If both sites in a pair comes up with the same assign-, ment of topics, one can try again until they get a different random order. B. The order that systems will be run is left to right as given above. C. Concerning sites with more than the minimal 3 participants (ideally all), it seems desirable, because of possible carry-over effects from one system to the next, for sites, where possible, to try to balance the number of participants assigned to each of the 6 possible system sequences, which are spread over the 2 latin squares in the 2-site design. This could mean using the latin square for the other site for every other group of 3 participants and trying for multiples of 6 participants. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Features of the experiment: A. The two latin squares given above are orthogonal latin squares and have the property that each system immediately follows every other system twice. B. Note that the second latin square was formed by reversing the order of systems from the respective row of the first square. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Proposed evaluation: A. For each site i, estimate E2(i) - E1(i) direct comparison of the experimental systems E2(i) - Co(i) comparison of 2nd experimental system & the control E1(i) - Co(i) comparison of 1st experimental system & the control B. Estimate the site-INDEPENDENT comparison E2 - E1 by averaging the direct site-dependent comparison E2(i) - E1(i) over all sites. C. Estimated the site-INDEPENDENT comparison E2 - E1 using the control as will be done in the main experiment:[E2(2) - Co(2)] - [E1(1) - Co(1)]. D. Assess the effectiveness of the control system in removing the site effect by comparing the two estimates of the site-independent difference E2 - E1 from steps 2. and 3. above. We can estimate the "true" difference in the two estimates with a confidence interval. With only a few sites participating, the true difference will be estimated with fairly substantial uncertainty . - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Topics as seen by the searchers: ============ Number: 258i Topic: Computer Security Description: Identify instances of illegal entry into sensitive computer networks by non-authorized personnel. Please save at least one document that identifies EACH DIFFERENT case of illegal entry into a computer network. If one document discusses several cases or instances, then you need not save other documents which mention the same ones, since your goal is to identify the different instances of illegal entry. Narrative: Illegal entry into sensitive computer networks is a serious and potentially menacing problem. Both 'hackers' and foreign agents have been known to acquire unauthorized entry into various networks. Items relevant to this subject would include but not be limited to instances of illegally entering networks containing information of a sensitive nature to specific countries, such as defense or technology information, international banking, etc. Items of a personal nature (e.g. credit card fraud, changing of college test scores) should not be considered relevant. ============ Number: 274i Topic: Electric Automobiles Description: What are the latest developments in the production of electric automobiles? Please save at least one document that identifies EACH DIFFERENT recent development in the field of electric automobiles. If one document discusses several developments, then you need not save other documents that repeat those developments, since your goal is to identify the different ones that have been discussed. Narrative: The economic feasibility of electric automobiles appears to be limited by a number of factors, including the limited range of operation between recharges of batteries. What progress has been made in addressing these factors? ============ Number: 292i Topic: Worldwide Welfare Description: Identify social programs for poor people in countries other than the U.S. Please save at least one document that identifies EACH DIFFERENT social program in EACH DIFFERENT COUNTRY. If one document discusses several programs in different countries, then you need not save other documents which mention those same programs, since your goal is to identify the different such programs which exist. Narrative: To be relevant a document would identify a welfare program in a foreign country and explain how it works to aid citizens who have little or no income. It would include those who can't work because of a disability and people who have the extra burden of small children. The document should indicate how these people are supported or not supported. A relevant document should identify the source of the monies used to support such welfare programs. ------------------------------------------------------------------------------- 2. Pre-experiment as run Two sites, Rutgers and City University, London provided experimental systems: RU-Inquery(Ru) and Okapi(Ok), respectively. NIST provided a control system: ZPRISE(Zp). All three sites installed and ran all 3 systems. The following combinations of the 6 possible system sequences and 3 topic-sequences were run at the 3 sites with the specified searcher: (Site : Searcher) 274 292 258 258 274 292 292 258 274 -------------- ------------------------ ----------------------- Ok Ru Zp \ s001 bra ns8 s007 Zp Ok Ru > Rutgers: s002 City: att NIST: ns2 Rutgers: s008 Ru Zp Ok / s003 col ns3 s009 -------------- ------------------------ ----------------------- Zp Ru Ok \ mcf s004 ns4 Ru Ok Zp > City: lea Rutgers: s005 NIST: ns5 Ok Zp Ru / aji s006 ns6 -------------- ------------------------ ----------------------- ------------------------------------------------------------------------------- 3. Results - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - A. For each of the 3 sites, estimated E2(i) - E1(i) direct comparison of the experimental systems E2(i) - Co(i) comparison of 2nd experimental system & the control E1(i) - Co(i) comparison of 1st experimental system & the control Let: E1 = Ru (Ru-Inquery) E2 = Ok (Okapi) Co = Zp (ZPRISE) Differences in Differences in Avg. Precision Avg. Recall ---------------------- --------------------- Ok-Ru Ok-Zp Ru-Zp Ok-Ru Ok-Zp Ru-Zp # of searchers Sites ------ ------ ------ ------ ------ ----- - ----- City -0.231 -0.026 0.205 0.112 -0.006 -0.118 6 Rutgers 0.048 0.000 -0.048 0.040 0.017 -0.023 9 NIST 0.264 0.333 0.069 0.061 0.158 0.097 6 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - B. Estimated the site-INDEPENDENT comparison E2 - E1 by averaging the direct site-dependent comparison E2(i) - E1(i) over all 3 sites. Precision: 0.030 = ( (6)(-0.231) + (9)0.048 + (6)0.264 ) / 21 Recall: 0.067 = ( (6)0.112 + (9)0.040 + (6)0.061 ) / 21 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - C. Estimated the site-INDEPENDENT comparison E2 - E1 using the control as will be done in the main experiment:[E2(2) - Co(2)] - [E1(1) - Co(1)]. City(Ok-Zp) - Rutgers(Ru-Zp): Precision: 0.022 = -0.026 - (-0.048) Recall: 0.017 = -0.006 - (-0.023) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - D. Assessed the effectiveness of the control system in removing the site effect by comparing the two estimates of the site-independent difference E2 - E1 from steps 2. and 3. above. We can estimate the "true" difference in the two estimates with a confidence interval... With only a few sites participating, the true difference will be estimated with fairly substantial uncertainty... Confidence intervals [95% t.025,v=(n-1)=2] for "true" difference E2-E1: "True" E2-E1 >= (mean of y's) - ( t(0.05/2) (stddev/sqrt(n)) ) and <= (mean of y's) + ( t(0.05/2) (stddev/sqrt(n)) ) Precision: y's = {-0.231, 0.048, 0.264} n = 3 t(0.05/2) df=2 = 4.303 mean of y's = 0.027 stddev = 0.248 -0.589 <= E2-E1 <= 0.643 Recall: y's = {0.112, 0.040, 0.061} n = 3 t(0.05/2) df=2 = 4.303 mean of y's = 0.071 stddev = 0.037 -0.021 <= E2-E1 <= 0.163 On the one hand, the confidence intervals for the "true" difference E2-E1 in terms of precision and recall do contain the estimates of the differences based on the control - more narrowly for recall than precision. On the other hand, the confidence intervals are so wide that the results of this small experiment provide no strong support for the effectiveness of the control in isolating the site-effect. ------------------------------------------------------------------------------- 4. Supporting results data Within-searcher system differences in precision and recall Difference in Difference in Avg. Precision Avg. Recall --------------------- --------------------- Site Searcher Ok-Ru Ok-Zp Ru-Zp Ok-Ru Ok-Zp Ru-Zp city bra -0.750 0.000 0.750 -0.755 -0.275 0.480 city att 0.700 0.150 -0.550 0.604 0.049 -0.555 city col -0.446 -0.596 -0.150 0.198 -0.116 -0.314 city mcf -1.000 -0.250 0.750 -0.091 -0.385 -0.294 city lea -0.286 0.714 1.000 0.237 0.545 0.308 city aji 0.395 -0.176 -0.571 0.478 0.147 -0.331 ----- ----- ----- ----- ----- ----- mean -0.231 0.026 0.204 0.111 -0.006 -0.118 std 0.658 0.442 0.710 0.488 0.334 0.411 rutgers s001 0.175 -0.100 -0.275 0.384 -0.007 -0.391 rutgers s002 0.000 -0.800 -0.800 0.000 -0.636 -0.636 rutgers s003 -0.067 -0.043 0.024 -0.014 -0.066 -0.052 rutgers s004 0.077 0.077 0.000 0.071 0.071 0.000 rutgers s005 0.031 0.657 0.626 0.265 0.656 0.391 rutgers s006 0.381 -0.119 -0.500 0.176 -0.356 -0.532 rutgers s007 -0.200 -0.208 -0.008 -0.483 -0.169 0.314 rutgers s008 -0.786 0.047 0.833 -0.769 0.160 0.929 rutgers s009 0.818 0.485 -0.333 0.727 0.496 -0.231 ----- ----- ----- ----- ----- ----- mean 0.048 0.000 -0.048 0.040 0.016 -0.023 std 0.432 0.416 0.517 0.446 0.398 0.500 nist ns8 -0.500 -0.833 -0.333 -0.324 -0.513 -0.189 nist ns2 0.000 1.000 1.000 -0.328 0.308 0.636 nist ns3 0.917 1.000 0.083 0.422 0.636 0.214 nist ns4 1.000 1.000 0.000 0.455 0.455 0.000 nist ns5 0.667 0.167 -0.500 0.231 0.140 -0.091 nist ns6 -0.500 -0.333 0.167 -0.091 -0.077 0.014 ----- ----- ----- ----- ----- ----- mean 0.264 0.333 0.069 0.061 0.158 0.097 std 0.688 0.796 0.523 0.357 0.411 0.296 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - All measures by search Site | Searcher | | System | | | System Sequence | | | | Topic | | | | | Search | | | | | | Number of docs saved | | | | | | | Elapsed time (secs) | | | | | | | | Precision | | | | | | | | | Recall | | | | | | | | | | Covered aspects | | | | | | | | | | | 1 bit per, leftmost =#1 city aji ok 1 258 a31 13 1172 0.538 0.692 1101110111010 city aji ru 3 292 a33 7 1212 0.143 0.214 00010011000000 city aji zp 2 274 a32 7 1204 0.714 0.545 11100111000 city att ok 2 274 b22 20 1186 0.900 0.818 11111011110 city att ru 3 292 b23 5 1197 0.200 0.214 00010011000000 city att zp 1 258 b21 8 1203 0.750 0.769 1011111011110 city bra ok 1 258 b11 8 1197 0.250 0.154 0000000001001 city bra ru 2 274 b12 13 1203 1.000 0.909 11110111111 city bra zp 3 292 b13 16 1245 0.250 0.429 00011111100000 city col ok 3 292 b33 13 1141 0.154 0.429 11110011000000 city col ru 1 258 b31 5 1096 0.600 0.231 1000000101000 city col zp 2 274 b32 4 1133 0.750 0.545 11110001100 city lea ok 2 274 a22 7 1136 0.714 0.545 11110001100 city lea ru 1 258 a21 2 1118 1.000 0.308 0001110000010 city lea zp 3 292 a23 6 1167 0.000 0.000 00000000000000 city mcf ok 3 292 a13 17 994 0.000 0.000 00000000000000 city mcf ru 2 274 a12 1 1182 1.000 0.091 10000000000 city mcf zp 1 258 a11 8 998 0.250 0.385 1010001001100 nist ns2 ok 2 258 ns2-ok-258 2 1200 1.000 0.308 0001110010000 nist ns2 ru 3 274 ns2-ru-274 6 1290 1.000 0.636 11110001110 nist ns2 zp 1 292 ns2-zp-292 3 1200 0.000 0.000 00000000000000 nist ns3 ok 3 274 ns3-ok-274 6 1200 1.000 0.636 11111001100 nist ns3 ru 1 292 ns3-ru-292 12 1200 0.083 0.214 00010011000000 nist ns3 zp 2 258 ns3-zp-258 1 660 0.000 0.000 0000000000000 nist ns4 ok 3 274 ns4-ok-274 4 1200 1.000 0.455 10100001110 nist ns4 ru 2 258 ns4-ru-258 1 1200 0.000 0.000 0000000000000 nist ns4 zp 1 292 ns4-zp-292 4 1200 0.000 0.000 00000000000000 nist ns5 ok 2 258 ns5-ok-258 3 1200 0.667 0.231 1000001001000 nist ns5 ru 1 292 ns5-ru-292 3 1200 0.000 0.000 00000000000000 nist ns5 zp 3 274 ns5-zp-274 2 1200 0.500 0.091 10000000000 nist ns6 ok 1 292 ns6-ok-292 0 1200 0.000 0.000 00000000000000 nist ns6 ru 3 274 ns6-ru-274 2 1265 0.500 0.091 10000000000 nist ns6 zp 2 258 ns6-zp-258 3 1200 0.333 0.077 0000000000001 nist ns8 ok 1 292 ns8-ok-292 6 1210 0.167 0.214 00010011000000 nist ns8 ru 2 258 ns8-ru-258 9 1205 0.667 0.538 1001110101010 nist ns8 zp 3 274 ns8-zp-274 7 1187 1.000 0.727 11111011100 rutgers s001 ok 1 274 s001.t274 10 1237 0.300 0.455 10100011100 rutgers s001 ru 2 292 s001.t292 8 1262 0.125 0.071 00000000100000 rutgers s001 zp 3 258 s001.t258 10 1235 0.400 0.462 1010001001101 rutgers s002 ok 2 292 s002.t292 11 1000 0.000 0.000 00000000000000 rutgers s002 ru 3 258 s002.t258 7 1290 0.000 0.000 0000000000000 rutgers s002 zp 1 274 s002.t274 5 1337 0.800 0.636 11111001010 rutgers s003 ok 3 258 s003.t258 10 1077 0.100 0.077 0000000000001 rutgers s003 ru 1 274 s003.t274 6 1360 0.167 0.091 10000000000 rutgers s003 zp 2 292 s003.t292 14 1171 0.143 0.143 00000000001001 rutgers s004 ok 3 292 s004.t292 13 1904 0.077 0.071 00010000000000 rutgers s004 ru 2 274 s004.t274 1 1363 0.000 0.000 00000000000 rutgers s004 zp 1 258 s004.t258 1 1212 0.000 0.000 0000000000000 rutgers s005 ok 2 274 s005.t274 10 1320 0.800 0.727 11111001110 rutgers s005 ru 1 258 s005.t258 13 1610 0.769 0.462 0011110011000 rutgers s005 zp 3 292 s005.t292 7 1178 0.143 0.071 00010000000000 rutgers s006 ok 1 258 s006.t258 7 1229 0.714 0.462 1001110101000 rutgers s006 ru 3 292 s006.t292 3 700 0.333 0.286 00010100110000 rutgers s006 zp 2 274 s006.t274 6 1169 0.833 0.818 11111111100 rutgers s007 ok 1 292 s007.t292 12 1284 0.167 0.286 00010000001110 rutgers s007 ru 2 258 s007.t258 30 1014 0.367 0.769 1011111011110 rutgers s007 zp 3 274 s007.t274 8 1094 0.375 0.455 11010001001 rutgers s008 ok 2 258 s008.t258 14 1418 0.214 0.231 0001000011000 rutgers s008 ru 3 274 s008.t274 15 1402 1.000 1.000 11111111111 rutgers s008 zp 1 292 s008.t292 6 1296 0.167 0.071 00010000000000 rutgers s009 ok 3 274 s009.t274 11 1263 0.818 0.727 11110001111 rutgers s009 ru 1 292 s009.t292 5 1398 0.000 0.000 00000000000000 rutgers s009 zp 2 258 s009.t258 3 992 0.333 0.231 0001110000000 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Precision and Recall by searcher, topic, and system Precision: Searcher | | 258 274 292 | --------------------- --------------------- --------------------- | ok ru zp ok ru zp ok ru zp ----- ----- ----- ----- ----- ----- ----- ----- ----- bra 0.250 1.000 0.250 att 0.750 0.900 0.200 col 0.600 0.750 0.154 mcf 0.250 1.000 0.000 lea 1.000 0.714 0.000 aji 0.538 0.714 0.143 --------------------- --------------------- --------------------- s001 0.400 0.300 0.125 s002 0.000 0.800 0.000 s003 0.100 0.167 0.143 s004 0.000 0.000 0.077 s005 0.769 0.800 0.143 s006 0.714 0.833 0.333 s007 0.367 0.375 0.167 s008 0.214 1.000 0.167 s009 0.333 0.818 0.000 --------------------- --------------------- --------------------- ns8 0.667 1.000 0.167 ns2 1.000 1.000 0.000 ns3 0.000 1.000 0.083 ns4 0.000 1.000 0.000 ns6 0.333 0.500 0.000 ns5 0.667 0.500 0.000 --------------------- --------------------- --------------------- Recall: 258 274 292 --------------------- --------------------- --------------------- ok ru zp ok ru zp ok ru zp ----- ----- ----- ----- ----- ----- ----- ----- ----- bra 0.154 0.909 0.429 att 0.769 0.818 0.214 col 0.231 0.545 0.429 mcf 0.385 0.091 0.000 lea 0.308 0.545 0.000 aji 0.692 0.545 0.214 --------------------- --------------------- --------------------- s001 0.462 0.455 0.071 s002 0.000 0.636 0.000 s003 0.077 0.091 0.143 s004 0.000 0.000 0.071 s005 0.462 0.727 0.071 s006 0.462 0.818 0.286 s007 0.769 0.455 0.286 s008 0.231 1.000 0.071 s009 0.231 0.727 0.000 --------------------- --------------------- --------------------- ns8 0.538 0.727 0.214 ns2 0.308 0.636 0.000 ns3 0.000 0.636 0.214 ns4 0.000 0.455 0.000 ns5 0.231 0.091 0.000 ns6 0.077 0.091 0.000 --------------------- --------------------- --------------------- ------------------------------------------------------------------------------- 5. Various higher-level views of the data for exploration Precision and recall by site across searchers within site, systems, and topics Precision Recall ---------------- ------------- mean std mean std # of searches ----- ----- ----- ----- -- City .512 .357 .404 .278 18 Rutgers .339 .316 .319 .302 27 Nist .440 .424 .234 .256 18 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Precision and recall by system across sites, searchers within sites, and topics ok 0.456 0.358 ru 0.426 0.292 zp 0.369 0.307 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Precision and recall by system sequence number across sites, searchers, systems and topics 1 0.320 0.259 2 0.503 0.362 3 0.428 0.337 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Precision and recall by topic across sites, searchers within sites, and systems 258 0.426 0.304 274 0.722 0.524 292 0.102 0.129 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Precision and recall by site and system across topics and searchers within sites Precision Recall Ok Ru Zp Ok Ru Zp City 0.426 0.657 0.452 0.440 0.328 0.446 Rutgers 0.354 0.307 0.355 0.337 0.298 0.321 NIST 0.639 0.375 0.305 0.307 0.247 0.149 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Precision and recall by system and topic across sites and searchers within sites Precision Recall 258 274 292 258 274 292 ok 0.498 0.790 0.081 0.308 0.623 0.143 ru 0.486 0.667 0.126 0.330 0.403 0.143 zp 0.295 0.710 0.100 0.275 0.545 0.102 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Precision and recall by site and topic across systems and searchers within sites Precision 258 274 292 258 274 292 city 0.565 0.846 0.124 0.423 0.576 0.214 rutgers 0.322 0.566 0.128 0.299 0.545 0.111 nist 0.444 0.833 0.042 0.192 0.439 0.071 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Precision and recall by searcher within site across various system-x-topic combinations Precision Searcher Recall bra 0.500 0.497 att 0.617 0.600 col 0.501 0.402 mcf 0.417 0.159 lea 0.571 0.284 aji 0.465 0.484 s001 0.275 0.329 s002 0.267 0.212 s003 0.137 0.104 s004 0.026 0.024 s005 0.571 0.420 s006 0.627 0.522 s007 0.303 0.503 s008 0.460 0.434 s009 0.384 0.319 ns8 0.611 0.493 ns2 0.667 0.315 ns3 0.361 0.283 ns4 0.333 0.152 ns5 0.389 0.107 ns6 0.278 0.056 ------------------------------------------------------------------------------- 6. Assessment results Topic-document-aspect matrix (aspectual content as assessed by NIST) Topic | TREC Document # | | Covered aspects 1 bit per possible aspect; | | | "1" = "covered by this document"; | | | leftmost bit = Aspect #1 | | | 258 FT911-3918 0000000000000 258 FT921-10278 0000000000000 258 FT921-11682 0000000000000 258 FT921-7240 0000000000000 258 FT922-13695 0000000000000 258 FT922-2176 1100000000000 258 FT922-2486 0000000000000 258 FT922-2528 0000000000000 258 FT922-468 0000000000000 258 FT922-5072 0000000000000 258 FT923-10088 0000000000000 258 FT923-11228 0000000000000 258 FT923-14087 0000000000000 258 FT923-2796 0000000000000 258 FT923-449 0010001000100 258 FT923-8456 0000000000000 258 FT924-13926 0000000000000 258 FT924-5109 0000000000000 258 FT924-8561 0000000000000 258 FT931-11384 0000000000000 258 FT931-1338 0000000000000 258 FT931-1831 0001000000000 258 FT931-2005 0000000000000 258 FT931-2806 0001110000000 258 FT931-2937 0001010000000 258 FT931-2938 0001010000000 258 FT931-3482 0000001000000 258 FT931-7090 0001000000000 258 FT931-7297 0000010000000 258 FT931-7553 0001110000000 258 FT932-14128 1000000100000 258 FT932-2922 0000000000000 258 FT932-3590 0000000000000 258 FT932-7637 0010100011000 258 FT932-7643 0010100011000 258 FT932-7897 0000000010000 258 FT932-9585 0000000000000 258 FT933-10240 0000000000000 258 FT933-11258 0000000000000 258 FT933-13340 0000000000000 258 FT933-13759 0000000000000 258 FT933-8055 0000000000000 258 FT933-9071 0000000000000 258 FT934-12464 0000000000000 258 FT934-13338 0000000000000 258 FT934-15659 0000000000000 258 FT934-5580 0000000000000 258 FT934-6622 0000000000000 258 FT941-11158 0000000000000 258 FT941-4379 0000000000000 258 FT941-6108 0000000000000 258 FT942-2223 0000000000000 258 FT942-5241 0000000000000 258 FT942-578 0000000000000 258 FT942-7826 0000000000000 258 FT942-7890 0000000000000 258 FT943-1215 0000000000000 258 FT943-12981 0000000000000 258 FT943-5307 0000000000000 258 FT943-9656 0000000000000 258 FT944-11064 0000000000000 258 FT944-15438 0000000000000 258 FT944-15440 0000000000000 258 FT944-16521 0000000000010 258 FT944-1726 0000000000001 258 FT944-7040 1000000001000 258 FT944-7314 0000000001000 258 FT944-7649 0000000001000 258 FT944-9024 0000000000000 258 FT944-9459 0000000000000 274 FT921-15508 11111000000 274 FT921-1556 11000111000 274 FT921-15760 00000000000 274 FT921-2636 00000000000 274 FT921-5193 01100001100 274 FT921-5494 00000000000 274 FT923-10730 01100001000 274 FT923-11292 10000000000 274 FT923-13374 00000000000 274 FT923-14293 10000000000 274 FT924-2212 00100001000 274 FT924-2358 00000010100 274 FT924-5561 00000000010 274 FT924-6107 00000000000 274 FT924-7255 00000000000 274 FT931-12869 00000001000 274 FT931-13738 00000000000 274 FT931-14618 10000001000 274 FT931-14778 10000001000 274 FT932-11975 00000000000 274 FT933-11925 00000000000 274 FT933-12570 00000000000 274 FT934-14844 10100000000 274 FT934-2530 00000000000 274 FT934-4790 00000000000 274 FT934-4995 00000000000 274 FT934-9597 00100000000 274 FT941-13218 00000000000 274 FT941-16334 00000000000 274 FT941-16609 10100000010 274 FT941-17640 00100001100 274 FT941-7609 00000001000 274 FT941-9725 00000000000 274 FT942-12506 00000000000 274 FT942-12711 00000000000 274 FT942-15407 00000000001 274 FT942-15408 00000000001 274 FT942-9950 10000000000 274 FT943-11205 00000000000 274 FT943-11627 00100000100 274 FT943-11755 00000000000 274 FT943-11934 10000001110 274 FT943-14882 00000000000 274 FT943-16085 00000000000 274 FT943-3737 00000000000 274 FT943-8478 10000000000 274 FT943-985 00000000000 274 FT944-16551 00000000000 274 FT944-16970 10010001000 274 FT944-17283 01000000000 274 FT944-17513 11010001000 274 FT944-18127 10100000000 274 FT944-18230 11100001100 274 FT944-3280 11010001000 274 FT944-4751 01000000100 292 FT911-1070 00000000000000 292 FT911-1691 00000000000000 292 FT921-10164 00000000000000 292 FT922-1215 00000000000000 292 FT922-12153 00000000000000 292 FT922-1568 11100000000000 292 FT922-2754 00000000000000 292 FT922-4425 00010000000000 292 FT922-5307 00000000000000 292 FT922-5936 00000000000000 292 FT922-7329 00000000000000 292 FT922-770 00000000000000 292 FT922-9180 00000000000000 292 FT922-9224 00000000000000 292 FT922-9330 00000000000000 292 FT923-2405 00000000000000 292 FT923-2790 00000000000000 292 FT923-289 00001100000000 292 FT923-8922 00000000000000 292 FT923-9849 00000000000000 292 FT924-10895 00000000000000 292 FT924-1107 00000000000000 292 FT924-11314 00010011000000 292 FT924-1207 00000000000000 292 FT924-1255 00000000000000 292 FT924-12824 00010100110000 292 FT924-14680 00000000000000 292 FT924-15607 00000000000000 292 FT924-2474 00000000000000 292 FT924-4469 00000000000000 292 FT931-1043 00000000000000 292 FT931-10935 00000000000000 292 FT931-14988 00000000000000 292 FT931-15653 00000000000000 292 FT931-258 00000000000000 292 FT931-2783 00000000000000 292 FT931-3537 00000000000000 292 FT931-7362 00000000000000 292 FT931-9432 00000000000000 292 FT932-1015 00000000000000 292 FT932-11049 00000000000000 292 FT932-12315 00000000000000 292 FT932-12747 00000000000000 292 FT932-12865 00000000000000 292 FT932-12991 00000000000000 292 FT932-3822 00000000000000 292 FT932-4859 00000000000000 292 FT932-4864 00000000000000 292 FT932-5425 00000000000000 292 FT932-5668 00000000000000 292 FT932-6510 00000000000000 292 FT932-7223 00000000000000 292 FT933-10651 00000000000000 292 FT933-14061 00000000000000 292 FT933-1422 00000000000000 292 FT933-14927 00000000000000 292 FT933-15869 00000000000000 292 FT933-2471 00000000000000 292 FT933-249 00000000000000 292 FT933-6509 00000000000000 292 FT933-9986 00010000000000 292 FT934-10045 00000000000000 292 FT934-10072 00000000000000 292 FT934-10231 00000000000000 292 FT934-1035 00000000000000 292 FT934-10921 00000000000000 292 FT934-11014 00000000000000 292 FT934-11906 00000000000000 292 FT934-12730 00000000001100 292 FT934-12800 00000000000000 292 FT934-13781 00000000000000 292 FT934-2265 00010000001010 292 FT934-2293 00000000000000 292 FT934-2514 00000000000000 292 FT934-4297 00000000000000 292 FT934-6695 00000000000000 292 FT934-7624 00000000000000 292 FT934-8104 00000000000000 292 FT934-8863 00000000000000 292 FT941-11779 00000000000000 292 FT941-12323 00000000000000 292 FT941-12334 00000000000000 292 FT941-12373 00000000000000 292 FT941-12426 00000000000000 292 FT941-16163 00000000000000 292 FT941-5213 00000000000000 292 FT941-5683 00000000000000 292 FT941-8918 00000000000000 292 FT942-10229 00000000000000 292 FT942-13791 00000000000000 292 FT942-15092 00000000000000 292 FT942-3981 00000000000000 292 FT942-4141 00000000000000 292 FT942-5248 00000000000000 292 FT942-5651 00000000000000 292 FT942-5904 00000000000000 292 FT943-14063 00000000000000 292 FT943-14531 00000000000000 292 FT943-15578 00000000000001 292 FT943-1684 00000000100000 292 FT943-1865 00000000000000 292 FT943-2556 00000000000000 292 FT943-274 00000000000000 292 FT943-3011 00000000000000 292 FT943-3430 00000000001000 292 FT943-5084 00000000000000 292 FT943-8923 00000000000000 292 FT944-10159 00000000000000 292 FT944-15084 00000000000000 292 FT944-2202 00000000000000 292 FT944-6246 00000000000000 292 FT944-642 00000000000000 292 FT944-6725 00000000000000 292 FT944-8202 00000000000000 292 FT944-9133 00000000000000 292 FT944-9190 00000000000000 292 FT944-9193 00000000000000 - - - - - - - - - - - - - - - - - Aspect glosses from NIST assessors Topic | Aspect # | | Descriptive Phrase 258 1 Prince Philips Electronic Mailbox 258 2 Earth to Space Shuttle Atlantis Comm's 258 3 NASA 258 4 Financial Times 258 5 Headquarters of the European Community (EC) 258 6 European Organization for Cancer Research/Treatment 258 7 Pentagon 258 8 CERN Nuclear Research Lab in Geneva 258 9 Bank, SG Warburg 258 10 British Telecommunications 258 11 NATO 258 12 Hospital Computer System 258 13 General Electric (U.S.) Sensitive Material 274 1 governmental requirements/incentives 274 2 manufacturers' decision to produce/sell 274 3 commercial trials 274 4 improved battery pack 274 5 quick recharge facility 274 6 lightweight materials 274 7 low-drag aerodynamics/tires 274 8 industrial collaboration 274 9 hybrid-elect-petrol 274 10 govt.-industrial collaboration 274 11 fuel-cell technology 292 1 Cuts in Capital Spending 292 2 Making Poor Pay For Medicines 292 3 Free Medical Treatment 292 4 Pensions (Monthly or Weekly Income For Poor) 292 5 Annual Income Support Entitlement (Unemployment Comp) 292 6 Social Security (For Poor) 292 7 Community-Related Work 292 8 Entitlement (Monies for Collateral-Loan) 292 9 Child Benefit 292 10 No Benefit (For Those With Sufficient Capital) 292 11 Unemployment - Received 2/3 of Former Salary For 32 Mo. 292 12 Self-Financing / Contribution-Driven 292 13 Percentage of GNP 292 14 Family Credit ------------------------------------------------------------------------------- 7. Search data submitted for evaluation site search searcher system system-sequence topic time (secs) doc-seq doc city a11 mcf zp 1 258 998 1 FT941-6108 city a11 mcf zp 1 258 998 2 FT944-15438 city a11 mcf zp 1 258 998 3 FT944-7040 city a11 mcf zp 1 258 998 4 FT911-3918 city a11 mcf zp 1 258 998 5 FT934-12464 city a11 mcf zp 1 258 998 6 FT923-449 city a11 mcf zp 1 258 998 7 FT922-2528 city a11 mcf zp 1 258 998 8 FT924-5109 city a12 mcf ru 2 274 1182 1 FT923-11292 city a13 mcf ok 3 292 994 1 FT944-6246 city a13 mcf ok 3 292 994 2 FT923-8922 city a13 mcf ok 3 292 994 3 FT931-2783 city a13 mcf ok 3 292 994 4 FT933-14061 city a13 mcf ok 3 292 994 5 FT931-7362 city a13 mcf ok 3 292 994 6 FT932-12747 city a13 mcf ok 3 292 994 7 FT922-5936 city a13 mcf ok 3 292 994 8 FT931-14988 city a13 mcf ok 3 292 994 9 FT941-12373 city a13 mcf ok 3 292 994 10 FT923-2790 city a13 mcf ok 3 292 994 11 FT933-2471 city a13 mcf ok 3 292 994 12 FT934-13781 city a13 mcf ok 3 292 994 13 FT943-1865 city a13 mcf ok 3 292 994 14 FT922-9224 city a13 mcf ok 3 292 994 15 FT941-16163 city a13 mcf ok 3 292 994 16 FT933-14927 city a13 mcf ok 3 292 994 17 FT943-14063 city a21 lea ru 1 258 1118 9 FT944-16521 city a21 lea ru 1 258 1118 10 FT931-7553 city a22 lea ok 2 274 1136 1 FT943-8478 city a22 lea ok 2 274 1136 2 FT931-12869 city a22 lea ok 2 274 1136 3 FT941-17640 city a22 lea ok 2 274 1136 4 FT931-13738 city a22 lea ok 2 274 1136 5 FT944-16551 city a22 lea ok 2 274 1136 6 FT944-3280 city a22 lea ok 2 274 1136 7 FT934-14844 city a23 lea zp 3 292 1167 1 FT934-12800 city a23 lea zp 3 292 1167 2 FT934-10921 city a23 lea zp 3 292 1167 3 FT934-8104 city a23 lea zp 3 292 1167 4 FT934-4297 city a23 lea zp 3 292 1167 6 FT942-5651 city a23 lea zp 3 292 1167 5 FT922-12153 city a31 aji ok 1 258 1172 6 FT922-2528 city a31 aji ok 1 258 1172 1 FT944-7649 city a31 aji ok 1 258 1172 2 FT933-8055 city a31 aji ok 1 258 1172 3 FT932-7897 city a31 aji ok 1 258 1172 4 FT931-7553 city a31 aji ok 1 258 1172 5 FT911-3918 city a31 aji ok 1 258 1172 7 FT944-7040 city a31 aji ok 1 258 1172 8 FT942-578 city a31 aji ok 1 258 1172 9 FT931-2005 city a31 aji ok 1 258 1172 10 FT943-1215 city a31 aji ok 1 258 1172 11 FT922-2176 city a31 aji ok 1 258 1172 12 FT932-14128 city a31 aji ok 1 258 1172 13 FT944-16521 city a32 aji zp 2 274 1204 1 FT921-1556 city a32 aji zp 2 274 1204 2 FT931-14778 city a32 aji zp 2 274 1204 3 FT931-14618 city a32 aji zp 2 274 1204 4 FT944-17283 city a32 aji zp 2 274 1204 5 FT941-16334 city a32 aji zp 2 274 1204 6 FT942-12711 city a32 aji zp 2 274 1204 7 FT924-2212 city a33 aji ru 3 292 1212 1 FT941-12426 city a33 aji ru 3 292 1212 2 FT942-5651 city a33 aji ru 3 292 1212 3 FT922-12153 city a33 aji ru 3 292 1212 4 FT924-11314 city a33 aji ru 3 292 1212 5 FT944-6725 city a33 aji ru 3 292 1212 6 FT932-3822 city a33 aji ru 3 292 1212 13 FT931-1043 city b11 bra ok 1 258 1197 1 FT944-7314 city b11 bra ok 1 258 1197 2 FT943-1215 city b11 bra ok 1 258 1197 3 FT944-1726 city b11 bra ok 1 258 1197 4 FT922-2528 city b11 bra ok 1 258 1197 5 FT942-5241 city b11 bra ok 1 258 1197 6 FT932-2922 city b11 bra ok 1 258 1197 7 FT933-13340 city b11 bra ok 1 258 1197 8 FT933-10240 city b12 bra ru 2 274 1203 26 FT942-9950 city b12 bra ru 2 274 1203 29 FT942-15407 city b12 bra ru 2 274 1203 41 FT944-18230 city b12 bra ru 2 274 1203 42 FT943-11934 city b12 bra ru 2 274 1203 45 FT921-1556 city b12 bra ru 2 274 1203 53 FT943-8478 city b12 bra ru 2 274 1203 54 FT944-16970 city b12 bra ru 2 274 1203 55 FT943-11627 city b12 bra ru 2 274 1203 56 FT944-3280 city b12 bra ru 2 274 1203 57 FT944-4751 city b12 bra ru 2 274 1203 58 FT941-17640 city b12 bra ru 2 274 1203 59 FT924-2358 city b12 bra ru 2 274 1203 60 FT923-11292 city b13 bra zp 3 292 1245 1 FT943-1684 city b13 bra zp 3 292 1245 2 FT942-5248 city b13 bra zp 3 292 1245 3 FT934-8104 city b13 bra zp 3 292 1245 4 FT934-10921 city b13 bra zp 3 292 1245 5 FT934-7624 city b13 bra zp 3 292 1245 6 FT944-9193 city b13 bra zp 3 292 1245 7 FT942-10229 city b13 bra zp 3 292 1245 8 FT934-2293 city b13 bra zp 3 292 1245 9 FT922-5307 city b13 bra zp 3 292 1245 10 FT942-5651 city b13 bra zp 3 292 1245 11 FT922-12153 city b13 bra zp 3 292 1245 12 FT924-11314 city b13 bra zp 3 292 1245 13 FT923-289 city b13 bra zp 3 292 1245 14 FT931-1043 city b13 bra zp 3 292 1245 15 FT922-4425 city b13 bra zp 3 292 1245 16 FT944-10159 city b21 att zp 1 258 1203 1 FT931-2806 city b21 att zp 1 258 1203 2 FT932-7643 city b21 att zp 1 258 1203 3 FT934-12464 city b21 att zp 1 258 1203 4 FT944-7040 city b21 att zp 1 258 1203 5 FT923-449 city b21 att zp 1 258 1203 6 FT944-16521 city b21 att zp 1 258 1203 7 FT921-11682 city b21 att zp 1 258 1203 8 FT932-7897 city b22 att ok 2 274 1186 1 FT923-10730 city b22 att ok 2 274 1186 2 FT941-7609 city b22 att ok 2 274 1186 3 FT924-5561 city b22 att ok 2 274 1186 4 FT943-8478 city b22 att ok 2 274 1186 5 FT934-9597 city b22 att ok 2 274 1186 6 FT943-3737 city b22 att ok 2 274 1186 7 FT931-13738 city b22 att ok 2 274 1186 8 FT931-14618 city b22 att ok 2 274 1186 9 FT931-14778 city b22 att ok 2 274 1186 10 FT944-18230 city b22 att ok 2 274 1186 11 FT941-17640 city b22 att ok 2 274 1186 12 FT924-2358 city b22 att ok 2 274 1186 13 FT921-15508 city b22 att ok 2 274 1186 14 FT944-3280 city b22 att ok 2 274 1186 15 FT931-12869 city b22 att ok 2 274 1186 16 FT921-5193 city b22 att ok 2 274 1186 17 FT923-11292 city b22 att ok 2 274 1186 18 FT944-4751 city b22 att ok 2 274 1186 19 FT944-16970 city b22 att ok 2 274 1186 20 FT934-14844 city b23 att ru 3 292 1197 4 FT932-12865 city b23 att ru 3 292 1197 5 FT911-1691 city b23 att ru 3 292 1197 9 FT911-1070 city b23 att ru 3 292 1197 10 FT924-11314 city b23 att ru 3 292 1197 12 FT943-14531 city b31 col ru 1 258 1096 1 FT944-7040 city b31 col ru 1 258 1096 2 FT922-13695 city b31 col ru 1 258 1096 12 FT944-7649 city b31 col ru 1 258 1096 15 FT942-578 city b31 col ru 1 258 1096 16 FT932-14128 city b32 col zp 2 274 1133 1 FT943-8478 city b32 col zp 2 274 1133 2 FT944-3280 city b32 col zp 2 274 1133 3 FT934-4790 city b32 col zp 2 274 1133 4 FT941-17640 city b33 col ok 3 292 1141 1 FT942-3981 city b33 col ok 3 292 1141 2 FT942-4141 city b33 col ok 3 292 1141 3 FT942-5651 city b33 col ok 3 292 1141 4 FT931-258 city b33 col ok 3 292 1141 5 FT922-1568 city b33 col ok 3 292 1141 6 FT942-13791 city b33 col ok 3 292 1141 7 FT922-9180 city b33 col ok 3 292 1141 8 FT933-1422 city b33 col ok 3 292 1141 9 FT924-11314 city b33 col ok 3 292 1141 10 FT922-770 city b33 col ok 3 292 1141 11 FT931-1043 city b33 col ok 3 292 1141 12 FT922-7329 city b33 col ok 3 292 1141 13 FT932-12315 nist ns2-ok-258 ns2 ok 2 258 1200 1 FT932-7897 nist ns2-ok-258 ns2 ok 2 258 1200 2 FT931-2806 nist ns2-ru-274 ns2 ru 3 274 1290 1 FT931-12869 nist ns2-ru-274 ns2 ru 3 274 1290 2 FT944-17513 nist ns2-ru-274 ns2 ru 3 274 1290 3 FT944-3280 nist ns2-ru-274 ns2 ru 3 274 1290 4 FT941-17640 nist ns2-ru-274 ns2 ru 3 274 1290 5 FT941-16609 nist ns2-ru-274 ns2 ru 3 274 1290 6 FT944-4751 nist ns2-zp-292 ns2 zp 1 292 1200 1 FT934-10921 nist ns2-zp-292 ns2 zp 1 292 1200 2 FT932-7223 nist ns2-zp-292 ns2 zp 1 292 1200 3 FT932-4859 nist ns3-ok-274 ns3 ok 3 274 1200 1 FT943-8478 nist ns3-ok-274 ns3 ok 3 274 1200 2 FT944-3280 nist ns3-ok-274 ns3 ok 3 274 1200 3 FT924-2212 nist ns3-ok-274 ns3 ok 3 274 1200 4 FT944-18127 nist ns3-ok-274 ns3 ok 3 274 1200 5 FT944-18230 nist ns3-ok-274 ns3 ok 3 274 1200 6 FT921-15508 nist ns3-ru-292 ns3 ru 1 292 1200 1 FT943-5084 nist ns3-ru-292 ns3 ru 1 292 1200 2 FT932-4864 nist ns3-ru-292 ns3 ru 1 292 1200 3 FT944-9190 nist ns3-ru-292 ns3 ru 1 292 1200 4 FT922-2754 nist ns3-ru-292 ns3 ru 1 292 1200 5 FT924-14680 nist ns3-ru-292 ns3 ru 1 292 1200 6 FT942-5248 nist ns3-ru-292 ns3 ru 1 292 1200 7 FT924-11314 nist ns3-ru-292 ns3 ru 1 292 1200 8 FT934-8104 nist ns3-ru-292 ns3 ru 1 292 1200 9 FT944-8202 nist ns3-ru-292 ns3 ru 1 292 1200 10 FT923-2405 nist ns3-ru-292 ns3 ru 1 292 1200 11 FT924-10895 nist ns3-ru-292 ns3 ru 1 292 1200 12 FT944-6725 nist ns3-zp-258 ns3 zp 2 258 660 1 FT934-15659 nist ns4-ok-274 ns4 ok 3 274 1200 1 FT941-17640 nist ns4-ok-274 ns4 ok 3 274 1200 2 FT931-12869 nist ns4-ok-274 ns4 ok 3 274 1200 3 FT943-11934 nist ns4-ok-274 ns4 ok 3 274 1200 4 FT942-9950 nist ns4-ru-258 ns4 ru 2 258 1200 1 FT943-5307 nist ns4-zp-292 ns4 zp 1 292 1200 2 FT943-8923 nist ns4-zp-292 ns4 zp 1 292 1200 3 FT942-5248 nist ns4-zp-292 ns4 zp 1 292 1200 4 FT922-1215 nist ns4-zp-292 ns4 zp 1 292 1200 5 FT924-4469 nist ns5-ok-258 ns5 ok 2 258 1200 1 FT931-3482 nist ns5-ok-258 ns5 ok 2 258 1200 2 FT931-2005 nist ns5-ok-258 ns5 ok 2 258 1200 3 FT944-7040 nist ns5-ru-292 ns5 ru 1 292 1200 1 FT942-5904 nist ns5-ru-292 ns5 ru 1 292 1200 2 FT933-15869 nist ns5-ru-292 ns5 ru 1 292 1200 3 FT924-2474 nist ns5-zp-274 ns5 zp 3 274 1200 1 FT943-8478 nist ns5-zp-274 ns5 zp 3 274 1200 2 FT921-2636 nist ns6-ok-292 ns6 ok 1 292 1200 1 0 nist ns6-ru-274 ns6 ru 3 274 1265 1 FT943-8478 nist ns6-ru-274 ns6 ru 3 274 1265 2 FT931-13738 nist ns6-zp-258 ns6 zp 2 258 1200 1 FT944-15438 nist ns6-zp-258 ns6 zp 2 258 1200 2 FT944-1726 nist ns6-zp-258 ns6 zp 2 258 1200 3 FT923-2796 nist ns8-ok-292 ns8 ok 1 292 1210 1 FT934-13781 nist ns8-ok-292 ns8 ok 1 292 1210 2 FT924-11314 nist ns8-ok-292 ns8 ok 1 292 1210 3 FT924-1255 nist ns8-ok-292 ns8 ok 1 292 1210 4 FT941-11779 nist ns8-ok-292 ns8 ok 1 292 1210 5 FT924-1207 nist ns8-ok-292 ns8 ok 1 292 1210 6 FT941-12426 nist ns8-ru-258 ns8 ru 2 258 1205 1 FT932-14128 nist ns8-ru-258 ns8 ru 2 258 1205 2 FT944-7040 nist ns8-ru-258 ns8 ru 2 258 1205 3 FT934-12464 nist ns8-ru-258 ns8 ru 2 258 1205 4 FT931-2005 nist ns8-ru-258 ns8 ru 2 258 1205 5 FT924-8561 nist ns8-ru-258 ns8 ru 2 258 1205 6 FT944-7314 nist ns8-ru-258 ns8 ru 2 258 1205 7 FT944-16521 nist ns8-ru-258 ns8 ru 2 258 1205 8 FT931-2806 nist ns8-ru-258 ns8 ru 2 258 1205 9 FT931-7090 nist ns8-zp-274 ns8 zp 3 274 1187 1 FT941-17640 nist ns8-zp-274 ns8 zp 3 274 1187 2 FT923-11292 nist ns8-zp-274 ns8 zp 3 274 1187 3 FT944-18127 nist ns8-zp-274 ns8 zp 3 274 1187 4 FT931-12869 nist ns8-zp-274 ns8 zp 3 274 1187 5 FT944-18230 nist ns8-zp-274 ns8 zp 3 274 1187 6 FT924-2358 nist ns8-zp-274 ns8 zp 3 274 1187 7 FT921-15508 rutgers s001.t258 s001 zp 3 258 1235 1 FT942-578 rutgers s001.t258 s001 zp 3 258 1235 2 FT931-2005 rutgers s001.t258 s001 zp 3 258 1235 3 FT944-1726 rutgers s001.t258 s001 zp 3 258 1235 4 FT941-6108 rutgers s001.t258 s001 zp 3 258 1235 5 FT911-3918 rutgers s001.t258 s001 zp 3 258 1235 6 FT934-12464 rutgers s001.t258 s001 zp 3 258 1235 7 FT944-7314 rutgers s001.t258 s001 zp 3 258 1235 8 FT923-449 rutgers s001.t258 s001 zp 3 258 1235 9 FT944-7040 rutgers s001.t258 s001 zp 3 258 1235 10 FT941-11158 rutgers s001.t274 s001 ok 1 274 1237 1 FT943-11755 rutgers s001.t274 s001 ok 1 274 1237 2 FT943-985 rutgers s001.t274 s001 ok 1 274 1237 3 FT943-8478 rutgers s001.t274 s001 ok 1 274 1237 4 FT943-3737 rutgers s001.t274 s001 ok 1 274 1237 5 FT921-2636 rutgers s001.t274 s001 ok 1 274 1237 6 FT933-11925 rutgers s001.t274 s001 ok 1 274 1237 7 FT934-4995 rutgers s001.t274 s001 ok 1 274 1237 8 FT934-2530 rutgers s001.t274 s001 ok 1 274 1237 9 FT924-2358 rutgers s001.t274 s001 ok 1 274 1237 10 FT941-17640 rutgers s001.t292 s001 ru 2 292 1262 1 FT943-1684 rutgers s001.t292 s001 ru 2 292 1262 2 FT941-11779 rutgers s001.t292 s001 ru 2 292 1262 3 FT942-10229 rutgers s001.t292 s001 ru 2 292 1262 4 FT934-8104 rutgers s001.t292 s001 ru 2 292 1262 5 FT934-6695 rutgers s001.t292 s001 ru 2 292 1262 6 FT944-10159 rutgers s001.t292 s001 ru 2 292 1262 7 FT931-9432 rutgers s001.t292 s001 ru 2 292 1262 8 FT934-11906 rutgers s002.t258 s002 ru 3 258 1290 1 FT923-8456 rutgers s002.t258 s002 ru 3 258 1290 2 FT944-9459 rutgers s002.t258 s002 ru 3 258 1290 3 FT942-7826 rutgers s002.t258 s002 ru 3 258 1290 4 FT922-468 rutgers s002.t258 s002 ru 3 258 1290 5 FT923-14087 rutgers s002.t258 s002 ru 3 258 1290 6 FT911-3918 rutgers s002.t258 s002 ru 3 258 1290 7 FT931-1338 rutgers s002.t274 s002 zp 1 274 1337 1 FT921-15508 rutgers s002.t274 s002 zp 1 274 1337 2 FT943-8478 rutgers s002.t274 s002 zp 1 274 1337 3 FT931-14618 rutgers s002.t274 s002 zp 1 274 1337 4 FT932-11975 rutgers s002.t274 s002 zp 1 274 1337 5 FT941-16609 rutgers s002.t292 s002 ok 2 292 1000 1 FT922-9330 rutgers s002.t292 s002 ok 2 292 1000 2 FT934-8863 rutgers s002.t292 s002 ok 2 292 1000 3 FT943-2556 rutgers s002.t292 s002 ok 2 292 1000 4 FT921-10164 rutgers s002.t292 s002 ok 2 292 1000 5 FT923-9849 rutgers s002.t292 s002 ok 2 292 1000 6 FT922-12153 rutgers s002.t292 s002 ok 2 292 1000 7 FT943-3011 rutgers s002.t292 s002 ok 2 292 1000 8 FT932-5425 rutgers s002.t292 s002 ok 2 292 1000 9 FT931-3537 rutgers s002.t292 s002 ok 2 292 1000 10 FT941-5683 rutgers s002.t292 s002 ok 2 292 1000 11 FT943-8923 rutgers s003.t258 s003 ok 3 258 1077 1 FT931-2005 rutgers s003.t258 s003 ok 3 258 1077 2 FT944-1726 rutgers s003.t258 s003 ok 3 258 1077 3 FT934-12464 rutgers s003.t258 s003 ok 3 258 1077 4 FT924-8561 rutgers s003.t258 s003 ok 3 258 1077 5 FT911-3918 rutgers s003.t258 s003 ok 3 258 1077 6 FT941-6108 rutgers s003.t258 s003 ok 3 258 1077 7 FT924-13926 rutgers s003.t258 s003 ok 3 258 1077 8 FT933-10240 rutgers s003.t258 s003 ok 3 258 1077 9 FT934-5580 rutgers s003.t258 s003 ok 3 258 1077 10 FT944-9024 rutgers s003.t274 s003 ru 1 274 1360 1 FT943-14882 rutgers s003.t274 s003 ru 1 274 1360 2 FT921-5494 rutgers s003.t274 s003 ru 1 274 1360 3 FT924-6107 rutgers s003.t274 s003 ru 1 274 1360 4 FT921-2636 rutgers s003.t274 s003 ru 1 274 1360 5 FT943-8478 rutgers s003.t274 s003 ru 1 274 1360 6 FT943-16085 rutgers s003.t292 s003 zp 2 292 1171 1 FT934-10921 rutgers s003.t292 s003 zp 2 292 1171 2 FT934-4297 rutgers s003.t292 s003 zp 2 292 1171 3 FT941-12323 rutgers s003.t292 s003 zp 2 292 1171 4 FT934-10072 rutgers s003.t292 s003 zp 2 292 1171 5 FT942-5248 rutgers s003.t292 s003 zp 2 292 1171 6 FT943-3430 rutgers s003.t292 s003 zp 2 292 1171 7 FT934-11014 rutgers s003.t292 s003 zp 2 292 1171 8 FT934-7624 rutgers s003.t292 s003 zp 2 292 1171 9 FT932-11049 rutgers s003.t292 s003 zp 2 292 1171 10 FT924-15607 rutgers s003.t292 s003 zp 2 292 1171 11 FT924-1107 rutgers s003.t292 s003 zp 2 292 1171 12 FT934-10231 rutgers s003.t292 s003 zp 2 292 1171 13 FT934-12800 rutgers s003.t292 s003 zp 2 292 1171 14 FT943-15578 rutgers s004.t258 s004 zp 1 258 1212 1 FT944-15438 rutgers s004.t274 s004 ru 2 274 1363 1 FT921-5494 rutgers s004.t292 s004 ok 3 292 1904 1 FT941-12334 rutgers s004.t292 s004 ok 3 292 1904 2 FT933-9986 rutgers s004.t292 s004 ok 3 292 1904 3 FT934-10045 rutgers s004.t292 s004 ok 3 292 1904 4 FT932-1015 rutgers s004.t292 s004 ok 3 292 1904 5 FT931-15653 rutgers s004.t292 s004 ok 3 292 1904 6 FT933-6509 rutgers s004.t292 s004 ok 3 292 1904 7 FT942-5248 rutgers s004.t292 s004 ok 3 292 1904 8 FT944-2202 rutgers s004.t292 s004 ok 3 292 1904 9 FT942-15092 rutgers s004.t292 s004 ok 3 292 1904 10 FT931-10935 rutgers s004.t292 s004 ok 3 292 1904 11 FT934-13781 rutgers s004.t292 s004 ok 3 292 1904 12 FT933-10651 rutgers s004.t292 s004 ok 3 292 1904 13 FT932-12991 rutgers s005.t258 s005 ru 1 258 1610 1 FT944-7314 rutgers s005.t258 s005 ru 1 258 1610 2 FT931-7553 rutgers s005.t258 s005 ru 1 258 1610 3 FT944-7649 rutgers s005.t258 s005 ru 1 258 1610 4 FT922-13695 rutgers s005.t258 s005 ru 1 258 1610 5 FT932-7897 rutgers s005.t258 s005 ru 1 258 1610 6 FT932-7643 rutgers s005.t258 s005 ru 1 258 1610 7 FT931-2806 rutgers s005.t258 s005 ru 1 258 1610 8 FT923-2796 rutgers s005.t258 s005 ru 1 258 1610 9 FT931-1831 rutgers s005.t258 s005 ru 1 258 1610 10 FT944-15438 rutgers s005.t258 s005 ru 1 258 1610 11 FT931-7297 rutgers s005.t258 s005 ru 1 258 1610 12 FT931-7090 rutgers s005.t258 s005 ru 1 258 1610 13 FT932-7637 rutgers s005.t274 s005 ok 2 274 1320 1 FT944-18127 rutgers s005.t274 s005 ok 2 274 1320 2 FT923-14293 rutgers s005.t274 s005 ok 2 274 1320 3 FT943-11934 rutgers s005.t274 s005 ok 2 274 1320 4 FT923-11292 rutgers s005.t274 s005 ok 2 274 1320 5 FT924-7255 rutgers s005.t274 s005 ok 2 274 1320 6 FT941-17640 rutgers s005.t274 s005 ok 2 274 1320 7 FT944-18230 rutgers s005.t274 s005 ok 2 274 1320 8 FT921-15508 rutgers s005.t274 s005 ok 2 274 1320 9 FT944-3280 rutgers s005.t274 s005 ok 2 274 1320 10 FT941-9725 rutgers s005.t292 s005 zp 3 292 1178 1 FT944-642 rutgers s005.t292 s005 zp 3 292 1178 2 FT941-5213 rutgers s005.t292 s005 zp 3 292 1178 3 FT934-12800 rutgers s005.t292 s005 zp 3 292 1178 4 FT931-1043 rutgers s005.t292 s005 zp 3 292 1178 5 FT941-8918 rutgers s005.t292 s005 zp 3 292 1178 6 FT922-4425 rutgers s005.t292 s005 zp 3 292 1178 7 FT932-12865 rutgers s006.t258 s006 ok 1 258 1229 1 FT931-7553 rutgers s006.t258 s006 ok 1 258 1229 2 FT931-11384 rutgers s006.t258 s006 ok 1 258 1229 3 FT931-7090 rutgers s006.t258 s006 ok 1 258 1229 4 FT942-578 rutgers s006.t258 s006 ok 1 258 1229 5 FT944-7649 rutgers s006.t258 s006 ok 1 258 1229 6 FT932-14128 rutgers s006.t258 s006 ok 1 258 1229 7 FT931-2937 rutgers s006.t274 s006 zp 2 274 1169 1 FT921-1556 rutgers s006.t274 s006 zp 2 274 1169 2 FT944-3280 rutgers s006.t274 s006 zp 2 274 1169 3 FT941-17640 rutgers s006.t274 s006 zp 2 274 1169 4 FT931-12869 rutgers s006.t274 s006 zp 2 274 1169 5 FT933-12570 rutgers s006.t274 s006 zp 2 274 1169 6 FT921-15508 rutgers s006.t292 s006 ru 3 292 700 1 FT942-5248 rutgers s006.t292 s006 ru 3 292 700 2 FT932-1015 rutgers s006.t292 s006 ru 3 292 700 3 FT924-12824 rutgers s007.t258 s007 ru 2 258 1014 1 FT944-7040 rutgers s007.t258 s007 ru 2 258 1014 10 FT931-7553 rutgers s007.t258 s007 ru 2 258 1014 2 FT944-15438 rutgers s007.t258 s007 ru 2 258 1014 3 FT911-3918 rutgers s007.t258 s007 ru 2 258 1014 4 FT943-1215 rutgers s007.t258 s007 ru 2 258 1014 5 FT921-11682 rutgers s007.t258 s007 ru 2 258 1014 6 FT923-2796 rutgers s007.t258 s007 ru 2 258 1014 7 FT944-7314 rutgers s007.t258 s007 ru 2 258 1014 8 FT931-2806 rutgers s007.t258 s007 ru 2 258 1014 9 FT944-11064 rutgers s007.t258 s007 ru 2 258 1014 11 FT933-11258 rutgers s007.t258 s007 ru 2 258 1014 12 FT931-7090 rutgers s007.t258 s007 ru 2 258 1014 13 FT931-1831 rutgers s007.t258 s007 ru 2 258 1014 14 FT931-2937 rutgers s007.t258 s007 ru 2 258 1014 15 FT931-2938 rutgers s007.t258 s007 ru 2 258 1014 16 FT944-16521 rutgers s007.t258 s007 ru 2 258 1014 17 FT922-2486 rutgers s007.t258 s007 ru 2 258 1014 18 FT933-13759 rutgers s007.t258 s007 ru 2 258 1014 19 FT923-10088 rutgers s007.t258 s007 ru 2 258 1014 20 FT923-449 rutgers s007.t258 s007 ru 2 258 1014 21 FT943-9656 rutgers s007.t258 s007 ru 2 258 1014 22 FT934-6622 rutgers s007.t258 s007 ru 2 258 1014 23 FT933-9071 rutgers s007.t258 s007 ru 2 258 1014 24 FT944-15440 rutgers s007.t258 s007 ru 2 258 1014 25 FT932-7637 rutgers s007.t258 s007 ru 2 258 1014 26 FT941-4379 rutgers s007.t258 s007 ru 2 258 1014 27 FT921-7240 rutgers s007.t258 s007 ru 2 258 1014 28 FT934-13338 rutgers s007.t258 s007 ru 2 258 1014 29 FT923-11228 rutgers s007.t258 s007 ru 2 258 1014 30 FT932-9585 rutgers s007.t274 s007 zp 3 274 1094 1 FT944-3280 rutgers s007.t274 s007 zp 3 274 1094 2 FT943-8478 rutgers s007.t274 s007 zp 3 274 1094 3 FT944-3280 rutgers s007.t274 s007 zp 3 274 1094 4 FT943-8478 rutgers s007.t274 s007 zp 3 274 1094 5 FT943-3737 rutgers s007.t274 s007 zp 3 274 1094 6 FT921-2636 rutgers s007.t274 s007 zp 3 274 1094 7 FT942-15408 rutgers s007.t274 s007 zp 3 274 1094 8 FT923-13374 rutgers s007.t274 s007 zp 3 274 1094 9 FT943-11205 rutgers s007.t274 s007 zp 3 274 1094 10 FT921-15760 rutgers s007.t292 s007 ok 1 292 1284 1 FT942-5248 rutgers s007.t292 s007 ok 1 292 1284 2 FT934-2265 rutgers s007.t292 s007 ok 1 292 1284 3 FT941-11779 rutgers s007.t292 s007 ok 1 292 1284 4 FT944-15084 rutgers s007.t292 s007 ok 1 292 1284 5 FT933-249 rutgers s007.t292 s007 ok 1 292 1284 6 FT932-7223 rutgers s007.t292 s007 ok 1 292 1284 7 FT934-2514 rutgers s007.t292 s007 ok 1 292 1284 8 FT934-1035 rutgers s007.t292 s007 ok 1 292 1284 9 FT944-9133 rutgers s007.t292 s007 ok 1 292 1284 10 FT934-12730 rutgers s007.t292 s007 ok 1 292 1284 11 FT932-6510 rutgers s007.t292 s007 ok 1 292 1284 12 FT942-10229 rutgers s008.t258 s008 ok 2 258 1418 1 FT942-2223 rutgers s008.t258 s008 ok 2 258 1418 2 FT932-3590 rutgers s008.t258 s008 ok 2 258 1418 3 FT921-10278 rutgers s008.t258 s008 ok 2 258 1418 4 FT942-578 rutgers s008.t258 s008 ok 2 258 1418 5 FT931-7090 rutgers s008.t258 s008 ok 2 258 1418 6 FT942-7890 rutgers s008.t258 s008 ok 2 258 1418 7 FT922-2528 rutgers s008.t258 s008 ok 2 258 1418 8 FT943-12981 rutgers s008.t258 s008 ok 2 258 1418 9 FT934-12464 rutgers s008.t258 s008 ok 2 258 1418 10 FT922-5072 rutgers s008.t258 s008 ok 2 258 1418 11 FT944-7314 rutgers s008.t258 s008 ok 2 258 1418 12 FT933-10240 rutgers s008.t258 s008 ok 2 258 1418 13 FT931-2005 rutgers s008.t258 s008 ok 2 258 1418 14 FT932-7897 rutgers s008.t274 s008 ru 3 274 1402 1 FT944-18230 rutgers s008.t274 s008 ru 3 274 1402 2 FT941-16609 rutgers s008.t274 s008 ru 3 274 1402 3 FT944-3280 rutgers s008.t274 s008 ru 3 274 1402 4 FT941-17640 rutgers s008.t274 s008 ru 3 274 1402 5 FT924-2358 rutgers s008.t274 s008 ru 3 274 1402 6 FT923-14293 rutgers s008.t274 s008 ru 3 274 1402 7 FT923-10730 rutgers s008.t274 s008 ru 3 274 1402 8 FT944-18127 rutgers s008.t274 s008 ru 3 274 1402 9 FT921-1556 rutgers s008.t274 s008 ru 3 274 1402 10 FT942-15407 rutgers s008.t274 s008 ru 3 274 1402 11 FT923-11292 rutgers s008.t274 s008 ru 3 274 1402 12 FT931-12869 rutgers s008.t274 s008 ru 3 274 1402 13 FT944-4751 rutgers s008.t274 s008 ru 3 274 1402 14 FT921-15508 rutgers s008.t274 s008 ru 3 274 1402 15 FT943-11934 rutgers s008.t292 s008 zp 1 292 1296 1 FT944-6725 rutgers s008.t292 s008 zp 1 292 1296 2 FT932-5668 rutgers s008.t292 s008 zp 1 292 1296 3 FT934-10231 rutgers s008.t292 s008 zp 1 292 1296 4 FT932-11049 rutgers s008.t292 s008 zp 1 292 1296 5 FT934-10921 rutgers s008.t292 s008 zp 1 292 1296 6 FT922-4425 rutgers s009.t258 s009 zp 2 258 992 1 FT931-2005 rutgers s009.t258 s009 zp 2 258 992 2 FT931-7553 rutgers s009.t258 s009 zp 2 258 992 3 FT924-8561 rutgers s009.t274 s009 ok 3 274 1263 1 FT931-12869 rutgers s009.t274 s009 ok 3 274 1263 2 FT944-3280 rutgers s009.t274 s009 ok 3 274 1263 3 FT923-11292 rutgers s009.t274 s009 ok 3 274 1263 4 FT943-11934 rutgers s009.t274 s009 ok 3 274 1263 5 FT942-15408 rutgers s009.t274 s009 ok 3 274 1263 6 FT944-16970 rutgers s009.t274 s009 ok 3 274 1263 7 FT941-13218 rutgers s009.t274 s009 ok 3 274 1263 8 FT943-11627 rutgers s009.t274 s009 ok 3 274 1263 9 FT942-12506 rutgers s009.t274 s009 ok 3 274 1263 10 FT944-18127 rutgers s009.t274 s009 ok 3 274 1263 11 FT941-17640 rutgers s009.t292 s009 ru 1 292 1398 1 FT931-1043 rutgers s009.t292 s009 ru 1 292 1398 2 FT941-12426 rutgers s009.t292 s009 ru 1 292 1398 3 FT942-5651 rutgers s009.t292 s009 ru 1 292 1398 4 FT932-3822 rutgers s009.t292 s009 ru 1 292 1398 5 FT943-274 ----------------------------------------------- 8. Aspect Analysis by the NIST Assessors - - - - - - - - - - - - - - - - - - Topics as seen by the NIST assessors: ============ Number: 258i Topic: Computer Security Description: Identify instances of illegal entry into sensitive computer networks by non-authorized personnel. Narrative: Illegal entry into sensitive computer networks is a serious and potentially menacing problem. Both 'hackers' and foreign agents have been known to acquire unauthorized entry into various networks. Items relevant to this subject would include but not be limited to instances of illegally entering networks containing information of a sensitive nature to specific countries, such as defense or technology information, international banking, etc. Items of a personal nature (e.g. credit card fraud, changing of college test scores) should not be considered relevant. Aspects: each different case of illegal entry into a computer network. ============ Number: 274i Topic: Electric Automobiles Description: What are the latest developments in the production of electric automobiles? Narrative: The economic feasibility of electric automobiles appears to be limited by a number of factors, including the limited range of operation between recharges of batteries. What progress has been made in addressing these factors? Answers: each different recent development in the field of electric automobiles ============ Number: 292i Topic: Worldwide Welfare Description: Identify social programs for poor people in countries other than the U.S. Narrative: To be relevant a document would identify a welfare program in a foreign country and explain how it works to aid citizens who have little or no income. It would include those who can't work because of a disability and people who have the extra burden of small children. The document should indicate how these people are supported or not supported. A relevant document should identify the source of the monies used to support such welfare programs. Answers: each different social program in each different country. - - - - - - - - - - - - - - - Instructions to the assessors: You will be presented with a topic, which is the description of an information need. Stated or implied in the topic is a question with multiple aspects or answers. Here is an example topic: Common Treatments for High Blood Pressure in the U.S. and Western Europe <desc> Description: Identify the various common alternatives available for the treatment of hypertension in the U.S. and western Europe. <narr> Narrative: A relevant document will discuss one or more treatments commonly used for the treatment of high blood pressure in the U.S. or western Europe. Treatments might include drug therapy, life-style, etc. One or more specific treatments must be identified for a document to be relevant. Documents containing only generalities are not relevant. Neither are documents about treatments no longer in common use in the U.S. or western Europe. <aspects> Aspects: EACH DIFFERENT treatment of the sort described above You will also be given a set of documents - articles from the Financial Times of London 1991-1994. Your task will be to read through the documents looking for relevant ones which contain one or more answers to the question posed by the topic. You are to gradually compile a list of all the different answers contained in relevant documents and identify which answer appears in which document. On the following pages are materials for an example you should work through for practice. The example includes: - instructions - an example topic - 6 documents (to save time these are summarized in a sentence or 2) - a form on which to record the aspects - a form on which to record which aspects occur in which documents You can ask questions at any time. Instructions: Your task is to make a list of all the different aspects of the topic - answers to the question posed by the topic - which are contained in relevant documents. You should do the following: - read the topic carefully noting what sort of aspects you are to identify and what documents are considered relevant - then read each of the documents from the pool for the topic and gradually: - create a list of the aspects found somewhere in the documents - record a word or short phrase, describing each aspect found, on the "Aspects of Topic" form. - determine which documents contain which aspects - bracket each aspect in the text of the document in which it was found - on the "Document-Aspect-Matrix" form, for each document, circle the number of each aspect/answer the document covers/contains Feel free to read the documents as many times as needed and to revise your list of answers as well as the list of which answers are provided by which documents. Topic: Sample-Topic <title> Common Treatments for High Blood Pressure in the U.S. and Western Europe <desc> Description: Identify the various common alternatives available for the treatment of hypertension in the U.S. and western Europe. <narr> Narrative: A relevant document will discuss one or more treatments commonly used for the treatment of high blood pressure in the U.S. or western Europe. Treatments might include drug therapy, life-style, etc. One or more specific treatments must be identified for a document to be relevant. Documents containing only generalities are not relevant. Neither are documents about treatments no longer in common use in the U.S. or western Europe. <aspects> Aspects: EACH DIFFERENT treatment of the sort described above Sample pseudo-document (summaries): Doc-1: Talks about the increasing popularity of the mediterranean diet in the U.S., its recommendation by some physicians for the treatment of mild hypertension. It mentions the common use in Spain of a tea made from the snakeroot plant to lower blood pressure. Doc-2: Talks about how the ancient Roman physicians used leeches to lower blood pressure. Doc-3: Talks about how high blood pressure can lead to health problems and about how more people should get their pressure checked. Doc-4: Talks about use of diuretics and beta-blockers in Japan as cheap treatments for elevated blood pressure levels. It states biofeedback techniques have been found to be useful in managing hypertension in studies in California and is being used by physicians with increasing numbers of patients. Doc-5: Talks about the high stress levels among workers at blood collection agencies where there is increasing pressure on workers to do more on smaller budgets. Doc-6: Talks about Merck's sales of a drug Vasotec in the U.S., used to lower blood pressure. It mentions biofeedback as an available and popular treatment for high blood pressure in the U.K's government hospitals.