IN THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF COLUMBIA ELOUISE PEPION COBELL, et al., ) ) Plaintiffs, ) ) v. ) Case No. 1:96cv01285 ) (Judge Robertson) DIRK KEMPTHORNE, ) Secretary of the Interior, et al., ) ) ) Defendants. ) __________________________________________) DEFENDANTS’ NOTICE OF FILING OF EXPERT REPORT OF FRITZ SCHEUREN, PURSUANT TO RULE 26(a)(2) OF THE FEDERAL RULES OF CIVIL PROCEDURE Defendants hereby give notice of filing and attach hereto the Expert Report of Fritz Scheuren. Dated: August 24, 2007. Respectfully submitted, PETER D. KEISLER Assistant Attorney General MICHAEL F. HERTZ Deputy Assistant Attorney General J. CHRISTOPHER KOHN Director /s/ Robert E. Kirschman, Jr. ROBERT E. KIRSCHMAN, Jr. (D.C. Bar No. 406635) Deputy Director Commercial Litigation Branch Civil Division P.O. Box 875 Ben Franklin Station Washington, D.C. 20044-0875 Phone (202) 616-0328 Fax (202) 514-9163 CERTIFICATE OF SERVICE I hereby certify that, on August 24, 2007 the foregoing Defendants’ Notice of Filing of Expert Report of Fritz Scheuren, Pursuant to Rule 26(a)(2) of the Federal Rules of Civil Procedure was served by Electronic Case Filing, and on the following who is not registered for Electronic Case Filing, by facsimile: Earl Old Person (Pro se) Blackfeet Tribe P.O. Box 850 Browning, MT 59417 Fax (406) 338-7530 /s/ Kevin P. Kingston Kevin P. Kingston Expert Report Fritz Scheuren, PhD. VP Statistics and Methodology NORC University of Chicago In what follows I quote extensively from reports that the National Opinion Research Center (NORC) has already prepared for the Office of Historical Trust Accounting (OHTA). All of the 63 such reports delivered, so far, to OHTA were ones that I participated in: analyst, author, writer and reviewer. They are brought forward here to frame for the Court what I see as the main statistical issues (Section 1). The key statistical issue is first the role of statistical sampling (Section 2); a summary is then given of the Litigation Support Accounting Project, which was a turning point in the understanding of what was possible (Section 3). My expert opinion on what the new sampling design should be going forward is found in Section 4. A discussion of the connection between litigation risks and assurance levels comes next (Section 5). My expert opinions are summarized in Section 6. Appendix A provides selected standard statistics texts, and Appendix B provides a list of NORC reports prepared for OHTA. Of course, there are many more statistical challenges coming and some of the steps to come may not yield such favorable results. But, so far, accuracy rates are reasonably high. While NORC has much more to do to ascertain completeness of the account system – even for the Electronic Ledger Era – so far, though small issues have been encountered, no major problems have been detected. 1 Introduction NORC’s involvement on Indian Trust Historical Accounting started in August 2001, shortly after OHTA was founded, in July of that year.1 Hence this expert report covers almost all the historical accounting work done by that Office from its very beginning. Over that period there have been five major OHTA Report deliverables: • Blueprint for Developing the Comprehensive Historical Accounting Plan (“Blueprint”) • Report Identifying Preliminary Work for the Historical Accounting (“120 Day Report”) • Report to Congress on the Historical Accounting Individual Indian Money Accounts (“July 2, 2002 Report to Congress”) • Historical Accounting Plan for Individual Indian Money Accounts (“Purple Plan”) • Revised Historical Accounting Plan for Individual Indian Money Accounts (“Green Plan”) 1 Attachment 1 contains NORC’s proposal to OHTA to provide statistical consulting services for the Historical Accounting. NORC has contributed to all of these in one way or another. For example, NORC has provided population analyses, sample prototypes, research into new sample designs suitable for sampling the very complex and unique Individual Indian Money (IIM) transaction population, statistical support for OHTA’s quality control, and a series of reports describing actual samples drawn nationwide. From almost its inception, the OHTA has employed a mixture of approaches to accomplish its mission. One of NORC’s principal objectives has been statistically assessing the accuracy and completeness of transaction histories of IIM accountholders. In discharging this objective, statistical sampling has played a major role. In earlier reports by NORC, the role of statistics has been given prominence. To start off my report I will briefly describe how the results from a statistically drawn random sample can provide useful information about an entire population. Later sections will cover some of the sampling details and contrast results from the current project with earlier similar efforts. 2. Random Samples Statistical inference relies on the use of random sampling to select part of a population so that it is representative of the entire population. If the samples are large enough, this “representativeness” allows sound statistical inferences or valid conclusions to be drawn from the sample and applied to the entire population. Sampling is a commonly used tool in many facets of our lives. Statistical samples randomly selected to be representative of populations of interest save time and money. Examples include: • Medicine – Scientists test the efficacy of a medical treatment. For example, a clinical trial might use a sample of adults that is representative of the entire adult population. Results from that sample would then allow valid conclusions to be drawn about the efficacy of the treatment for all adults. • Manufacturing – Engineers test the quality of a factory’s daily production. For example, they might examine a sample of light bulbs that is representative of all the light bulbs produced in the factory that day. Results from the sample would allow valid conclusions to be drawn about any quality issues that may exist in the day’s production and what they are. • Records Auditing – Auditors test the accuracy of records. For audits of travel expense reports, the IRS might examine whether the entries in a representative sample2 of expense reports are supported by documentation (credit card receipts, hotel bills, etc.) that demonstrate the accuracy of the expense claims submitted. 2 Statistical sampling is routinely used in audit practice, as described in texts such as D.M. Guy’s Audit Sampling – An Introduction, 1994, John Wiley and Sons, New York. Mr. Guy was an auditing standards executive of the American Institute of Certified Public Accountants for many years. This last example most closely mirrors the way sampling is being used by OHTA. Each individual Indian’s Historical Statement of Account is a history of the opening balance and all the subsequent credit and debit transactions in that accounts that have been recorded in the manual and electronic systems – no sampling is involved in compiling this account history. Sampling is only used to determine the confidence one should have in the accuracy and completeness of the reported transaction histories. To this end, a representative sample of credit and debit transactions is selected and reconciled against source documents. If the IIM transaction and documentation match well (e.g., there are few differences and those that exist are minor), it can be confidently concluded that the recorded histories as a whole are supported by the underlying documents.3 One frequently asked question is how taking a sample that represents only a very tiny fraction of the population can make valid statements about a population with adequate precision and high confidence. To answer this question, I will present some basic concepts and how sampling works. 2,1 Sample Size, Confidence, and Precision In order to make inferences about an entire population based on the sample drawn from it, the sample must be sufficiently large for statements or conclusions to be made with high level of confidence. Probability statements are used to quantify confidence or assurance in the results. Take the well-known coin tossing example. Suppose we are testing whether or not a coin is “fair”, i.e. equally likely to land heads or tails. For illustration purpose, suppose further that we always observe 30% Heads (and 70% Tails). Our confidence level about the “fairness” of the coin can be stated as the probability of observing 30% or fewer heads if the coin were fair. A small probability indicates that the coin is not fair. The following illustrates how the confidence level changes as the sample size changes. Suppose we toss the coin 10 times and observe 3 heads and 7 tails. Using the binomial probability distribution, the probability of observing 3 or fewer heads in 10 independent tosses of a fair coin is approximately 0.17. That is, there is a 17% chance of seeing such a result, when the coin is fair. Most would not consider this outcome strong evidence that the coin was not fair. If we toss the coin 20 times, there is only a 6% chance of observing 6 (or 30%) or fewer heads if the coin was fair. The evidence is stronger that the coin is not likely to be fair. However, if we toss the coin 50 times, the chance of observing 15 heads or less drops to .3% and most people would say, with strong confidence, that the coin was not fair. If in 100 tosses, we observe 30 heads, the chance of such a result, given a fair coin, drops to less than .01%. Therefore, while increasing the sample from 20 to 50 provides a noticeable increase in our “confidence” or “assurance” that the coin is not fair, increasing 3 This has been OHTA’s experience so far, as detailed in the NORC reports that have been regularly released as the historical accounting work has proceeded. the sample size to 100 does not produce – for most practical purposes – significantly more confidence in the result. One other factor affecting the sample size is the desired precision around the estimates. Using the same coin tossing example, suppose we have an unfair coin (e.g., probability of head is not 50%) and we want to know the ‘degree of unfairness’ by tossing the coin n times. Intuitively, as the number of times the coin is tossed increases, the sample result gets more precise – gets closer to the ‘true’ degree of unfairness. In other words, if one was to repeat the same experiment over and over, the sample result will fluctuate less from one trial to another as n gets large. For example, if the ‘true’ probability of head is 60%, the observed percentage of heads will, 95% of the times, range from 40% to 75% for sample size of 20. The range will, however, narrow to 48% to 70% for sample size of 50 and 51% to 68% for 100. As with the confidence level, the precision improves greatly when the sample size is increased from 20 to 50 but not much from 50 to 100. In the OHTA setting, an assurance statement is desired from a sample about the accuracy of all transactions in the entire population. The goal is to make general statements about the percentage of transactions in error. For example, “the error rate of debit transactions is estimated to be 0.4%. With an assurance level of over 99%, we can state that, at most, 0.9% of the transactions may be in error.” In this example statement, the precision or margin of error associated with the estimated error rate of 0.4% is plus/minus 0.5% at a 99% confidence level, making the 99% upper bound of the error rate to be 0.9%. Similar assurance statement can be made on the total dollars in error – “we are 95% confident that the total dollars in error for all debit transactions is no more than $1 million.” 2.2 Population Size. While it is true that higher confidence and better precision requires larger samples, the size of the population is not a particularly important factor in determining sample size unless the population is very ‘small’ (i.e., in a few hundreds). Some quotes taken from standard sampling books are shown below on the effect of the population size on the sample size. “The number of sampling units in the frame that the sample was drawn from conveys no information in regard to the precision of the sample. The proper size of sample for a universe of 100,000 accounts might be identical with the size for a universe of 1,000,000 similar accounts.” W. Edwards Deming, Sample Design in Business Research, p. 29. “Often laymen are surprised to hear that precision depends only on the size of the sample and not on the population size. But population size affects only the factor (1 -n/N)4, and this can usually be ignored in designing the sample.” Leslie Kish, Survey Sampling, p 50. “A sample of 500 from a population of 200,000 gives almost as precise estimates of the population mean as a sample of 500 from a population of 10,000. Persons unfamiliar with sampling often find this result difficult to believe and, indeed, it is 4 The term, n/N, is the fraction of population sampled. remarkable. To them it seems intuitively obvious that if information has been obtained about only a very small fraction of the population, the sample mean cannot be accurate. It is instructive for the reader to consider why this point of view is erroneous.” William G. Cochran, Sampling Techniques, p. 24-25. "A common misconception is that a population ten times as large needs ten times as much sample, but the population size usually only plays a minor role. If the records are the same type and variability, a population of five thousand and a population of five million normally require about the same sample size to meet the same design specification." Wendy Rotz, Compliance Today, p. 2 The OHTA’s accuracy testing of transactions posted to Land-Based IIM accounts in the Litigation Support Accounting (LSA) Project is equivalent to another common example used in introductory probability courses – colored balls in an urn where two different colors can denote transactions with and without differences. Applying the hypergeometric distribution for 2 different confidence levels, the following table illustrates how the population size has virtually no impact on the sample size except when the population is very small. Table 1. Sample Size Needed for 1% Margin of Error Assuming No Error is Observed Population Size 95% Confidence 99% Confidence 500 224 297 1,000 257 364 5,000 289 434 10,000 293 444 50,000 297 452 100,000 298 453 1,000,000 298 453 10,000,000 298 453 Statistical inference is based on probability calculations such as above. The upper bound of errors and the assurance statements are determined by the final sample results (percentage of observed differences) and the sample size. For the precision of the assurance statements, the sample size is an important factor, but not as a percentage of the population size. This concept explains why statistical sampling is so frequently used in election polling (up to 150 million potential voters), military munitions (the tested bullets, for example, are destroyed in the firing test), consumer products (daily production, e.g., bags of peanuts exceeds 200,000 per 8 hour shift), etc. 2.3 Examples of Statistical Samples in the US Government. Most Federal Government surveys utilize statistical sampling where only a tiny fraction of the population is selected. The Survey of Consumer Finances performed every three years by NORC for the Federal Reserve Board is about the same size as the sample drawn for the LSA Project – approximately 5,000 representing the income and wealth characteristics of over 100 million US households consisting of 300 million individuals. Similar successful sampling activities are typical of many other US government agencies, including the US Census Bureau, the Departments of Agriculture, Education, Labor, and the Internal Revenue Service. 3 Litigation Support Accounting Results One of the biggest parts of NORC’s work for OHTA has been its involvement in the design, implementation and analysis of the LSA project. The LSA project for the Electronic Ledger Era (1985-2000) for the land-based IIM accounts, began in 2004, was continued during 2005, with the additional reconciliation of a random subsample of the originally selected transactions which had not been completed in 2004. Combined with the 2004 reconciliation results, over 99% of the randomly selected transactions used to make population estimates were reconciled for all 12 Bureau of Indian Affairs (BIA) Regions. This very high completion rate for searching and locating documentation and the reconciliations should put to rest concerns about the impact that the 1% remaining unreconciled transactions might have on the results. Reconciliation results show the debit difference rate to be 0.4%. NORC has concluded with an assurance level of 99% that the difference rate for debits is no more than 1.3%. NORC’s estimate for the rate of differences that are disadvantageous to the accountholders with an assurance level of 99% is 0.7%. This rate, at a 95% assurance level is 0.6%. With an assurance level of 99%, NORC estimates the dollar exposure for debit differences that disadvantage IIM accountholders to be no more than $4 million. At a 95% assurance level, the dollar exposure is slightly over $2 million. Reconciliation results show the credit difference rate to be 1.3%. With an assurance level of 99%, NORC states that the difference rate for credits is no more than 7.0%. NORC’s estimate for the underpayment rate with an assurance level of 99% is under 4.0%. This rate, at a 95% assurance level is 3.0%. With an assurance level of 99%, NORC estimates the dollar exposure for credit differences that disadvantage IIM accountholders to be no more than $86 million. At a 95% assurance level, the dollar exposure is $42 million. NORC found no evidence suggesting that under- and over-payments were statistically different. That is, the under- and over-payments occur at about the same rate and the distribution of the difference amount is statistically equivalent (whether the differences were under or over the recorded amount).5 The completion of the LSA project provided the following information: -Supporting documents can be found. 5 The LSA results and NORC’s conclusions are taken from [52]. -Identifying and locating all supporting documents is much more resource intensive than expected. -The attribute error rate is estimated to be very low overall – close to 0% for debit transactions and less than 1.5% for credit transactions. -The dollar error rate, i.e. the percentage of dollars in error, is estimated to be even lower. This confirmed the original design assumption. -There are many very small credit transactions that contribute minimally to the total throughput (e.g. 37% of all credit transactions in the population are less than $1 and account for only 0.1% of the total credit throughput). The primary concern here was that there could be very large relative errors, e.g. a $1 recorded transaction that should have been $100. Therefore the very small transactions were included in the sample design. No cases were found with large variances on a small transaction. At the time NORC finished its LSA report it offered for consideration several recommendations for how to proceed in piloting the sampling of the LSA accounts that had a so-called paper tail – that is, that went back before the Electronic Ledger Era. These recommendations after further consideration were not followed. Instead, it was decided to go with the Paper Ledger Era Design shown in Section 4 below. 4 Sample Design Planning Report 2007 (Green Plan) The application of statistical sampling procedures has not changed from the January 6, 2003 Historical Accounting Plan for Individual Indian Money Accounts (2003 Plan). In the historical accounting, sampling has been and will continue to be used to test the accuracy and completeness of the land-based IIM account statements. It is never used, nor was ever intended, to construct the actual account statements. The general approach has not changed either, namely samples are designed to provide an unbiased estimate of an overall error rate that is assumed to be small, but by using an adaptive approach, the sample procedures can deal with unexpected or systematic patterns of problems, if these are detected. The scope of the testing, however, has changed since 2003 because a lot was learned from the work that has been done during this time frame. 4.1 Electronic Ledger Era. The scope of the 2003 Plan was the transaction population of IIM land-based accounts where information was currently available, i.e. those transactions referred to as being in the Electronic Ledger Era. Due to Court decisions regarding the 2003 Plan, the sample design described there was not carried out. However, as part of the LSA Project, a smaller sample was used to obtain information on the likelihood and size of errors in this same population. The LSA sample design was applied to the same population of transactions as the proposed 2003 Plan but it differed from the 2003 Plan, as described below. -The LSA certainty stratum contained transactions of $100,000 or more (compared to $5,000 or more in the 2003 Plan). -A random sample of transactions under $100,000 was selected using a two-stage process: first, randomly selecting IIM accounts; and then, selecting transactions from within the selected accounts (compared to a direct random selection of transactions as had been proposed earlier). -Because of the first stage sample of accounts, it was not feasible to use fixed dollar strata for the selection of transactions. Some stratification by the size of the transaction was used within selected accounts, but the predominant stratification was based on type of account, type of transaction and time period. -The LSA sample was stratified by BIA Region, but not by Agency, and therefore the sample does not cover every Agency, as envisioned earlier. -The LSA sample was significantly smaller than the design proposed in the 2003 Plan. In 2003, given the possibility that the plaintiffs were right about government’s inability to find a significant fraction of IIM supporting records, NORC proposed an approach that would limit litigation risk by completely examining all transactions of $5,000 or more. This definition allowed for more transactions selected with certainty than would typically be used. The LSA Project, however, showed the contrary – the government could find the IIM supporting records; moreover, the percentage of dollars underpaid was less than 0.1% for the certainty stratum of transactions of $100,000 or more, so even a large sample of transactions $5,000 or more seemed unwarranted. The LSA Project also provided data on the time and cost of reconciling a transaction and these resources were much greater than what was projected for the 2003 Plan. Therefore, in combination the low dollar error rate and the high cost of reconciliation made reconciling additional transactions with certainty not only not necessary but even ill advised. For the transactions of less than $100,000 in the Electronic Ledger Era, the LSA results also provide assurance statements about the accuracy of the transactions. For debit transactions NORC provides with 99% assurance that the underpayment rate is less than 1% (i.e., less than 1% of the debit transactions have an error that is disadvantageous to the accountholder) and for credit transactions the underpayment rate is less than 4% (i.e. less than 4% of the credit transactions have an error that is advantageous to the accountholder). Selecting and reconciling additional transactions could lower these assurance bounds, say from 4% to 3%, but the cost of reconciliation is sufficiently high that the potential savings in a settlement from narrower assurance limits would most likely be less than the cost of reconciling the additional transactions. Based on the results of the LSA sample, I conclude that only relative modest follow-up work is required in order to make assurance statements about the error rates on transactions posted during the Electronic Ledger Era. There are two areas not yet covered: 1. Posting tests for the receipts in the Electronic Ledger Era – also referred to as completeness testing 2. Clean-up tests of transactions discovered later to be in the population of interest, but not included in the original sampling frame 4.2 Posting Test. The posting test checks the completeness of the account statements – whether there is any evidence that some expected collections were not made and/or the collected monies have not been posted to the ledger. The LSA Project has provided sound estimates of the error rate among posted transactions, but because the starting point for this accounting project was a recorded transaction, any failure to collect, deposit, and record collections would likely not have been discovered in LSA testing. Therefore OHTA plans to conduct additional tests where “revenue” is selected from a sampling frame independent of the electronic transaction ledger and traced to posting in the system to verify whether monies due have been deposited. The posting test will be conducted by BIA Region in order to provide summary information as the statements are sent. Statistical estimates will not be made until the entire sample, over all BIA Regions, is completed. This plan assumes that few random errors of this type were made. The sample will test this assumption, of course, and if errors found indicate systemic problems, adaptive procedures will be used to increase the sample where further testing is needed. 4.3 Clean-up Testing. This test is done on those transactions that should have been subject to sampling in the LSA Project but were not because they were not identified as such until the data validation work and the interest recalculations were completed. Ideally, the data validation and the interest recalculation would have been completed prior to the definition of the sampling population for the LSA Project. This was not feasible. There are relatively few transactions which were not included in the sampling frame, but because they had no chance of selection in the LSA Project, as a matter of due diligence, a small random sample should be selected and reconciled to test that the error rate for these transactions is the same as in the original LSA sampling population. 4.4 Sample Design for the Paper Ledger Era. For the Paper Ledger Era transactions the approach begins by testing the hypothesis that any differences found will be statistically similar to those in the earlier LSA Project. In other words, the difference rate will be statistically indistinguishable between the two Eras and that the average dollar differences will also be statistically indistinguishable. The LSA Project had to establish a baseline value for the difference rate and that required a very large sample. Having completed the LSA work, a simpler statistical question can now be addressed – how similar, then, are the two Eras? If the hypothesis can be accepted that the two transaction populations are similar, less work will need to be performed and the historical accounting can be completed more quickly. Should the results be different or inconclusive, then an expanded effort could still be performed. There are two categories of transactions included here: 1. transactions occurring in the IRMS system whose electronic record is not currently available, and 2. transactions occurring prior to the IRMS system. It will be necessary to obtain both the paper ledgers and the printed computer output in order to construct the information for the transactions in the Paper Ledger Era. However, a test of the accuracy of the transactions in the Paper Ledger Era can be done prior to the location, imaging, and digitizing of all of these transactions. 4.5 Proposed Paper Ledger Era Sampling Procedures. The proposed sample design would be a two-stage sample, similar to the LSA Project. The first stage sample would select a stratified random sample of accounts from a list of accounts in the population that may have transactions in the Paper Ledger Era. For the selected accounts, the paper ledgers and computer printout would be obtained and the Paper Ledger Era transactions would be digitized. The second stage sample would select a stratified random sample of transactions from the Paper Ledger Era transactions in the selected accounts. The steps necessary to select the sample are outlined below. 1. Define the sampling population of accounts with Paper Ledger Era transactions. From the data validation work OHTA has done, a list is available of accounts in the population that may have account activity in the Paper Ledger Era. These accounts can be stratified by BIA Region, by Agency, and by type of Paper Ledger Era records (i.e. IRMS or prior to IRMS or both).6 2. Select a stratified random sample of accounts with transactions in the Paper Ledger Era. 3. Locate the ledgers and computer printout for the selected accounts and construct the Paper Ledger Era transaction history for each selected account. 4. Develop the second stage sample design and select a stratified random sample from the transactions within the selected accounts. 6 By definition, there is no direct electronic information available on revenue activity during the Paper Ledger Era. It is not known how many years the account had activity, prior to the Electronic Ledger Era, nor what the number, size or type of Paper Ledger Era transactions may be in a given account. However, the revenue activity observed in the Electronic Ledger Era can be used to predict revenue activity for prior years, and in some cases, the birth year of the account holder is available to estimate the time period covered by the Paper Ledger Era transactions. Therefore the Paper Ledger Era accounts could be stratified by BIA Region; by Agency (if desired); and by the predicted time period and type of revenue activity. Separate sample designs will be used for credit (incoming money including receipts and transfers from another account) and debit (outgoing money including disbursements and transfers to another account) transactions. In addition, the second stage stratification will address the need to select a sample that covers the important factors such as time period and type and size of transaction. However, for the efficiency of the design, the selection will be such that the final selection probabilities are as nearly constant as possible, i.e. the sample will be approximately proportionately allocated. Certainty Stratum. By performing the test prior to the construction of all transactions in all accounts in the Paper Ledger Era, it is not possible to define a certainty stratum. It cannot be guaranteed, therefore, that all transactions of a certain size will be reconciled because such transactions cannot initially be identified. Having observed from the LSA Project, however, that the percentage of dollars underpaid was less than 0.1% for the certainty stratum of transactions of $100,000 or more, the exposure risk of not being able to reconcile all transactions above a certain size should be low – an aspect of the very hypothesis to be tested. Sample size. The accuracy of debit and credit transactions will be estimated and reported separately and, therefore, the samples designed separately for debit and credit transaction populations. In the LSA Project a 99% assurance level was used at the design stage. That was an high assurance level and a major factor in determining the large sample sizes in the 2003 Plan. Those large samples were considered essential to guard against a higher than the expected 1% error rate. As it turned out, the error rate for debit transactions was much less that 1%, while that for credit transactions was at about 1%. With these experiences in mind, smaller samples designed from the beginning made sense. Exactly how small the assurance level, and hence sample size, was still to be decided. In the end a 95% assurance level was recommended; remaining on the high side but quite common, unlike the earlier 99% rate, which is seldom used in business settings. Under the assumption that there will be moderate variation in the sample weights (but significantly less variation than in the LSA Project), the following tables show examples of the types of assurance statements that can be made using a sample size of 1,250 (Table 2) and a sample of size 750 (Table 3). For example, with a sample of size 750 in Table 2, if the estimated error rate is 2%, one has 90% assurance that the true error rate is less than 4.5%, and 95% assurance that the true error rate is less than 5.2%. Table 2. Upper bounds for three assurance levels with a sample of n=1250 Table 3. Upper bounds for three assurance levels with a sample of n=750 Estimated Assurance Level Error Rate 90% 95% 99% 0% 1.0% 1.3% 2.0% 2% 3.7% 4.2% 5.0% 4% 6.2% 6.7% 8.0% Estimated Assurance Level Error Rate 90% 95% 99% 0% 1.6% 2.0% 3.3% 2% 4.5% 5.2% 6.5% 4% 7.0% 7.8% 9.5% For initial design work, NORC recommends using sample sizes of the order of 750 for credit transactions and 750 for debit transactions. Since the analysis of the accounts in the Paper Ledger Era should provide information that may refine this recommendation, final transaction sample sizes will not be determined until the selected account histories are constructed. For credit transactions, a sample of 750 would yield an upper bound that is comparable to the LSA Project results, at the 99% assurance level, if the observed error rate is 2%, and at the 90% level if the observed error rate is 4%. This sample size should also allow a hypothesis test that the Paper Ledger Era transactions are no different than the Electronic Ledger Era transactions in accuracy with a significance level of .05 and 90% power to detect a difference of 5.5% or more. For debit transactions, this sample size would yield an upper bound that is comparable to the LSA results at the 90% assurance level, as the observed error rate is expected to, again, be close to 0%. However, I believe that the sample design for debit transactions will be more efficient than for credits; therefore the final assurance levels for this upper bound will be closer to 95%. In order to determine the number of accounts to select in the first stage sample, NORC will need to estimate the number of Paper Ledger Era transactions to expect per account. This can be estimated only after the analysis of the account information, which cannot be done until the ledgers are located and the transaction history is provided. Therefore, NORC proposes to divide the randomly selected sample of accounts into replicates. When the first replicate of accounts is constructed, NORC will have information on the number and type of transactions in the selected accounts. If the number of transactions in the Paper Ledger Era has been under-estimated, the second replicate may have to be included as well. This approach will allow NORC to increase, or decrease, the sample of accounts in a principled way, depending on the additional information. 4.6 Adaptive Sampling. The proposed sample is very small – allowing, on average, fewer than 70 debit transactions and 70 credit transactions per BIA Region. Therefore, an adaptive approach is needed so that the sample size can be increased if the results are very different than expected. The main goal of the LSA Project was to estimate the overall difference rate of all transactions recorded during the Electronic Ledger Era. The purpose of the Paper Ledger Era accuracy testing, however, is different. The purpose is to test the hypothesis that the difference rate of the Paper Ledger Era transactions is statistically similar or not worse than the Electronic Ledger Era transactions. An implication is that the sample size needed for hypothesis testing does not have to be as large as for estimation. If the tested hypothesis is rejected, then the data need to be analyzed to see if there are any discernable patterns that can be identified by location, time period, or type of transactions, and to pinpoint the source of the discrepancy. Additional samples would then be drawn from any identified cluster of transactions with potentially different error rates and, then, separate estimates for the Paper Ledger Era produced. In what ways do these differences in detail matter when doing hypothesis testing versus estimation, as in the LSA Project? Not oversampling the largest transactions should make very little difference in conducting hypothesis testing. A different summary statistics may have to be used when the hypothesis is tested, but this should present no extra issues. A 99% upper bound can always be calculated from the final sample results, if desired. What has been avoided is the cost in time and money of a larger sample size. Of course, if the sample results depart for what is expected, then the option of increasing the sample size remains. 5 Assurance Level Options (99%, 95%, 90%) To frame the issues here, let me state what I knew at the end of 2002 and then move on to what has been learned since, ending up with my opinion regarding an assurance level to propose going forward. 5.1 State of Knowledge of Litigation Risks, Circa 2003. OHTA was formed in mid 2001 and NORC was hired just over a month later. By the end of 2002, NORC had done a series of small-scale statistical studies of aspects of the Individual Indian Trust System, notably studies of the Land Title and Records Offices (LTROs) around the country. I was quite impressed with the quality of the record keeping found, albeit NORC’s results were limited by the fact that the original approach was hampered by the District Court’s shutdown of computerized aspects of the LTRO system. NORC had also begun a detailed statistical examination of IIM Alaska records, which was the first place the government proposed to undergo a full test for feasibility. At the time of the Plan’s submission, costs and timing information were unknown. NORC had suggested there be a pilot and Alaska was chosen by OHTA. Plaintiffs had asserted that a significant fraction of the needed IIM supporting records would not be found. The government and plaintiffs interpreted work done on the records of the five named plaintiffs in the Cobell case variously, with the court siding with the plaintiffs. NORC found the results of the historical accounting work for the named plaintiffs inconclusive and not necessarily applicable to a system-wide test. The preliminary pilot work done in Alaska suggested that with enough effort supporting records could virtually always be found. How typical Alaska was of the lower 48 remained unknown, however. What to recommend? Caution required that the Historical Accounting should allow for the possibility that the Plaintiffs might be right about the government’s ability to find the supporting records. NORC, thus, devised an adaptive strategy described in the Plan that called for an approach that permitted OHTA to modify its sampling as results came in. Results adverse to the government’s contention would lead to possibly even a bigger sample, while results supporting the government’s contention that errors were few and scattered could lead to a much reduced sample. In the January 6, 2003, Plan, given the possibility that government might not be able to find a significant fraction of IIM supporting records, NORC proposed an approach that would limit litigation risk by completely examining all transactions of $5,000 or more.7 Balancing this government risk NORC recommended a 99% assurance level. This high an assurance level would signal the government’s seriousness in meeting its fiduciary responsibilities under the Trust. The large sample sizes, on the other hand, would keep the precision of the Plan’s findings very high. That is, it would keep the gap between the actual finding of the historical accounting and the 99% upper bound small, if the government was right that, while errors had been made, a full accounting would show them to be on the order of 1% or less. 5.2 What has been Learned in the Four Years 2003-2006? NORC’s recommendation of a 99% level in the 2003 Plan was not controversial, even though it favored the plaintiffs on its face. Such an assurance level would have vindicated the government position, if the error rates were low. Its very thoroughness was impressive. Of course, the 2003 Plan was never implemented. Instead, the LSA Project was designed and implemented. NORC analyzed the results obtained and these results were in many respects as good as the government expected. The following key results from the LSA Project are important to consider in developing any further plans to test the transactions. • Virtually all the supporting records were found, making it possible, except in a trivial number of cases to make an accounting determination. • The overall error rates were low, as expected. • When there was an error, the amount of the error was usually a very small percentage of the transaction amount. • The time and resources required for the accounting was higher than expected – over $3,000 per transaction. Therefore, NORC and OHTA also learned in the LSA project that the sample sizes recommended by NORC in the Plan, given the costs of over $3,000 per transaction to do an accounting, were unbearably large, well above what was estimated in the Plan itself. 7 All of the NORC’s senior statisticians had worked at the IRS when in the federal government. When IRS examiners found the taxpayer to be untrustworthy all instances of unsupported transactions could be disallowed. Thus, if the accounting was not handled properly, the astronomical claims made by the plaintiffs, who at one point totaled well over $100 billion could be reached. The NORC design guarded against this, in case some significant fraction of the supporting records might not be findable. Moreover, to complete the Plan would protract the historical accounting process enormously.8 5.3 What I Recommend Going Forward. I do not recommend continuing with the Plan as is. Instead, it might make sense now to separate the assurance levels reported on from the assurance level designed for, allowing a much smaller sample to be used initially. • Sample Design: Assurance levels in conjunction with desired precision levels are used to estimate the required sample size. • Reporting the Findings: When the study is completed, assurance levels are used to make statements about the findings. When reporting the results, assurance levels of at least 95% are the standard and assurance levels of 99% may be required. However, at the design stage, it makes sense to allow a smaller sample size to be used provided that the error rates obtained continue low and OHTA’s ability to find records remains very high. Under these assumptions OHTA could escape the media/litigation hit that would be taken by lowering the assurance levels reported on (which could be kept at 95% and 99%), while designing at a much more affordable level (90%, say). Since OHTA has the option of increasing sample sizes later, if there is a surprise in the results, this could be a good compromise. NORC actually has already used this approach in that the GAO/Treasury project9 was designed at 90% but could have been reported at 95% or 99%, had there been a need to do so. I am less sure about reporting at 95% instead of 99%. The best practice may be to continue to make both available, as OHTA has now. But the more conventional 95% might be given greater emphasis in the future. 6 Summary of Expressed Opinions When developing the sampling plan for a historical accounting described in the January 6, 2003, Plan, a very large sample had been envisioned in the expectation that the error rate might be high and that it would be difficult, even impossible, to find a sizable percentage of the records needed to assess the accuracy of the accounting. The LSA Project, however, established that the accuracy was better than anticipated and that supporting records could be located. It did turn out, though, to be even more expensive than anticipated to find the records needed to reconcile the sampled transactions. But it could be done, with enough ingenuity and determination. 8 A note on this issue of Friday, January 26 (attached) goes into this in more detail. 9 See [59]. The idea of taking an adaptive approach was always central statistically, given what was known at the beginning, and spelled out in the 2003 Plan. Over the course of the LSA Project, since the results were so much better than expected and the work so much more expensive, the sample was reduced. While this approach was probably close to optimal for credits, the sample for debits may have been even larger than needed. • In my opinion, in light of the LSA baseline error rate established to date, OHTA’s new sampling plan submitted to the Court in May 2007 (to start with a smaller sample size and, only if needed, to adapt upwards) seems wisest since the unit cost of the accounting for the Paper Ledger Era is expected to be even higher than for the Electronic Ledger Era. • In my opinion, reconciling additional transactions is not needed for the Electronic Ledger Era since the dollar error rate found in LSA is low and the cost of reconciliation high. Selecting and reconciling additional transactions could lower assurance bounds, say from 4% to 3%, but the cost of reconciliation is sufficiently high that the potential reduction in dollars in error from lowering the upper assurance limit would most likely be less than the cost of reconciling the additional transactions. Therefore, only relatively modest follow-up work is required in order to make assurance statements about the error rates on transactions posted during the entire Electronic Ledger Era. • In my opinion, the more conventional assurance level of 95% that is given a greater emphasis in the 2007 Plan may be justified. At the design stage, especially, it makes sense to allow a smaller sample size to be used, provided that the error rates obtained continue to be low and OHTA’s ability to find records remains very high. Since OHTA has the option of increasing sample sizes later, if there is a surprise in the results, this could be a good compromise. I am less sure about reporting only at 95% instead of 99%. The best practice may be to continue to make both available, as OHTA has now. Fritz Scheuren, PhD Appendix A Some Standard Sampling and Statistics References Cochran, W. G. (1977). Sampling Techniques, 3rd ed. New York: John Wiley and Sons. Deming, W. E. (1950). Some Theory of Sampling. New York: John Wiley and Sons. Deming, W. E. (1960). Sample Design in Business Research. New York: John Wiley and Sons. Guy, D.M. (1994), Audit Sampling – An Introduction, New York: John Wiley and Sons. Hansen, M. H., Hurwitz, W. N., and Madow, W. G. (1953). Sample Survey Methods and Theory, Vols. I and II. New York: John Wiley and Sons. Hogg, R. V. and Craig, A. T. (1978). Introduction to Mathematical Statistics, 4th ed. New York: Macmillan Publishing Co., Inc. Kalton, G. (1983). Introduction to Survey Sampling, Series: Qualitative Application in the Social Sciences. Newbury Park: SAGE Publications Kish, L. (1965). Survey Sampling. New York: John Wiley and Sons. Lahr, S. L. (1999). Sampling: Design and Analysis. Pacific Grove, CA: Brooks/Cole Publishing Company. Rotz, W. and Dorn-Havlik, S (2001). Compliance Today. Health Care Compliance Association Thompson, S. K (2002). Sampling, 2nd ed. New York: John Wiley and Sons. Tucker, H. G. (1998). Mathematical Methods in Sample Surveys. River Edge, NJ: World scientific Publishing Co. Appendix B Listing of the NORC Deliverables to OHTA As noted in the main body of my report, NORC began its engagement with OHTA in August 2001, just over six years ago (See Appendix B for the actual technical NORC Proposal). All the expert reports since NORC was engaged are listed here to help document how I arrived at my current views and also how those views have changed as more and more data became available. Some of the early reports will be dealt with only briefly here and do not get mentioned at all in the main report. Many of the later reports, on the other hand, are quoted extensively in the main body of this report. Early Period In the beginning of the NORC engagement, I was asked to read the files in order to statistically frame the work to be done on the historical accounting (as called for in the contract). This “statistical thinking” step was taken and resulted in two early reports that looked at the system as a whole and tried to set out the statistical challenges expected. Rereading these reports now, with the hindsight that subsequent events permits, two major observations come readily to mind: I expected to have difficulty in locating all the records that would be needed. The order of the magnitude, in time and money, that the accounting search would require was not clear. I did not learn the scope of this search effort until we actually did the LSA project. There was some expectation that OHTA’s contractors would be unable to obtain all the records to do an accounting. This was wrong because virtually all were found. What was not envisioned was how resourceful our colleagues would turn out to be. They nearly always found the needed supporting document, whether that was a primary or secondary source. When reading OHTA’s files to get oriented I had a chance to examine the excellent Case Studies done by Historical Research Associates (HRA). This seemed an ideal place to begin getting a detailed knowledge of the specifics of the kinds of transactions for which the accounting was to be done. This third NORC report found that these studies were good for even more. While the recommendations made were never carried out, the Studies did find their way into the meta-analysis efforts done in an effort to integrate all that is known scientifically and to potentially use this body of knowledge to confirm or contradict the conventional wisdom and anecdotal evidence that has so characterized what is known or believed to be known. Getting Started Most of the early work NORC did was in response to the goals set out in the so-called “120 Day Report” Four (4) of NORC’s early reports, in part already mentioned, were aimed at familiarization with the data and administrative procedures on the financial and realty side of the Indian Trust [1 – 3, 5].1 In addition, there are a series of three (3) reports documenting the continuing evolution of NORC’s sampling plans for the historical accounting. These reports lead up to, but do not include the first full NORC design – the design implemented for Alaska [9, 11, and 17]. Number Early Planning Steps Date [1] Pre-Design Report on Sampling and Economic Applications Oct-01 [2] Design Report on Sampling and Economic Applications Nov-01 [3] An Analysis of the Application of HRA Case Studies to the Historical Accounting of Individual Indian Trust Moneys Apr-02 [5] Indian Trust Account Division (ITAD) Facility Lanham Maryland Jun-02 [9] Anadarko Agency IIM Transactions Described Dec-02 [11] NORC Sample Design Planning Report Feb-03 [17] Alaska Region IIM Receipt Transactions Described Jul-03 Land Title Studies Land Title and Cadastral Resurvey Project: NORC’s research into land title and probate documents held by the Bureau of Indian Affairs (BIA) is described in four (4) reports [4, 6, 12, and 13]. This land title work was also called for in the “120 Day Report” Related is a set of three (3) publications about the impact of boundary errors, as measured in Bureau of Land Management (BLM) cadastral surveys, on the historical accounting for Individual Indian Monies (IIM) that is to be done [14, 15, 16]. The first of these reports was written by the BLM at OHTA’s request and provides the basis for NORC’s work on the other two. Number Land Title and Ownership Reports Date [4] Land Title Pilot Procedural Documentation Mar-03 [6] Individual Land Title and Record Office Reports2 Mar-03 [12] NORC Analysis of LRIS Tract History Reports Feb-03 [13] Land Title Pilot Project Summary Mar-03 [14] Cadastral Resurvey Pilot – BLM Acreage and Location Results Jun-03 [15] Cadastral Resurvey Pilot: Statistical Analysis and Interpretation Jun-03 [16] Cadastral Resurvey Pilot - General Summary Jun-03 1 Report numbers came from the compendium reports. 2 Originally issued between April 2002 and May 2003 Quality Control Four (4) Quality Control reports were written as part of the statistical support of OHTA’s quality control efforts, carried out by Grant Thornton [8, 46-48]. Incidentally, NORC’s role in quality assurance has been largely in designing samples and assisting in the preparation of statistical summaries, on an as needed basis. Number Quality Control Reports Date [8] Sampling and Quality Control for IIM Account Statement Mailings Nov-02 [46] Quality Control Sampling Support for Grant Thornton Jul-04 [47] Mersenne Twister Technical Documentation Jul-04 [48] Calculating Confidence Intervals When the Non-Sampling Error is Measured by Statistical Quality Assurance Sep-04 Miscellaneous Under this heading are four (4) smaller reports that cover statistical issues of special interest [7, 10, 28, 50]. Report [7] on IIM Trust fund receipts provided an earlier look at the total. Number Specialized Statistical Reports Date [7] A Statistical Estimate of Receipts Credited to IIM Trust Funds Jul-02 [10] Procedures for Imaging and Coding Documents Jan-03 [28] Median Balancing [Sample Selection] Oct-03 [50] Compendium of NORC Reports, FY 2001 through FY 2004 Dec-04 Alaska Four (4) technical reports were written to describe detailed technical aspects of the originally proposed Alaska sample Design and Prototypes [18, 22, 23, 29]. There were also seven (7) technical reports that describe in detail the design and implementation of the first debit and credit samples for the Alaska region. These reports contain detailed descriptions of the sampling definitions and algorithms, and provide descriptions of the population and the actual sample drawn. Even though these samples were never completed due to legal developments in the Cobell class action case, this work still represents the first in-depth study of the transaction population, and the knowledge gained from these projects was an invaluable asset for much of OHTA’s later sampling work [19 – 21, 25-27, 43]. Number Alaska Sample Design and Prototype Date [18] Report on the Use of Adaptive Sampling in the Historical Trust Accounting of Individual Indian Money Transactions Aug-03 [19] Alaska Region Disbursement Prototype Sample – Design and Findings Sep-03 [20] Electronic Payment Prototype for the Alaska Region Sep-03 [21] Alaska Region – Identifying Related Payments over Time Sep-03 [22] Drawing the Credit Sample for Alaska Sep-03 [23] On the Assignment of the Alaska Receipt Sample To Individual Accounting Firms Sep-03 [25] Alaska Region Receipt Prototype Sample – Design and Findings Oct-03 [26] On the Relevance of Central Office Accessions In the Document Collection for the Alaska Pilot – Part I Oct-03 [27] On the Relevance of Central Office Accessions In the Document Collection for the Alaska Pilot – Part II Oct-03 [29] Drawing the Debit Sample for Alaska Dec-03 [43] Alaska Region Special Deposit Account (SDA) Debit Sample Design Report Apr-04 Litigation Support Accounting Litigation Support Accounting: A series of seventeen (17) reports about the design and analysis of eleven regional samples for OHTA’s Litigation Support Project. [30-41, 44, 45, 49]. Included here are also the two (2) reports about the Eastern Region Design, and the Eastern Region results [24, 42]. As the designs, except for Alaska and Eastern, were broadly similar, the work got progressively more efficient. Number Litigation Support Accounting Reports Date [24] Eastern Region Sample Design and Selection Sep-03 [30] Alaska Region Sample Design Report Feb-04 [31] Drawing the Account Sample for the Litigation Support Feb-04 [32] Western Region Sample Design Report Feb-04 [33] Pacific Region Sample Design Report Feb-04 [34] The Great Plains Region Sample Design Report Feb-04 [35] Southwest Region Sample Design Report Feb-04 [36] Southern Plains Region Sample Design Report Feb-04 [37] The Rocky Mountain Region Sample Design Report Feb-04 [38] Midwest Region Sample Design Report Feb-04 [39] Eastern Oklahoma Region Sample Design Report Feb-04 [40] Navajo Region Sample Design Report Mar-04 [41] Northwest Region Sample Design Report Mar-04 [42] A Statistical Evaluation of Preliminary Eastern Region Sample Results Mar-04 [44] Litigation Support Accounting Project Interim Report On Debit Transactions May-04 [45] Analysis of the Alaska Sample Jun-04 [49] Reconciliation of the High Dollar and National Sample Transactions Dec-04 Additional Reports There were 13 additional NORC reports in supporting OHTA’s development of the plan. The list is quite wide ranging in scope. Many of these were excerpted in the main part of my expert report Number Miscellaneous NORC Report deliverables Date [51] CP&R Subsample Report Sept 04 [52] Reconciliation of the High Dollar and National Sample Transactions from Land-Based IIM Accounts (All Regions) – LSA Project For the Electronic Records Era (1985-2000) Sept 05 [53] Qualitative Meta-Analysis of Audit and Reconciliation Studies of Indian Trust Accounts: Vol I, Summary Jun 06 [54] Qualitative Meta-Analysis of Audit and Reconciliation Studies of Indian Trust Accounts: Vol II, Bibliography and Index Jun 06 [55] Sampling IIM Transactions in the Paper Records Era Nov 06 [56] Reconciliation Results of the Remaining High Dollar Transactions for the Litigation Support Accounting Project Dec 06 [57] Litigation Support Accounting Project Results and the Sampling Plan Submitted to the U.S. District Court Jan 07 [58] Assurance Level Options (99%, 95%, and 90%) Jan 07 [59] Government Monitoring of the Indian Service Special Disbursing Agents’ Accounts: A Quality Assessment Based on the Settlement Packages, 1890-1950 Mar 07 [60] Land To Dollar Completeness test at the Horton Agency Mar-07 [61] The Role of Statistics and Statistical Sampling in the Individual Indian Monies (IIM) Historical Accounting May-07 [62] Sample Design Planning Report 2007 May-07 [63] Initial Hypothesis Testing Approach for the Paper Ledger Era May-07 All 63 of the preceding reports were transmitted to the Court as part of the Administrative Record. A detailed bibliography about most of them has been created. The work has grown in complexity as we got deeper and deeper into it. As you will see from our administrative record, NORC has done a great many specific tasks. Some turned out to be exploratory and some turned out to be central to what has been learned. Attachment 1 Expert Services Support for the Development and Implementation of Options for Conducting an Historical Accounting of Individual Indian Money Accounts Response to RFP 1435--01--01--RP--18159 1. INTRODUCTION We are pleased to respond to the request by the Office of Historical Trust Accounting (OHTA) for “Expert Services Support for the Development and Implementation of Options for Conducting an Historical Accounting of Individual Indian Money (IIM) Accounts.” As required, we give only a brief summary of our expertise and past performance on similar projects, plus, of course, our technical approach. Separately, a price proposal provides our rates and gives an overall cost estimate that is consistent with the total effort assumptions envisioned in the RFP. In preparing the response, we have carefully examined the background materials provided and also consulted other publicly available records on the operation of the Individual Indian Money Accounts. 2. EXPERTISE The team we offer you is composed of three individuals, a senior statistician and two journeymen—an economist/statistician and, as needed, also a mainframe computer programmer. While we are prepared to offer the three team members up to 100% of the time for the next three years, we have envisioned the senior statistician working at 80% level of effort overall and the other two team members at 60%, although this would undoubtedly be closer to 100% at the beginning. Dr. Fritz Scheuren, the senior member of our team, is a world famous statistical expert. He rose to be the Chief Mathematical Statistician at the Social Security Administration (SSA) and then for many years was the Director of the Statistics Division at the Internal Revenue Service (IRS). After leaving the federal government, he was a Professor of Statistics at The George Washington University, then a Principal at Ernst and Young, LLP, and most recently a Senior Fellow at the Urban Institute. Because of the page limitation on this response we include only a one-page version of his resume. Dr. Scheuren has been selected for the team because he has what we judge to be the right expertise in sampling hard to locate and hard to use operating records that may be spread across a host of media: from paper, to microfilm, to microfiche, to legacy mainframe computer systems and other more modern storage devices. His management and auditing experience should also prove invaluable to you and add greatly to the interdisciplinary team you correctly envision as needed to reach a fair and equitable resolution of IIM account issues. Dr. Kirk Wolter, our Senior Vice-President for Statistics, will provide firm oversight and review all deliverables. We are offering 10% of his time to support your requirements and to be sure that the full strength of NORC is brought to bear on your behalf. Again, for space reasons only, a one-page version of Dr. Wolter’s resume is included. We have yet to select the supporting members of the team but have several individuals at NORC that would fit your requirements. There is no doubt that NORC has the track record you need, being able to handle large projects of long duration that bring together the right people for the statistical and technical tasks at hand, however difficult and involved. 3. EXPERIENCE NORC is renowned for its sophisticated high-level scientific and technical performance. As a firm, NORC offers a wide range of experiences in collecting and using complex data that we can draw upon to meet your needs. Our general corporate capabilities are given as an attachment. Here we will concentrate on a few of the specific examples of the wide range of experiences that Dr. Scheuren could invoke to effectuate successful results in our combined efforts. These are his work on the National Bankruptcy Study, his statistical oversight of Ernst and Young’s work at the Swiss Bank (characterized by the media as the search for “Holocaust Gold”), and his representation of corporate clients in audit and inventory sampling cases before several state and federal regulatory bodies: . National Bankruptcy Study. This first-ever study was designed and carried out under Dr. Scheuren’s direction. It involved large and hard to collect samples of paper, microform and computer records obtained from every bankruptcy court across the country. The results were selected and processed to an exacting set of standards. The highest quality requirements had to be met as the issues were very controversial and subject to dispute. His role was, incidentally, not a policy one but entirely statistical. In fact his reputation for integrity and objectivity was one of the main reasons his representations before Congress were not disputed but accepted by both sides. The Study was one of the primary inputs that led to the recent passage of bankruptcy reform legislation. . Swiss Bank Study. Ordinarily, specific client engagements are confidential but the Swiss Bank Study is on the public record and, hence, can be discussed. Dr. Scheuren’s role was to provide statistical oversight on the work of compiling and cataloguing all relevant records material to the dispute. In this capacity he led a small team that visited the locations where records were stored and checked to see if all microfilm and microfiche documents relevant to the case had been found. Then he designed and oversaw the execution of a sample of the final records as these were turned over to the court for trial. Not only was his usual thoroughness employed in this case but, at various points, he introduced small adaptations in statistical theory that not only increased the assurance that could be given about accuracy but also that would consequently reduce costs. In this case the Swiss Bank eventually settled the case out of court, agreeing to a payment of about $2 billion to indemnify Holocaust survivors for the unclaimed wealth stored in its vaults during World War II. . Audit and Inventory Sampling Engagements. Dr. Scheuren has designed and managed multiple audit and sampling studies for clients. While no specifics can be given, these were often cases that brought him to represent a client before the IRS or some other regulatory body, such as the Federal Communication Commission or a State Public Service Commission. The way these projects worked was NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” that Dr. Scheuren was brought in by a corporate client and given full access to the client’s internal records (like in this case). A statistical approach would be agreed upon, often but not always involving sampling. He would then be asked, after the results were in and had been analyzed, to opine statistically on the valuation of inventories or the justification for a business expense or business practice. These efforts, done with a small interdisciplinary team of professionals (as is being bid on in this case) had elements in common with the work needed by the OHTA. Precision standards, sometimes exceedingly stringent, had to be met. The need to achieve frugal designs that minimized costs was paramount too; but not, of course, at the expense of accuracy. Many messy data handling problems, including missing data had to be faced (something expected when examining IIM accounts). Representation of the results in a contentious setting had to be anticipated too (another parallel with the IIM effort). Usually achieved, the goal was to employ a transparent approach that would minimize any concerns about the statistical results, however much the parties might disagree otherwise. In these three examples we see demonstrated success in meeting the stated needs outlined in the RFP in providing: . Expertise in the application of statistical methodologies to accounting issues (all three) . Expertise in the forensic use of statistics (the second and third example, but especially the Swiss bank work) . Analysis and documentation of statistical methods employed in the development of the accounting (all three) . Design and evaluation of statistical and economic methods, and the corollary means to address missing or incomplete information (again in all three, but especially in the third) . Review of historical accounting proposals and oversight of historical accounting implementation (his inventory sampling projects have this flavor, as does some of Dr. Scheuren’s SSA and IRS work, which has not been described.) . Ability to make inferences from statistical information (all three were characterized by proposing and implementing a sound comprehensive approach that then led to a sound statistical conclusion.) . Support to inform beneficiaries and others on the results and implications of statistical and economic analyses employed in the accounting (representation of the results in contentious settings was integral to the first and third, and, had it been needed, to the second.) No amount of experience, by Dr. Scheuren (or even NORC), can prepare us fully for what we are going to need to do, if engaged. The work you are seeking help on is far from routine. Dr. Scheuren has made a great number of theoretical contributions to the profession of statistics as a result of oftentimes having to construct “work-arounds” in situations like yours. His many NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” publications give ample support to his proven track record as an innovator when innovation is required. The difficulty of your problem, plus the high public service value that can be ostensibly achieved if we help you succeed, are frankly among the greatest attractions of this engagement. One last comment before describing the strategy we will employ in helping the OHTA develop and conduct a reconciliation of all IIM accounts: Despite the natural cautions voiced above (that come from often hard-earned experience), we also feel confident that bits and pieces from past solutions to related problems can be successfully spliced together with new elements that may have to be customized for your work. The net result can be predicted to lead to an affordable and successful effort. As the saying goes, “something old, something new, something borrowed, ....” 4. APPROACH The RFP asks us to provide statistical support services in phases. The first three phases (familiarization, plan development and testing) are overlapping and well developed for an activity of this type and scope. We will provide a combined approach to OHTA needs below, although each of the phases will be addressed individually. In phase IV our statistical work changes to mainly an oversight role. Another contractor is implementing the plans decided on in earlier phases. This is again appropriate and we provide our approach below. The RFP, however, also makes mention of the possibility that— “At the Department’s option, the statistical expert contractor selected may participate in conducting portions of the historical accounting.” While we want to be fully supportive of OHTA’s requirements, we do not think this option preserves the essential independence needed for us to act for you in providing a fully objective assessment of what gets learned in Phase IV and executed and internalized in Phase V. (This need for separation of consulting and oversight is why the big five accounting firms recently split off their business consulting practice from their audit and tax practices.) The final phase is to support the reporting that the OHTA has to do and to help with its settlement and resolution. This is the natural cumulation of our work for OHTA and we are sure, given our past work in contentious settings and what we will learn in the earlier phases, that we can again meet your needs. Our approach to Phase V concludes this section of our response. A brief summary ends our technical presentation in Section 5. 4.1 Familiarization (Phase I) During this first phase, we will, as required, become more familiar with the Historical Accounting Project, its needs, and the existing sources and types of data related to IIM accounts. The key familiarization objectives, as we see them, are to review, assess and assimilate the existing information and to summarize our understanding and test it against agency experts and other specialists carrying out other parts of the overall task. For Phase I to be successful we will need to complete four sub-tasks: . We will need to develop strong working arrangements with the other OHTA teams NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” and be sure coverage exists on all tasks and redundancy is avoided. This is not discussed directly in the RFP but may be the most important familiarization task, a sine quo non for the later work. . We will need to summarize in a “Pre-Design Report” what we acquired by reading reports and listening to experts, including the checking we did against the data themselves. This is amply covered in the RFP, with many sensible steps set out that do not deserve detailed comment. . The Pre-Design Report would be augmented, after being internally reviewed by the results of “benchmarking” against what other organizations have done in similar situations. The words in the RFP are “Compare this historical accounting project with other statistical historical accounting efforts that may bear on the type and feasibility of statistical methods Interior may use, and document these findings.” . There would be an examination of the suggestions already made on how to approach the historical IIM accounting. Again, the RFP words are “Obtain, examine, and develop summaries and analysis of all prior proposals and recommendations for the historical accounting of IIM accounts. The Pre-Design Report would be a Phase I deliverable and, after internal review, would be built upon in Phase II. The benchmarking steps and the review of the suggestions made by others (some quite interesting, albeit incomplete) would be chapters in the Statistical Design report - a deliverable under Phase II. The RFP, as we read it, envisions extensive meetings and considerable travel for meetings. What is not emphasized is the need for several first-hand examinations of the various storage media on-site in BIA offices and at two or more federal record centers. Routinely in past engagements what the Head Office believes is happening is at best incomplete and highly general. Nonetheless, whatever we learn from reports and Washington office staff and their consultants will almost certainly be insufficient for us to develop the sampling and other statistical procedures that we will employ in testing alternatives during Phase III. Early and regular access to the physical files and to those who manage them is essential. In the jargon of a sampler, we need to gain a hands-on familiarity with the “population” and with the ways it can be accessed, the nature of any lists of accounts (or “frames”) kept that could be sampled from and whether these (ideally computerized) frames are centralized or decentralized. How “searchable” are the files, how much activity do they have, how much leakage has there been, what is the quality of the items they contain, how good is the documentation, etc. These are all integral considerations to the successful completion of the task at-hand. Incidentally, no cost budgeting has been provided for the travel involved in this or later phases, since we do not know the locations and nature of the various record storage arrangements that we will be visiting. Presumably these trips, if concurred upon, would be arranged and paid directly by the OHTA. That is the assumption we are making at this point, unless informed otherwise. 4.2 Development of the Historical Accounting Plan (Phase II) This second or design phase begins almost at the inception of our work and will run through at NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” least Phase III, and arguably into Phase IV (should we learn in Phase IV that something thought to be feasible at an affordable cost was not). In our discussion of the work to be done under Phase I, there may have been an implication that we were preparing to design one or more samples in Phase II. We do not see our role as simply samplers, however. It is actually much broader and truly comprehensive. Sampling will almost certainly be one of the “statistical methods whose structure we define” in this phase; but there will undoubtedly be other methods needed too, especially given the incompleteness of the records available. Put another way, we see ourselves as identifying options for the statistical system you need and then working with the other OHTA teams to make recommendations on the options that deserve to be tested (then jointly statistically testing them in Phase III). The overarching objective of Phase II, as we see it, is to obtain a draft Historical Accounting Plan, the final version of which, according to the RFP, will have to go to Congress for approval. The word “draft” does not appear in the RFP and that is why it has been given emphasis here. All the sharing and coordination steps can take place just as set out in the RFP and we can support you fully on them but only if it is clear and with the stated caveat that the report remains a draft. Until it has been fully tested in Phase III we would not be comfortable in recommending that a full-stage implementation be undertaken. Cost estimates can be prepared and confidence intervals calculated but only under assumptions that we would not have been able to adequately test at that point. Some examples of the tasks we see ourselves performing, or participating with others in performing, in this phase include but are not limited to the development of: . Direct data collection designs, including those that employ sampling. For some account reconstructions, a geographically clustered design that combines elements of stratification and random selection as the last step may be tenable . Methods to process any new data directly collected and to limit data capture errors . Measurement designs that check at some level of assurance for data quality, both of newly acquired data and existing macro and micro information . Methods, such as micro models, to handle missing data, especially when there are gaps such that whole blocks of past transactions have been destroyed . Econometric macro models that attempt to build up from directly observed information to accounts for which none or very limited information is able to be gauged . Validation of prediction methods (where the prediction is done via imputation or an econometric model) will be key to integrating all known facts, especially Historical Trust Fund aggregates for which supporting details may be wholly or partially destroyed Typically in a statistical design activity as complex as you will require, it may be worth building a computer simulation model of how the pieces fit together. This would allow us to determine, NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” through sensitivity analyses, the places where small mistakes (or knowledge gaps) stay small and, hence, only modest efforts are needed. Conversely, it might also help us to more systematically identify the places where the statistical design does not “fail safe” and where, therefore, more of an effort (through additional direct data collection, for example) may be needed to reduce uncertainties. This computer modeling, if you wished, could also be an integrating device for the overall efforts of other teams and a way to consistently refine our collective ability to learn what options work and how to improve on them, as testing commences. Linking this modeling up with the resolution effort in Phase V might also prove worthwhile. 4.3 Pilot Testing (Phase III) Testing of supposed “facts” and design assumptions are basic at some level to all statistical undertakings and in this sense are part of all phases of the RFP. We have already touched on the modest amount of testing that would be nearly continuous in the first two phases. In this phase, Phase III, we give a central role to testing, so that the draft Historical Accounting Plan can be refined and readied for the Congressional approval process. Here the testing we perceive as needed would be of three types which might be termed individual component-by-component or simply “component” testing, joint component testing at the “join” points, and “ensemble” or complete systems testing. The component testing would separately check the assumptions made concerning separate pieces of the overall process, such as the sample selection, data entry, new accounting procedures, or the treatment of missing data. Once we were comfortable that the components of the plan worked separately and the linkages between them were adequate, a complete dress rehearsal would be launched and the entire assembled plan monitored. Below we go into a bit more detail on these distinctions and what their implications are for the successful completion of the whole project: “Component” Testing. As each of the teams comes up with its portion of the Historical Accounting Plan, the approach they recommend would need to be tested on live data, at a scale large enough to find major bugs should there be any. We are assuming that the component testing would be done by the team responsible for the component, with the statistical team providing support and an independent assessment. “Joint” Testing. In component testing, an examination of how each component fits with those that come before and after would be a secondary objective. Often good procedures are written and found workable for a component of a complex system but adequate planning at the join points is lacking. The individual component teams and the integration team would presumably be responsible for these “joints,” but the statistical team would help devise measurable ways of confirming the reliability of these parts of the overall system or plan. “Ensemble” or Complete Systems Testing. Again the integration team would be principally responsible here, working in concert with the statistical team. Operating properties and overall performance measures would be the focus in this ensemble testing, NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” where quantities like total time, total cost and final confidence intervals would be obtained. If the computer simulation model we recommend in Phase II gets built, then the ability to scale up these pilot test results to the full implementation envisioned for Phase IV would be enhanced. Notice that the testing of components can proceed independently to some extent and be done as the work on them is completed. However, if the components get done on different schedules, the testing of how the pieces work at the joints may require another separate examination. It may be necessary for the statistical team to construct test data or simulate the testing in some cases to check components or join points. This would be undesirable but may be inevitable. Clearly to carry out the roles set forth above, the statistical team would have to advise the Historical Accounting Project Manager and Departmental officials on the amount of testing needed and provide costs estimates, written proposals and justifications for all activities to be undertaken. The statistical team members would travel to on-site locations to assess first-hand the adequacy of the testing of the work of other contractors and the results being achieved. The “Component” and “Join” tests would be places where the Plan could be refined, as needed, and yet still go forward, as long as fixes could be found and confirmed. A failure in the “Ensemble” or full systems test would be much more serious and even if a fix were found, the statistical team might recommend a second complete test. In any event, the statistical team would take the lead in developing a testing report and would make recommendations on whether the testing supported going forward with the (revised) Historical Accounting Plan or whether further work was needed at that juncture. 4.4 Full Historical Accounting (Phase IV) During Phase IV, we are confident that we can successfully serve, along with other initial contractors, as a consultant to the Historical Accounting Project Manager and the Department to develop the historical accounting contract requirements and related statements of work, review other contractor proposals and products, and provide expert advice and recommendations on implementation of the historical accounting. Once Phase IV got underway we would play an ongoing monitoring role, including potential regular visits to the General Contractor’s work sites, plus the quality assessment of a sample of all intermediate work products. This role for the statistical team might be designed (and even tested, as well) during Phase III, so that it could be low in cost and effective in detecting problems should these arise during full implementation. Phase IV can fail in many ways. The RFP lays out those for which the statistical team would be responsible for guarding against. As we read the RFP, these are primarily (1) the detection of material flaws in the Plan not evident earlier in the testing phase (Phase III) and (2) execution problems by the General Contractor or subcontractor(s). As requested in other parts of this RFP, the statistical team would provide advice and make written recommendations on an ongoing basis to the Historical Accounting Project Manager and Departmental officials. This time (1) on the performance and results being achieved by the Historical Accounting Project General Contractor and subcontractors and (2) on any NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” modifications needed to the Historical Accounting Plan to ensure success. The statistical team, using its own quality assurance and on-site reviews, would comment and make written recommendations to the Historical Accounting Project Manager concerning the General Contractor’s periodic and final reports, and develop reports and analysis, as requested, based on the contractor/subcontractor deliverables. Statistical team members would also regularly participate in meetings with the Historical Accounting Project General Contractor as a Departmental expert and advisor to the Historical Accounting Project Manager and Departmental officials on how the results being obtained feed into the eventual resolution and settlement of IIM issues. 4.5 Reporting, Settlement, and Resolution (Phase V) As the RFP states, Phase V will likely begin before the completion of Phase IV. Depending on how large and realistic the ensemble testing is in Phase III, it might be possible to begin anticipating, even at that point, the final Historical Plan Accounting results and to explore how to report on them, develop settlement proposals and create resolution options that have various price tags. In any case, the statistical team, in consultation with the other teams that are part of this interdisciplinary effort, will continue to provide expert advice and recommendations to the Historical Accounting Project Manager and the Department on how to use and apply the information developed by the historical accounting to create starting balances for IIM accounts, resolve the Cobell v. Norton litigation through a decision of the Court or settlement, and develop any needed legislation. The actual tasks that we would be ready and able to perform include making oral and written recommendations to the Historical Accounting Project Manager and Departmental officials on how to apply the statistical results of the historical accounting to satisfying the Department’s historical accounting obligations and creating starting balances for all IIM accounts; how to explain, justify, and defend (forensically and otherwise) these results as satisfying the need for a full historical accounting; how to use the results obtained in settlement negotiations with plaintiffs (including developing written evaluations and recommendations for government and plaintiff’s settlement proposals), and, finally, if requested, helping in the development of statutory language to implement a settlement with plaintiffs or a legislative solution, including developing written evaluations of and recommendations for legislation. There are many conventional ways we could carry out these tasks, including providing a summary Recommendation Report that draws out the essential issues. To aid in this process and because the negotiations might be long and protracted, micro-simulation models might be built, like those now used in tax and welfare policy analyses. Dr. Scheuren has particular expertise here and could supply that skill in building such a model. 5. SUMMARY Throughout the period of performance, we will provide advice, recommendations, and forensic statistical analyses on such topics as data assessment, sampling, feasibility of methods, and the inferences that can be drawn from the accounting done in the sampled cases when applied to NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” other IIM accounts. At each phase in our support of the OHTA, we plan to provide a deliverable that will document the statistical process and help improve its transparency: . In Phase I we will provide, as a deliverable, a Pre-Design Report that summarizes what is known statistically about the problem. . During Phase II, we will develop a Statistical and Economic Design Report that explores econometric, sampling, and other statistical options. This would be an input to the overall Historical Accounting Plan being prepared jointly. . A Testing Report will be a deliverable in Phase III that describes how we refined and improved upon the Historical Accounting Plan contributed to by NORC and other contractors in Phase II. . During Phase IV, a Monitoring Report will be prepared that objectively assesses the implementation of the full historical accounting work done by the General Contractor eventually selected. . Finally, if requested, as part of our support of Phase V, we will develop a Recommendation Report that addresses settlement and resolution issues insofar as these are statistical or economic. We have noted elsewhere in this response that a micro-simulation policy modeling approach could be a natural follow-on step to Phase V. We would be prepared to assist in its implementation, as part of the services offered here. However, to build such a model would go beyond the scope envisioned by the RFP, although we would be happy to supply this expertise separately if called upon to do so. We have used the original list of contractor responsibilities in the RFP as a checklist to see that we have been fully compliant. We believe we have. Rather than repeat every instance where we have described how we would meet OHTA needs, we will here mention just a few of the major ways. The RFP language comes first, followed by one or more examples of how we will comply: . “Review and assess existing information on IIM accounts and records, and develop documentation and reports.” The Phase I Pre-Design Report would be one of the deliverables here. . “Review and assess related work products of Federal Government agencies, private firms, consultants, and others, and develop documentation and reports.” The Design Report in Phase II is an example of how we intend to meet this responsibility. The Monitoring Report and related support activities in Phase IV are other examples of the same. . “Review and examine the existing IIM accounts at macro and micro levels to determine the potential utility of statistical and economic methods to advance the historical accounting, and develop documentation and reports.” This responsibility is partially met in Phase II but aspects of Phase III develop it further, including the Testing Report. NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” . “Participate in discussions with key OHTA, Departmental, and other contractor personnel on historical accounting, and related decisions, policies, procedures, statements of work, other contractor proposals, and provide expert advice and recommendations as appropriate.” This role is thoroughly woven into all phases of the work we would do under this RFP, if we are selected. . “Contribute statistical expertise to formulate historical accounting plans, methodologies, procedures, justifications, uses, applications, and interpretations, and prepare recommendations and supporting documentation and reports.” Again, this is interlaced into virtually all aspects of our response. Given our expertise, this is yet another place where NORC will exceed RFP requirements. . “Review and prepare responses to questions and comments from all sources including Departmental leadership, the Administration, Congress, the Court, and others.” Our objectivity and reputation for fairness and thoroughness should be invaluable to the OHTA in addressing questions and concerns from every quarter. . “Assist and support the Department and other Federal Government agencies in meetings with other agencies, plaintiffs’ representatives, Congress, the Court, and others.” The Recommendation Report which may be produced as part of Phase V would be an example here of what might be responsive and appropriate. In summary, NORC and its statistical team stands ready to ably support the OHTA as it develops and implements an Historical Accounting Plan that leads to a resolution of outstanding IIM issues. We have the demonstrated expertise, ample experience, and a firm understanding of the requirements at every particular phase of the work. NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” CORPORATE CAPABILITIES NORC is a full-spectrum research organization with 60 years experience in conducting efficient and technologically advanced data handling and data analysis. NORC prides itself on doing high-quality work and is one of only a few firms in the United States that undertakes a significant volume of large-scale, complex, national, longitudinal studies where individuals are tracked over many years. This longitudinal analysis experience, as a firm, will be one of the strengths NORC brings to reconstructing a Historical Accounting of Individual Indian Money (IIM) records. NORC research capabilities are organized into five departments: Economics and Population; Education and Child Development; Health Studies; Statistics and Methodology; and Substance Abuse, Mental Health and Criminal Justice. The Directors of these departments report directly to Dr. Craig Coelen, NORC President. Each department has its own staff of experienced and dedicated researchers who perform administrative, data collection and data analysis work needed for completion of important research. Data Collection Capabilities. NORC employs an experienced and extensive national field staff, including the range of skills needed to think through how to capture the data that might be required to resolve IIM issues. The field structure divides the country into four large districts and further subdivides it into thirteen divisions. Twelve Regional Managers, each of whom have been supervisors at NORC for a minimum of eight years and many for more than 20 years, provide oversight management in their respective divisions and a staff of 50 front-line Field Managers supervise the interviewers. The approximately 500 interviewers are all hired, trained, and evaluated by Field Managers or their supervisors. Computing Capabilities. NORC Computing Applications Development staff includes programmers, systems analysts, and data processing project managers. The staff has the full range of mainframe and other programming skills to tackle the expected wide range of legacy systems that the Historical Accounting project will require. Recognizing the critical role of information technology, NORC has been a leader in developing, testing, and implementing state­of-the-art computer-assisted personal interviewing (CAPI) capabilities, with interviewers using laptop computers to serve the needs of large-scale studies requiring face-to-face interviewing. Project Management. NORC designs project management plans tailored for the specific needs of the agency and the study. Typically, such plans designate a senior NORC researcher or survey specialist as Project Director. For the Historical Accounting Project we are taking the highly unusual step of bringing in one of the firm’s Senior Vice-Presidents. Project Directors are supported by other staff with expertise in particular aspects of data collection relevant to a specific project as needed. Also supporting the Project Director as necessary are individual task leaders who are experts in particular aspects of any work relevant to a project. Financial Management. In addition to project management, NORC also provides extensive financial management for projects. NORC has established an accounting system that is in full compliance with federal regulations. Our system has been approved by the U.S. Department of Health and Human Services and meets government standards for accumulating and reporting project-related cost. NORC’s record in successfully fielding large-scale, complex studies has depended in part on its commitment to working closely and responsively with clients. We design NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” communication plans to satisfactorily meet any client’s needs for timely and appropriate contact during all phases and over the entire duration of a project. The communications plan is designed to keep our clients updated on project status, to assure rapid and fully documented communication on critical project decisions, and to assure project responsiveness to the agency’s contract requirements and technical direction. Technical Infrastructure. In Chicago, NORC maintains four facilities interconnected by T1 telecommunications trunk lines for voice and data transmission. These include our corporate headquarters on the campus of the University of Chicago, offices and telephone interviewing and data preparation operations at two locations in Chicago’s Loop, and a suburban telephone interviewing center. The Loop, U of C campus, and Washington offices have Polycom videoconference installations served by 384Kb lines. All NORC office staff has high- performance, multi-media desktop PCs that use Pentium III or higher level microprocessors and the Microsoft Windows NT operating system. These are connected to NORC’s T1 line which provides instant access to NORC’s Intranet, Internet and World Wide Web gateways. NORC uses a distributed processing model. Our Wide Area Network (WAN) currently offers its users 0.75 terabytes of online disk storage, located on file servers, application servers, and UNIX hosts. An additional 325 gigabytes is available from a 500-CD/ROM data server. Ancillary resources located on the WAN include nine-track tape drives, CD-ROM readers, CD-ROM write capability, database servers, and high-volume laser printers. Our UNIX production server, located in the Chicago downtown facility, features four RISC 7200 processors with 2 gigabytes RAM. Disk capacity is HADA, dual-channel RAID-5 configuration with continuous, uninterruptible power supply (UPS), resulting in zero down time for disk accessibility. Systems Security. NORC ensures data integrity through a combination of system access restrictions, screen update rules, and system maintenance and backup procedures. The goal is to protect against unauthorized system access, mistakes in case information entry, and data loss. These procedures are consistent with the guidance provided in the National Institute of Standards and Technology Federal Information Processing Standard Publication (FIPSPUB) 41, Computer Security Guidelines for Implementing the Privacy Act of 1974. Respondent Confidentiality. NORC has meticulously safeguarded both client and our individual case confidentiality for 60 years, and we are widely respected for our responsible behavior toward research participants. Protection of respondent confidentiality actually encompasses three basic issues: the systematic treatment of individual respondents, computer hardware and software features that protect the data collected, and post-processing procedures implemented to minimize the possibility of deductive disclosure. NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” KIRK M. WOLTER, PH.D. EDUCATION Ph.D. Iowa State University, Statistics, 1974 M.A. Iowa State University, Statistics, 1972 B.A. St. Olaf College, Mathematics and Economics, cum laude, 1970 PROFESSIONAL EXPERIENCE Recent Positions 1994-present Senior Vice President, Statistics, NORC at the University of Chicago 1994-present Professor, Department of Statistics, University of Chicago 1991-1994 Vice President, Statistical Design Worldwide, ACNielsen 1988-1991 Vice President, Statistical Design, ACNielsen U.S. Bureau of the Census, 1974-1988 Chief, Statistical Research Division Senior Mathematical Statistician Assistant Division Chief, Statistical Research Div. American Statistical Association, 1972-present Chair, Survey Research Methods Section Chair, Committee on Federal Statistical Appointments Member, Publications Committee Coordinator, ASA/Census Fellowship Program Chair, Committee on Short Courses, Survey Research Methods Section Program Chair, Business and Economic Statistics Methods Section International Association of Survey Statisticians (IASS), 1985-2000 President, President Elect, Vice President Scientific Secretary, Council Member Program Chair, IASS, Cairo, Egypt George Washington University, 1975-1988 Adjunct Professor, Professorial Lecturer, Associate Professorial Lecturer Selected Panel Memberships Member, Federal Economic Statistics Advisory Committee (2000) Member, Panel on Cost of Living Indexes, National Academy of Sciences (1999) Member, Bureau of Labor Statistics Advisory Panel on the Current Employment Statistics Program (1993) Member, Special Advisory Panel, appointed by the Secretary of Commerce, regarding completeness of the 1990 Census (1989-1991) SELECTED PUBLICATIONS Moore, W., S. Pedlow, P. Krishnamurty, & K. Wolter. 2000. National Longitudinal Survey of Youth 1997 (NLSY97): Technical Sampling Report. Chicago: NORC. Harter, R., K.M. Wolter, & M. Macaluso. 1999. Small domain estimation of employment using CES and ES202 data. Paper presented at the Federal Committee on Statistical Methodology Research Conference, Rosalyn, VA. Huff, L., J. Shao, & K.M. Wolter. 1998. Variance estimation for the Current Employment Statistics Program. In Proceedings of the Survey Research Methods Section, 1998, American Statistical Association. Wolter, K.M. 1998. Introduction to Variance Estimation, Chinese Edition. New York: Springer-Verlag. Wolter, K.M. 1996. Statistics under the spotlight. Improving the Consumer Price Index: Statement. In Proceedings of the Section on Government Statistics, American Statistical Association. FRITZ J. SCHEUREN, Ph.D. NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” Résumé EMPLOYMENT Senior Fellow, The Urban Institute, 1999 to present. National Technical Director and Principal, Statistical Sampling Economics Group, Ernst and Young, LLP, 1996 to 1999. Professor of Statistics, The George Washington University, 1994 to 1996. Director, Statistics of Income Division, U.S. Internal Revenue Service, 1980 to 1994. Chief Mathematical Statistician, Social Security Administration, 1978 to 1980. RECENT PROFESSIONAL SERVICE ACTIVITIES Vice-president American Statistical Association, 1999 to present. Scientific Secretary, International Association of Survey Statisticians, 1997. National Academy of Sciences, Applied and Theoretical Statistics, 1994 to 1997. President, Washington Statistical Society, 1991 to 1992. Associate Editor, Journal of the American Statistical Association, 1989 to 1996; Survey Methodology, 1986 to present; Journal of Business and Economic Statistics, 1983 to 1989. Adjunct Professor of Statistics, The George Washington University, 1997 to present. Advisory Board Member, George Mason University Statistics Department, 1999 to present; USDA Graduate School Statistics Advisory Board, 1989 to present. EDUCATION Ph.D. (Statistics) The George Washington University M.A. (Statistics) The George Washington University B.A. (English Literature) Tufts University HONORS ASA Founders Award (1998); Shiskin Award for contributions to U.S. economic statistics (1995); Finalist, Senior Executive Association Executive Excellence Award (1992); Elected Member, the International Statistical Institute (1988); Elected Fellow, the American Association for the Advancement of Science (1984); Elected Fellow, the American Statistical Association (1981). PUBLICATIONS Over one hundred and eighty applied and theoretical papers, monographs, and books focused on the sampling of operating records, process quality, auditing, and the handling of missing data. NORC “Use or disclosure of data on this sheet is subject to the restriction on the title page of this proposal or quotation.” Attachment 2 August 24, 2007 Statement of Fritz Scheuren, PhD Re: Cobell v. Kempthorne I submit the following statement regarding my compensation in connection with service as an expert in this matter: I am employed through NORC, which is under contract with the Department of the Interior. NORC receives in compensation for my work the hourly rate of $226.48. Neither NORC nor I am being separately compensated for this report or for any testimony I may give. Fritz Scheuren, PhD Attachment 3 FRITZ J. SCHEUREN, PH.D. EDUCATION Ph.D. (Mathematical Statistics) The George Washington University (1972) M.A. (Statistics) The George Washington University (1970) B.A. (English Literature) Tufts University (1963) PROFESSIONAL EXPERIENCE 2001 to Present Vice President, Statistics, National Opinion Research Center (NORC) at the University of Chicago Dr. Scheuren joined NORC as Vice President for Statistics and Methodology to work on a major effort just getting underway in late 2001 to help the Department of Interior with its handling of Individual Indian Money (IIM) Trust fund records. This effort has already yielded hundreds of NORC products, including nearly 60 major research reports, most of which have been compiled into compendia. He is an overall editor in this series and a principal author. This role allows him to employ his considerable sampling and audit expertise in a setting where, as at the Internal Revenue Service (IRS) and Social Security Administration (SSA), there are extensive government records from which to draw. Following the 2000 Presidential Election NORC became increasingly involved in US Election measurement issues. In fact, since 2004 Dr. Scheuren has become very active on voting. For example, he led a pro bono exit polling activity in Albuquerque NM in 2004 and in Columbus OH in 2006, where he was the NORC Co-Project Director. The 2006 experience arose because NORC, as a public service, sponsored a telephone survey of eligible voters in two Ohio counties to determine whether they experienced any problems with the voting process. NORC also conducted a follow-up opinion poll regarding the late decision on a close House race. As an expert private citizen, Dr. Scheuren has submitted pro bono testimony before both the New Jersey Legislature and later the US Senate on how to statistically audit voting in 2008. Recently Dr. Scheuren has been heavily involved in overseas consulting to support US efforts to address our country’s response to the Millennium Development Goals. Outside of NORC, Dr. Scheuren also advises on HIPAA privacy protection matters, as he remains heavily involved in privacy and confidentiality issues beginning with his days at IRS and SSA. Occasionally he also engages on minor one-time statistical business consulting efforts. 1999 to 2001 Senior Fellow, The Urban Institute Dr. Scheuren was in overall charge of the Urban Institute’s National Survey of America’s Families (NSAF), part of the effort made to measure the impact of US welfare reform. In addition to his managerial duties, he was the editor and a principal author in the 1997 and 1999 NSAF Methodology Series (33 volumes). That survey was a major part of the Urban Institute’s Assessing the New Federalism project, with nearly 300 publications on welfare reform and related issues. At the time his distribution of NSAF data and metadata won praise as an example of best practice among web-distributed statistical datasets. 1996 to 1999 National Technical Director and Principal, Statistical Sampling & Economics Group, Ernst and Young, LLP (E&Y) Dr. Scheuren was a Principal at E&Y, working in the National Tax Division where his practice covered many federal as well as state taxation issues, including sales and use taxes. He had considerable representational experience before regulatory and legislative bodies, including the US Congress. His sample designs built extensively on existing client operating records and hence were very informative and economical. He led a team that critically examined an audit of the major telecoms by the Federal Communications Commission and was able to achieve a useful compromise for all concerned. 1994 to 1996 Visiting Professor of Statistics, The George Washington University (GWU) Professor Scheuren taught the entire sampling sequence while a visiting professor at The George Washington University, plus many service courses. While there, he set up a graduate certificate program in survey research, on which he still advises. His extensive consulting eventually drew him away from full-time teaching. His consulting on tax issues, both audit and information issues, was what brought him to Ernst and Young. 1980 to 1994 Director, Statistics of Income Division, Internal Revenue Service (IRS) As the long time Director of the Statistics of Income Division, Dr. Scheuren, using the quality improvement ideas of Deming, completely transformed the organization. Its statistical activities, computer hardware and software, and customer focus were all modernized. His efforts mainly dealt with issues of national importance to the economy and he was recognized for this with the prestigious Shiskin Award in Economic Statistics (1995). 1973 to 1980 Chief Mathematical Statistician, Social Security Administration (SSA) Dr. Scheuren, as the chief spokesperson at Social Security for statistical methodology, made major advances in the formulation and especially the delivery of statistical data to outside researchers and the public at large. His work included, among other matters, assessing the results of large-scale administrative and survey studies and occasionally representation of the agency before Congress. He sponsored and participated in seminal work on the handling of missing data, including a role in the creation of “Multiple Imputation” (as described in his November 2005 paper in the American Statistician.) 1968 to 1973 Mathematical Statistician, Office of Economic Opportunity (OEO) During this part of his career, Dr. Scheuren was completing his doctoral studies by going to night school part-time at the George Washington University. Even so, at OEO he was, with George Sadowsky, the chief architect and principal developer of what was then a state-of-the-art metadata system for the 1966 and 1967 Surveys of Economic Opportunity. This work even played a part in his keynote talk on Paradata at the Statistics Canada Methodology Conference in October 2005. 1963 to 1968 Statistician, Internal Revenue Service This was Dr. Scheuren’s first professional job after graduation from Tufts University in English Literature. He started at IRS as a management intern and was sent to night school at GWU to become a statistician. Even though just beginning his career he ran several statistical studies at IRS. These were to lead to his earliest publications on income and wealth, including groundbreaking statistical work on the Estate Multiplier Method of wealth estimation. RECENT PROFESSIONAL SERVICE ACTIVITIES Human rights projects include US and overseas work done involving Armenia, East Timor, Guatemala, Peru, Republic of Georgia, Russia, South Africa, Vanuatu, plus Afghanistan refugees in Pakistan and Kosovar refugees in Albania. Pro Bono work by him has also been taking place involving Iraq. He has recently been training Iraqi pollsters in Jordan and Turkey. Work in Darfur is planned but progress has been slow. NORC AT THE UNIVERSITY OF CHICAGO President, American Statistical Association, 2005 Member Board of Scientific Councilors, National Center for Health Statistics, 2003 to 2004. Member Advisory Board for Evaluation of AmeriCorp Program, 2003 to present Member, Committee on Scientific Freedom and Responsibility, American Association for the Advancement of Science, 2002 to 2007 Vice-President American Statistical Association, 1999 to 2001 Scientific Secretary, International Association of Survey Statisticians, 1997 National Academy of Sciences, Committee on Applied and Theoretical Statistics, 1994 to 1997 President, Washington Statistical Society, 1991 to 1992 Associate Editor, Journal of the American Statistical Association, 1989 to 1996 Associate Editor, Survey Methodology, 1986 to present Associate Editor, Journal of Business and Economic Statistics, 1983 to 1989 Associate Editor, The American Statistician, 2003 to 2006 Adjunct Professor of Statistics, The George Washington University, 1985 to present USDA Graduate School Statistics Advisory Board, 1989 to present Advisory Board Member, George Mason University Statistics Department, 1999 to 2006 HONORS Distinguished George Washington University Alumni Achievement Award (2006) American Immigration Lawyers Association Human Rights Award (2005) Harry V. Roberts Statistical Advocate Award (2004) Chartered Statistician, Royal Statistical Society (2003) American Statistical Association Founders Award (1998) Julius Shiskin Award for contributions to U.S. economic statistics (1995) Finalist, Senior Executive Association Executive Excellence Award (1992) Elected Member, the International Statistical Institute (1988) Fellow, the American Association for the Advancement of Science (1984) Fellow, the American Statistical Association (1981) PUBLICATIONS Over 251 applied and theoretical papers, monographs, and books focused on the sampling of operating records, survey design, process quality, auditing, and the handling of missing data. Dr. Scheuren also submitted many reports orally and in writing before regulatory and judicial bodies, here in the United States and internationally. Full citations of Dr. Scheuren’s publication record are provided separately. An attempt has been made to be complete, especially in the last ten years. Some presentations that were not submitted later to proceedings may have been omitted. NORC AT THE UNIVERSITY OF CHICAGO Attachment 4 Fritz Scheuren, Ph.D. List of Publications and Technical Papers 1995-2007 Provided here is a “due diligence” list of my recent statistical publications and technical papers. There was no attempt to catalogue the talks I regularly give using PowerPoint and that are made from informal notes that never get into a final written form. For example, I have given at least 5 such talks in July and August, almost all at the Salt Lake City Joint Statistical Meetings (JSM), just concluded. At best one of these will appear in the JSM Proceedings. That paper will involve NORC client work with the State Department, Millennium Challenge Corporation, in Georgia (in the former Soviet Union, where I will be going again in September). The material here is organized or grouped by broad topic. The list begins with current major publication projects, mainly books now underway or just finished. For the other groupings, usually there is a brief sentence or two about the topic, followed by a list of publications, beginning with the most recent. Proprietary client work is not included; most of this proprietary work recently has involved occasional Health Insurance Portability and Accountability Act (HIPAA) de-identification certifications. Recent Major Publications and Major Works in Progress, 2006 to Present There is a Wiley book, with Wendy Alvey, now in preparation, provisionally titled Exit and RDD Election Polling, dedicated to the late Warren Mitofsky (to appear in the spring of 2008). There is also the website, currently being updated, on election system issues entitled www.votingsystems.us. My recent co-authored paper on exit polls, published in Chance, will appear there, along with my written testimony on how to statistically audit the vote. Also underway at the moment is a book with Susan Ross tentatively titled Presidential Statistical Papers: The First 100 ASA Presidents (I was the 100th ASA President). This will appear in late 2008. Recent books include Data Quality and Record Linkage Techniques, June 2007, with T. Herzog and W. Winkler. Also Human Rights and Statistics, scheduled out in November 2007. This is a book I am co-editing with J. Asher and D. Banks. I am proofing galleys now. Finally, work continues on the still heavily cited web booklet, entitled What is a Survey, available at www.whatisasurvey.info. That booklet is already published in both English and Arabic. Chinese and Spanish versions are being added to the website this fall. Fritz J. Scheuren, Ph.D. Page 2 The American Statistician History Corner, 2003-2006 For three years 2003 to 2006 I produced a column each quarter on statistical history for The American Statistician. The 12 columns are cited fully below, in chronological order, working backwards from the last column, February 2006 to the first May 2003. Vol. 60, no. 1, February 2006 pp. 32 - 33 Vol. 59, No. 4, November 2005 pp.308ff Vol. 59, No. 3, August 2005 pp.223ff Vol. 59, No. 2, May 2005 pp.162ff Vol. 59, No. 1, February 2005 pp.57ff Vol. 58, No. 4, November 2004 pp.290ff Vol. 58, No. 3, August 2004 pp.224ff Vol. 58, No. 2, May 2004 pp.144ff Vol. 58, No. 1, February 2004 pp.12ff Vol. 57, No. 4, November 2003 pp.253ff Vol. 57, No. 3, August 2003 pp.189ff Vol. 57, No. 2, May 2003 pp.94ff Each column began with an introduction that provided context to the submissions received. In many cases I supplemented the submission with heavily excerpted historical papers of my own choosing. My recollections about the emergence of Multiple Imputation also appear in the November 2005 History Corner. Prior to the History Corner in The American Statistician, circa 1995-1997, I had done a column for several years in AMSTAT NEWS excerpting material from landmark papers in survey sampling. Work Products Produced For the Department of Interior, 2001 to Present The NORC reports and deliverables produced for the Office of Historical Trust Accounting (OHTA) are discussed in my deposition. But since they are proprietary they have not been listed here. This is unlike many other client reports (see below) that were developed over the years and results of which have made their way into the scientific literature. National Survey of America’s Families (NSAF) Publications, 1999-2001 Listed below are the paper and web publications that I sponsored, edited and in some cases co-wrote and that formed the backbone of the metadata produced in connection with the 1997 and 1999 National Survey of America’s Families (NSAF). 1999 NSAF Public Use File User's Guide: Report No. 11 (Methodology Report), (2001), The Urban Institute, with N. Converse, A. Safir, R. Steinbach, K. Wang. Fritz J. Scheuren, Ph.D. Page 3 1999 NSAF Collection of Papers: Report No. 7 (Methodology Report), (2001), The Urban Institute, with T. Black, K. Feingold, B. Garrett, A. Safir, K. Wang, D. A. Wissoker, The Westat Group. Survey Methods And Data Reliability, 1997 and 1999 (State Data Profiles), (2000), The Urban Institute, with A. Safir and K. Wang. 1997 NSAF Impact of Census Undercount-Adjusted Weights on Survey Estimates: Report No. 14 (Methodology Report), (2000), The Urban Institute, with S. R. Schmidt, J. Capizzano, The Westat, S. Vandivere. NSAF Methodology Report No. 1: National Survey of America’s Families: Survey Methods and Data Reliability, (2000), The Urban Institute. 1999 Data Editing and Imputation: Report No. 10 (Methodology Report), (2000), The Urban Institute, with T. Black, J. Coder, N. Converse, V. Cox, A. Lhila, The Westat Group. 1997 NSAF Technical Papers: Report No. 16 (Methodology Report), (2000), The Urban Institute, with N. J. Brennan, G. M. Kenney, S. Rajan, K. Wang, S. Zuckerman, The Westat Group. 1997 NSAF MKA Public Use File Documentation and Codebook: Report No. 13 (Methodology Report), (1999), The Urban Institute, with The Westat Group. 1997 NSAF Survey Methods and Data Reliability: Report No. 1 (Methodology Report), (1999), The Urban Institute, with G. M. Kenney, K. Wang. 1997 NSAF Child Public Use File Documentation and Codebook: Report No. 11 (Methodology Report), (1999), The Urban Institute, with The Westat Group. 1997 NSAF Methodology Report No. 15: National Benchmarking Measures, (1999), The Urban Institute. The corresponding report for the 1999 survey is Report No. 6 in the 1999 NSAF Methodology Series 1997 NSAF Methodology Report No. 16: Selected NSAF Research Papers (1999), The Urban Institute. The corresponding report for the 1999 survey is Report No. 7 in the 1999 NSAF Methodology Series. 1997 NSAF Methodology Report No. 10: Data Editing and Imputation, (1999), The Urban Institute, with S. Dipko, M. Skinner, N. Vaden-Kiernan, J. Coder, and S. Rajan. 1997 NSAF Methodology Report No. 11: Child Public Use File Codebook, (1999), The Urban Institute, with B. Russell, and M. Leonard. Fritz J. Scheuren, Ph.D. Page 4 1997 NSAF Methodology Report No. 13: Most Knowledgeable Adult (MKA) Public Use File Codebook, (1999), The Urban Institute, with B. Russell, and N. Converse. 1997 NSAF Methodology Report No. 14: Undercount Adjusted Weights, (1999), The Urban Institute, with M. Petron, E. Engstrom and K. Wang. 1997 NSAF Methodology Report No. 15: National Benchmarking Measures, (1999), The Urban Institute. 1997 NSAF Methodology Report No. 16: Selected NSAF Research Papers (1999), The Urban Institute, with M. Brick, D. Judkins, G. Shapiro, J. Kenney and J. Waksberg and others. 1997 NSAF Methodology Report No. 17: non-MKA Public Use File Codebook, (1999), The Urban Institute, with A. Wigton and Harris Associates. Business and Audit Sampling Research, 1996-1999 While at Ernst and Young much of the statistical work done was client-confidential and so can only be described in general terms. There have been, however, a fair number of audit sampling studies undertaken (involving IRS tax cases, plus some other regulatory agencies, notably the FCC). Inventory sampling was a mainstay too; however, the technical advances made in achieving more efficient inventory designs are proprietary and have not been published. The cases with Yan Liu are the man exceptions. Some of the completed work, notably for government agencies or given before Congress is public and has been cited below. “Efficient Sampling Design and Estimation in Audit Data (II),” (2005), American Statistical Association Proceedings Section on Survey Research Methods, with Y. Liu and M. Batcher “Efficient Sample Design in Special Ratio Type Data,” (2003), American Statistical Association Proceedings Section on Survey Research Methods, with Yan Liu and Mary Batcher. Interim Statistical Analysis for BellSouth Telecommunications (1998), Ernst and Young, LLP, (with S. Hinkins, and E. Mulrow). This was an extensive report to the Louisiana Public Service Commission. It analyzed data required to be reported for regulatory purposes. Subsequent regulatory filings, not cited but which continue, have been less extensive. “Surveying Nonbanked Households: Early Results from a Pilot Study,” (1998.) Proceedings of the Section on Survey Research Statistics, American Statistical Association, with C. Dunham, and D. Willson. Fritz J. Scheuren, Ph.D. Page 5 1997 National Bankruptcy Petition Study, Prepared for VISA and delivered to the House of Representatives in April 1998, with Tom Neubig. Surveying the Financial Service Needs of Non-Banked Households: Final Survey Design (1997), Ernst and Young, LLP. This was prepared for the Office of the Controller of the Currency and led to two surveys, the second of which is described in a paper given in 1998 (see above). Survey Literacy, 1995 to Present The booklet, entitled What Is a Survey and found at www.whatisasurvey.info was mentioned already. It grew out of a small booklet of the same name finished in 1980. An effort to update the material led to a series of pamphlets. The titles and dates of the pamphlets, produced by me, now in modified form chapters on the website, are shown below. “More About Telephone Surveys,” (1999), Tenth pamphlet in series (written by J. Lepkowski). Published by the American Statistical Association. “Designing a Questionnaire,” (1999), Ninth pamphlet in series (written by L. Stinson). Published by the American Statistical Association. “What Is a Margin of Error?” (1998), Eighth pamphlet in series (written by Lynne Stokes and T. Belin). Published by the American Statistical Association. “More About Mail Surveys,” (1998), Seventh pamphlet in series (rewritten by me from a student group project in my sampling class at The George Washington University with help from D. Dillman). Published by the American Statistical Association. “What Are Focus Groups?” (1997), Sixth pamphlet in series (rewritten by Scheuren, based on a student group project in my sampling class at The George Washington University). Published by the American Statistical Association. “How to Conduct Pretesting,” (1997), Fifth pamphlet in series (rewritten by me from a Census Bureau Pamphlet on the same topic, with help from T. Demaio). Published by the American Statistical Association. “Judging the Quality of a Survey,” (1996), Fourth pamphlet in series (authored by W. Kalsbeek). Published by the American Statistical Association. “How to Collect Survey Data,” (1995) Third pamphlet in series (rewritten by me in part from original outdated booklet, What Is a Survey). Published by the American Statistical Association. Fritz J. Scheuren, Ph.D. Page 6 “How to Plan a Survey,” (1995) Second pamphlet in series (rewritten by me in part from original outdated booklet, What Is a Survey). Published by the American Statistical Association. “What Is A Survey?” (1995) First pamphlet in series (rewritten by me from original outdated booklet of the same name). Published by the American Statistical Association. Other Topics, 1995-2007 “Vanuatu Household Income and Expenditure Survey,” (2007). To be presented at the Comparative Survey Design and Implementation (CSDI) International Conference on Survey Methods in Multicultural, Multinational, and Multiregional Contexts with S. Johnson and C. Ryan. “Vanuatu Country Background and Challenge,” (2007). To be presented at the Comparative Survey Design and Implementation (CSDI) International Conference on Survey Methods in Multicultural, Multinational, and Multiregional Contexts with S. Johnson and A. Sewen. “Paradata Inference Applications,” (2007). To be presented at the International Statistical Institute, 56th Biennial Session. “Eight Rules of Thumb for Understanding Survey Error,” (2007). RTI International Gertrude M. Cox Seminar Series. Book Review of Statistical Matching: Theory and Practice. (2007) Journal of the American Statistical Association. “The Pro Bono Statistician” (2007) Journal of the American Statistical Association. “Explaining Discrepancies between Official Votes and Exit Polls in the 2004 Presidential Election” (2007) Chance with S. Kyle, D. Samuelson, and N. Vicinanza. “Paradata from Concept to Completion,” (2005). Proceedings of Statistics Canada Symposium 2005, Methodological Challenges for Future Information Needs. “Seven Rules of Thumb for Nonsampling Error in Surveys,” (2005) National Institute of Statistical Science (NISS) Total Survey Error Conference. Model-Ready Designs for Client Chosen Models (2005). Published talk given at a series of Washington Statistical Society (WSS) lectures in 2004 and 2005, with S. Hinkins, M. Batcher, Y. Liu, A. Mushtaq, and P. Baier. Model Ready Efficient Sample Designs (2004). Published talk given at a series of Washington Statistical Society (WSS) lectures in 2004, with S. Hinkins, M. Batcher, Y. Liu, A. Mushtaq, and P. Baier. Fritz J. Scheuren, Ph.D. Page 7 “From Survey Data to Multiple Types of Data in Historical and Real Time,” (2004), American Statistical Association Proceedings Section on Survey Research Methods, with J. Lott, J. Keller, and D. Banks. “The Still Unfinished Journey from Single to Multiple Imputation,” (2004), American Statistical Association Proceedings Section on Survey Research Methods. “Regression-based Statistical Matching: Recent Developments,” (2004). American Statistical Association Proceedings Section on Survey Research Methods, with C. Moriarity. “Information integration: Case studies and emerging principles.” (2004), American Statistical Association Proceedings of the Joint Statistical Meetings, Section on Survey Research Methods, with D. Judson, C. Popoff, & D. Culhane. “From survey data to multiple types of data in historical and real time.” (2004) American Statistical Association Proceedings of the Joint Statistical Meetings, Section on Survey Research Methods, with J. Lott, J. Keller, & D. Banks. “A Note on Rubin’s Statistical Matching Using File Concatenation with Adjusted Weights and Multiple Imputation,” (2003) Journal of Business and Economic Statistics, with C. Moriarity. “The Use of Overlapping Replicates to Reduce Bias in Replicate Variance Estimates,” (2003), American Statistical Association Proceedings Section on Survey Research Methods, with S. Hinkins and C. Moriarity. “Dalenius Days at the U.S. Federal Committee on Statistical Methodology,“ (2003), American Statistical Association Proceedings Section on Survey Research Methods. “Statistical Matching with Assessment of Uncertainty in the Procedure: New Findings,” (2003), American Statistical Association Proceedings Section on Survey Research Methods, with C. Moriarity. “Efficient sample design in special ratio type data.” (2003) American Statistical Association Proceedings of the Joint Statistical Meetings, Section on Survey Research Methods, with Y. Liu & M. Batcher. “Afghan Refugee Camp Surveys: Pakistan, 2002,” (2002), American Statistical Association Proceedings Section on Survey Research Methods, with J. Bell, D. Nolle, R. Citrin Killings and Refugee Flows in Kosovo in March – June 1999, (2002), with P. Ball, W. Betts, J. Dudukovich, and J. Asher. Published by the American Association for the Advancement of Science. Fritz J. Scheuren, Ph.D. Page 8 “Longitudinal Attrition in SIPP and SPD” (2002), American Statistical Association Proceedings Section on Survey Research Methods, with D. Vaughan. “Survey Sampling Issues in Human Rights Research to begin the 21st Century” (2002) AMSTAT NEWS. “Estimating illegal immigrants: The Three Card Method: Estimating Sensitive Survey Items—With Permanent Anonymity of Response,” (2001), American Statistical Association Proceedings Section on Survey Research Methods, with J. Droitcour and E. Larson. “Statistical Operations and Studies in the SOI Program of the IRS,” (2001), Turning Administrative Systems into Information Systems, Internal Revenue Service; Washington, DC, with T. Petska and J. Hobbs. “Statistical Matching: A New Validation Case Study,” (2000), American Statistical Association Proceedings Section on Survey Research Methods, with D. Ingram, J. O'Hare, and J. Turek. “A Confidentiality Fable,” (2000), American Statistical Association Proceedings Section on Survey Research Methods, with J. Mulrow. “Increasing Public Accessibility to Complex Survey Data by Using Inverse Sampling,” (2000), American Statistical Association Proceedings Section on Survey Research Methods, with S. Hinkins and V. Parsons. “Estimating Residency Rates for Undetermined Telephone Numbers,” (2000), American Statistical Association Proceedings Section on Survey Research Methods, with J. M. Brick and J.Montaquila. Later Published in Public Opinion Quarterly (2003). “Preserving Both Confidentiality and the Ability to Calculate Variances in the National Health Interview Survey – Part II,” (2000), American Statistical Association Proceedings, Section on Survey Research Methods, with S. Hinkins and V. Parsons. “Metadata and paradata In the National Survey of America’s Families’ (2000) an invited paper presented at the 2000 Joint Statistical Association Meetings (included in report No. 7 of the 1999 NSAF methodology series). “Inverse sampling algorithm for NHIS confidentiality protection.” (1999) American Statistical Association Proceedings, Section on Survey Research Methods, with S. Hinkins and V. Parsons. “Human Rights Violations in Guatemala,” (1999), American Statistical Association Proceedings, Section on Survey Research Methods, with P. Ball, H. Spirer, and W. Seltzer. Fritz J. Scheuren, Ph.D. Page 9 “NSAF Quality Challenges and Responses in Surveying the Poor,” (1999). Presented at the Association for Public Policy Analysis and Management Conference, (with K. Wang). “Improving Stratified Sampling by Employing Median Balancing” (1999). Proceedings Section on Survey Research Methods, American Statistical Association, with Y. Liu. “Linking Data Sets: Information Needs Versus Privacy in the Computer Age—A Balancing Act?” (1999). Presented at the American Evaluation Associate Conference, with J. Droitcour. “Preserving Both Confidentiality and the Ability to Calculate Variances in the National Health Interview Survey,” (1999), American Statistical Association Proceedings, Section on Survey Research Methods, with S. Hinkins and V. Parsons. “Administrative Records and Census Taking,” (1999). Survey Methodology. Based on an earlier unpublished report submitted to the Census Bureau in May 1999. “The Confidentiality Beasties: A Fable About the Elephant, the Duck, and the Pig,” (1998), Turning Administrative Systems Into Information Systems, Internal Revenue Service, with J. Mulrow. “Design-Free Data Analysis,” (1998). Presented at the 1998 Statistical Society of Canada Meetings, with S. Hinkins, and Y. Liu. “Methodological issues in surveying the nonbanked population in urban areas.” (1998) American Statistical Association Proceedings, Section on Survey Research Methods, with C. Dunham, R. and D. Willson. “Replicate variance estimates: Reducing bias by using overlapping replicates.” (1997). American Statistical Association Proceedings, Section on Survey Research Methods, with S. Hinkins and H. Oh. “Applying mass imputation using the schools and staffing survey data.” (1997) American Statistical Association Proceedings, Section on Survey Research Methods, with S. Kaufman. “Inverse Sampling Design Algorithms,” (1997) Survey Methodology, with S. Hinkins and H. L. Oh. “Merging and Analyzing Administrative Lists and Data,” (1997), Survey Methodology, with W. Winkler. “IRS Test Call System,” (1997). Survey Measurement and Process Quality, Wiley: New York, (with Mary Batcher – 1996). Fritz J. Scheuren, Ph.D. Page 10 “The Census Sampling Controversy: When Can A Sample Be Better Than A Census?” (1997), Consortium. “Using Permanent Random Numbers in Enterprise Surveys,” (1996) American Statistical Association Proceedings, Section on Survey Research Methods, with S. Hinkins and C. Moriarty. “New Approaches to Teaching Survey Sampling,” (1996), The American Statistician, (with Ron Fecso, Sharon Lohr, Elizabeth Stasny, William Kalsbeek, and William Schaeffer) “Median Balanced Sampling Design”, (1996), American Statistical Association Proceedings Section on Survey Research Methods, with Y.Liu. “Total Quality Management in an Administration Setting,” (1996). Data Quality, with J. Mulrow). “Replicate Variance Estimation in Stratified Sampling With Permanent Random Numbers”, (1996), American Statistical Association Proceedings Section on Survey Research Methods, with S. Hinkins and C. Moriarity. “Where Will it all End? Some Alternative SASS Estimation Research Opportunities,” (1996), American Statistical Association Proceedings Section on Survey Research Methods, with S. Kaufman. “Recursive Merging and Analysis of Administrative Lists and Data,” (1996), Proceedings of the Section of Government Statistics, American Statistical Association, with W. Winkler. “Recursive Analysis of Linked Data Files,” (1996). Proceedings of the 1996 Census Bureau Annual Research Conference, with W. Winkler. Discussion of Janet Norwood’s book “Organizing to Count,” (1996), The American Statistician. “Mass Imputation in the Private Schools Survey,” (1996). Proceedings of the Section on Survey Research Statistics, American Statistical Association, with Steve Kaufman. “Administrative Record Opportunities in Education Survey Research,” (1996), From Data to Information: New Directions for the National Center for Education Statistics, U.S. Department of Education Office of Educational Research and Improvement. Exploring Nonresponse Bias in the School and Staffing Survey, (1996) National Center for Education Statistics, U.S. Department of Education. Fritz J. Scheuren, Ph.D. Page 11 “Generalized Least Squares as a Method for Achieving Consistency across School Surveys,” (1995). Proceedings of the Section on Survey Research, National Center for Education Statistics, U.S. Department of Education. “Intersurvey Consistency in NCES Private School Surveys”, (1995), National Center for Education Statistics, U.S. Department of Education, with B. Li. “An Administrative Record Census in the U.S.?” (1995), Chance. “Trust in The U.S. Statistical System,” (1995). Turning Administrative Systems into Information Systems, Internal Revenue Service; Washington, DC. “Linking Data to Create Information,” (1995), Proceedings of Statistics Canada Symposium, with W. Winkler. “Linking Health Records: Human Rights Concerns,” (1995). Commissioned paper for the Science and Human Rights Program, American Association for the Advancement of Science. Also reprinted in Record Linkage Techniques – 1997. “Review of Private Lives and Public Policy,” (1995), Journal of the American Statistical Association. “Improved GLS Estimation in NCES Surveys”, (1995), American Statistical Association Proceedings Section on Survey Research Methods, with S. Kaufman and B. Li. “Using an inverse sampling algorithm for tests of independence based on stratified samples,” (1995), American Statistical Association Proceedings Section on Survey Research Methods, with S. Hinkins and H. Oh. “Improved GLS estimation in NCES surveys,” (1995), American Statistical Association Proceedings Section on Survey Research Methods, with S. Kaufman and B. Li. “Multivariate modeling of unit nonresponse for 1990-91 schools & staffing surveys,” (1995), American Statistical Association Proceedings Section on Survey Research Methods, with S. Salvucci, F. Zhang, D. Monaco, and K. Gruber.