IN THE UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 

ELOUISE PEPION COBELL, et al.,  )  
)  
Plaintiffs,  )  
)  
v.  )  Case No. 1:96cv01285(JR)  
)  
DIRK KEMPTHORNE,  )  
Secretary of the Interior, et al.,  )  
)  
)  
Defendants.  )  

__________________________________________) 
DEFENDANTS’ FILING OF RESPONDING EXPERT REPORT 
OF FRITZ SCHEUREN, PURSUANT TO RULE 26(a)(2) 
OF THE FEDERAL RULES OF CIVIL PROCEDURE 

Defendants hereby file and attach hereto the Responding Expert Report of Fritz 
Scheuren. 
Dated: September, 17, 2007. 
Respectfully submitted, PETER D. KEISLER Assistant Attorney General MICHAEL F. HERTZ Deputy Assistant Attorney General 
J. CHRISTOPHER KOHN Director
 /s/ Robert E. Kirschman, Jr. ROBERT E. KIRSCHMAN, Jr. 
(D.C. Bar No. 406635) Deputy Director Commercial Litigation Branch Civil Division 
P.O. Box 875 Ben Franklin Station Washington, D.C. 20044-0875 Phone (202) 616-0328 Fax (202) 514-9163 
CERTIFICATE OF SERVICE 
I hereby certify that, on September 17, 2007 the foregoing Defendants’ Filing of Responding Expert Report of Fritz Scheuren, Pursuant to Rule 26(a)(2) of the Federal Rules of Civil Procedure was served by Electronic Case Filing, and on the following who is not registered for Electronic Case Filing, by facsimile: 
Earl Old Person (Pro se) Blackfeet Tribe 
P.O. Box 850 Browning, MT 59417 Fax (406) 338-7530 
/s/ Kevin P. Kingston Kevin P. Kingston 

Expert Rebuttal Report 
Fritz Scheuren 
NORC, University of Chicago 
scheuren-fritz@norc.org 
September 17, 2007 

This is the second of two rebuttal reports that the National Opinion Research Center ~ORC) has prepared to respond to comments that Plaintiffs’ expert Mr. DuncanI has made. Here I will confine attention to his comments on the scope and value of the Qualitative Meta-Analysis (QMA) done2 by NORC as part of its engagement with the Office of Historical Trust Accounting (OHTA). 
As co-leader (with Dr. Susan Hinkins) of the NORC team providing statistical support services to OHTA, I have a very deep knowledge of the role of the QMA and how the analysis was conducted. In what follows, I respond to Mr. Duncan’s claim that the QMA is "misrepresentative, misleading, starkly inconsistent with the underlying studies, and ultimately irrelevant." Before doing so, I would like to first describe the role of the QMA with respect to DOI’s 2007 Historical Accounting Plan For Individual Indian Money Accounts (2007 Plan). 
QMA Role 
Mr. Duncan’s claim that the QMA will play a central role in the 2007 Plan is false and unfounded. In point of fact, the NORC Qualitative Meta-Analysis was conducted primarily for planning and hypothesis setting purposes. The focus was on "looking ahead" to the new work proposed in the 2007 Plan by "looking backward" at what others had already done. Working hypotheses, not conclusions, were the goal. 
Such an approach is analogous to what an investor would do when deciding to invest in a mutual fund. A review of the funds prospectus would provide the investor with a history of past performance. Along with other information, the prospectus’ past performance report would help the investor decide whether an investment in the fund should be made; and, if so, how much of an investment. 
In a similar way, NORC wanted to review past history to determine the likelihood of finding errors in the Paper Ledger Era population. Continuing the analogy, NORC had to decide how much sampling capital (i.e., sample size) to recommend be invested in the Paper Ledger Era testing. Using perceived past performance made sense as one input. Of course there were other inputs--the results from the Litigation Support Accounting (LSA) project were extremely valuable because they gave NORC a way to benchmark these "perceived past performances" against the scientifically collected data obtained by 
~ Referenced in the August 23, 2007 report to the court entitled Expert Report Of Dwight J. Duncan, CFA, especially Appendix C.
2 Qualitative Meta-Analysis of Audit and Reconciliation Studies on Indian Trust Accounts (NORC 2006). 
NORC and OHTA’s other contractors. Another important input was gleaned from reviewing a sample of the 60 years of GAO and Treasury Settlement Packages stored at National Archives II in College Park, MD.3 
There was never any expectation that the QMA would replace the hard tasks that lay ahead. It was hoped that it would simply shed light on the possible scope of the work, the likely accuracy rates, and the availability of supporting documents. 
In summary, the task of QMA was to review, integrate, and synthesize the findings of the many past accounting studies of the Indian Trust system in order to set the stage for statistically grounded conclusions about the quality of administration of the Indian Trust Fund. The variety and breadth of these previous studies makes meta-analysis a valuable tool for accomplishing that task.4 
In aggregate, the reports reviewed do not point to a non-performing system. However this conclusion does no._3t mean that testing of the Paper Ledger Era transactions is unnecessary. Instead, it supports the position in the 2007 Plan that a large sample is not likely to be needed for the Paper Ledger Era. The 2007 Plan proposes a smaller sample than used in the LSA to test the hypothesis that error rates similar to those found in the Electronic Ledger Era will be found in the Paper Ledger Era. Despite the QMA findings, if the Paper Ledger Era sample findings turned out to be inconsistent with the LSA findings, then the 2007 Plan, as described in my expert report submitted to the court on August 24, 2007, calls for expanding the sample adaptively to produce sound independent error rates for the Paper Ledger Era. 
Response to Mr. Duncan 
In what follows, I present factual responses to Mr. Duncan’s criticisms, as stated in Appendix C of his expert report filed August 23, 2007. Generally, Mr. Duncan has misunderstood our purpose in doing the QMA and so, even when he is factually correct, he uses his facts to support irrelevant criticisms. 
3 Working with Reznick Group PC (one of the accounting firms engaged by OHTA), NORC selected and 
examined a sample of the audit records prepared by the Treasury Department (during the period 1890­1920) and then, later, the General Accounting Office (from 1921 to 1950). These Account Settlement Packages contain I!M ledgers, supporting financial documents, and the work papers of GAO and Treasury auditors, who, prior to 1951, audited the accounts of Indian Service Special Disbursing Agents.
4 In the NORC Qualitative Meta-Analysis Report there are over a dozen general references given to meta­
analysis, so I feel no need to repeat them here. An additional reference that may be of interest is the Federal Judicial Center’s Reference Manual on Scientific Evidence, Second Edition, (2000), which can be found at http://www.t]c.gov. Typically, meta-analysis is a quantitative tool that makes use of quantitative 
information from comparable, relevant study findings. Well-known uses are in epidemiology and medical clinical trials, to integrate the results of different studies. One example, from Gross, A. (1993) Does exposure to second-hand smoke increase lung cancer risk? Chance, Vol. 6, No. 4, is the combined examination of data from 33 epidemiological studies to assess more definitively the possible relationship between second-hand smoke and lung cancer risk. NORC considers its meta-analysis a "qualitative analysis" because it cannot result in a single set of numerical summaries. The work was too variable for that. NORC was left, therefore, with verbal summaries. 
2 
My style for responding to Mr. Duncan is as follows: 
I directly quote Mr. Duncan’s report to frame my comments. When this is done 
his text will be italicized, indented and in 10-point type. 
My responses to his accusations and misstatements will not be italicized or 
indented. They will appear in standard 12-point type. 
~, 	If Mr. Duncan added emphasis to his writing, I left the emphasis alone. I have not added any emphasis of my own to Mr. Duncan’s statements. 
Mr. Duncan’s footnotes are not included here, but the footnote marks in his text 
remain. 
If Mr. Duncan quoted other sources in his report, I have changed the font to make 
his source clear. 
I take up Mr. Duncan’s points in order and provide side-headings that follow his 
numerical subheadings. 
On some occasions Mr. Duncan makes only a flat assertion, without citing factual evidence or argument. In all my responses I try to cite evidence or argument, but sometimes his unsupported accusations compel a response in kind. 
Duncan 1.1 
The Meta-Analysis Report draws "audits" from the numerous organizations. The studies are then 
grouped into subcategories (e.g. "qualified opinions", "unqualified opinions"). Some of the 
reports are so named by the accountants that conducted the actual audits. Many others are not so 
named NORC does not purport to be a professional accounting organization, auditors or 
accountants. The distinction between qualified and unqualified opinions is not within the purview 
of statisticians. Therefore, NORC ’s summary of "audit" reports is beyond NORC ’ s credentials 
and is without basis, and the conclusions regarding the "audit" reports can be disregarded 
Mr. Duncan has posited an erroneous statement to reach an unsupported conclusion. A sertior accountant was specifically hired by NORC to work on the QMA reviews of the audit studies. Therefore, NORC did have an accountant, and its conclusions regarding audit reports cannot be disregarded. 
The conclusions of the Meta-Analysis Report rely heavily on NORC’s analys~ of the "audit" reports. Therefore, any errors, misstatements, or exaggerations made by NORC related to the audit portion of the Meta-Analysis will adversely affect the conclusions and implications of the Meta-Analysis Report. 
The main point that NORC makes in its QMA work regarding the Indian Trust audit 
reports is that they are often aimed at risk assessment and failures to comply with stated 
standards, not the actual detection of errors. Reconciliations looked for actual errors by identifying any instances in which the records of the transactions do not match the source 
documents. 
The NORC team went to great effort in its report to distinguish the 900+ studies it located into audit and reconciliation categories. "Reconciliations," as NORC used the term, usually report quantitative results, such as the percentage of dollars in error for accounts or transactions examined. The LSA project NORC worked on is an example of what we considered a reconciliation. "Audits," were studies categorized by NORC that focused on specific processes used to administer the Indian Trust Funds, including identification and documentation of risks and weaknesses. The NORC category of Indian Trust audits usually contained limited quantitative results, if any, and consisted primarily of recommendations for process improvements. 
Now, as NORC defined them, the audits and reconciliations broadly come out in very 
different places. The Indian Trust audits found considerable system risk5 and 
noncompliance with requirements, while the reconciliations found few errors. Recall that 
one of the primary purposes of the QMA was to help decide how much sampling capital 
should be invested in the Paper Ledger Era testing. We used the LSA project results in 
conjunction with the QMA findings as a guide. 
The LSA project examined transactions in the period 1985 to 2000. There were 36 audit studies used in the QMA work that were performed during that same time period.6 Nearly all of these audits identified systems risks caused by perceived noncompliance. The LSA results showed overall error rates at or under 1%. This is very similar to the QMA overall results--audits suggest system risks while reconciliations suggest very low error rates. Thus NORC felt comfortable recommending a small sample size for the Paper Ledger Era testing under the assumption that results would be similar to the LSA error rates. As I mentioned in my first expert report, I also recommend that the Paper Ledger Era sample size be adapted if initial sample results indicate more problems exist than suggested by the QMA or the LSA. 
Duncan 1.2 
The Meta-Analysis Report conclusion regarding the reconciliation studies is, at best, a subtle 
misstatement of the underlying studies and at worst, a blatant misrepresentation designed to 
mislead the reader into believing the studies matched nearly all of the transactions to source 
documents and found little to no errors. 

There is no misstatement by NORC. The QMA simply "tells it like it is." In conducting the QMA, NORC operated under the premises that reliable data from past studies can "speak for themselves." For the QMA it was important to make sure that reliable data 
5 Slovie, P. (2000) The Perception of Risk (Earthscan Publications Ltd, London and Sterling, VA) is a 
good general source. Statistics is based in part on risk measurement and NORC certainly has professional standing on the topic. 6 The titles of the 36 audits studies can be found in Qualitative Meta-Analysis of Audit and Reconciliation 
Studies on Indian Trust Accounts, Volume II: Bibliography and Index (NORC 2006). 
were considered, as it would be in any meta-analysis. Therefore, NORC set a standard 
that a past reconciliation study needed to have a reconciliation rate of 80% or higher to be 
considered for the study. NORC excluded studies that (a) had less than 80% transactions 
reconciled,7 (b) could not be obtained during the study period or (c) were irrelevant--no 
reconciliation was done or only a partial report was obtained and so the NORC team 
could not make a determination as to the results of the study.8 NORC considered this 
reasonable, even though the LSA met a much higher standard. The LSA was 
unprecedented in this respect, so a lower, but still reasonably high reconciliation rate was 
used. 
The task of QMA was to review, integrate, and synthesize the findings of the many past accounting studies of the Indian Trust system in order to set the stage for statistically grounded conclusions about the quality of administration of the Indian Trust Fund. As a starting point, we accepted the conclusions of past studies; we did not try to re-evaluate any conclusions. We simply categorized each "in-scope’’9 study as described in the NORC report. 
In Section 1.2, Mr. Duncan shows a table that categorized reports by the organization that conducted the reconciliation, and by whether or not the report was included in the QMA. Mr. Duncan notes that, overall, 388 (63%) past reconciliation reports were left out of the QMA. My understanding of Mr. Duncan’s concern here is that any conclusion based on approximately one-third of the available reports cannot be meaningful. But, in fact, a conclusion based on a larger set of reports that would mix low-quality data with high-quality data is what would be meaningless. 
As I have already noted, Mr. Duncan wants to make it appear that the QMA plays a role in the 2007 Plan that was not intended. He does not understand the goals of the QMA or the goals ofmeta-analysis in general. 
In a meta-analysis study, it is typical that studies that do not meet pre-defined selection criteria are excluded. In the NORC team’s view, there were no conclusive reconciliation results that could be drawn from the studies that were excluded. Never were studies excluded from the QMA because they "did not feed into the conclusions." 
7 By restricting the studies reviewed to those with at least an 80% reconciliation, NORC was eliminating studies with inconclusive results. This is analogous to what a statistician might do when a survey has a low response rate and the responses received are not representative of the population.
8 For example, the majority of the studies left out of the QMA Report were Indian Trust Accounting 
Division (ITAD) studies. The omitted ITAD studies were mostly dockets, letter reports or accounting schedules pertaining to litigation without any actual reconciliation study being done.
9 "In-scope" studies were those studies left over after "out-of-scope" studies were removed. Excluded 
studies were those that (a) had less than 80% transactions reconciled, (b) could not be obtained during the study period or (c) were irrelevant--no reconciliation was done or only a partial report was obtained and so the NORC team could not make a determination as to the results of the study. 
The NORC team’s approach is consistent with the meta-analysis literature. The NORC 
team did not redo the studies but aggregated the results of individual studies with 
reasonably high reconciliation rates in order to make a collective statement. This too is 
consistent with usual meta-analysis practice. 
Duncan 1.2.1 
Based on the information EconLit was able to obtain, the Meta-Analysis Report appears to present 
a single AA project (the tribal trust reconciliation projecO as 294 different reconciliation reports. 
While it is technically correct that AA conducted certain reconciliation procedures with respect to 
294 different tribes, the procedures and criteria for reconciliation were all the same. Therefore, 
for purposes of summarizing the AA reconciliation findings, the Meta-Analysis Report’s 
conclusion misrepresents the underlying studies. 
The Arthur Andersen (AA) work was done under a single contract but it consisted of several hundred reports of individual tribal reconciliation studies, which, individually, had varying reconciliation results. Because AA issued separate reports for each tribe, NORC similarly treated these as separate reports. The QMA considered for inclusion AA reports with the pre-defined reconciliation rate of at least 80%, as was done for all reconciliation reports considered in the QMA. On this basis not all tribal AA results proved usable for inclusion. Similar procedures and criteria were used for all the AA reports. This similarity is good because it reduces study-to-study variability in methodologies used. 
It should further be pointed out that most of the non-AA reconciliation studies were 
focused on one Tribe or Agency. That the AA work be divided and considered as 
individual Tribal reconciliation studies is appropriate in the context of all the reconciliation studies identified. 
Furthermore, the AA studies identified address tribal trust reconciliations. The Meta-Analysis 
Report fails to disclose this fact and fails to disclose the fundamental differences between tribal 
trust accounts and individual trust accounts. 
The QMA report clearly states that the AA studies pertained to tribal accounts. For example, on page 3 of the first volume of the QMA Report, is a list of the three sources from which the majority of studies were obtained. The source definition of the AA work is given as: "Tribal account reconciliations conducted by Arthur Andersen LLP." Also, the second volume of the QMA Report contains a list of all the AA reports--they are clearly labeled as AA tribal accounting reports. 
While there are differences between tribal trust accounts and individual trust accounts, the QMA work was an examination of the entire Indian Trust Fund administration. NORC never maintained the QMA was limited to IIM. It is our understanding that management of the Tribal and IIM Indian Trust System has not been totally separate historically, with a portion of Tribal Trust Fund transactions processed through the IIM structure.’° Because of this, and the fact that the QMA is only meant to provide us an indication of what to expect in Paper Ledger Era testing, in my expert opinion, it was appropriate to review both Tribal and individual accounts in order to evaluate the system as a whole. 
It & informative to know that AA attempted to reconcile both types of accounts and ultimately only reconciled tribal trust accounts over a limited time period (as discussed further infra). The Meta-Analysis Report’s summary of findings with respect to tribal trust accounts is misrepresentative of individual lndian trust accounts.. 
While it is true that the original intent of the AA work was to cover individual trust accounts, my understanding is that after many cost overruns, the scope of AA’s work was limited. This point was discussed in an a 1992 House report.|1 The QMA report tried to give a summary statement of the Indian Trust system, without specifically looking at tribal or individual accounts. 
Duncan 1.2.2 
As is clear from Mr. Homan ’s testimony, the ,t,4 tribal trust reconciliation project was simply a set of agreed-upon procedures selected only after AA concluded a unqualified audit was impossible. 
The AA project was undertaken under agreed-upon procedures to check the posted transactions against the supporting documents. In the QMA report, NORC treated this as a reconciliation, and what AA learned was judged still useful for planning further work in the Paper Ledger Era. That it had limitations has been amply covered in the NORC’s two volume QMA report and in NORC’s world papers. 
NORC’s failure to reveal the true history of the AA tribal trust reconciliation project reveals a significant bias in the underlying Meta-Analysis Report and draws the other conclusions in the Meta-Analysis Report (many subject to substantial professional judgment) into question. 
NORC’s approach in the QMA study, as it is in all the studies NORC has conducted for OHTA, is to let the data speak with all their relevant limitations cited. There is no personal bias with such an approach. That an unqualified audit could not be conducted actually agrees with the results presented by NORC in the audit studies section of the QMA report. 
Just because AA could not perform an unqualified audit does not mean that the reconciliation work done is not fit for use. Reconciliation studies and audit studies should not be confused here. Again, NORC’s working interpretation for the QMA was that an audit documents the risk of an error occurring, and a reconciliation documents the actual occurrence of errors. 
10 See the Morgan Angel & Associates January 2006 report entitled Tribal Trust Fund Policies and Procedures: 1946- 2005. 
~ H. R. Rep. No. 102-499, April 22, 1992. 
The "Study Summary Template" in the Meta-Analysis Report also reveals blatantly misleading 
statements regarding the AA studies. For example, under the "Study Approach" section, the Meta-Analysis Report presents the following statement in answer to the self-posed question, "How was the accounting work done?" 
"Basic Reconciliation verified non-investment receipts and disbursements posted to the Tribal account ledger with the relevant supporting documents (financial only). Verified the dollar amount, transaction date, and account number." 
However, the AA tribal trust reconciliation study EconLit reviewed reveals the misleading nature of the Meta-Analysis Report. Under the "Basic Reconciliation" section of the AA tribal trust reconciliation study, it discusses the following procedure for verification of a receipt: 
"We verified (i.e. traced and agreed) non-investment receipts posted to each of the Tribe’s accounts, to the extent source documentation was provided, as to amount, date, and account number to deposit tickets and/or related collection vouchers..."(emphasis added [by Mr. Duncan]). 
The highly relevant qualification was not disclosed in the Meta-Analysis Report. Even under the Meta-Analysis Report section inquiring about limitations of the study, this material qualification was omitted. By failing to disclose that the AA tribal trust reconciliation studies were limited to documents provided, the Meta-Analysis Report is misleading. 
Mr. Duncan appears to be accusing NORC of trying to hide the statement "to the extent source documentation was provided." NORC has not hidden any aspects of the AA reconciliation studies. It is my understanding, based on discussions with OHTA accounting contractors along with my work experiences at Ernst & Young, that report language such as that highlighted by Mr. Duncan is a fairly standard disclaimer in 
accounting reports.~2 NORC took into account that some reconciliations had limited documentation. 
As I have stated previously, reconciliation rates were considered by NORC as making results more or less reliable. All AA reconciliations were done "to the extent source documentation was provided" and NORC assumed that the 80% or more reconciliation rate in the AA studies it considered was achieved after using documents provided. So the lack of documentation was considered because this would have caused smaller reconciliation rates--and NORC would have excluded such reports. 
NORC omitted reconciliation studies with a low reconciliation rate to be conservative, not to artificially improve results as seems to be Mr. Duncan’s conviction. It is illustrative to note that a disproportionate number of the AA reconciliation studies, excluded from the NORC analysis have a 0% error rate. If these studies were included and we assume that the studies are missing-at-random,~3 the reported error rate for the AA studies would actually be lower than that given in the QMA. The highest error rates in the AA work were (on average) found in the reconciliation studies with higher reconciliation rates. NORC, for the QMA studies purpose, used the higher error rates. 
lz Mr. Duncan himself uses such language in his August 23, 2007 report. In section 2 he states "Accordingly, EconLit’s analysis is limited to the information available at this time."
13 Rubin, D. B. (1976). Inference and Missing Data. Bioraetrika 6.3, 581-592. 
Duncan 1.2.3 
The Meta-Analysis Report "Study Summary Template" also contains statements that are directly and starkly inconsistent with the underlying studies. For example, under the "Results" section, the Meta-Analysis Report presents numerical conclusions with respect to the level and dollar amount of transactions reconciled and concludes that the results are "representative". However, based on the AA tribal trust reconciliation study EconLit reviewed, NORC must have selected a tribe that had extremely high "reconciliation rates" and ignored the summary of all tribes found in the same report. Specifically, the Meta-Analysis Report concluded that 98 percent of noninvestment dollars were reconciled By contrast, the actual overall reconciliation figure was only 87 percent. 
Incredibly, the 87 percent figure was only obtained by AA after attempting to do a "full reconciliation" but due to inadequate documentation then having to accept the "Bureau’s standard of reasonableness " for purposes of labeling a transaction as "reconciled".3 
The Study Summary Template that Mr. Duncan refers to is Appendix A of QMA volume 
1. In the QMA introduction section (page 5), it clearly states: "In Appendix A, we illustrate two of the reviews actually conducted--an audit and a reconciliation." (emphasis added) NORC’s work papers include individual templates for all AA tribal reconciliations. Appendix A of the QMA includes just one of these templates. It is never stated in the QMA report that the illustrations in Appendix A represent the QMA conclusions. 
The noninvestment dollars reconciliation rate from the QMA report that Mr. Duncan refers to is for one reconciliation, specific to one tribe.14 Mr. Duncan is trying to imply that the reconciliation report for this one tribe represents all the tribal reconciliati6n reports conducted by AA. It does not. 
With respect to NORC’s conclusion that 98% of noninvestment dollars were reconciled for the one specific AA report described in the reconciliation example of the Study Summary Template, I am not sure why Mr. Duncan has a problem. If he is trying to argue that the example report is not representative of all AA reports because AA concluded that the reconciliation rate was 87%, and the example report of one single tribe shows 98%, then his comparison is misplaced. The Summary Study Template example is the result for the one tribe that is illustrated. The fact that it differs from an AA Tribal Trust reconciliation that Mr. Duncan reviewed should not be a big surprise. It is quite common when reviewing observations from a collection of data, such as the reconciliation reports, that numerical values of observations will vary from one report to the next. Some values will be "close" to the typical or average value of the whole data set, and some will not be "close." A single average value for noninvestment dollars-­which NORC did not include in the QMA--cannot full3~ describe the reconciliation rate of the totality of all AA reconciliation reports considered for the QMA. 
The Study Summary Template does not disclose the tribe names. This would be inappropriate for the QMA report, which is available to a wide audience. 
NORC just considered studies with reconciliation rates of at least 80% for the QMA; not 
all studies, just those deemed to have enough conclusive information. So the only 
conclusion that NORC would make about the overall reconciliation rate is that it is 80% 
or more, but less than 100%. The example report in the Study Summary Template is 
consistent with this, and if any report Mr. Duncan reviewed were within the scope of the 
QMA, then it would also be consistent with the reconciliation rate range. 
With respect to AA’s 87% reconciliation rate that Mr. Duncan refers to, it is based on all AA reconciliations, including studies with low (less than 80%) reconciliation rates. The studies with lower reconciliation rates tended to have lower error rates, so by excluding these studies, we have reported a higher error rate. NORC did not include studies with low reconciliation rates because this means the reconciliations were mostly incomplete and the results could not be viewed as conclusive. The fact that he may be able to find a single reconciliation report that appears to tell a different story does not mean that the values NORC presents in its report are misrepresentative. 
The Meta-Analysis Report "Study Summary Template" also includes a section titled "Analysis". In this section, NORC concludes that the sample report falls into the "zero (0%) net dollar amount in error"4 category. EconLit was not provided with the 294 AA tribal trust reconciliation reports that formed the basis for NORC’s opinions. However, based on the AA tribal trust reconciliation report EconLit did have to review, the conclusion of "zero net dollar" error is simply wrong. 
Again, Mr. Duncan is taking the result of the example AA reconciliation shown in the 
Study Summary Template illustration, and assuming that it represents all of the studies reviewed in the QMA. This is not true, and it was never suggested by NORC in the QMA report. The fact that the AA tribal trust reconciliation report that Mr. Duncan reviewed does not show the same results as the Study Summary Template example does no____~t mean that NORC’s results are--as Mr. Duncan is suggesting--starkly inconsistent with underlying studies. Once again, the nature of the data is that variation can arise from one observation to the next. Statistical science is basically the study of this variation. 
AA, in fact, noted substantial errors in the reconciliation effort. For example, in conducting the investment yield analysis, AA identified 9 categories for errors that fell outside of the "error tolerance range" (e.g. insufficient information to complete the work on the appropriation, interest posted to the wrong account, and no interest was credited to the account.) 
AA did many different tests, including a Basic Reconciliation of non-investment transactions and the investment yield analysis referred to by Mr. Duncan. Because other non-AA studies included in the QMA were restricted to the Basic Reconciliation of non-investment transactions, we only included this testing from the AA studies so that comparisons across all studies could be made. 
Another example of the substantial errors is in the receipts and disbursement reconciliation which identified 4 categories of errors (e.g. an error occurred that resulted in a misstatement of the account and a correction was recommended, error occurred that DOI claimed was later corrected in the account, errors less than $1,000 each). It is incomprehensible that NORC couM categorize the AA studies as "zero (0%) net dollar amount in error" in light of AA defining an error tolerance range and then specifically identifying transactions that fell outside of that range.5 
In calculating the dollars in error, NORC did not apply the tolerance range and included errors less than $1,000 in the error rate calculations. 
Mr. Duncan’s footnote 5 refers to the fallacy of a net error rate. Dr. Hinkins’ report addresses this issue in more detail for the 2007 Plan. With respect to tribal trust accounts, calculating a net underpayment rate is appropriate. For example, if a tribe has several accounts, but money is incorrectly deposited into account A when it should have been deposited into account B, there is no underpayment to the tribe as a whole. While there is a transaction error for the tribe, the net dollar difference to the individual tribe is zero. 
Duncan 1.2.4 
As previously discussed, the overall analysis that the Meta-Analysis purportedly supports is irrelevant to providing accurate and complete accounting to the 11M account holders. The 2007 Plan will only be able to reach a conclusion regarding the average misstatement of transacaons. lgnoring the myriad of other deficiencies in the 2007 Plan, statistical results regarding an average misstatement of transactions do not provide insight into how accounts wouM potentially be misstated and individual accounts have been affected Furthermore, the average misstatement offsets mistakes in one account with mistakes in another. Again, this is irrelevant to individual accounts. 
The QMA report focused on making a statement about the Trust System by using previous reports. It was not limited to IIM accounts only. Mr. Duncan is extending incorrect arguments from his main report onto the QMA. My colleague Dr. Hinkins, in her expert report, provides the background and reasons why Mr. Duncan’s statements are incorrect in this matter. 
Duncan 1.3 
Recall that NORC’s overall conclusion from the Meta-Analysis Report found "no evidence of widespread fraud" and "considerable evidence that systems performed far better than generally recognized". The overall conclusion is unfounded 
The QMA, to put matters more technically, did not find any mention of fraud, except in one minor instance it was suggested. 15 Once again, we merely summarized facts found in the reports. We felt comfortable with the statements that there was no evidence of widespread fraud. 
15 The Griffin and Associates, P.C. 1997 Audit noted an instance where an "accounting technician made unauthorized changes to an Integrated Records Management System ("IRMS") account holder master record." 
The Meta-Analysis Report did not perform a fraud investigation nor did NORC reference even one reviewed study that purported to be a fraud investigation. Public sector auditing has specific procedures and policies relating to conducting a fraud audit. NORC does not purport to be a professional accounting firm, auditors or accountants. NORC is not qualified to opine as to the findings of an audit or a fraud audit. 
None of the reviewed studies were undertaken to study fraud in the Indian Trust system. One study included a note about improprieties that took place. While I agree that the studies were not investigating fraud, NORC assumed that if improprieties that would suggest fraud were widespread in the system then the incidents would have been mentioned in many of the reports if they were observed by the organization conducting the study. Based on this assumption, I believe NORC was within its authority to conclude that no widespread fraud was found. 
Studies do exist that specifically sought to identify fi’audulent acth, ity related to lndian Trust Funds. For example, in 1989, the Senate Select Committee on lndian Affairs was asked to investigate fraud, corruption, and mismanagement related to lndian Trust Funds. The Senate Select Committee report concluded that, "We indeed found fraud, corruption, and mismanagement prevailing the institutions that are supposed to serve American lndians." Therefore, NORC’s conclusion regarding no evidence of underlying fraud is unfounded and meaningless. 
The "study" that Mr. Duncan refers to would not have been included in the QMA because it is not an audit or reconciliation of the Indian Trust system. Mr. Duncan has taken wide latitude in describing the Senate Select Committee’s report as one that is specific to the Indian Trust. Based on my reading of the report, the Senate Select Committee was tasked "...to uncover fraud, corruption and mismanagement in American Indian affairs, no matter where or to whom it led." Thus, the Senate Committee’s study was very broadly focused, not specific to any one area of the government institutions that are supposed to serve American Indians. That is not to say that the Indian Trust is not mentioned in the report.16 It is just not an audit or reconciliation study that specifically focuses on account transactions. Therefore, it was out-of-scope for the QMA. 
The Meta-Analysis Report’s finding that the overall performance of the "systems" is "far better" than generally recognized is baseless and meaningless. The overall conclusions are nothing more than feel good statements (predominately based on the tribal trust system) designed to support NORC’s predetermined conchtsions regarding the Individual lndian Trust system. 
Previous responses to Mr. Duncan’s statements in Section 1.3 of his report provide my reasoning as to why NORC’s conclusions are neither baseless nor meaningless. As I have already mentioned, NORC approach was, and will always be, to let the data speak for themselves. Mr. Duncan has no basis for his accusation. 
~6 Final Report and Legislative Recommendations; A New Federalism for American Indians (Select Committee on Indian Affairs, 1989), p. 10. 

Duncan 1.4 
It is noteworthy that the vast majority of identified studies relate to the tribal trust system. All of the 294 identified AA reconciliation reports address the tribal trust system. 


Yes, to reiterate, the QMA report focused on making a statement about the Trust System by using several previous reports. It did not only focus on IIM accounts. 
Amazingly, the most directly relevant AA reconciliation report was not even identified by NORC. 7 In March 1992, AM prepared a report titled, "Summary Report of lndividual Indian Monies (IIM) Reconciliation Work Performed at Agency Offices - Phase I of Contract No. CMK 00129391 ­Reconciliation of the Bureau of lndian Affairs’ Trust Accounts." ("AM IIM Report"). 
The original scope of work for the AM IIM Report was "all of the IIM accounts at the three Agencies indicated will need to be reconciled from inception to the most current month-end at the time the reconciliations are being performed ,,s Reconciled was defined as "reconciled as accurately as possible". 9 However, AM identified obstacles to a "complete reconciliation" (emphasis in original) such as unavailable accountholder statements, inconsistencies in land ownership and leasehoM interest records, and extensive time requirements to create a database of transactions and perform the reconciliation work (over a time period truncated to only 5 years). 

The report referenced by Mr. Duncan does not itself provide reconciliation results, so it 
would nothave been included in the QMA. The purpose of the AA report is to give anoverview of the work that was done and to summarize the difficulties in carrying out a complete reconciliation of IIM funds. The reconciliation work done by AA was used primarily to "determine the feasibility of complete account reconciliations.’’~7 It does reference the IIM reconciliation work done by AA in three individual agencies; reconciliation results of that work are not given in the AA Summary Report. I was indeed aware of the IIM reconciliation work done by AA here, but NORC was never able to obtain the results of that work~8. If obtained, this work would have constituted three additional reconciliation studies in NORC’s final QMA report.19 However, it is my understanding that reconciliation work in these three agencies was halted before meaningful results could be obtained. Therefore, the reconciliation work would likely have been categorized as out-of-scope. 
NORC is not hiding the results of this reconciliation work, as seems to be implied by Mr. Duncan. The reconciliation results were simply not available. The internal control weaknesses noted by AA in this Summary Report are also included in the AA audits. These audits were a part of the QMA work. 
In this section of the report, Mr. Duncan goes on to quote some observations and recommendations from the AA IIM Report. I agree with these AA statements, and I believe that some of AA’s recommendations have been addressed. As mentioned in my 
17 Summary Report of Individual Indian Monies (IIM) Reconciliation Work Performed at Agency Offices- Phase I: Reconciliation of BIA ’s Trust Accounts (Arthur Andersen, 1992), Page 4
18 It is not even known if results of this AA work exist outside its work papers. 19 Incidentally, NORC never said the QMA was complete, although I think it is nearly so. 
first expert report, NORC undertook a thorough review of legal ownership records, which 
included the Land Title and Record Offices’ (LTRO’s) series of 10 studies. Other examples of how AA’s observations and recommendations are being taken seriously are the LSA Project and NORC’s recommendations that went into the 2007 Plan. 
What is puzzling is why Mr. Duncan cites the AA observations and recommendations. NORC does not recommend that reconciliations be undertaken in the absence of source documents. Reconciliation was certainly difficult in the LSA project but, and this would be best left to OHTA’s accounting contractors, in virtually all cases it was accomplished in accordance to the Accounting Standards Manual. 
The Meta-Analysis Report either intentionally excluded or inadvertently ignored the most directly relevant reconciliation report. The conclusions of the AA 11M Report completely undermine the conclusion and inferences from the Meta-Analysis Report. 
This is yet another statement of the same point made by Mr. Duncan and addressed by me earlier. NORC tried at every point to include as many studies as possible into the QMA Report in order to provide a more accurate overall statement about the Trust System. We have not fully considered how we would change the QMA Report beyond including the AA transmittal as yet another Report. We do not think it would change the QMA report’s findings, as the individually available tribal studies are all included already. 
Duncan 1.5 
In this section of Mr. Duncan’s meta-analysis report, he reiterates many of the claims and accusations that he has stated throughout his report. I have already responded to all of these, so that is little value in me repeating my answers in a point-by-point fashion. Instead, I will provide an overall summary of my expert rebuttal report. 

Rebuttal Report Summary 
NORC’s conclusions in the QMA report were based on the aggregate summary of Indian Trust audit and reconciliation reports reviewed. On aggregate, the reports reviewed do not point to a non-performing system. NORC did not itself carry out any audits or reconciliations but based its conclusion on results from the many studies it reviewed. The QMA considers results as presented in reports without redoing the study. Key points that I would like to make regarding Mr. Duncan’s claims and accusations are: 
The QMA was carefully performed and provides ample support for its rather 
limited conclusions. Mr. Duncan has simply assumed a role for the QMA that it 
was never intended to have. The main issue became one of planning. In 
particular, whether NORC should recommend an approach that was based on the 
results of reconciliations or the risk identified in the audit studies. 
14 
The QMA work is part of an overall, thorough set of planning activities,s° no one of which alone was relied upon. That each was basically consistent was sufficient for NORC to make the sample design recommendations it has for the 2007 Plan. 
Mr. Duncan has assumed that illustrative examples of NORC’s approach to categorizing data represent the study’s conclusions. The QMA report never says this. Mr. Duncan is making arguments based on a comparison analysis between two observations out of the many hundreds in the QMA analysis. This is not sound professional reasoning. 
NORC, having temporarily augmented its staff with an accountant for the purpose of guiding the team on the technical aspects of audits and reconciliations, was qualified to catalogue the accounting opinions in the audit studies. 
The fact that the AA IIM Report was not in the QMA does not make the QMA useless for the purpose that it was intended. The QMA was conducted primarily for planning and hypothesis setting purposes. Even if the missed report were included in the QMA, we would still recommend the approach to the 2007 Plan that is included in my first expert report. Namely, the Paper Era Ledger testing should begin with a relative small sample size, and adapt the sample size should an unexpected error rate for IIM accounts be realized. 
One further note, the most curious aspect of Mr. Duncan’s report is that he wants to insist that NORC’s QMA is a central fixture in DOI’s 2007 Plan. In the main body of his expert report, he states: 
The Meta-Analysis performed by NORC plays a central role in purportedly strengthening and corroborating DOI’s conclusions, including the results for the statistical sampling allegedly completed (i.e., Electronic Ledger Era), as well as currently contemplated (i.e., Paper Ledger Era). ~1 
I do not understand how Mr. Duncan can draw this conclusion. The QMA only gets a brief mention in the 2007 Plan in Section III entitled "Lessons Learn To Date." How one can go from the short meta-analysis paragraph in the 2007 Plan to concluding that the QMA is central to the plan is incomprehensible. 
To be clear, the QMA does not play a central role in the 2007 Plan. The QMA report conclusions do no____~t imply that testing of the Paper Ledger Era is unnecessary. Instead, it supports the position in the 2007 Plan that a large sample is not likely needed for the Paper Ledger Era. 
so These include the LSA experience, the audits done by Treasury and GAO prior to 1951, and of course, 
the QMA Reviews themselves. 
~ Expert Report Of Dwight Y. Duncan, CFA, p. 29 of 79. 

15 
Respectfully submitted, 
PhD