0950 1 UNITED STATES DISTRICT COURT FOR THE DISTRICT OF COLUMBIA 2 3 ELOUISE PEPION COBELL, : Civil Action 96-1285 et al. : 4 Plaintiffs : : Washington, D.C. 5 V. : Wednesday, October 17, 2007 : 6 DIRK KEMPTHORNE, Secretary : of the Interior, et al. : 7 : Defendants : MORNING SESSION 8 TRANSCRIPT OF EVIDENTIARY HEARING 9 DAY 5 BEFORE THE HONORABLE JAMES ROBERTSON 10 UNITED STATES DISTRICT JUDGE 11 APPEARANCES: 12 For the Plaintiffs: DENNIS GINGOLD, ESQUIRE LAW OFFICES OF DENNIS GINGOLD 13 607 14th Street, NW Ninth Floor 14 Washington, DC 20005 (202) 824-1448 15 ELLIOTT H. LEVITAS, ESQUIRE 16 WILLIAM E. DORRIS, ESQUIRE KILPATRICK STOCKTON, L.L.P. 17 1100 Peachtree Street Suite 2800 18 Atlanta, Georgia 30309-4530 (404) 815-6450 19 KEITH HARPER, ESQUIRE 20 JUSTIN GUILDER, ESQUIRE KILPATRICK STOCKTON, L.L.P. 21 607 14th Street, N.W. Suite 900 22 Washington, D.C. 20005 (202) 585-0053 23 DAVID C. SMITH, ESQUIRE 24 KILPATRICK STOCKTON, L.L.P. 1001 West Fourth Street 25 Winston-Salem, North Carolina 27101 (336) 607-7392 0951 1 For the Defendants: ROBERT E. KIRSCHMAN, JR., ESQUIRE JOHN WARSHAWSKY, ESQUIRE 2 MICHAEL QUINN, ESQUIRE J. CHRISTOPHER KOHN, ESQUIRE 3 U.S. Department of Justice 1100 L Street, N.W. 4 Washington, D.C. 20005 (202) 307-0010 5 JOHN STEMPLEWICZ, ESQUIRE 6 Senior Trial Attorney 7 U.S. Department of Justice 8 Commercial Litigation Branch 9 Civil Division 10 Ben Franklin Station 11 P.O. Box 975 12 Washington, D.C. 20044 13 (202) 307-1104 14 15 16 Court Reporter: REBECCA STONESTREET 17 Official Court Reporter 18 Room 6511, U.S. Courthouse 19 333 Constitution Avenue, N.W. 20 Washington, D.C. 20001 21 (202) 354-3249 22 23 24 Proceedings reported by machine shorthand, transcript produced 25 by computer-aided transcription. 0952 1 2 3 4 5 6 C O N T E N T S 7 8 WITNESS DIRECT CROSS REDIRECT RECROSS 9 10 DR. FRITZ SCHEUREN 11 By Mr. Warshawsky 960 -- -- -- 12 By Mr. Gingold -- 1026 -- -- 13 14 15 E X H I B I T S 16 17 NUMBER ADMITTED 18 19 20 DEFENSE: 21 22 6 - 7 1026 0953 1 P R O C E E D I N G S 2 THE COURT: Good morning, everybody. Mr. Kirschman? 3 MR. KIRSCHMAN: Good morning, Your Honor. Before we 4 continue with the examination of our witness, I would like to 5 bring two matters to your attention if I could, please. 6 The first is a scheduling issue related to the 7 presentation of our responsive witnesses, those witnesses who 8 have prepared reports in response to plaintiffs' experts. 9 This is a matter that, until about 9:30 last night, the 10 parties appeared to be in agreement on. But there's a 11 divergence now, and because of the need to clarify the schedule, 12 we thought we should bring this to the Court's attention. 13 We have a total of six responsive reports. Our 14 presentation of most of our responsive witnesses was, we 15 anticipated, going to be after you actually first heard the 16 reports and the testimony they were responding to, so that you 17 would have context for what they were saying, and that they 18 would be responding to what actually was entered into the 19 record. That includes specifically Caren Dunne, 20 Professor Langbein, and Dr. Hinkins. We thought -- and we think 21 that is the orderly way to progress, so that you're not hearing 22 testimony from our witnesses that is speculative and based on 23 what they assume they will hear. 24 There's also a question as to whether, for example, 25 plaintiffs -- certain of plaintiffs' experts will testify, or 0954 1 whether you will favorably entertain motions we have regarding 2 their expertise. And there may be things that our witnesses 3 would testify that plaintiffs would end up not raising once they 4 call their witnesses. 5 So for the efficient, orderly presentation of the 6 issues being addressed, we had assumed that our responsive 7 witnesses would proceed after you heard what was first said by 8 plaintiffs. 9 As of yesterday, plaintiffs in fact had told us 10 regarding Dr. Scheuren, they would oppose the presentation of 11 our responsive witnesses in our case-in-chief. The issue had 12 come up because we had suggested that Dr. Scheuren should only 13 appear once. The issue he's responding to, the meta-analysis, 14 is a fairly narrow issue, and we had suggested that he just 15 address that while he's on the stand today. 16 So that was how the issue surfaced. And we seemed to 17 be in agreement that, for the other responsive witnesses, they 18 would only testify after plaintiffs had a chance to present 19 their case. That changed, I got a phone call last night about 20 9:30. 21 But obviously the parties need to know -- we feel 22 strongly that it only makes sense that our witnesses actually 23 get to respond to something you've heard. But resources will 24 have to be diverted if there's now something different afoot. 25 THE COURT: All right. Well, like all trial questions, 0955 1 I find that a little bit abstract. Maybe I can hear from the 2 plaintiffs, and hear how that relates to individual witnesses. 3 MR. KIRSCHMAN: Your Honor, could I give you one 4 example, to be more concrete? And certainly Mr. Harper can add 5 his. 6 THE COURT: Okay. 7 MR. KIRSCHMAN: One of plaintiffs' expert witnesses, 8 for example, is Dwight Duncan, who has challenged the 9 statistical sampling employed by the Department of the Interior 10 through NORC. In their pretrial statement, plaintiffs indicated 11 that Dwight Duncan would testify for up to 12 hours on direct. 12 THE COURT: And I said nobody is going to testify for 13 12 hours. 14 MR. KIRSCHMAN: We anticipate that's not going to 15 happen. Nevertheless, we also anticipated that there would be a 16 fair amount of testimony coming from Dwight Duncan on these 17 issues. And Susan Hinkins, Dr. Susan Hinkins from NORC, has 18 prepared a detailed response to Mr. Duncan's detailed report. 19 So for you to hear Dr. Hinkins give a detailed response 20 before -- 21 THE COURT: I hear that. I hear that. Let me hear 22 what the plaintiffs have to say about it. 23 MR. DORRIS: Your Honor, good morning. 24 THE COURT: Good morning. 25 MR. DORRIS: You'll recall some of this came up where 0956 1 there was a question about whether these experts were responding 2 experts or rebuttal experts. And there was prior argument, and 3 the government took the position that they were responsive 4 experts and not rebuttal experts. 5 What Mr. Kirschman says, the issue had not been 6 discussed between the two parties as to whether or not these 7 other experts should be testifying or not. We called him back 8 at 9:30 last night after he had asked us the question in an 9 earlier call in the evening. That's the reason why that call 10 was made, where the parties first discussed this overall issue. 11 Now, with respect to their experts. For example 12 Mr. Scheuren, I'm convinced, today is going to be talking about 13 Mr. Duncan and what he says. That happens in every case, where 14 an expert that's called in the whoever-goes-first case is 15 talking about their opinion and why it should be accepted as 16 opposed to the other expert's opinion. 17 And that's exactly what should happen here. They 18 should be put in a position to put on their entire case. If 19 that includes experts that they want to call that are going to 20 give testimony that is different than our experts, that's what 21 they ought to do in moving forward. That's our position, 22 Your Honor. 23 THE COURT: Well, I don't know how this relates to 24 other witnesses, but as it relates to Duncan and Hinkins, 25 accepting the representation of Mr. Kirschman as accurate and in 0957 1 good faith that Hinkins' testimony is nothing but a response to 2 Duncan, then Duncan goes first. Okay? 3 Now, I don't know how that rule applies to other 4 witnesses, but that's my ruling on that one. 5 MR. KIRSCHMAN: Your Honor, would you like me to raise 6 the other two witnesses where this most aptly applies? 7 THE COURT: Might as well. Let's hear about them. 8 MR. KIRSCHMAN: Caren Dunne, Your Honor, is a CPA with 9 Clifton Gunderson. By the way, all of these witnesses that I'm 10 mentioning we had identified as may-call, largely because of 11 this, depending on what we hear when plaintiffs put on their 12 case. 13 But Caren Dunne was listed as a may-call, and she will 14 be responding to the report prepared by Don Pallais, 15 P-A-L-L-A-I-S, regarding the Accounting Standards Manual and 16 issues related to that. Her report is detailed and tracks his 17 opinions -- he issues an opinion; she responds in kind, related 18 to the Accounting Standards Manual. So to hear her response to 19 Mr. Pallais' positions on the use of the ASM just again wouldn't 20 be logical and wouldn't be efficient. 21 Plaintiffs had suggested if that one of our witnesses 22 like Ms. Dunne heard something new, we could call her back a 23 second time. And certainly, that wouldn't make sense. So 24 Caren Dunne is another one specifically. 25 A third is Professor Langbein, who has offered a 0958 1 responsive expert report on the standards governing Trust 2 accounting. Professor Langbein is responding to Robert Vaughn, 3 who provided a report for plaintiffs on the standards of Trust 4 accounting. 5 This is particularly notable because Mr. Vaughn cited 6 his experience as the basis for his opinions on the standards. 7 We don't know what those experiences are, and until we actually 8 hear him testify, Professor Langbein will be limited in how he 9 can respond to that. 10 THE COURT: Okay. Counsel, I don't want to spend the 11 morning on this. I want to think about it a little bit. We are 12 in some sense, all of us, making this up as we go along, because 13 I don't know that there's ever been a trial quite like this one 14 before. 15 But as I see it, in the broadest outline, the reason 16 the government is going first -- even though it is ultimately 17 the plaintiffs who have the burden of proof in this case on most 18 concepts, the government is going first because the government's 19 case is: Here's what we're doing. The plaintiffs are going to 20 say: That's not good enough. And the government is going to 21 respond with its responsive witnesses. 22 That suggests to me that the ordinary and logical 23 course of this trial is for the government to put on the rest of 24 its here's-what-we're-doing witnesses; for the plaintiffs to 25 then attack it, as they will be doing with expert witnesses; and 0959 1 for the government then to respond with their experts. 2 So it sounds to me -- but I'm not ruling on this, I 3 just want to think about it. It sounds to me like Dunne and 4 Langbein come after the witnesses to whom they're responding. 5 But let's think about it, and we may talk about it a little bit 6 more. We don't have to decide that this morning. I want to get 7 a witness on the stand and get moving. 8 MR. KIRSCHMAN: Your Honor, I'm sorry. I have one more 9 matter to bring to your attention. 10 THE COURT: Quickly. 11 MR. KIRSCHMAN: It will be quick, Your Honor. We're 12 moving orally for leave to present a new defendant's exhibit. 13 It is marked as DX-365. It is a demonstrative exhibit that has 14 been prepared by Dr. Abe Haspel, with the assistance of 15 contractors. The document was just completed late yesterday, 16 early evening. It is a demonstrative that will assist the Court 17 in visualizing what Dr. Haspel will be testifying to regarding 18 the relationship between throughput and its different 19 variations, and the historical accounting work. 20 There is information in this document that is already 21 contained in other documents, other defendants' trial 22 exhibits -- 23 THE COURT: When is Haspel going to testify? 24 MR. KIRSCHMAN: The order now, Your Honor, that we have 25 provided to the plaintiffs is that Dr. Scheuren will testify 0960 1 today, he will be followed by Dr. Lasater, and then Dr. Haspel 2 will testify. 3 We anticipate therefore he will probably be testifying 4 tomorrow morning. That's a rough approximation, depending on 5 the length of the direct and the cross of the two witnesses who 6 precede him. 7 THE COURT: All right. Well, he can use the 8 demonstrative. Whether it comes into evidence remains to be 9 seen, but he can use the demonstrative. 10 MR. KIRSCHMAN: Thank you, Your Honor: 11 THE COURT: Let's get Dr. Scheuren on the stand, 12 please. 13 MR. WARSHAWSKY: Good morning, Your Honor. 14 THE COURT: Good morning, Mr. Warshawsky. 15 CONTINUED DIRECT EXAMINATION 16 BY MR. WARSHAWSKY: 17 Q. Good morning, Dr. Scheuren. 18 A. Hi. 19 Q. Dr. Scheuren, I went to move into the specific facts 20 underlying your various recommendations in this matter. But one 21 last point I do want to tie up very quickly. 22 When we were talking yesterday about precision and 23 confidence, or margin of error or assurance levels, whatever, 24 and we talked about sample size, is there a linear relationship 25 between sample size and precision or confidence; in other words, 0961 1 twice as big a sample means twice as much confidence, twice as 2 precise? 3 A. Unfortunately for clients who have to spend the money, it's 4 not linear. It's quadratic. If you want to narrow the 5 confidence interval, say shorten it by half, you have to have a 6 sample that's four times as large. 7 Q. And at some point, does increasing the size simply not make 8 a difference? 9 A. In the context of the client's needs for precision, I would 10 say yes, it does. 11 Q. Referring you again to your expert report, Defendant's 12 Exhibit 6 we have up on the screen here table one on page five. 13 Can you just generally explain what is presented in this table? 14 A. This is a table that looks at sample size. If you have a 15 world, hypothesized world of a one percent error rate, and you 16 want a margin of error around that, very narrow margin of error 17 around that -- excuse me. It's assuming no errors in the 18 observations, and a margin of error of one percent. 19 And there are three columns. The first column is 20 population size, and the second column is a 95 percent 21 confidence interval, and 99 percent confidence interval. These 22 are the ones that we've been talking about yesterday. 23 And what you're seeing in the table is, if you have -- 24 if you are at a population size of 500, the sample size that you 25 need to achieve this is 224, which is a very large fraction of 0962 1 that total population. 2 As you move up the table, or actually down the table, 3 the sample size grows, but very slowly. Very slowly. And the 4 fraction that it becomes of the population drops dramatically. 5 That's the point of the table. 6 Q. And in fact, for a population 100,000 to 10 million, there's 7 no change in the sample size required. Is that correct? 8 A. That's correct. Rounds to that, right. 9 Q. Dr. Scheuren, I would like you to refer now to what's been 10 marked, it's in the administrative record, AR-402. The Bates 11 numbers on that is 00049-6. I have the cover page on the 12 screen. 13 Now, Dr. Scheuren, would you identify what this 14 document is, please? 15 A. This is a report that we produced for the Office of 16 Historical Trust Accounting on our sample design plans for the 17 2003. This would have been the earlier accounting plan. 18 Q. You're referring to the plan that was submitted to the 19 District Court in 2003? 20 A. Correct. 21 Q. And you were one of the authors of this report? 22 A. Yes, I was. 23 Q. Referring now to page three of this document, which is on 24 the screen -- yes? 25 A. I need to have the point size come up a little bit. That's 0963 1 why I had a problem yesterday. 2 Q. And as I said, Dr. Scheuren, we don't have your notebook of 3 hard copies in front of you. If that would assist you, we can 4 give you that as well. 5 A. Let's do this point size. 6 Q. Okay. Is that big enough? 7 A. Absolutely. 8 Q. Dr. Scheuren, there's a section captioned "General 9 Approach." What was the purpose of including this in your 10 report? 11 A. Well, people like me get in the weeds a little too much. 12 Maybe I did that yesterday, even. This is something that was to 13 give the senior people in the government access to what our 14 general ideas were. 15 Q. And the first two bullet points on the screen here propose 16 sampling transactions under 5,000, and propose that all 17 transactions over 5,000 be directly reconciled. Is that 18 stratification? 19 A. Yes, that's the -- 5,000 is a boundary, a stratification 20 boundary. 21 Q. Why did you propose different strata for your sampling 22 model? 23 A. Well, we had concerns here at the beginning of this about 24 whether we were going to be able to find all the records. And 25 the larger amounts - okay, 5,000 is a reasonably large amount - 0964 1 cause considerable concerns if you can't find the records. 2 Q. Why is that? 3 A. The reason for that is, based on my background, when I was 4 at the IRS, when you have a situation where you have 5 controversies of the kind that we are seeing here, if you can't 6 find the document, you can't -- if you can't support the 7 document, then it's treated as an error. 8 And indeed, that is what we have done in our work here. 9 When we couldn't find the document, when the team of people that 10 did the work, including us, were unable to find the documents in 11 the litigation support work that we did, all of those were 12 treated as errors. And that was a concern. 13 Q. And you're referring to the statistical analysis of those 14 items? 15 A. That's correct. 16 Q. In fact, if you couldn't find support, does that mean that 17 the transaction was erroneous? 18 A. No. 19 Q. Why? 20 A. But there is no evidence that it is accurate. And that 21 worst case approach was what I've seen done in other settings; 22 I've mentioned the IRS, and it's what we did here. 23 Q. Let me ask you something. Yesterday, I'm not sure if you 24 were in the courtroom when Judge Robertson asked this, when 25 Ms. Herman was testifying about the reconciliation efforts for a 0965 1 $0.58 item, and the estimate was that it cost several thousand 2 dollars to reconcile that item, why would you go to the effort 3 of reconciling a $0.58 item? 4 A. Well, you have to believe that the Office of Historical 5 Trust Accounting was not too pleased with that, when they came 6 back that it cost so much money. 7 The reason we did this is because, how do we know it 8 was $0.58? What if it was $50,000 but had been mis-keyed as 9 $0.50? 10 So we did look at the very small items, and an awful 11 lot of very small transactions, especially credits, and we 12 looked at them to see if in fact they were understated. 13 We almost found none of them understated. I think 14 there were two that were slightly understated. Maybe they were 15 $0.50 above or something like that, a dollar above. If I may 16 continue with this, since it is relevant? 17 Q. Please. 18 A. When we Went back after we did the first round of the 19 litigation support work, we cut the sample size for the very 20 small items because we weren't learning anything. We didn't cut 21 it to zero, because there was a chance the ones we hadn't found 22 were different than the ones we had. But we didn't cut it to 23 zero, but we cut it down a way lot, to save some of this money. 24 I mentioned yesterday that one of the steps, this 25 four-step process, is the redesign step: You plan it, you carry 0966 1 it out, you make inferences from it, and then you look at what 2 you've done and you redesign it. 3 And that's what we did here, and we reduced the work 4 and reduced the money. We weren't learning anything. We 5 weren't sure we were right, but we were sure enough that we 6 could cut the sample size. And we did that dramatically. 7 Q. But how does a $0.58 item get selected in the first place? 8 A. Because it's part of the population. Every item in the 9 population, every transaction in the population that we were 10 given, had a positive probability of being selected. 11 Q. Now, in your second bullet point on the screen, you have a 12 sample statement in quotes, "With 99 percent confidence, more 13 than 99 percent of all transactions up to $5,000 are accurate." 14 What does that mean, that kind of a statement? 15 A. Well, it is a decision -- the client makes this decision, 16 but we offered it to them, that they want to be that accurate. 17 It's a very important case. It's appropriate, to give respect 18 to all of the parties, that we have a high degree of accuracy. 19 And this was what we proposed. And -- 20 Q. Go ahead. I'm sorry. I didn't mean to cut you off. 21 A. Remember our confidence interval idea of yesterday. Okay? 22 We have here an example of the application. 99 percent 23 confidence, that's a confidence interval. In fact, it's a 24 one-sided confidence interval, because we're only looking at one 25 kind of issue here. 0967 1 And what this means is that the error rate -- to put it 2 in a negative way, the error rate is one percent or less. 3 Q. What do you mean by one-sided confidence level? 4 A. Well, when I drew the picture yesterday, I had a possibility 5 of being -- of making a mistake on both sides. That possibility 6 to make a mistake on both sides always exists, of course, if one 7 of the sides isn't zero, but the interest here in view of the 8 case is really a one-sided interest. 9 You heard me mention that the way we're dealing with 10 errors is to look -- I guess we haven't talked about this yet. 11 We're looking at the issues in terms of the way the plaintiffs 12 would look at them. Mistakes are made in all directions, in 13 large activities. This is normal human behavior. There are 14 only certain kinds of errors are really of concern to the 15 plaintiffs; those in fact which the plaintiffs are 16 disadvantaged. And so that's what we were focusing on. 17 Q. Is 99 percent -- let me try that again. 18 99 percent confidence level, is that standard for this 19 type of analysis? 20 A. Actually, it's extreme. 21 Q. In what way? 22 A. In most business settings, I do a lot of work in business 23 settings, a 95 percent standard would have been used. But 24 because of the nature of this case, the importance of it, I 25 recommended a 99 percent, which means that the sample sizes are 0968 1 considerably larger as a result of that. 2 Q. And why don't we go ahead and refer to page six now of this 3 exhibit, AR-402. There's a table at the top which I have on the 4 screen. Would you explain what was depicted in that table 5 within your report? 6 A. This is a table, again with three columns. One of them is a 7 column which -- the first column which looks at different 8 classifications, different sizes of transactions; the second one 9 are the total number of such transactions in the population that 10 we were sampling; and the third is the sample size that at this 11 time we recommended, very large sample size. 12 Q. Okay. And then referring back to -- we'll leave the table 13 up here on the screen. But earlier when we were talking about 14 your bullet points on the first page, you talked about a 15 certainty stratum. I think that's sometimes used as a term. Is 16 that correct? 17 A. Or take-all. If you look at the 5,000-plus row here, you 18 see that the number of transactions in the population, which is 19 in the second column, and the number in the third column, are 20 the same. We were taking them all. 21 Q. That's not a sample, is it? 22 A. I would argue it's a sample, but you could argue it's not a 23 sample. 24 Q. Well, you're the expert. 25 In any event -- 0969 1 A. It's a 100 percent sample. 2 Q. Very good, sir. For the other two strata, though, under 500 3 and 500 to 5,000, you were proposing sampling for those. Right? 4 A. That's correct. 5 Q. And a total of 160,000 from those two strata? 6 A. That's correct, as well. 7 Q. And just to be clear, that was in the electronic ledger era? 8 A. Yes. 9 Q. Is that a large sample size for that population? 10 A. Yes, extremely large. 11 Q. Why did you recommend such a large sample size? 12 A. We had carefully examined what people thought was wrong with 13 the system. The possibility of local problems in some agencies 14 led us to have -- ask for a reasonable sample size at the agency 15 level in order to detect that, and, if it was there, to 16 estimate, estimate that error rate with some precision. 17 Q. Well, elaborate on that a little bit. What kinds of things 18 were you looking at when you said you considered what people 19 thought was wrong with the system? 20 A. Well, there was information that we had seen - and a lot of 21 it is conversational so I don't have documentation for this - 22 that particular problems existed in particular agencies, okay, 23 in some various points during the historical period that we're 24 looking at here. We're looking at from '85 to 2000. And that 25 led us to want to seek to know that well enough to be able to 0970 1 detect it. 2 And indeed, this isn't the complete sample size here. 3 We're recommending an adaptive approach. And in that world, we 4 were recommending adapting up in those situations where there 5 was some softness in the data, there were some problems with the 6 work that had been done historically, that we would actually 7 increase the sample size even more. 8 Q. Well, what kinds of problems were you talking about, though? 9 A. We're talking about mistakes in the accounting. That's the 10 focus of this engagement. 11 Q. Did you consider the potential for missing data? 12 A. Indeed, that's one of the factors here. That dominated our 13 concern about the 5,000-plus, the 100 percent strata. The 14 missing data problem also was a concern everywhere. How we 15 would handle the missing data problem was something we hadn't 16 fully faced here at this point. But we were expecting to 17 encounter a lot of it, and that was why we had such large 18 samples, one of the reasons we had such large samples. 19 Q. So yesterday when we talked about provisional sample sizes, 20 do you remember that, sir? 21 A. Yes, I do. 22 Q. Is that basically what we're talking about? 23 A. I would consider this a provisional sample size, right. We 24 had not done -- my apology. I interrupted you. 25 Q. No, you didn't. 0971 1 A. I have already apologized to the Court Reporter, and she 2 told me that she would punish me later depending on how often I 3 did it. 4 We had not actually got into the work at this point. 5 We had not been able to do any significant amount of pilot work, 6 for many reasons. So this was very much a provisional sample 7 size. 8 Q. So missing data, problems at the agencies, those were 9 hypotheses? 10 A. Correct. Concerns -- 11 Q. And -- go ahead. 12 A. Concerns about high error rates. Plaintiffs thought the 13 error rates were low -- excuse me. The government thought the 14 error rates were low, the plaintiffs thought they were high. We 15 had to balance those considerations in designing the sample. 16 Q. Did NORC actually draw sample transactions using this sample 17 design? 18 A. We did for Alaska. 19 Q. And describe the process where you drew the sample. 20 A. In the Alaska case, which was unusual in that Alaska was 21 not -- you know, not very active until very late in the history 22 of the Indian Trust, all of the transactions were available in 23 one place in Alaska. And we simply scanned all of them. All of 24 them were scanned. 25 The work was then divided up in equal parts among the 0972 1 accounting firms - there were four accounting firms - in order 2 to see if the accounting firms were doing the work the same way. 3 And the work was never completed because there was a hold on the 4 work. But the design was done. 5 Q. What do you mean by a hold on the work? 6 A. We were not allowed to -- the work could not be finished for 7 reasons that had to do with, I believe the combination of a 8 Court concern, opposed by the Court, and also later on a 9 Congressional concern about funding. 10 Q. I see. What population, by the way, were you designing for 11 on this? In other words, what types of accounts were included 12 in your population? 13 A. We were focusing on land-based accounts here. Judgment and 14 per cap are all being done, at least up to this point, on 15 100 percent basis. A good nature of the population makes that 16 the appropriate thing to do. But these were land-based 17 accounts. 18 Q. And was it every account that ever existed in the IIM Trust? 19 A. Every one that was in the electronic era, called the 20 electronic ledger era, yes. In Alaska. 21 Q. Was that as of -- 22 A. Well, in 1985. Of course, Alaska didn't start until '87, 23 but right. 24 Q. There's been some discussion here -- and just to be clear, 25 the accounts that you sampled from, were they accounts that 0973 1 existed as of October 1994? 2 A. Yeah, they were on or after October 1994. That was the 3 population that the client defined for us to sample from. 4 Q. So just to be clear, if an account ceased to exist prior to 5 October 1994, you didn't sample from it? 6 A. It was not in the population eligible for selection. 7 Q. Now, you mentioned that there was a hold on the 2003 plan 8 sampling. Right? 9 A. Yes, I did. 10 Q. What was the next thing that you did with regard to 11 sampling? 12 A. Well, we did do some analysis of what we had obtained from 13 Alaska, of course. The next important activity was the 14 litigation support activity that we engaged in later that year. 15 Q. And that's sometimes called the LSA project? 16 A. That's correct. The acronyms are... 17 Q. How did you first learn about the LSA project? 18 A. Well, I think Abe Haspel went and talked to Jeff Zippin and 19 Bert Edwards about it. And it was something that we were asked 20 to do, and all of us got involved rather readily. It was in the 21 fall of that year. 22 Q. And just if you would identify who these individuals were at 23 the time. 24 A. Abe Haspel is in the role he has now. 25 Q. And that is? He hasn't testified yet. 0974 1 A. Well, he's an undersecretary, I don't know what quite the 2 title is, at Interior. I think Cason -- he reports to Cason, 3 but he's right at that level, next level down. 4 Q. Got you. 5 A. And of course Bert Edwards is the director of the Office of 6 Historical Trust Accounting, and Jeff Zippin is his deputy. 7 Q. What did you do next with regard to the LSA? 8 A. Well, in October/November we were talking about this, and 9 then it was something that was going to happen right away. 10 Q. And I'm sorry, just to interrupt. What year are you talking 11 about? 12 A. This is, I think 2003. Yes, fall of 2003. 13 Q. Okay. All right, sir. 14 A. So we were asked to draw a sample. We recommended a sample 15 of 1,000 accounts, with an average of 20 transactions per 16 account. That's a pretty large sample also. That was in 17 keeping with our earlier thinking, but it was... 18 Our focus was different now. Our focus was on trying 19 to get an overall understanding of what was happening with the 20 system, not the kind of -- not necessarily the kind of estimates 21 that could be used for an historical accounting. If I might 22 continue. 23 If the results were very favorable, very low error 24 rates, then they would have a role in the historical accounting. 25 If they were what we expected to see, okay, then much more would 0975 1 have been needed. 2 Q. When you say what you expected to see -- 3 A. We expected to find a higher error rate than we found. 4 Q. I would like to have you now look at, from the 5 administrative record, AR-438, which we'll have put up on the 6 screen. The Bates number is 000040-2. 7 Dr. Scheuren, what is on this particular document from 8 the administrative record? 9 A. These are the -- 10 Q. Do you need this blown up again? 11 A. Yes. Not the first page, but when we get to something else. 12 Don't pick on me because I'm older than you are. 13 These are our results, one of the results reports from 14 that effort. 15 Q. Okay. And this is a report dated September 30, 2005? 16 A. That's correct. 17 Q. Who prepared this report? 18 A. Well, I'm one of the authors, I think, and a number of other 19 people were involved. 20 Q. Who are these other individuals? Dr. Hinkins, we've heard 21 about. 22 A. Robin Lee. She's actually here in the courtroom, and 23 she's -- she and Susan and I have been the central people in 24 this engagement. 25 We have different roles, and you'll learn that as time 0976 1 goes on. But when anything is decided, we all end up agreeing. 2 If one of us disagrees, we end up keep working it out until we 3 all agree. That's the basic approach we've taken. 4 I think it's pretty good. Robin is very, very 5 experienced, excellent statistician, ABD. Patrick is also a 6 Ph.D. He's no longer with us; he's gone back to England. 7 Ali Mushtaq has a master's degree in computing, and is an 8 amazingly good statistician. Mike Kwanisai, who is someone I 9 mentioned yesterday, was a student of Steve Thompson. 10 Steve Thompson is the leading expert in the world on 11 adaptive sampling, and we brought Steve in as a consultant to 12 have him take a look at this problem. We thought it was a 13 problem that adaptive sampling could apply to. He not only said 14 yes to that, but he offered us a student who has worked with us 15 since then. 16 Q. I know we talked about it yesterday, just very, very -- one 17 or two sentences. Would you remind us what adaptive sampling 18 is? 19 A. Adaptive sampling is a technique -- 20 THE COURT: I remember. 21 MR. WARSHAWSKY: Okay, very good. That's the only 22 audience that matters. 23 BY MR. WARSHAWSKY: 24 Q. And what was the purpose of this report, sir? 25 THE COURT: Might not remember your name, but I 0977 1 remember what he said about adaptive sampling. 2 MR. WARSHAWSKY: That's all I care about, Your Honor. 3 A. The purpose of this report was to tell the client what the 4 results were, and to present them in a way that we felt 5 comfortable represented the enormous work that had been done by 6 an awful lot of contractors here to get to this point. 7 BY MR. WARSHAWSKY: 8 Q. And this is reporting the LSA results? 9 A. That's correct. 10 Q. Now, let me refer you -- we're going to go to page four of 11 this report, a highlight section again. And let's just focus on 12 the first two bullet points right now. 13 A. Okay. 14 Q. Are you there, Dr. Scheuren? 15 Okay. Would you explain the point that's being made in 16 the second bullet point about reconciling over 99 percent of the 17 items? 18 A. Well, despite our pessimism based on the information we had 19 before we did the 2003 plan, and astonishing based on that 20 expectation, an astonishingly successful outcome occurred, and 21 we found virtually all of the records. And over 99 percent of 22 the randomly selected records were found. 23 Q. And you referenced all 12 BIA regions being sampled? 24 A. All regions, yes. 25 Q. Very good. Let's go to the third bullet. I gather this 0978 1 largely follows the point that you just made? 2 A. That's correct. 3 Q. What did you mean about the reconciliation -- this is the 4 wording here, quote, "The reconciliation should put to rest 5 concerns about the impact that the remaining one percent 6 unreconciled transactions might have on the results," end quote. 7 A. What I meant was that, even had they been wrong, that would 8 not have changed our results. In fact, what we did do was we 9 treated them as if they were wrong in reporting our results. 10 One more moment. 11 That is a model assumption. I think it's a severe 12 model assumption. It's probably they're not as bad as that, but 13 we could not -- the accountants could not support them, and so 14 we treated them as incorrect. 15 Q. Well, let's look at some of the quantification that you put 16 down in these bullet points that follow. The next bullet point, 17 the fourth bullet point, you refer to a debit difference rate. 18 What are debits? 19 A. They are disbursements, things that leave the system. 20 Q. Okay. And would you summarize what you've concluded here? 21 A. Well, the debit difference rate that we found in the data 22 was 0.4 percent, okay, which is very low error rate. Even in a 23 commercial world, that would be arguably a low error rate. 24 We calculate -- because we're talking about a sample, 25 we calculate an assurance rate, assurance level. That's kind of 0979 1 a confidence interval that we talked about, a one-sided 2 confidence interval. And with that 99 percent assurance level, 3 we think we can say with 99 percent assurance that the debit 4 rate in the population is 1.3 percent or less. 5 Q. You also refer then to a rate of differences that are 6 disadvantageous to the account holders -- I'm sorry. This is 7 the last sentence. "Our estimate for the rate of 8 differences" -- you state that, "Our estimate for the rate of 9 differences that are disadvantageous to the account holders, 10 with an assurance level of 99 percent, is 0.7 percent." 11 A. That's correct. 12 Q. What are you referring to when you talk about 13 disadvantageous? 14 A. Mistakes can be made in both directions. From the 15 plaintiffs' side of view, the fact that some plaintiffs received 16 more money, okay, not really of interest. What's of interest is 17 that some received less than they should have received. And so 18 we focused only on the disadvantageous portion of this. The 19 error rate that you see here is 1.3 percent. About half of 20 that, when you round it, is the .6 percent. 21 Later on in this page, and in some supporting work that 22 is not before the Court, we have tested the hypothesis that the 23 system is, quote, "fair," unquote; that mistakes are made in 24 both directions about at the same rate. And this is a rate 25 estimate. These are percentages of transactions in error. We 0980 1 then talk a little bit later about dollars in error, but that 2 comment about fairness applies to both estimates. 3 Q. You may have just misspoken, I think, when you referred to 4 a .6 percent rate. If you put your glasses on. 5 A. Oh, I should have paid -- pointed a sentence up, to 6 the .7 percent for the 99 percent. That's correct. 7 Q. What was the .6 percent that's referenced in your bullet 8 point there? 9 A. If I narrow the confidence interval from a 99 percent to a 10 95 percent, okay, that's the answer I get for that. 11 Q. Now, is this an attribute analysis or a variable analysis? 12 A. This is an attribute analysis. 13 Q. And why is that? 14 A. We're looking -- we're counting mistakes. They happened or 15 they didn't happen. 16 Q. Well, would you contrast that now with the next bullet 17 point? Summarize what you're saying in that bullet point, 18 please. 19 A. Here we're looking at the dollars, the dollars in error for 20 debits, still. The dollars that were disadvantageous to the 21 account holders. 22 Q. What did you conclude -- first of all, is that a variable 23 analysis? 24 A. Yes, it is. 25 Q. What did you conclude with regard to the debit differences? 0981 1 A. The disadvantageous estimates were no more than four 2 million, okay. That is, there was an opportunity to make dollar 3 errors. And in terms of the whole system over this electronic 4 era, we estimated it at a high assurance level, 99 percent, that 5 it's $4 million or less. 6 Q. $4 million from 1985 through 2000? 7 A. That's correct. Given -- I want to emphasize, because there 8 were issues about this that may come up, this is given the 9 population that we were sampling from. It's based on the 10 population that was subject to being sampled. 11 Q. And you also have a conclusion there using this 95 percent 12 level, which you said -- 13 A. That's correct. 14 Q. -- which is more common in business settings. What did you 15 find there? 16 A. That instead of four million, there was two million. 17 Q. Let's move down to the next bullet point. You have a 18 comparable finding with regard to credit differences. What are 19 the credits that you're referring to here? 20 A. These are receipts that go into the accounts, and there are 21 transfers also included. 22 Q. And would you summarize your conclusions with regard to 23 that? 24 A. We found a higher error rate for credits. 1.3 percent was 25 the estimate we had. At the assurance level of 99 percent, 0982 1 we've gone through this now before, we can state that the 2 difference for credits is no more than seven percent. That's 3 not a small number, in my opinion. But still, that's what it 4 is. So it was seven percent. 5 But the disadvantageous -- can I continue with this? 6 Q. Please. 7 A. The disadvantageous fractions is about half of that. Again, 8 we can support that, that the fairness of the system, half of 9 those credit mistakes were in favor of the government and half 10 were in favor of the plaintiffs. 11 Q. Is that the four percent that you're referring to here? 12 A. That's right. 13 Q. And again, you also have a similar finding for 95 percent. 14 Correct? 15 A. We're rounding the percentages, and that's why you're 16 getting these -- they're not exactly half. 17 Q. And in the next bullet point you quantified -- you came up 18 with a variable analysis. Is that correct? 19 A. That is correct. 20 Q. And what did you find for credits? 21 A. Same conditions. 86 million was the -- disadvantageous to 22 the account holders. 23 Q. That was at the 99 percent level? 24 A. That's correct. And at 95, it's 42 million. 25 Q. Finally, the last bullet point, the one that appears below 0983 1 that, I think you may have referred to this earlier, but explain 2 what that bullet point is about. 3 A. It's worth repeating. One of the considerations we had was, 4 to the extent -- and I'll say it again: To the extent to which 5 the system was fair, I mean, that the error rates are low, or 6 lower than expected anyway, was a good thing. But were they all 7 in one direction, or not? 8 And they were not. They were about equally -- 9 statistically, anyway. A lot of uncertainty here. But we could 10 not reject -- I'll say it this way a little more formally: We 11 could not reject the null hypothesis that the underlying 12 processes which generated the errors were the same for both 13 advantageous and disadvantageous. 14 Q. I would like to refer within this document to Appendix A. 15 This begins on page 22 -- 16 THE COURT: Would you accept "even-handedly erroneous" 17 as a substitute for "fair"? 18 THE WITNESS: I would accept that, absolutely. In 19 fact, I think I like it better. I believe we use the word 20 "fair," though. But I like even-handedness. Thank you. Thank 21 you, Your Honor. 22 BY MR. WARSHAWSKY: 23 Q. Right. Refer now to page 22, and we'll blow up the first 24 couple of paragraphs here. What's presented in Appendix A to 25 this document? 0984 1 A. What we're talking about is how we did the sampling here, 2 for all 12 regions. 3 Q. Let's start with the big one here, A.1, the 10-Region group. 4 What does that refer to? 5 A. May I start in a different order? We had done Alaska 6 already. We had had -- we had a good population for Alaska, and 7 Eastern Region had been done as a result of the 120-day report. 8 Q. Okay. Why don't we -- if you prefer to go in that order, 9 let's go first to page 26, then. 10 A. All right. Excuse me. 11 Q. It's okay. Whatever makes it simpler. A.2, is this the 12 Alaska sample that you were talking about? 13 A. That's correct. 14 Q. And describe what you did with the Alaska sample. 15 A. We had already been sampling in Alaska as a result of the 16 2003 plan, and so we drew a sample of that sample and then 17 proceeded to -- it was then passed to the accountants for their 18 work. 19 Q. Now, in drawing your sample, did you directly select 20 transactions for Alaska? 21 A. We went to accounts again. 22 Q. So you started with accounts. And what did you do with the 23 accounts? 24 A. We sampled within the accounts, to choose transactions 25 within the accounts. We favored it -- since we've already 0985 1 selected some transactions randomly, if we had already selected 2 a transaction randomly in the account that we selected randomly, 3 we used that transaction. If we had not selected that 4 transaction in the earlier sampling, we added more transactions 5 to sample. 6 Q. And I believe you indicated you did the Eastern Region 7 before the 10-Region? 8 A. Correct. In the 120-day report -- 9 Q. I'm sorry. Why don't we go first to page 27, then. 10 MR. WARSHAWSKY: Blow that up more for me, even. 11 BY MR. WARSHAWSKY: 12 Q. It indicates OHTA's first reconciliation was completed for 13 the Eastern Region in 2003. 14 A. Correct. 15 Q. So that preceded the Alaska sample? 16 A. Yes. 17 Q. Tell us about the Eastern Region sample, how that was drawn. 18 A. Going back to the 120-day report, the plan was to do them 19 all, to take a census. It wasn't a very big population. That 20 made sense. But it was very difficult to do. And it was one of 21 those things that needed to get done before other things that 22 were bigger needed to be done. 23 And because it wasn't happening, the accounting firm 24 that had the responsibility of doing the 100 percent census, 25 which you don't need a statistician for, came to us. And they 0986 1 said, "Help us out." And we said "Sure. Let's draw a sample. 2 Let's take what you've already done and treat them as 3 100 percent, give them a weight of one, a probability of one, 4 since they were selected with certainty. And then we'll sample 5 the rest for you, and then you'll have to find them all." 6 And so we developed a sample. In fact, my colleague, I 7 believe Robin Lee, who's sitting here, is actually the one who 8 drew this sample. And we gave them the sample to do, and then 9 they went out and did their best. And the results were 10 reasonable, although there were some transactions - I think 11 there were eight - that they did not find. 12 Q. So you started trying to do 100 percent -- 13 A. They did, yes. 14 Q. -- and then drew a sample. Were you still able to perform a 15 statistically valid analysis on the -- 16 A. Because we were keeping track of the probability of 17 selection, we were able to make a inference, an estimate for the 18 Eastern Region. Even if it were done in this sort of "try this 19 way, then try that way" kind of style, fundamentally -- one more 20 moment. 21 Fundamentally, if you're keeping track of how you do 22 the work and you're systematic about it, even though you 23 encounter challenges along the way - and when you're doing 24 something for the first time, you should expect challenges along 25 the way - and you expect a procedure which you can modify and 0987 1 still accomplish your goal. 2 One of the problems in some of the work that had been 3 done prior to our work was that others had encountered problems 4 that they didn't anticipate, and they, quote, "ran out of 5 money," unquote, and -- 6 Q. I'm sorry -- 7 A. -- they were unable to finish. 8 Q. Let me just interrupt a moment. You've talked about 9 Andersen running out of money. 10 A. I am talking about Andersen, yes. 11 Q. What project are you talking about there? 12 A. The work they did on Tribal, and they started for IIM, yes. 13 Q. And what time frame was this, do you know? 14 A. About three or four years before we were working on this, 15 yeah. I've heard the presentations, I've looked at some of 16 their data. I'm afraid the exact dates are not in my head at 17 the moment. 18 Q. But this predates your involvement? 19 A. They do. I want to make the comment that excellent work was 20 done by them, but they didn't finish it. Okay? And that they 21 didn't finish it was really a major problem as a result. 22 Q. Are you familiar with the House committee report called 23 "Misplaced Trust"? 24 A. In general terms. 25 Q. Do you know if that report describes the effort that you're 0988 1 talking about? 2 A. I believe that's probably the case, yes, where it's 3 described. 4 Q. Now, after OHTA did the Eastern Region and the Alaska 5 Region, you moved on to the 10-Region? 6 A. We did the 10 Regions, yes. 7 Q. Why don't we pull up page 22 now. And there were a total of 8 12 regions. Is that correct? 9 A. Correct. 10 Q. So this is the rest of the regions? 11 A. Remaining 10. 12 Q. And in the second sentence under A.1 it states, quote, "The 13 data used for designing and drawing the sample was provided by 14 FTI Consulting, Inc., an OHTA professional accounting 15 contractor, in December 2003." 16 Why did you draw the sample from FTI data? 17 A. Well, the way the population is set up did not give us a way 18 to put the units we wanted to observe together automatically. 19 We had to do some work -- Michelle, whom you've heard here, led 20 that effort to bring accounts together, because there can be 21 multiple account numbers for the same account holder, and that 22 had to be addressed so that we would get an unambiguous 23 selection probability at the account holder level. 24 And she went ahead and did that, and did what you call 25 at that point a validation effort. And she was churning them 0989 1 out -- after all, we were all pushed pretty hard by this 2 initiative. And it happened in the fall. She was churning them 3 out agency by agency -- excuse me, region by region, and passing 4 them to us as she did them. 5 And we spent our holiday - and she did, too - working 6 on this, in order to meet a delivery in the first week of 7 January, which we met. And it was an astonishing achievement, 8 even though we didn't celebrate Christmas fully. 9 Q. And so the effort that you've just described for 10 Ms. Herman's team, that's what you were referring to in the next 11 sentence by the data validation procedures"? 12 A. That's correct. 13 Q. The second paragraph of this section talks about this 14 process again. And this is near the bottom of that paragraph, 15 where you talk about the sample of transactions under $100,000 16 being selected in two phases. 17 A. Correct. 18 Q. First, a sample of accounts, and then a sample of 19 transactions. 20 A. We sampled 1,000 accounts, and then averaging 21 20 transactions per account. That was the design. 22 Q. Let me take a couple of points here. Why did you use 23 $100,000 as the cutoff? 24 A. There needed to be an upper cutoff of some kind, to protect 25 against errors that occurred in the large accounts that might be 0990 1 different than the small accounts, in the large accounts those 2 100,000 in this definition. 3 The actual boundary, the strata boundary, was 4 negotiated with the client. I think that number comes from 5 Abe Haspel. 6 Q. So for $100,000 and over, what was done? 7 A. We were going to take all the transactions, no matter what 8 accounts they were from. 9 Q. And then you sampled under $100,000. Why did you allocate 10 among the regions? You referred to 20, I think each region, at 11 least? 12 THE COURT: Excuse me. I'm missing something here. 13 100 percent reconciliation of every transaction involving 14 $100,000? 15 THE WITNESS: Yes, sir. 16 THE COURT: But what's the difference between that 17 choice and the choice made in an earlier table, which was to run 18 it down, everything of $5,000 or more? 19 THE WITNESS: May I answer that? 20 THE COURT: Yes. The question is to you. 21 THE WITNESS: The 100 percent strata was changed from 22 5,000 to 100,000. The belief was, we're not doing an historical 23 accounting at this point. We didn't know we were doing a 24 historical accounting. It turns out, I think, we may have 25 accomplished that, to a degree. But we were not doing it. 0991 1 And we were trying to take a look at what was happening 2 in this population, and you have to concern yourself with the 3 large transactions. 4 So there does have to be a take-all strata at some 5 level, and since we were trying to move quickly through this 6 process, we moved that boundary from 5,000, where there were 7 73,000 accounts, up to 100,000. 8 THE COURT: And that was for this litigation support -- 9 THE WITNESS: That's correct. 10 THE COURT: -- project. Okay. 11 BY MR. WARSHAWSKY: 12 Q. By the way, there's been some suggestion that litigation 13 support -- the litigation support project was designed with an 14 eye towards coming up with as small a number as possible for 15 exposure. Is that true? 16 A. I have no knowledge of that. That's not the way we do our 17 work. We took a risk, by having a small sample, that we have a 18 large variance. But the data are unbiased. The work is done by 19 a reputable accounting firm, four reputable accounting firms, 20 and checked by a fifth reputable accounting firm. And I do not 21 believe that anything was done to push it in one direction or 22 another. It just was what was it. 23 MR. GINGOLD: Objection, Your Honor. I think it's 24 beyond the scope of the witness' expertise. He states in 25 several of his own reports that his firm did not perform the 0992 1 basic reconciliation work. And specifically he says, in 514-6, 2 "It is beyond the scope" of his work and his expertise to 3 comment on the accounting done. 4 MR. WARSHAWSKY: Your Honor, he's simply -- 5 THE COURT: The objection is sustained insofar as it 6 supposes that the witness is giving an expert opinion about the 7 quality of the accounting firms. 8 MR. GINGOLD: Thank you, Your Honor. 9 MR. WARSHAWSKY: And just to be clear, I wasn't asking 10 for an opinion. 11 BY MR. WARSHAWSKY: 12 Q. Let me ask you to refer now to the third paragraph -- 13 THE COURT: We don't do that in this classroom, Doctor. 14 He asks the questions. 15 THE WITNESS: Thank you, judge. 16 BY MR. WARSHAWSKY: 17 Q. You referred to this a little earlier, and I just wanted to 18 get this elaborated on a little bit. The first sentence of the 19 third paragraph talks about, "Allocating the sample among 20 10 regions, roughly proportional to the number of accounts in 21 the regions, minimum of 20 accounts per region." Why did you 22 use that approach? 23 A. We wanted to achieve representativeness across the 24 geography, and later on across time, and even still later on in 25 the design, across type of transaction. And this is a starting 0993 1 point for that effort. 2 Q. Okay. 3 MR. WARSHAWSKY: Let's go to page 23. And at the 4 bottom, the very last few sentences, you can even cut off the 5 table. 6 BY MR. WARSHAWSKY: 7 Q. The statement reads, quote, "The work was to be divided into 8 four replicates," end quote. 9 Now, I didn't know about replicates before this case. 10 What's a replicate? 11 A. A replicate is a representative sub-sample of the overall 12 sample, such that if you had designed a smaller sample, it would 13 have looked, for design purposes, exactly like the larger 14 sample. 15 In this case we chose four replicates because of our 16 concerns about how hard this would be to do. After all, we had 17 just seen how difficult Eastern Region was. And if you get into 18 the details of Eastern Region, that was not a challenging, 19 compared to later activities, a challenging activity. It was 20 difficult enough. So we divided into four replicates. 21 And we had a very tight time frame, so we recommended 22 to the client that we start with a single replicate. And then, 23 if there were resources and time available, other replicates 24 would be added. 25 Q. Earlier in your testimony you talked about a sample of 0994 1 20,000 transactions, I believe? 2 A. That was the full sample, if we had done all of it. 3 Q. Is that divided among the four replicates? 4 A. That's correct. We only did a fourth of it, yes. 5 Q. So it's 5,000 transactions per replicate, roughly? 6 A. Correct. But we did -- we did all of the 100 percent cases, 7 so that made it 6,000. 8 Q. Now, if you only ended up doing one of the four replicates, 9 that's what you said. Right? 10 A. Correct. 11 Q. How does that affect your statistical analysis? How does it 12 affect the inferences you draw from the single replicate? 13 A. It makes the margin of error larger. Since there were four 14 replicates, and since we've already established this quadratic 15 relationship with sample size, we take the square root of four, 16 that's two; so that means our margin of error is twice what it 17 would have been if we had done the whole thousand. 18 Q. And when you talk about your margin of error, again that 19 refers back to the numbers you talked about in your highlights 20 section? 21 A. Yes. 22 Q. Now, let's go up to page -- skip a couple of pages, to 23 page 25. And just very quickly, this is below table A.5. Is 24 this what you were just referring to when you said OHTA 25 subsequently made the decision to reconcile all four replicates 0995 1 of the high-dollar transactions, but only one replicate of the 2 national sample? 3 A. That's correct. 4 Q. And the national sample was the 20,000 transactions? 5 A. That's correct. 6 Q. You seem to be pausing on that response. 7 A. The 20,000 was of the sample. Okay? The 100 percent strata 8 was on top of that, so it was actually -- you know, that's why I 9 paused. 10 Q. Let's jump back to pages 12 through 14, and I think we can 11 go through this in relatively quick order. But you've got a 12 section captioned "Debit results." And that continues, if you 13 want to just show the next two pages. 14 Okay. You're familiar with pages 12 through 14 of this 15 report. Correct? 16 A. Yes, I am. 17 Q. And what was the purpose of these three pages in your 18 report? 19 A. They were to describe in summary fashion the way we 20 approached the estimates, and some details around those numbers. 21 Q. And do these support what you set out in your highlights? 22 A. They support the highlights. That's exactly what they're 23 for. 24 Q. Go to table four on page 13, please. 25 Can you see that, sir? 0996 1 A. Yes, I can. 2 Q. I'm sorry. What is depicted in table four? 3 A. Let me -- these are the in-scope debit transactions that 4 were selected for reconciliation. This is the sample, and there 5 is the 100 percent strata. 6 At this stage of the game there were two -- we divided 7 the 100 percent strata up into two pieces: Those over half a 8 million, which are in the first -- in the beginning of this were 9 all to be obtained; and when we ran into some challenges, those 10 between 100,000 and 500,000, we did not have all of them 11 available at that time. 12 The rest is a national sample; that's the third row. 13 And then the total at the bottom. And then across the columns, 14 how much throughput there was in the sample, the number of 15 reconciled transactions, the reconciled throughput, and the 16 number of differences that were found. 17 Q. What's the difference between the sample size and reconciled 18 transactions? 19 A. Sample size are the cases that we originally selected and 20 passed to the accountants to do their reconciliation. And the 21 reconciled transactions are the ones that came back to us from 22 the sample. They didn't find them all, sometimes. 23 Q. I beg your pardon? 24 A. They didn't always find them all, or couldn't always 25 reconcile them anyway. 0997 1 Q. Let's look at the national sample numbers now, here. 2,372, 2 is that the total number of transactions that were selected for 3 debits? 4 A. That is correct. 5 Q. What does it mean that they reconciled all but nine of them? 6 A. Well, it means that they didn't find the supporting 7 documents that they needed in order to be able to make a 8 determination as to whether the transaction was correct or was 9 in error. That's what it means. 10 Q. The zero under "number of differences," what does that mean? 11 A. For the 2,363 transactions which they reconciled, they found 12 no differences. No errors, in a more casual way. 13 Q. As a statistician, did you draw any conclusions when you saw 14 they were able to reconcile all but nine out of 2,372? 15 A. Well, it's an astonishingly good result, frankly. I've done 16 a lot of other projects in a lot of other places , and this is 17 probably the very best result, in terms of what you've targeted 18 to do and what you actually achieved, that I've ever seen. 19 Q. And let's go to that section on page 14 near the bottom, 20 where we have some -- I don't know why I ended up highlighting 21 everything here, but what's set out in your "inference for debit 22 transactions" section. Just generally, what is that? 23 A. Well, this is a somewhat longer statement of you've already 24 asked me to talk about earlier. And I'm not sure I need to do 25 more than that, unless you want me to read something 0998 1 specifically. 2 Q. Fair enough. Let me ask you about the first sentence in the 3 final paragraph where it says, "The above figures are based on a 4 very conservative estimate of the effect of the unreconciled 5 transactions by calculating the estimates assuming that, if 6 reconciled, these transactions would have differences," end 7 quote. 8 What did you mean by that? 9 A. Those cases that were not found were treated as if they were 10 errors. Okay? And that's a very conservative assumption. 11 Q. You're talking about the nine transactions, the nine 12 transactions? 13 A. Yes. 14 Q. Why did you make that assumption? 15 A. I had earlier used the analogy with the work we did at IRS. 16 Many of us are from IRS and have done this kind of work at IRS 17 earlier. Dr. Hinkins and Robin Lee, and now Ed Murrow, who just 18 joined us, they're all from the IRS. And IRS does an awful lot 19 of samples of these sorts, and so it was appropriate that we 20 look at their practice. And I felt comfortable with adopting 21 it, and that's what we recommended. 22 Q. So in your analysis, you assumed all nine transactions were 23 in error? 24 A. Correct. 25 Q. In fact, you believe that would have been the case if you 0999 1 had found support for the nine transactions? 2 A. You're asking me to speculate. Do you want me to speculate? 3 Q. Please. 4 A. No. 5 Q. Why is that? 6 A. No, I don't think so. I have actually -- 7 MR. GINGOLD: Objection, Your Honor. This is outright 8 speculation. 9 THE COURT: Sustained. You're slurring in your older 10 age, Mr. Gingold. 11 MR. GINGOLD: Just tired. 12 THE WITNESS: Am I allowed to thank Mr. Gingold? 13 BY MR. WARSHAWSKY: 14 Q. Let me ask you this: Based on everything you saw, do you 15 have any reason to believe that all nine of those transactions 16 in fact were erroneously recorded? 17 A. Let me answer a slightly different question from that, if 18 you would. 19 Q. No, you're only allowed to answer my questions, 20 Dr. Scheuren. 21 A. My answer is, I do have no knowledge that they were in 22 error. 23 Q. Let's look at pages 15 to 17 of this report. And we don't 24 need to look at any specific section right now. You're familiar 25 with these pages of your report. Right? 1000 1 A. Yes. These are parallel to the debit stuff we've just seen. 2 Q. And why don't we go to table seven on page 16. And is that 3 a similar -- 4 A. Yes, it is. 5 Q. -- table to the one you've described for the debit analysis? 6 A. Yes, it is. 7 Q. And would you summarize what you found for the national 8 sample there? 9 A. The national sample line, there were 2,128 transactions 10 sampled. And the sample throughput is the next column. And the 11 number of reconciled transactions is the column after that, 12 2,117, which means there were 11 that were not reconciled. And 13 then the throughput that was reconciled, and the number of 14 differences found; here there were 36 differences found. 15 Q. So in this case, is it fair to say there were some items 16 that were reconciled, but with differences? 17 A. That's correct. 18 Q. As opposed to items treated as errors, for your purposes, 19 because of no support? 20 A. That's correct. 21 Q. Now, why don't we move down to the bottom of page 16, 22 carrying on to 17. Again, I think my over-highlighted sections 23 are there. Let's go to the top of page-17. 24 Again, this section on inferences for credit 25 transactions, is that comparable to what we said earlier to -- 1001 1 A. Yes, it's a somewhat lengthier version of the highlights 2 that we talked about awhile ago. 3 Q. And again, now, the first sentence in this final paragraph, 4 quote, "These figures incorporate the most conservative missing 5 data model for the transactions selected but not reconciled. 6 The unreconciled transactions have differences," end quote. 7 A. That is correct. 8 Q. And again, that's your assumption about where an item 9 couldn't be reconciled: You treated that as an error? 10 A. We didn't find the supporting documents to make a decision. 11 The accountants didn't, and so we treated it as an error. 12 Q. And in the course of your work, did you see anything that 13 caused you to believe that in fact those 11 items were 14 erroneous? 15 A. I have no knowledge of that. 16 Q. All right. Why don't we move on to a document in the 17 administrative record, AR-437. The Bates number on this is 18 000040-1. 19 Dr. Scheuren, are you familiar with this document? 20 A. Yes, I am. 21 Q. What is this document? 22 A. This is a follow-on of the previous document, where we're 23 looking at the reconciliations remaining high dollars. In the 24 earlier document there was a distinction between those high 25 dollars that were between 100,000 and 500,000, and those that 1002 1 were 500,000 and over. And we had recommended to the client 2 that they not complete the transactions between 100,000 and 3 500,000, because they probably wouldn't learn very much from it 4 and we had already made an estimate. 5 They decided to complete it, and this is what the 6 results were when that happened. 7 Q. And now let's move to AR-442, Bates number is 000042-2. 8 Are you familiar with this document, sir? 9 A. I am. 10 Q. And what is this? 11 A. It's another one of the results. And here we're talking 12 about how the litigation support led us to the sampling plan 13 that we submitted. 14 Q. Okay. And if we move down this document, there are two 15 bulleted items in the first paragraph. 16 A. Yes. 17 Q. Do you recognize -- 18 A. Those are the two documents you just asked me to talk about; 19 the first one at length, the second one just by reminding the 20 Court of what it was. 21 Q. Okay. So that was AR-438 and 437? I'm sorry, the 22 September 30, 2005 document -- 23 A. Yes. 24 Q. -- and the December 31, 2000 -- I'm sorry. The December 31, 25 2006 document? 1003 1 A. Yes. These are the two that we've just looked at, yeah. 2 Q. Very good. 3 Now, I'm a little more judicious with highlighting 4 here. You see I've highlighted a sentence that reads, quote, 5 "These analyses have caused NORC to rethink the sampling 6 proposals made in the January 6, 2003 historical accounting 7 plan, the one that was submitted to the District Court." Do you 8 see that, sir? 9 A. Yes. I wrote those words. 10 Q. I'm sorry? 11 A. I wrote those words. 12 Q. Why did you write those words? 13 A. Because we have rethought it. One learns something over 14 time, and that's what happened. I mean, we started out very 15 knowledgeable about our skill sets, but not about the 16 population. We now know a lot about the population, too. And 17 we concluded that the original plan, now that we've done the 18 litigation support activity, was much too large, was 19 over-engineered, as the phrase goes, and unnecessarily large. 20 Q. Okay. And in what -- what was it specifically that you 21 rethought as a result of your work? 22 A. Well, most fundamentally was, we could find the documents. 23 We actually -- I mean, we didn't do it. We did a little of it. 24 My colleague Robin Lee was heavily involved in that. But most 25 of the work was done by the accountants, by experts on searching 1004 1 at the various Indian repositories that we worked in. And they 2 found the records. It was astonishing. 3 The second thing was that the error rate was low 4 relative to what our expectations were. Quite low. 5 Q. Okay. We'll go into this in a little bit more detail. Not 6 too much, but a little bit more. 7 Bottom of page two. First of all, there's a 8 highlighted statement, which you can see on the screen, "Thus, 9 the large sample size proposed in the 2003 plan was not 10 necessary." Your words again? 11 A. Those are my words, and that's a summary statement of what I 12 just said. We didn't need that big a sample for the electronic 13 ledger era. 14 Q. There's a highlighted statement above there. That was your 15 recommendation as a result of this work? 16 A. That is correct. 17 Q. And that recommendation was? 18 A. That we did not recommend any further reconciliations of the 19 posted transactions. Can I elaborate why I said that, or do you 20 want to do it later? 21 Q. Please. That would have been my next question. Go ahead. 22 A. It has to do with how much it costs to do the 23 reconciliations, as to how much exposure or how much reduction 24 in our understanding -- how much better our estimates would be. 25 Let me make something very clear, Your Honor. When we 1005 1 increase the sample size, we don't expect to change the point 2 estimate, the estimate of the error rate. That's not going to 3 happen. What's going to happen is, we're going to shrink the 4 margin of error around the estimate. Okay? And that's the 5 issue. So when you shrink the margin of error, of course, you 6 change some of these assurance levels that we talked about 7 earlier. 8 And we looked at this issue. Let's say we had done a 9 second replicate. We had four, and we did a second replicate. 10 The litigation support activity cost well over $20 million. 11 Okay? Very expensive. Some of that was for the 100 percent 12 certainty strata, and the rest was for the sample. 13 Well, the certainty strata has already been done. 14 You're not going to spend that money again. Those cases that 15 were over 100,000, 100,000 or over, we've been calling them the 16 certainty strata. And then -- but we could have done another 17 replicate, another sample replicate. 18 The order magnitude on cost of that was about what we 19 would have been able to reduce the assurance level by. So we 20 were not going to gain a great deal because of the fact that 21 this increasing sample sizes doesn't give you a very big result 22 in terms of lowering the assurance levels or lowering the 23 exposure or whatever phrase you want to use. It's about 24 comparable. 25 So it didn't make sense to us. You're going to have to 1006 1 spend real money, okay, and there's always a chance, okay, that 2 what you're expecting to happen doesn't happen, and you might 3 end up not even being pleased with what you came up with. 4 And by then -- may I add one more thing? By then, 5 everyone involved was totally exhausted. I mean, the amount of 6 effort that people stretched themselves to do was such that at 7 least you give them a break before you do it. So that was also 8 a factor here. 9 Q. But you said, if you had done a second replicate, it would 10 have cost about 20 million? 11 A. Probably a little under 20 million. 12 Q. And so to complete that sample, as you originally designed 13 it, another $60 million? 14 A. Close to that, yeah. 15 Q. Near the top of page two, a little bit further up, the first 16 sentence under "2003 plan versus LSA design." You've touched on 17 this already a couple of times, but explain what you meant here 18 when you said the 2003 plan indicated an adaptive approach would 19 be followed. 20 A. Our plan then and our plan now is to base what we end up 21 with what the final sample size is based on what we're learning. 22 And if, as in this case, we thought that the error rates -- if 23 the error rates were higher than we expected them to be, say in 24 a particular agency, we would have increased the sample size in 25 that agency in order to achieve the same overall level that we 1007 1 expected to achieve. 2 We have a set of assumptions we're making. Those 3 assumptions lead us to a tolerable error. The client agrees 4 that that's what they want. If the assumptions are wrong, and 5 they're wrong in that the error is bigger than assumed, in order 6 to still achieve what the client wants, you have to increase the 7 sample size. The error is what it is. You can't change the 8 error rate, but you can reduce the sampling error around that 9 number. 10 Q. Next to last paragraph in this section, right above the 11 highlighted section, "In the LSA project, however," do you see 12 that paragraph? 13 A. Yes, I do. 14 Q. That summarizes what you've already said, right, about the 15 record find and the reconciliation rates? 16 A. Yes. 17 Q. And that was a basis for your recommendation? 18 A. Yes, it is. 19 Q. Now, let's go to the final page, your summary conclusions. 20 You've already talked about your strong recommendations. 21 Explain -- you've got slightly different analysis on the debit 22 transactions and the credit transactions. 23 A. That's correct. 24 Q. And would you explain the distinction between your analysis 25 here? 1008 1 A. We actually already thought at this point, given the results 2 of debits, that we had drawn a sample that was too big for 3 debits already. We didn't actually need as big a sample as that 4 to achieve what we thought the client wanted. 5 Credits are a closer call, a trade-off on costs. Okay? 6 Because there was some lumpiness in the credit results, and 7 there was -- the error rates were higher, slightly higher, 8 1.3 percent. Remember, the client expectations were 9 one percent; that's what we designed it at. So it was slightly 10 higher, at 1.3 percent. 11 What saved us was the fact that the find rate was so 12 exceedingly good. We found virtually everything. Well, the 13 accountants and others found virtually everything. 14 Q. Now explain, though, what you're talking about in the credit 15 transactions about cost effectiveness. 16 A. Well, you've just heard that, and I'll just recount it 17 again. If you had increased the sample size for credits, you 18 would have reduced the margin of error on credits. The estimate 19 would have been about the same, 1.3 percent, but it has 20 uncertainty so it could have wobbled a little bit. But you were 21 going to spend a lot of money to do that, and it just -- why not 22 just take -- we asked the client, "Why don't you just take the 23 higher assurance level," okay? It's reasonable to do that, and 24 that would be fair. 25 Q. Dr. Scheuren, I would like you to refer now to AR-427. 1009 1 THE COURT: How much more direct do you have, 2 Mr. Warshawsky? 3 MR. WARSHAWSKY: I would say a half an hour, at most. 4 THE COURT: In that case, let's take a mid-morning 5 break. 6 Mr. Smith, could I see you at side-bar, and 7 Mr. Kirschman? 8 (Recess taken at 11:02 a.m.) 9 THE COURT: All right, sir. Please continue. 10 BY MR. WARSHAWSKY: 11 Q. Dr. Scheuren, I'd like to get back to one point that you 12 mentioned before the break. When we were talking about -- you 13 were explaining reasons not to go ahead and do a second 14 replicate. Do you remember that? 15 A. Yes, I do. 16 Q. One of the reasons you said was you might not get the result 17 you liked. What did you mean by that? 18 A. Well, let me -- an ill-chosen word, I might add. You have a 19 point estimate of 1.3 percent - we'll talk about credits - and 20 you're going to do another sample, you're going to get a bigger 21 sample, but you're still going to have uncertainty in the 22 results. 23 And the point estimate might not be 1.3 when you've 24 added the second replicate. It could be smaller, could be 25 larger, depending on whether you like a smaller or larger 1010 1 number. You might not like that result, and you have spent a 2 lot of money. 3 From the government's point of view, what really 4 matters is not so much the point estimate but the upper bound. 5 The upper bound is seven percent, as you'll remember. Okay. 6 99 percent upper bound. You would expect, given the larger 7 sample, that that upper bound would shrink, be smaller, if you 8 get the same 1.3 estimate. You can't guarantee the 1.3. After 9 all, it's an uncertain world. 10 So does it pay to reduce the upper bound and spend 11 about as much money in order to reduce it, as you would gain if 12 it were smaller? 13 So I didn't think it was worth doing, and I recommended 14 it to them and they've subsequently adopted that. 15 Q. Dr. Scheuren, I'd like you to refer to -- from the 16 administrative record AR-427, Bates numbers 000052-3. 17 MR. WARSHAWSKY: We may have to blow this up a little 18 bit. There we go. 19 BY MR. WARSHAWSKY: 20 Q. March 15, 2007 memo from you to Bert Edwards. Do you 21 recognize this document? 22 A. Yes, I do. 23 Q. And what is this? 24 A. It's our proposal to the Office of Historical Trust 25 Accounting. Okay. 1011 1 Q. Okay. Your proposal with regard to what, I'm sorry? 2 A. The 2007 plan. 3 Q. And so this ultimately is what led to the plan -- or this is 4 with regard to the plan that's before the Court now? 5 A. That's correct. 6 Q. Now, I'd like you to refer, bottom of page two -- first of 7 all, to put it in context, on the first page, not the cover 8 memo, but the page one that follows it, you have a section 9 captioned, electronic ledger era. 10 A. I'll take your word for it at the moment. 11 Q. Here, it's blown up a little bit. Does that help? 12 A. Yes, I do. 13 Q. And it's within this section that I want to ask you some 14 questions. This carries over to page three. 15 A. I'm not sure -- this has been moving around in front of me, 16 so... 17 Q. You should have worked with the paper copy. 18 Starting at the bottom of page two. 19 MR. WARSHAWSKY: Can you get the bottom of page two and 20 that section at the top of page three on one screen? I don't 21 know if you can do that or not. Perfect, I think. 22 BY MR. WARSHAWSKY: 23 Q. You have this paragraph that begins at the bottom of page 24 two where you're talking about, "based on the results of the LSA 25 sample, you conclude only relatively modest follow-up work?" 1012 1 A. Correct. 2 Q. And you refer to two areas not yet covered. What did you 3 mean by the posting tests? 4 A. We talked about the -- initially about the fact that we were 5 only looking at transactions that had been posted. So our 6 sample frame population that we're sampling from are posted 7 transactions. 8 There is a possibility that there are unposted 9 transactions that were never posted, and we need to look at 10 that. That's an important issue. Okay? 11 And I recommended that we do that. That's the first of 12 these two on posting. 13 Q. And what are you recommending be done with regard to any 14 transactions ultimately found to have been unposted? 15 A. That they would be errors, that there would probably be 16 errors if they were -- should have been posted and were not 17 posted, that would be errors. And that would be a serious 18 errors. 19 Q. How would you assess those? 20 A. Well, we're working on that right now. I happened to be 21 here at the end of the day yesterday because I was getting ready 22 to testify when Michelle was talking about Holton (sic). 23 Q. Was it Horton? 24 A. Horton, yes. One of the places we've been testing, testing 25 in that case, procedurally how we would do this. Okay? 1013 1 MR. WARSHAWSKY: Just so the record's clear, I saw a 2 puzzled look on the reporter's face. 3 THE WITNESS: Well, I knew I misspoke when I said it, 4 because I'm not too good at names. 5 A. Anyway, that was a procedural test to see how hard it was 6 going to be to do this. 7 We went out and did another sample, okay, to get 8 transactions that had not come to us as posted; by going back to 9 the land or leases or combinations, we were then exploring those 10 issues in Horton. 11 And subsequently, we've done three more of these 12 activities, three more agencies; more complicated revenue 13 streams, et cetera, et cetera. Do you want me to mention them? 14 Maybe it doesn't matter. We're learning how to do this, and of 15 course we can't do this ourselves. The accountants have to come 16 in. 17 So we found the records, the records are ready to be 18 reconciled. But this is a different kind of accounting, so I've 19 been told, than the accounting that was done for the 20 transactions. The Accounting Standards Manual doesn't apply to 21 the kinds of activities that were engaged here. 22 And so the accountants have to come together and 23 developed their procedures. And I believe in the context of 24 earlier activities, they have to be submitted to the Court. 25 Q. Now, this is work that still needs to be done -- 1014 1 A. It's under way. We've done our part. 2 Q. Dr. Scheuren, just so we have a clean record, this is work 3 that you believe still needs to be done with regard to the 4 electronic ledger era? 5 A. Indeed. And we are doing pilots of that. 6 Q. The second item that you identified in your memo is, 7 "cleanup tests of transactions discovered later to be in the 8 population of interest but not included in the original sampling 9 frame." What were you referring to there? 10 A. Here again, Michelle is involved. She has been talking 11 about this for quite some time. Those cleanup transactions, 12 those added transactions, quite a few of them, need to be tested 13 just like the other transactions that we've already tested in 14 the LSA. And we recommended that. They all should be subjected 15 to the same sort of testing that we've been doing for the LSA. 16 Q. And that includes sampling? 17 A. That indeed includes sampling, yes. 18 Q. Dr. Scheuren, I'd like you to refer now -- let's shift 19 topics a little bit to administrative record 426. This is Bates 20 number 000052-2. We're looking at a May 25, 2007 memorandum 21 authored by you. Correct? 22 A. Signed by me, yes. Usually authored by -- as I've already 23 indicated, it's authored by Robin and Susan and I. If you've 24 looked at these memos, and maybe others have noticed this, some 25 are signed by Susan and some are signed by me. But it's usually 1015 1 a joint work product of all of us. This I wrote more of than 2 some of the others. 3 Q. But you put your initials on it. Right? 4 A. Yes, indeed. 5 Q. And what was the purpose of this memo? 6 A. It was to give them a strategy for the sampling to be done. 7 And the focus was shifting the emphasis from making an estimate, 8 which is what we did in the LSA and what we intended to do in 9 the 2003 plan, to testing a hypothesis that the paper ledger era 10 would produce an error rate similar to the electronic ledger 11 era. 12 Q. Now, this proposal is with regard to the paper ledger era. 13 Correct? 14 A. Correct. 15 Q. What is meant by the paper ledger era? 16 A. These are the transactions that were not included in the 17 electronic population, as augmented by the work that Michelle 18 has been doing, they were not included in the electronic 19 population. 20 Q. And this was not part of the LSA, then? 21 A. No. 22 Q. I've highlighted in the subject, "initial hypothesis 23 testing." What is meant by the term "initial hypothesis 24 testing"? 25 A. We are trying to develop an approach here where -- we don't 1016 1 claim we know enough to call it -- the word "initial" is just a 2 starting point. Okay? But we want to look at the hypothesis, 3 as I've already stipulated, that the error rates for these two 4 eras are virtually the same, same magnitude. 5 Q. Why are you wanting to test that hypothesis? 6 A. Because it allows us to draw a smaller sample. And if we're 7 wrong, we can increase it later. But if we're right, we save 8 the money, or the government saves the money. 9 Q. Why not simply approach the paper ledger the same way that 10 you did the electronic ledger era? 11 A. Well, first of all, we know a lot more than we did when we 12 started. It would be, I think, wasteful not to use that 13 information. This is a way, not the only way, this is a way to 14 use that information. 15 Q. On the second page of this document, you have a section 16 captioned, "recommended paper ledger" -- 17 THE COURT: Just excuse me a minute. 18 This hypothesis that the paper ledger era error rate 19 will be the same or roughly the same as the electronic era, 20 whose hypothesis is that? 21 THE WITNESS: We are proposing that as a hypothesis to 22 the client. 23 THE COURT: Okay. But the real hypothesis is -- well, 24 I suppose you can comment on this rather than answer the 25 question. The real hypothesis is, let's try to do it for less 1017 1 money, and by doing it for less money, one hypothesis that flows 2 from the less money assumption is same error rate, which we 3 contest. 4 I mean, it's driven more by cost than by anything else, 5 is it not, this hypothesis? 6 THE WITNESS: Cost and time, yes. I mean, it takes a 7 long time to do this. We didn't carry you through this, but in 8 the earlier LSA effort, we ran out of money. We started in 9 January. Remember I told you we all worked Christmas? 10 THE COURT: Right. 11 THE WITNESS: And we started in January and we ran 12 until somewhere in mid August and we ran out of money. Had to 13 stop. Okay? And that gave us a chance to resample, redesign, 14 and resume at a smaller sample size. So money mattered, and 15 time mattered, too. 16 THE COURT: Okay. I got it. 17 BY MR. WARSHAWSKY: 18 Q. Well, is cost a valid consideration when you design a 19 sample? 20 A. It always is in all the work I've ever done, sure. You're 21 dealing with a client who has to put the money out, and the 22 client wants a result within a certain tolerance band, okay, for 23 the resources that they have available. 24 And you go to the client and you see the client doesn't 25 have enough money, and you say, you don't have enough money to 1018 1 do this; you've got to change your tolerance level or you've got 2 to get some more money. We'll do our best to come up with a 3 good sample design which uses frugally your resources -- 4 THE COURT: I don't think we have to quarrel about 5 that, Mr. Warshawsky. If there's one proposition that is law of 6 the case, it is that money is relevant. So proceed from there. 7 MR. WARSHAWSKY: Very good. Very good. 8 BY MR. WARSHAWSKY: 9 Q. So in your basic sample design, you proposed two stages. 10 Correct? 11 A. Correct. 12 Q. And would you describe what those are? 13 A. These two stages are the same two stages we used for the 14 LSA. We would take a sample of accounts from a list of all the 15 accounts - the problem here is getting the list - and known or 16 believed to contain transactions, and then we'll sample the 17 transactions. 18 Q. Now, we can refer to the next page, page three. And you've 19 already described to some extent what I've highlighted there, 20 the purpose of your hypothesis test. 21 What happens if -- or what do you propose if, in fact, 22 you run the test and the hypothesis is accepted, that, in fact, 23 the difference rate for the paper ledger era is statistically 24 similar to or even smaller than the electronic ledger era? 25 A. We will have designed the sample size to be sufficient, so 1019 1 that should that hypothesis be accepted, we'll be finished. 2 We'll be able to produce results that will be satisfactory for 3 the client's goals. 4 Q. And what happens if that hypothesis is rejected? 5 A. We're going to need a larger sample, and we'll adaptively 6 increase the sample to achieve -- after going back to the 7 client, the client can change their goals, but if they want to 8 keep their goals what they are, we're going to have to have a 9 bigger sample. 10 Q. Now, Dr. Scheuren, you're aware that there's been some 11 debate in the prehearing pleadings about the role of 12 meta-analysis. Correct? 13 A. Yes, I am. 14 Q. What is meta-analysis? 15 A. Meta-analysis is a technique, one of many, to combine 16 studies from many sources on similar or related subjects to try 17 to draw together what is known about the subject. And perhaps 18 in drawing it together, to gain some advantage, some additional 19 information from it. 20 If I might elaborate one more time. I've been alluded 21 a couple of times to the difference in ages. I began, and 22 sampling began, in an era of where the world was data poor. 23 Q. Data poor? 24 A. Data poor. There was very little data available. And the 25 sample that was drawn in those days - this is in the '30's, for 1020 1 example - were small to begin with. And then the world became 2 data rich, and then, of course, there's lots of information that 3 comes from data. 4 And then, from the information, how we combine all this 5 information? We're now moving to that stage in science. We're 6 combining information and using techniques, like meta-analysis, 7 to do that. Certainly not the only one, but it's certainly a 8 valid one. 9 Q. Is it an accepted technique in the world of statistics? 10 A. At least accepted, and our reports cited, a dozen or near a 11 dozen reports which describe it. It's heavily used, especially 12 in the medical area. 13 Q. I'm sorry, you said your reports. What are you referring 14 to? 15 A. We did two volumes, a summary of the work that we did on the 16 meta-analysis. And in one of those lines, I guess the first 17 one, we talked about or cited other reports that were done, 18 other approaches in the literature on meta-analysis. 19 Q. What was the role of meta-analysis in terms of formulating 20 your recommendations for the paper ledger era? 21 A. We were looking for, I think the phrase actually is a trout 22 in the milk. We're looking for something that said to us, this 23 hypothesis just doesn't work. Why even start on this without 24 looking to see if there's a major, major result out there which 25 says that this hypothesis isn't going to be satisfactory. 1021 1 So we did that and we looked at all of the studies we 2 could find. 3 So it's basically a due diligence step, and if we had 4 not found that, we would not -- if we had found something big 5 out there, we would not have been making this recommendation as 6 to the hypothesis. 7 MR. WARSHAWSKY: Would you pull up Defendant's 8 Exhibit 7, page 15, please? And I would like you to blow up the 9 last three paragraphs there. 10 BY MR. WARSHAWSKY: 11 Q. This is from your rebuttal report. Correct, sir? 12 A. Yes. 13 Q. And you quoted a statement from Mr. Duncan's analysis there. 14 Do you see that? 15 A. I do now. 16 Q. Okay. Do you agree with the statement that, "the 17 meta-analysis performed by NORC plays a central role in 18 purportedly strengthening and corroborating DOIs conclusions", 19 and then continues on? 20 A. Adjectives and adverbs are always dangerous. The word 21 "central," I do find a problem with that. It plays a role, 22 among a bunch of other things, in order to plan ahead before we 23 actually undertake the early pilot activities, which we've 24 already done some of those, too, on this 2007 plan. It plays a 25 role, not a central role. 1022 1 Q. Okay, sir. 2 A. I would say it's a minor role. 3 Q. Go ahead. 4 A. It's a minor role given what we found. If we had found a 5 problem, a serious problem, it would have been -- it would have 6 stopped us. Probably would have stopped us. But that didn't 7 happen. 8 Q. Aside from the meta-analysis work, what activities, if any, 9 have been undertaken to develop the design for the paper ledger 10 era? 11 A. One of the things we did was to look at the work done by 12 Treasury and General Accounting Office, and for the period 1890 13 to 1950, 60-year period, to see whether we were going to find 14 problems in that 60-year period. And that's one of the reports 15 in our administrative record. In a very one sentence or one 16 phrase way, we didn't find the kind of problems that would have 17 led us to change our mind. 18 Another thing we did, and we're almost finished now, 19 were their reservation histories. In the 2003 plan we said -- 20 or whoever wrote that plan said that there would be reservation 21 histories produced. They're almost all finished now. They're 22 very useful to see if there's anything special going on in a 23 particular area that might draw you to a different conclusion as 24 to the initial - important to emphasize "initial" - hypothesis. 25 I haven't read all of those myself. I've only read a 1023 1 few of them. I haven't seen everything yet. We're not through 2 the end of that. Others will look at those. But I think we're 3 going to end up probably where we are here. 4 We've actually done, as I mentioned, some early pilot 5 activity, looking for some -- you know, you to have to go get 6 the ledger, remember? We don't have these ledgers 7 electronically. So we have to find a ledger, find the account, 8 and then follow the account back historically to see what's 9 going on. And we've begun that process. 10 Have we done accounting for that yet? No, we have not. 11 But we're looking at cost issues here, too, after all. That 12 bears on this. 13 MR. WARSHAWSKY: Would you put up Defendant's 14 Exhibit 6, please, page 16? 15 BY MR. WARSHAWSKY: 16 Q. Dr. Scheuren, this is your expert report, page 16 of it? 17 A. Yes. 18 Q. And do you recognize the three bullet points on this page? 19 A. Yes. 20 Q. And what is set forth in these bullet points? 21 A. These are three recommendations that grew out of the work we 22 did. There are many other recommendations we've made to the 23 client, but these seem to be important enough to bring to this 24 level of attention. 25 Q. And would you summarize the opinions set forth in your first 1024 1 bullet point? 2 A. We've been talking about this a lot. In the light of the 3 baseline error rate that we established in the study, we thought 4 it was wisest -- the word "wisest" is something that I should 5 have put in quotes because it's not necessarily -- I'm not an 6 expert on wisdom. Wisest, since the unit cost of the accounting 7 ledger was expected to be even higher, that we would start with 8 a smaller sample size and focus on that as a way of starting. 9 We had started with a larger sample size in 2003; 10 turned out to be overengineered, enormously overengineered, in 11 our opinion. We'll start with a smaller sample size, and if we 12 needed to, we would go up in size. 13 Q. Would you summarize your second opinion? 14 A. I recommended that we didn't need to do any more 15 transactions from the same population in the electronic ledger 16 era. I've already elsewhere - and you've drawn this out of me - 17 recommended that we do additional testing in the electronic 18 records era to make sure that the transactions were complete, or 19 complete enough, and to test the new transactions that were 20 added by the work that Michelle's team has done. 21 Q. And finally, in your third paragraph you talk about the more 22 conventional assurance level of 95 percent. Do you see that? 23 A. Because of the importance of the case, and because of the 24 concerns about potentially higher error rates than was initially 25 thought to be the case, we set the assurance level at 1025 1 99 percent. And setting it at 99 percent causes you to have a 2 much larger sample than if you set it at 95 percent. 3 It's entirely possible, whether you set it at 95 or 99, 4 to provide the estimate at the other level. It's just you're 5 not using it to design the sample size. If you set it at 95, 6 you're going to have a smaller sample at the get-go, and it's a 7 cost issue. 8 And given what we found out, I think it's reasonable. 9 And you could always go back and make the estimate at 99. 10 Q. And how would you do that? 11 A. Simple. Mathematically, do it. I mean, you go -- a 12 95 percent confidence interval, to increase it to a 99 percent 13 confidence level, you multiply it by a constant that depends on 14 the nature of the sample design you're talking about. But it 15 would be about 30 percent, 40 percent more. 16 THE COURT: In other words, you keep the sample size 17 the same, you just report different results, whether it's 95 or 18 99 percent? 19 THE WITNESS: Exactly. And it's a design issue at the 20 estimation stage. At the analysis stage, the client can do 21 either one, a very minor change. 22 MR. WARSHAWSKY: Thank you, Dr. Scheuren. 23 Your Honor, we would move into admission Defendant's 24 Exhibit 6 and Defendant's Exhibit 7. That's Dr. Scheuren's 25 expert and rebuttal expert report. 1026 1 THE COURT: All right. I'll receive 6 and 7. 2 (DEFENDANT EXHIBIT Number 6 and 7 were moved into 3 evidence.) 4 THE COURT: Mr. Gingold? 5 MR. GINGOLD: Thank you, Your Honor. Good morning, 6 Your Honor. 7 THE COURT: Good morning. 8 CROSS-EXAMINATION 9 BY MR. GINGOLD: 10 Q. Dr. Scheuren, my name is Gingold and I represent plaintiffs. 11 I'd like to discuss with you at the outset some general 12 issues. You talked about cost as being an important factor at 13 least as your task is assigned. Is that correct? 14 A. That's correct. 15 Q. But is cost something that can affect reliability? 16 A. Of course it can. 17 Q. How can it do that? 18 A. Larger sample and more costly sample would increase 19 reliability. 20 Q. What about replication of results? 21 A. You want to -- your first few words were a little blurred 22 there. Can you repeat? 23 Q. Replication of results? 24 THE COURT: You said, "what about replication of 25 results." 1027 1 A. You mean if you repeat the results? What do you mean -- I 2 mean, we talked about replication in the context of sampling in 3 which we had four replicates. Is that the sense in what you 4 mean? 5 BY MR. GINGOLD: 6 Q. No, I'm asking you generally about reliability. And with 7 respect to statistical sampling, isn't the measurement process 8 itself required to be reliable? 9 A. Of course it is. 10 Q. And isn't replication or reproduction of the results 11 important to determine consistency? 12 A. Replicability I think is the word you want to use. And the 13 answer is, of course it is. And in the samples we're drawing, 14 we are producing replicable -- the results can be replicated. 15 Q. So your position is the ability -- your belief in the 16 ability to replicate them satisfies the requirement of 17 reliability. Correct? 18 A. I didn't say that. 19 Q. Okay. 20 A. Replicability is one dimension of reliability. Reliability 21 has a lot to do with the margin of error you have in mind. 22 Q. I'm sorry, Doctor -- 23 A. May I? As I've indicated, that's a client decision, and our 24 role as samplers is to meet the client requirement. 25 Q. You made it very clear that the client determines the nature 1028 1 and scope of your engagement. Correct? 2 A. Yes. 3 Q. And you operate to the best you can within the scope set 4 forth by the client. Correct? 5 A. Yes. But I also stipulated the independent role that I 6 would have not -- we would have not have done this if we had 7 not -- if we hadn't the ability to provide an independent role. 8 Q. I understand that. But my question was, you operate within 9 the scope -- 10 A. Yes, I do. Resources are resources. 11 Q. Are you familiar with the Reference Manual on Scientific 12 Evidence? 13 A. Yes, I am. 14 Q. Are you familiar with the discussion in the reference manual 15 concerning the accuracy of measurements and reliability, 16 referring to the reproduction of results? 17 A. I don't have that in my head right now. I mean, you want to 18 provide it to me on the screen? 19 Q. No, I'm asking if you're familiar with it. That's all. 20 A. I'm generally familiar only. 21 Q. So it's your understanding that the reference manual talks 22 about the ability to reproduce the results, and not the actual 23 production of the results to ensure reliability of the process. 24 Correct? 25 A. Why don't we divide that up a little bit more? 1029 1 Q. Does the reference manual, based on your recollection, 2 require the reproduction of results to ensure reliability? 3 A. The results should be ones you can repeat. That's correct. 4 Q. It is correct, isn't it? 5 A. You should be able to get the same -- but you're obviously 6 doing it with sampling error and so on. So you may be -- I'm 7 not really sure what you're asking here. 8 Q. I'm asking whether or not for you to state that your 9 conclusions are reliable, statistically reliable, that you need 10 to reproduce the results. 11 A. In an estimated way, the expected value is yes. Okay. 12 Q. Again, statistics is all estimates. Correct? 13 A. Pretty much. 14 Q. You're not talking about what has actually occurred, you're 15 dealing with probabilities and estimates. Correct? 16 A. That is correct. But there was an interpretation of what 17 you said that I actually go through and reproduce the actual 18 calculations of every step along the way. Okay? That is a form 19 of replication as well that's used typically in an IT setting. 20 And I take it you were talking about a statistical setting only. 21 But both apply here, in fact. 22 Q. But you are testifying as an expert in statistics, not IT. 23 Correct? 24 A. Correct. 25 Q. The questions I'm asking you are with respect to what is 1030 1 stated as your expertise. 2 A. Thank you. 3 Q. So my question for you is: If you chose not to replicate 4 the results, does that affect reliability as discussed in the 5 Reference Manual on Scientific Evidence? 6 A. Internally, our sample already allows us to replicate the 7 result. Okay? A new sample, a fresh, completely new sample is 8 not necessary, in my opinion, as an expert, to achieve this 9 standard of replicability. 10 Q. So if, in fact, the Reference Manual on Scientific Evidence 11 states that reliability refers to the reproductability of 12 results, that's what you believe is the case, not the need to 13 reproduce the results to prove the consistency. Correct? 14 A. My answer is yes. Yes. 15 Q. And what we're talking about, whether you can possibly 16 produce it or you have, doesn't go to the accuracy of the 17 results, does it? 18 A. No. 19 Q. So you can actually reproduce the results 10,000 times, and 20 it's not stating anything about its accuracy, is it? 21 A. Depending on how you did it. I mean, what we're doing in 22 probability sampling is developing methods that allow us to 23 measure the accuracy at the same time we do the work. 24 Q. Because you're providing consistency. Correct? 25 A. That's one of the factors, yes. 1031 1 Q. But you're not providing a statement of validity of the 2 results, are you? 3 A. You're moving around in your concepts here, and I suppose 4 that's your prerogative. The results of validity are somewhat 5 different from the issue of replicability. I have a process 6 which is going to produce a certain result. Okay? Is that 7 result on target? Is it valid? That's a separate issue. 8 Q. Is it accurate? 9 A. That's -- replicability doesn't mean -- here's the result, 10 okay, that I'm going to get from this process, and here's the 11 true value over here. Okay? And if on the average this process 12 hits that true value, even though it doesn't in any case, that's 13 valid. Okay? All right? That's valid. That's validity. 14 Okay? 15 If it's a good process and you've got a nice narrow 16 margin of error and so forth, it's reliable. 17 Q. If you stepped on the scale 14 times and produced a weight 18 of 300 pounds, it's consistent, isn't it? 19 A. That's correct. 20 Q. And it's statistically reliable, isn't it? 21 A. Yes. 22 Q. But it's inaccurate, isn't it? 23 A. Yes. 24 Q. Thank you. 25 And you're not determining the validity of the 1032 1 underlying transactions in your statistical analysis, are you? 2 A. I'm not an accountant. 3 Q. That's right. Or you're not a consultant who does the 4 analysis, whether or not he or she is an accountant. Correct? 5 A. Some of the analysis we did do based on the -- 6 Q. No, no. You're not doing the analysis of whether or not 7 there's an error in the transactions. Correct? 8 A. That's correct, we did not do that. 9 Q. So whichever consultants, accountants or otherwise, make 10 that judgment, you're accepting that judgment. Correct? 11 A. We did look at the quality assurance samples. We actually 12 designed the quality assurance samples that were used by Graham 13 Thornton, which is one of the accounting firms they engaged. 14 And we've seen that work, we've actually incorporated 15 the errors that they found - very few errors - but we 16 incorporated those errors in the statement of the uncertainty in 17 these results. 18 Q. But you're a man of precision, aren't you, with regard to 19 what you do and how you testify. Correct? 20 A. Doing my best. 21 Q. You don't determine what is an error and what isn't an 22 error, do you? 23 A. No. 24 Q. The consultants and others, including accountants, make 25 those determinations. Correct? 1033 1 A. That is correct. 2 Q. So whatever the error result is, is not determined by you? 3 A. We analyze the resulting errors that are determined by 4 others. That's what we did. 5 Q. Right. So if a transaction that's not on the system is not 6 considered an error, whatever your acceptable error rate is 7 doesn't include that. Correct? 8 A. I've already told you that we are going to be looking for 9 transactions that should have been on the system and were not on 10 the system, and once we have identified them, we will sample 11 them. 12 Q. That wasn't my question. And you responded to 13 Mr. Warshawsky in a direct examination. You weren't talking to 14 me. 15 And you haven't testified as an expert before, have 16 you, Dr. Scheuren? 17 A. Not in this setting. My apology. 18 MR. GINGOLD: If I may, Your Honor. 19 BY MR. GINGOLD: 20 Q. Your answers should actually be directed to the Court. I'm 21 asking questions. 22 A. Certainly. I knew that. 23 Q. That's all I'm doing. 24 A. All right. Please resume. 25 Q. Thank you. 1034 1 If transactions are not on the system, as, for example, 2 within what is defined as the electronic records era, they are 3 not viewed as errors, are they? 4 A. It's an odd way of saying this. My answer is, the 5 population that we're making an inference about, okay -- within 6 the population that we're making an inference about, they're not 7 viewed as errors. 8 Q. What is your understanding of the definition of the 9 electronic records era? 10 A. The original version of this -- 11 Q. How about the version you're working with. 12 A. You walked away from the mic. You want to -- 13 Q. Yeah, yeah, I would like you to -- 14 A. The electronic -- 15 Q. If I may, I'm sorry. 16 What is the definition of the electronic records era as 17 you understand it, and how you are working with it in your 18 statistical analysis? 19 A. The electronic records era, or ledger era, are all those 20 transactions that were electronically available, okay, or have 21 subsequently been added in this work that you've heard Michelle 22 Herman talk about. They were not -- 23 Q. I'm sorry. 24 A. That was not the population about which we made our 25 statements for the LSA. The population that we made our 1035 1 statements for the LSA were transactions for account holders 2 that existed on or after October 1994. Those are the only ones 3 we looked at. 4 Q. If I may, for precision purposes, because this will help and 5 probably expedite the cross-examination, so what you're talking 6 about as you understand the electronic records era are, it's 7 from a date, a period in time, a date certain. Correct? 1985 8 to 2000? 9 A. That isn't what I said, sir. 10 Q. Oh, it is not the case? 11 A. No, it's not what I said. 12 Q. Okay. 13 A. What I said -- may I repeat what I said? 14 Q. Please do. Because I didn't understand it. 15 A. I'll leave that alone. Let me say it this way -- 16 Q. Please. 17 A. What I said was all of the transactions that were on the 18 system, okay, electronically, or added by Michelle Herman's 19 team, okay, are in the electronic ledger era. That was not the 20 population that we sampled. We sampled a smaller population. 21 Q. So dates are unimportant in your understanding of the 22 electronic era. Correct? 1985 to 2000 is irrelevant, based on 23 your testimony. Correct? That's not the issue, is it? 24 A. The dates that -- the records that existed on the system, 25 okay, is what I mean by electronic records era. The 1985 is 1036 1 actually a tricky problem, as you know. 2 Q. Well, please explain. Because I'm trying to get an 3 understanding. Because much of the information that's been 4 produced in the administrative record and testimony of other 5 witnesses have suggested that the electronic record -- 6 electronic ledger era is a period of time from 1985 to 2000. 7 But please explain why my understanding is incorrect. 8 A. It is that. But it's also those transactions which were -- 9 that were in that time period that were electronically 10 available. Not all of the transactions in that time period 11 remained electronic. Okay? Some of them were written over, as 12 you've heard. I assume you've heard. 13 But the ones that were available or have been added 14 subsequently by Michelle's team, I would consider the definition 15 of electronic records era, or ledger era. 16 Q. Why did you say the 1985 period is a tricky time period? 17 A. Because at that point the process that we're talking about 18 began. But because of the nature of the system, it didn't begin 19 everywhere at the same time. 20 Q. So you haven't defined the era. That's not your definition, 21 is it? 22 A. No. 23 Q. Who defined the era? 24 A. I don't think I know the answer to that directly. I mean, 25 it's been a working definition that I've heard others use, okay, 1037 1 and we've been looking at a subset of that population, as you 2 heard me testify to. 3 Q. Is it your understanding -- 4 THE COURT: This is for both of you. We don't have 5 time for this sparring. I have heard the term "electronic era" 6 described to be a time period between 1985 and 2000. The 7 witness says that's true. I've also heard him say that some 8 documents created in that period of time are not available 9 electronically. I've also heard him say that the LSA -- or 10 excuse me, that his analysis, maybe the LSA, uses a different 11 concept of electronics era that is for transactions that 12 occurred on or after October 1994. Is all that correct? 13 THE WITNESS: Yeah. 14 THE COURT: Let's move on. 15 BY MR. GINGOLD: 16 Q. One last clarification. 17 MR. GINGOLD: Your Honor, there's only one reason I was 18 asking the question. The electronic era and the IRMS system 19 began in the 1970's, so I was trying to understand why it 20 started in 1985. 21 THE COURT: If he knows. 22 BY MR. GINGOLD: 23 Q. So do you know why the electronic era, as defined and as it 24 has been -- 25 A. I don't know. 1038 1 Q. And as a result, the pre-1985 transactions on the IRMS 2 system are part of the paper ledger era. Is that correct? 3 A. That's the definition we're using, yes. 4 Q. And you don't know why, do you? 5 A. I've answered that. The answer: I don't know. 6 Q. Okay. Thank you. 7 What is your target population for the electronic 8 records era? 9 A. Well, the target population ultimately will be all of the 10 records. 11 Q. Do you have a population number in mind with regard to 12 transactions? 13 A. I don't think we're ready -- we're finished yet. I don't 14 think Michelle is finished yet, so I don't know the number. 15 Q. Have you heard the term 28 million? 16 A. I don't think that's the number, but no, I haven't. I don't 17 know what that means. There's lots of numbers out there. 18 Q. Okay. So with respect to -- you don't have a target 19 population of transactions yet, do you, for the electronic 20 records era? 21 A. That's correct. It's coming close, though. 22 Q. Do you have an understanding of the number of transactions 23 for the paper records era? 24 A. There have been estimates made. In fact, I'm one of the 25 people that made those estimates. And they're subject to a 1039 1 great deal of uncertainty. 2 Q. So is your answer you don't know what the number of 3 transactions are at this time? 4 A. There are -- estimates have been made. The answer is no, I 5 don't know. 6 Q. Thank you. 7 Is it true that the further back in time you go, there 8 is a greater problem with regard to finding records? 9 A. I don't know that. 10 Q. You don't know that? 11 A. In fact, may I conjecture? 12 THE WITNESS: Am I allowed to conjecture, Your Honor? 13 THE COURT: No. If you don't know, you don't know. 14 BY MR. GINGOLD: 15 Q. With regard to the determination -- or your hypothesis, I'm 16 sorry, that the paper records era has similar characteristics to 17 the electronic record era transactions, you stated, I believe, 18 that the meta-analysis is a minor part of your hypothesis. 19 Correct? 20 A. That's correct. 21 Q. And I believe you stated -- and this is not a direct 22 restatement of what you testified to, but I think you've 23 referenced the GAO settlements package. Is that correct? 24 A. GAO Treasury settlement packages, yes. 25 Q. Right. Is that a minor part as well? 1040 1 A. It actually plays somewhat more of a role, I would say. 2 It's pretty good work. We liked what we saw, but it's still a 3 minor role, yes. Still minor. 4 Q. So the meta-analysis is minor and GAO's settlement packages 5 are? 6 A. Also minor. 7 Q. And you then I think stated reservation histories. Correct? 8 A. Correct. 9 Q. And is that also minor? 10 A. Yes, also small. Yes. 11 Q. So you identified three particular points that support your 12 hypothesis that are all minor. What is the major basis of your 13 hypothesis? 14 A. Of course, those are additive in my -- they are additives. 15 They all add up to -- the word "minor" is -- I would use the 16 word "small." 17 The fourth one I mentioned to you is that we've been 18 doing some early pilot activity, okay, to see whether what we're 19 saying about the paper ledger era is going to turn out to be the 20 case. 21 Q. And the pilot work is representative, do you believe? 22 A. No, it's not representative at this point. We're still 23 learning how to do this. 24 Q. Okay. So is this part of the LSA? 25 A. No. 1041 1 Q. So this is independent of the LSA? 2 A. It's independent. 3 Q. So you've identified four factors which are minor. They all 4 add up to be your conclusion. Is that what that is? 5 A. They make it possible for us to support the idea of starting 6 with a test rather than with a large sample. We might end up 7 with a large sample at the end. 8 Q. So you've taken samples right now in your pilots to allow 9 you to reach the conclusion that the hypothesis is correct. Is 10 that fair? 11 A. We haven't got that far yet, but we will get that far. 12 Q. So you haven't done a sample? 13 A. We're offering the plan. We haven't completed it. 14 Q. What I'm asking you is a little different, though. I'm 15 asking you, your hypothesis, as I understand it, is that the 16 paper ledger era is similar to the electronic ledger era. 17 Correct? 18 A. That's correct. In errors, in error rates, yes. 19 Q. Sufficient for you to be able to apply whatever procedures 20 you've used in the electronic era to the paper era. Correct? 21 A. No, that's not what we're doing. We're starting with a 22 different idea altogether. The hypothesis testing is not -- 23 we're not making estimates as our main focus, we're trying to 24 look at whether -- what we're going to learn is similar in error 25 rate, not in process, to what we did earlier. 1042 1 Q. So it has a similar characteristic, and that characteristic 2 is -- 3 A. At the end of the day, yes. 4 Q. And you haven't taken any samples yet. Correct? 5 A. No. 6 Q. So what you've done -- I'm sorry. Go ahead. 7 A. My apology. Not representative samples of sufficient size 8 to make us feel comfortable at this point statistically. 9 Q. Have you taken samples of any size? 10 A. Yes, we have. 11 Q. Where have you done that? 12 A. We've been looking at them. 13 Q. Beg your pardon? 14 A. We've been looking at them. We went out and got some 15 ledgers. I didn't do this myself. We obtained some ledgers and 16 we've examined the information there. 17 We need to bring, I think -- and we're moving away from 18 my expertise here. We need to bring the accountants together to 19 look at whether the Accounting Standards Manual needs to be 20 modified as you move backward in time to the paper era. That 21 hasn't happened. 22 Q. That hasn't happened yet? 23 A. No. 24 Q. And whatever you're doing, whether it's the electronic 25 ledger era or the paper ledger era is dependent upon the results 1043 1 of what the accountants and other consultants do and conclude. 2 Correct? 3 A. Yes. 4 Q. And if their work happens to be wrong, then your hypothesis 5 may need to be changed. Correct? 6 A. The answer is yes. Although I might add, if I might add, 7 that there's a lot of checking going on in this process. It's 8 not a single accountant, or accounting firm, even, making these 9 decisions. 10 Q. No, but the result is the same. If the accountants and the 11 consultants are correct, that will help support your hypothesis, 12 and if they're not, you have to make changes. Correct? 13 A. That's correct. 14 Q. With regard to the meta-analysis, since it's one of the four 15 you've identified as supporting your hypothesis, you have stated 16 in your rebuttal report and you've stated in other reports - 17 that's you and NORC - that you've reviewed approximately 900 18 studies. Is that fair to say? 19 A. The word "reviewed" needs to be modified. We've discovered, 20 we've identified about 900 studies. We've reviewed I think 800 21 something. I don't have that number in my head. 22 Q. So the references to 900 in the various reports only refer 23 to the fact that you've identified them, not reviewed them. 24 Correct? 25 A. You have looked at the reports. You know the answer to that 1044 1 is yes. 2 Q. If you can please answer the question. 3 A. My answer is yes. Yeah, we tabulate exactly what we did in 4 the reports. 5 Q. Does anyone in your firm review -- I mean, did your firm, 6 either individually or collectively, review all of the reports 7 that you identified? 8 A. No one person looked at all the reports. 9 Q. Collectively, did NORC look at all the reports? 10 A. Remember, some of the reports that we identified as existing 11 were ones we could not find. They're out of file. And some of 12 them that we found, after quick looking were simply not 13 applicable. And the ones that were applicable were reviewed and 14 analyzed, and are in the report. 15 Q. And did you, yourself, review any of the reports, Doctor? 16 A. A couple dozen, yeah. 17 Q. Were they judgmentally selected for review or randomly 18 selected? 19 A. Some in early going were randomly selected. Then I began to 20 ask questions, do you have anything like this, anything like 21 this, and people would bring things to me. 22 Q. Did your firm create a database with the reports and the 23 summaries that were prepared for the reports? 24 A. The reports were scanned. Okay? And a summary and a 25 database was created, a partial database was created. The work, 1045 1 as you know because we've had exchanges of correspondence, was 2 not finished in terms of a complete database. In one of our 3 reports, said we were going to do this, but it fell in priority 4 such that it has not been finished. 5 MR. GINGOLD: For the record, the correspondence was 6 between the Department of Justice and plaintiffs' counsel. 7 Dr. Scheuren was not involved in it, as far as I know. 8 BY MR. GINGOLD: 9 Q. With respect to the database, it was not then utilized as a 10 searchable database by NORC. Is that a fair statement? 11 A. It was used but not as a searchable database, no. 12 Q. So any suggestions that it was may not be precisely correct. 13 Correct? Any suggestion that it had been used as a searchable 14 database is not correct. Is that a fair statement? 15 A. We didn't say it had. 16 Q. We can go into that later, but I just want to make sure I 17 understand what the facts are. 18 MR. GINGOLD: I would like to identify a document, it's 19 called -- it's NORCMA, underline, and the last three digits are 20 231. 21 BY MR. GINGOLD: 22 Q. Doctor, do you recall seeing this particular -- this is -- 23 for purposes of clarity, this is a summary of what appears to 24 have been performed by your firm with regard to one of the 25 documents that was produced as a meta-analysis document. 1046 1 Have you ever seen this? 2 A. I've seen this template before. I'd have to look carefully 3 to see if I had seen this specific document. 4 Q. The template is a template that was prepared by your firm. 5 Is that correct? 6 A. Correct. 7 MR. GINGOLD: And I would like to go to what would be 8 identified on the Bates number as 234. 9 BY MR. GINGOLD: 10 Q. Did you recall seeing this document at all, Doctor? 11 A. I don't have any recollection of that. 12 Q. Was it a problem in your firm when you were reviewing these 13 documents that the documents were missing page numbers? 14 A. We discussed that in our analysis. And we separated those 15 where there were page numbers missing from the source -- quote, 16 "source document" on original. They weren't always original. 17 That was the problem from those where there were significant 18 gaps. In those that there were significant gaps, we made them 19 out of scope. We couldn't analyze them. But when there was 20 just a page or two here and there, we were able for our purposes 21 to be able to make a determination. 22 Q. Do you have an understanding of how many documents were 23 missing pages? 24 A. I have an understanding, and we have reported how many were 25 sufficiently incomplete that we couldn't use. 1047 1 Q. Do you remember -- 2 A. Those which have a random page here or there missing, I 3 don't have that estimate. 4 Q. But the significant pages that were missing, how many 5 documents, do you know? 6 A. It's in the report. I don't have that in front of me. 7 Q. You also said, I believe, that you had an accountant 8 review -- 9 A. We did. 10 Q. -- the reconciliations. What is the name of the accountant? 11 A. Carolyn Haase. 12 Q. Could you please spell that? 13 A. H-A-A-S-E. 14 Q. And what firm is Carolyn Haase with? 15 A. She had been controller at a small law firm, and we asked 16 her to come to work for us to help us do this. She wanted to 17 make a transition, she agreed to. She came with us for eight or 18 nine months, I forgot how long, and then she worked for Deloitte 19 for another year on the historical accounting. 20 And now she is a controller, maybe chief financial 21 officer, I'm not sure what the title is, in another small firm. 22 Q. And she's a certified public accountant? 23 A. She is not. She has a master's degree in accounting. 24 Remember, she didn't do any accounting, all she did was help us 25 categorize these reports. 1048 1 Q. Well, I just asked whether or not she was a certified public 2 accountant. 3 A. I've answered that. 4 Q. Is it your understanding that it is important that computer 5 systems have adequate document controls in order to rely on the 6 data housed in those systems? 7 A. I'm not sure what you're going to here. But computer 8 systems need to be adequately documented, yes. 9 MR. GINGOLD: I would like to call up Bates 5520-1, and 10 I would like to go to a document which is 12, page 12 of the 11 document. 12 THE COURT: You're calling up documents by Bates 13 numbers. They're not in the administrative record. I don't 14 know whether they have exhibit numbers or not. 15 MR. GINGOLD: The only thing we have is a Bates number 16 out of the administrative record, and it's 5520 -- 17 THE COURT: You're just blowing them by the witness on 18 the witness stand. They're not going to be exhibits. If you're 19 going to move them into evidence, you need to put exhibit 20 numbers on them so we can keep track of them. Are they on your 21 exhibit list? 22 MR. GINGOLD: Yeah, these are from the administrative 23 record. 24 THE COURT: Oh, from the administrative record. All 25 right. 1049 1 MR. GINGOLD: That's all we're using, Your Honor. 2 THE COURT: That's fine. Okay. 3 MR. WARSHAWSKY: Your Honor, just for clarification, in 4 some cases, I think, for example, the NORC document that you 5 referred to earlier, some of those were not, I believe, in the 6 administrative record. But the Bates numbers that he's utilized 7 right now, this one is. I can certainly give the AR -- 8 THE COURT: If somebody says Bates number, I'm to 9 understand that it's in the administrative record? 10 MR. GINGOLD: It's my understanding that everything we 11 received in the administrative record is Bates stamped. 12 MR. WARSHAWSKY: And just to be clear, you used a 13 document with a number NORC, I think, earlier? 14 MR. GINGOLD: That's correct. 15 MR. WARSHAWSKY: Was that in the administrative record? 16 MR. GINGOLD: No, it was not. That was subsequently 17 produced. 18 MR. WARSHAWSKY: Right. That's what I wanted to be 19 clear about. Where you have a Bates number, I can certainly 20 give a cross reference to the administrative record here. I 21 have that here. I can look it up. 22 MR. GINGOLD: Your Honor, what we will do -- 23 THE COURT: Do you think Bates has anything to do with 24 these numbers anymore? 25 MR. GINGOLD: Your Honor, there were approximately 1050 1 17,000 pages of documents in the administrative record that were 2 produced. When we input the information in the system, it was 3 input -- it was not segregated other than by Bates number at the 4 time we received the information. 5 There have been some subsequent productions, but it was 6 a significant process. 7 THE COURT: Fine. I was just musing on whether Bates 8 really deserves to have his name on all those numbers that we're 9 using. 10 THE WITNESS: Clearly from a different century. 11 THE COURT: Really, it was. And I remember that 12 century. Go ahead, Mr. Gingold. 13 MR. GINGOLD: It was only a couple of years ago, 14 Your Honor. 15 BY MR. GINGOLD: 16 Q. Let's look at page one. Page one of the document, 17 Dr. Scheuren -- 18 A. How do I know what I'm supposed to have in front of me? 19 Q. This is page one. 20 Now, do you recall sending this e-mail 10/3/01 to 21 Bert Edwards? That's October 3, 2001. 22 I would like you to turn -- 23 A. I need a little longer with this, please. This is a long 24 time ago. Yes. 25 MR. GINGOLD: If we can please call up page 12. I 1051 1 would like to focus on the last paragraph. 2 BY MR. GINGOLD: 3 Q. Do you see, Doctor, the head note is, "period of Legacy 4 (1985 to 1998) or existing (1998 to date)." Do you see that? 5 A. Yes, I do. 6 Q. And this refers to computerized accounts. Correct? 7 A. Yes. 8 Q. I would like you -- let's see, one, two, three, four, five 9 lines from the bottom. Did you write this, by the way, Doctor? 10 A. Yes, I did. 11 Q. Okay. And it states as follows: "To make full use of these 12 computer files for a historical accounting, we would need to 13 check on how the continuing accounting data are being collected 14 now and what is being done to limit data capture and other 15 errors. How are such errors prevented in earlier legacy 16 systems? What procedural handbooks and documented system checks 17 exist? Can a conformance audit be used rather than relying 18 mainly on a search for supporting transaction detail? Bottom 19 line, here is how tight are, (were), the controls for these 20 computer systems, and hence, how much reliance can be placed on 21 them?" Close, question mark. 22 Do you recall saying that? 23 A. Yes. It's actually in the first predesign document that 24 NORC has. I don't know where this came from, but the final 25 version was in the predesign. 1052 1 Q. So that statement is correct, isn't it? 2 A. I said that. 3 Q. And you believe it was correct then and it's correct now. 4 Correct? 5 A. That's a little bit of an interest to say. The answer is 6 basically it's correct, still. 7 Q. Thank you. 8 A. But the issues there are different now than they were then. 9 Q. Can you tell me if it is no longer true that "the bottom 10 line is how tight were the controls for these computer systems, 11 and hence, how many reliance can be placed on them?" Is that 12 true? 13 A. What we're talking about here is a contrast between two 14 different approaches to doing this work. One is conformance 15 audit, which if the controls were tight enough, you wouldn't 16 have to do a reconciliation. 17 And the other is, if you're not sure about the 18 controls, you can do -- you have to do a reconciliation. Much 19 more expensive process. 20 Q. So basically what you're still saying here isn't -- remains 21 to be true, isn't it? What you said here is true today? 22 A. There were two approaches, and that is true. There were two 23 approaches. 24 Q. Thank you. 25 Now, do you, yourself, have any knowledge whether or 1053 1 not the IRMS system has had sufficient controls? 2 A. I do not. 3 Q. Do you know if any of the other systems that house the data 4 had sufficient controls? 5 A. Not in the way you're using the phrase, no. 6 Q. How about the way you're using the phrase? We're using your 7 terms now. 8 A. The answer would still be the same. I do not know. 9 Q. Did anyone inform you whether or not they had sufficient 10 controls? 11 A. No. We did undertake to do some testing of the systems, 12 okay, that we're looking at to see what kind of results we could 13 have expected from them, but the issue of control is really 14 beyond our scope. 15 Q. It's beyond your scope? 16 A. That's correct. 17 Q. And you're only making conclusions about the consistency of 18 the information, not about the accuracy of the data. Correct? 19 A. We're going back to that again? 20 Q. Well, I'm just trying to understand. 21 A. My answer is, if I have a transaction that I'm testing with 22 supporting documents, and I find the supporting documents, okay, 23 even though it's very difficult to find them, okay, but I do 24 find them, okay, I'm going to be -- if I'm an accountant -- 25 Q. Doctor, you're not an accountant. We're talking to you 1054 1 about your opinions -- 2 MR. WARSHAWSKY: Objection, Your Honor. The witness -- 3 THE WITNESS: You're asking me a question outside of my 4 scope, then. 5 THE COURT: Let him finish his answer, counsel. 6 THE WITNESS: May I finish? 7 THE COURT: Yes. 8 A. And the accountants actually are pulling this information 9 from a system, okay, and they're making a comparison with a 10 transaction that's on an electronic database, and they end up 11 concluding that the two match, okay, or that they don't match. 12 But they end up concluding something that they believe is true. 13 That's what I relied on when we did our work. 14 If they had a different kind of system, then you could 15 use a different kind of approach, but that kind of approach was 16 not judged suitable. 17 BY MR. GINGOLD: 18 Q. So the answer is the accountants or consultants are making 19 those conclusions. Correct? 20 A. That's correct. 21 MR. GINGOLD: I'd like to turn to what is identified as 22 Bates number 421. 23 BY MR. GINGOLD: 24 Q. And I would like to first point out the first page, which is 25 page one. This is a January 29th, 2007 memorandum from you to 1055 1 Jeffrey Zippin. 2 A. Yes. 3 Q. Do you recall this, document? 4 A. Yes, I do. 5 MR. GINGOLD: I'd like to turn to Bates number two, 6 footnote one. 7 BY MR. GINGOLD: 8 Q. This is similar to what you testified in response to a 9 question from Mr. Warshawsky. 10 A. Yes. 11 Q. Do you see that? 12 A. Yes, I do. 13 Q. It talks about when you were at the IRS and how you viewed 14 taxpayers as whether or not they were trustworthy? 15 A. Yes. This is -- 16 Q. And what did you say in this footnote, Doctor? 17 A. In our IRS background, all instances of unsupported 18 transactions could be disallowed. 19 Q. And why is that? 20 A. Pardon me? 21 Q. Why is that? 22 A. Because there was no trust in the taxpayers involved. 23 Q. Does that also apply to systems, if the systems are 24 untrustworthy? 25 A. That wasn't the context in which I used this. 1056 1 Q. I asked you if it also applies to the systems. 2 A. I don't have the answer to that question. That's not what 3 we were addressing here. 4 Q. But you've made -- I think you've made statements in your 5 opinion, or rebuttal opinion, and also in other reports that 6 you've filed with the Department of Interior that I think you 7 found that the systems have been performing, or there's no 8 evidence that they haven't been performing. Do you recall 9 making those statements? 10 A. I do. 11 Q. And are you referring in that regard to the paper ledger era 12 when you made those statements? 13 A. In that particular context, which is I'm referring to the 14 system as a whole, yeah. 15 Q. So that would include both the paper and the electronic? 16 A. Yes, sir, yeah. 17 MR. GINGOLD: I would like to call up an exhibit which 18 is identified as PPX4174. 19 BY MR. GINGOLD: 20 Q. Doctor, just for clarification, this was not one of the GAO 21 reports that was included in your meta-analysis. Have you ever 22 seen this before? 23 A. I'm not sure. What's the date on it, 1982? 24 Q. 1982, that's correct. So it is shortly before the defined 25 electronic ledger era. Do you see that? 1057 1 A. Uh-huh. 2 Q. Doctor, I would like you to turn to what would be the 3 page 2-I of this exhibit. And there's a head note. It says, 4 "Causes of the System's Breakdown." Do you see that? Sorry. 5 A. Thank you. 6 Q. And I would like to read the first sentence. "GAO 7 identified two basic causes for the breakdown of the accounting 8 system." 9 I'm trying to move this a little rapidly, but this 10 deals with the Individual Indian Trust, and also the Tribal 11 Trust systems. 12 MR. GINGOLD: If the Court would like me to go through 13 them in more detail, I will. 14 BY MR. GINGOLD: 15 Q. But in any event, were you aware that at least the 16 Controller General in 1982 believed the accounting systems had 17 broken down? 18 A. I'm aware of the general work done by other organizations 19 who have audited this system over the years. Okay? And -- 20 Q. So you're aware there were at least identifiable problems. 21 Correct? 22 A. Oh, yes. 23 Q. But you hadn't been told the systems had broken down, were 24 you? 25 A. I remember that phrase, yeah. 1058 1 Q. I would like to turn to page 3-I, second paragraph from the 2 bottom above the head note. And I would like to read the first 3 part of it, the first few, I think, five lines of the paragraph. 4 "These matters have been pointed out repeatedly over a 5 number of years by GAO, Interior's inspector general, and 6 various system study groups. Bureau managers were well aware of 7 the problems and acknowledged that the information in the 8 automated accounting and finance system was unreliable." 9 Had you been informed that by your client, the 10 defendant? 11 THE COURT: Mr. Gingold, this is a 25-year old 12 document. Right? 13 MR. GINGOLD: That's correct, Your Honor. 14 THE COURT: I gather this is all in aid of your 15 proposition that if was once unreliable, you can consider it 16 unreliable for all purposes, since you used to work for the IRS. 17 Is that what you're saying? 18 MR. GINGOLD: Not entirely, Your Honor, no. It was 19 unreliable at that point in time. If it has ever become 20 reliable, it should be viewed as reliable from that point 21 forward, but not for the historical data. 22 THE COURT: So you're focused on old-fashioned 23 pre-electronic era information? 24 MR. GINGOLD: Your Honor, that was -- your hypothesis 25 with regard to the paper records era, as I understand 1059 1 Dr. Scheuren, is that the characteristics in the paper era are 2 similar to the electronic era. 3 THE COURT: And you want him to say that was not a 4 reasonable hypothesis because of all this old stuff from the 5 GAO. Right? 6 MR. GINGOLD: Well, the question is whether or not his 7 hypothesis is supportable based on four points that he 8 identified which support his hypothesis. 9 THE COURT: Well, you know, rather than showing him 10 these 25-year old words, seems to me that you ought to stick 11 within his expertise, which is statistics. I think he's told us 12 where the hypothesis came from. It came from the dollars. You 13 try the hypothesis, if it works, fine; if it doesn't work, you 14 look at something else. 15 We keep going back to these old documents, which 16 frankly aren't doing much for me. Let's break for lunch. 17 (Recess taken at 12:31 p.m.) 18 19 20 21 22 23 24 25 1060 1 CERTIFICATE OF OFFICIAL COURT REPORTER 2 3 I, Rebecca Stonestreet, certify that the foregoing is a 4 correct transcript from the record of proceedings in the 5 above-entitled matter. 6 7 8 9 _______________________________ _________ 10 SIGNATURE OF COURT REPORTER DATE 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25