AMERICAN STATISTICAL ASSOCIATION COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION DAY 2 OF 2 WASHINGTON, D.C. FRIDAY, APRIL 20, 2007 2 1 PARTICIPANTS: 2 NAGARAJ K. NEERCHAL, Ph.D., Chair 3 DEREK BINGHAM, Simon Frasier University 4 EDWARD A. BLAIR, University of Houston 5 JAE EDMONDS, Ph.D., Pacific Northwest Laboratory 6 MOSHE FEDER, D.Sc., Research Triangle Institute 7 BARBARA FORSYTH, Westat 8 WALTER W. HILL, St. Mary's College of Maryland 9 NEHA KHANNA, Binghamton University 10 EDWARD KOKKELENBERG, Ithaca, New York 11 12 13 * * * * * 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 3 1 P R O C E E D I N G S 2 (8:40 a.m.) 3 DR. NEERCHAL: What I'd like to do 4 -- I think you start with -- I will ask any 5 Committee member who is not here and -- but 6 here today to go ahead and introduce 7 themselves and also the audience members, 8 anybody in the audience who is -- who was not 9 here, but who is here today to go to the mike 10 and introduce themselves. 11 And also I want to remind you that 12 please register and put your e-mail down when 13 you're coming in because of -- on the 14 registration table there. 15 Any new persons here? I do. 16 MS. CARLSON: Mary Carlson. 17 DR. NEERCHAL: You have to come to 18 the mike. 19 MS. CARLSON: All right. 20 SPEAKER: You have like those 21 foreign national fingerprint IDs to get into 22 the building. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 4 1 MR. BURNS: Hello. I'm Eugene 2 Burns. I'm with the Office of Energy Markets 3 and End Use in the Energy Consumption 4 Division. 5 MR. CONRAD: I'm Lenny Conrad with 6 SMG. 7 MS. BATTLES: I'm Stephanie 8 Battles. I'm Director of the Energy 9 Consumption Division. 10 MS. CARLSON: Mary Carlson, Office 11 of Oil and Gas, Natural Gas Division. 12 MS. MICHAELS: Joelle Michaels, 13 Energy Consumption Division. I'm the survey 14 manager for the Commercial Buildings Energy 15 Consumption Survey. 16 MS. O'BRIEN: Eileen O'Brien, 17 Survey Manager of the Residential Energy 18 Survey in the Energy Consumption Division, 19 EMEU. 20 MR. SWENSON: Alan Swenson, Energy 21 Consumption Division. 22 DR. NEERCHAL: Before we start off BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 5 1 with (off mike) today, I think we should take 2 a moment and remember the people who are 3 affected by the Virginia Tech tragedy. Today 4 is a national day of mourning, and, you know, 5 we all feel for them and that our colleagues, 6 and, you know, fellow citizens of Virginia 7 there. So maybe we should observe a moment 8 of silence and then. 9 (Moment of silence.) 10 DR. NEERCHAL: Thank you. Today's 11 first -- we have two plenary sessions today, 12 and the first plenary session speaker is 13 Jacob Bournazian? 14 DR. BOURNAZIAN: Yes. 15 DR. NEERCHAL: And he's going to 16 talk about the microdata. 17 DR. BOURNAZIAN: Yes. 18 DR. NEERCHAL: Okay. 19 DR. BOURNAZIAN: Well, Members of 20 the Committee on Energy Statistics, Fellow 21 EIA staff and guests, I'm happy to be here 22 this morning because this topic is an area BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 6 1 that's developing and it's on a fast pace 2 with the federal statistical agencies. 3 It's also an area that's been 4 growing with the federal statistical 5 agencies, and it's a long time since we've 6 been able to talk about this here at EIA, and 7 so my presentation today really comes at a 8 timely moment. We're talking about expanding 9 research or access to microdata or EIA's 10 microdata. 11 All right, Larry. EIA is 12 considering three options for expanding 13 research or access. There's a program that 14 Ruey-Pyng spoke about yesterday afternoon 15 with the National Institute on Statistical 16 Sciences. How many Committee members heard 17 Ruey-Pyng's presentation? 18 One, two, three, four. Okay. A 19 majority of you. All right. So I'll be 20 going each one of these options in detail, 21 but the NIST-EIA student research program, a 22 nice program on site here at the Forrestal BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 7 1 Building. I'll be going over that again. 2 A second option, though, is to 3 place our residential energy consumption 4 survey data and our commercial building 5 energy consumption survey at the Census 6 Bureau's RDC facilities. 7 There's nine facilities and the 8 Census Bureau conducts the Manufacturing 9 Energy Consumption Survey, MECS, which is 10 already housed at the Census Bureau's RDC. 11 So by placing these two surveys with the 12 Census Bureau, you'd have the whole sweep of 13 consumption surveys in one RDC facilities set 14 up. 15 There's a third option, and that's 16 to place the same two surveys at the National 17 Opinion Research Council's data enclave 18 that's going live May 1st, in a few weeks. 19 The big feature of NORC's data 20 enclave program is remote access, where the 21 researcher stays right there in their office, 22 right from their PC, dials in and can do the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 8 1 research. 2 It's a private VPN network. No 3 cutting and pasting allowed, no downloading, 4 no printing. You just -- you can do your 5 research and connect to those company-level 6 data files. 7 So we'll be talking about these 8 three options and at the end of my 9 presentation, I'm going to be asking you for 10 some input to help us evaluate which is most 11 appropriate and be effective for EIA. 12 Now, EIA has never engaged in this 13 activity to any significant extent as far as 14 allowing researcher access to the 15 company-level files. But there are benefits 16 and many federal, if not all of the federal 17 statistical agencies have moved in this 18 direction for this very reason. 19 There are substantial benefits by 20 allowing researcher access. First, you're 21 fostering an open research environment. 22 Basically, this benefit here is replication BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 9 1 of your results, the transparency of the 2 data, and of the information you're putting 3 out. 4 However, in every single agency's 5 case we've talked with that has allowed 6 researcher access, they found another 7 by-product of benefit here: It improves data 8 quality. Well, if you heard Ruey-Pyng's 9 presentation, he mentioned a few selected 10 topics dealing with statistical methodology. 11 Agencies learn about their data by 12 allowing researcher access. They evaluate 13 their statistical methods, but they also 14 learn a great deal about their data. In 15 fact, talking with RDC Administrator, a lot 16 of outliers, misreporting of problems with 17 the data get identified. Not that you can do 18 anything about it five years after the fact, 19 but you learn about your data and it has -- 20 and it impacts you on your analysis. 21 Now, in the old days at EIA, we 22 used this, okay? We went down through BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 10 1 output, company by company, and we analyzed 2 what those companies reported. 3 We don't do that anymore. Resource 4 constraints. We have automated edits. We're 5 going to sort the data, look for companies 6 are sufficient distance away from the crowd. 7 We're going to look at what companies make an 8 impact on a cell. We don't have time to call 9 three -- make 300 phone calls to validate our 10 data, and the automated edits work well. 11 But what does a researcher do? 12 Researcher wanders right into the forest, 13 looks at every tree, every branch, leaf on 14 the, you know, branch. Right, he's going to 15 look at a small company's reporting history 16 regardless if it falls in the geographic area 17 they're interested in. 18 So if every other agency has 19 realized these benefits, no doubt EIA will 20 benefit by the same way. 21 Strongest reason down here at the 22 bottom: Microdata they permit in-depth BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 11 1 analysis for many policy-relevant issues that 2 the agency just is not able to analyze. We 3 don't have the resources to do that. As 4 Ruey-Pyng pointed out on his research 5 projects, there's a lot of methodology 6 issues. We just don't have the time. 7 They're wonderful research projects, but 8 staffing and time is limited. 9 However, if you heard Guy Caruso's 10 opening remarks yesterday, he talked about 11 service reports. Well, we get a lot of 12 questions, besides questions from Congress. 13 We get questions on whether electric power 14 producers from Montana are leveraging 15 residential electricity consumers in Idaho. 16 You got questions why gasoline prices in 17 Georgia are more expensive than Florida when 18 Georgia's tax is only seven cents and 19 Florida's is 20. 20 Okay. We don't have time to get 21 into that real local state, not local, but 22 state analysis. In fact, what Guy said was BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 12 1 he has to prioritize. He can't do every 2 single service request. And what is he 3 doing? 4 He's regionalizing, nationalizing 5 it, and if there are other requests that 6 match up, because we're going to focus on the 7 national supply picture or regional. 8 Who is going to be able to analyze 9 the difference between Arkansas and 10 Tennessee? Maybe a professor from the 11 University of Tennessee would be interested 12 in that, but certainly EIA doesn't have the 13 analysis resources to go after every single 14 analysis question. So we would benefit a 15 great deal by using our data in this fashion. 16 Okay. Now I've made this point in 17 my paper, but I have to admit I probably 18 didn't do it effectively because my 19 discussant approached me at lunch time 20 yesterday and he said, you know, what do you 21 mean by this statement expanding researcher 22 access is going to increase EIA's investment BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 13 1 in the data collection. In fact, I used Ray 2 to return to my paper. All right? 3 Well, I promised that. I said, I'm 4 going to make it up to the whole Committee 5 this morning with this one slide. And I may 6 have failed in my paper; we'll come back with 7 a picture. 8 Well, we looked at where EIA spends 9 its resources. Now, Guy Caruso mentioned 10 yesterday that it was a strong wind blowing 11 through Congress yesterday or this year and 12 we got $90 million in the budget. 13 We didn't always have that strong 14 wind. Okay. Budgetary financing was more of 15 the '80s. Let's round down to $80 million. 16 And what's two- thirds of $80 million? I 17 think it's a little over $50 million, but I'm 18 going to round down to $50 million. And Guy 19 also mentioned that we've been around for 20 what? Thirty years? What's 30 times $50 21 million? $1.5 billion, to be exact. Where 22 are our resources and money going? It's BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 14 1 going to collect data. We processed the data 2 and now we have a bunch of master files with 3 company-level data on there sitting on the 4 shelves. Nobody is using it. We've got the 5 aggregate tabular data that's been released 6 and we're real good about that. We've got 7 online query systems that get you that 8 aggregate tabular data right from the web. 9 But nothing, nothing is going on with our 10 company-level data. It sits there, year 11 after year, decade after decade. I don't 12 even know if we could link up with our 13 company-level files from the '80s to even do, 14 you know, some long-term analysis with the 15 files in the '90s or even in this decade. 16 But we just keep on going out, 17 collecting, compiling, and stacking on 18 shelves, and it sits there. It sits there, 19 and so I'm going to point this out, because 20 I'm going to be talking about some options 21 that are cost money, tens of thousands of 22 dollars. I'm going about spending tens of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 15 1 thousands of dollars, talking about 2 increasing our data utility. 3 And when you compare the tens of 4 thousands of dollars I'm going to ask this 5 agency to spend, compared to the opportunity 6 cost of having over a billion dollars of 7 resources sit there, the benefits should be 8 there and the money as well worth spending. 9 Now, I hoped that addressed on what 10 I mean by rate of return, because Ed's a New 11 Yorker. Right? So what does he say in the 12 lunch line? Are you looking for seven 13 percent, eight percent -- the typical New 14 York question. And I thought about it on my 15 second pass through the linguini line is that 16 there's plenty of metrics. We could look at 17 users' accessions to the files. We could 18 look at research proposals. We could look at 19 products produced. You could even look at 20 citations on the research. There's many 21 metrics we could develop to measure the 22 usefulness and utility of this activity. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 16 1 Okay. Well, then why haven't we 2 just run out there and done this? I mean if 3 this is such a great activity and every other 4 federal agency is doing it, why aren't we out 5 there with the pack? 6 Well, you got some risks. And so 7 let's talk about those risks 'cause you want 8 to -- that's one reason, main reasons, why we 9 have not engaged in expanding researcher 10 activity. 11 Well, this is a general policy that 12 there's concerns over confidentiality. 13 Across federal statistical agencies, 14 confidentiality impacts response rates and 15 refusal rates, well documented in the 16 research. One thing that's unique about EIA, 17 we've got mandatory data collection 18 authority. You have very high response 19 rates, and so even though there can be 20 confidentiality problems, our response rates 21 would never be sucking wind as bad as what we 22 see in like Census or BLS that are down to BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 17 1 the 40 percent range on voluntary surveys. 2 Mandatory surveys in general always have a 3 higher rate. 4 So do we win automatically? Can I 5 dismiss this? No, not really? Cause lack of 6 trust negatively affects data quality. 7 Actually, lack of trust negatively affects 8 communication and even in a personal level; 9 right? 10 When trust is high, easy to 11 communicate with one another. When trust is 12 low, it's very difficult. The same in a 13 business setting. Okay. So what happens to 14 EIA if we cause a breach in confidentiality 15 or engage in a program that diminishes and 16 denigrates our pledge? 17 Well, I'll tell you one thing: 18 Volume data is going to get rounded down and 19 cost data is going to get rounded up; right? 20 That's what we get. EIA is going 21 to force me to report, you'll get your file 22 and report on time. Is it going to be the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 18 1 right data? Maybe not. Is that going to 2 impact us? Yeah. That's real important. We 3 have a good reputation for accuracy. 4 There's another common risk. The 5 perceived risk of a breach can be just as 6 damaging as an actual breach. Again, it's 7 dynamics of human relationships; the same in 8 a business setting. 9 However, there's one caveat on the 10 perceived risk. It's how you handle it. 11 It's how you handle the perceived risk versus 12 an actual breach. An actual breach you're 13 stuck. You're burned on that one. But on a 14 perceived -- how do you respond could 15 boomerang and be a huge deposit with your 16 trust and goodwill with your respondents. 17 Getting cast in a false light and then 18 correcting it with the truth always comes 19 back for strengthening that bond. 20 However, it's very important to 21 handle on your -- we have certain rules now 22 and notification of breaches and then BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 19 1 policies and procedures. 2 Here's a huge, another big risk: 3 EIA may get blamed for bad research. Now you 4 researchers are a suspect group. Why? 5 Because you're intermittently playing around 6 with the data. You come in and you want to 7 do energy studies. Then you want to do a 8 labor study. And then you come back and do 9 another energy study. Whereas hard-core 10 staffers, consultants, well, they're always 11 in the energy industry; right? 12 Are you guys going to get it right 13 this time? Wow, what if you get it wrong and 14 EIA gets blamed 'cause, after all, you used 15 EIA data. Mm. 16 Well, federal agencies handle it 17 two ways: Disclaimers and or extensive 18 disclosure -- product review. And you can 19 pick any flavor or a combination of the two. 20 A very interesting observation. 21 Actually this one here has led some program 22 officers to altogether pull out and say we're BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 20 1 not going to let any researchers touch the 2 data because bad research -- the ironic 3 thing, though, is on the aggregate data, and 4 I've talked about three core cases in the 5 Midwest where professors were called in as 6 expert witnesses, testified on jobbers 7 getting squeezed by refiners in the upstream 8 market; used EIA data, petroleum data and 9 won. The jobbers won. 10 The researchers did a great job, a 11 professional job of explaining that to the 12 courts and they used it correctly. 13 So somehow, maybe it's 14 intrinsically that they're able to be bright 15 and clever and on point using aggregate data, 16 but when you come to microdata, you guys just 17 mess it up every time; right? 18 You know, just like you can handle 19 magnitude, but not frequency. But it's 20 there. 21 The cost of providing access -- 22 normally that should not be a risk, except BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 21 1 for when you work at EIA. All right. This 2 is not always going to be a strong wind 3 blowing through Congress. All right? 4 And if you're going to commit the 5 agency to an annual expenditure of fifty 6 grand, ten grand, a hundred thousand dollars, 7 okay, that is a constraint. 8 And so it's a risk -- are you going 9 to be able to continue with these programs? 10 We have to look at cost. In fact, cost is 11 going to be the number one bullet item when 12 we go through these three options. 13 But I do want to put in context, 14 'cause I been talking about what other 15 federal agencies have been doing, and it's 16 pretty exciting. Remember when I opened, I 17 said this is a fast-developing area. 18 Well, if I gave this speech five 19 years ago, I'd have four agencies up here on 20 the top. But it keeps on growing. Now, it's 21 not just that little light bulbs are going on 22 and more and more agencies are getting into BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 22 1 this game and engaging in this activity. You 2 see -- all right. It's a weak laser pointer, 3 so good thing I had Chinese food last night. 4 Chopsticks seems to be in the morning. 5 Agency for Health Research and 6 Quality, they have a nice researcher access 7 program and what do they do? They host their 8 data at the National Center for Health 9 Statistics. They're -- they have an RDC in 10 Hyattsville. They pay an annual fee. Sort 11 of similar to the option I'm going to be 12 explaining for what we're considering with 13 the Census Bureau. The Bureau of Economic 14 Analysis, the Bureau of Labor Statistics 15 on-site researcher programs. 16 The Census Bureau, (off mike) they 17 have the nine research data center 18 facilities. National Agriculture Statistics 19 Service. They have a data lab right down the 20 street, and also 40 regional offices that are 21 connected on a secure LAN for researchers to 22 come in and access microdata. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 23 1 The National Center for Education 2 Statistics. Well, one of the leaders in it. 3 It started in the late '80s, mainly using 4 licensing, but now moving on. 5 The National Center for Health 6 Statistics and National Science Foundation. 7 Now, even the agencies that have 8 been in this activity for a while are going 9 through their second and third generation of 10 evolving; okay? It's not like they went and 11 got engaged in it; they're staying with it. 12 They're moving well ahead forward. 13 For example, NASS has a data lab, 14 but now NASS has also signed a contract with 15 NORC to place their data in their data 16 enclave. Why? They're looking for that 17 remote access feature. Okay. 18 Also, NCHS, they've got a research 19 data center. They signed a contract with 20 Census Bureau and placed their data at the 21 RDCs in Census. What are they doing? Just 22 increased their RDCs by nine- fold. Okay? BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 24 1 Now, besides all this, too many CDs 2 are probably ending up in researchers' 3 attics, but I don't know why, but NCES, the 4 Census Bureau, NASS, NCHS moving to online 5 query systems. You got CDC Wonder, in which 6 you can query a database and get, you know, a 7 company, household respondent-level records. 8 The Census Bureau American Fact 9 Finder query level you can do data restricted 10 queries, but you can go and access -- do your 11 research on company-level data. NASS has the 12 Agricultural Research Methods Survey. NCES 13 just started this in the last year. Besides 14 licensing, now they are also putting up, and 15 they have on their Web site their 16 company-level files. Again, buffered access 17 for approved researchers. 18 So you can see these agencies are 19 moving forward in developing multiple modes 20 for researcher access, not just one. 21 Okay. So who's been left behind? 22 The Bureau of Transportation Statistics, a BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 25 1 very small agency about one-third the size of 2 EIA. It's been around for about 10 years. 3 Internal Revenue Service and that's 4 administrative data. I don't think it's a 5 fair comparison, except they have one unit, 6 Statistics of Income. 7 The Social Security Administration, 8 once again administrative data. Now look at 9 this little agency right here. I'm not 10 saying we should do this because everybody 11 else is doing it. I'm saying everybody else 12 is doing it because there's real benefits and 13 they're growing in the area and if EIA is not 14 going to engage in it, EIA just gets left 15 behind. And that's been the story of the 16 last 10 years. 17 Okay. Can we do this? Maybe 18 that's why we've never done it. Well, yes, 19 we can. We do have the legal authority to 20 engage in it, because if we didn't, end of 21 the presentation. 22 This is what we're relying on in BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 26 1 our negotiation with the Census Bureau. They 2 require legal authority. 3 All right. Also, expanding 4 researcher access promotes EIA's mission. 5 Now EIA's mission statement is kind of 6 boring, but if you step back and you look at 7 it as a trilogy within a trilogy, it's a 8 really a great mission statement, because why 9 are we here as an agency? Well, to provide 10 the data, the forecasting work, and analysis. 11 But so why do we provide that? 12 Well, we want to promote sound policy making, 13 operating efficient markets, and a public 14 understanding. Public understanding of what? 15 Well, first of all energy. We want to 16 promote an understanding of energy -- 17 energy's interaction with the economy; 18 energy's interaction with the environment. 19 Now, when I see words like 20 interaction with the economy, interaction 21 with the environment, I'm thinking data 22 linkage. Right? That's kind of the analysis BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 27 1 that provides for that kind of work. You 2 want to link the files and really get some 3 powerful research done. 4 Now how's that going to benefit 5 this program, researcher access? Right here. 6 Promote sound policy making and public 7 understanding, two big areas we can get a 8 boost from. 9 Got some outside support for this 10 activity, too. The External Study Team 11 issued the report last year, 2006, and 12 laudable goal. They recommended that EIA 13 increase its level of interaction with the 14 research community to a level that is on par 15 with EIA's level of interaction with the 16 energy industry. Good luck. Right. Right. 17 I agree, because it's wonderful we have these 18 energy conferences. It's the only time 19 really researchers come here on site and we 20 have this kind of candid interaction. 21 The only other interaction you guys 22 doing is presentations at JSM in August, a BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 28 1 minor showing at American Public data users 2 conference, but this really is a core 3 activity for us. Compare it to EIA's 4 interaction at energy conferences. Okay. So 5 it's going to be a long time, and I don't 6 think researcher access is going to balance 7 the equation, but it's a step in the right 8 direction. 9 But this quote I felt was really 10 useful. Later on in the report, they write, 11 ideas to promote this goal, right up above, 12 include providing a program of visiting 13 scholars to bring academic researchers to 14 EIA. 15 Well, that's what Ruey-Pyng was 16 talking about yesterday. And expanding 17 researcher access to microdata made under 18 appropriate safeguards to protect the 19 confidentiality of the reporting entities. 20 No one's looking to undermine our data 21 protection pledges. No one's looking to 22 throw that away, and no one's looking to BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 29 1 undermine the goodwill and trust we have 2 built up over the last 30 years. 3 We have the resources to maintain 4 those data safeguards and yet permit this 5 activity, and they were highlighting it. 6 Yes, we recognize the importance of 7 preserving that trust and goodwill with our 8 respondent community. 9 Okay. So let' go to the three 10 options and wrap up. 11 The first option that research data 12 center at Census Bureau. A hundred thousand 13 dollars to start up; $75,000 annually. Well, 14 I just want to brag, because a (inaudible). 15 Well, we started off the negotiations with 16 $250,000 up front and a $125,000 a year, so 17 we did the best we could, but that's where 18 the Census Bureau has drawn the line in the 19 sand for offering their services. 20 It's off site. It's a full service 21 operation here. They do everything from, you 22 know, receive the proposals, monitor the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 30 1 employee when they're working there, do the 2 disclosure review, do the product review. 3 Okay. 4 As I mentioned, we already have the 5 Manufacturing Energy Consumption Survey data 6 there at the same location. Now, I was 7 talking with Dan Weinberg (ph.), who is the 8 Director of the Center for Economic Studies, 9 and he is in charge of the RDC program at 10 Census Bureau. And I said, you know, can't 11 you come down below a hundred grand because 12 quite frankly you got competition. You got 13 competition, buddy. You could lose our 14 business if you can't come down below a 15 hundred. 16 And so Dan reminds me, he says, 17 Jake, I serve on the Board of Directors for 18 NORC, and I'm helping them build their data 19 enclave. We do not view this competition at 20 all. In fact, I want to help my colleagues 21 develop their program. Okay. 22 You know why? Right here. Right BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 31 1 here. He said, Jake, Census Bureau is still 2 and will always probably have the largest 3 register of business data available. 4 So if you're a researcher and 5 you've got some interesting topic, most 6 likely this is the place that you're going to 7 need to go. So Dan's not worried about 8 competition. 9 Now we got some downsides. When he 10 was (off mike) working at RDC, you got to 11 submit a proposal to the Census Bureau. It 12 goes to their Disclosure Review Board; a copy 13 would come over to us. You got about a month 14 or two, probably two months time lag there. 15 Once you get the proposal approved, then you 16 get -- go in to getting special sworn status. 17 They're not going to waste time doing a 18 background check if your proposal is not 19 accepted. Right? This takes six months, the 20 FBI background check. 21 What are we talking about? It 22 could take (off mike) away, just like eight BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 32 1 months, eight months right down there, and 2 it's not for free. Okay. We're paying a 3 hundred grand. Researchers got pay $3,750 a 4 month. That's about the average cost of a 5 researcher. On top of it is $18,500. The 6 National Center for Health Statistics has a 7 similar fee, about $18,000 per researcher to 8 use that facility. 9 So -- it's plus or minus in there. 10 Okay. Ruey-Pyng talked about this from the 11 National Institute of Statistical Sciences -- 12 $10,000. Great program. Nice deep 13 affiliations than this has with its other 14 universities. They've got -- it would be 15 done here on site, so EIA needs to provide a 16 work area, dedicated PC, some staff to (off 17 mike). Selected research topics, so it may 18 involve other survey data besides RECS and 19 CBECS data. That's pretty appealing. Also 20 you got a list of selected research topics 21 that, you know, we can channel out for (off 22 mike). But it goes to a selected group of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 33 1 researchers, a selected group of researchers. 2 How many people in this room got 3 their Ph.D. after the year 2000? That's what 4 I'm talking about. See. That's right. See, 5 you know, if you woke up this morning, and 6 said, you know, I'm getting kind of old to do 7 a lot of things in life. When you came here 8 to this meeting, I bet you never thought 9 you're too obsolete to do any research on 10 company-level data at EIA, but, in fact, 11 you're all discriminated against. You're 12 out. You're axed. You're off the list. 13 All right. This goes to grad 14 students; right? That's why we have selected 15 topics, because that's what grad students are 16 for. They, you know, they like to do what 17 they're told. Give them a topic, and it may 18 work. But it's a great program, and it's a 19 nice little module that adds on. 20 Okay. How do we control that? The 21 researcher is an agent of EIA. It's 22 important. I'll be pointing this out in each BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 34 1 one of my options, because you have to look 2 at what control mechanisms are in place to 3 make sure there is no diminishment of the 4 confidentiality pledge. What safeguards are 5 in place? What rules, remedies, you know, 6 regs do you have? 7 DR. NEERCHAL: Jacob, just a 8 clarification. 9 DR. BOURNAZIAN: Mm hmm. 10 DR. NEERCHAL: The (off mike) what 11 this option means or include what you're 12 going to (off mike), at that time (off mike) 13 just a list. 14 DR. KIRKENDALL: (off mike). 15 DR. BOURNAZIAN: Right. How much 16 do grad students make; right? I'm just 17 joking. No. 18 DR. NEERCHAL: (off mike) 19 DR. BOURNAZIAN: No, that works 20 through NISS. Right. There is a payment, 21 right, that goes with that. 22 DR. NEERCHAL: (off mike) BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 35 1 DR. BOURNAZIAN: Right. 2 DR. NEERCHAL: (off mike) 3 DR. BOURNAZIAN: Good point, 4 Nagaraj. In fact, you might as well say $10K 5 plus, $10K plus. 6 Last option is from the National 7 Opinion Research Council. Annual fees 50,000 8 plus. Why? Because they're offering a 9 portfolio approach. They're saying, okay, we 10 got this remote access feature in which 11 researchers can use it, but what other 12 services do you need as an agency? You need 13 disclosure protection training. You need 14 file work -- coordinating with the 15 researcher, working with him on that. We can 16 give them training. How about archival 17 services? You got files from 10 years ago 18 and your documentation for that and link them 19 up. All right. Is there any archival work 20 that needs to be done? 21 Typical private sector consultant. 22 They do it cost plus; right? But it allows BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 36 1 you to choose your boutique selection of 2 services you want. But the baseline $50,000. 3 The researcher is also going to get 4 charged. I think it was about, (off mike), 5 but it was somewhere around $100 a week. 6 They were going to do it in weekly units of 7 measurement, being that when you offer it for 8 free to researcher, it's not valued as much 9 if they maybe have to pay a de minimis fee. 10 So they wanted something applied (off mike) 11 to them. 12 Oh, they do have two off-site 13 facilities -- Chicago and D.C. 14 Now, we've got to apply those data 15 safeguards, right, because we don't want to 16 change the playing field. We keep the level 17 of risk the same. 18 And so you have monitored -- (off 19 mike) argues buffered remote access. 20 We're going to apply some filters 21 to the queries and have some queries 22 restricted. Everything else at the default BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 37 1 is let the query go to the data. 2 Okay. Now, a lot of authentic -- 3 you have two-way authentication for the IP 4 addresses. Not only is the researcher 5 approved by NORC to log into the system 6 through that VPN network, there's an 7 authentication data safeguard for the IP 8 address. 9 And like I said, they have a 10 features (off mike), you can't download (off 11 mike) through the system. It's like your own 12 private VPN network (off mike) research. 13 The researcher now becomes a 14 subcontractor or agent. (off mike) the 15 contractor. Now, oh, also NASS, ERS has 16 signed up and they're putting their data 17 there. They're using a -- working as a 18 subcontractor, but EIA can go either way -- 19 subcontractor or agent. 20 Okay. I showed you some cost data. 21 That was the first bullet on every slide. I 22 just want to summarize that. NISS is coming BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 38 1 in at $10,000; NORC $10,000 plus. 2 Everybody's plus. NORC $50,000 plus except 3 the Census Bureau. I have it at $75,000, but 4 I didn't tell you it's a $100,000 in the 5 first year. They had some start-up costs 6 getting a -- purchase additional equipment 7 and service to host our data, but you got to 8 say if we engage in this activity, it's not 9 for a year or two. It's a commitment to this 10 activity. And so this really would be the 11 recurring annual cost to the agency who did 12 that. 13 You got -- (off mike) do some work. 14 It's not something we can just jump into 15 outside of the NISS program, which, like I 16 said, a nice contained module (off mike) 17 organization. 18 We got to document those files. A 19 lot of our file are (off mike) on shelves 20 (off mike) the documentation is pretty 21 sparse. If you want a researcher to work 22 with your files, they got to have the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 39 1 documentation to tell them what the file 2 structure is. 3 Okay. Researcher training has to 4 go along with this or else the research is 5 going to get bogged down real quickly. 6 Disclosure protection training can 7 be modules, a training module, but that has 8 to be addressed. You got to let the 9 researcher know how you treat confidentiality 10 at your agency, how it applies to you. 11 Cyberinfrastructure work. I've 12 told you how NORC is establishing their data 13 enclave, but, you know, every time you think 14 you got something set up and your IT people 15 have a solution for you, there's some hacker 16 out there waiting to unlock the key. It just 17 happens. It's the nature of this business 18 and in cybersecurity. EIA has received -- it 19 receives a lot of cyberattacks on a daily 20 basis (off mike) 300 in the first three 21 months of this year. But we're also getting 22 what they call this root matrices where the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 40 1 hacker comes in and places a false IP address 2 on an employee's PC. They don't know it, and 3 then can either steal data or try to corrupt 4 the database and through that the employee's 5 PC. We've had about 10 attacks on that. 6 So maybe that dual IP 7 authentication that NORC is proposing may not 8 work. What do they have? 9 They have solutions to that, too, 10 but once again more cyberinfrastructure would 11 probably be needed to put in place the remote 12 access researcher program. 13 Resource constraints. You got to 14 be looking at money and where the funding is 15 going to come from to do that. 16 Okay. So let's summarize and turn 17 it over to you Committee members (off mike) 18 want to get some input. We put on some 19 questions. 20 On these three options that I've 21 just been talking about this morning, which 22 modes of researcher access should EIA provide BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 41 1 and be providing or getting into developing? 2 Which mode of access works best with the 3 research community? (off mike) not only the 4 ability to do the research, but the type of 5 research. You remember that trilogy within 6 our mission statement regarding energy, 7 energy's interaction with the economy, 8 energy's interaction with the environment. 9 What kind of research and not only the 10 substantive areas, but also statistical 11 methodology. 12 What kind of research would people 13 be interested in, and if they had access to 14 company- level files here at EIA. 15 And then what I'd like what I'd 16 like to hear from -- if anyone is engaged in 17 accessing microdata from federal statistical 18 agency, a nice war story, your anecdote that 19 sets, laid out what you found useful or 20 beneficial or what you found as an obstacle 21 or a difficulty so that maybe EIA wouldn't 22 have to repeat that same mistake and then BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 42 1 develop once again an effective researcher 2 access program. 3 So in summarizing, I turn it back 4 to you Nagaraj, our Chair, for comments. 5 DR. NEERCHAL: Thank you. Thank 6 you, Jacob. 7 I will go to Moshe for the first 8 (off mike) discussion. 9 MR. FEDER: First of all, I wanted 10 to say great presentation. In fact, we have 11 two great ideas the last two days. Ruey-Pyng 12 spoke about the N-I-S-S plan, and Jake spoke 13 about -- and I think both are great ideas and 14 I'm looking forward to hearing more about 15 (off mike). 16 I just had one question before you 17 went to summary. Since NISS and the option 18 is relatively very cheap, why not do that and 19 consider yet another option? Is there 20 anything in the contract that would be signed 21 that would preclude the other from being 22 done? BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 43 1 DR. BOURNAZIAN: No. No, in fact, 2 that is one of the options. We can do 3 multiple modes. 4 MR. FEDER: So even if I let's say 5 go with the data enclave, why not do that at 6 the NISS as well? I mean I don't -- I don't. 7 DR. BOURNAZIAN: They're not 8 mutually exclusive. 9 MR. FEDER: (off mike) Why is it so 10 cheap and -- 11 DR. KIRKENDALL: It's not -- no 12 matter -- it's actually not a competition 13 with NISS. We are going to pursue that one. 14 MR. FEDER: Yeah. 15 DR. KIRKENDALL: Because it's no 16 particular additional cost. 17 MR. FEDER: No, no, no. I meant 18 the NISS data arrangement. 19 DR. KIRKENDALL: That's part of the 20 same thing. 21 MR. FEDER: A part of it. Okay. 22 Great. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 44 1 DR. KIRKENDALL: So it's a -- we 2 are doing that and I think on that one, this 3 came from a little fellowship program we had 4 through ASA, and we actually did have a few 5 researchers to come through that program that 6 worked here at EIA on site. They 7 particularly accessed the Electric Power 8 data, and they thought that it worked really 9 well for them. And, you know, agreeable to 10 do some work. 11 Electric power thought it was good, 12 too. 13 MR. FEDER: I remember seeing Joe 14 Guspid (ph.) here. Was he working (off 15 mike)? 16 DR. KIRKENDALL: No, Joe wasn't. 17 But anyhow, that's the kind of program we 18 were thinking about where the program 19 officers review the proposals and (off mike) 20 to, and because they're the ones who have to 21 provide the access and make sure the data are 22 protected according to their specifications. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 45 1 MR. FEDER: Now, Jake, you 2 mentioned this is a developing area. Indeed, 3 it is, and about a week and a half ago, I saw 4 an announcement for a new journal on data 5 confidentiality. I know one of the editors 6 is Christina, and I can send you -- I think I 7 saw Paula here. She's also an expert in this 8 area, so I'll send you guys information on 9 it. In fact, I would consider publishing a 10 paper in that journal when you make your 11 choice of the considerations that might be 12 useful, either that or (off mike), but 13 because it's a very good piece of information 14 for other agencies and government agencies 15 across the world to know what you're doing, 16 because you were very right about all the 17 benefits to EIA and to the research community 18 of giving people access to the data that 19 cannot be used otherwise because of 20 constraints, and the constraints are not 21 necessarily because your budget is not (off 22 mike) enough. It's because there's so much BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 46 1 data. Even if you had billions of dollars, 2 you would still be able to get a lot of 3 things out of the data. 4 So I think it's worthwhile to 5 consider publishing once you've (off mike) on 6 this -- on that. 7 Let's see. Now, my experience has 8 been mostly with data collected from human 9 subjects, which has particular constraints. 10 We have to get every piece of data collection 11 approved by the IRB. That's the 12 Institutional Review Board. And everything 13 that's collected has to be -- any use of that 14 data has to be approved. So if we cannot 15 even get someone to be our agent, so to 16 speak, and use the data, if that has not been 17 approved, and usually it has to be done prior 18 to collection. 19 Now, in your case, it's mostly 20 business data, but I do believe you have some 21 household-level data, so you have to check on 22 the legalities and if you can iron this out, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 47 1 again that would be worth publishing so 2 people will know what is -- what can and 3 cannot be done, I think that's one thing to 4 consider. 5 I really liked your comments about 6 the perceived risk of breaching of the 7 confidentiality pledge. It's something 8 that's been a lot on my mind because I think 9 all it takes is one case in the Washington 10 Post, let's say about an agency that released 11 information that was collected under pledge 12 of confidentiality with response rates, 13 accuracy of reporting and so on would be in 14 jeopardy. 15 So it's a good point, and I think 16 has to be kept in line. 17 So in summary, I liked -- Ed will 18 probably talk more about more substantive 19 aspects of use of the data, but I liked the 20 idea of leveraging researches, you know, 21 resource (off mike) students, whatever they 22 have to make more of the data. I think the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 48 1 first NISS option should be done regardless 2 of which other options you chose, because 3 it's cheap and probably very effective, and, 4 again, I think when you develop your approach 5 and make your own choices, I would encourage 6 you to consider publishing it in the 7 disclosure literature. 8 Yeah, that's about everything I 9 wanted to say. 10 DR. NEERCHAL: Thank you, Moshe. 11 DR. KOKKELENBERG: You know, a lot 12 of different observations, but let me give 13 you as background. I was one of the first 14 people over the falls with the longitudinal 15 establishment of data files at Census, and 16 (off mike) how many years ago that was. 17 Somebody asked me how working on that file 18 was, and after a year of working on it, I 19 said did you ever read Thomas Mann's Der 20 Zauberberg, the Magic Mountain, where time 21 all of a sudden goes extremely slowly and 22 nothing happens. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 49 1 Well, that was the Bureau of the 2 Census. I did want to underscore one of the 3 things that Moshe said right at the end about 4 the Washington Post, not that I have it in 5 for the Washington Post, but last fall, if 6 you will recall, there were some misplaced 7 laptops in various agencies, and there was a 8 great, great media uprising on that, but not 9 quite as bad as some we've seen elsewhere and 10 other times, but a lost laptop in an agency 11 that has eve 50 or 60 computers is not an 12 uncommon thing; right? 13 And so if that is going to be the 14 kind of reaction that happens, there is a 15 catastrophic risk if somebody does somehow 16 breach the confidentiality rules. So that 17 really has to be a concern. 18 Since I did this work, there was a 19 lot -- there's been a lot of developments and 20 basically because of these off-site centers 21 like the Census off-site centers, the 22 researcher needs to come to these problems BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 50 1 with an awful lot of background. I didn't 2 have it, and I learned that I had to have it, 3 and I got some of it. 4 First of all, the researcher really 5 needs to know what the underlying -- 6 understand underlying data generating process 7 is and what the questions were that were 8 asked to get this data. 9 Without that, you just got numbers. 10 And there's an awful lot of economists who 11 will take any dataset and run it through 12 their algorithm and go and say, see, I showed 13 the efficiency of or the cost function is or 14 the utility function is, and most of it is 15 not read very much deservedly. 16 This agency is really good on 17 statistical reliability. It's probably one 18 of the leaders in the area, so I think that 19 when you hand out data for whatever purpose, 20 emphasizing your understanding of the 21 reliability of the data is useful. 22 One of the big things that datasets BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 51 1 have in them at the micro level are an awful 2 lot of imputations, and they generally are 3 flagged if the agency is doing the work 4 properly. And the researcher has to 5 understand that particular aspect of things. 6 Now, the agency unfortunately is 7 going to have to have virtually a consultancy 8 to be able to help the researcher use the 9 data. 10 The researchers come in and they 11 don't know what the data -- I mean they have 12 an idea, and they spend a lot of time working 13 on a research plan. They submit it. It gets 14 approved. It has a basic understanding of 15 the problem, but it really doesn't understand 16 what data is there and how it will be used 17 and how to use it. 18 And so there's a lot of backing and 19 forthing. You can't say submit your request 20 for the variables you want or the 15 firms 21 you want for the time period that you want, 22 and we'll give you the data, and then expect BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 52 1 the person to go away fully satisfied. 2 They're going to be back saying don't you 3 have something and (off mike) so yes, and 4 they don't know what the -- the rules of 5 engagement that you lay out are going to have 6 -- be able to have them put a request in 7 advance knowing that the data exists and that 8 means they have to have access to the data 9 before they ask the question. 10 So one thing that has been proposed 11 is synthetic data sets. Okay. Are you 12 familiar with those? 13 DR. BOURNAZIAN: Yes. 14 DR. KOKKELENBERG: Okay. That 15 would be the way to go I think. If you gave 16 a synthetic dataset, which is all hokey 17 numbers, and the person -- but it's complete. 18 It shows all the variables that you really do 19 have and in the loving detail you have, and 20 then the researcher takes the synthetic 21 dataset and works on the problem, understands 22 what variables they need, what the strengths BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 53 1 and weaknesses of the models they have are, 2 but then gets access to the real live data. 3 And it's a one- shot or maybe two-shot access 4 to the data. But they're not -- 5 DR. BOURNAZIAN: So they first do 6 research on the synthetic data? 7 DR. KOKKELENBERG: On the synthetic 8 data to try to understand what the 9 limitations of the data are and what are the 10 variables that are available and their 11 strengths and weaknesses of each variable and 12 the time pass that they can look at. 13 So I understand some of the 14 research data center stuff is going to be 15 that way out at Census. 16 The other thing you're going to 17 need is you're going to need to be able merge 18 this data with other datasets, and that means 19 that this cut and paste prohibition for -- 20 might be legitimate. You've got to work your 21 way around that. But you want to match it up 22 with Census data or Ag data that's available, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 54 1 maybe also microdata, but without being able 2 to match it up, you're going to have problems 3 with that, or limitations. 4 Now, I would agree with Moshe when 5 he said that you should try multiple methods 6 of researcher access. The best way to 7 control confidentiality is only the access 8 through the agencies with computers that are 9 not connectable to the Internet and don't 10 have USB ports on them, you know, that sort 11 of thing. 12 DR. BOURNAZIAN: Mm hmm. 13 DR. KOKKELENBERG: And can't burn a 14 disk. The next best step is probably these 15 RDCs that the Census Bureau has, because 16 they've had a lot of experience with this 17 problem. 18 I don't know anything about the 19 National Opinion -- 20 DR. BOURNAZIAN: Research Council? 21 DR. KOKKELENBERG: Yeah, that -- 22 what their centers are going to be like so I BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 55 1 couldn't say anything about them. 2 DR. KIRKENDALL: That's relatively 3 new. 4 DR. KOKKELENBERG: Right. But even 5 the Research Data Centers take years to set 6 up. I know Cornell has been trying to get a 7 Research Data Center, and the best they could 8 do was get one I think in Buffalo and in one 9 in New York City, where they have access. In 10 fact, they teach a whole course on how to use 11 these Research Data Centers in the graduate 12 level. It's called Information 747, and John 13 Abowd teaches it, and I think he's a person 14 you should really strongly talk to, because 15 he's had a lot of experience working with the 16 data centers. He was Arnold Zelmer's 17 student. 18 DR. BOURNAZIAN: I know John. 19 DR. KOKKELENBERG: You do know him? 20 DR. BOURNAZIAN: Yep. 21 DR. KOKKELENBERG: Okay. He's an 22 expert in the -- he knows what BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 56 1 confidentiality laws apply and so forth, and 2 so he's the kind of guy you can (off mike) to 3 bring into have as a discussant when you get 4 to the point where you really want to make 5 some decisions on these (off mike), and get 6 some better advice. 7 Let's see. I think that's about 8 (off mike). Well, one other point: The 9 Census have one man, Jim Monahan was his name 10 who knew the dataset that they were making 11 available. He had spent years working it, so 12 it -- this takes a heavy- duty investment of 13 resources on the part of the agency to have 14 somebody who can shepherd the research or 15 into accessing the data and what's out there 16 and what isn't out there. 17 DR. BOURNAZIAN: Right. 18 DR. KOKKELENBERG: That's all. 19 DR. NEERCHAL: Neha? 20 DR. KHANNA: I have a question, one 21 question and another comment. My first 22 question is before the options that you BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 57 1 considered the three different options, does 2 it -- for any of those options does it matter 3 whether the researcher is a foreign national 4 and located in a U.S. university? 5 DR. BOURNAZIAN: Yes. If they're a 6 foreign national, they're going to have a 7 very difficult time coming on site here, and 8 then if it's Census Bureau RDC that FBI 9 background check could take a year. 10 DR. KHANNA: That's what I thought, 11 because that's a big disadvantage for an 12 onsite kind of arrangement because you're 13 losing more than half your research community 14 right there. 15 DR. KOKKELENBERG: But then you'll 16 lose it even for off-site -- 17 DR. BOURNAZIAN: Yeah. 18 DR. KOKKELENBERG: -- if they have 19 to go through this background check. 20 DR. KHANNA: Yes, but if you're 21 going -- the background check has to be done 22 anyway, whether you're coming on-site or BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 58 1 off-site, but on-site is -- and it's (off 2 mike) today, because I'm a foreign national 3 and they didn't like where I worked before. 4 I don't know. And I just -- it makes it more 5 difficult to me whether the load I think some 6 good fraction (off mike) because a foreign 7 national grad students from Georgia (off 8 mike) foreign national. That's just 9 something I wanted to put on the table 10 because not everyone in this room has to deal 11 with it as much as maybe I have to. 12 Another question that I -- another 13 question or comment that I had was I think 14 Ed's point about being able to match the data 15 with other publicly available data is really 16 important. An incubator by itself may be 17 nice, but it may not be very useful to do the 18 research project. 19 And there are two ways of doing 20 that. I mean if you went through let's say 21 the off-site VPN access route, because of the 22 fact that NORC might have let's say other BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 59 1 Census data also available that matching 2 issue may not actually be as great; whereas, 3 if you have it on-site and come and work on 4 our computer her, then I kind (off mike) the 5 Census data, and I have to -- 6 DR. BOURNAZIAN: Right. 7 DR. KHANNA: And I have to be 8 physically -- 9 DR. KIRKENDALL: I'm not sure 10 you'll be able to access Census data from 11 NORC either. 12 DR. KHANNA: No? 13 DR. FORSYTH: But you can from 14 Census. 15 DR. KIRKENDALL: But you certainly 16 could at Census. 17 DR. KHANNA: No, but if I have an 18 off- site, if I have the VPN, let's -- if I 19 have, you know, approval and I can use the 20 VPN from my computer at home or in my office, 21 and through the (off mike) can I access both 22 the Census and the EIA data? BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 60 1 DR. BOURNAZIAN: No. Right now, 2 NASS and ERS are putting their agriculture 3 data with NORC in a -- but, no, there's -- 4 they don't have the depth of the business 5 registries at the Census RDC has. 6 DR. KHANNA: But then another 7 question is: If a researcher, let's say, has 8 some publicly available Census data, let's 9 say at the Census track level or block group 10 level, something that you can just go to the 11 Census Web site and download, would it be 12 possible for the researcher to bring those 13 data to the NORC Web site? 14 DR. BOURNAZIAN: Yes. NORC 15 provides those portfolio services to a 16 researcher can bring in exogenous or 17 administrative data. 18 DR. KHANNA: Okay. 19 DR. BOURNAZIAN: And match to 20 whatever they're accessing. 21 DR. KHANNA: Okay. So that -- 22 DR. BOURNAZIAN: Enhance it. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 61 1 DR. KHANNA: Enhance it a little 2 bit. But okay. 3 DR. NEERCHAL: Moshe. 4 DR. FEDER: I wanted to comment on 5 the synthetic data. It's an appealing idea, 6 but it has a number of limitations. Number 7 one is the cost. To do it, you really need 8 to spend way more than a hundred thousand 9 dollars to develop it, but the main problem 10 is how do you do that? You hypothesize a 11 certain model. You feed the model parameters 12 from your data and then you generate data 13 from that model. 14 And the problem is if the 15 researchers were interested in validating 16 that same very model that was used to 17 generate the data, they cannot, because the 18 data were using that model. And that has 19 been raised many times. 20 So synthetic data has this inherent 21 limitation that cannot be overcome, well, 22 because maybe use some semi-parametric then BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 62 1 approach, then you would really develop some 2 methods that haven't been created yet. 3 I don't think we have been working 4 on it, but it really -- real data is real are 5 -- sorry -- real data and simulated data 6 costly and not as good. So that's why I 7 still like the idea of direct access to the 8 data, plus what Neha was talking about, being 9 able to link it with Census data, which you 10 would not be able to do with synthetic data, 11 because you need some -- 12 DR. KOKKELENBERG: The Census data 13 is only used for you to develop your research 14 proposal pertinent to the points of where you 15 know what the real data is, but that's 16 available and then use it in that sense, not 17 to fit any models necessarily but to try to 18 understand data availability. 19 DR. FEDER: I would use Census data 20 to get company variables that are not 21 available in the EIA data. 22 DR. KOKKELENBERG: Oh, yeah, well, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 63 1 that's a good point. 2 DR. FEDER: Yeah. 3 DR. KOKKELENBERG: You know one way 4 to that I would have thought of doing this is 5 let's say you have the electricity generation 6 by generator. So if you take this vector of 7 just add to it or multiply it or take it to a 8 power of a random number, and there's your 9 synthetic data. 10 DR. NEERCHAL: I think that one 11 quick point on that one I think knowing the 12 data structure, even if I don't know what the 13 numbers are in the beginning really help a 14 researcher focus their data request a lot. 15 DR. BOURNAZIAN: Absolutely. 16 DR. NEERCHAL: I think in terms of 17 that you can just basically, you're going to 18 give me a blank matrix with all the data -- 19 you know, the variables in there and what the 20 components are and so on. That will be 21 extremely useful. 22 DR. FEDER: Well, you could make BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 64 1 the code book available. 2 DR. NEERCHAL: No, no. On the 3 other hand, I think Moshe is absolutely 4 right. I think really doing -- putting 5 synthetic data out there, which is kind of, 6 you know, unbeatable is a very difficult 7 task. I think that that's a -- but I do want 8 to make one comment, you know. 9 From the presentations, the way I 10 understood is that, you know, you have two 11 important objectives. One is to address all 12 these questions, many of these questions that 13 come in and you don't have time to answer it. 14 You just say, you know, that information is 15 over there; you know, help yourself. That 16 kind of, you know, you don't need to spend 17 too much time on this. 18 The other objective is to promote 19 research on your data. I think the -- maybe 20 the first one should be the, you know, the 21 first objective. I think that maybe you want 22 to kind of prioritize it and say how can we BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 65 1 really, like, cut down some of this work that 2 we need to do periodically and not -- don't 3 really enjoy doing it. At the same time, we 4 don't want to say no. I think this is the 5 negotiation that Guy was talking about 6 yesterday. 7 I think, you know, come with 8 something that really helps you to say oh, it 9 will be very hard, we have been thinking 10 about what you would ask and we put this 11 information out there. Help yourself. So, 12 but, you know, that's about it. 13 On the other hand, promoting 14 researchers (off mike) certainly it's not 15 possible for us to sit down here and think 16 about all the things a researcher might ever 17 ask for. No matter what you put out there, 18 somebody is going to come and say, I need 19 more. Okay. I think that that's something 20 -- that is why what promoting research, you 21 really want to go the "proposal" route. If 22 someone wants data, let them write down what BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 66 1 they want. You do whatever you can to the 2 best of your available resources. 3 You may want to kind of bifurcate 4 your approach like that. 5 DR. KIRKENDALL: Actually, all of 6 these proposals involve us responding to a 7 proposal for research by a researcher. A 8 person would have to send in a proposal 9 either to NISS or to NORC or to the Census 10 Bureau and that would proposal would have to 11 be reviewed any number of different places 12 and accepted in all of those places before 13 they would actually get the access. 14 So everybody would have to look at 15 it and make the sure the research was okay, 16 according to the plan and then the people who 17 owned the data would have to be sure it was a 18 good idea, too. 19 DR. NEERCHAL: And I think we need 20 to -- we may have to (off mike) better. 21 Barbara, who invested (off mike)? 22 DR. BINGHAM: No, go ahead, because BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 67 1 I think she wanted to -- 2 DR. FORSYTH: It was, but I think 3 it might be less important (off mike) what 4 you just said, Nancy. 5 What I was going to suggest is, you 6 know, the synthetic data and giving -- 7 helping people to understand the structure of 8 the data. I liked your idea of a course for 9 users, teaching (off mike) of the structure 10 of the microdata, and you have a ready 11 outlet. You know, JPSM, for example, could 12 help you put together like a short course, a 13 two-day course, and possibly in the context 14 of that course, depending on how you 15 structure it, maybe give people some kind of 16 access to some kind of subset or limited set; 17 at least give them some experience, 18 opportunities to experience the structure of 19 the microdata. So that's what I was going to 20 suggest. 21 DR. KIRKENDALL: Actually, both of 22 these surveys already have public use BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 68 1 datasets, so we have -- it's just that 2 there's been some top coating and maybe not 3 all the variables are there -- 4 DR. FORSYTH: Right. Right. 5 DR. KIRKENDALL: But other methods 6 to (off mike) of it. That's at least a way 7 you could get started understanding the data. 8 DR. FORSYTH: Right. The 9 microdata, though, would have a somewhat 10 different structure, because it would be more 11 of it; right? 12 DR. KIRKENDALL: Yeah. It would -- 13 SPEAKER: The actual -- 14 DR. FORSYTH: Right. And the 15 matching across datasets. 16 DR. BINGHAM: So I'm curious (off 17 mike) change the subject. I'm curious about 18 what the -- I'm curious what the -- how easy 19 is a breach of confidentiality insofar as 20 when you're looking -- so what is 21 confidentiality in this case? I mean that's 22 the first question and does that mean I'm BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 69 1 able to identify a respondent's exact 2 information and lots of detailed information 3 about them and is that bad? 4 DR. KIRKENDALL: Well, we tell them 5 we're not going to give that data out, and 6 that is the kind of identification we want to 7 protect against, and we really don't really 8 want somebody to say that they've identified 9 it, even if they're wrong. 10 DR. BINGHAM: Right. (Laughter.) 11 DR. BINGHAM: Okay. So -- 12 DR. KIRKENDALL: And that's a 13 little harder to protect against. 14 DR. BINGHAM: So I know that you -- 15 the temptation is always there to do that; 16 right? If you're working with that level of 17 data. Okay. So this happens -- so there's a 18 lot of work that goes on with confidentiality 19 in surveys, for instance, where 20 health-related things and they collect things 21 all the way down to the county level and 22 suppose you're looking at HIV counts and BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 70 1 there are two people who live in the county; 2 right? And so the other person knows that 3 they don't have HIV, so they know who does; 4 right? 5 DR. KIRKENDALL: Mm hmm. 6 DR. BINGHAM: And so there are lots 7 of efforts to either aggregate to another 8 level or randomly switch around counties and 9 things like that and my question is any of 10 this taking place at -- you know, on this 11 microdata when you're going to send it out, 12 you know, because you know whether it's a 13 graduate student or it's another company, 14 just a guy in a company who says, you know 15 what, I know enough about this. I should be 16 able to identify these individual records 17 here, and is this fiddling? Is it pos -- 18 what's being done? 19 DR. BOURNAZIAN: Well, the -- 20 there's a lot in that area. The pledge is 21 that you're not -- there's not going to be 22 unauthorized disclosure of reported value of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 71 1 a respondent by EIA. 2 That's why in a lot of the slides 3 you're saying that this researcher is going 4 to be an agent or a subcontractor. We all 5 look at company-level data. In fact, 70 6 percent of Americans can be identified just 7 by zip code and birth date. So you've got 8 these survey records that have a lot of 9 direct identifiers, even though you can strip 10 off the name and company ID. 11 The researcher, he's interested in 12 some kind of statistical analysis, so he 13 might be in a small geographic area. He 14 knows this is a certain type of company. He 15 might not know the name, and then, yes, you 16 know, if he's well, you know, involved in 17 this research, he knows this market and he 18 knows the players in the market, and so he 19 knows really who he's looking at when he's 20 looking at the record. 21 The point is when he finishes his 22 research and he writes up his paper, is he BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 72 1 going to be talking about, you know, X, Y, Z 2 power generator, you know, or is he going to 3 be talking about this, the power generating 4 industry in this area. 5 So it still has to be statistical 6 analysis. That's what Nancy was saying. 7 Those proposals get reviewed quite 8 rigorously. What's the purpose for allowing 9 access? What do they want to study? 10 And if it has a nice statistical 11 purpose, yes, it can go forward. Researcher 12 is going to have access to it, but he can't 13 take it out. He can't take it home. And he 14 can't be publishing, you know, a 15 company-level responses in his paper. 16 But the access is there. 17 DR. KIRKENDALL: And he's signed an 18 agreement that he won't reveal the data. So 19 there has to be an agreement with the 20 researcher that they, you know, I'm sure -- 21 even though they may have looked at something 22 and they have figured out that it's company BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 73 1 X, Y, Z, he can't -- he's not supposed to 2 make use of that information. 3 DR. BOURNAZIAN: Right. 4 DR. KIRKENDALL: And he's not 5 supposed to reveal it, and, in fact, if it's 6 CIPSEA data, then he's subject to big 7 penalties if he does. 8 DR. BINGHAM: Okay. So I guess the 9 only thing -- so in that case, so, like I 10 think I said, I suspect with somebody 11 fiddling will figure something out. 12 You can also guess that if 13 somebody's doing a detailed enough micro 14 micro analysis on something via, I guess 15 somebody has to vet this in order to make 16 sure that something inadvertently doesn't get 17 published. 18 You know, I don't know to what 19 level people are using the information, what, 20 you know, like how fine down something that's 21 going to appear in a publication, how far 22 down somebody will have dug, and, if they BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 74 1 will have -- I guess it's the only thing that 2 I would -- the only question I have is okay, 3 can something -- okay, can confidentiality be 4 inadvertently breached, and) I guess it seems 5 like from the spirit of what you're saying, 6 the answer is no. But -- 7 DR. KIRKENDALL: We hope the answer 8 is no, and there are safeguards. You know, 9 the papers are reviewed -- they'd be reviewed 10 either by the Census Bureau Research Data 11 Center or-and or by the NORC folks. It would 12 also be reviewed by EIA people, and so you 13 hope that you haven't allowed something that 14 breaches confidentiality to get out. 15 And that's what the level of review 16 is about. 17 DR. BOURNAZIAN: And this has been 18 going on for many years. 19 DR. KIRKENDALL: But that doesn't 20 guarantee either, of course. 21 DR. BOURNAZIAN: But it's been 22 going on for almost 20 years, and we have not BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 75 1 found any confidentiality breaches from 2 researcher access among the federal 3 statistical agencies. There's been breaches, 4 but it hasn't come from this activity where a 5 researcher has gone in and done something 6 intentional or even foolish or deliberate. 7 They're usually -- well, they're 8 all dedicated in good faith and in those 9 reviews is your final check. And so you got 10 to say, look at the track record. Zero 11 breaches across eight, nine agencies engaging 12 in this activity over the last 20 years. 13 DR. KIRKENDALL: Well, it's bound 14 to happen like tomorrow. 15 DR. BOURNAZIAN: Right. You can 16 never say it will never happen. Right. 17 Well, we're building up to it. 18 DR. NEERCHAL: Ed, you had your -- 19 DR. KOKKELENBERG: Well, much of 20 what I was going to respond to, Derek brought 21 up and it was responded to by Nancy and Jake. 22 One thing I did want to add, though, is that BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 76 1 you do know, as a researcher, you know that 2 company's name? 3 DR. KIRKENDALL: Mm hmm. 4 DR. KOKKELENBERG: And you -- I 5 don't care whether you strip the name out or 6 not, but in the Census data, it was not -- at 7 the establishment level, I knew that company, 8 and I knew its quarterly labor bills and its 9 production, every -- you know, the details. 10 So that's available to me. All 11 these other safeguards are there, plus that 12 there are a lot of penalties under the laws 13 that enable the Census, which, you know, 14 could put you in prison if you really wanted 15 to go out and breach the confidentiality 16 arrangement. 17 And the final review of your paper 18 is before -- even with Moshe raised the issue 19 about confidential data in human-or for 20 individuals. 21 I know I've been working with 22 student- level records, and I get that paper BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 77 1 vetted by the FERPA enforcer, I guess I even 2 called her, at SUNY Binghamton before I can 3 let the paper go out the door or what have 4 you. 5 So it's the same with Census. But 6 you really want to go look over those papers 7 very carefully to make sure that I wasn't 8 revealing anything to anybody. That's all. 9 You just have to stand on the, you 10 know, trust the ethical responsibilities of 11 the individual researcher that requires the 12 -- perhaps this long vetting of the 13 researcher to do that; that the FBI does for 14 me to commit to do it. 15 DR. NEERCHAL: Thank you. Thank 16 you and the presenters, discussants and (off 17 mike). I think we break and get back here 18 around 10 maybe? We're a little bit behind, 19 but that should be okay. We need a little 20 break, I think. 21 DR. KIRKENDALL: Okay. Yeah. 22 (Recess.) BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 78 1 DR. NEERCHAL: So we are onto our 2 last plenary session. Can I just go and ask 3 any audience member who was not here before 4 to go to the mike and mention your name and 5 affiliation, please. 6 SPEAKER: You may even need to turn 7 your switch on. 8 MR. SIMBALSKI: John Simbalski. I 9 work for EIA, and I'm the NEMS residential 10 (off mike), and I'm here to listen to how 11 George is going to pick apart our work. 12 (Laughter.) 13 MS. BOEDECKER: Erin Boedecker. I 14 work on the commercial module of the NEMS 15 system. 16 MR. McDOWNEY: Preston McDowney. 17 DIA, Systems and Methods Group. 18 MR. RADICH: Tony Radich. I work 19 on the Petroleum Market Model in NEMS. 20 MR. SKELLY: Dan Skelly, EIA. 21 DR. NEERCHAL: Thank you. So the 22 last session speaker is George Lady. It's on BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 79 1 Forecast Evaluation Methodology. 2 DR. LADY: Thank you very much. I 3 very much appreciate a chance to talk about 4 my project and John is wrong. In fact, 5 without his help, a good deal of what I'm 6 going to report on today couldn't have been 7 done. So. 8 In fact, I should say in advance, 9 just to be clear, OIF has more and more taken 10 an interest and supported what I am doing and 11 by the time we're done today, you'll see my 12 point of view in terms of what I recommend is 13 going to depend very much on that. 14 The project that I'm reporting on 15 is at this point taking thought on what would 16 be a good way to evaluate the accuracy of 17 NEMS forecasts. 18 The truth is that EIA has published 19 for some years comparisons of the NEMS 20 projections with the eventual values of the 21 energy statistics, and I think they reported 22 on something like 17 or 18 of the major BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 80 1 series that they're forecasting. 2 There's a literature on this, and 3 part of the literature found that, although 4 the forecasts would often look quite 5 accurate, if you disaggregated them, for 6 example, if you took the consumption forecast 7 and re-expressed it in terms of residential, 8 commercial, transportation, and industrial 9 sector consumption that the errors would be 10 somewhat larger and then offsetting when 11 added together in the total. 12 My idea was to pursue sort of that 13 point of view and to look at the forecasts 14 and try to understand what were the major 15 influences that would lead to the differences 16 between what was forecast and what eventually 17 happened. 18 And here on the slide is the list I 19 came up with of the sorts of things that 20 would explain why, in the end, the projection 21 would not be accurate. 22 From the standpoint of what I can BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 81 1 do and have been trying to develop a point of 2 view about, I can analyze the weather. I'm 3 trying to develop a method where I can 4 organize major explanatory variables for 5 supply and demand, like price and gross 6 domestic product. And I think in the end, I 7 can evaluate structural change by comparing 8 different versions of NEMS and can detect 9 whether or not there has been a change in the 10 model logic. 11 What has been pointed to me is that 12 there are a number of other things that can 13 make a big difference, too, perhaps the most 14 important one is the fact that the data 15 definitions can change over time, and this 16 can make the forecasts appear inaccurate in 17 terms of historical values, but really the 18 world is changing underneath of the model. 19 And I think that's very important, 20 and I'm hoping really that this project 21 starts an enterprise of worrying about all of 22 the possible sources of forecast error, but BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 82 1 some of them, such as changes in data, I 2 can't speak to. I hope that a tradition 3 starts where that particular issue is taken 4 care of. 5 As far as what I'm doing, the 6 methods at stake really are three: 7 The point is to understand at the 8 level of explanation why the supply and 9 demand forecasts in NEMS are not accurate, 10 and the idea is to take the errors that are 11 found and to try to partition them across 12 such as price forecasts which are inaccurate, 13 driver forecasts, like GDP, that are 14 inaccurate, innocent things like the weather, 15 which can be different, in fact, than was 16 assumed for NEMS; and try to parse out all of 17 these influences. 18 The ideal -- and I received some 19 very nice written comments from the Committee 20 earlier in the week -- the ideal would be to 21 save each version of NEMS, to archive it, and 22 when the time comes to rerun the model using BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 83 1 actual values for assumed values and maybe 2 even reconfiguring the model logic to account 3 for changes in laws or other structural 4 changes in the energy system. 5 In fact, just to show you how old I 6 am, back in 1980, I actually, when I worked 7 here, initiated a program to archive the 8 versions of all of the models that were used. 9 At that time, one of the more, one of the 10 things that was in mind was that when a 11 forecast was published, the archive model 12 could be given to a third party, who could 13 then inspect what was the basis for the 14 agency's projections. 15 But strictly speaking, it could be 16 used in this project. The problem is that 17 it's very hard to do, and it's been well over 18 a decade I think since there has been any 19 routine archive program. It's very expensive 20 and as far as I know, there hasn't been any 21 initiative to use archives versions of the 22 model to speak to the kinds of matters that I BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 84 1 think I would like to organize. 2 But it obviously would be the best 3 thing, and that's what the reviewers say, and 4 I agree with that. But I fear that it is too 5 hard to do. 6 What I will propose today is that 7 there is a good second best that we can do; 8 that I think is a lot easier as far as 9 forecast evaluation compared to archiving the 10 model, and subject to the errors of linear 11 approximation will do probably as nearly a 12 good job. 13 And in particular it's my item two 14 here. I'm advocating now that each major 15 version of NEMS, the AEO versions would be 16 the first to come to mind, have a structure 17 of runs made which are designed to pick out 18 major influences. 19 OIF has already done this several 20 times with respect to prices, and John's 21 group, very nicely, gave me over 70 runs that 22 they did a little over a year ago where they BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 85 1 changed each fuel price one by one and just 2 in terms of the demand models were able 3 through the solution to see what difference 4 the price changes made. 5 Well, I believe this can be done in 6 general, and that what I would advocate in 7 the end is that there be some routine 8 protocol developed, some experimental design 9 agreed to, where some number of solutions are 10 created, designed in a comparative static 11 sense to pick out influences and rather than 12 archive the models, we'll archive the 13 solutions. Much, much simpler. And then a 14 statistical analysis of the solutions can be 15 used to pick out the sources of error in a 16 fashion that I'm proposing. 17 For what I did, not having, except 18 in the one case, such an inventory of 19 solutions was to use the solutions that are 20 published for the Annual Energy Outlook, 21 which I have back through 1998. 22 The reviewers quite rightly in BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 86 1 considering the statistical approach I 2 brought to those solutions worried that there 3 is what in economics is called an 4 identification problem. In other words, when 5 I was picking out the influence of price, the 6 truth is that price is an endogenous variable 7 in the model. And as a result, in general, 8 if you took market data, you wouldn't be able 9 to just run a simple least squares and 10 isolate the price effect. 11 And here is the graphic that the 12 reviewers provided demonstrating that. 13 So here is supply and demand 14 shifting around and in terms of the 15 elasticity that you want to estimate, there's 16 no way from these solutions to identify the 17 slope here, which is what you need. 18 But since NEMS is not the world, we 19 can control it more. And the truth of the 20 matter is that the principal scenarios for 21 the AEO were designed by me, in fact, in 1977 22 to show you how old I am, to speak to the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 87 1 identification problem. So specifically if 2 you consider the oil price variation cases, 3 what they are doing in principle is freezing 4 the demand configuration and then shifting 5 supply. 6 So the elasticity estimates that 7 I'll show you before we're done came from 8 these solutions pooled together. And 9 although the reviewers were quite right, 10 there's a simultaneity problem in general, I 11 believe it is very successfully spoken to by 12 the data that I pooled. 13 But my proposal number two I think 14 is even better. If in the end, which has 15 been done, we generate solution sets where 16 the variables are changed one by one in terms 17 of the supply and demand components 18 individually, then when you pool those data, 19 you will have completely avoided the 20 simultaneity problem at least in terms of the 21 first) order subject to the errors of a 22 linear approximation. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 88 1 So in terms of method, what I am 2 proposing in the end is that we -- I -- 3 somebody agree to an experimental design and 4 EIA devote the resources so that each model 5 version can be run through this drill and the 6 solution saved to then be used in later years 7 to assess the accuracy of the forecast. 8 Now for the rest of my talk, I want 9 to just go through the sorts of things that I 10 was able to do in the several months that 11 I've had a chance to work on this. All of 12 this is with -- is prospective. In other 13 words, my words my purpose was to investigate 14 how this might be done. Nothing that I will 15 provide you was intended to be, and whether 16 it was or not is certainly not definitive in 17 terms of the way in which the evaluation 18 should be conducted, but rather it's my 19 effort to try to get all of the methods in 20 hand and to show examples of how they might 21 be used. 22 Certainly, for adjusting forecasts BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 89 1 with respect to differences in the historical 2 values, weather would be an obvious influence 3 on energy consumption, so I looked at the 4 weather data that EIA keeps and found some 5 things I think probably should be attended 6 to. 7 The Annual Energy Review publishes 8 heating degree day and cooling degree day 9 time series, which are the major weather 10 variables in terms of understanding what 11 weather might have to do with energy 12 consumption. 13 NEMS I learned -- and I didn't 14 really know this until I got into the project 15 -- does have weather variables in it, and in 16 terms of my proposal two OIF very nicely ran 17 two weather cases for me, where they changed 18 in one case the heating degree day variable 19 and in the other case the cooling degree day 20 variable each by 10 percent that I could then 21 compare to the base case for the current 22 version of NEMS and then could quite readily BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 90 1 estimate what the weather effect is. 2 These data come from those cases 3 that were provided me. 4 The short-term model also uses 5 weather, and these data come from oh, 6 probably eight weeks ago what was on the 7 Internet in terms of what the short-term 8 model was doing. 9 The point of my remarks is when you 10 look at the numbers, where you would assume 11 they would be the same -- here's the AER 12 definition of normal. Here is NEMS forecasts 13 for normal weather in this year, and here's 14 the short-term models forecast. They're all 15 different. And I don't know why they're 16 different, but I think it's probably not good 17 that they're different. So a byproduct of 18 this particular discussion is my thought that 19 this should be somehow cleared up. 20 When I went ahead and used the data 21 to calculate the sensitivity, I went about it 22 three ways just to see what difference it BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 91 1 would make. I have now cases where I'm doing 2 discreet changes. It's perfect. It's 3 absolutely just what you want. So the base 4 percent of elasticity, as I understood it, 5 was the way in which OIF would process the 6 data, which was to simply take the actual 7 changes and then use the base case as the 8 percentage basis for calculating elasticity. 9 A more standard economic approach 10 would use the average of the two cases. But 11 I decided, because of what I wanted to do 12 with the data that I would use the constant 13 elasticity form as the mathematical form to 14 estimate the sensitivity. 15 I then looked at what difference it 16 made, and it made almost no difference as to 17 which method was used, so this is the method 18 I used, and what I'll show you in a minute, 19 and these are actual elasticities that come 20 out of the 2007 Annual Energy Outlook version 21 of NEMS. It's very interesting. 22 I see that they change over time. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 92 1 I don't know why, but it's very interesting 2 to me. 3 In any case, I took the 4 elasticities and calculated them and then 5 compared them to the elasticities in the 6 short-term model. 7 And as you see, I found 8 differences. I'm not sure why they're 9 different. It's just -- was out of my scope 10 to be able to track that down. There's even 11 a sign difference, although I suspect that 12 these small numbers really should be nil 13 rather than an actual number. But still, 14 there is a sign difference. 15 I think that this should also be 16 cleared up. The methods are different, and 17 here we all know that the NEMS method versus 18 the more econometric method that the 19 short-term model uses can lead to different 20 results, but outside, say, if I'm explaining 21 this to my class at Temple, they don't 22 understand why they're different. They think BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 93 1 there should just be one sensitivity. So I'm 2 recommending that that somehow be dealt with 3 as well. 4 In any case, I did use the results 5 that I was nicely given and in the results 6 I'm going to show you in a minute, I took the 7 constant elasticity form that I estimate from 8 the data and then applied it to the 9 historical versus the forecast values in the 10 following way: When I have the actual data 11 and I have the elasticity and I have 12 historically what the consumption number was, 13 I can back out the value of the 14 multiplicative constant in this functional 15 form. 16 And given that, I can plug in what 17 the normal weather consumption would be and 18 then difference them and this would give me 19 my adjustment due to the weather that 20 happened versus the assumption of normal 21 weather. 22 And in a minute, this particular BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 94 1 method will be showing up in the impact 2 analysis that I'll show you. 3 One of the most interesting things 4 I had an opportunity to do was to see the 5 degree to which I could look at the price 6 sensitivity of NEMS across years worth of 7 NEMS versions, 1998 all the way through this 8 year's 2007 version. 9 I used a very austere specification 10 and the reviewers quite correctly pointed out 11 that when you got down to doing this really, 12 the specification that I used was certainly 13 too austere, and I agree with that entirely 14 and, in fact, my very next order of business 15 is to I hope get some advice from the 16 modelers as to what the proper specification 17 would be for some of these regressions. 18 In the meantime, to get on with it, 19 I ran this regression for the data pooled 20 across the base case and oil price cases for 21 all 10 years worth of versions of NEMS. And 22 because EIA published elasticities in BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 95 1 long-run and short-run cases, since I have 2 the lagged endogenous variable in this 3 specification, I went ahead and backed out 4 the one year and the two year and the long 5 run elasticities as well using these formula 6 just to compare them. 7 There is a problem potentially, 8 which is to say the regression doesn't have 9 to come out right. Specifically, the price 10 elasticity certainly should be negative and 11 the driver elasticity certainly should be 12 positive, and for the long-run adjustment to 13 converge the parameter that you estimate for 14 the lag term has to be positive but less than 15 one for the whole business to work. 16 And although it generally worked, 17 it didn't always work, and here is for the 18 four fuel types for the 10 years and for both 19 residential and commercial sector 20 consumption. Here is a scorecard, and, as 21 you can see, distillate really didn't work 22 very well at all. The others worked most of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 96 1 the time. I think the problem is probably 2 partly that the data series that I'm using 3 are a little sparse and if I get the custom 4 runs, that this will be better. But I think 5 probably the problem is more that I've left 6 something out. There's something excluded in 7 my simple specification that is causing the 8 trouble. 9 But most of them did work. And 10 there's a hard copy. This is fun to look at, 11 so we can't really enjoy this as much as it's 12 worth. But here, think of it: Here are the 13 elasticities for the versions of NEMS between 14 1998 and this year for these four fuel type, 15 both long-run and short-run. All of the rows 16 which are labeled SIM are my regressions. 17 The rows that are labeled AEO9903 and are 18 from the much better experimentally designed 19 data that OIF created, so there was a report 20 that gave me the elasticities for 1999 and 21 2003 and then, although there wasn't a 22 report, I was given the solutions and can BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 97 1 read out of the spreadsheets where they've 2 done it what the values would be for 2006. 3 As you can see, they move around a 4 lot. I think part of that is because the 5 specification I'm using is too austere, but 6 the better designed values also move around a 7 lot, so I think there's a lot of interesting 8 groundwork that we can go through to see 9 what's going as far as the model is 10 concerned. 11 So these are all available to you. 12 I have a lot of fun looking at them, and it 13 is a very interesting result. 14 But in terms of the bottom line, 15 what I wanted to do was to take what I had 16 done and assuming that it was good enough, to 17 go ahead and formally partition the 18 differences between the NEMS projections and 19 historical data insofar as my statistical 20 specification would allow me to do it. 21 Here's an example of the way it 22 would work: Here is a regression for the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 98 1 year 2000 version of NEMS, where I'm looking 2 at commercial- delivered energy, and the 3 regression is from data pooled across the 4 base case in the two oil price cases. The 5 outcome of the regression in terms of the 6 implied elasticities is very plausible; 7 certainly consistent. And this then was the 8 mathematical underpinnings of how and then I 9 would look at the projections from that 10 version of NEMS and try to see what might 11 explain the differences between the 12 projections and historical data. 13 Well, this is a good one. The 14 projection is very close. It's only two 15 percent low. So the first line here shows 16 the difference between what was projected by 17 NEMS and what actually happened. 18 The other lines in the table take 19 each of these influences -- the price, the 20 driver, which for commercial energy 21 consumption that I used I used floor space -- 22 the lag endogenous variable and then weather BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 99 1 adjusting in the fashion that I mentioned 2 earlier and went ahead and looked at what 3 actually happened compared to what was 4 assumed and then determined how that would be 5 expected to influence the forecasts. 6 So, for example, the price that was 7 assumed in the year 2000 for the year 2005 8 was 24 percent low. So you would think as a 9 result that the NEMS forecast would be high 10 compared to what actually happened with the 11 price a lot higher. So that's what the 4.41 12 percent is -- how high you would think the 13 forecast would be given the fact that 14 historically the price was 24 percent higher 15 than was assumed. 16 Then almost perfectly offsetting 17 that is the fact that the actual drivers 18 values were larger than those assumed. So 19 for that you would think the forecast would 20 be low. And so these two influences offset. 21 In any case, by the time you take 22 each of these influences identify them and BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 100 1 add them all up, you would think the forecast 2 would be one percent low, but instead it was 3 a little over two percent low, so the 4 uncertainty number that I present is what 5 balances that out. In other words, the 6 forecast was a little lower than you would 7 have thought given the explanatory variables 8 that I was keeping track of. 9 So I went ahead and did this and 10 again you have the hard copy and you can 11 enjoy this more at leisure than we can in 12 this talk, but I did it for five variables 13 across the AEO versions up through 2004 for 14 the year 2005. 15 I'm advised that for the versions 16 of NEMS which are quite close to the year 17 that's being projected that actually the way 18 the projections are published relates to the 19 short-term model to some degree as well as 20 NEMS and it's a little bit not right to view 21 this as accuracy of NEMS per se. But for the 22 ones that are five and six years into the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 101 1 past, I think that is fine for NEMS. 2 Interestingly, I found that using 3 my specification and the adjustment process 4 that I've outlined, I found for four of the 5 five fuels that I kept track of that the 6 forecast ended up being lower than I would 7 have thought, consistently, year by year, 8 which could be because I'm just leaving 9 something out, except for commercial 10 electricity where it was just the other way 11 around. 12 And although I think there is a lot 13 of room for improvement in terms of the 14 particular statistical methods that I'm 15 using, I think, nevertheless, this kind of 16 result is indicative as to what can be 17 brought out of this kind of analysis to 18 assist the modelers in understanding what the 19 model is doing. 20 Accordingly, is my proposal that we 21 proceed along the following lines. 22 First, I really do think that the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 102 1 weather business should be at least 2 explained. I could just not understand 3 something, which is very clear, but the way 4 it looks from the outside, there's too many 5 differences. 6 In any case, otherwise, I believe 7 that we should move towards the direction of 8 until we do archive NEMS, which I suspect is 9 not going to happen, we should move towards 10 developing an experimental design to run 11 versions of the models through so that the 12 solution sets that you bring together from 13 that design can be used to isolate the 14 influences of the explanatory variables. 15 My idea would be that at least for 16 the time being as these solutions are 17 created, I can then test the specifications 18 that I'm given and then there can be 19 interaction -- will it work or won't, to some 20 degree -- and we can iterate through that 21 analysis. But the target will be to bring 22 into being I would think an annual report BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 103 1 which presents -- and there's much to con -- 2 in the template that's missing yet -- but 3 still the general idea would be to present 4 how NEMS is doing and to try to understand 5 the major sources of differences between what 6 the projections said and what actually 7 happened, and not only cope with the things 8 that I can talk about, but also try to deal 9 with data differences, and in the end have a 10 good, solid quality control handle on what 11 that model is doing, which would be a 12 credential to the users of the mode and I 13 also believe a diagnostic tool that would 14 help the modelers. So my efforts are to try 15 to point towards something much better than 16 I've been able to do, but try to demonstrate 17 that this is an important and useful 18 approach. 19 DR. NEERCHAL: Thank you. Can we 20 start with the discuss -- (off mike) 21 discussions and then we'll open it up. Jae, 22 you want to go first? BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 104 1 DR. EDMONDS: I thought you were 2 listed first. 3 DR. NEERCHAL: And the thing is 4 that a good gesture for me to say go first. 5 DR. EDMONDS: Well, let me just 6 first say that -- two things. One is that I 7 was part of that session last time, where we 8 sort of began looking at this issue, which 9 was an opportunity to sort of follow up with 10 George's career. I've been learning from 11 George for as long as I've been working in 12 the energy business, so it's always a 13 pleasure to get back in touch. 14 In terms of trying to reflect back 15 and give you some helpful comments, I wasn't 16 -- I read through the material, and I wasn't 17 quite clear exactly what the -- you know, 18 what the questions you'd like us to deal with 19 were, so what I'm going to say may or may not 20 be at all helpful, so, you know, I apologize 21 if that's the case. 22 And just thinking through it, I was BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 105 1 having, you know, trouble getting my mind 2 wrapped around exactly what we're trying to 3 do. The basic idea is to try to understand 4 how do you make NEMS a more accurate forecast 5 or if there is a -- you know, if there is -- 6 will always irreducible uncertainty; that is, 7 if you're given a set of input drivers and, 8 you know, they're coming over from the 9 Council of Economic Advisors and it's 25 10 percent higher than, you know, your 11 assessment, you know, it's coming from the 12 Council of Economic Advisors and this is what 13 you're going to run, and so, you know, some 14 of that is just going to be inevitable; you 15 know, things that are just going to be 16 outside your control. 17 But the model itself is inside your 18 control and so, you can actually go back and 19 say well, what if I had the right, you know, 20 gross domestic product? What if I had, you 21 know, the right -- you know, any set of 22 exogenous variables that go into NEMS, and BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 106 1 you're going to ask, you know, how -- you 2 know, how well did I do if I had that 3 information correct, and how do I make it a 4 better forecast. And that seems to me to be 5 what this activity is really geared at and 6 when you just sort of go back to the 7 foundations. How do you make the NEMS a 8 better modeling system and so with that in 9 mind, you know, NEMS, of course, to anyone 10 who has tried to, you know, to -- has 11 experience with NEMS knows that it's a -- 12 it's not a simple model. It's a 13 sophisticated system of models and so how do 14 you get your arms around it? 15 I guess the first thing to 16 recognize is that, as George has pointed out, 17 the forecast accuracy really depends on the 18 variable that you're looking at. I mean it's 19 going to be different depending upon whether 20 you're looking at, you know, total primary 21 energy or whether you're asking yourself what 22 was the household consumption of natural gas, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 107 1 and each one of those is going to give you a 2 different problem to deal with and the 3 strategy that, as I understand it, you've 4 laid out is to say I don't always want to 5 have to deal with the entirety of NEMS. I 6 actually want to be able to get the essential 7 NEMS and so there is this process by which 8 you're trying to create a reduced form 9 representation; is that correct? So you're 10 actually going back and saying, you know, I 11 want, you know, I want the simplified 12 version. 13 And the right simplified version, 14 at least in my experience -- of course, 15 depends on that prior question -- that is, 16 what is it I'm trying to explain. If I'm 17 trying to explain the price, I'm going to 18 have one reduced form model. But if I'm 19 trying to explain total primary energy, I'm 20 going to have a different reduced form model, 21 and so forth. 22 From the perspective of these BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 108 1 questions then the question as I would see it 2 is if I'm looking at the aggregate 3 performance of NEMS, I have to kind of -- and 4 I've tried to create a reduced form version 5 of that model -- then, in fact, I have to 6 kind of put a wrapper around the model with 7 its, you know, what's exogenous and what's en 8 -- and what is the projection, and everything 9 else then kind of becomes a black box in the 10 middle. If you're looking at sub-modules, 11 as, in fact, you're doing here, where you're 12 kind of taking out a piece and actually 13 asking how's the performance of that piece, 14 you actually end up going back to the same 15 kind of question. What's the wrapper around 16 that piece and how do you actually get your 17 arms around that wrapper. 18 From the presentation, I couldn't 19 quite tell -- I mean there are well developed 20 methodologies for creating reduced form 21 models, and I'm going to presume that the, 22 you know, the sensitivity uncertainty BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 109 1 analyses that you would do to actually pull 2 -- you know, create the parameterizations 3 (ph.) and to do the attributions are being 4 done below that wrapper, but it might be -- 5 and it might be a good, you know, exercise to 6 just sort of go through and, in fact, discuss 7 how that, you know, that happens. 8 But once you've done that, then, of 9 course, you do begin to get those insights. 10 How much of the -- of the error was, in fact, 11 attributable to the fact they got the price 12 wrong? How much of it was attributable to 13 the fact that they just didn't have the 14 square feet of floor space right? How much 15 of it was the fact that they missed -- they 16 were using an average winter and, in fact, 17 the winter turned out to be much warmer than 18 normal and so forth? 19 And then, of course, you're left 20 with that uncertainty piece, which I presume 21 is what leads you to try to do this on an 22 annual basis to figure out how much of that BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 110 1 uncertainty is, in fact, coming out of the 2 structure, and, as the structure changes -- 3 the structure of the model changes, is, in 4 fact, that structure leading to a more 5 accurate representation of the behavior or is 6 that structure actually, you know, it's 7 evolving. It may be more attractive in some 8 -- in some aesthetic way, but it's not 9 actually performing better, and I think that, 10 you know, that actually is a really 11 interesting piece because, in fact, in the 12 world of modeling, what you always 13 implicitly, if not explicitly, assume is that 14 the model is, in fact, the true 15 representation of the world, and it's only 16 the data that carries the uncertainty. 17 Well, of course, you know, there 18 may be occasions when you actually had the 19 wrong structure of the world, and so this is, 20 in fact, a way of asking as the structure of 21 the world, according to NEMS, evolves is 22 that, in fact, evolving in a way that is BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 111 1 improving the accuracy or not. 2 And those are just -- those are the 3 thoughts that came to me in sort of trying to 4 reflect back. If there's some specific 5 questions you would have for us about how to 6 -- you know, that we could help use to help 7 you, you know, just let us know. 8 DR. NEERCHAL: Thanks. Yeah, I 9 want to make a couple of comments basically 10 starting with an apology. I do not know a 11 whole lot about -- I know very little about 12 NEMS, except that I know it's a very 13 important model, you know, it comes up every 14 other presentation that I've been through 15 this current year. I think it's a very 16 important thing, and I definitely think it 17 deserves full-scale sensitivity analysis, the 18 kind of thing you're proposing. 19 And, you know, from what I heard, I 20 think who else should be doing it, the one 21 who has created it. I think it seems like a 22 good choice. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 112 1 But I have a couple of comments. I 2 think that one thing when I look at or when I 3 hear about accuracy about something that I 4 don't know much about, so I'm going to be 5 doing it at the totally at the top level -- 6 actual, predicted (off mike). So there is 7 very layman's level assessment of accuracy. 8 I think that's what you're sort of referring 9 to your proposed action number one. I didn't 10 say look at a table and say here's an actual 11 column and here's the proposed column and if 12 there is a big difference, put a star saying 13 that this happened because of this weather 14 event that is not part of the NEMS. 15 So it's a very top level, if you 16 will. A kind of top level explanation of why 17 some things were not captured by NEMS' 18 forecast. I think that will be very useful. 19 The other thing would be I would 20 say the other extreme, and for, as a 21 researcher in NEMS, they might want to (off 22 mike) have a whole lot more and very detailed BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 113 1 sensitivity analysis, and I think that what 2 you're proposing, coming up with a design, 3 you know, properly chosen, I think that's the 4 way to do it, and I think in this (off mike) 5 literature there are papers -- I think Gerry 6 Sachs and Cohen -- but many papers on how to 7 analyze computer models, how to analyze the 8 uncertainty in computer models. There are 9 quite a few papers. I can give you a couple 10 of references on that work. 11 Statistical Sciences about maybe 12 six or eight years ago had a, you know, 13 leading article by Gerry and (off mike), a 14 very useful reference. 15 DR. BINGHAM: '93. 16 DR. NEERCHAL: '93. 17 DR. BINGHAM: Sachs is the first 18 author of that. 19 DR. NEERCHAL: And I think -- 20 something like that would be very useful in 21 deciding what the design points should be and 22 so on. That's one thing. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 114 1 I do want to ask you a question: 2 How much -- what do you mean by a full 3 archive of NEMS and how much effort is 4 involved in it? I was curious about that. 5 What do you mean by full archive? 6 And I was thinking is the code itself it 7 takes three hours to run or 10 hours? 8 DR. LADY: Well, the idea of -- 9 originally was that in some sense a third 10 party could run the model and (off mike) the 11 basis for any published projection. In terms 12 of the reviewers of this report and what I 13 would say, too, and did say, it would be a 14 version of the model that can be run when the 15 time comes with historical data substituted 16 for assumed data and then further run against 17 an experimental design where you can isolate 18 each influence. 19 Again, while that clearly is the 20 best thing to do, I suspect it's not 21 practical, at least for the moment, and I 22 further argue that the proposal that I've BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 115 1 made will probably do very well in terms of 2 serving the same purpose and be much simpler 3 to work with. 4 DR. NEERCHAL: I will open. Yes. 5 DR. KOKKELENBERG: How many -- you 6 mentioned you had a set of AEOs back in '98. 7 DR. LADY: Yes. 8 DR. KOKKELENBERG: But how -- when 9 did the AEOs start? When was the first year 10 it was published? 11 DR. LADY: 1974 would be the first 12 time that the -- 13 DR. KOKKELENBERG: Okay. So that 14 data from '74 to '97 is not available or? 15 DR. LADY: That I don't know. 16 DR. KOKKELENBERG: Okay. 17 DR. LADY: I -- there's a graphic 18 interface for NEMS that allows -- that first 19 of all requires that the solutions be 20 generated in a form that the program can 21 read, and then I can use that program to take 22 data out to do statistics with or do anything BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 116 1 I want with, and that starts with the '98 2 version. 3 DR. KOKKELENBERG: I see. Okay. 4 So then when you're doing your tests and 5 you're calculating your elasticities, how 6 many observations did you have for let's say, 7 you calculated the elasticity of residential 8 natural gas consumption? 9 DR. LADY: All I'm probably -- 10 DR. KOKKELENBERG: How many years 11 do you have? 12 DR. LADY: I probably did it -- I 13 would do it for three solutions, 20 years per 14 solution, so I would have 60 -- 60. 15 DR. KOKKELENBERG: So you would 16 have it for -- those three solutions were 17 let's say the high, low, and medium GDP 18 growth or something like that? 19 DR. LADY: Well, the oil price 20 cases would be the ones that would be -- 21 DR. KOKKELENBERG: Okay. 22 DR. LADY: -- proper. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 117 1 DR. KOKKELENBERG: So you'd have 20 2 observations -- 3 DR. LADY: Per solution. 4 DR. KOKKELENBERG: Per solution. 5 DR. LADY: And then I would pool 6 them. 7 DR. KOKKELENBERG: Okay. So you'd 8 pool those. Would that maybe explain why 9 you're getting this high variability in your 10 elasticities. 11 I mean first of all, you have a 12 short number of years, 20 observations, and 13 then you're pooling them and so the standard 14 error on an individual elasticity, evaluated 15 at any particular year, is probably going to 16 be pretty high? 17 DR. LADY: Well, not in the 18 regressions they're not. 19 DR. KOKKELENBERG: You mean the 20 standard error -- do you calculate the 21 standard error of the elasticity, not the 22 coefficients but -- it would be if you used BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 118 1 the log form to do that. 2 DR. LADY: No, I didn't do that. I 3 could use -- I didn't use the log form, so 4 I'd only have the P-statistic on the 5 coefficient. 6 DR. KOKKELENBERG: Okay. 7 DR. LADY: Not on elasticity. 8 DR. KOKKELENBERG: Okay. If you -- 9 you can calculate the elasticity standard 10 errors, but you have to recognize whatever 11 form you have and how many times that 12 variable enters in. But that may be one way 13 to looking at why you get these -- I'm really 14 disturbed by these elasticities that jump all 15 over the place, from 1.4 down to negatives 16 for some (off mike). 17 DR. LADY: Well, I suspect that the 18 specification is too austere. I don't think 19 -- I think that there's some reason to think 20 that this is my fault for not quite using a 21 sufficiently detailed specification. 22 DR. KOKKELENBERG: I'm not sure BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 119 1 that -- I don't know. What -- do you think 2 that would help? I don't think it would. 3 DR. KHANNA: It's hard to say. 4 DR. KOKKELENBERG: It's hard to say 5 if it would help, but I think, particularly 6 since you're using the lagged value, which 7 should be capturing everything that 8 influences the lagged value. Theoretically, 9 that model makes a lot of sense. 10 DR. LADY: I agree. 11 DR. KOKKELENBERG: And I believe 12 people have said this for years that they 13 can't forecast what influences electricity 14 demand, but if you use last year's 15 electricity demand, it's a good proxy for all 16 that equipment that's in place that demands 17 electricity. 18 And -- 19 DR. LADY: Well, I can tell you 20 that I have run the log form, these 21 regression utility that makes it easy to do, 22 and the T-statistics on the coefficients, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 120 1 which are the elasticities in that form are 2 very high. In other words, the regression 3 doesn't make it look like there's a large 4 variation for each case. 5 DR. EDMONDS: Well, the first 6 question is, you know, in that module, did 7 you actually do a full sensitivity analysis 8 so that you know every, you know, every ex -- 9 in that module, every exogenous variable and 10 what its potential contribution to -- you see 11 that, you know, that's what may be what he 12 means when he says too austere -- 13 DR. LADY: Yes. 14 DR. EDMONDS: -- that, in fact, if 15 you went back and you did a -- you know, if 16 you systematically went through and just -- 17 which is kind of the first step and when 18 you're trying to do this reduced form model 19 is you just take everything that's outside 20 the model, outside that module, and you just 21 perturb it, you know, one percent or whatever 22 the appropriate value and just see what's the BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 121 1 impact on the dependent variable of interest 2 and that's kind of where you start and then 3 you have to put weights on those to get a 4 sense of, you know, are some of these things 5 just, you know, they never change, you know, 6 it's the solar constant or something, which 7 actually does change, but let's set that 8 aside. 9 But anyway, so, you know, that, in 10 fact, may be what you're referring to in it's 11 too austere. There may be something in there 12 which actually is more important than you are 13 giving it credit for a priori. 14 DR. NEERCHAL: Derek. 15 DR. BINGHAM: So I'm going to 16 change the topic here slightly so. Just 17 regarding kind of your overall plan, what 18 you're describing of, describing in the 19 statistics literature would be called a 20 design and analysis of (off mike) which 21 Nagaraj was referring to. So there's been a 22 thematic year this year at SAMSI and at also BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 122 1 at NCAR on this. And I work in this area, so 2 I can actually tell you a little bit about 3 this. 4 So let me ask you a quick question: 5 Is NEMS deterministic? I mean if you put the 6 same inputs in, do you get the same output? 7 DR. LADY: I don't believe so. 8 It's a numerical algorithm, so there is a 9 tolerance definition to solution. 10 DR. BINGHAM: So if I were to run 11 it and you were to run it, we would get 12 different answers? 13 DR. LADY: No, you should get the 14 same answer, yeah. It's deterministic. 15 DR. BINGHAM: Okay. So any 16 regression function that you fit to this, 17 okay, is going to be a wrong model because 18 all of the model -- your errors are not 19 independent because everything comes from -- 20 there's no randomness. It's all model mis- 21 specification, and that's why typically what 22 people will do instead is use something like BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 123 1 a Gaussian process model where you kind of 2 view the function as a random function. 3 And so there's a whole literature 4 out there on this, and there are also design 5 experiments that kind of go along with this 6 as well to basically what you want to -- the 7 worst thing you could do is do a one at a 8 time sort of thing. 9 What you want to do is you have an 10 input space and you want to fill that space; 11 right? So there are inputs into your model 12 and what you want to do is somehow fill this 13 as best as possible and run the model at a 14 whole bunch of different places, and you can 15 either keep those runs as archive or you 16 could try to build surrogate models from 17 those. So rather than keeping all the runs, 18 you could build a series of surrogate models, 19 either for component wise or for a global 20 model, based on -- you wouldn't try a 21 regression model. It would be -- the order 22 of the polynomial or something that you would BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 124 1 need. It would be way too high. And you 2 would use something called a Gaussian process 3 model or some sort of non- parametric 4 regression model. 5 And right and so -- what the folks 6 at NASA and Los Alamos and NCAR will do is 7 they'll archive the -- that model and use it 8 as a surrogate for the computer code later 9 on, and also with associated uncertainties. 10 And if you're looking through that 11 literature, there's a slight disconnect, so 12 one of the things that you mentioned that you 13 wanted to do was design experiments and also 14 identify the important drivers and in that 15 literature it would be called -- that would 16 be called screening; right? 17 So you're looking for the active -- 18 the big guys, the things that are really 19 driving the system. Whereas sensitivity 20 analysis is typically a post-hoc thing, so 21 I'm just giving you the definitions in that 22 literature. If you decide to dive into it, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 125 1 the sensitivity analysis is usually deciding, 2 you know, what is a functional form. So 3 would something have a quadratic effect or, 4 you know, some sort of periodic effect or 5 linear effect on any type of output. So that 6 would fall under sensitivity analysis and 7 there's a set of references -- there's 8 actually a book by Cytelli on sensitivity 9 analysis, which has a different -- that 10 phrase has a different meaning in almost 11 everybody's field. 12 When you're looking at computer 13 models and doing experiments on computer 14 models and simulate or computer simulators of 15 physical processes so sensitivity analysis is 16 usually a post talk, you know, an 17 explanatory, you know, explaining what's 18 going on; whereas screening is usually 19 identifying the big guys. And there is a 20 broad literature on that, and we can give you 21 a list of references offline if you want. No 22 need to -- BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 126 1 DR. LADY: Okay. 2 DR. BINGHAM: -- list them all now. 3 But I also think that that parts of this 4 would be so given that there's been a lot of 5 effort to try to get -- there was a summer 6 school that was held on computer experiments 7 last year that was part of this thematic year 8 at SAMSI, which is an applied math and 9 statistics institute affiliated with NISS. 10 There's a lot of people out there who are now 11 trained in it, young people, graduate 12 students, who might fit very well into this 13 graduate program that you're trying to set 14 up. It would be a perfect place for an 15 internship to kind of transfer some of that 16 knowledge both on the design and on the 17 analysis side. 18 So I think this is a methodology 19 that would be very useful for you that's kind 20 of sitting out there, and there's lots of 21 code available -- 22 DR. LADY: Okay. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 127 1 DR. BINGHAM: -- on the Web, so you 2 could probably steal their code and start 3 there. 4 DR. NEERCHAL: Good refer (off 5 mike) in the summer. Advice. I'll say. 6 Keep them off the street. Walter? 7 DR. HILL: Nagaraj, you introduced 8 that you know very little about these models. 9 I know less. This has been a great break out 10 session. 11 DR. NEERCHAL: You mean (off mike)? 12 DR. HILL: Yeah, I've made a note 13 in reference to your comment that it looks 14 like their first model of observation, so 15 that may partially explain the reason why 16 you're getting a fairly large error. 17 You also brought the -- your list 18 of unknowns. It looks like those -- a number 19 of those will be unknown that you will not 20 know the number of (off mike) or accidents 21 (off mike) by definition. It won't be known 22 ahead of time. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 128 1 DR. LADY: I can't hear. I'm 2 sorry. 3 DR. HILL: That number of (off 4 mike) -- some of these -- that number of 5 accidents will not be known ahead of time, 6 and so it will be difficult to put that in 7 the original model, although you did mention 8 I think (off mike) in the current model. 9 DR. LADY: You mean that (off mike) 10 definition model? 11 DR. HILL: Some of these transitory 12 influences, like accidents, for example, you 13 will not -- 14 DR. LADY: Right. 15 DR. HILL: -- you will not know. 16 You either do not know. You will not know 17 how many accidents that will be in the summer 18 of this year. 19 DR. LADY: Sure. I don't know 20 about (off mike) Katrina. 21 DR. HILL: Yes. 22 DR. LADY: That's right. But I BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 129 1 think it should be part of what's done? 2 DR. HILL: Right. And (off mike) 3 that, but in -- 4 DR. LADY: Mm hmm. 5 DR. HILL: -- eventually that's put 6 in the (off mike). 7 DR. NEERCHAL: You want to use the 8 microphone for the transcription. Yeah. 9 SPEAKER: There's a problem. He 10 can't hear Walter until she stands there, and 11 then she can't hear him when he talks. 12 SPEAKER: There's a lavaliere mike. 13 SPEAKER: Hello. Hello. 14 SPEAKER: Speak louder perhaps. 15 DR. LADY: I -- it is my proposal 16 that the institutionalization of this idea 17 would include accounting for what you might 18 call offline influences as well as the sort 19 of analytical things that we've been talking 20 about, like Katrina; yes. 21 DR. HILL: Perhaps those are put in 22 the later model. Those are not in. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 130 1 DR. LADY: I'm not so sure that 2 that's -- by model you mean what's done to 3 generate -- 4 DR. HILL: Post hoc. Post hoc. 5 What is the effect of the Katrina? 6 DR. LADY: I would view that 7 probably to be simply offline, and I'm not so 8 sure you would want to try to model that. 9 DR. HILL: But it must affect 10 energy use? 11 DR. FORSYTH: The star. 12 DR. LADY: Mm? 13 DR. HILL: It must affect -- 14 DR. NEERCHAL: The star, yeah. 15 Star. That's -- you know, you're talking 16 about the weather and I think something that 17 you want a, you know, a regular web browser 18 will catch saying wait, this actual doesn't 19 fit the predicted outcome rate. 20 DR. LADY: Mm hmm. 21 DR. NEERCHAL: I think those are 22 easy -- I mean (off mike) easy, but you can BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 131 1 have some identifiable causes saying if this 2 Katrina; this is, you know -- 3 DR. FORSYTH: The or -- 4 DR. NEERCHAL: -- or whatever, you 5 know. Whatever else is there -- I think that 6 something, you know, understandable in a 7 layperson's language; whereas, elasticity and 8 those kinds of things are not understandable 9 by laypersons' language. If there is a 10 problem due to issues like Ed is raising and 11 common people cannot really -- but they don't 12 really get into that kind of level, so that 13 is a different level of, you know, accuracy 14 check. You know, so I was trying to divide 15 them into groups in my mind to understand 16 this NEMS accuracy check one, from a very top 17 level -- top or bottom, I don't know -- 18 easier perspective; right? I don't want to 19 get into that. 20 So at one extreme of very low level 21 knowledge like me about NEMS and the other 22 one is someone who is more -- (off mike) more BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 132 1 expert level knowledge of NEMS. You know, 2 how do you -- how do you convince them? 3 So the expert people will not 4 bother with the small change. They know that 5 it's a model. It's -- I mean it's supposed 6 to be wrong sided most of the time. It's an 7 accident if it keeps right on the target. 8 You know, so experts shouldn't really be 9 excited about matching. Experts should 10 expect a certain error, because it's a model. 11 But what the expert will be divided about, 12 for example, if you look at the NEMS -- those 13 are beautiful runs -- the last table, if you 14 look at 1998, '99, 2004, and actual listed in 15 the next column, most of the time the numbers 16 are actually moving towards the actual, 17 almost always. You know it starts with a big 18 gap and then slowly they converge. 19 I mean those are the things that, 20 you know, an expert might notice, but not a 21 layperson. You know, he's going to -- oh, I 22 was right on 2001, because, you know, I put a BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 133 1 thousand dollars in the Vegas or something 2 like that. The layperson is going to worry 3 about instantaneous issues. 4 So for those people when they -- 5 when you are not matching or when you're 6 significantly different, you may want to be 7 able to explain the identifiable cause and 8 say this was not -- this is not (off mike) 9 this way. That's what I was saying. I don't 10 know whether you are referring to the same? 11 DR. HILL: Yes. 12 DR. KIRKENDALL: I think I sort of 13 -- 14 DR. NEERCHAL: I think Neha has 15 been waiting for a while and then I think now 16 I forgot we tell the two Eds. 17 DR. KHANNA: So my comment is 18 possibly completely off the (off mike) 19 because I may have (off mike) misunderstood 20 what you do, but it's actually is related to 21 what Derek was trying to say. The different 22 outcomes that you get you essentially say BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 134 1 it's the weather. Heating degree days were 2 higher than what we had modeled. You know, 3 here's one heating degree day where we look 4 at the output in electricity demand, et 5 cetera. 6 There's another heating degree day 7 now going to look at what electricity demand 8 number NEMS spits out for a different value 9 heating degree day; right? 10 And that's how you calculate (off 11 mike). So because the model is 12 deterministic, the only reason that those 13 numbers differ and you can't predict a priori 14 how they're going to differ or exactly what 15 the difference is going to be, it's because 16 of the inherent non-linearity in the model; 17 right? 18 And we don't know exactly what the 19 shape of that non-linearity is. 20 DR. LADY: I suspect not, because 21 my experience is that what you might call the 22 variable parameter manifold that describes BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 135 1 the model is pretty flat and, in fact, for 2 many of the major series, just using linear 3 relationships, you could get a very good fit 4 with the NEMS solution. So I suspect that 5 the differences in the elasticities that 6 we're commenting upon are due, in part, to 7 the need for a better specification than the 8 one I used. But even in the NEMS experiments 9 themselves across the three years where it 10 was done properly, (off mike) statically with 11 the modules, you got big differences as well. 12 So the modelers would be the ones who could 13 explain why the model is changing that way I 14 think. 15 I mean part of this is to reveal 16 what should excite interest and explanation. 17 DR. NEERCHAL: Sorry. 18 DR. BLAIR: Actually, I'll defer to 19 him. 20 DR. NEERCHAL: Okay. 21 DR. KOKKELENBERG: Oh, well, I 22 guess I'm totally confused. The NEMS itself BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 136 1 is deterministic and that -- evaluating the 2 computer models was what our deterministic 3 was addressed by Derek as far as I can tell. 4 But what you're doing, the way I 5 understand it, is taking the forecasts that 6 NEMS made and then comparing that to the 7 actual, which is a test of forecast accuracy. 8 So here there is a stochastic element. So 9 it's -- you've got about three different ways 10 you're trying to evaluate NEMS. One is the 11 way that Derek suggested, taking a hard look 12 at how deterministic computer models arrive 13 at solutions and what could be done to 14 enhance that. 15 The second is to look at forecast 16 accuracy, and see, you know, take the worst 17 things you forecast and why are they so bad. 18 And then the third thing is what 19 Neha was talking about was scenarios 20 essentially; tweak a variable, and then see 21 if that comes closer to the actual in your 22 model. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 137 1 But then you need a rather complex 2 -- well, you really need the NEMS model to do 3 the tweaking. You can't just use the similar 4 equations. 5 So and there's probably five other 6 way to test the accuracy that I haven't 7 immediately thought of. But I think you're 8 going to have to -- well, didn't we publish 9 or EIA publish a study of the NEMS 10 forecasting accuracy very recently as part of 11 the -- 12 DR. KIRKENDALL: One of the NEMS 13 people want to speak to that. 14 SPEAKER: Yeah. We can clear up a 15 lot of this stuff right here. 16 DR. KIRKENDALL: Well, come up. 17 DR. KOKKELENBERG: Okay. 18 DR. KIRKENDALL: Come on up to the 19 table and talk. 20 DR. KOKKELENBERG: That's what I 21 was going to say. What's their experience? 22 DR. NEERCHAL: Come. Come, come, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 138 1 come. 2 DR. KIRKENDALL: There's a couple 3 of seats here. 4 DR. NEERCHAL: Grab one of the 5 chairs here. 6 SPEAKER: So I (off mike) spoke up 7 earlier about it. I didn't know what the 8 protocol was. 9 Okay. So the reason the 10 elasticities change over time, the portion of 11 heating and cooling to total energy, which is 12 what you're looking at is changing. So in 13 2030, the amount of energy attributable to 14 heating and cooling is less than it was in 15 the beginning of the forecast. So you're 16 going to get a smaller elasticity, and you 17 can see that on your handout. 18 DR. LADY: I think it was the price 19 elasticities, not the weather elasticities 20 that they were talking about. 21 DR. KOKKELENBERG: Well, both 22 elasticities do vary immensely. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 139 1 DR. LADY: Well, I'm sure there's a 2 good reason. 3 SPEAKER: And it still applies. 4 DR. LADY: The weather elasticity 5 -- I'm sure that you would know. I didn't 6 think that was an anomaly. It was just I 7 didn't know why that (off mike). Yeah. 8 SPEAKER: No. All right, and we 9 did publish the forecast. The evaluation 10 just came out a couple of weeks ago? 11 So we do that, and we do the 12 tweaking and I think I've shared with you the 13 table where I actually went with the 14 residential model went back and changed like 15 four or five key variables; put in the 16 history that I knew; and the forecast -- and 17 then the rest we would consider model error. 18 So we got down to about one to three percent. 19 DR. LADY: Right. I propose that 20 what you did in that one case be generalized 21 and done regularly. Yeah. You're right. 22 That's -- BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 140 1 SPEAKER: Yeah. I mean you got to 2 understand that the models run. It's not a 3 regression model. It's an NU structural 4 model and so the uses that are affected by 5 weather and price are different now than they 6 are forecasted so the elasticities will 7 definitely change. 8 Refrigeration, if your electricity 9 price goes up, there's nothing in the short 10 run you're going to do to change your 11 refrigerator energy consumption. 12 However, heating you can: You can 13 change the thermostat, so the concept here I 14 think is very, you know, for me, 'cause I do 15 it, it's easy to understand, but there's no 16 reason to think that the elasticity would not 17 change over time, whether price or -- weather 18 or price. 19 DR. LADY: I was just measuring it. 20 I didn't -- 21 SPEAKER: And then why the weather 22 is different? I'll address that real quick BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 141 1 while I'm up here. What I do is I use the 2 last 30 historical years, so that's '76 to 3 2005. That's my normal. The NOAA official 4 normal is '71 to 2000, and then from 2000 on 5 they used fixed population weights for the 6 states. I actually used the actual from 7 Census that they have. So that's why they 8 don't mesh. 9 DR. BLAIR: Ed Blair. I had some 10 comments along the lines of what the other Ed 11 just made, but I -- the main problem I have 12 is I lost track somewhere along the way of 13 just where we're going and what the end 14 objective of the exercise is. I did get the 15 idea that our goal was to in effect try to 16 partition the forecasting error and try to 17 figure out where it was coming from. But 18 then for what purpose? You know, simply to 19 have that, and it seemed to me that in a 20 sense where we were heading toward was the 21 possibility of targeting model inputs for 22 measure improvement potentially of saying, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 142 1 Jeez, you know, we're consistently taking a 2 beating here. You know, this is a place 3 where we really need to get better. 4 Or potentially revising the model, 5 and I don't know if this language will even 6 make sense to better statisticians in the 7 room, but that in a sense what happens is as 8 we take our in sample information and make 9 predictions, you know, well, we're always 10 making predictions to an out of sample 11 environment, you know, which the new year. 12 And it turns out that a variable that 13 performs very well, you know, on a backcast 14 basis just, you know, it's sort of a mean 15 square error kind of context. It's - - you 16 know, it's got a lot of variance associated 17 with it and we, you know, we'd be better with 18 a model specification that just doesn't rely 19 so heavily on that variable or a different 20 form of the model that is more robust and I'm 21 not sure if I should be saying less sensitive 22 or -- BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 143 1 (Laughter.) 2 DR. BLAIR: -- or I'm not sure 3 which word, you know, which word I should be 4 using, but in a sense it seemed to me that 5 that was the direction we were going. In 6 that sense, it was not clear to me that the 7 screening, sensitivity, I'm not sure what 8 word to use -- it wasn't sure to me that the, 9 you know, sort of the experimental design of 10 varying the parameters and seeing how the 11 model responds to these -- I mean the only 12 thing I could see that you were going to get 13 out of that was that you would identify which 14 variables of the model showed responsiveness 15 to or the forecasts showed responsiveness to 16 and when you sort of thought about which of 17 those are frail, either because they're 18 inherently high variance or because they're 19 poorly measured, that you would say, whoa, 20 you know, we've got a problem there. Okay. 21 But if we just on a sort of a historical 22 basis go back and say where, you know, in BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 144 1 effect try to partition where did we get 2 problems, that's sort of built in. It's all 3 built in. 4 So I wasn't clear what the 5 experimental design would do for us at the 6 end. I'm sorry if I've been confusing as 7 I've wandered through that. 8 DR. LADY: My meaning of it was 9 that -- and I would have to look at the 10 literature that I've been advised to look at 11 -- was that the design would create a dataset 12 in essence that would allow an analysis of 13 the sources of error differentially. That's 14 what I meant in essence. 15 And that isn't. I didn't present a 16 resolved such design, and I've been told that 17 I should go look and I will at the work 18 that's already -- there's already a 19 literature on this. So that's what I meant, 20 just at the -- an overview level. A proper 21 design so that the dataset that was gotten 22 would serve the purpose that the archive BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 145 1 model would have served, and yet it would be 2 much cheaper and easier to accomplish. 3 DR. NEERCHAL: Jae. Last one. 4 DR. EDMONDS: I was just going to 5 go down that same, you know, path. We've 6 talked about some specific representations. 7 My thought -- my question is sort of stepping 8 back one step and what is -- you know, which 9 of these variables have been chosen for 10 examination so that you do a better job of 11 representing them in the future and what was 12 that process? In other words, you're 13 creating reduced, you know, you're creating a 14 set of reduced form models to predict certain 15 -- to -- for particular outputs of NEMS. 16 That is, you're trying to be able to say what 17 would the NEMS model have predicted if I'd 18 run the whole thing and, you know, but I'm 19 not -- I'm going to actually use a reduced 20 form representation of it. 21 So that's the problem that you -- 22 you know, you're addressing, which is how do BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 146 1 I do that and, you know, how do I learn from 2 that so that, in fact, I reflect back to the 3 real NEMS information that helps that real 4 NEMS do a better job. 5 Well, the very first question I 6 have is what are the variables that have been 7 selected that you're trying to say, all 8 right, NEMS has got a vector of outputs and 9 there are N of them -- maybe N is eight; 10 maybe N is 10; I don't know what it is -- how 11 were those chosen and, you know, why were 12 those chosen and, you know, what was the 13 process by which this whole arg -- the whole 14 design, you know, the whole project 15 architecture was created? 16 DR. LADY: Well, they haven't been 17 chosen at this point, and the specification I 18 used, as was mentioned, is just a traditional 19 one that an economist would come up with. 20 My proposal is to talk to the 21 modelers. They know how it works, and to get 22 their estimate as to the right set of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 147 1 important parameters to keep track of and 2 base the specification on their 3 recommendations. And that is the very thing 4 that I hope and propose to start doing next 5 week. 6 In other words, they know. I don't 7 know. I'm good at some things, but I -- but 8 they know the model, and they know far better 9 than I how to set up a proper regime of 10 sensitivities. Now, there may be some 11 mathematical twists that I need to learn as 12 to what to do with that, but they are the 13 experts. 14 DR. NEERCHAL: Yes, just a quick 15 question, more of a clarification. Any model 16 that produces an output is essentially 17 producing a point estimate because of the 18 parameters that are built into the model. 19 So one of the issues to keep in 20 mind is when you're talking is when you're 21 talking about forecast accuracy, we've got a 22 point estimate and we can (off mike) that BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 148 1 point estimate with non- stochastic 2 observation. And I mean I would worry about 3 the forecast accuracy of the model if it was 4 systematically underpredicting or 5 overpredicting. But if it is randomly 6 predicting differently, I don't know if we 7 have a problem at all. 8 DR. LADY: From my own perspective, 9 to have some protocol that assesses accuracy 10 and understands in some sense the underlying 11 reasons for it is proper science, whether it 12 does the modelers any good or not. In other 13 words, as a credential for published 14 information, I believe that that should be 15 known, although I do believe that if this is 16 done in the end that the diagnostic 17 assistance that can come from it could be 18 significant as well. 19 But from an outsider's standpoint, 20 to make these projections and to do it with 21 quite a substantial analytical engine, you 22 should know how it's working at the level of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 149 1 what the model is trying to do simply as a 2 credential for the information. 3 DR. NEERCHAL: I think we should 4 wrap up - - oh, you know, well, one of -- 5 regardless of what the objective is, I think 6 what Derek said I think -- because it is 7 computer model. It's a numerical 8 deterministic model, there is a literature 9 out there how statisticians can look at it 10 and they actually go through these different 11 scenarios, you know, what they are looking at 12 and so on where the answer (off mike) is 13 coming from. 14 Sometimes the inputs are 15 deterministic, some numbers that are pretty 16 viable, and sometimes there is measurement 17 error in the inputs themselves. You know, 18 they have to be treated differently in this 19 context. 20 So I think it would be really 21 helpful to take a look at that literature I 22 think. So. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 150 1 Please identify yourself just for 2 the record. 3 MR. SKELLY: I'm Dan Skelly. You 4 know, we publish five main scenarios each 5 year. High macro. Low macro. High price. 6 Low price. 7 The impact analysis George 8 presented, was for the reference case. Now, 9 if you did that impact analysis for the other 10 cases, I can imagine you might get two types 11 of results. One it would be similar across 12 all cases and maybe that would validate the 13 approach. But what if you got different 14 impact analyses across these five scenarios? 15 What would that tell you? 16 DR. LADY: Well, I would be alarmed 17 if it was very different across the 18 scenarios. In other words, I would think 19 that the approximation, if that's the right 20 word to use, that's being used to parse the 21 errors would not show great differences, so I 22 would worry about the method if I found that. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 151 1 MR. SKELLY: So doing that would be 2 a -- 3 DR. LADY: Yep. 4 MR. SKELLY: Something you should 5 do as part of this? 6 DR. LADY: Sure. 7 DR. NEERCHAL: Ed. 8 DR. BLAIR: Ed Blair. I'm really 9 hesitant to speak because I feel I'm better 10 to remain silent than let people think you're 11 a fool than open your mouth and (off mike). 12 (Laughter.) 13 DR. NEERCHAL: That's not stopping 14 us, by the way. 15 (Laughter.) 16 DR. BLAIR: Going back to storing 17 the model, going back to the idea as an 18 alternative to storing the model, running 19 conditions and retaining a record of those, 20 it seems to me that if the -- how to put this 21 -- that at one level, I don't need to know 22 what the model was. I just need to know what BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 152 1 the forecast was and I can go back and I can 2 look at differences between the inputs that 3 were used in the model and the now known 4 correct inputs. 5 Now, I would appreciate that some 6 of those inputs which may be in here are 7 intermediate inputs, which is going to be a 8 problem. But I can go back and weather, 9 strikes, you know, accidents, embargos, you 10 know, changes in the laws. I can go back and 11 I can correct and I can in a sense run the 12 correction against model error and that gives 13 me a direct sense at the variable measurement 14 level of which variables are killing me and 15 then everything that rolls to the residual or 16 the uncertainty is a candidate for 17 specification error. I don't know if this 18 makes any sense to anybody else. 19 But then this is a candidate for 20 specification error and is the objective here 21 to try to get a window into specification 22 error, because it seems to me that where we BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 153 1 were going was we were getting essentially 2 variable-level results, which I could have 3 gotten directly. 4 So that's why I'm just not getting 5 this. 6 DR. HILL: I was trying to raise a 7 point like that, that these things are 8 unknown ahead of time. If there -- if the 9 model is properly specified, the essence of 10 the (off mike). So it's just along a 11 variation (off mike). 12 DR. BLAIR: So to me, I would not 13 be running scenarios, well, then and I guess 14 this is where I lost track of exactly what's 15 happening here. I would not be running 16 scenarios of imagine a particular input being 17 10 percent high or 10 percent low anything 18 like that. In a sense, I'd be running 19 variations in the specification and then 20 being able to go back and see how the 21 variation in specification related to the 22 ultimate error. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 154 1 DR. LADY: In my opinion, this does 2 not necessarily imply the project itself that 3 there's anything wrong with NEMS. 4 DR. BLAIR: No, it doesn't. 5 DR. LADY: Or that it's 6 misspecified at all. In fact, it could be 7 perfect. I just think that when the time 8 comes you should be able to report why the 9 projection was different than the event and 10 to do it in a way which conforms to the 11 economic logic of NEMS and can speak to the 12 supply and demand interactions that NEMS is 13 trying to project. There doesn't have to be 14 anything wrong with NEMS specification at 15 all. If the price was reasonably assumed in 16 the year 2005 to be 24 percent less than it 17 turned out to be five years later, which is 18 not anybody's fault, you should have a way of 19 seeing the degree to which this would explain 20 the fact that the forecast that was made was 21 higher than it should have been, because it 22 didn't take into account that higher price. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 155 1 Now I believe that there's a lot of 2 learning by doing that this project will give 3 rise to and I think that from the modeler's 4 standpoint if this is done right, that it can 5 be of help to things like specification. But 6 there is no presumption in my mind that there 7 is such a problem. In other words, there's 8 nothing wrong -- as far as I'm concerned NEMS 9 is one of the most sophisticated models 10 anywhere outside of defense anyway that I 11 know anything about, and I want a credential 12 for its results rather than a criticism of 13 its nature. 14 DR. NEERCHAL: Walter? 15 DR. HILL: Well, this could be and 16 since it's a verification of the model is 17 correct if you -- 18 DR. FORSYTH: Right. So it's just 19 random. 20 DR. HILL: Yeah. That there's 21 random variation. You won't be able to do 22 any better, because you can't make (off BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 156 1 mike). You don't know ahead of time what the 2 value of some of these variables are. 3 DR. LADY: You can't predict the 4 weather. 5 DR. HILL: You don't know how many 6 heating degree days will be in the summer, 7 this summer. And who (off mike) this summer. 8 DR. NEERCHAL: Yes. I think so 9 we'll go. Thank you very much and thank you 10 for the discussants and all the comments, and 11 so let's -- next time we have in the agenda 12 here Committee suggestions for topics at the 13 fall 2007 meeting. I think, you know what? 14 Spending a few minutes. Email to Nancy. 15 (off mike). So you don't have to think that 16 this is where you have to come up with a 17 topic right off the top of your head. 18 DR. KIRKENDALL: The other thing 19 just to remind you last -- yesterday, Guy 20 talked about the 30th anniversary of EIA, 21 which is also the anniversary of the 22 Committee, and if you have any ideas for what BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 157 1 we could do that would involve the Committee. 2 DR. KOKKELENBERG: Oh, for the 30th 3 anniversary? 4 DR. KIRKENDALL: Yeah, for the 30th 5 anniversary. John Paul is the Chair of the 6 History Committee and he's going to be taking 7 an active role in planning things, and so he 8 was particularly interested in any ideas you 9 might have, so I thought I'd bring it up 10 again in case you've forgotten that (off 11 mike) that you were invited. 12 DR. KOKKELENBERG: Can we have one 13 conversation? 14 DR. NEERCHAL: Yeah. I think -- 15 DR. WEINIG: George, Derek. Would 16 you take that off mike so that our 17 transcription -- 18 DR. KHANNA: Well, one thing I can 19 (off mike) I didn't think about is somehow 20 incorporate the celebration, it would be 21 great at least for the Committee members 22 because you know the history of the Committee BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 158 1 -- what exactly -looking back 30 years, what 2 are the great things this Committee has done 3 here? Why do you insist on having us around? 4 DR. NEERCHAL: We are very 5 expensive? 6 DR. EDMONDS: I certainly want to 7 ask that question. 8 DR. NEERCHAL: I think another 9 question, a related question will be that is 10 it possible to have the celebration around 11 the same time as the Committee meeting so 12 that it reduces travel. 13 DR. KIRKENDALL: So you could 14 participate more broadly -- 15 DR. NEERCHAL: Right. 16 DR. KIRKENDALL: If you wanted to? 17 DR. NEERCHAL: Exactly. That may 18 be a -- I mean for me, it's okay. I have a 19 short travel (off mike). In terms of topics, 20 one thing I thought, you know, I (off mike) 21 the most exciting thing I learned in this 22 meeting is fellows and the graduate students BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 159 1 (off mike) that you are thinking, 2 undoubtedly, the most exciting thing I think. 3 As a (off mike) I think there's no doubt 4 about it in my mind it's the most exciting 5 thing and the most important thing you can do 6 is to train the workers. 7 So I think that could be a kind of 8 a regular thing in your agenda, make a place 9 for it, instead of people, when you have 10 identified those people, if they're working 11 here, make sure that they have some 12 presentation time, and because they are 13 students, don't put them in the exact same 14 format as everybody else. You know, make it 15 a little more friendly looking, because it 16 almost -- this is very much like a Ph.D. 17 defense. 18 DR. KIRKENDALL: Yeah. 19 DR. NEERCHAL: And (off mike) I 20 don't know that that's good or not. It could 21 be good also, you know. You know, I put my 22 kids through that weekly. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 160 1 DR. KIRKENDALL: Oh, no, your poor 2 kids. 3 DR. NEERCHAL: We have a graduate 4 student seminar where the student presents 5 and then you have the faculty criticizing 6 their presentation right in front of 7 everybody else. Every week, there are two 8 (off mike) for this torture, and our kids are 9 very good because of this. It has been going 10 on for about 10 years, and I think that will 11 be nice. 12 And also I think before you go on 13 ahead, I'm sure, you know, this might come to 14 you with a specific plan. I think this (off 15 mike) good place to really (off mike). Many 16 of us actually have thought about this early. 17 DR. KOKKELENBERG: I was going to 18 add John Paul might be able to get a hold of 19 (off mike) of the names of all the past 20 Committee members and the chairs and so 21 forth. I know you can't afford to invite 22 them all here, and they won't all come here BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 161 1 even if you invited them, but you might 2 consider at least having some sort of 3 biograph (off mike) since it's been in 4 existence for 30 years and you're probably 5 talking upwards of 500 people, because, you 6 know -- 7 DR. WEINIG: It's a six-year span. 8 DR. KOKKELENBERG: Well, some of 9 them are only three-year span. Some of them 10 are (off mike). 11 Maybe -- okay 300 so far. 12 DR. KIRKENDALL: Still, it's a 13 reasonable number by this time. 14 DR. KOKKELENBERG: But also it 15 might be useful to build on what has been 16 said already. Nagaraj may have been going 17 there, and I may have missed this, but if the 18 Committee has some participation or has a 19 meeting around that time, it might be useful 20 to allow the Committee to invite -- and I'm 21 not going to talk about out of who's pocket 22 book this goes out of -- maybe one of their BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 162 1 grad students that, if appropriate, could 2 come to the meeting to see what goes on a 3 these kinds of meetings. It might help you 4 for this research fellowship or whatever 5 we're going to be calling it, and it may also 6 -- I don't think we'd invite so many that 7 there wouldn't be any room for the staff to 8 come to the meeting. But, because if you 9 send about -- well, you could invite one if 10 you thought it was appropriate and why it's 11 appropriate. But I'm not sure that that's 12 the best idea, but it's an idea. 13 DR. FORSYTH: This is different 14 (off mike), but actually I liked Moshe's idea 15 earlier this morning about publishing how you 16 structure your decision about or your 17 approach to, if I understood correctly, your 18 approach to making decisions about how to 19 release data, and I was thinking of it as 20 kind of like the cost-benefit structure that 21 you work through. And if you do something 22 like that, then I'd love to hear about it -- BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 163 1 like that; further structure (off mike). 2 DR. NEERCHAL: And the one place, 3 easy place for it to be published, there's a 4 section on Garman statistics of (off mike) 5 thing. You have the electronic kind of (off 6 mike) kind of, and they actually publish, you 7 know, at the bottom of that electronic news 8 listserv, you know, they put a -- you know 9 put some slightly technical information at 10 the end, and you (off mike) like a couple of 11 pages how you did it, that will be very 12 useful for a lot of people and (off mike). 13 DR. FORSYTH: The factors to 14 consider. 15 DR. NEERCHAL: I can give you the 16 person who is actually -- who sends out that 17 (off mike). 18 DR. FORSYTH: Okay. 19 DR. HILL: I mentioned an offhand 20 comment that it might be of interest to look 21 at the effect of the weather. I don't know 22 if that's -- we should think about it a BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 164 1 little bit more whether or not that's worth 2 pursuing. So maybe, maybe. 3 DR. NEERCHAL: I think it's my time 4 for inviting the public for comments. 5 DR. KOKKELENBERG: Could? I -- 6 DR. NEERCHAL: Oh, sorry. 7 DR. KOKKELENBERG: I guess I got 8 confused. I thought we were discussing the 9 30th anniversary and now you want topics? 10 DR. KIRKENDALL: Yeah. I guess for 11 future -- 12 DR. KOKKELENBERG: Okay. 13 DR. KIRKENDALL: -- yeah, it was 14 sort of both. I was -- 15 DR. KOKKELENBERG: Transmission 16 reliability data issues. I don't know if you 17 have data on transmission, but electricity 18 transmission and pipeline transmission 19 reliability issues I think are -- we had that 20 blackout. What two years ago? And there was 21 a huge study on why we had that blackout. 22 The main thing they pointed up was BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 165 1 that the reliability responsibility has been 2 smeared out all over the place and we talked 3 about it this time when we talked with -- we 4 just alluded to it, when we talked about the 5 electricity transmission problems as being a 6 constraint on what kind of plants come in and 7 out. 8 So it seems to me that perhaps 9 transmission, which is not a regulated 10 industry (off mike) as far as I understand 11 it, is something that maybe we should be 12 paying some attention to or the EIA should be 13 paying attention to. 14 Another topic I had is that when 15 you collect solar energy data here, all we 16 collect, it was my understanding, is the 17 shipment of solar panels. We don't have any 18 idea of how much electricity is generated by 19 solar devices. 20 DR. KIRKENDALL: We have some idea. 21 DR. KOKKELENBERG: Okay. I didn't 22 -- I wasn't aware of that. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 166 1 DR. KIRKENDALL: Yeah. But that 2 has to be from big solar establishments. We 3 certainly don't have data on the amount 4 generated by home solar panels. 5 DR. KOKKELENBERG: Or even 6 industrial. 7 DR. KIRKENDALL: Or -- well, I 8 don't know whether (off mike). 9 DR. KOKKELENBERG: Or commercial? 10 DR. KIRKENDALL: Yeah, I don't know 11 whether RECS, MECS, CBECS have any of that or 12 not. 13 DR. KOKKELENBERG: Yeah. Well, it 14 may be an (off mike) topic because -- 15 DR. KIRKENDALL: Okay. 16 DR. KOKKELENBERG: -- this is one 17 of the things that is really touted and the 18 same for wind generation, although I think we 19 have much more on wind generation. 20 Forced outage modeling. I was 21 surprised to hear Phillip say that the forced 22 outages were such a high percentage that he BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 167 1 had to reduced capacities essentially by as 2 much as seven or eight percent just from 3 forced outages. That works out to be about 4 three weeks out of the year for a plant, and 5 that's a lot of time to be down on a non- 6 scheduled I would assume he meant by forced 7 outages. 8 And I just want to underline what 9 Walter said. My work that I've done on a 10 variety of things suggests that things like 11 Katrina really didn't have a whole heck of a 12 lot of effect on the economy as a whole. It 13 certainly did on the South economy. It 14 certainly had an effect on the energy prices 15 for a while, but on as far as the economy as 16 a whole, it didn't have a whole heck of a lot 17 of impact. And so when you get to the NEMS 18 modeling system, it's got to take, you know, 19 maybe a global rise in temperature or a 20 global winter for a year before it's going to 21 have the impact that is felt in the GDP 22 estimates, and, hence, the energy (off mike). BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 168 1 And so the sensitivity to the weather of the 2 NEMS, it's high -- Neha and I were just 3 discussing that -- it's high when it comes 4 to, say, fuel prices and natural gas prices, 5 but it may not be high in the sense that it 6 feeds back through the GDP as much as 7 sometimes we would think it might. And so 8 the incomplete NEMS modeling sensitivity to 9 modest changes in weather might be revealing. 10 And you probably want (off mike). But would 11 that be useful to discuss that or not? 12 Where's Dan when you want him; right? 13 MS. BOEDECKER: We're off? We're 14 off? 15 COURT REPORTER: No, you're on. 16 MS. BOEDECKER: I'm on? Okay. We 17 do run scenarios where we do weather 18 sensitivities. In fact two AEOs ago, I think 19 there was a piece in the Issues and Focus 20 about it. Excuse me. 21 But in the residential and 22 commercial modules, when we have weather BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 169 1 affecting our fuel use, we have it only 2 affecting the heating and cooling fuel use 3 and not the other uses. And because the 4 heating and cooling are becoming a smaller 5 percentage of the energy use, you're not 6 going to have as great an effect as you would 7 for, say, a price change, which affects all 8 fuel uses. 9 DR. KOKKELENBERG: So maybe that 10 isn't' a topic that's worth looking at? 11 MS. BOEDECKER: It's been looked at 12 -- 13 DR. KOKKELENBERG: Yeah, that's why 14 I'm saying this. That's why -- 15 MS. BOEDECKER: But it's up to the 16 Committee I guess as far whether they want to 17 see something on it, but there is something 18 published on it already. 19 DR. NEERCHAL: Thanks. I mean it 20 is something -- if you think of something, 21 send an e- mail to Nancy or Bill. It's an 22 ongoing discussion. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 170 1 So now, it's my pleasure to invite 2 the public for their comments. Last chance. 3 I think -- and it doesn't have to be this 4 presentation. It could be the previous 5 presentation or any of the (off mike) day and 6 a half. Please identify yourself for the 7 transcription. 8 MS BATTLES: Okay. Stephanie 9 Battles. I'm the Division Director for the 10 Energy Consumption Division. We're the home 11 of the RECS and the CBECS. 12 I just have a few comments. As far 13 as the NISS it's quite intriguing, because I 14 do have a student right now from Stafford who 15 would love to get to historic data, because 16 we do not have very much on the Internet at 17 the moment. 18 What we are doing, though, is I do 19 have one staff. Right now, the staff is like 20 half the size of what it was 10 years ago, 21 but I do have one who is working on the 1990 22 RECS microdata, but this is public data, BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 171 1 putting it up on the Internet. 2 And my goal is to get every last 3 microdata set for the RECS and CBECS up on 4 the Internet, but it does take time and it 5 takes staff. 6 We find that the public data is 7 used a lot, a lot by researchers, especially 8 students and what have you. We take very few 9 variables off of the public data. I mean 10 like for the RECS we take state identifiers 11 off. The main reason we do that is because 12 the sample size is so small. CBECS we've had 13 to, because we have problems with large 14 buildings, we've had to top code the number 15 of stories and the floor space because big 16 buildings you can point them out. You know 17 everybody knows the Empire State Building as 18 an example. I'm not saying it's in our 19 sample. But that's just an example there. 20 And so that is -- to me, it's low 21 hanging fruit to get as much data out as 22 possible. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 172 1 Another thing that we've done we're 2 going to be getting it up on the Net in I 3 think within a month -- I've had every last 4 report that we've ever done scanned into PDFs 5 and so this is all, but this is aggregated 6 data. All the tables going back like the 7 RECs to '78 and any other reports, you know, 8 that we've done, and we're getting this up on 9 the Internet. 10 So we've done things that we can; 11 we can afford to do. Yes, we do want to get 12 the researchers to get the unmasked data. We 13 get very few requests for it, but they are 14 available. But a lot of times they're 15 requests because they want one or two 16 variables. You know, they want -- they want 17 that -- we had to take some large buildings 18 off in the CBECS. They want those large 19 buildings because of sample size and things 20 of that nature; like the number of stories we 21 had to do. So they do want that data for the 22 monthly building data that we get, but I say BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 173 1 it does take a lot of time and money to do 2 that. 3 So, anyway, you know, we're just 4 trying to do what we can do with the 5 resources we have at the moment. And just as 6 an aside, we know -- we do the MECS also. 7 And we've never had the opportunity to go to 8 the (off mike) of the RDC at the Census, 9 because basically we can't afford it, so 10 we've never -- we don't have access to our 11 historic MECS data basically unless we have 12 the funds. 13 Thank you very much for the 14 opportunity. 15 DR. NEERCHAL: Thank you. I mean 16 just include it in public if you want to make 17 a comment. 18 MS. BOEDECKER: This is Erin 19 Boedecker again. Commercial module and NEMS. 20 I did want to make one slight 21 clarification. We do archive the model every 22 year, the Annual Energy Outlook version of BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 174 1 the model. 2 But it is an evolving model when 3 over time there are changes made and we just 4 haven't -- don't have the resources for one 5 thing to go back and make changes to input 6 what the actuals were instead of what we 7 assumed at the time I think, because some 8 models are more modular than others, and in 9 some cases, you're changing the actual code 10 instead of just inputs. So I believe it 11 would be very resource intensive, even if you 12 just chose a limited number of variables. 13 But we do have archives of the AEO version of 14 the NEMS module every year. 15 DR. KOKKELENBERG: Can I say -- I 16 don't think you should be changing it. I 17 think if you made a forecast based on the 18 best knowledge you had at that time, the best 19 specification you had at that time, then 20 that's the forecast. Revisiting that and 21 saying, well, really temperatures were higher 22 or a (off mike) changed, you really don't BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 175 1 serve much of a useful purpose unless you're 2 trying to identify why there was an error in 3 the model. And that should be left to the 4 individual investigator rather than 5 rearchiving the model with those changes. 6 MS. BOEDECKER: Okay. Thank you. 7 And to some extent, we do do that, as John 8 Cymbalsky mentioned earlier. 9 DR. FEDER: But, Ed, if you want to 10 improve the model, you want to know where the 11 error is. 12 DR. KOKKELENBERG: Oh, yes. Yes, 13 yes. That's right. I really forecast that, 14 but this is what I really (off mike) forecast 15 that you're kind of (off mike). 16 DR. NEERCHAL: I don't know. Do 17 you have any (off mike) on it? 18 With that, I think we are ready for 19 adjourning to -- adjourning this meeting and 20 go to lunch I guess. So lunch is like 21 yesterday, downstairs the same room. 22 DR. WEINIG: Yes. BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382 176 1 DR. FEDER: Could you remind us 2 which room? Which floor? 3 DR. NEERCHAL: 2E-081. 081. It's 4 the same corridor, second floor. 5 (Whereupon, at 11:45 a.m., the 6 PROCEEDINGS were adjourned.) 7 * * * * * 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com (202) 464-2400 800-522-2382