December 23, 1991 MEMORANDUM SUBJECT: East Helena Lead SIP - Protocols for Design Value Determination, and Model "Verification" FROM: Tom Coulter, Environmental Scientist Techniques Evaluation Section, SRAB (MD-14) Dean Wilson, Meteorologist Techniques Evaluation Section, SRAB (MD-14) TO: Mindy Mohr, Lead SIP Coordinator, Region VIII (8AT-AP) This is in response to your memorandum of December 5, 1991. The Model Clearinghouse has completed its review of the subject documents and has included its comments in the two respective attachments. If you have any questions or need additional information, please contact Tom Coulter at FTS 629-0832. Attachments cc: T. Coulter L. Ostrand D. Wilson ATTACHMENT 1 Comments on Inter-model Comparison Protocol (Mindy Mohr 12/5/91 memo to Coulter et al.) We have thoroughly reviewed the "Protocol for Comparison of Additional Results from Reconciled and CMB Models at East Helena, Montana" (hereinafter, the "12/2/91 Protocol") and have several concerns. We are not convinced of the utility of "comparing the ISCST model results to CMB results over an equivalent study period ..." (p. 1-2). This process was rigorously performed for the third and fourth quarters of 1990 in the initial study (CPP, 1991) and it "was deemed to have been successful relative to the targets set by the study." The process used in the initial study was much more sophisticated and comprehensive than anything described in EPA's Protocol for Reconciling Differences among Receptor and Dispersion Models (March 1987; EPA-450/4-87-008). In the 12/2/91 Protocol, we are reminded that "... 21 of the 22 comparable source categories met the criterion for acceptable model agreement at each of the monitoring sites" (p. 1-1). It must be noted, however, that EPA had no such criterion in its reconciliation protocol, nor is such reconciliation absolutely necessary for dispersion model acceptability in SIP development. Thus, while the innovative design of the initial reconciliation process was commendable, it was incommensurate with any strict EPA guidance or policy. A substantial portion of the narrative of Section 2.2 of the 12/2/91 Protocol is given to statistically justifying the use of a wider overlap criterion for daily inter-model comparisons as a result of a smaller sample size. While this approach has some statistical basis, it must be noted that the comparison interval described in the reconciliation protocol was not statistically derived, nor was the choice of 25 days in the initial study statistically sacred. It follows that the coefficient 1.58 is likewise not statistically sacred, so the effort to expand the comparison interval is somewhat academic. The pseudoscience involved in NEA's analysis notwithstanding, a potential problem arises from the fact that "the comparison will not be an iterative process in the same sense as the previous inter-model reconciliation process" (p. 1-2). If only "data entry errors will be changed in the model inputs", what will happen if the new criterion is not met? Do they then go back to first base? It seems like eventually one needs to declare a dispersion model "reconciled" and move forward with the business of SIP work. The bottom line is that the intention to perform a recomparison between the models is not founded in EPA guidance and is superfluous. The Dispersion Model Performance Verification (Section 2.3) would seem to be the most important objective at this point in the modeling analysis. While the verification effort is well described in this section, much of the narrative parallels that in the previous section, i.e., expansion of the comparison criterion to compensate for a smaller sample size (n = 25 vs. 10). And much of our response would echo what we offered in the previous paragraph; i.e., the moral of the story would be essentially the same. While their effort to establish such a stringent verification goal is commendable, there was never anything sacred in NEA's criterion for the top 25 measured daily lead values to compare "within a factor of two, for both the mean and standard deviation ..." (p. 2-6). That is, this criterion is not rooted in any definitive EPA guidance or policy. As such, the expanded factor of 3.16 (p. 2-9), while having some statistical basis, is arbitrary. The establishment of such a criterion begs the question of what will happen in the event that the criterion is not met since it was stated that the "ISCST dispersion model will use the same hourly emission factors, percent lead compositions, and AP-42 emission equations as in the previous study" and that "[t]he model algorithms themselves will also be identical to those used previously" (p. 2-1). Again, the overall impression is that the level of sophistication in the verification procedure has been taken to a level not specified in any EPA guidance or policy. Perhaps it would be more meaningful for NEA to simply run the reconciled dispersion model for the selected days and document the performance. These concerns notwithstanding, the procedures described in the 12/2/91 Protocol appear acceptable, though perhaps superfluous. ATTACHMENT 2 Comments on Design Value Protocol (Mindy Mohr 12/5/91 memo to Coulter et al.) 1. Page 1; Part A. We realize that the March 11, 1991 protocol indicates that design values and demonstration modeling is to be done using July 1990 to July 1991 meteorological data. However, since the Guideline recommends the use of all available on-site data, credibility would be enhanced if all quarters since January 1990 for which meteorological data are available would be included. 2. Page 2; first bullet under Part B. We don't believe that the method used to derive the emission input to ISCST is consistent with Table 9.1 of the Guideline. Table 9.1 assumes that one starts with a short term emission limit, or maximum emission at design capacity, which is multiplied by an operating factor which represents the average historical percentage of the time that the unit was operating. If we understand ASARCO's proposal they want to start with a maximum quarterly emission and divide by 91 to obtain an average daily emission rate, which is then input to the model. We believe that number will be different from the Table 9.1 derived value. We would be willing to be convinced that the ASARCO method is as conservative as the Guideline method. 3. Page 3; second bullet. The pattern of emissions variation is more complicated than we normally have encountered in this type of analysis. Emission inventory experts should be consulted to see if this is a reasonable request. In any event, ASARCO would need to take a permit condition limiting them to operating in the manner they have described unless it can be shown that 24-hour operation of these sources would not affect the design value(s), or physical limitations prevent operating the sources more than 8 hours per day. 4. Page 3; third bullet. It is unclear what the model emission input implications are for "2/3 uncontrolled, 1/3 controlled." Unless the October "controlled" emissions can be treated explicitly in the model for that month, we recommend that uncontrolled emissions be input for the entire quarter. 5. Page 4; first bullet under Part C. See Comment 2 above regarding the average quarterly emission rate.