May 19, 1993 MEMORANDUM SUBJECT: Technical Comparison Document--Phelps Dodge Smelter FROM: Joseph A. Tikvart, Chief Source Receptor Analysis Branch, TSD (MD-14) TO: Gerald Fontenot, Chief Air Programs Branch, Region VI (6A-AP) As follow up to telephone conversations with your staff and in response to the request from Shawn Kendall of Phelps Dodge, the Model Clearinghouse has reviewed the subject document. Although we have a few comments, which are provided below, in general we believe that the plan for model evaluation in the document is sound and the Company should plan to go ahead with the data collection as planned. Resolution of the relatively minor issues we have identified can take on whatever schedule you want to set up with the source over the next several months. Comments on the Technical Comparison Document (TCD) 1. Comments from Bill Cox are attached. His first comment refers to the role of a reference model. The normal protocol for determining the most appropriate model for a given application through a performance evaluation is to compare performance of the proposed model and a reference model. If the proposed model performs better than the reference model, based on criteria set up in the scoring scheme, then it may be used for the application. If the proposed model does not perform better, then the reference model should be used. However, the Interim Procedures for Evaluating Air Quality Models (Revised) does allow for the use of other technical criteria to make a decision in the case of comparable performance. But the basis for making such a judgment is usually highly subjective and could be argued either way in most cases. Even though we believe that the reference model should generally be the default, unless the proposed model clearly performs better, an exception to this philosophy may be appropriate in the Phelps Dodge case. We tend to agree that the MPDM model may be used in the event of a performance evaluation tie. This is the case because MPDM appears to be a technically better model for this application since it handles transport in complex terrain in a more rigorous manner. The real key to its technical superiority is, however, the fact that the sophisticated data bases necessary to run the model should be available from the field study. Bill's other two comments are self explanatory and should be addressed in revisions to the document. 2. An important part of the criteria that we have insisted be included in model evaluation protocols of the type being discussed here is to include provisions to guard against underprediction of the design concentration(s). Phelps Dodge should include such a provision in their application scheme for MPDM should that model be determined to be appropriate, yet still underpredict the critical concentrations. Their consultant, ENSR, is familiar with underprediction adjustment factors used in other situations, and should be able to develop an acceptable scheme. 3. As a general observation, we agree with the monitor siting and with the performance evaluation criteria in the TCD, and we do not have a better scheme to suggest. However, we really wonder whether the state of the science is commensurate with the degree of sophistication used in developing the scoring schemes. In other words, while we agree that it is important to consider such things as concentration patterns, weighting of monitor readings, and cross-cutting between averaging times, we question whether having a highly sophisticated rigorous procedure for accomplishing these considerations leads to an unwarranted sense of certainty about the believability of the results. Perhaps a simpler scheme, based more on subjectivity, would better convey to the public the true state of the science in developing a scoring scheme. While these are our thoughts, we are not insisting or even recommending that any changes be made in the scheme contained in the TCD. In summary, we are in general agreement with the material contained in the Phelps Dodge TCD. The Company should address our comments in a revision to the document. We are particularly interested in reviewing the underprediction scheme before the Environmental Protection Agency agrees to the protocol. If you have any questions please contact Dean Wilson at 919-541-5683. Attachment (1 p.) cc: G. Blais, MD-15 W. Cox, MD-14 T. Diggs, Region VI J. Yarbrough, Region VI To: Dean Wilson April 20, 1993 Subject: Comments on the ENSR-Phelps Dodge Project From: Bill Cox The criteria for model selection compares each competing model with "the most technically sound model" which they have determined to be MPDM (page 11-1). They propose to use MPDM unless the MCM calculation of at least some other model is statistically superior to MPDM. Question: I thought that comparisons had to be made against a reference model--a model I assumed was other than the model being proposed. Have I missed something? ENSR apparently follows the EPA approach for model comparison fairly closely with the exception of (1) weighting assigned to monitoring stations and (2) inclusuion of a 3-hour fractional correlation statistic. I personally am luke warm to the 3-hour correlation statistic (see page 10-8) and do not see that it has any particular value. If they wish to include it, I am not sure we should necessarily object; however, I do believe that they could present a more cogent argument for it. They should include the Atmospheric Environment reference in addition to the EPA reference for comparing models. It is: Cox, William M. and Tikvart, J.A. (1990). A Statistical Procedure for Determining the Best Performing Air Quality Simulation Model. Atmospheric Environment, 24A.