Introduction

The two main types of statistical guidance available to forecasters in preparing maximum/minimum temperature forecasts and probability of precipitation (PoP) forecasts are AVN Model Output Statistics (MOS) and NGM MOS, hereafter referred to as AVN and NGM. The local AFOS-Era Verification software package (Meier and Barker, 1993) compares the forecast performance of the NGM and the local forecaster but does not include comparisons with the AVN.

A study was conducted to compare the forecast performance of the NGM to that of the AVN for three sites in Montana that are located in different climatic regions. The sites chosen were Missoula (MSO), which lies in a valley west of the Continental Divide; Great Falls (GTF), which lies along the east slopes of the Rocky Mountains; and Glasgow (GGW) which is in the eastern plains of Montana. AVN and NGM forecast maximum/minimum temperatures and PoPs for forecast periods 1 through 4, including the 00Z and 12Z forecast cycles for September 1995 through August 1996, were compiled for each location. PoPs were rounded to the nearest 10 percent. For temperature comparisons, Fall covered the period September through November; Winter, the period December through February; Spring, the months of March through May; and Summer, the period June through August. With precipitation comparisons, the cool season covered the months of October through March, while the warm season covered the months of April through September.

It should be noted that both temperatures and precipitation averaged below normal for the period of study at both GGW and GTF, while at MSO, precipitation averaged above normal and temperature averaged a little above normal. Mean absolute error (MAE) (Meier, 1993) was used to compare temperature forecasts while the Brier score (Brier, 1950) was used to compare PoP forecasts. The Brier score involves squaring the difference between the forecast PoP and the observed PoP (0% for no precipitation and 100% for precipitation). Scores range from 0 (best) to 100 (worst). Biases of the AVN and NGM were examined for both temperaures and PoPs. Reliability curves (Meier, 1993) were used to examine PoP biases. A reliability curve displays the percentage of times measurable precipitation occurred for each forecast PoP along with an idealized line representing zero bias. Points above the idealized line indicate underforecasting while points below this line show overforecasting. .

Temperatures

Figure 1 compares MAEs for forecast periods 1 through 4 for the three sites. Overall, the NGM had a smaller MAE than the AVN at GGW and MSO, while the reverse was true at GTF. These trends also applied to each of the four periods with the exception of the third period at GTF, where the MAEs of both the AVN and NGM were the same.

Figure 2 compares MAEs by season for the three sites. At both GGW and MSO, the MAE of the NGM was smaller than that of the AVN for each season, but the MAEs were the same for Summer at MSO. At GTF, the MAE of the AVN was smaller than that of NGM for each season with the exception of Spring.

Figure 3 compares MAEs of the three sites for forecast maximum (periods 1 and 3 of the 00Z cycle and periods 2 and 4 of the 12Z cycle) and forecast minimum (periods 2 and 4 of the 00Z cycle and periods 1 and 3 of the 12Z cycle) temperatures, as well as a composite of all periods. At both GGW and MSO, the composite MAE of the NGM was smaller than the AVN for both maximum and minimum temperature forecasts. This result also applied to each forecast period with the exception of period 3 for forecast minimum temperatures at GGW and period 2 for forecast maximum temperatures at MSO. For GTF, results were mixed for individual periods for both maximum and minimum temperatures forecasts, but overall, the AVN had a slightly lower MAE than the NGM for both maximum and minimum temperature forecasts.

Figure 4 compares MAEs for large temperature changes by season for the three sites. For this study a large temperature change was defined as an observed temperature change of 10øF or more for consecutive maximum or minimum temperature periods (Beasley, 1995). At GGW, the AVN had a slightly smaller MAE than the NGM for the year. Comparing individual seasons, the AVN had a smaller MAE than the NGM for Fall and Winter while the reverse was true for Spring and Summer. Similar results were observed at GTF, with the exception that the MAE for the AVN was slightly smaller than that of the NGM for Summer. At MSO, the NGM had a slightly smaller MAE than the AVN for the period of study. Whereas at both GGW and GTF, the MAE of the AVN for Winter was about one-half degree less than that of the NGM, the reverse was true at MSO. For Spring and Summer the NGM had a slightly smaller MAE than the AVN, while the reverse was true for Fall.

At GGW and GTF, both the AVN and NGM had a warm bias, but the NGM was not as warm as the AVN. This result applied to Winter, Spring, and Summer except the NGM had a warmer bias than the AVN for the Summer at GTF. At MSO the AVN had a warm bias while the NGM had a cold bias. For Spring and Summer both the AVN and NGM had warm biases but the NGM was not nearly as warmly biased as the AVN. For the Fall the AVN had a stronger cold bias than the NGM. For Winter, the NGM again had a cold bias while the AVN had a warm bias.

PoPs

Figure 5 shows a comparison of Brier scores for forecast periods 1 through 4 for the three cities. For the year, the AVN had a lower Brier score than the NGM at both GGW and MSO. This result held for each period at both stations with the exception of the first period at GGW. Conversely, at GTF, the NGM had a lower Brier score than the AVN for the year and also for each period.

Figure 6 shows Brier scores for the warm and cool seasons for the three sites. For both seasons at GGW, the AVN had a lower Brier score than the NGM, while the reverse was true at GTF. At MSO, the AVN had a lower Brier score than the NGM for the cool season while the NGM had a slightly lower Brier score than the AVN for the warm season.

Figure 7 illustrates average PoPs for each site for the warm and cool seasons for each of the following two situations: 1 - cases when there was no measurable precipitation; 2 - cases when there was measurable precipitation. At each of the three locations for the cool season, the AVN had a higher average PoP than the NGM for cases when there was measurable precipitation and when there was no measurable precipitation. For the warm season the opposite was true, with the exception that the NGM had a slightly lower average PoP than the AVN at MSO for cases when there was no precipitation.

Figures 8, 9, and 10 show reliability curves for each site for the cool season, warm season, and composites. At GGW, the NGM had better reliability than the AVN for the cool season while the reverse was true for the warm season. Overall, it appeared the AVN had the better reliability. At GTF, the AVN had better reliability than the NGM for the cool season while the reverse was true for the warm season. But similar to the results for GGW, it appeared the AVN overall had better reliability than the NGM. At MSO, the AVN had better reliability than the NGM for the cool season while the NGM appeared to have slightly better reliability than the AVN for the warm season. Here, too, the AVN overall had the better reliability. At GTF and MSO, the NGM tended to underforecast precipitation, particularly during the cool season.

Particular Temperature Forecast Problems at GGW and GTF

At GGW, one problem is forecasting temperatures in an arctic airmass. During the period from January 17, 1995 to February 5, 1995, temperatures at GGW were generally below zero. Averaging all four forecast periods, the MAE of the AVN was 8.0øF while for the NGM it was 10.8øF. This is in contrast to the result for the entire period of study that the NGM had a smaller MAE than the AVN. Averaging all four periods, the mean error for AVN was 4.4øF too warm while for the NGM it was 9.6øF too warm. At GTF, a forecast problem is predicting maximum temperatures when an arctic front moves into and also out of the area. Five such cases were observed during the period November 1995 to January 1996. In each case, consecutive maximum temperatures dropped over 30øF during the arctic airmass intrusion and climbed over 30øF when the arctic airmass retreated. Averaging all four periods, the MAE for arctic air intrusions was 14.1 øF for the AVN and 18.6øF for the NGM. In each case, the AVN had a lower MAE than the NGM. For cases where the arctic air retreated, the MAE for the AVN was 6.2øF while for the NGM it was 9.6øF. In four of the five cases, the AVN had a lower MAE than the NGM. The corresponding mean errors were 12.2øF too warm and 18.2øF too warm for arctic air intrusions, and 5øF too cold and 9.6øF too cold for cases where the arctic air retreated.

Summary

Overall at both GGW and MSO, NGM temperature forecasts had a smaller MAE than AVN temperature forecasts, while the reverse was true at GTF. This generally held true for each forecast period and for each season. In contrast to the results for temperature forecasts at GGW and MSO, the AVN PoP forecasts had a lower overall average Brier score than the NGM, while the reverse was true at GTF. With regard to temperature biases, both the AVN and the NGM had a warm bias at all three sites with the exception that the NGM had a cold bias at MSO. Averaging all three stations, the AVN was more warmly biased than the NGM. Results from this study indicate that the AVN was better than the NGM in forecasting temperatures associated with arctic airmasses.

For precipitation biases, the AVN had a little better reliability than the NGM at all three sites. During the cold season, the AVN had a higher average PoP than the NGM both for cases with and without precipitation. Opposite results were observed during the warm season at all three sites with the exception of non-precipitation cases at MSO.

Due to the limited nature of this study ,it is important not to generalize any of these results to locations in similar climatic regimes.

Acknowledgment

The author wishes to thank Dave Bernhardt, Science and Operations Officer at NWSFO GTF, and Ken Mielke, Meteorologist-in-Charge of NWSFO GTF, for their review of the paper and also Donald Britton of NWSFO GTF for his assistance with the figures.

References

Beasley, R.A., 1995: "AFOS-Era Forecast Verification". NOAA Techniques Development Laboratory Computer Program NWS TDL CP No.95-2, August 1995.

Brier, G.W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev.,78, 1-3.

Dallavalle, P., 1997: personal communication.

Meier, K.W., and Barker, T.W., 1993: "AEV Local Verification for Aviation, Precipitation, and Temperature Programs: AV, REL, TEM". NOAA Western Region Computer Programs NWS WRCP No.42, September 1993.

National Verification Plan, 1982: National Oceanic and Atmospheric Administration, U.S. Department of Commerce.