Long-Range Forecast Skill Maps

How to interpret the skill map for the Environment Canada temperature and precipitation anomaly forecast ?

The temperature and precipitation anomaly forecasts for each month and season during the period indicated on the forecast charts were compared with the observed anomalies (based on 3 categories: ABOVE, BELOW and NEAR NORMAL). The skill maps show the percent correct. This score is calculated using a 3 by 3 contingency table. The higher the percentage, the better the forecast over the verification period. A purely random "chance" forecast would be, on the average, 33 percent correct. However, the data used to calculate the percent correct cover only 30 years . Thus, the threshold to be statistically significant is close to 40% (with 10% confidence level) and not 33%. With a longer dataset of, say, 10000 years the threshold would have been 33%. The fact that the threshold is 40% ensures that a forecast system is not better than 33% only by pure chance (true 18 times out of 20, 90% of the time). This means that a percent correct of 40% or less indicates that the forecast does not show skill. These areas are identified by the grey color on the skill maps. The percent correct can then be use to evaluate the confidence in the forecast for the current season (month).

In general, seasonal and monthly forecasts for countries located in the mid-latitudes (e.g. Canada) show low skill. However, since the skill of the forecasts varies with the season (month) and the geographical location, some useful information can still be obtained for many locations in Canada. It has to be note that when a large ENSO phenomena occur (like the El Nino of 1997-1998) the confidence in the forecast is much higher in Canadian areas teleconnected with the tropical Pacific oceans (see El Niño and La Niña web pages for details). For example during the El nino of 1997-1998, Environment Canada forecasted above normal temperature for nearly all of Canada and the percent correct of this forecast was 88%. Due to a number of factors the surface air temperature forecasts are generally much better than the precipitation forecast (Plante and Gagnon, 2000; Servranckx et al., 2000).

How to use the percent correct maps ?

Locate the area of interest on the map and check if the percent correct there is equal to or greater than 40%. If it is, ex. 60% , this means that in 60% of the cases during the period indicated on the chart, the correct category (ABOVE, BELOW or NEAR NORMAL) was forecasted for the season (month) considered. If the value is lower than 40% (grey areas), the model is not statistically better than pure chance and hence the confidence on the forecast is very low.

As of December 2011 seasonal forecasts at time ranges up to a year ahead are produced in a uniform manner from two comprehensive coupled atmosphere-ocean-land physical climate models. (This contrasts with the two different methods used prior to December 2011, numerical models up to 2-4 month range and a statistical method for longer range forecasts.)The skill of the forecasts generally decreases with increasing lead time, for example the 1-3 month forecasts are more skillful than the 2-4 month forecasts, which are more skillful than the 4-6 month forecasts, etc.

References

Kharin, V. V., Q. Teng, F. W. Zwiers, G. J. Boer, J. Derome, J. S. Fontecilla, 2009: Skill assessment of seasonal hindcasts from the Canadian Historical Forecast Project. Atmos. Ocean., 47, 204-223.

Plante A. and N. Gagnon, 2000: Numerical Approach to Seasonal Forecasting. In the "Proceedings of the sixth workshop on operatiOnal meteorology", Halifax, November 1999, 162-165. (PDF Version) (Requires Acrobat Reader to view)

Servranckx, R., N. Gagnon, L. Lefaivre and A. Plante, 2000: Environment Canada Seasonal Forecasts: Products, Methods, Procedures and Verification. In the "Proceedings of the sixth workshop on operational meteorology", Halifax, November 1999, 172-176. (PDF Version) (Requires Acrobat Reader to view)