![]() | ||
CPT Help Home -> How to use CPT -> Viewing the Results -> Model Validation Measures Model Validation MeasuresForecast performance scores and graphics can be obtained for the cross-validated forecasts, and if the retroactive forecast option was selected then results for these forecasts is also available. Using the Tools ~ Validation menu item, select whether it is the cross-validated or the retroactive forecasts that are to be verified, and then whether performance statistics, bootstrap confidence interval and permutation significance tests, or scatter and residual plots should be provided for an individual series, or a map/bar chart (depending on whether the Y data are in gridded/station format) for all series. A validation window will open.The performance window for an individual series provides a variety of forecast performance scores divided into those based on continuous measures, and those based on measures in which the observations, and in some cases the forecasts as well, are divided into three categories. The continuous forecast measures calculated are:
Beneath the categorical measures are relative operating characteristic (ROC) graphs for the above- (red line) and below-normal (blue line) categories. The observations are categorised using cross-validated category definitions (see Contingency Tables for further details on the definitions of the categories), but the forecasts are considered on the continuous scale. The forecasts are ranked, and the forecast with the highest value is taken as the most confident forecast for above-normal conditions, and that with the lowest value is taken as the least confident forecast. For forecasts of below-normal conditions, this ranking is inverted so that the forecast with the highest value is taken as the most confident forecast for below-normal conditions, and that with the highest value is taken as the least confident forecast. The areas beneath the graphs are given under the categorical skill measures above the ROC graph. Scores and graphs are shown for one series at a time. Information for the desired series can be shown by setting the appropriate number at the top left of the validation window. A series that has been omitted in the calculations is skipped when cycling through the series using the arrows. The Bootstrap window provides confidence limits and significance tests for a variety of forecast performance scores. The confidence limits are calculated using bootstrap resampling, and provide an indication of the sampling errors in each performance measure. The bootstrap confidence level used is indicated, and can be adjusted using the Options ~ Resampling Settings menu item. The actual sample scores are indicated, and are the same as those provided by Performance Measures. As well as providing confidence limits, significance levels are also provided. The p-value indicates the probability that the sample score would be bettered by chance. Permutation procedures are used to calculate the p-values. The accuracy of the p-values depends upon the number of permutations, which can be set using the Options ~ Resampling Settings menu item. It is recommended that at least 200 permutations be used, and more if computation time permits. If the Skill Map option is chosen, a window showing a map (if the Y data are gridded/stations) or a bar chart (otherwise) for all series will be shown. It is possible to choose which score to use for the map, simply by checking the button next to the dsired score. The actual scores can be saved to files, and the graphics as JPEG files by right-clicking anywhere in the child window, and then selecting the desired output to be saved. A prompt for the name of the file in which to save the scores is given. For the JPEG file, a default name is given, but this name can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. The titles of all graphs can be customised by right-clicking on the graph, and selecting Customise for theappropriate graph. The Scatter Plots option shows a graph of the forecast residuals (differences between the forecasts and the observations), as well as a scatter plot of the observations against the forecasts. The scatter plot includes horizontal and veritical divisions that indicate the three categories. In both cases the divisions are defined by the terciles of the observations using all the cases. A best fit linear regression line is shown on the scatter plot, but only over the range of the forecasts.
|