The validation window provides a variety of forecast performance scores together with a graph of forecast (green line) and observed (red line) values, and ROC graphs for the above- (red line) and below-normal (blue line) categories. The categorical measures and the ROC graphs are based on three categories (see Contingency Tables for further details on the definitions of the categories). Scores and graphs are shown for one series at a time. Verification for the desired series can be shown by setting the appropriate number at the top left of the validation window. A series that has been omitted in the calculations will be skipped.

The graphics can be saved as JPEG files by right clicking anywhere in the child window, and then selecting the desired graphic to be saved. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. The graphics titles can be set using the Customise option upon right-clicking in the child window.

Further details are available under Viewing results.

The bootstrap window provides confidence limits and significance tests for a variety of forecast performance scores. The confidence limits are calculated using bootstrap resampling, and provide an indication of the sampling errors in each performance measure. The bootstrap confidence level used is indicated, and can be adjusted using the Options ~ Resampling Settings menu item. The actual sample scores are indicated, and are the same as those provided by Performance Measures.

As well as providing confidence limits, significance levels are also provided. The p-value indicates the probability that the sample score would be bettered by chance. Permutation procedures are used to calculate the p-values. The accuracy of the p-values depends upon the number of permutations, which can be set using the Options ~ Resampling Settings menu item. It is recommended that at least 200 permutations be used, and more if computation time permits.

The validation window provides a map (if the Y data are in gridded or station format) or a bar chart (otherwise) of a specific performance measure for all series at once. Graphics for the desired score can be shown by selecting the appropriate button.

The graphic can be saved as a JPEG file by right clicking anywhere in the child window. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. The graphic title can be set using the Customise option upon right-clicking in the child window.

Further details are available under Viewing results.

The Scatter Plots option shows a graph of the forecast residuals (differences between the forecasts and the observations), as well as a scatter plot of the observations against the forecasts. The scatter plot includes horizontal and veritical divisions that indicate the three categories. In both cases the divisions are defined by the terciles of the observations using all the cases. A least-squares linear regression line is shown on the scatter plot, but only over the range of the forecasts.

The graphics can be saved as JPEG files by right clicking anywhere in the child window. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. The residuals and scatter plots titles can be set using the Customise option upon right-clicking in the child window.

Further details are available under Viewing results.

text here

The attributes diagrams are constructed using probabilistic retroactive forecasts. The reliability statistics are calculated using predictions for all gridpoints / stations / series. The probabilistic forecasts are binned to the nearest 10% (in accordance with the WMO's Standardized Verification System for Long-Range Forecasts - SVSLRF), but the point on the x-axis is shown at the average of the forecasts in the bin rather than at the centre of the bin. Results are shown separately for the three categories, and for all the categories pooled together (top left).

The reliability curve is shown by the thick line, while the solid 45 degree diagonal shows the line of perfect reliability. The dashed diagonal is the so-called no-skill line (a reliability curve that is shallower than this will score a negative Brier skill score against climatology). The horizontal line represents the line of no resolution and crosses the y-axis at the observed relative frequency for all years. Note that his value is not necessarily the same as the climatological probability of an event since the observed relative frequency is defined only over th retroactive period.

The reliability statistics can be saved by right-clicking on the graphs. In a similar manner the graphs themselves can be saved and the titles customised.

The ROC graphs are constructed using probabilistic retroactive forecasts. The ROC statistics are calculated using predictions for all gridpoints / stations / series. The probabilistic forecasts are binned to the nearest 10% (in accordance with the WMO's Standardized Verification System for Long-Range Forecasts - SVSLRF). Results are shown separately for the three categories, and the areas beneath the graphs, calculated using the trapezium rule, are shown.

The ROC statistics can be saved by right-clicking on the graphs. In a similar manner the graphs themselves can be saved and the titles customised.

Weather Roulette is a suite of verification procedures aimed at providing an indication of the potential value of forecasts. The underlying assumption is that a user of the forecasts invests an initial sum of money on the first forecast, apportioning the investment across the three possible outcomes according to the forecast probabilities. The investment pays out fair odds against climatology. For example, if the three categories are climatologically equiprobably the investment will return 3 times (1 / 33%) the investment on the verifying outcome. Whatever returns are made after the first forecast are available for the second forecast. If there are many locations, results are averaged across them all. The change in the investors fortunes can be viewed in one of two ways, as a cumulative profits or effective interest rate diagram.

The cross-validated contingency table window provides frequency and contingency tables for the cross-validated forecasts. The frequency table gives the counts of forecasts for each of three categories, marked below-normal (B), normal (N), and above-normal (A), and the number of times each of the three catgories verified. The contingency table indicates the percentages of times that each of the three categories verified given the forecast category, and can be obtained from the frequency tables by simply dividing each element in the table by the respective column total. The column and row totals indicate the percentages of times that each of the three categories were forecast and observed, respectively.

Further details are available under Viewing results.

The retroactive contingency table window provides frequency and contingency tables for the retroactive. The frequency table gives the counts of forecasts for each of three categories, marked below-normal (B), normal (N), and above-normal (A), and the number of times each of the three catgories verified. The contingency table indicates the percentages of times that each of the three categories verified given the forecast category, and can be obtained from the frequency tables by simply dividing each element in the table by the respective column total. The column and row totals indicate the percentages of times that each of the three categories were forecast and observed, respectively.

Further details are available under Viewing results.

The scree plots show the percentage of variance explained by each EOF mode. The percentages are shown for all modes regardless of the numbers that were specified for use. The plots can be useful in selecting the appropriate number(s) of EOF modes to retain. One method for selecting the number of modes involves identifying "elbows" in the scree plot. The number of modes up to, but not including, the elbow are retained on the basis that the additional modes explain similar amounts of variance, which are assumed to be largely noise. Elbows are often easier to identify when the percentage variance is plotted on a logarithmic axis. An option to use logarithmic axes is available by right-clicking on the scree plots. Another simple method for selecting the number of modes is to retain a sufficient number to ensure that the total variance retained exceeds a predefined minimum. The cumulative percentage of variance explained by the EOF modes can be shown using the Cumulative option by right-clicking on the scree plots. Note that the cumulative percentages are shown only if the logarithmic axis option is switched off.

Another method for selecting the number of models is to compare the explained variance of each mode to that expected from random data. A computationally and conceptually simple such approach is based on the random stick theorem, which involves randomly dividing the total variance into segments and considering the expected size of the nth largest segment. The sizes of the random stick segments can be included on the scree plot by an option available by right-clicking on the plot. The random stick is shown only if the cumulative option is switched off.

The graphics can be saved as JPEG files by right clicking anywhere in the child window, and then selecting the desired graphic to be saved. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. The scree plot titles can be set using the Customise option upon right-clicking in the child window.

If gridded or station data are used, a map of the spatial loadings for each mode is shown, together with the temporal scores. If the data are unreferenced, a bar chart of the loadings is shown instead. The loadings are shown in the form of correlations between the original gridded/station/index data and the temporal scores. If the loadings are saved to an output file, the actual loadings are saved rather than the correlations shown in the map. The graphics for each mode can be cycled by using the arrows at the top left of the graphics window, or by typing in the requisite number. Depending on your computer speed, you may need to wait a short time for the new graphics to display. The graphics can be saved as JPEG files by right clicking anywhere in the child window, and then selecting the desired graphic to be saved. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. The map and graph titles can be set using the Customise option upon right-clicking in the child window.

If gridded or station data are used, a map of the spatial loadings for each mode is shown, together with the temporal scores. If the data are unreferenced, a bar chart of the loadings is shown instead. The loadings are shown in the form of correlations between the original gridded/station/index data and the temporal scores. If the loadings are saved to an output file, the actual loadings are saved rather than the correlations shown in the map. The graphics for each mode can be cycled by using the arrows at the top left of the graphics window, or by typing in the requisite number. Depending on your computer speed, you may need to wait a short time for the new graphics to display. The graphics can be saved as JPEG files by right clicking anywhere in the child window, and then selecting the desired graphic to be saved. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. The map and graph titles can be set using the Customise option upon right-clicking in the child window.

CCA Maps The CCA maps (or bar charts if the data are unreferenced) show similar results to the EOF Loadings and Scores, but show the weights for the X and Y components of the CCA modes together. The corresponding temporal scores on the centre graph (the red line is for the X variables, and the green line is for the Y variables). It is the correlation between the curves shown on this central graph that CCA attempts to maximize. The canonical correlation is a function partly of any relationships between the X and Y data that CPT exploits to make predictions, but also partly a function of the numbers of X EOF modes retained. Thus the canonical correlation can be artificially inflated simply by forcing COT to retain a large number of EOFs. High canonical correlations should not necessarily be interpreted as a sign of a good predictive model.

The weights and scores for different CCA modes can be displayed using the control at the top left of the CCA maps window, but, unlike the EOF modes, only those CCA modes that CPT retains can be viewed. The results can be saved to an output file, but the actual weights are saved rather than the correlations between the original gridded / station / index data and the temporal scores for the corresponding X and Y components of the CCA modes. The weights are known as "homogeneous maps" (even for unreferenced data).

The graphics can be saved as JPEG files by right clicking anywhere in the child window, and then selecting the desired graphic to be saved. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. The map and graph titles can be set using the Customise option upon right-clicking in the child window.

Tables of the forecasts for individual gridpoints/stations/series and a graph of the historical and current forecast(s) are shown. The table shows the years for which the forecasts apply, some information about the definitions of the categories, and the forecasts themselves, presented in different formats.

In the thresholds box (top left) the climatological period is first indicated, and defines the data used to define the categories. The thresholds dividing the categories are then given in the units of the original data, possibly standardized depending on the standardization option. For example, if the original data are in mm, and the anomaly standardization is selected, the thresholds will be in mm anomalies from the climatological average. The climatological probabilities, below, show the proportion of years in the climatological period that were in each category. By default these probabilities will be 33% (assuming there are no ties, in which case the probabilities may deviate from 33% somewhat), but the actual probabilities will depend upon the threshold settings. The climatological odds are equal to the probability that the category in question was observed, divided by the probability that it was not observed. For example, if the climatological probability of "above-normal" is 33% the odds will be 0.5 ("above-normal" occurs half as often as "normal" and "below-normal" combined, i.e., it occurs half as of often as it does not occur).

The forecasts box presents the forecast in different formats. Probabilistic forecasts and odds are presented first. If the climatological probabilities are equiprobable and the forecast probability for the "normal" category is the lower than for the outer categories, it may be worth transforming the Y data so that they are normally disrtibuted. Applying the transformation will require CPT to reset. The probabilities are derived from the best-guess forecast (listed as "Forecast" under "Forecast ranges") by assuming that the errors in the best-guess forecast will be normally distributed (or, more strictly, follow a Student's t distribution), with the variance of the errors defined by the variance of the errors in the cross-validated predictions (alternative error variances can be set using Options ~ Forecast Settings). If the Y data have been transformed it is assumed that the forecast errors of the transformed data will be normally distributed. From the best-guess forecast and the assumed error variance and distribution the probabilities of exceeding the various thresholds can be calculated. Prediction intervals can also be calculated for a given level of confidence. The prediction intervals are shown as the upper and lower limits under "Forecast ranges". The default is to define the intervals as one standard error from the best guess, which translates to a confidence level of about 68%. The default can be adjusted using Options ~ Forecast Settings.

The graph shows the history of the cross-validated forecasts (green line) and observations (red line), and the current forecast(s) shown by green crosses. The prediction intervals for the forecasts can be added by right-clicking on the graph, and following the "Customize" option. The vertical coloured divisions show the three categories. Note that when calculating the cross-validated forecasts, the threshold defintions will have fluctuated slightly, and so the current categorization fo the forecasts and observations may not exactly match the cross-validated classification. This fluctuation is the reason for any apparent discrepancies between the graph and the contingency tables. The graph can be saved as a JPEG file by right-clicking anywhere on the graphic. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. In most cases, the highest quality image should be manageable. The map titles can be set using the Customise option upon right-clicking on the graphic. The forecast values can be saved by using File ~ Output Results.

Tables of an ensembles of forecasts for individual gridpoints/stations/series and a graph of the historical and current forecast(s) are shown. The table shows an ensemble of forecast values for the current variable and forecast target period.

The ensemble is generated by defining equi-probable intervals between the forecasts, based on the cross-validated (by default) error distribution. The default number of ensemble members is nine, so that there is a 10% probability of the observed value falling between and two sorted member, and a 10% probability of the observed value being larger than the largest member, and another 10% probability of it being smaller than the smallest member. The number of ensemble members can be reset using Options ~ Forecast Settings), as can the method for calculating the error variance. The retroactive and fitted error variances are alternative options.

The graph shows the history of the cross-validated forecasts (green line) and observations (red line), and the current ensemble of forecast(s) shown by green crosses. The prediction intervals for the forecasts can be added by right-clicking on the graph, and following the "Customize" option. The vertical coloured divisions show the three categories. Note that when calculating the cross-validated forecasts, the threshold defintions will have fluctuated slightly, and so the current categorization fo the forecasts and observations may not exactly match the cross-validated classification. This fluctuation is the reason for any apparent discrepancies between the graph and the contingency tables. The graph can be saved as a JPEG file by right-clicking anywhere on the graphic. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. In most cases, the highest quality image should be manageable. The map titles can be set using the Customise option upon right-clicking on the graphic. The forecast values can be saved by using File ~ Output Results.

Tables of an ensembles of forecasts for individual gridpoints/stations/series and a graph of the historical and current forecast(s) are shown. The table shows an ensemble of forecast values for the current variable and forecast target period.

The ensemble is generated by defining equi-probable intervals between the forecasts, based on the cross-validated (by default) error distribution. The default number of ensemble members is nine, so that there is a 10% probability of the observed value falling between and two sorted member, and a 10% probability of the observed value being larger than the largest member, and another 10% probability of it being smaller than the smallest member. The number of ensemble members can be reset using Options ~ Forecast Settings), as can the method for calculating the error variance. The retroactive and fitted error variances are alternative options.

The graph shows the history of the cross-validated forecasts (green line) and observations (red line), and the current ensemble of forecast(s) shown by green crosses. The prediction intervals for the forecasts can be added by right-clicking on the graph, and following the "Customize" option. The vertical coloured divisions show the three categories. Note that when calculating the cross-validated forecasts, the threshold defintions will have fluctuated slightly, and so the current categorization fo the forecasts and observations may not exactly match the cross-validated classification. This fluctuation is the reason for any apparent discrepancies between the graph and the contingency tables. The graph can be saved as a JPEG file by right-clicking anywhere on the graphic. A default name is given to the file, but this can be changed using the browse button. The quality of the JPEG file can be adjusted using the slider or by the quality indicator, which ranges between 0.01 and 1.00. The highest quality is obtained using 1.00. The size of the JPEG file is affected by the quality chosen, with larger files being generated the higher the selected quality. In most cases, the highest quality image should be manageable. The map titles can be set using the Customise option upon right-clicking on the graphic. The forecast values can be saved by using File ~ Output Results.

A set of maps can be drawn to illustrate the forecasts for all grids / stations / series for a given forecast time. The forecasts can be displayed either as actual forecast values (or anomalies, or % departures, etc, based on the standarization setting) with prediction intervals, or as probabilities.