Statistical foundations

Michael K. Tippett

International Research Institute for Climate and Society

The Earth Institute, Columbia University

ERFS Climate Predictability Tool Training Workshop May 4-9, 2009

Ideas

▶
▶

Linear regression models

ŷ = Ax + b

▶ y = the predictand, the quantity to be predicted. E.g.,

rainfall, temperature. ▶ ŷ = the prediction from the regression model. ▶ x = the predictor. ▶ A = regression coefficient(s). ▶ b = constant = ⟨y⟩ − A⟨x⟩

Linear relation between predictor and predictand.

Generalized linear models can be used for nonlinear relations

Linear regression models

Select A so that the sum of squared error is minimized.

min⟨∥y − Ax∥^{2}⟩= min∑_{i=1}^{n}∥y(t_{i}) − Ax(t_{i})∥^{2}

data at time t_{i}.

Univariate linear regression model

When x and y are scalars, regression finds the line that minimizes the sum of squared vertical differences between the line and the data.

Nonlinear relations.

Line fits the data poorly.

Nonlinear relations.

Line fits the data poorly. GLM (Poisson regression) does well.

Goodness of fit

SSE = sum of square error = ∑(y − ax − b)^{2}

Goodness of fit

SSE depends on the magnitude of the data.

Goodness of fit

Normalize SSE by something proportional to the variance. SSA = sum of squared anomalies = ∑(y − b)^{2}

\frac{SSE}{SSA} = fraction of variance explained = "signal" Related to the linear correlation r by

\frac{SSE}{SSA} = r^{2}

1 − r^{2} = fraction of unexplained variance = "noise"

Goodness of fit

Correlation does not depend on the magnitude of the data.

Goodness of fit

Regression and correlation sensitive to outliers. Assumption of Gaussian distributions.

Multiple linear regression

ŷ = a_{1}x_{1} + a_{2}x_{2} + … a_{m}x_{m} + b

▶ ŷ = the predictand, the quantity to be predicted. E.g.,

rainfall, temperature. ▶ x_{1}, x_{2}, …, x_{m} = the predictors. ▶ a_{1}, a_{2}, …, a_{m} = regression coefficients. ▶ b = constant = ⟨y⟩ − a_{1}⟨x_{1}⟩ − a_{2}⟨x_{2}⟩ − … a_{m}⟨x_{m}⟩

Data and number of predictors

Simple counting arguments show that at least n = m + 1 data points are required to estimate the m + 1 parameters of the regression model.

But with n = m + 1 data points and any m predictors the data can fit perfectly.

Such a regression while fitting the data perfectly would like have little skill on independent data.

Example

m predictors are used to form a regression using n data points. Check goodness of fit to the data.

Apply the regression equations to n additional independent data points. Check goodness of fit in the independent data.

Example

m = 20 predictors n = 21 samples Predictors are unrelated to the predictand.

Example

m = 20 predictors n = 40 samples Predictors are unrelated to the predictand.

Example

m = 20 predictors n = 80 samples Predictors are unrelated to the predictand.

Example

m = 20 predictors n = 160 samples Predictors are unrelated to the predictand.

Rule of thumb

5–10 samples per predictor. n ≈ (5 × m) – (10 × m)

Key points

The skill of a regression based forecast should be computed on independent data.

▶ Independent of the data used to compute the regression

coefficients. Over-fitting. ▶ Independent of the data used to select the predictors.

Selection bias.

Selection bias

Example: Suppose the regression coefficients are estimated half of the data and the skill is computed using the other half of the data. Sounds good, but if the predictors are selected using all the data, skill appears better than it really is.

Selection Bias: Example

m = 10 predictors n = 50 samples

Predictors are unrelated (r^{2} = 0) to the predictand as before, but were selected based on their correlation with the predictand in the entire data set.

Selection Bias: Example

m = 10 predictors n = 50 samples

Predictors are unrelated (r^{2} = 0) to the predictand as before, but were selected based on their correlation with the predictand in the first half of the data set.

Avoiding selection bias

3 independent data sets:

▶ Estimate regression models on data set 1.

▶ Select a “best” model on the basis of skill on data set 2. ▶ Evaluate skill of best model on data set 3.

Example: Select 2 predictors from a pool of 100. One predictor has r^{2} = 0.5, the rest are unrelated. Each data set has 25 samples. “Best” model r^{2} = 0.45

Avoiding selection bias (2)

2 independent data sets: ▶ Estimate regression models on data set 1. ▶ Select a “best” model on the basis an in-sample criteria

like AIC or BIC. Rewards model fit but penalizes model complexity.

▶ Evaluate skill of best model on data set 2.

Cross-validation

A method for mimicking real forecasts.

An alternative to splitting the data.

▶ Remove some number k of samples from the data set. ▶ Compute the model on the remaining n − k samples. ▶ Use that model to predict the left-out samples.

▶ Sometimes a set of k contiguous in time samples are left out and only the middle one is predicted to deal with temporal correlation.

▶ Repeat.

Data reduction and PCA

What if the predictors are spatial fields such as the SST or the output of a CGCM?

The number of grid point values (100’s, 1000’s)is large compared to the number time samples (10’s for climate)

Need to represent the information in the spatial field with fewer numbers.

▶ Pick a few “representative” grid points

▶ Noisy

▶ Spatial average e.g., NINO 3.4. All India Rainfall.

▶ Principal component analysis (PCA). Also called EOF.

▶ Weighted spatial average.

▶ Weights are chosen in an optimal manner to maximize explained variance.

Example: PCA of tropical SST

Weight (EOF) 1 Time series (PC) 1

Explains 26% of the total variance of the SST field.

Computed in the Data Library web interface.

Example: PCA of tropical SST

Weight (EOF) 2 Time series (PC) 2

Explains 12.5% of the total variance of the SST field.

Computed in the Data Library web interface.

Example: PCA of tropical SST

Weight (EOF) 3 Time series (PC) 3

Explains 7.4% of the total variance of the SST field.

Computed in the Data Library web interface.

PCR

Principal component regression (PCR). Predictors are PCs.

CCA Summary

▶