北大暑期课程《回归分析》(Linear-Regression-Analysis)讲义PKU5汇编

Class 5: ANOVA (Analysis of Variance) and F-tests

I. What is ANOVA

What is ANOVA? ANOVA is the short name for the Analysis of Variance. The essence

of ANOVA is to decompose the total variance of the dependent variable into two additive components, one for the structural part, and the other for the stochastic part, of a regression. Today we are going to examine the easiest case.

II. ANOVA: An Introduction

Let the model be

εβ+= X y .

Assuming x i is a column vector (of length p) of independent variable values for the i th'

observation,

i i i εβ+='x y .

Then b 'x i is the predicted value.

sum of squares total:

[]

∑-=2

Y y SST i []

∑-+-=2

'x b 'x y Y b i i i

[][]

[][]

∑∑∑-+-+-=Y -b 'x b 'x y 2Y b 'x b 'x y 2

2

i i i i i i

[][]

∑∑-+=2

2

Y b 'x e i i

because [][][]∑∑=-=--0Y b 'x e Y b 'x b 'x y i

i i i i .

This is always true by OLS. = SSE + SSR

Important: the total variance of the dependent variable is decomposed into two additive parts: SSE, which is due to errors, and SSR, which is due to regression. Geometric interpretation: [blackboard ]

Decomposition of Variance

If we treat X as a random variable, we can decompose total variance to the between-group portion and the within-group portion in any population:

()()()i i i x y εβV 'V V +=

Prove:

()()

i i i x y εβ+='V V

()

()()

i i i i x x εβεβ,'Cov 2V 'V ++=

()()i

i

x εβV 'V +=

(by the assumption that ()

0 ,'Cov =εβk x , for all possible k.)

The ANOVA table is to estimate the three quantities of equation (1) from the sample.

As the sample size gets larger and larger, the ANOVA table will approach the equation closer and closer.

In a sample, decomposition of estimated variance is not strictly true. We thus need to

separately decompose sums of squares and degrees of freedom. Is ANOVA a misnomer?

III. ANOVA in Matrix

I will try to give a simplied representation of ANOVA as follows:

[]

∑-=2

Y y SST i (

)

∑-+=i i y Y 2Y y 2

2

∑∑∑-+=i i y Y 2Y y 2

2

∑-+=2

22

Y n 2Y n y i (because ∑=Y n y i )

∑-=2

2

Y n y i

2

Y n y 'y -=

y J 'y n /1y 'y -= (in your textbook, monster look)

SSE = e'e

[]

∑-=2

Y b 'x SSR i

()()[

]

∑-+=Y b 'x 2Y b 'x 2

2

i i

()[]

()∑∑-+=b 'x Y 2Y n b 'x 2

2

i i

()[]()∑∑--+=i i i e y Y 2Y n b 'x 2

2

()[]∑-+=2

2

2

Y n 2Y n b 'x i

(because ∑∑==0e ,Y n y i i , as always)

()[]∑-=2

2

Y

n b 'x i

2

Y n Xb X'b'-=

y J 'y n /1y X'b'-= (in your textbook, monster look)

IV. ANOVA Table

Let us use a real example. Assume that we have a regression estimated to be y = - 1.70 + 0.840 x

ANOVA Table

SOURCE SS DF MS F with Regression 6.44 1 6.44 6.44/0.19=33.89 1, 18

Error 3.40 18 0.19 Total 9.84

19

We know

∑=100x

i

, ∑=50y i , 12.509x 2

=∑i , 84.134y 2

=∑i , ∑=66.257y x i i . If we

know that DF for SST=19, what is n?

n= 20

5.220/50Y ==

84.95.25.22084.134Y n y SST 2

2

=??-=-=∑i

()[]

∑-+=0.1250.84x 1.7-SSR 2

i

[]

∑-???-?+?=0.125x 84.07.12x 84.084.07.17.12

i i

= 20?1.7?1.7+0.84?0.84?509.12-2?1.7?0.84?100- 125.0

= 6.44

SSE = SST-SSR=9.84-6.44=3.40

DF (Degrees of freedom): demonstration. Note: discounting the intercept when calculating SST. MS = SS/DF

p = 0.000 [ask students]. What does the p-value say?

V. F-Tests

F-tests are more general than t-tests, t-tests can be seen as a special case of F-tests.

If you have difficulty with F-tests, please ask your GSIs to review F-tests in the lab. F-tests takes the form of a fraction of two MS's.

MSR/MSE F , df2df1

An F statistic has two degrees of freedom associated with it: the degree of freedom in

the numerator, and the degree of freedom in the denominator.

An F statistic is usually larger than 1. The interpretation of an F statistics is that

whether the explained variance by the alternative hypothesis is due to chance. In other words, the null hypothesis is that the explained variance is due to chance, or all the coefficients are zero.

The larger an F-statistic, the more likely that the null hypothesis is not true. There is a

table in the back of your book from which you can find exact probability values.

In our example, the F is 34, which is highly significant.

VI. R 2

R 2 = SSR / SST

The proportion of variance explained by the model. In our example, R-sq = 65.4%

VII. What happens if we increase more independent variables.

1. SST stays the same.

2. SSR always increases.

3. SSE always decreases.

4. R 2 always increases.

5. MSR usually increases.

6. MSE usually decreases.

7. F-test usually increases.

Exceptions to 5 and 7: irrelevant variables may not explain the variance but take up degrees of freedom. We really need to look at the results.

VIII. Important: General Ways of Hypothesis Testing with F-Statistics.

All tests in linear regression can be performed with F-test statistics. The trick is to run

"nested models."

Two models are nested if the independent variables in one model are a subset or linear

combinations of a subset （子集）of the independent variables in the other model.

That is to say. If model A has independent variables (1, 1x , 2x ), and model B has

independent variables (1, 1x , 2x ,3x ), A and B are nested. A is called the restricted model; B is called less restricted or unrestricted model. We call A restricted because A implies that

0=3β. This is a restriction.

Another example: C has independent variable (1, 1x , 2x +3x ), D has (1, 2x +3x ). C and A are not nested.

C and B are nested. One restriction in C: 32ββ=

.

C and

D are nested. One restriction in D: 0=1β. D and A are not nested.

D and B are nested: two restriction in D: 32ββ=

; 0=1β.

We can always test hypotheses implied in the restricted models. Steps: run two

regression for each hypothesis, one for the restricted model and one for the unrestricted

model. The SST should be the same across the two models. What is different is SSE and SSR. That is, what is different is R 2. Let

()()df df SSE ,df df SSE u u r r ==;

df df ()()0u r u r r u n p n p p p -=---=-<

Use the following formulas:

()()()()()

,SSE SSE /df SSE df SSE F SSE /df r u r u dfr dfu dfu u u

---=

or

()()()()()

,SSR SSR /df SSR df SSR F SSE /df u r u r dfr dfu dfu u u

---=

(proof: use SST = SSE+SSR)

Note, df(SSE r )-df(SSE u ) = df(SSR u )-df(SSR r ) =df ?,

is the number of constraints (not number of parameters) implied by the restricted model

or

()()()2

2

,2R R /df F 1R /df

u

r dfr dfu dfu

u

u

--?=- Note that

df 1df ,2

F t =

That is, for 1df tests, you can either do an F-test or a t-test. They yield the same result. Another way to look at it is that the t-test is a special case of the F test, with the numerator DF being 1.

IX. Assumptions of F-tests

What assumptions do we need to make an ANOVA table work?

Not much an assumption. All we need is the assumption that (X'X) is not singular, so that the least square estimate b exists.

The assumption of ε'X =0 is needed if you want the ANOVA table to be an unbiased estimate of the true ANOVA (equation 1) in the population. Reason: we want b to be an unbiased estimator of β, and the covariance between b and εto disappear.

For reasons I discussed earlier, the assumptions of homoscedasticity and non-serial correlation are necessary for the estimation of ()i V ε.

The normality assumption that εi is distributed in a normal distribution is needed for small samples.

X. The Concept of Increment

Every time you put one more independent variable into your model, you get an increase in 2

R . We sometime called the increase "incremental 2

R ." What is means is that more variance is explained, or SSR is increased, SSE is reduced. What you should understand is that the incremental 2

R attributed to a variable is always smaller than the 2

R when other variables are absent.

XI. Consequences of Omitting Relevant Independent Variables

Say the true model is the following:

0112233i i i i i y x x x ββββε=++++.

But for some reason we only collect or consider data on 21,,x and x y . Therefore, we omit

3x in the regression. That is, we omit in 3x our model. We briefly discussed this problem

before. The short story is that we are likely to have a bias due to the omission of a relevant variable in the model. This is so even though our primary interest is to estimate the effect of

1x or 2x on y.

Why? We will have a formal presentation of this problem.

XII. Measures of Goodness-of-Fit

There are different ways to assess the goodness-of-fit of a model. A. R 2

R 2 is a heuristic measure for the overall goodness-of-fit. It does not have an associated test statistic.

R 2 measures the proportion of the variance in the dependent variable that is “explained” by the model: R 2 =

SSE

SSR SSR

SST SSR +=

B. Model F-test

The model F-test tests the joint hypotheses that all the model coefficients except for the constant term are zero.

Degrees of freedoms associated with the model F-test: Numerator: p-1 Denominator: n-p.

C. t-tests for individual parameters

A t-test for an individual parameter tests the hypothesis that a particular coefficient is equal to a particular number (commonly zero).

t k = (b k - βk0)/SE k , where SE k is the (k, k) element of MSE(X’X)-1, with degree of freedom=n-p.

D. Incremental R 2

Relative to a restricted model, the gain in R 2 for the unrestricted model:

?R2= R u2- R r2

E. F-tests for Nested Model

It is the most general form of F-tests and t-tests. ()()()()()

,SSE SSE /df SSE df SSE F SSE /df r u r dfu dfr u dfu u u

---=

It is equal to a t-test if the unrestricted and restricted models differ only by one single parameter.

It is equal to the model F-test if we set the restricted model to the constant-only model.

[Ask students] What are SST, SSE, and SSR, and their associated degrees of freedom, for the constant-only model?

Numerical Example

A sociological study is interested in understanding the social determinants of mathematical

achievement among high school students. You are now asked to answer a series of questions. The data are real but have been tailored for educational purposes. The total number of observations is 400. The variables are defined as: y: math score

x1: father's education

x2: mother's education

x3: family's socioeconomic status x4: number of siblings x5: class rank

x6: parents' total education (note: x6 = x1 + x2) For the following regression models, we know: Table 1 SST SSR SSE DF R 2 (1) y on (1 x1 x2 x3 x4) 34863 4201 (2) y on (1 x6 x3 x4) 34863 396 .1065 (3) y on (1 x6 x3 x4 x5) 34863 10426 24437 395 .2991 (4) x5 on (1 x6 x3 x4) 269753 396 .0210

1. Please fill the missing cells in Table 1.

2. Test the hypothesis that the effects of father's education (x1) and mother's education (x2) on math score are the same after controlling for x3 and x4.

3. Test the hypothesis that x6, x3 and x4 in Model (2) all have a zero effect on y.

4. Can we add x6 to Model (1)? Briefly explain your answer.

5. Test the hypothesis that the effect of class rank (x5) on math score is zero after controlling for x6, x3, and x4.

Answer: 1. SST SSR SSE DF R 2 (1) y on (1 x1 x2 x3 x4) 34863 4201 30662 395 .1205 (2) y on (1 x6 x3 x4) 34863 3713 31150 396 .1065 (3) y on (1 x6 x3 x4 x5) 34863 10426 24437 395 .2991 (4) x5 on (1 x6 x3 x4) 275539 5786 269753 396 .0210

Note that the SST for Model (4) is different from those for Models (1) through (3). 2.

Restricted model is 01123344()y b b x x b x b x e =+++++

Unrestricted model is ''''''0112

23344y b b x b x b x b x e =+++++

(31150 - 30662)/1

F 1,395 = -------------------- = 488/77.63 = 6.29 30662 / 395 3.

3713 / 3

F 3,396 = --------------- = 1237.67 / 78.66 = 15.73 31150 / 396

4. No. x6 is a linear combination of x1 and x2. X'X is singular.

5.

(31150 - 24437)/1

F 1,395 = -------------------- = 6713 / 61.87 = 108.50 24437/395

t = 10.42t ===