SPSS加权最小二乘法及应用

SPSS数据统计分析与实践主讲：周涛副教授北京师范大学资源学院2007-12-4教学网站：/Courses/SPSS第十五章：加权最小二乘法（Weighted Least Squares）本章内容：一、最小二乘法的应用领域z根据需要人为地改变观测量的权重z Remedial Measures for Unequal Error Variances二、SPSS提供的WLS过程z Linear Regression procedure (with weight variable)z Weight Estimation procedure三、相关输出结果的比较z OLS与WLS比较z SPSS提供的两种WLS方法比较加权最小二乘法应用（一）根据需要人为地改变观测量的权重根据需要人为地改变观测量的权重－－实例实验中收集的15对数据，每对数据都是将n份样品混合后测得的平均结果，但各对数据的n大小不等，试求出X对Y的线性方程。

数据源：郭祖超，《医用数理统计方法》第三版P249根据需要人为地改变观测量的权重－－实例方法一：如果不考虑样品混合量的差异，则该问题是一个非常简单的线性回归问题，可直接拟合回归方程，结果如下：Model Summary b.987a.975.973.11330Model1R R SquareAdjustedR SquareStd. Error ofthe Estimate Predictors: (Constant), xa.Dependent Variable: yb.Coefficients a7.454.17343.143.000-.015.001-.987-22.468.000 (Constant)xModel1B Std. ErrorUnstandardizedCoefficientsBetaStandardizedCoefficientst Sig.Dependent Variable: ya.Y = 7.45 –0.015 * X (R2=0.98)根据需要人为地改变观测量的权重－－实例方法二：由于每对测量数据都是将n份样品混合后测得结果，显然混合的样品越多，测得的结果越稳定，即变异越小。

如果直接拟合方程，则是将所有测量值均一视同仁，1份样品的测量结果与15份样品混合后的测量结果等价对待，这显然不太合理。

为此可以考虑在分析中将样品数n作为权重变量，n 值越大的观测量在计算中给予的权重越高，对方程的影响越大，即按照加权最小二乘法来拟合回归方程。

根据需要人为地改变观测量的权重－－实例z SPSS操作步骤：z AnalyzeÆRegressionÆLinearz Dependent: Yz Independent: Xz WLS Weight: nWLS WeightWLS: Weighted Least SquaresWLS 输出结果Model Summary b,c.982a.965.962.29365Model 1RR SquareAdjusted R SquareStd. Error of the EstimatePredictors: (Constant), x a. Dependent Variable: yb. Weighted Least Squares Regression - Weighted by nc. Y = 7.19 –0.014 * X (R 2=0.97)Coefficients a,b7.190.18838.316.000-.014.001-.982-18.816.000(Constant)xModel 1B Std. Error Unstandardized Coefficients BetaStandardizedCoefficientstSig.Dependent Variable: ya. Weighted Least Squares Regression - Weighted by nb.WLS 与OLS 输出结果比较1.在OLS中，测定系数为0.975, 而在WLS中测定系数降低为0.965。

2.由于测定系数是按照普通最小二乘法进行计算，因此加权后的方程测定系数必然小于普通最小二乘法，即此时不能使用测定系数来判断模型的优劣。

WLS 与OLS 输出结果比较3. 通过绘制OLS和WLS的回归直线加以比较，如下图所示，WLS更靠近中部那些混合样品数据n较大的测量值，而对两端n较小的测量值则比OLS回归直线更远一些，显然这些测量值在计算时对方程的影响程度是不同的。

实现WLS的另一种方法z事实上，如果使用SPSS的Weight Case过程，将n指定为频数变量，然后进行普通的线性回归，得到的分析结果与上述加权最小二乘法完全相同。

z操作过程如下所示：实现WLS的另一种方法步骤：（1）调用Weight Case过程DataÆWeight Cases…）（2）调用线性回归过程（采用OLS）实现WLS 的另一种方法—输出结果Model Summary b,c.982a.965.962.29365Model 1RR SquareAdjusted R SquareStd. Error of the EstimatePredictors: (Constant), x a. Dependent Variable: yb. Weighted Least Squares Regression - Weighted by nc. Y = 7.19 –0.014 * X (R 2=0.97)Coefficients a,b7.190.18838.316.000-.014.001-.982-18.816.000(Constant)xModel 1B Std. Error Unstandardized Coefficients BetaStandardizedCoefficientstSig.Dependent Variable: ya. Weighted Least Squares Regression - Weighted by nb.加权最小二乘法应用（二）Unequal Error Variances Remedial Measures ----Weighted Least SquaresVariation of Errors Around the Regression LineX1X 2XYf(e)1. y values are normally distributedaround the regression line.2. For each x value, the “spread”orvariance around the regression line isthe same.Regression LineEqual Error Variancesip i p i i i X X X Y εββββ+++++=−−1,122110K 110,,−p βββK are parameters1,21,,,−p i i i X X X K are known constantsi εare independent ),0(2σN ni ,,1L =Equal error variance(1)111)(××−××′′=p p p pp p YX X X b (2)ip i p i i i X X X Y εββββ+++++=−−1,122110K 110,,−p βββK are parameters1,21,,,−p i i i X X X K are known constantsi εare independent ),0(2iN σni ,,1L =Unequal error variance⎥⎥⎥⎥⎥⎦⎤⎢⎢⎢⎢⎢⎣⎡=×222212000000}{n n n σσσεσK MM M K K (3)z The estimation of the regression coefficients in generalized model (3)could be done by using the estimators in (2)for regression model (1)with equal error variances. These estimators are still unbiased and consistent for model (3),but they no longer have minimum variance.z To obtain unbiased estimators with minimum variance, we must take into account that the different Y observations for the n cases no longer have the same reliability.z Observations with small variances provide more reliable information about the regression function than those with large variances.Error Variances KnownError Variances KnownWe first consider the estimation of the regressionfunction coefficients when the error varianceare known . This case is usually unrealistic, but it provides guidance as to how to proceed when the error variances are not know.2i σError Variances KnownWhen the error variances are known, we can usethe method of maximum likelihood to obtainestimators of the regression coefficients in (3).2i σ])(21exp[)2(1)(21,1110212/12−−=−−−−−=∏p i p i i i n i i X X Y L βββσπσK βwhere βdenotes the vector of the regression coefficients. We define the reciprocal of the variance as the weight w i :2i σ21i i w σ=(4)(5)⎥⎦⎤⎢⎣⎡−−−−−⎥⎦⎤⎢⎣⎡=−−=∑∏21,1110112/1)(21exp )2()(p i p i i n i n i i X X Y w w L βββπK β(6)Error Variances Known⎥⎦⎤⎢⎣⎡−−−−−⎥⎦⎤⎢⎣⎡=−−=∑∏21,1110112/1)(21exp )2()(p i p i i n i ni i X X Y w w L βββπK β(6)We find the maximum likelihood estimators of theregression coefficients by maximizing L(β)in formula (6)with respect to .110,,,−p βββK Since the error variances and hence the weightsare assumed to be known, maximizing L(β)with respect to the regression coefficients is equivalent to minimizing the exponential term :2i σi w 21,11101)(−−=−−−−=∑p i p i i n i i w X X Y w Q βββK (7): weight least squares criterionw QError Variances Known•Since the weight is inversely related to the variance , it reflects the amount of information contained in the observation Yi.•Thus, an observation Yi that has a large variance receivesless weight than another observation that has a small variance.•Intuitively, this is reasonable. The more precise is Yi (i.e.,the smaller is ), the more information Yi provides aboutE{Yi }and therefore the more weight it should receive infitting the regression function.2i σiw2iσLet the matrix W be a diagonal matrix containing the weight :i w ⎥⎥⎥⎥⎦⎤⎢⎢⎢⎢⎣⎡=×n n n w w w W K M M M K K 00000021The normal equations can then be expressed as follows:WYX b WX X w′=′)(and the weighted least squares and maximum likelihood estimators of the regression coefficients are:WYX WX X b w ′′=−1)(where b w is the vector of the estimated regressioncoefficients obtained by weighted least squares.(8)(9)(10)•The weighted least squares and maximum likelihood estimators of the regression coefficients in formula (10) are unbiased, consistent, and have minimum variance among unbiased linear estimator.•Thus, when the weights are known, bw generally exhibitsless variability than the ordinary least squares estimator b.Error Variances UnknownError Variances Unknownz If the variances of errors were known, the use of weighted least squares with weights wi would be straightforward.z Unfortunately, one rarely has knowledge of the variances of errors. We are then forced to use estimates of the variances.Estimation of Variance Function z The magnitudes of error variances often vary in a regular fashion with one or severalpredictor variables Xk or with the meanresponse E{Yi}.X ez Such a relationship between and one or several predictor can be estimated because the squared residual obtained from an ordinary least squares regression fit is an estimate of , provided that theregression function is appropriate.X e2i σ2i e 2i σ}{22ii E εσ=Some possible variance functions:1.A residual plot against X 1exhibits a megaphone shape . Regress the absolute residuals against X 1.2.A residual plot against exhibits a megaphone shape. Regress the absolute residuals against .3.A plot of squared residuals against X 3exhibits an upward tendency. Regress the squared residuals against X 3.4. A plot of the residuals against X 2suggests that the varianceincreases rapidly with increases in X 2up to a point and then increases more slowly. Regree the absolute residualsagainst X 2and .Y ˆY ˆ22XSummary of the WLS estimation process1.Fit the regression model by unweighted leastsquares and analyze the residuals.2.Estimate the variance function by regression eitherthe squared residuals or the absolute residuals onthe appropriate predictor(s).e the fitted values from the estimated varianceto obtain the weights w.i4.Estimate the regression coefficients using theseweights.SPSS Example of WLSExamplez A health researcher, interested in studying the relationship between diastolic blood pressure(舒张压) and age among healthy adult women 20 to 60 years old, collected data on 54 subjects.Step 1: Preliminary analysesBlood_Pressure= 56.157 + 0.58003 * Age Nonconstant errorvarianceStep 2: Estimating the variance functionThe plot of the absolute residuals against X suggests that a linearrelation between the error standard deviation and X maybe reasonable.Age s×+=198.0549.1ˆdenotes the estimated expected standard deviationsˆTo obtain the weights w i , the analyst obtained the fitted values from the standard deviation function in (11). For case 1 (X 1=27)(11)801.327198.0549.1ˆ1=×+=s0692.0)801.3(1)ˆ(12211===s wStep 2: Estimating the variance functionX i Yiei|ei|isˆi w21)ˆ(1swi=Step 3: Weighted Least SquaresX Y596.0566.55ˆ+=Coefficients a,b55.566 2.52122.042.000.596.079.7227.526.000(Constant)AgeModel 1B Std. Error Unstandardized Coefficients BetaStandardizedCoefficientstSig.Dependent Variable: Blood_Pressurea. Weighted Least Squares Regression - Weighted by wib. (12)Step 4: Comparison of WLS and OLSCoefficients a56.157 3.99414.061.000.580.097.6395.983.000(Constant)AgeModel 1B Std. Error Unstandardized Coefficients BetaStandardizedCoefficientstSig.Dependent Variable: Blood_Pressurea. Coefficients a,b55.566 2.52122.042.000.596.079.7227.526.000(Constant)AgeModel 1B Std. Error Unstandardized Coefficients BetaStandardizedCoefficientstSig.Dependent Variable: Blood_Pressurea. Weighted Least Squares Regression - Weighted by wib. OLSWLSb w generally exhibits less variability than the ordinary least squares estimator bIt is interesting to note that this Std. is somewhat smaller than the Std.of the estimate obtained by OLS. The reduction of about 18% is the result of the recognition of unequal error variances when using WLS.Step 4: Comparison of WLS and OLSCoefficients a56.157 3.99414.061.000.580.097.6395.983.000(Constant)AgeModel 1B Std. Error Unstandardized Coefficients BetaStandardizedCoefficientstSig.Dependent Variable: Blood_Pressurea. Coefficients a,b55.566 2.52122.042.000.596.079.7227.526.000(Constant)AgeModel 1B Std. Error Unstandardized Coefficients BetaStandardizedCoefficientstSig.Dependent Variable: Blood_Pressurea. Weighted Least Squares Regression - Weighted by wib. OLSWLSSince the regression coefficients changed only a little , the analyst concluded that there was no need to reestimate the standard deviation function and the weights based on the residuals for the weighted regression in formula (12).Step 4: Comparison of WLS and OLS If the estimated coefficients differ substantially from the estimated regression coefficients obtained by OLS, it is usually advisable to iterate the WLS process by using the residuals from the WLS fit to reestimate the variance or Std. Function and then obtain revised weights. This iteration process is often called iteratively reweighted least squares.SPSS Weight Estimation procedureSPSS Weight Estimation procedurez尽管采用SPSS Linear Regression过程（人为确定WLS Weight变量）可以用于WLS，但两个因素使得使用该过程比较麻烦：1.有时WLS Weight变量值不容易确定。