Econometrics: Mathematical Derivation of Least Squares
Econometrics: Mathematical Derivation of Least Squares Econ 4400
Popular in Elementary Econometrics
Popular in Economcs
This 0 page Class Notes was uploaded by Noah Scovill on Sunday March 20, 2016. The Class Notes belongs to Econ 4400 at Ohio State University taught by Anthony Bradfield in Fall 2015. Since its upload, it has received 38 views. For similar materials see Elementary Econometrics in Economcs at Ohio State University.
Reviews for Econometrics: Mathematical Derivation of Least Squares
Report this Material
What is Karma?
Karma is the currency of StudySoup.
Date Created: 03/20/16
asdf The Mathematical Derivation of Least Squares The Bivariate Case For the case in which there is only one independent variable the classical OLS ordinary least squares regression model can be expressed as follows Mm me H where yis dependent response variable Xi is the independent explanatory variable Bois the regression constant 81 is the regression coef cient for the effect of X and eis the error we make in predicting yfrom X Now in running the regression model what are trying to do is to minimize the sum of the squared errors of prediction ie of the evalues across all cases Mathematically this quantity can be expressed as SEE Bi 2 Speci cally what we want to do is nd the values of b0The estimate ofe0and b1The estimate of l that minimize the quantity in Equation 2 above So how do we do this The key is to think back to differential calculus and remember how one goes about nding the minimum value of a mathematical function This involves taking the derivative of that function If we want to nd the values of boand In that minimize SSE we need to express SSE in terms of boand b1 take the derivatives of SSE with respect to boand b1 set these derivatives to zero and solve for boand b1 However since SSE is a function of two critical variables boand 191 we will need to take the partial derivatives of SSE with respect to 1Page asdf boand b1 In practice this means we will need to take the derivative of SSE with regard to each of these critical variables one at a time while treating the other critical variable as a constant keeping in mind that the derivative of a constant always equals zero In effect what this does is take the derivative of SSE with respect to one variable while holding the other constant We begin by rearranging the basic OLS equation for the bivariate case so that we can express ein terms of y Xi b0 and b1 This gives us eiyib0b1xi Substituting this expression back into Equation 2 we get SSEy bo blx2 4 where n the sample size for the data It is this expression that we actually need to differentiate with respect to boand b1 Let s start by taking the partial derivative of SSE with respect to the regression constant b0 ie BSSE a n b b 2 8190 ab0 y 0 1x1 In doing this we can move the summation operator 2 out front since the derivative of a sum is equal to the sum of the derivatives aSSE39 a 2 l7b abo yl 0 1x1 We then focus on differentiating the squared quantity in parentheses Since this quantity is a composite we do the math in parentheses and then square the result we need to use the chain rule in order to obtain the partial derivative of SSE with 2Page asdf respect to the regression constant In order to do this we treat yi b1 and was constants This gives us aSSE39 abo 2yi bo b1x1 Further rearrangement gives us a nal result of ESSE39 8190 2ilty bo 42x 5 For the time being let s put this result aside and take the partial derivative of SSE with respect to the regression coef cient b1 ie MM 3 n 2 b b ab ab gm 0 1x1 Again we can move the summation operator 2 out front BSSE a 2 E n b b We then differentiate the squared quantity in parentheses again using the chain rule This time however we treat yi b0 and Xas constants With some subsequent rearrangement this gives us ESSE39 ab 2 2ixiyi bO b1xl 6 With that we have our two partial derivatives of SSE in Equations 5 and 6 The next step is to set each one of them to zero O 2iyi bO b1xl 7 3Page asdf 02 2ixiyi b0 bx 8 Equations 7 and 8 form a system of equations with two unknowns our OLS estimates boand b1 The next step is to solve for these two unknowns We start by solving Equation 7 for b0 First we get rid of the 2 by multiplying each side of the equation by 12 0Zy bo b1x1gt 1 Next we distribute the summation operator though all of the terms in the expression in parentheses 0271 Z 90 2 91351 1 1 1 Then we add the middle summation term on the right to both sides of the equation giving us 2 b0 223 1 Z 91351 1 1 1 Since boand b1 the same for all cases in the original OLS equation this further simpli es to quot70 ziyi 2 71751 1 1 To isolate boon the left side of the equation we then divide both sides by n gyl b X1196 g n 1 n 90 Equation 9 will come in handy later on so keep it in mind Right now though it is important to note that the rst term on the right 4Page asdf of Equation 9 is simply the mean of y while everything following b1 in the second term on the right is the mean of Xi 1903 191 10 Now we need to solve Equation 8 for b1 Again we get rid of the 2 by multiplying each side of the equation by 12 Oinyi b0 b1x1 1 Next we distribute Xthrough all of the terms in parentheses Ozixiyi xib0b1xi2 1 We then distribute the summation operator through all of the terms in the expression in parentheses Oixiyi i xibO i 91392 1 1 1 Next we bring all of the constants in these terms ie boand b1 out in front of the summation operators as follows 0xiyi b0 xi b1 x3 1 1 1 We then add the last term on the right side of the equation to both sides 91 xi2 zixiyib0 xi 1 1 1 Next we go back to the value for bofrom Equation 9 and substitute it into the result we just obtained This gives us 5Page asdf n n yi ixi n b1z xi2 inyi In b1 1 2 xi 1 1 1 Multiplying out the last term on the right we get n n yii xi lixi b1 xi2 inyibl 1 n If we then add the last term on the right to both sides of the equation we get 2 n in n yii xi n 2 2 1 Z 1 1 b1 Xi b1 Xiyi 1 n 1 n On the left side of the equation we can then factor out b1 If we divide both sides of the equation by the quantity in the large brackets on the left side we can isolate b n iyiixi xiyi 1 1 b1 1 Finally if we multiply top and bottom by n we obtain the least square estimator for the regression coef cient in the bivariate case This is the form from lecture 6Page asdf nixiyi yixi 1 1 1 2 n2 xzx 1 1 b1 MD The Multiple Regression Case Deriving OLS with Matrices The foregoing math is all well and good if you have only one independent variable in your analysis However in practice this will rarely be the case rather we will usually be trying to predict a dependent variable using scores from several independent variables Deriving a more general form of the leastsquares estimator for situations like this requires the use of matrix operations The basic OLS regression equation can be represented in the following matrix form YXBe HE where Y is an nxl column matrix of cases dependent variable X is an nxk1 matrix of cases scores on the independent variables where the rst column is a placeholder column of ones for the constant and the remaining columns correspond to each independent variable B is a kl column matrix containing the regression constant and coef cients and e is an nxl column matrix of cases errors of prediction As before what we want to do is nd the values for the elements of B that minimize the sum of the squared errors The quantity that we are trying to minimize can be expressed as follows 55E e e 13 If you work out the matrix operations for the expression on the right you ll notice that the result is a scalar a single number consisting of the sum of the squared errors of prediction ie multiplying a lxN matrix by a le matrix produces a lxl matrix Le a scalar In order to take the derivative of the quantity with 7Page asdf regard to the B matrix we rst of all need to express e in terms of Y X and B e Y39 B Substituting the expression on the right side into Equation 13 we get SSE Y XB Y XB Next the transposition operator on the rst quantity in this product Y XB can distributed SSE Y B39X Y XB When this product is computed we get the following SSE Y Y Y39XB B X Y B X XB Now if multiplied out the two middle terms Y XB and B X Y are identical they produce the same scalar value As such the equation can be further simpli ed to 55E Y39Y 2Y39XB B39X39XB 14 We now have an equation which expresses SSE in terms of Y X and B The next step as in the bivariate case is to take the derivative of SSE with respect to the matrix B M w 2ng B X XB BB First since we are treating all matrices besides B as the equivalent of constants the rst term in parentheses based completely on the Y matrix has a derivative of zero Second the middle term known as a quotlinear formquot in B is the equivalent of a scalar term in which the variable we are differentiating with respect to is raised to the rst power Le a linearterm which means we obtain the derivative by dropping the B and taking the transpose of all the matrices in the expression which remain giving us 2X39Y Finally the third term known as a quotquadratic formquot in B is the equivalent of a scalar term in which the variable we are 8Page asdf differentiating with respect to is raised to the second power Le a quadratic term This means we obtain the derivative by dropping the B39 from the term and multiplying by two giving us 2X XB Thus the full partial derivative is aSSE aB 2X Y2XXB 15 The next step is to set this partial derivative to zero and solve for the matrix B This will give us an expression for the matrix of estimates that minimize the sum of the squared errors of prediction We start with the following 0 2X39Y 2X39XB We then subtract 2X XB from each side of the equation 2X39XB 2X39Y Next we eliminate the 2 on each term by multiplying each side of the equation by 12 X39XB X39Y Finally we need to solve for B by premultiplying each side of the equation the inverse of X X ie X X1 Remember that this is the matrix equivalent of dividing each side of the equation by X UK B X39x391 X Y 16 Equation 16 is the OLS estimator To tie this back to the bivariate case note closely what the expression on the right does While X Y gives the sum of the crossproducts of X and Y X X gives us the sum of squares for X Since premultiplying X Y by X X1 is the matrix equivalent of dividing X Y by X X this expression is basically doing the same thing as the scalar 9Page asdf expression for b1 in Equation 11 dividing the sum of the cross products of the IV or IVs and the DV by the sum of squares for the IV or IVs 10 Paaglta