What you see on this slide is where we essentially stopped on Tuesday.
I stated the theorem that the unbiased estimator of the variance of the disturbances u is given
by the sum of squared residuals in vector notation written as u hat prime u hat divided
by n minus k, where n is the number of observations and k is the number of regressors.
So we divide by the degrees of freedom and not by the number of observations or the number
of estimated residuals.
And the theorem states that this estimate would be an unbiased estimate.
I gave you already an intuition of why we have to divide by less than the number of
observations because I indicated that u hat, the residuals which were constructed by a
minimization problem, by the objective of making residuals as small as possible.
So the residuals are therefore smaller in absolute value than the true disturbances, at least in an
expected value sense.
The expected value of the squared sum of residuals is smaller than the expected value of the
squared sum of true residuals, even though we do not know the true residuals.
We can compute that and it is intuitive since the least squares estimation is designed precisely
with the aim of making the residuals as small as possible.
Now I started with a proof of this theorem, gave you the definition which you see here
of an item potent matrix and then I prove properties of an item potent matrix, which
you find here, in particular the properties that all eigenvalues are equal to one or zero
and that the trace of an item potent matrix is equal to its rank.
And when I gave you the proof, I realized that perhaps you are not so familiar with
the trace.
I assume that this has been covered in lectures on linear algebra, but perhaps it's not as
common to have knowledge on the trace of a matrix as it is common to have knowledge on
the rank of a matrix.
So I inserted a new slide here for today's lecture in which I just review properties
of the trace and actually define the trace of the matrix of a matrix in case you do not
know or did not know what the trace of a matrix is.
So here is essentially the definition, the trace of a matrix A, A shall be a square matrix
by the way, is just the sum of its diagonal elements.
So if this is matrix A, then what the trace does is that all elements which are on the
main diagonal of the matrix are summed.
So notation would be just take the sum over all aii elements.
So where the row index is the same as the column index, both cases i and then i goes
from 1 to n, this gives us the sum over all the diagonal elements here.
And this is the trace of a matrix.
I think it is useful to review the properties of the trace because I will make use of these
properties now in the proof that is to follow.
So here are the main properties of the trace.
If we have a linear combination of matrices, so alpha times A plus beta times B, where
A and B are essentially arbitrary quadratic matrices, but of the same size, obviously
otherwise we could not sum them, then the trace of alpha A plus beta B is equal to alpha
times the trace of A plus beta times the trace of B and alpha and beta of course are
just scalar values, so just some numbers taken from the space of real numbers.
Moreover what is trivial is that the trace of a matrix A is the same thing as the trace
of A prime, so of the transpose of A, because obviously the elements of the main diagonal
are they remain on the main diagonal.
So the trace of A must be equal to the trace of A prime.
And what is less obvious is that the trace of a matrix product A times B is the same
thing as the trace of the matrix product B times A.
So if these are two different matrices, it is not always clear of course that they are
also conformable in the opposite way, so if the product AB exists, this does not mean
that we can also take the product BA, but if it is possible to write the matrices in
reverse order, so if this reverse product is also well defined, then the trace of AB
is equal to the trace of BA.
And what is even more useful is that this not only holds for two matrices but generalizes.
So here for instance you have it for three matrices, the trace of ABC is equal to the
trace of CAB, so you may take the last matrix in this product and use it as the first term
in the product, make out of the ABC a CAB and the trace would be just the same.
And of course by the same token you can also take the B here and make this the first matrix
in this product here, so this is the same thing as the trace of BCA.
That's a very useful property as you will see now when we complete the proof of the
unbiasedness of the estimator 1 over n minus k times u hat prime u hat.
And I thought it useful to just provide this information to you in case you didn't have
it ready from your linear algebra lectures.
Well then we had this lemma here which I have already read out and we proved the lemma last
lecture and now comes the proof of the theorem.
And the proof of the theorem essentially is here, I will go through the first through
lines a little later, first look at this, it starts a little unusual because I start
with n minus k times the expectation of the estimator.
What I will prove is that n minus k times the expectation of the estimator is the same
thing as sigma square u times n minus k.
So if I can prove that this thing here sigma square u times n minus k is the same thing
as n minus k times the expectation of the estimator, then obviously the n minus k here
and the n minus k here cancel and we are left with what we want to prove, namely that the
expectation of the estimator is the true parameter sigma square u.
And this would then mean that the estimator is unbiased.
So that's what we do in this proof here.
Now in the proof we will use various properties of matrices of idempotent matrices and properties
of the trace.
So recall please that we had this matrix M here defined as identity matrix minus x x
plus so minus the Bupenrose inverse or if you want to write it in an explicit way, recalling
what the Bupenrose inverse is, it's i minus x x prime x inverse e x prime.
This matrix M is a symmetric matrix and as I have already pointed out is idempotent and
it does satisfy that M times the regressor matrix x is equal to zero, which you may perhaps
verify.
I think I asked you already once to verify it, but you can verify it again, it's always
useful to know.
So M and x are orthogonal to each other.
Now what we also know is that the estimated residuals u hat, which we use in estimating
now the residual variance or the error variance, these estimated residuals u hat are defined
as y minus x beta hat where beta hat is the least squares estimator of beta.
So beta hat is x prime x inverse e x prime y or is x plus times y.
So we may also write this as y minus x x plus y and this is then the same thing as M times
y times, so this is this matrix here times y as we have already stated elsewhere in the
lecture somewhat earlier.
And y is M x beta plus u of course, so just substituting in the model equation y is equal
to x beta plus u since M times x is zero, we know that this is equal to M u, right?
So u hat is equal to M u and this we will use now in the actual proof which comes here
and as I said starts out with n minus k times the expectation of sigma square u hat.
Maybe this is the expectation of u hat prime u hat because well the estimator here is one
over n minus k times u hat prime u hat since I multiply this by n minus k we just have
the expectation of u hat prime u hat, so we don't need to have the n minus k factor in
our computations all the time, this is actually why I pre-multiply here by n minus k.
u hat prime u hat is a scalar, right?
So we have a row vector here and a column vector here, so this is a scalar product which
we have here, we just sum the squared residuals and then we have this sum as just value taken
from R, so it's just a real number here.
The trace of a real number is just the real number, so the real number is always equal
to its trace, so we may also write this as the expectation of the trace of m u prime
times m u because m u is equal to u hat and m u prime is equal to u hat prime, so that's
the same thing.
Now you know that when we take the transpose of a product then we can also multiply the
two factors in reverse order transposed, so this thing here m u prime is the same
thing as u prime m, right?
u prime m prime, m is symmetric so we don't need the prime there but it's u prime m actually,
so if we multiply u prime m with m u and we know the matrix m is idempotent then we see
this is the same thing as u prime m u of which we have to take the trace here.
So we take the trace of u prime m u and the expectation of this matrix.
Obviously u prime m u is still a scalar, so this is still one by one what we have here.
But now we make use of the properties of the trace and rewrite the matrices of which the
trace is taken as u u prime m, so we take as I have shown you on the last slide the
last matrix here and use it as the first matrix in this matrix product, so we have the trace
of u u prime m here of which we take the expectation.
And this is convenient because the expectation of the trace is the same thing as the trace
of the expectation, another property which I might also have stated but I think that
should be clear and that this property holds.
So I have the trace of the expectation of u u prime times m.
I do not need to have m within the expectation operator because I take the matrix x as being
deterministic so I can pull it out of the operation.
So it's the expectation of u u prime and the expectation of u u prime is by our assumptions
a3 and a4 just sigma square u times identity matrix.
So that diagonal matrix with sigma square u's on the main diagonal and all other entries
are zero.
So we have here the trace of matrix sigma square u which is a scalar times identity
matrix times m, well identity matrix times m is of course just m and the sigma square
u is a constant factor which I can pull out of the trace, recall property one on the previous
slide so that's equal to sigma square u times the trace of m.
So m we had defined it is this matrix here so we have sigma square u times the trace
of an identity matrix with dimension n so n rows and columns minus x, x prime, x inverse,
x prime.
As I told you the trace is a linear operator so I can write the trace of this difference
here also as the difference of the traces so that's equal to sigma square u times trace
of identity matrix with dimension n minus the trace of x prime, x inverse, x prime,
x inverse.
Why do I now have x prime, x times x prime, x inverse because I have again taken the last
matrix here the x prime and put it in front of the product here so that I get x prime,
x, x prime, x inverse, this is something I can do within the trace.
But you now see x prime, x times x prime, x inverse is just the identity matrix.
But now careful what size, what row and column dimension does this matrix have?
Usually I would ask you that in the lecture and wait for your answer now but this is so
complicated with online lectures I will tell you x is a matrix of dimension n by k so x
prime is a matrix of dimension k by n right and x prime, x inverse was a matrix of dimension
k by k so this here is a k by k matrix so it's a k by k identity matrix.
So we have the trace of an n by n identity matrix minus the trace of a k by k identity
matrix obviously the case of an n by n identity matrix is just n because we have n elements
on the main diagonal and each of which is one and the trace of a k by k identity matrix
is k because we have k elements on the main diagonal each of which is one so this term
in parenthesis here is just n minus k the whole expression is sigma square u times n
minus k so we have proven that the expectation of sigma hat square u is equal to sigma square
u so sigma hat square u is an unbiased estimator of the error variance in our model.
All right are there any questions concerning this proof or anything else I have said?
No questions?
I don't see any sign then let's move on.
A simple exercise here prove now that the rank of m is equal to n minus k that's very easy
and if you want to convince yourself by sort of random experiments that what I have said
here is correct you can do the same thing in MATLAB you generate again as the previous exercises
n by k matrices x so regressor matrices which consist of any numbers random numbers
with of course more observations than columns and then just construct the matrix m as identity
matrix minus x x plus and verify that this matrix is symmetric and that it is idempotent
and that it satisfies the properties m x is equal to zero.
Okay recall we needed a estimate of sigma square u because we are interested in the covariance
matrix of the estimates beta hat we would like to know when we estimate the unknown parameter
beta in our regression model by the least squares estimator beta hat we would like to know which
standard errors the estimates have so that we are able to construct confidence intervals and for
instance check whether a parameter is significantly different from zero.
We have derived an expression for the covariance matrix of beta hat in the last lecture remember
this was just x prime x inverse c times sigma square u and now we have an estimator for sigma
square u this estimator gives us an estimate of the variance of the regression what is usually
given in regression output is however not this variance estimate but the estimate of the standard
error so it is the square root of our estimate which is reported in regression output we call
this square root sigma hat u or say sigma u hat we call this the standard error of regression
you read this often in econometrics literature and it is precisely this expression here the
square root of the estimated error variance the regression output usually reports this
sigma u hat not sigma u hat squared obviously when we have sigma u hat sigma u hat we can
easily compute sigma u hat squared by just squaring this why is this standard error of
regression reported well because sometimes we are interested in some type of confidence interval
for the dependent variable for instance when we forecast we would like to know what is the forecast
error we would like to know within which interval will probably the dependent variable the y variable
b and for these type of confidence intervals we would need the standard error of the regression
the confidence interval would be more or less the value of the random variable the value of x
beta hat plus minus two standard errors so plus minus two two standard errors of regression
this is what i write here if we know the x beta for some forecasting purpose already
so if we know the true beta and not the beta hat this is another problem then the y will not deviate
by more than plus minus two standard errors from x beta in approximately 95 percent of the samples
to be processed plus minus 1.96 standard errors of course but so approximately 95 percent of the
samples and that can basically be used or is most often used for forecasting purposes so that we can
construct confidence intervals for all forecasts after all this theoretical material let's go back
to our example equation or example regression where we have estimated the Keynesian consumption
function what i have done so far was that i explained to you the coefficient estimates
how they are derived namely as x prime x inverse the x prime beta and how the standard errors
are derived when you go back in my slides you see the same slide here with either the
coefficients highlighted or the standard errors highlighted what i want you to understand is the
meaning of all or almost all of the figures that are given in standard regression output because
mostly the same type of information is given in any statistics package which does linear regression
and so i will go through the the other numbers here too and now i explain to you what this number
is right s e is the standard error the standard error of regression we have just defined this
would be 0.33 in our regression here that's the square root of the estimated variance of the use
right this is this number here and as i say you can use it for constructing confidence intervals
for the dependent variable this is why it's called the standard error of the whole regression
meaning actually the standard error of the y variable which is part of the regression
now what does this tell us here well first recall that the regression is in logs so we have the log
of private consumption here and the log of income here this regression is in logs and since it is
logs the 33 here the 0.33 can be interpreted as a percentage so when we know that the standard error
can or the standard error is basically 33 percent of the level of consumption so not the logged
value of consumption but 33 percent of the level of consumption then this means that we know for
a given income consumption may fluctuate by plus or minus 66 percent with 95 probability
so the confidence interval is actually huge and this tells you already that this regression here
is not a particularly good regression actually you would usually not use it as a
as a basis for modeling a consumption function there are many problems in this regression which
i will partially comment on later on and partially i won't discuss that at all because these are time
serious econometrics methods matters which come up in later lectures but in fact the regression
is not very good even though we have such high t values here and zero probabilities for
coefficient being zero there are serious problems with this regression but that's sorry this is my
phone i have sorry for the disturbance for disturbing you here but let me just
turn off my phone which i forgot
okay done um so it's not a particularly good um uh regression um but obviously when you do
econometric berg you are confronted with not so good regressions much more often than with
a final regression which you think is perhaps reasonably good so you must learn to interpret
regression output also from regressions which are misspecified or not specified very well
which can be improved upon and this would certainly be a regression where you can improve
uh the uh the specification so there was a question coming in
would you mind explaining why it is plus minus 66 percent uh the standard error should be
33 percent uh so why isn't it plus minus 33 percent well precisely for the following reason
that as i have said here we know with 95 approximate confidence that y will be in the
range of plus minus two standard errors right and this is why i have 66 percent this is one standard
error but the 95 percent confidence interval is approximately uh two standard errors plus minus
two standard errors so it's approximately uh 66 percent okay
good okay that's thanks for the question yeah please um pose your questions is very good if
i get some feedback from you all right there's actually another problem here which i have noted
in the last line here the regression residuals are not normally distributed so they do not follow
normal distribution but we haven't discussed tests of the normal distribution yet or of the property
of having normal disturbances we haven't discussed tests of these yet so that's another problem in
this regression that we actually do not have normal error distribution and therefore the
approximation of plus minus two standard errors is actually inaccurate but leave that aside for the
time being since you have the data on log consumption and log income another exercise
here for you would be to verify the highlighted estimate of the standard error in MATLAB by just
computing by hand that you get this number if you run the regression not with pre-programmed
software but in MATLAB with your own matrix operations i see another question coming in
okay the question is can you please clarify again why this model is not
good only because the big 66 percent or another reason thanks um what why i said the model is not
good is indeed that the standard error is so be here we have not modeled the error process
so you cannot really see it immediately from the the statistics which are here but the error
process is not uncorrelated errors as we have supposed it to be in the classical linear
regression model and this causes rather huge standard errors here so it's more a matter of
experience here than that you can infer this directly from the regression output but the
rather high standard error here is a warning signal that the equation is not very well specified
okay now let us think about whether we can measure the goodness of the regressions
fit so what i'm aiming at now is for those of you who already know this and actually you should all
know is what is r squared and what is the r adjusted r squared which as you know are measures
of it so as you know y hat is x beta hat and y is y hat thus you had we had these relationships
already y hat obviously measures how much of the observed y we explain by the regression
and this fact gives rise to the idea that we can express in some standardized way how much of
y is explained by our regression and therefore have a measure of the goodness of the regression
of the goodness of the regressions fit right and this is what we wanted to do now in order to do
this we will suppose that the regressor matrix contains a constant term which usually is the
case right so that's not a very restrictive assumption remember a constant term i denoted
by a Greek letter uta so uta is just a column vector of ones and remember that the x matrix
is orthogonal to the residuals so when x contains a constant term then we know that uta prime
u hat is equal to zero so the vector of ones is orthogonal to the estimated residuals therefore
we know that the residuals all sum to zero that we also had already previously now let's look at
the minimized sum of squares that is to say the solution of the least squares problem this is as
you know u hat prime u hat by the way i think i didn't mention it i'm not sure i will mention it
later the sum of squared residuals you also find in this output here right so this is this number
here obviously sum of squared residuals so this is the u hat prime u hat of which i'm talking
right now the 20.29 okay u hat prime u hat is equal to u prime m u u are the true disturbances
u hat are the residuals and m is this idempotent matrix which we had used previously in the proof
of the theorem of unbiasedness of the variance estimator so that's the same thing as u prime m u
and we know this is the same thing as y prime m y this again exploits the fact that the m matrix
is orthogonal to x right we have had this also in the previous proof so y prime m y we may
also write as y prime y minus y prime x x plus y now using the definition of m m recall is i
minus x x plus so if we multiply the i the identity matrix with y prime y we just get y
prime y and here we get then the y prime y with x x plus plus in between x x plus is idempotent
so we can just double it and so y prime y is the same so this is the same thing as y prime y minus
y prime x x plus x x plus y um i said already this is the case because x x plus is idempotent
you could also say this is because we use the properties of the pseudoinversity of
the moor-penrose inversity because recall one of the properties of the moor-penrose inversity was
x plus x x plus is equal to x plus right this is one of the properties of the moor-penrose in
an inverse so um this thing here is exactly the transpose of this thing because x x plus is also
symmetric so we can write this the same thing as y prime y minus x x plus y prime so transposed
times x x plus y not transposed but x x plus y is the same thing as x beta hat because x plus y
is the same thing as beta hat this is the old estimate so this the same thing as y prime y
minus x beta hat prime x beta hat and x beta hat is of course just y hat right so we have here y
prime y minus y hat prime y hat so what we have arrived at is the sum of the squared residuals
is exactly the difference between the sum of the squared observations minus the sum of the
squared fitted observations of the squared explained observations or to write it in a different
way we can write the sum of the squared observations y prime y is decomposed into
the sum of the fitted observations the squared values of fitted observations
and the squared value of the residuals this is sometimes stated as the total variation
so the total variation is the variation in the observations is equal to the explained variation
plus the unexplained unexplained variation and perhaps you say well this seems rather natural
that this is the case it is actually not completely natural because the important message is here that
we do not have any mixed terms so we don't have any terms y hat prime u hat or u hat prime y hat
something like this we just have the quadratic terms but we do not have the cross product terms
the sum here because of those orthogonality properties
now another property which is perhaps important to note it's rather trivial is that the averages
of y and y hat are identical so if i first sum all the y values by pre-multiplying with yuta prime
with a vector of ones transposed this would be the sum of all the y values then obviously this
is equal to the sum of all the y hats plus the sum of all the u hats but the sum of all the u hats
is equal to zero so this is equal to the sum of all the y hats so this shows us already that these
two sums are identical and if i divide by the number of observations to get the means then i
know that y bar the mean of the observations is the same thing as y hat bar so the mean of the
fitted observation of the explained observations of the explained wise i should perhaps better say
now make use of this and subtract n times y bar square from equation 31 to make a
variance like thing out of that right this here would be basically the first moment or related
to the first non-central moment now also subtract the squared mean by subtracting n times y bar
squared from the left hand side here so we would get one oh yeah and then divide by n so we would
get one over n y prime y minus n y bar squared is the same thing as one over n y hat bar y hat
prime y hat minus n y hat squared y hat squared bar right because this thing here is equal to
this thing by virtue of what we found out here and here obviously i do not need to subtract
anything because the u hats have a mean of zero anyway we know that you that that yuta prime u
hat is equal to zero so now what i have here are the naive variance estimators which i had denoted by
small var hat small var hat or so the estimated variance of the y's is equal to the estimated
variance of the y hats plus the estimated variance of the u hats if i use as estimator for the
variance a biased estimator they mean an estimator where i divide by one over n everywhere rather
than by dividing through the degrees of freedom we have seen that actually we should estimate
the variance of the u hats by dividing through by n minus k here rather than by n only then would
this variance estimator be unbiased similarly here we would have to divide by n minus one because we
subtract the sample mean of the observations so that would be the appropriate degrees of freedom
and similarly here and so this is not a good variance estimator we use here but it is sort
of the most intuitive variance estimator which people sometimes use and i have my special notation
for this there was a question why i used this notation already earlier in the lecture and
perhaps now you understand it is a type of variance estimator which we will mostly not use
because it is biased but it is a variance estimator and this is not a completely bad
variance estimator because it is unbiased asymptotically so if n goes to infinity then the
difference between n minus k and n is negligible so asymptotically these estimates are right they
are what we later would call consistent but they are not unbiased and in terms of these biased
estimators we now know that the variance the estimated variance of the y is equal to the
estimated variance of the y hats plus the estimated variance of the u hats
so what we wrote here total variation is equal to explained variation plus unexplained variation
can now be written as total variance actually i should write total estimated variance is equal to
estimated explained variance and estimated unexplained variance so to the sum of the two
and this gives rise to what is called the coefficient of determination or r squared
this coefficient r squared is the share of the variance of the dependent variable which is
explained by the regression so it's the ratio of the estimated variance with this biased estimator
y hat over the variance of the estimated variance of the observations y
so this ratio here is r squared and in some sense it's a natural and intuitive measure
we use the explained variance as a ratio of total variance to describe how well our regression
fits the data and due to the fact that we can decompose total variance and explained
variance and unexplained variance both also all three in terms of this biased estimator
we can also write r squared as one minus the estimated variance of the u hats
divided by the estimated variance of the y from which we immediately see that r squared is always
positive or perhaps in extreme cases equal to zero but certainly not negative let's say always
positive because variances are always positive and it is smaller than one right so if we say
variances are always positive and estimated variances are always positive then from this
representation of r squared we see that r squared must be positive and from this representation here
we see that must be smaller than one because we have one minus something positive so smaller
so r squared is a scaled measure of fit which is always between zero and one and therefore we can
interpret r squared as a percentage we can say it is the percentage of the variance of the
observations which is explained by the regression because this is the explained variance here and
this is the total variance so that's the percentage of the regressions variance which is explained by
the regression and r squared gives us this percentage so for instance if r squared is 0.8
then we would say 80 percent of the variance of the observations is explained by this regression
this sounds rather good and attractive and i just have to warn you it is not quite as good as you
may think it is but at first sight it looks pretty good this is the r squared a useful
measure of fit but a measure of fit which has its weaknesses and we come to the weaknesses
okay here's the exercise very funny that r squared is between zero and one i've done this
just for you so you don't need to do the exercise anymore just make sure that you understand it
and then well yeah i think i will also do this here because it's perhaps important and the
solution is here anyway compute or let us compute now the r squared by hand so how would we do this
for our example regression where we know what the variance the estimated variance of the y would be
if we use this biased estimator because we have in the regression output information on the standard
deviation of the dependent variable right let's see where this is
here standard deviation of the dependent variable so this information is given in the
regression output it is roughly 1.09 right remember that 1.09 is the standard deviation
of the y's and this is given in the regression output now you have to know this standard deviation
is estimated here or computed with the unbiased estimator so dividing by n minus one not dividing
by n okay so what we have to do here is if we want to get the estimator which divides true by n
then we have to first square the standard deviation to get the variance so this is why i
square the value of 1.09 and then multiply by n minus one and divide by n since this variance
here has been estimated with dividing by n minus one i reverse this division by n minus once
by multiplying by n minus one and then then i divide through by n to get the biased estimator
bar hat of y if i do this well we have 187 observations as you know from the regression
output n minus one would be 186 so i would have to say this is equal to 186 over 187 times 1.09
squared 1.09 was the standard deviation of the dependent variable the squared is then the
variance of the dependent variable why do i have approximately equal here because the 1.09
is rounded right so it's not quite exact quite the exact number but otherwise i have nothing here
otherwise so computing this you will get a value of 1.1805 okay for the biased estimator of
the error variance u hat of the sorry of the residual variance u hat here so bar hat of u hat
we would do the same thing we take the standard error of regression i showed you already this
is very good n squared to get the sum of the squared residuals and then we multiply by n
minus k because we have actually divided by n minus k and divide by n to get this bias into
our estimate so we would have here since k is 285 this is n minus k divided by n which is 187
times 0.33 squared so that's our standard error here of regression squared if you compute this
this is 0.1085 so from this we now know the r squared is one minus the estimated variance
of the u hats with biased estimator and the estimated variance of the y divided by the
estimated variance of the y's also with a biased estimator so this is one minus with the ratio of
those two numbers two numbers which we have computed here and there and so this is 0.908
if you want to verify that and now let's go back and check from the regression output
here it is right so 0.908 this is the r squared and we have now reproduced the r squared from
other information in this regression object so actually you may say this r squared information
here is redundant yes indeed it is you could compute it already from other output which
you have here but obviously this is cumbersome and you may easily make mistakes with all those
n's and n minus ones and n minus k's and this stuff so the computer computes it right away for
you this is the r squared but now you know how the r squared is computed and how you can verify
that this is the true value of the r squared by going through the numbers in this example here
here's an exercise for you to compute some information which you do not have in the regression
output so compute y prime y and verify that y prime y is 13,322 just to get a feeling of how
to use regression output you should do this exercise there's no particular value in knowing
what y prime y is as i said it's not given in the regression output because there is no particular
value in it but you should think of a way of how you can retrieve the information on the sum of the
squared y's from the numbers you have in the regression output to get a feeling of how you
can use the regression output are there any questions so far
i don't see anything so now suppose we have estimated regression with as usual k regressors
and we have obtained an estimate beta hat which i index now with k to denote that this is an
estimate beta hat where we have used k regressors and we have also obtained residuals u hat and
again i indicate that we have used k regressors in finding these residuals by equipping them
here with a subscript k so this index here does not mean that i take the kth component of the beta
vector or the kth component of the u hat vector but rather i index the vector to indicate that
these are results from a regression with k regressors and now suppose that we rerun
the regression because we think perhaps we have not included all relevant regresses perhaps there
is a kth regressor and if this is an error here this should be the k plus one th regression not
the n plus one th regression sorry for that so suppose we re-estimate the regression adding a
k plus one th regressor and then we obtain an estimate beta hat k plus one and residuals
u hat k plus one or the sum of squared residuals u hat k plus one prime u hat k plus one
now what we know is that the residuals u hat k plus one prime u hat k plus one or the sum
of squared residuals from the regression with k plus one regresses is always smaller than the
sum of squared residuals from the regression with k regresses does anyone know or can explain
quickly why this is the case why is this here always less than or equal to the sum of the
residuals when we use just k regresses rather than k plus one regresses
any quick answer on that i don't want to go into the meeting to
let you give the answer but if somebody can type it in or just indicate that he's about to type it in
well if not then i mean you just have to read what i've written on the slide um
there's the answer good because we explain more well yeah hmm do we explain more
not really um because we use an additional regressor um um is actually uh the reason and
in more detail um you see it here in the new regression we can always make the residuals
as small as in u hat k prime u hat k by just choosing the old regression coefficients
and assigning zero to the new and added k plus first regressor right so if i just assign the new
regressor a value of zero then essentially i have the same result as before i have the same beta hat
values estimated so i get the same residual so i can always in a regression with one additional
regressor obtain a result which is at least as good in terms of the sum of the squared residuals
as the regression was with just the first k regresses but this would be the result if i
choose zero for the eight k plus first regressor regressor obviously there are many other choices
for this coefficient here and the least squares routine will with probability one actually
choose different value it will not choose zero uh this never happens it will choose some other
value which then is chosen because the residuals become even smaller with a different value than
zero so the residuals u hat k plus one or their squared sum must be less than or equal
than the residuals if we just use k regresses so one problem in uh practical econometrics
is that when we have a regression which has a certain goodness of fitment as measured by the
r squared we can always improve the quality or seemingly improve the quality of the regression
by just adding future uh further regresses by just adding more regressors regresses the more
regressors we add the higher will the r squared be the more variance will we explain and that is
obviously not useful because we may add just nonsense regresses which have no explanatory
potential whatsoever and still our square would rise so um this is one of the weaknesses of our
square look uh at this result here again our consumption function estimated with two regressors
constant and log y so the only economic regressor is actually uh log y here gives us an r squared of
0.9 0 8 9 0 9 okay now let's see what happens if we add a nonsense regressor to this regression
for instance i have generated uh random numbers uh here um so i have just generated random
numbers which have nothing to do with private consumption and i add this vector of 187
randomly chosen numbers to my regression first you see here a scatter diagram you see the random
numbers which i have generated somewhere distributed between minus three and plus three
and you see log private consumption here and you immediately see that there is no relationship
whatsoever between private consumption and my random numbers as was to be expected right so we
know this regressor here actually explains nothing can explain nothing because it has nothing to do
with private consumption so now i add this regressor to my regression rerun the regression
with three regressors regresses constant log income and random numbers what happens
well you see this here the coefficient which we estimate for the random numbers is um small 0.0256
its p value is 32 percent so it's actually not significantly different from zero that's exactly
what how it should be right so every researcher would now say um this regressor we don't need it
explains nothing it is probably equal to zero because the p value is 32 percent it's actually
quite close to zero in numerical value so explains nothing we just throw it out of the regression
but the r squared has increased right so now we have an r squared of 0.90858
so remember the 85 right uh here we had 80 it's not much by which the r squared has increased
um but it has increased suggesting that the goodness of fit of this regression here is better
than the goodness of fit of the previous regression but of course this is not the case
this is a nonsense regression as far as this regressor the random random numbers is concerned
and actually this superfluous regressor even makes the regression harder to explain
or makes it more difficult to explain by the regression the data have a look for instance
at the standard errors here the standard error for lny is estimated here as 0.0200
oh sorry um i have to go further 0014 there's a 14 here right at the end
and here there is one one so marginally uh the estimate has become less precise the standard
error is greater here it is greater here because we have introduced unnecessary
unnecessary noise into the regression this also spills over to the regression estimates
to the standard errors of the coefficients so this standard error here will also be a little
bigger than the standard error that we had uh in the previous regression so it's here 179485
179473 so it's a little uh smaller here and the same will be true for the standard
error of regression it's here 33116 it's here 331168 so all the standard errors have increased
because we have introduced a regressor which explains almost nothing but inserts additional
noise in the regression and makes our estimates less precise so these coefficients here are
slightly less trustworthy than the coefficients we had in the correctly specified regression
even though the r squared here is lower than the r squared r squared there now if i do not behave
as stupid as i have just behaved and do not use a regressor from which i already know that it
is useless because it's just random numbers which i have generated if i add some regressor which
possibly explains something and possibly does not you know then you may have many
candidates which you might also want to add to this regression equation for instance what could
be the case you could say i also add income interest rates here because interest rates also
generate income for instance or i add unemployment because unemployed people perhaps consume less
or i add wages or whatever i mean you may think of many possible regressors which you
add to a talk consumption function like this here and you will see whichever additional regressor
you add to this consumption function the r squared will increase
regardless of whether this regressor actually has any explanatory power so you may be misguided
if you just try to maximize your r squared this is the main message i want to convey here
it is not a useful strategy in empirical econometrics to try to maximize the r squared
or as i put it here r squared is not a good measure of fit unfortunately even though it
was quite nicely constructed and scaled to be between zero and one it is not a good and reliable
measure of fit so we should think about the possibility of modifying r squared such that
there is a penalty for increasing the number of regressors so if we increase the k then r square
should contain a penalty which penalizes us for having an additional regressor so that the measure
r square would only increase if the gain in explained variance is greater than the penalty
imposed on us for using an additional regressor where there are many possibilities to impose a
penalty and in practice we use only one which is embodied in the so-called adjusted r squared
the adjusted r squared follows a very intuitive idea the adjusted r squared
says well why actually have we bothered to define this r square here by virtue of these biased
estimators of variances i mean what why do we did we use these estimates or these estimators of which
we know that they are biased in finite samples why did we use them in the first place why didn't
we use the unbiased estimators so this is what the r bar squared does the bar denotes the adjusted
r squared we use the unbiased estimator of the error variance here and we use the unbiased
estimator of the variance of the observations here and say our r bar squared is one minus this ratio
which is the most natural thing actually to do once we have understood how r squared is constructed
this is obviously different from the r squared right because here we divide by n and n so these
two terms actually cancel whereas here we divide by n minus k and by n minus one so the r squared
r bar squared is equal to the r squared only in the special case where we have just one regressor
they're the two coincide but for other for other numbers of regressors the r bar squared is of
course different from the r squared and what does this mean that is different where we see this
already when the number of regressors increases right then the numerator here decreases
the numerator here decreases the denominator sorry the yeah the denominator stays constant
because this is not dependent on the k right so just the numerator here decreases so we
uh sorry i have to be careful the numerator in this this ratio here decreases right which means
that this thing here increases so the numerator of this ratio of the whole ratio here
increases becomes greater which means that the r squared bar squared becomes smaller right sorry
i think i first bespoke but now i formulated it the right way the r bar squared will
be smaller than the r squared for regressions in which the number of regressors exceeds two
and here it is this is the adjusted r squared so this is r bar squared and you see it is smaller
slightly smaller 0.90 uh 0.907596 okay so basically 908 um here uh but a little
smaller than uh the r squared if we compare now with the other regression where we had
the random regressor in there the superfluous nonsense regressor regressor we now see that
the adjusted r squared actually decreases whereas the r squared has increased so here we have 0.591
at the end and here we have 0.596 so uh the adjusted r squared has decreased a little bit
due to this uh unnecessary regressor here and tells us it was not a good idea to include this
superfluous regressor in the regression whereas the r squared would have conveyed exactly the
opposite message so why do we have the r squared at all in the regression output well the
um unfortunate thing about r bar squared is that it can be outside the interval 0 1 so it is not
scaled anymore to be within 0 and 1 and therefore it cannot be uh considered a percentage
of the variance of the observations that we have explained by our uh regression to see this in an
example look here at the regression again where i have taken out log y from the list of regressors
and i just run a regression where i regress log private consumption on a constant and on this
nonsensical random number regressor well obviously this regression cannot explain much right well
there's a constant but nothing else is explained and you see that the adjusted r squared here is
actually negative so what does this mean i mean we cannot explain this anymore as a percentage
of explained variance the percentage of explained variance is actually zero but a negative value
would not make sense now this shows to you already that r bar squared can be outside the interval 0 1
by taking on negative values um an exercise for you here is that you think about the question
whether r bar squared can actually be greater than one and then please give a proof uh for
whatever you assert there
yeah again an exercise for MATLAB if you like these type of exercises just take the benchmark
regression again and verify manually the regression outputs for r squared and r squared
now this is one problem with r squared or with r bar squared so both of them have their own
problems r squared is misleading when we add new regressors and r bar squared is not easy to
interpret because it's not a percentage so both of them are not ideal but there is another problem
which is perhaps even more confusing at least when you start with econometrics and this is
particularly present pertinent in time series models once with time dimension so let's look
at a model where we have time series data so y t has the index t because this now
indicates the teeth observation the observation in period t so y here is not a vector but y shall be
the observation real number in period t and suppose now the model is that period t observation
is explained by a constant b not plus a coefficient b one times the time trend t so this becomes
greater and greater as time runs plus b two times some other regressor x t plus a disturbance u t
suppose we have capital t observations so small t is an index which goes from one to capital t
and this relationship here is supposed to hold for all values of t so for all values of t between one
and capital t so for instance it also holds for period t minus one observation for period t minus
one observation we have y t minus one is equal to b zero plus b one times t minus one now because
the trend term is now nowhere time has not progressed as much as it had progressed here
so we use t minus one here plus b two x x t minus one plus the disturbance of period t minus one
so this is the model equation written for periods t this is the same model equation written for
period t minus one and what we can now do with time series data and remember our example equation
is time series no sorry this cross section data but we could also do it for time series data and
what we can do with time series data is that we just take the first difference here and in this
case we would get the change in y when going from period t minus one to period t is of course the
difference between the two right hand sides so b naught minus b naught is zero that just cancels
okay b one times t minus b one times t minus one is b one times t minus t minus one
well this is just b one b two x t minus b two x t minus one is b two times delta x t
plus it disturbs nu t nu t which is the difference between ut and ut minus one
okay nu t is ut minus ut minus one now the r squares from estimating 33 and 35 can be very
different i give you here an example of time series data because as i said the other example
was in cross section and that wouldn't make sense to compute the difference over time
but these here are time series data for stock prices in the united states that's the blue line
here that's the standard and poorest index from 1955 to i think i had 2000 yeah there it is 2007
right so that's a time series of the value of a representative stock market market portfolio in
the united states and this here is what we call total factor productivity and you probably have
heard of total product factor productivity of the concept of tfp in microeconomics lectures
either in undergraduates or now in the masters program total factor productivity is a measure
of the technological knowledge that is available in an economy it is typically a trending variable
because technological knowledge advances over time and we are often more interested in the
change in tfp so in new developments new technological developments new inventions
for instance which become available and give rise to new production possibilities
so this here is tfp there are methods to construct tfp we don't need to bother with
that just believe me that i have constructed tfp here in sort of state of the art methods
and we have these two time series here now there is one thing which is perhaps surprising when we
look at the data here because in general one would think that when there are innovations
technological developments which open up new production possibilities then this also opens up
profit opportunities and stock prices react to profit opportunities so actually we would perhaps
expect that stock prices and total factor productivity co-move that they move together
in this diagram it doesn't really look like that right i mean for some time perhaps tfp and stock
prices move together but then you see here beginning with the 1970s stock prices move
completely in the opposite direction as tfp while tfp growth was perhaps a little lower here than
it had been in the 1950s and in the 1960s actually actually there's also almost something like a
plateau phase here but except for the oil price shock which we see here tfp doesn't go down
right whereas stock prices go down over many many years actually in the united states
and then they start to rise again and much more strongly actually than the growth in tfp
and they rise even stronger than in the 1990s which were very bullish years and almost make
up for all the loss in value of stock market portfolio that we had over the 1970s and early
80s okay so here we are almost on the same level that we would have had if a long-run trend would
have continued for the stock price not quite but almost okay and then there's another stock market
crash and then it goes up and i discontinued the time series now in 2008 okay um just let's take
this these time series data and run a regression of stock prices on a constant on a linear trend
t and on tfp what kind of results do we get well bizarre results we would say we have 210
observations um here the constant is uh well has not negative coefficient or this is unimportant
this best tells us that stock prices here are below the tfp okay another actually this is not
true because uh well this is the scale for stock prices yeah it's below tfp right so that's the
scale for stock prices so the log here is like minus six point something and this is the scale
for tfp the log is minus one point or minus two point something depending on where you are
so obviously uh stock prices are lower in logarithms than tfp but this is a question
of scaling so this is why we have a negative coefficient here which is highly significant
then there is the trend and the trend is positive 0.008 no doubt actually about the significance of
this trend with a t statistic of 3.5 and then we have tfp interestingly with negative coefficient
so that would be very hard to interpret economically because this would say
the higher the level of technological knowledge in an economy is uh the lower stock prices this
doesn't really make any sense at all right so apparently there's something very badly specified
in this regression here another example of value regression even worse than the consumption of
regression by the way but that's not important now we are not now talking about um you know tfp
and what does tell us about uh stock prices it's just an illustrative example about weaknesses of
r squared you see the values of r squared and adjusted r squared here it's 24 here and it's
23.7 here okay now let's look what happens when we take the first differences i take the first
difference of the stock price and the first difference of tfp from period to period right
and then i regress on a constant and on uh the change of tf that is exactly what i should do
right the difference in stock prices delta yt here would be regressed on a constant b1
and a coefficient for the regressor b2 which is also given in difference from then
be aware that the constant which we estimate here b1 is not the same as the constant which we would
have estimated if we had estimated the regression levels so we cannot compare the two constants which
we have estimated because this b1 here corresponds actually to the coefficient of the trend uh so
this is what is left over of the trend right and the b2 is of course the same as the b2 here for
the regressor x so um where am i this is the regression um this is the coefficient for
the trend 0.008 it becomes 0.0005 in this regression here so it's not quite the same
estimate but in terms of orders of magnitude it's at least closer to it than uh the to then
to the c here okay and then there is the change of uh tfp which now has a positive coefficient
right so completely different from the coefficient that we had estimated before we had this one was
negative so in some way this makes more sense uh because now an innovation a new invention for
instance would drive up stock prices and actually the coefficient is significant as you see here
but look at what happened to r squared while this equation here actually makes slightly more sense
than the equation we've had before the r squared is much much lower and the adjusted r squared also
so here in terms of differences we would only explain like 3.2 or 3.7 of the variance something
like this whereas here we had explained seemingly explained 25% or 24% of the variance
so this is what i mean when i say that uh r squared or r bar squared are problematic
measures of fit because they are not invariant to linear transformations of a model
um if you take r squared or r bar squared at face value as measures of fit you might be completely
misguided by saying well this equation here is a much better equation than this equation here
which is not true at all actually this coefficient here makes more economic sense
than this coefficient here but true the r squared here is much much lower than the r squared is this
regression so please do never try to judge the quality of a regression by just comparing
r squares in particular when you take differences of data it is almost always the case that the r
square will go down dramatically and this has nothing to do with that equation differences
is worse than the equation levels often this is actually the case that they are approximately of
the same explanatory value we have just basically taken out constants so it doesn't really matter
much that we estimate either levels or differences but r squared is completely misled
by that and may give you misjudgments so when you use r squared or the adjusted r squared
you may use them as indicators for two rivaling specifications in terms of differences
that works reasonably well or you may use this for two rivaling specifications in terms of levels
so by introducing other level regressors here exchanging something that that may sometimes be
useful to look at r squared here don't overestimate the informative content of r squared but some
informative content there is but never compare r squared between levels estimation and differences
for growth rates estimation that would give completely wrong impressions about the explanatory
power of r squared okay so the r squared r squares are very different but as we see here the
coefficient estimates are also quite different even though actually this coefficient and this
coefficient should be approximately equal to this coefficient and that coefficient in particular
with respect to tfp this is not the case and this is one indication that well this is not a good
specification here and we should improve on the specification so some indication with time series
data of whether you have found a good specification would also be that when you move from levels to
differences the coefficients don't change by much and here they do change by much so the model is
not good which we estimate as i have already said so the difference in the coefficient
estimates suggests that the regression is misspecified does not capture the true model
why this is so i cannot explain here there are tedious issues they are related to time series
the behavior of time series this is a completely different chapter in
econometrics and i will give a lecture on time series econometrics next semester if you're
invited to to attend this lecture if you take an interest in that but we can't do it in this lecture
here there are some generalizations of r squared to logistic regressions which we may cover at the
end of this lecture so i give you a reference here nager kerke has a note on a general definition of
the coefficient of econometrics in biometric that's it for today as far as the webinar session is
concerned i will stop the recording now and