Thank you.
Hello and welcome to Estimation and Inference, the second lecture of this week.
May I just ask you if you can hear me well, because I received a message that my loudspeaker
doesn't function or doesn't function well.
Is there any?
So let me ask you this way.
Can you hear me?
Then please raise your hand.
Okay.
That seems to work.
Thank you very much.
Will, do you have any questions which relate to yesterday's lecture?
Again, if you do, please raise your hand.
This is apparently not the case.
You remember that we left off at this page, which derived the following result.
You see it again on the next slide.
The property which I have proven yesterday, the mean squared error of any estimator W
can be decomposed into the variance of this estimator and the squared bias of this estimator.
Or if you want to phrase it a little differently, you can say the mean squared error can be decomposed
into the second moment, the second central moment of this estimator and the square of a measure
which relates to the first moment because the bias relates to the expected value of the estimator.
So this is related to first moment and the variance is, of course, related to second moment.
It is actually the second central moment of the estimator W.
This is important because when we choose an estimator for a particular purpose,
then we often face a trade-off.
I should perhaps say when we have a choice between different estimators for a particular purpose,
then we may face a trade-off in the sense that some of those estimators have a larger bias
and others have a larger variance.
For instance, if you have just two possible estimators for which you want to choose,
it may be the case that the first estimator has small bias, perhaps is even unbiased,
but has a large variance, which means that while the estimator is unbiased,
it is quite likely that you receive estimates which are far off the true value,
either much greater or much smaller than true value.
On average, these errors compensate for each other.
So the mean of the estimator is the true value if it is an unbiased estimator
or perhaps it's just a small bias, but the variance of the estimator can be big.
And the second estimator can be just the opposite.
It can be that this is an estimator which has a substantial bias, but it has a small variance
or an estimator which has a small bias, whereas the first estimator was completely unbiased.
So in terms of bias, the second estimator is worse, but the variance of the estimator is smaller.
The variance of the second estimator is smaller than the variance of the first estimator.
So you have to make up your mind, then which estimator would you prefer?
If in terms of these two criteria, there is no clear ranking available,
how you would make your choice.
And one possibility then is to resort to the mean squared error and take this as the criterion,
which makes you choose a particular estimator.
So you may say, will I just take the estimator with the lowest mean squared error,
knowing that both the variance of the estimator and the bias of the estimator contribute to the mean squared error.
So it would be a reasonable choice to say, if you don't have any other guideline,
that you take the estimator with the lowest mean squared error as your preferred choice in your empirical investigation.
It is often also the case that the estimator depends on some parametric choice you have to make.
So you can basically vary the estimator by perhaps varying a particular parameter.
Then it is often the case that those two components of MSE are not independent of each other.
So when you change the estimator in such a way that you reduce the variance,
it may easily be the case that the bias increases or vice versa.
So variance and bias of an estimator are typically not independent of each other.
And changing one often involves also changing the other and often actually the opposite direction.
So you should be aware of that.
The property of unbiasedness of an estimator was a finite sample property,
and we now move to asymptotic or large sample properties of estimators.
So whereas so far we have studied a property for a given sample size, let's say n and n was fixed,
we now study the question what happens if we were able to increase the sample size
and making the sample larger and larger and even approaching an infinite sample size.
The key concept in this setting of asymptotic properties is the concept of consistency.
And here's the definition in the green box.
Let wn be an estimator of some parameter theta based on a sample of n random variables.
So we have some sample y1 up to yn.
So basically we draw n times from the population.
So wn is an estimator with size n fixed.
We call wn a consistent estimator of theta.
If it is true that this estimator here converges to the true value in a particular sense,
which I'm now going to explain.
I would first like to explain what do I mean by converge?
By converge I mean that actually not the estimator converges,
the sequence of estimators converges.
The sequence of estimators wn can think of wn as depicting a sequence with increasing n.
So one estimator would be w with 100 observations.
The next would be w with 101 observations.
And another estimator is w with 102 observations and so forth.
You would have a sequence of estimators.
When we speak of wn converging to something,
then we actually mean the sequence of estimators converges to a certain limit.
Now, convergence of random variables is somewhat more complicated than convergence
when you speak about real numbers.
I introduce you here to a particular type of convergence,
which you may or may not yet have encountered in undergraduate studies,
which is called plim convergence, the probability limit.
And this plim convergence is defined as follows.
The probability for the event that the estimator deviates from the true value of theta
by more than a chosen number epsilon,
this probability shall convert to zero as the sample size n goes to infinity.
And actually, it shall do so for every epsilon greater than zero.
So essentially, we think of epsilon of some small number.
And regardless how small you choose this number,
it shall be true that for every epsilon, as small as you may think of any positive number,
it is true that the probability for the event that the estimator deviates from theta
by more than just this tiny little number epsilon that this probability is almost zero.
It converges to zero as n goes to infinity.
So this is a somewhat complicated definition of convergence,
because we do not write typically that wn converges to something or converges to theta,
since it is just the probability of wn deviating from the true value, which converges to something.
And this is why the shorthand notation does not say the limits of wn is equal to theta,
so the limit of wn is equal to theta. That would be wrong to say,
but rather it is the probability limit which converges to zero or the probability limit of wn is theta.
If it is true that the probability of this event, which I've described, converges to zero as n goes to infinity.
In this case, we say that theta is the probability limit for wn.
And any estimator wn, which converges in this plimps sense to theta, is said to be a consistent estimator.
Obviously, an estimator which is not consistent for theta is then called inconsistent.
Are there any questions for this consistency issue, because that's important that you understand it well?
Yes, please raise your hand.
Okay.
Now, this is just a graphical illustration.
This is the distribution of a sequence of estimators for different sample sizes.
So essentially, it's the same estimator except for the fact that the sample size is different.
And each of these three estimators, which I show you in the graphical illustration, tries to estimate the same true parameter theta.
We know since this is done with artificial data what the true parameter theta is, which we want to estimate.
And then we can, just in the computer, simulate the distribution of the estimator with random numbers.
And then it turns out, for instance, that if we have just four observations to estimate the parameter theta, then our estimates are distributed according to the PDF, this red PDF, which I outlined here.
This red PDF, which is actually a histogram. But think of it as a PDF. Actually, you can theoretically derive this also as a PDF if you derive the distribution of the random variable of the estimator.
But you can think of it also as the distribution of the histogram of all the estimates in some kind of Monte Carlo study.
So this kind of distribution would be such that, well, okay, the mean is probably approximately theta, the mean of this distribution here.
But you see that the variance is big, and you can basically estimate anything between zero and some really big value, which lies far apart from theta.
Now, when you use the same estimator with a higher number of observations, larger number of observations, like n is equal to 16, then you get this kind of distribution here.
And you see this kind of distribution is, again, probably centered on theta, or not even quite, I would say by eyeball econometrics.
But it has a smaller variance, very clearly a smaller variance than the red estimator, either n equal to 4 estimator has had.
And when we have n equal to 40, then you see that the variance is even smaller because then we have the blue PDF here.
You see that the estimates are much more centered around theta than they were for n equal to 16 or even n equal to 4.
Note that for this particular estimator, it is not always true that the modus of the PDF, so the maximum of the PDF is the expected value, by far not.
So these are screwed distributions where the modus of the distribution lies to the left of the mean of the distribution.
But the modus comes closer to the mean of the distribution as the sample size increases.
Let me point out what the difference is between unbiasedness and consistency. Here's the first approach to understanding that.
Unbiasedness, as I have pointed out, is a finite sample property.
So when we prove unbiasedness, then this property is true for any finite sample size, whereas consistency is an infinite sample property.
So the consistency is valid only when we assume that we have so many observations that the size of the sample goes to infinity.
And only in the limit, asymptotically, is the consistency result true.
For any sample size, which is just large but not infinite, as is of course always the case, we never have a sample size which is infinite.
At the maximum we have really big samples, but they are all finite.
However, for large samples, the consistency result holds approximately.
And how well this result is approximated by the asymptotic result depends of course on the size of the sample and on the particular property we are trying to show asymptotic.
The intuition for unbiasedness and consistency is perhaps even more instructing when you try to understand on an intuitive basis what unbiasedness means, or consistency means.
Unbiasedness is basically the property which you would establish by saying if we can observe many different samples of the same finite size,
would then the resulting distribution of the estimates have a mean equal to the true value.
So you think of one population you have, and you draw repeatedly samples of the same size.
For each sample which you draw, you can estimate a certain parameter.
And unbiasedness would mean that all the estimates which are associated with different samples you draw from the population,
all the estimates should be distributed symmetrically around the true value. That would be unbiasedness.
So the idea is here many, many different samples which you draw, perhaps even infinitely many samples which you draw but each sample with a finite size, and then look at the distribution of the estimates associated with each sample.
Whereas for consistency, you do not think of repeated sampling, but rather you ask if we could make our sample size as large as we want.
Is it then true that our estimator gets arbitrarily close to the true value?
Now what is the relationship between unbiasedness and consistency?
Note that an estimator can be biased and yet be consistent.
And an example of that you have certainly encountered in undergraduate studies. For instance, when you estimate the variance of some sample using this formula here,
where you subtract the sample mean from each of the observations, then square this term and take the arithmetic mean of all those squared terms.
And you know that this is an estimator of the variance which is biased because you divide here by n rather than dividing by n minus 1.
Dividing by n minus 1 would deliver an unbiased estimator. Dividing by n is a biased estimator. And the reason why dividing by n minus 1 is correct.
You have probably also heard in econometrics, in undergraduate econometrics or undergraduate statistics, the reason is basically that we have made use of one degree of freedom already by estimating the sample mean here, which goes into this formula.
So this is this estimator here is biased, but it is consistent because when n becomes large, then it doesn't matter whether you divide by n or by n minus 1.
This difference vanishes asymptotically. So that's an example of an estimator which is biased and consistent and probably you're aware of this fact.
However, and this is perhaps less well known, it is also possible that you have an unbiased estimator. So it's unbiased in all samples, but it's inconsistent.
And an example of such an estimator would be an estimator for say the sample mean, which I call here new tool, right? Or the estimator I call new tool for some random variable y, where I would construct the estimator by saying, well, I just take the average of the first random variable which I draw and the last random.
So I exploit for my estimate just y1 and yn. I add the two up and then I take half of that, right? This is an estimator of the mean of a distribution. There's no question about that.
It is an unbiased estimator because the expected value of y1 is mu and the expected value of yn is mu. So in terms of expected values, the value of this parentheses here is 2 times mu multiplied by 0.5 gives exactly mu.
But the variance of this estimator remains greater than 0, even asymptotically, right? So since we discard all the information of random variables, y2, y3, y4, and so forth, up to yn minus 1, we do not benefit from increasing the sample size.
We still just exploit two observations, the first and the last random variable. And therefore, the variance of the estimator here remains constant, does not shrink so that the distribution of the estimates does not asymptotically concentrate on the true value.
So this estimator would be inconsistent, even though it is unbiased for all sample sizes.
What is true, but this is almost tautological, what is true is the fact that if we have an unbiased estimator, which converges in probability to some limit, then this estimator is always consistent.
And in that sense, you may say that unbiasedness implies consistency under the additional assumption that the estimator has a plan.
And if unbiasedness implies consistency, you may say that unbiasedness is the stronger property and consistency is the weaker property because consistency certainly does not imply unbiasedness, right?
But it requires the additional assumption that the unbiased estimator converges to a plim. Otherwise, it would not be possible to deduce that the unbiased estimator is also consistent.
Now, let's look again at the most primitive estimator we will talk about in our setting here, namely the sample B.
So take some random variable and consider just the sample average as the estimator for the population, right?
So our estimator w in this case would be just y and bar.
Y and bar I have already used, but here I define it again. It's one over n times the sum over all random variables yi.
That's an estimator, right? This is just the formula which describes us how we deal with the observations, but this is all random variable.
w is a random variable, y and bar is a random variable, the yi's are still random variables, so this is an estimator. It's not the estimate.
Now, there is an important and well-known property which holds for the sample average, and this is expressed by the law of large numbers.
Suppose that the random variables y1, y2, and yn are independent and identically distributed random variables with mean mu.
Of course, if they all have the same distribution, then they all have the same mean, right? And we call this mean mu.
If this is the case that all those yi variables, all those random variables here are independent and identically distributed random variables,
then it is the case that the plim of y and bar is equal to mu.
So the law of large number tells us the sample average converges to the true value, which is an important property and quite a useful property,
because we now know that we can estimate the sample average of any given sample, arbitrarily well, provided we have arbitrarily many observations.
So if we are interested in estimating the population mean of any distribution, then we can get arbitrarily close to the true value with a sufficiently large sample. This is what the law of large numbers says.
The important feature of the important property of this law of large number is that we do not have to assume anything about the distribution for which the sample is drawn, right?
We just say that the random variables y1, y2, and yn are independent. They have the same distribution, but we do not require that they come from any particular distribution.
They can come from whatever distribution they come from, as long as they all come from the same distribution, we still have this result that the plim of the sample mean, the sample average is equal to mu, is equal to the true parameter which we want to estimate.
We say therefore that the law of large number is a non-parametric theorem. Non-parametric means here that we do not use any kind of parametric assumption about the distribution which underlies the random variables in this theorem.
The law of large numbers is one of two well-known non-parametric theorems in basic statistics. The other one is the so-called central limit theorem.
So this is also non-parametric, which means that we do not make any particular assumption about the distribution of input random variables, y1 up to yn in the central limit theorem.
However, unlike the law of large numbers, the central limit theorem even makes or delivers a result for the distribution of the output variable even though we have not inputted any kind of distribution for the y variables.
So the central limit theorem says that if we have a random sample of independently and identically distributed random variables, y1 up to yn,
and these of course have some population mean which we call mu and some finite variance sigma square, but we make no assumption about the distribution of the y's except that we have to assume that all come from the same distribution.
If this is true that we have such a random sample from any given distribution, then the standardized variables zn, which is defined as the square root of n times yn minus mu over sigma,
has asymptotically a standard normal distribution.
So this transform variable zn converges to a standard normal distribution asymptotically.
That's very important and actually surprising result. Nowhere in this central limit theorem have we assumed anything about the normal distribution, but we get the normal distribution as a result and even standard normal distribution as a result.
Now you may ask where does this factor square root of n come from, right? If you look at the central limit theorem here again, you see that if this square root of n were not there, then you would immediately recognize this kind of standardization which we have already talked about in previous lectures.
So zn is defined as yn minus mu over a standard deviation. So you may ask, actually, why do we multiply by the square root of n? The answer for that is that you are mistaken, and I actually phrased it wrongly just a minute ago when I said divided by its standard deviation because sigma is not the standard deviation of n.
Rather, sigma is the standard deviation of each of the yi's here, y1 and y2 and up to yn. For these random variables, sigma is the standard deviation because the variance is sigma squared.
But for yn bar, sigma is not the standard deviation. Rather, sigma over the square root of n is the standard deviation.
Well, if you divide by sigma divided by square root of n, that's the same thing as multiplying by square root of n and dividing by sigma. This is where it comes from.
So everything's all right. Everything is done in the way we learned it so far, this type of standardization. It just happens to be the case that the standard deviation of yn bar is sigma over square root of n.
And then dividing yn bar minus mu by this term is the same thing as multiplying with square root of n over sigma.
So why is it the case or how can you see that the standard deviation of yn bar of the sample average is sigma over square root of n?
We've certainly done this in other random statistics too, but just for convenience, I repeat it here.
In order to understand it or understand it again, start out with a variance of yn bar. We can easily compute the variance of yn bar.
Well, how do we do this? First thing is that we plug in the formula for yn bar into this variance here, right?
yn bar is equal to 1 over n times the sum over all of the yi's.
Then we know 1 over n is a constant factor, so we can factor this out of the variance expression.
We know a constant factor being pulled out of the expression results in this factor squared, so it will be 1 over n squared times the variance of the sum now of all the yi's.
Well, the yi's are independent. We have assumed this in the central limit theorem and I assumed here again, so there are no covariances.
So this lets us know that the variance of the sum of the yi's is the same thing as the sum of the variances of the yi's.
And since the yi's all come from the same distribution, the variance is always the same. I mean, it is sigma squared, right?
So each variance of yi is equal to sigma squared and we have n terms to add up here, so the value of the sum is actually n times sigma squared.
Which means that we get 1 over n squared times the value of the sum, which is n times sigma squared.
So obviously the squared here and there cancel and we are left with sigma squared over n.
So the variance of the sample average is sigma squared over n, and therefore the standard deviation of the sample average, which I denote by sigma index y bar, is just the square root of that, which is sigma over the square root of n.
And this shows that we have divided in the central limit theory by the correct standard deviation of the random variable, which we are looking at, namely the sample average.
So what we know is that yn bar has a mean of mu and a variance of sigma squared n.
And since zn is asymptotically standard normal, we can then immediately infer the asymptotic distribution of yn bar, right?
Actually it would be more correct to say the asymptotic distribution of square root of n times yn bar, because the asymptotic distribution of yn bar is of course zero.
The standard deviation of yn bar goes to zero asymptotically, because the standard deviation of yn bar is sigma over the square root of n, and when n goes to infinity, the standard deviation of yn bar will go to zero.
So yn bar does not have a distribution asymptotically anymore.
yn bar rather shrinks and converges to a constant, and this constant is then mu.
So when I write here, we can immediately infer the asymptotic distribution of yn bar. This is slightly incorrect. I should better have written this asymptotic distribution of square root of n times yn bar.
Because only if we pull the sample average part, so if we stretch it by the factor square root of n, then we arrive at a well-defined asymptotic distribution, which actually has a positive variance. If we don't do it, the variance would converge to zero.
Still, the notation and often the usage of the language in literature is that we say the sample average yn bar converges to a normal distribution, and here we
encounter a second convergence concept for random variables. Namely, this is a convergence in distribution. In this case, we do not have this plim convergence.
We do not speak about the probability of certain events happening or not happening, but the kind of distribution or the kind of convergence which we establish in the central limit theorem is a convergence which occurs on the level of the PDF or the CDF.
So it is possible in the central limit theorem to show that the PDF of the sample mean pulled apart by the factor square root of n converges to the PDF of a normal distribution.
And this is why we write this form of convergence by this arrow with a superscript D for distribution. So this means convergence in distribution. Yn bar converges in distribution to a normal distribution with mean mu and with asymptotic variance sigma square over m.
Or sometimes, or actually equivalently, we say yn bar is asymptotically normally distributed, and then the notation is such a tilde with a superscript A for asymptotically. Yn bar is asymptotically distributed as a normally distributed variable with mean mu and asymptotic variance sigma square over n.
Quite obviously, the asymptotic distribution of yn bar is normal, but it is not standard normal, right? Because standard normal would mean that expected value is 0 at the variance as well, which is not the case here.
It's just this transformed variable Z, which has a standard normal distribution. Yn bar does not have standard normal distribution.
This central limit theorem underlines why the normal distribution is so important in statistics. Because even if we have no idea about the distribution of our random variables of our population, it can be any distribution, and there are many, many distributions around.
Then the central limit theorem shows us only of how certain estimators gain an asymptotically normal distribution. So the normal distribution is for some important estimators, the limiting case, and this makes the normal distribution so important.
The necessary condition for this central limit theorem is, of course, that the yi's are independently and identically distributed.
Not surprisingly, many people have worked about the question whether this assumption of identically and independently distributed variables can be relaxed, and in fact it can.
There are other versions of central limit theorems which allow for weak versions of dependence of the yi variable and weak differences in the distribution so that they don't need to be exactly identically distributed.
But that is very technical stuff which I don't cover here. We just stay with the core formulation of the central limit theorem, and this is that the yi variables are iid.
These things are really important. A law of large numbers and central limit theorem. I've covered this now or repeated it because I'm sure you have had this in undergraduate statistics. Are there any questions left over?
Please raise your hand if you have any question.
Okay, if this is not the case, then let's move on to confidence in our answer.
So far we have talked about what we call a point estimate. So the estimator W, or whatever its properties are, will give us a point estimate in the sense that we get some number, and this number is our estimate.
And we apply the estimator to real data.
But this point estimate doesn't provide us with any information about the fact how close the estimate is to the true population parameter.
And that is, of course, very important to know.
We do have information about the accuracy of this point estimate because we can compute the standard deviation of the estimator.
And we can just look at the second moments of the estimator and then compute the standard deviation as the square root of the variance of the estimator.
And this gives us an idea about the accuracy of the estimate, but it still doesn't tell us how far the estimate is from the true value which we wanted estimate.
So in order to overcome this difficulty, people have thought of the idea of constructing the confidence interval, which I abbreviate CI.
We start out here with the easiest case where we have the population normal distribution and where we assume somewhat unrealistically that we know the variance of our random variables, of the random variables we sample.
So to make it even easier, in the first step, we not only assume that we know the variance, but we also assume that the variance is equal to 1.
So suppose that we have a population and the population has normal distribution with mean u and with variance 1, and we know that the variance is 1.
As an exercise for you, show please that in this case, the sample average yn bar has normal distribution with expected value mu and variance 1 over n, even in finite samples.
So it's clear that this is the case asymptotically because we could apply the central limit theorem. But in this exercise, which is actually easy,
you shall show please that even in finite samples, the distribution of yn bar of the sample average is normal with mean u and this variance here.
So I assume that you have proven that and use this result now. If I standardize now the random variable yn bar by subtracting its mean and dividing through by the standard deviation, so the square root of the variance, then I know that this variable has a standard normal distribution, n of 0 and 1.
And using this fact, I can infer that the random variable, the standardized random variable, which I write down here again,
lies with a probability of 95% between the values negative 1.96 and positive 1.96.
But these values, these critical values here, negative 1.96, positive 1.96, I have taken from tabulated normal distribution, you know that negative 1.96 is the critical value for which we know that values even more negative than negative 1.96 have a probability under the normal distribution of 2.5%.
And more extreme values than 1.96 of the standardized variable here have a probability of 2.5%. So that together makes 5%, which implies that the probability for a standardized random variable being between these two critical values is exactly 0.95.
And then what can we do? Well, we just rearrange these inequalities here, because any kind of mathematical operation which we've applied to all three parts of this inequality here, if they are correctly done, do not change the probability.
So first thing is that I multiply the whole inequality by 1 over square root of n. So the standard deviation vanishes here in the middle term, and rather I have 1 over square root of n here and 1 over square root of n here.
I can do this because 1 over square root of n is always positive, so the inequality signs are the same.
Next thing I do is that I multiply the whole inequality by negative 1. Well, in this case, I would then get, well, minus times minus is plus, right? So I get 1.96 divided by the square root of n.
The inequality sign is reversed by multiplying by negative number is greater than mu minus y n bar, and this is greater than negative 1.96 over the square root of n.
And still the probability is the same, right? It's 0.95 because nothing has changed here. These are equivalent ways of stating the particular outcome of the random variable y n bar.
Last step I do is that I add y n bar to the inequalities, so I would get y n minus 1.96 divided by square root of n is less than mu is less than y n plus 1.96 divided by square root of n, and this probability is 0.95.
So now I can say with a probability of 95 percent, mu, the true parameter, lies between y n minus 1.96 over square root of n and y n plus 1.96 over square root of n.
This, however, is a statement done with respect to the random variables, and we will still have to see how we deal with it when we don't talk about the random variables, but talk about real numbers.
These are still the random variables, and actually in terms of the random variables, the statement is not yet so helpful because we don't really know which value the random variables take on when they realize.
Okay, this I have said here, so I can construct an interval, which is this interval here, which again shows you the random variable y n bar plus or minus 1.96 divided by the square root of n, and we would know that with a probability of 95 percent, the population lies in this interval.
But, as I say, we have to use real numbers, so we have to use estimates rather than the estimators, and this is done in the next step.
So one way to construct a 95 percent interval of mu is achieved by plugging in the specific sample average y n bar, which we have obtained from a data set.
So small y n bar here, plus or minus 1.96 by the square root of n, defines an interval which contains, or does not contain, mu with a certain probability, and I'll be careful about the burden here, because now when moving from the random variables to the actual realizations of the random variables,
we cannot say anymore that this interval contains the mu true value with 95 percent probability.
Rather, what we say is, or what we can say is, that if we sample many, many times independently of each other, and for each sample that we draw, we compute the sample mean y n bar,
then we can construct many, many different such intervals, and what we can then say is that in 95 percent of these intervals, we will know that the mu is in there, or we know that the mu is contained in these intervals in 95 percent of the cases.
That's the remark we can make. We cannot say that for each particular interval. It is true that the mu is in there with 95 percent probability, because it could be that we draw a sample which in some way yields a very bizarre and
rare estimate of the sample mean, which is perhaps far off true value. This can happen with low likelihood, and if this is the case, then maybe that the sample mean is actually not contained in the complex, in the computed data.
Okay, so these remarks I have written down here, it is incorrect to say that a specific confidence interval, like this one here based on real data, contains mu with a probability of 95 percent.
Rather, it is correct to say that for 95 percent of all random samples, the constructive confidence interval contains mu.
Okay, so this was the case where sigma was equal to one. Now we may ask what happens if the sigma is some other number rather than one, and we still know sigma. So we are still in case one, which is easy.
We assume that we know sigma. If sigma is known but different from one, then of course the 95 percent confidence interval is just the same with the one exception that we don't have one over square root of n, but rather we have sigma over the square root of n. But that should be clear, right?
Okay, case two, suppose now that we have a normal distribution, but we do not know the variance of the variables in this population. So when sigma is unknown, then we need to estimate it in order to construct confidence interval.
We know how an unbiased estimator of the sample standard deviation looks like. I call the estimator capital S n and the estimate small. A capital letter is always the random variable S n is the realization.
The typical estimator used for unbiased estimation of the variance is one over n minus one times then the sum over the squared discrepancies between observations and the sample.
So that's the usual formula for estimating variance. And in this case, I'm interested in the standard deviation. So I take the square root of all of this here.
But beware, we take here the factor one over n minus one to have an unbiased estimator.
Sorry.
What one can show and what we do not show here, but what I just tell you and you are allowed to just believe me because it's standard knowledge is that a standardized variable of the type we have studied in case one, just with the true standard deviation being replaced by an unbiased estimate of the standard deviation.
This standardized variable then follows a student t distribution with n minus one degrees of freedom.
I write it here in terms of random variables as one should do it. If we look at a random variable which is defined as random variable capital Y and bar minus mu divided by random variable S n capital S n.
Divided by square root of n. Then this random variable here has a distribution which is students t with n minus one degrees of freedom and this is denoted t index n minus one.
So if the standard deviation is unknown, we have to use the t distribution rather than the normal distribution, right? In the previous case, case one, we used the normal distribution to construct the confidence in Java.
Here we can basically do the same thing as we did in case one. Just we have to use the t distribution rather than the normal distribution because our standardized variable is not normally distributed anymore.
It is distributed as students t.
Paying attention to this fact that with an estimate of the standard deviation, we have to use the t distribution is irrelevant only in small samples because t distribution converges to the standard normal distribution rather fast.
So when the number of observations goes to infinity, the t distribution is exactly the same as the normal distribution.
Actually, already much, much earlier is the t distribution very close to the normal distribution. For instance, for t equal to, let's say 50, sorry, n equal to 50 or even n equal to 30 gives already a very, very good approximation.
For the t distribution, namely the normal distribution. So the t distribution is then already very, very close to the normal distribution.
The t distribution is symmetric and centered around zero. So exactly as the normal distribution, but it has better tingles than the normal distribution.
Or if you want to express it the other way, it has higher kurtosis. The kurtosis of a normal distribution is three and the t distribution has higher kurtosis depending, of course, on the degrees of freedom.
The special case is the t distribution with just one degree of freedom, which is also called the Cauchy distribution. This distribution doesn't even have a kurtosis.
It doesn't even have an expected value by the way. This is really kind of a very degenerate type of distribution, but we don't really usually use a t distribution with just one degree of freedom.
Actually, beware of t distributions with few degrees of freedom due to the fact that they have bad tingles. They can, well, they can be misleading inference rather likely, but always trying to have a sufficient number of degrees of freedom. If you have more than 20, then you're usually fine.
Okay, here are some graphs of students t distribution. This first graph here shows the t distribution for the case of just one degree of freedom.
That is, as I say, the so-called Cauchy distribution. Then students t or Cauchy distribution has a PDF which looks like this.
And you see that these are extremely fat tingles here, where the normal distribution, the blue curve, doesn't have any significant probability mass anymore, somewhere here beyond the value of three. Look how much probability mass there still is for students t distribution with one degree of freedom.
But things become better rather quickly. So this year would be the t distribution with four degrees of freedom. And you see that the tails are still thick, much thicker still than for the normal distribution, but they're far not as thick anymore as in the case of the Cauchy distribution.
And when you have even more degrees of freedom, and I told you, well, as soon as you have 20, you don't need to worry anymore.
Here is the case of 19 degrees of freedom. So that's basically saying we use 20 observations and we lost one degree of freedom for estimating the sample mean.
And therefore, I have a student t distribution with 19 observations. Now you see that the t distribution is only very, very marginally different from the standard normal distribution.
And here is the case for 49 degrees of freedom. Well, and actually, convergence has already occurred, at least graphically, you don't see any difference between the t distribution and the standard normal distribution anymore.
And just as a matter of notation, I will now drop the index n for sample averages and similar variables. So instead of writing y n bar or s n, I will just write y bar s.
And you will know, I hope, that such sample statistics always involve a certain number of samplings, a certain number of observations and we may call them typically n or one time with time series, then we'll call them t.
But we don't make this explicit notation, but we just know that there's a certain number of observations which gives rise to these sample statistics.
Now, how about confidence intervals in the case of a t distribution.
In order to construct such a confidence interval for some probability level, let's say one minus alpha, like the 95% we had for the normal distribution, you do the same thing now for the t distribution.
This would imply that the alpha is the 5% residual probability for more extreme events than the ones where we cut off the interval.
So, in order to construct such a confidence interval, we need to find the critical values c index alpha over 2.
We need to find this critical value from the t distribution.
Because we want to have half of alpha on the right hand side of the distribution and half of alpha is probability mass on the left hand side of the distribution.
And the confidence interval would then be the space of possible events or possible outcomes for the test statistic or for the statistic we're looking at in between those two critical values.
So, we are looking for these critical values, negative c alpha half and positive c alpha half, which have the property that the probability for our t statistic lying between these two critical values, this probability shall be one minus alpha.
So, shall we let's say 95%.
And there's no other way to go, then look into tabulated distributions and then your computer on the textbook.
You can easily find c alpha half in Woodridge Appendix G table G2, that's the table for the t distribution.
So, now everything is analogous to case one. The one minus alpha confidence for mu y is obtained by constructing an interval from any given sample, using the sample mean y bar.
So, the realization of the random variable minus the critical value u times the standard deviation, estimated standard deviation of the random variable capital Y bar and the same thing with plus here.
So, the shorthand notation for the confidence interval would be Y bar plus or minus critical value times standard error of Y bar.
And the standard error of Y bar is, of course, s over the square root of n, as you have seen.
So, that's a graphical illustration which doesn't give any new information just for your eye.
Right. Here is the one critical value. Here is the other critical value, both of them being critical values for alpha half as residual probability.
So, residual probability of 0.25, for instance, on this in this tail of the distribution and another 0.25 here gives us then confidence area of 95% for the values between negative c and positive c.
Now here's a brief example of what we can do in such a case, or how this can be useful. Suppose that we look at 20 firms in the United States which receive an investment tax credit in 2010.
Investment tax credit basically says that the government offered these 20 firms. If they invest in 2010, then they will get a reduction of the tax load. Right. So some of the investment expenditures can be credited for a tax bill.
Now, we may want to check if this policy of offering investment tax credit to certain companies has had the desired effect of raising investment.
We will do this in a very easy way. Let us assume that the change in investment or actually the change in log investment has a normal distribution. Right. So we have data on that and the average change in log investment from 2009 to 2010 is easily calculated.
Let's say we arrive at a sample mean of y bar is equal to 0.02.
Now we would like to know, does this number 0.02 tell us anything about the change in log investment. Can we infer from 0.02 that there was a positive effect on investment.
Let's say, let's stop here, positive effect on investment. We stopped short here of saying a positive effect of the investment tax credit because this would already be a causal statement and that's not so clear what this is actually causal, but it could be causal.
We just want to study here, did investment increase from 2009 to 2010.
Now we come up with a result. Well, the average change in log investment is 0.02 so 2%. And we would like to know, well, is this something which is significant, or is it just, I mean, could it have well have been zero and so it's small growth rate, right, the 2%.
So is it perhaps just a random event which happens, even though actually there was no sizable effect of the investment tax credit on investment activities.
Now, let's construct a 95% confidence interval for the mean change in log investment. This is what we note by you.
And following what I have said before is clear, this confidence interval is y bar. So the 0.02 plus 2.093 times the standard error of y bar, where, well, the standard error of y bar is just S over square root of n, which we have already encountered.
Well, y plus minus 2.093 because we use the t distribution, right, we would write 1.96 if it were the normal distribution, but since we estimate the standard error of y bar, we have to use the t distribution.
Now suppose that the standard error of y bar is 0.009.
This means now when we insert this in the confidence interval here, that the 95% confidence interval which we have constructed is 0.001 up to 0.039.
Can any one of you now tell me what would be conclude from this result here.
Please raise your hand if you have an idea of what this means.
You should have an idea now. What does this confidence interval now suggest.
No volunteer.
All right, see this confidence interval does not include the zero.
So the lower value of the confidence interval is still positive.
Since the zero is not in the confidence interval, we may conclude that the effect of the investment tax credit was non zero.
Actually, that's much stronger than we can make a statement.
But if you want to phrase it somewhat strongly, then you can say it that way.
Of course, we don't really know that is the effect of the investment tax credit.
But suppose that was the only thing which changed between 2009 and 2010, then you may say that any change in log investment has to have its roots in the change in the investment tax credit.
And since the change in log investment is positive may say or is positive with a confidence of 95%, you may be at least hopeful.
Let's put it cautiously that the investment tax credit has contributed to positive investment growth, which was its objective.
With a probability of 95% are new.
That's actually the wrong way to say it.
This is the way where I warned you that would not be quite correct.
Rather, we would say that if we sample many, many times, then we can be sure that with a probability of 95.
That 95% of the sample confidence that was cover the true value. And if this is not one of the confidence demands, which we have drawn and the only one we have access to because the sample is so limited that we can cautiously perhaps say, well, if it is not grossly
misstated confidence level, then the true value of the change in log investment is contained in this interval here with the likelihood of 95%. So we can be somewhat confident that investment has indeed increased from 2009 to 2010 possibly as effect of the investment tax credit.
Case three would be the case where we look at non-normal populations.
Case one, case two, we have assumed normal populations. The difference being that in case one, we knew the variance and in case two, we had to estimate it so that actually the distribution of our statistic came t rather than normal.
Now we give up the normal distribution altogether and do not assume any knowledge about the population, about the distribution in the population. So we allow specifically for non-normal distribution in the population.
This would give us a headache if we have a small sample. So let's forget about the headache and assume that we have a large enough sample for the central limit theorem to apply.
So if that is the case, then the central limit theorem would tell us that a good approximation for the distribution of the sample average y bar with a large sample is still the normal distribution.
So in that case, we would be able to say for large and an approximate confidence interval is the sample mean y bar plus or minus 1.96 times the standard error of y bar.
As a rule of thumb in empirical work, we often don't really bother about those 1.96. 1.96 is the exact 97.5 percentile of the standard normal distribution. So if you want to do everything exact, then use 1.96.
But as a matter of fact, 1.96 is almost 2. So often in empirical work, you use as a rule of thumb plus or minus 2 standard errors of y bar.
So just take the sample mean plus or minus 2 standard errors, and then you have an approximate confidence interval for your estimator.
These confidence intervals typically become smaller when the sample size increases, which is quite clear why this is the case because the standard error of y bar is S over the square root of n.
So when the sample size increases, then square root of n increases and the standard error of y bar becomes smaller. Then, of course, the confidence interval becomes less broad, becomes more and more narrow.
That's it for confidence intervals. I will move to hypothesis testing unless you have questions.
Raise your hand if there's a question.
Okay, no questions. Now, in terms of hypothesis testing, what do we need when we want to test a hypothesis?
Well, there are several things we need to specify.
You need to choose a test statistic and a significance level, but you also have to be specific about the null hypothesis, which you want to test. So you have to specify what actually is H0.
That is the hypothesis to be tested. And this is important because we value the test statistic always under the null hypothesis. So assuming that the null hypothesis is true.
Therefore, it is indispensable that you be very clear about what exactly is your null hypothesis when you do any test.
There's a very common error, not only among students, but also among many other people who do empirical work. Hopefully not too many academics, but not only academics do empirical work.
That people come up to you or come up to me and ask me, well, I've tested this and this. What do you think about my test results?
And then I often look at the test. And first thing I ask is, what actually is your null hypothesis? And then the people can't tell me, right? And this is exactly how the test looks like, right?
It is quite chaos because they haven't specified what the null hypothesis is. But you have to be very clear about what the null hypothesis is. There is no well-defined test without specifying what the null hypothesis is.
It is also necessary to specify an alternative hypothesis. So to be specific about what do you conclude if the null hypothesis is rejected by the data.
Of course, you have to choose a test statistic. A test statistic is typically just some function of the data you have. So some function of the random sample. And you have to choose a significance level of the test, let's say 5% or 1% or something like this.
The critical value is basically a function of the significance level. So the critical value associated with the significance level is determined by the distribution of the test statistic, assuming that the null hypothesis is true.
The rejection rule determines when the null hypothesis is rejected in favor of the alternative hypothesis H1. Based, of course, on comparing a value for the test statistic capital T. So some real number small t for the test statistic capital T to the critical value C.
Please note that this symbol t here has nothing to do with this t distribution, which I covered in the last subsection. This is just a new symbol here for the realization of a test statistic. A test statistic I denote by capital T if it is a random variable and then by small t if it is its realization. This has nothing to do with students t now.
All right. The rejection region is simply the values of t, if t realizes itself, which would result in the rejection of the null hypothesis.
Now we consider several cases here. Two, case one is the mean in a normal population.
So suppose we have a two-sided alternative, which we call H1. The null hypothesis is that the mean of the population is equal to some specific value mu naught.
So for instance, we say in the investment credit example, our null hypothesis is that investment has increased by 2%. So mu naught would be 0.02.
So that would be a null hypothesis we could test. H naught means that we fix the mu, the sample mean, at some specific value, which we spell out, let's say 2%.
And then if we have a two-sided alternative, the alternative hypothesis is that mu is different from mu naught.
Many times actually in economics and econometrics, we have two-sided alternatives, but not always.
It is also possible that the alternative is specified as a one-sided alternative. So for instance, we just test against the alternative that mu naught is greater.
Sorry, mu is greater than mu naught, or mu is smaller than mu naught. Not here. In the simple case, we just concentrate on what is mostly done. We say we have a two-sided alternative and mu is different from mu naught.
Now we compute the test statistic and its result, not the random variable, but its result is small t, which would be the square root of n times the sample mean, y bar, minus mu naught divided over s as the estimate of the standard error of the individual random variables.
So f divided by square root of n would be the standard error of the sample mean.
Note that here in this test statistic, I have computed the test statistic under the null hypothesis. So I do not write y bar minus mu, but I write y bar minus mu naught.
So I have assumed that the null hypothesis is true. And then our statistic, if the null hypothesis is true, would be this t statistic here, right?
The statistic which I call t. And that is, of course, nothing else but y bar minus mu naught divided by the standard error of y bar.
The rejection rule for a two-sided test with significance level alpha is then simply given by the rule that the absolute value of t is greater than the critical value.
Why do I take the absolute value here? Well, because I have a two-sided alternative, right? Essentially what I'm interested in is, does the sample average deviate by much from mu naught?
From the hypothesis, hypothesized value mu naught for the sample mean. If it is true that the sample average is way apart from mu naught, then probably mu naught is not the true sample mean.
And we would reject the hypothesis that true sample mean mu is equal to mu naught, right? So this depends on, essentially it depends here on the difference between y and mu naught, but we have to standardize this by dividing through by the standard deviation of y bar.
So the intuition is simply we reject the null hypothesis if y bar is far from mu naught in absolute value.
And the graphical representation of this is very similar to the one we have already seen. We have certain rejection regions, namely when the test statistic is either here in this 2.5% area or here in this 2.5% area.
And for the remainder, the remaining 95% of the probability mass, we will just accept the null hypothesis.
Now there are two possible errors which we can run into when doing such forms of test. They are called type I error and type II error.
The type I error is an error where we reject the null hypothesis, although it is true. So the type I error would occur if the null hypothesis is true, but we are unlucky to observe a test statistic which lies either here or lies there.
I mean, even if the null hypothesis is true, there is 5% probability that we observe a test statistic either here or there because these two regions have 2.5% probability each, right?
So if this happens, then we would permit a type I error. We would reject the null hypothesis because it seems that the t statistic is significant.
Even though the null hypothesis is actually true.
And the type II error is the error where we fail to reject the null hypothesis, although it is false. And what does this mean? Well, this means we observe a test statistic which is somewhere here between negative C and C, right?
We observe a test statistic somewhere here, even though the true value of mu is not mu naught. It's not mu naught, right? Let's say the true value of the mean is here, right?
That can easily be that we observe a test statistic which is, for instance, here or there. And in this case, we would not reject the null hypothesis that the true mean is zero here.
Therefore, we would commit a type II error and would not reject the null hypothesis, although it is true.
And we can compute the probability of making either type I error or type II error. The probability of a type I error is very easily computed. That is simply alpha, right? I mean, this is the significance level.
The probability that we observe a test statistic either here or there, even though the null hypothesis is true, this probability is, of course, the 2.5% here plus the 2.5% here. So the probability of a type I error is alpha, and that is the significance level.
The probability of a type II error, we cannot infer from this type of graph because the probability of a type II error depends on where the true mu lies. We know mu is not equal to mu naught. It's null hypothesis fault, but we do not know where the true value of mu now is.
And the probability of the type II error depends on whether the true value of mu is close to the mu naught that we have tested or whether it is farther apart. So we just say the probability of a type II error is denoted by beta.
And the power of the test is then called 1 minus beta. Often in the theoretical work, people try to convince other academics that a test is good by computing the power of the test by computing what the probability of a type II error is and subtracting this from 1.
Obviously, the smaller the beta is, the smaller the probability of a type II error is, the better is the power of the test.
Now suppose that we have just a one-sided alternative, so that H1 is either that mu is greater than mu naught or H1 is that mu is smaller than mu naught.
As I already said, these one-sided tests are less common in econometrics than two-sided tests, but it is rather straightforward how we handle the test procedure in these cases.
So we would reject the null hypothesis when the value of the sample average Y bar is sufficiently smaller than mu naught for the case, then let's say the alternative hypothesis is less than mu naught.
So we can just compute the test statistics as before, but the rejection rule would now be either T is greater than C alpha or T is less than negative C alpha, depending on which hypothesis here or there we want to test.
There's also a case two, not surprisingly, asymptotic arguments for populations which are not normal. So if the sample size n is large enough, then we can again go for the central limit theorem and use the standard normal distribution as a benchmark for our test statistic given the argument that we may be close to a normal distribution.
In finite sample size or close to the asymptotic standard normal distribution. So in such a case, we would compare the test statistic T with the critical values from a standard normal distribution because under the null hypothesis, it would be true
that the test statistic capital T is distributed as square root of n times Y bar minus mu naught divided by S and this thing is asymptotically normally distributed as n01.
You may be used in testing procedures as far as you have done this in other graduate studies to use the p-value, which is just another way to express the result of the T statistic.
The p-value is defined to be the largest significance level at which we could carry out the test and still fail to reject the null hypothesis.
That's the definition of the p-value, the largest significance level at which we could carry out the test without rejecting the null hypothesis.
So consider, for instance, the problem of testing a null hypothesis mu is equal to zero in some normally distributed population and assume that n is large enough for the test statistic T to have a standard normal distribution under the null hypothesis.
Suppose now that we have a T value of 1.52 for our sample. The largest significance level at which we would fail to reject H naught is the one associated with using T as the critical value.
So when we use T as the critical value, then we would just not yet reject the null hypothesis. So the p-value is defined to be the probability for the test statistic or random variable, the test statistic, to be greater than 1.52 given that H naught is true.
And that is nothing else but 1 minus phi of 1.52, and this is 0.065.
Okay, so when we have this test statistic of 1.52 here, then you see that the remaining probability mass in this case is 0.065. So the p-value of 1.52 would be 6.5%.
If we carry out a one-sided test, so if the alternative is just formulated with respect to one version of the inequality constraints, obviously you can do the same thing also for two-sided alternative.
Well, this test, if we carry it out, let's say at the 5% level, we would not reject H naught, right? Because there are still 6.5% probability mass left over. So with a test statistic of 1.452 and a significance level of 5%, we would not reject another.
But if we carry out the test at, let's say, 10% significance level, we would reject another because the test statistic just has a residual probability of 6.5%, which is less than the 10%.
Now, clearly this involves dangers that researchers manipulate their results. So what you should never do is that you say, well, you first look at the test statistic, 1.52.
And then you decide whether you apply a 5% level of significance or a 10% level of significance, sorry, 5% level of significance here or 10% level of significance approximately here, because then basically you can choose your result, right?
That would be a way of cheating, a way we would try to manipulate your result in such a way that you achieve the result which you desire.
But that is, of course, not legitimate. You should never do this. You have to fix the significance level before you actually look at the data.
So you have to say I use the 5% level. Or you cannot say I use the 10% level even though this decision is less common or the 1% level, but you have to fix the significance level before you look at the data. Otherwise, you will manipulate your results.
The p-value has also another interpretation.
The p-value, namely, is the probability that we observe a value of t as large as small t is equal to 1.52 when the null hypothesis is true.
So we may ask the question, is 6.5% small enough to reject null hypothesis?
Well, basically, this depends on our tolerance of type 1 errors.
Generally speaking, small p-values are just evidence gauged against the null hypothesis.
As they indicate that the outcome of the data occurs with small probability, well, if H0 is true.
So the way to use p-values is first you choose a significance level of alpha, then you choose your test statistic capital T and decide on the nature of the alternative, whether it is two-sided or one-sided.
And this determines, then, whether the rejection rule is either that t is greater than C alpha or t is less than negative C alpha or the absolute value is greater than C alpha half.
Only as a third step, you compute the p-value of the t-statistic and then you reject the null hypothesis if the p-value is smaller than alpha, or otherwise, if it is greater or equal to alpha, then you accept H0.
So that's the correct way to carry out a test.
This is all for today. Are there any questions?
Again, please raise your hand if you have a question.
As there is apparently no question, I can just recommend to you, please go through this material carefully if you don't have it remembered so well from undergraduate studies, which is often the case.
This is the reason why I repeat it here again, but as you see, I mean, we do this in more speed than you probably have had in undergraduate studies.
So if you have trouble following what I'm saying here, then you are really warmly encouraged to familiarize you again with this stuff, which I'm sure you've all had in some form in undergraduate studies,
because we will need it. I'm not quite sure now how I shall assess the fact that there are no questions, more or less, throughout the lecture, whether this all was much too trivial for you or whether you felt lost by what I was saying, because you're not so acquainted with the material anymore.
I assume there are at least some students who may have had more or less the feeling that they got lost, and that should not be the case. So I do this here for you to catch up. If you have deficiencies there from undergraduate studies, then please, if you feel that you were lost in this lecture, sit down at home and familiarize yourself again with this type of very essential elements of statistics.
Of course, if it was too trivial for you, then sorry for bothering you, but we will come to more and probably also new stuff for those of you who find it too trivial in the next lectures. So far for today, have a good afternoon and see you next week.