Thank you.
Welcome to today's lecture on estimation and inference.
Before I start with the lecture, let me announce, please, that next week, Monday, I won't be
able to give the lecture live like I usually do.
I will rather record the lecture in the next days, Friday at the latest, and then upload
it to Lecture2Go, where you usually find the lecture videos.
So please just download the video then and watch it to make sure that you know what I've
been talking about before the Tuesday lecture, naturally, because the Tuesday lecture will
then be live again.
But Monday's lecture will not, and therefore, please make sure that you download the video
and watch it in time.
OK, we start today with Section 5 of the lecture on randomized trials.
Do you have any questions which relate to the previous lectures?
Any question?
OK, I don't see any.
So let me start with randomized trials.
Excuse me.
Just to put this in perspective, no, I'm sorry.
Excuse me for a minute.
I think I have to drink a little bit.
OK.
Sorry for that, but I hope it goes better now.
So to just repeat what we did and why we do what we do today, we talk about randomized
trials.
These are experiment-like studies where researchers have full control over the design of the control
group and the treatment group.
So we are interested, again, in the effect of a certain treatment, and we would like
to compare a group of individuals which has been subjected to some type of treatment and
a comparison group, the control group, in which individuals have not been subjected
to the treatment.
We discussed last lecture that there may be a selection bias if we just take data which
are coming from observations which we make on our society, where we see that some people
have been subject of some treatment and others have not, then there may be a problem in the
composition of those two groups, the treatment group and the control group, because maybe
that individuals who took the treatment are not comparable across all possible dimensions
with the people who did not take treatment.
So there may be a correlation between having taken the treatment and certain characteristics
of the treatment group, which would inhibit us from conducting a fair comparison between
the treatment group and the control group, unless we control appropriately for this type
of correlation.
This problem is a problem which is pertinent to observational studies, so to studies where
we just pick data from observations we have made on our society and evaluate the data
under a specific research question which aims to identify the causal effect of some treatment.
Because we can usually not guarantee that the treatment group and the control group
are exactly equal in the distribution of the characteristics of the individuals in
these two groups.
However, we can think, and sometimes we have the chance to, we can think of designing studies
in which we randomly pick individuals from a population and assign them to two groups
randomly so that the probability of finding certain characteristics in one group is exactly
the same as the probability of finding these characteristics in the other group.
That would be what we call randomization or, in this case, randomized trials.
These experimental studies where we have full control over the control group and treatment
group are the easiest case of causal analysis, as you will quickly see, and this is why we
start with it.
On this slide here, you see the recommended readings, which are from Joshua Angrist, the
Sears Nobel Laureate, and Jan-Jefen Pischke, mostly harmless econometrics, chapters 1 and
the same author, Mastering Metrics, I mentioned this book already earlier, chapter 1, if you
want to read this as accompanying text.
Now, the main message of the first section of chapter 5 is that the random assignment,
which I just described, solves the selection problem, and the selection problem is actually
the problem that we may easily have a selection bias when we study two groups, which have
been selected, not by us as the researchers, not under a criterion, which was to have the
same probability distribution of observable or unobservable characteristics in both groups,
but by some other kind of development which was beyond our control, which may then have
induced a selection bias.
So the main message here of section 5.1 is that this selection problem, the existence
of a selection bias, is solved by random assignment.
So random assignment of the treatment means that the researcher himself randomly assigns
individuals from the population to the treatment and the control group.
So we just think of having some big population of individuals, many, many individuals in
the population, and obviously each individual has certain characteristics, and the characteristics
differ across individuals, but when we then randomly draw individuals from this population
and randomly assign the individuals to two different groups, then the probability distribution
of the characteristics of the individuals should be the same in the two groups.
So it should be the same in the treatment group and in the control group.
And this is what the researcher would be able to do if we assume that he is able to have
random assignment.
This means that the characteristics or the probability distributions of the characteristics
of the individuals are just the same in both groups, with one exception, and this one exception
is that the treatment is, of course, not randomly assigned, but the treatment is systematically
given to those people which are in the group we call the treatment group, and the treatment
is not given to any person in the control group.
So treatment is assigned systematically, just one group and not to the other group,
whereas everything else is assigned randomly.
And this difference in characteristics now, many, many characteristics which are just
randomly assigned and which have equal probability in the control group and treatment group,
and one characteristic, namely the treatment itself, which is non-randomly assigned but
systematically given to the treatment group and only to the treatment group, this difference
allows us to identify now any observed difference in outcomes as the effects of the treatment
which has been systematically assigned to the treatment group.
Note that this randomization also solves the problem of unobservable characteristics.
But naturally, we are usually only aware of a limited set of characteristics of individuals.
Most features, most characteristics which characterize individuals are not measured,
are not observed.
We do know that they exist.
For some of them, perhaps we don't even know that they exist, but we can think of many
which exist for which we have no knowledge at all, no observations at all.
This can be a big problem if we have non-experimental data, so data which are not chosen randomly
from one common population.
But if we do choose randomly, then the random assignment will also guarantee that all the
unobserved characteristics have the same probability in both groups, the control group
and the treatment group.
So random assignment is really ideal because it also helps us to solve the problem of unobservable
characteristics.
Now random assignment of DI means that the researcher randomly assigns individuals to
treatment and control group, which means I pick an individual from the population and
randomly choose either the treatment group or the control group for this individual.
So the treatment is assigned randomly to individuals.
Of course, I just said treatment is assigned systematically to the treatment group.
This is the defining property of the treatment group that all individuals in the treatment
group get the treatment.
But the way how individuals came into the treatment group was completely random.
They were randomly chosen from the population for the treatment group.
This means that the treatment itself has randomly assigned to the people in the treatment group,
even though it is, of course, systematic, that the treatment is given to all people
in the treatment group once they are in there already.
But the way to get into the treatment group is random, just to make sure that this is
not confused.
Now this random assignment of DI, as I have already explained, solves the selection problem.
I have to say it guarantees us that we do not have any bias anymore if the individuals
are randomly selected for the treatment.
Also the DI, the treatment status, recall DI is one when an individual is assigned for
treatment and it is equal to zero when an individual is not assigned for treatment.
This DI variable is independent of the potential outcomes.
Or this is actually what it says, the treatment status is completely independent of the potential
outcomes because we have this random assignment.
Therefore, we know that the expectation of why I, given that individual I was subject
or is subject to treatment, that DI is equal to one, minus the expectation of why I, given
that the same individual is not subject to treatment, this expectation here can be written
in various ways.
For instance, we can also write it as the expectation of the potential outcome in the
case of treatment for individual I, which is variable why one I, given treatment, minus
the expectation of the potential outcome of individual I, in the case that individual
I did not receive treatment, given individual I didn't receive treatment.
Quite obviously, it is the case that when we look at this variable here, the expectation
of why I, given that I is in the non-treatment group, did not receive treatment, this expectation
of why I is then the expectation of the outcome variable in the case of non-treatment.
This is true that the why I, this why I here, can be replaced by the why zero I if we are
talking just about those people which are not treated, which have DI equal to zero.
And the same thing here, this why I, given that I has been treated, it's the same thing
as the expectation of why one I, given that I has been treated, why one I being the potential
outcome in the case of treatment.
So we can write this difference in this way and we can also write the difference by leaving
the first term unchanged and changing the conditioning, the condition in the second
term.
So the second term here is now changed because here we have DI equal to zero and here we
have DI equal to one.
This is still the expectation of why zero I, given DI is equal to one, sorry, I said
strongly and it is still the expectation of why zero I, but the condition has not changed.
Here it is DI equal to one and here it is DI equal to zero.
It doesn't matter which of these conditions we have because the expectation of why zero
I must be the same since we have randomization, since the treatment status was just randomly
assigned.
Therefore, it doesn't matter whether we have DI equal to zero or DI equal to one as the
condition for the expectation of why zero I.
This is what I've written here on the rest of the slide.
So we can develop this further.
Just to note here, it is the independence between why zero I and DI, which allows us
to do this kind of swap, exchanging DI equal to zero for DI equal to one.
This is possible because why zero I and DI are independent, which is exactly what I had
as the first line on the previous slide, DI is independent of the potential outcomes.
So using this, I can swap the condition here and the way I've done that.
And then I can, of course, also develop the difference further because if I know that
it's the expectation of why one I minus expectation of why zero I, given the same status, treatment
status, DI equal to zero, and I can also write this as expectation of why one I minus
why zero I, given DI is equal to one.
And since it doesn't matter for the potential outcomes, what the treatment status is, since
treatment is independent of the outcomes, we can also write this as the unconditional
expectation of why one I minus why zero.
So these are different ways to write the causal effect, which is this expectation here.
And you see, we can do it in the easiest possible form without paying any respect to the condition,
because the condition has been randomly assigned anyway.
So it should not matter whether certain individual has been randomly assigned to treatment or
to non-treatment.
We can just concentrate on the unconditional expectation in the way in which I have written
it here.
All right, let me give you an example for that, which is kind of a weird example, actually,
which would mean that we assign hospitalizations or treatment in a hospital randomly across
the population.
We referred to this observed scenario already in previous lectures, and now I come back
to it.
It obviously doesn't make sense to basically draw a lottery everywhere in the population
and then tell certain people that they have been selected to go to hospital, even if they
are perhaps completely healthy.
But it doesn't make sense didactically to talk about this feature of a randomly assigned
hospitalization.
If it were the case that people in the population were actually randomly assigned to go to hospital
irrespective of whether they are ill or not, then the expected effect of hospitalization
on those people which are actually hospitalized is the same, of course, as the expected effect
of hospitalization on some randomly chosen person, right, some person which is perhaps
not at all assigned to hospitalization, to hospital treatment, and the effect would always
be the same.
So we would have the property that the average treatment effect is equal to the average treatment
effect of the treatment, and actually you could try to also be equal to the average
treatment effect of the non-treatment.
Since treatment is completely random, is randomly assigned, the average treatment effect and
the ATT and actually also the AT&T are the same, which also makes our life easier.
This is not only true in this example here, but it is a general feature.
There's no difference between the ATV and the ATT in randomized controlled trials.
In randomized controlled trials, I denote by RCT.
There is a question coming in, apparently, and we'll wait until it has been typed.
There it is.
So we don't have a huge problem with counterfactuals, then yes, this is true, right?
We don't have a huge problem with counterfactuals because the non-treated people already give
us the counterfactual, yes, that's quite correct what you're saying.
Okay, again, the property, of course, hinges on the fact that the random type of BI eliminates
the selected ones.
One example of the importance of designing studies as randomized trials versus the alternative
of having some non-experimental survey, let's say some observational data which we use without
having control about treatment and control groups.
So this is what I then call non-experimental, this is just observed data and not data which
has been shaped by random assignment by the researcher.
One example for this importance is a study which had been conducted for hormone replacement
therapy, and this hormone replacement therapy targeted middle-aged women.
You know that women in their mid-50s, late 50s, early 60s are in the face of their life
where the menopause starts, which is a change in the hormones of the female body which can
be quite unpleasant, you can experience sudden feelings of heat, heat waves, you can suffer
from depression, for instance, so it is something women have or some women have trouble going
through where they actually suffer from and the obvious possibility to alleviate the situation
of such a woman is a hormone therapy, which is called the hormone replacement therapy
or HRT here, which would try to make the change of hormones in the female body during menopause
smoother, easier to bear than perhaps a rather sudden change of hormones as it may be experienced
by some women. So the question was whether it is possible to give medication containing hormones
to middle-aged women and thereby have a significant effect on their well-being,
and this has been tested in a non-experimental study. First, there was a so-called nurses' health
study, which, as the name says, concentrated on women which worked as nurses in
the health industry, so in hospitals or with private doctors or whatever,
these nurses were given the hormone replacement therapy and there was a control group, of course,
also of nurses which did not receive the HRT therapy. What this nurses' health study then
showed was that the nurses who received HRT had significantly better health results
than the nurses in the comparison group, and therefore HRT was recommended as a medication
for all women which experienced this kind of hormonal change in their bodies.
But later than this influential nurses' health study, there was a randomized controlled trial
on the same question. This had been done because there was a critique that nurses are perhaps not
a random sample of middle-aged females in general. So nurses, one might perhaps suspect, are more
responsible in their use of medication than an average woman, or at least than some women which
are not nurses, which perhaps have quite different habits in taking or not taking medication for
various diseases which they may have. So testing HRT just on nurses, which perhaps are particularly
trained and particularly responsible in their usage of medication, may give results which are
not representative for all women in the population, and therefore they may have some kind of bias,
even though in this case the treatment group and the control group consisted both of nurses.
Still it is the case that the nurses in these two groups are not representative of all women
in the population. Therefore a second study was conducted which used a randomized controlled
trial, and the second study actually showed very few benefits of HRT. So the conclusion from the
second study was that the first study, the nurses' health study, had had a problem with selection
bias and gave misleading results which could not be reproduced, at least in terms of the
positivity of these results, which could not be reproduced by a randomized controlled trial.
Rather the randomized controlled trial revealed serious side effects that had not shown up in the
non-experimental data, so not shown up in the non-randomized data of the nurses' health study,
which illustrates that non-experimental studies, so studies which have not been randomized,
can be quite misleading because there may be a selection bias which has then been neglected.
The second example for the dangers affiliated with observational studies are attempts to quantify
what kind of effects, if any, a government subsidized job market program has, so job market
training programs. These government subsidized training programs typically aim at people which
are somehow disadvantaged on the labor market. People who have trouble finding a job,
they receive some kind of instruction, some kind of classroom instruction, some type of on-the-job
training, and often these are people which are really severely handicapped on the labor market,
so they may either be people which have been unemployed for a very long duration already
before they qualify for such a program, which is of course an expensive program, or they may be
groups which have particular psychological problems, which are alcoholics or drug addicts
or something like that, so really disadvantaged people as participants in the labor market.
Obviously the hope of these programs, of these training programs, is that with the government
training, these people improve their job market chances and have higher employment rates and
higher earnings. So there have been many, many studies actually of economists which try to
evaluate the effects of these program participants, and quite often the studies came up with a
surprising conclusion that the participants in the government subsidized training programs,
and after successful completion of the program, lower earnings than a comparison group,
which did not participate in the government subsidized training program. So the trainees
actually earned less than plausible comparison groups, which of course led to the recommendation
to abandon such government subsidized training programs, if the effect is actually that people
are less successful on the labor market after having been trained by the government. Now these
are studies for the United States. However, these studies were not studies which had a random
assignment of the treatment. So the selection bias may be a problem in these studies, and here
we would need to go into the details of different studies and see which study suffers possibly from
which detection bias or does not suffer from a selection bias. Why is there naturally a problem
or selection bias? Well, the government programs target actually the lowest
qualifications for people with the greatest problems in the labor market. So it may actually
be difficult to find a comparison group which is equally disadvantaged on the labor market,
but does not participate in such government program. If it were the case that there is
some selection bias in choosing the control group, then it is quite likely that the control
group has actually people which are at least slightly better qualified, slightly less
disadvantaged on the labor market than those people which have been selected for the program.
And therefore it would not be surprising to find that the control group which is assumed or assumed
that it may be the case which may be qualified a little bit better for jobs than the treatment
group, and that this group also has higher earnings after the people in the treatment group
finished their training hopefully successfully. So it may be that there are studies which used
observational data, non-experimental data on this research question, and that these studies had a
problem with selection bias which they did not recognize, which they did not control for
sufficiently, or as I put it here on this slide, maybe that the comparison groups were just not as
comparable or as plausibly comparable as it seems. Now the last example is an example of a
research question from the social sciences, whereas the formal studies were more from
biometrics, from health sciences, and both of them can of course also be studied with methods
which we typically employ in economics, but we also have to acknowledge that for many research
questions in economics, we are not in a position to conduct randomized trials. So while there are
some applications where randomized trials are possible also in the social sciences,
and perhaps they actually become more prevalent in recent years or in current years, and then they
have been prevalent in the previous years, but still most empirical investigations which we do
use observational data rather than randomized trials. We should however strive for randomization
because, as I have emphasized, this solves a couple of important problems, one of which is
the selection problem. So we would like to look now at an economic problem, basically
at least when we consider a human capital as an important economic variable, because we look
now at education, at schooling, or in this case at schooling in kindergarten age, so for
children which are between five and six years old. So one question which has been
discussed controversially in research was, is it useful to invest more in education,
and what is actually the benefit which is associated with a certain amount of additional
costs in education? We always have to weigh costs and benefits. And one important cost factor in
education is class size, because obviously when governments opt for smaller classes in kindergarten
or in schools, then they have to hire more teachers, and hiring more teachers is expensive.
Now, there were a number of studies actually which indicated from observational data that
class size just doesn't matter that much, and that the learning outcomes of students
did not really depend much on the question of whether the class was small or the class was
large, so the teacher had much time to devote to student or whether the teacher did not have that
much time. And these studies were criticized because they are not using randomized data,
and there is the possibility of selection bias, and I will explain in a minute what the selection
bias may be for such observational data being used in non-experimental studies. But first,
let me mention here one particular study which circumvented this problem of the selection
bias by using actually a randomization for the analysis of the question whether
small class sizes has better learning outcomes. And this was the so-called Tennessee Star
experiment, obviously an experiment which took place in Tennessee, the state of Tennessee in
the United States of America. It targeted exactly the question I've been talking about,
namely the question what kind of effect does a smaller class size have in primary school or
in kindergarten. The larger question which we are not looking at here is whether the expense,
the costs associated with smaller classes, actually pay off in terms of higher student achievement.
So this kind of cost-benefit analysis or this kind of weighing costs against benefits was
the ultimate objective of the Tennessee Star experiment. We will not look at this, but we
would rather just look at the question whether smaller classes lead to better learning outcomes.
We do not study the question whether the increase in learning outcomes is worth the costs associated
with it in terms of teachers. That's the second question that's difficult to assess in any case.
But here we are just interested in the question whether small classes lead to better learning
outcomes. As I have said in several observational studies, there was little evidence of any link
between class size and student learning. However, this is a little counterintuitive because usually
you would think that if we hire more teachers, then the teacher has more time to devote to each
single student. And if the teacher has no other business, then they should be able to achieve
better learning outcomes the more time they have to devote to a particular student. Or conversely,
it would be counterintuitive to think that hiring fewer teachers actually has no consequences for
students' achievements. Now, what may be the reason that observational studies still came to the
result that class size doesn't matter greatly or doesn't even matter at all for the learning
outcomes? Well, one possible selection bias in these observational studies may be that schools
have already deliberately put weaker students with learning problems into smaller classes
to help them. And if this is the case, if the school is actually controlled already for the
intellect and the ability of students and try to support the weaker students by putting them
into the smaller classes to have the benefit from more teacher attention, then it is, of course,
not surprising that studies have found that the results in small classes were inferior to the
results achieved in greater and larger classes. Because the students in these classes are
not comparable to each other, they do not have the same probability of characteristics which
describe them, their personality, their intellect, their learning ability, and these kinds of things.
So there would be a selection bias. This was actually the reason to start this big
Tennessee Star experiment. People wanted to study the question under randomization.
So that would ensure that we really have comparable control and treatment groups.
The students assigned to classes which are either big or small are comparable in every
other aspect except the class size in which they sit, so the class in which they sit and its size.
Any other aspect these students are comparable and the distribution of any
other characteristic is the same in the treatment group and in the control.
The Tennessee Star experiment then found a significant and lasting payoff to smaller
classes, quite a constant trust to the results produced in earlier observational studies.
We will come to this Star experiment now. Let me just mention that ideally in a randomized
experiment, the randomization should balance successfully the characteristics of the
individuals across the different groups, either treatment group or control group.
I write here across the different treatment groups. With this, I mean either the treatment
group or the one treatment group. So both are in that sense treatment groups, but I have so far
used the terms control group and treatment group and this is actually what is meant.
The randomization shall balance any kind of characteristic, be it observable or unobservable,
across the different groups which we have devised in order to study the question whether
smaller classes support better learning outcomes for students in these small classes.
Now the Star experiment had only a few characteristics which we may include here.
These were variables which were easily observable for the schools and the kindergartens which these
students went. I pick out three such variables here, namely one variable which is called free
lunch, another variable which is called student race, and a third variable which is student age.
Starting with the last one, it is quite obvious that one should think the older a student is
at the age of five or six. The higher is the student's ability to learn something, right?
The student is a little bit more mature and more experienced also in learning, so we would expect
that older students are a little better in learning than younger students at the age of five or six
years. Student race is of course also something which is controlled for, not probably for racial
arguments, which would imply something like mankind has different races of different intellectual
abilities, but rather because race often indicates a certain layer of society. You know that in the
United States of America, black people are usually coming from the lower layers of society,
and white or Asian people are typically coming from the higher layers. They have higher
incomes or better professional possibilities than black people, and this is why student race is being
controlled for. It may actually also be a proxy for the question of whether this is a one parent
or a two parent family. As we know that in black families, the divorce rate is much higher
than in white or Asian families, and obviously this can also have an impact on the learning
outcome of a student, whether he lives in a family with two parents or just in a family
with one parent. And the last variable, free lunch, is basically also a variable which
approximates parents' income because free lunches are handed out to those students who come from
the rather poor families where father and mother cannot pay for lunch, lunch package, which they
would supply there, their children. There is a question. Is there a rule of thumb on how many
individuals to use in the RCT, by the way, for the features to be balanced? No, there is no such
rule of thumb. In general, the more observations you have, the better it is provided that
observations are basically independent of each other, but there is no rule of thumb for how many
you may use. Typically, we aim for asymptotic properties, so we try to make the numbers as big
as possible. In the regular Tennessee classes in the year in which the study was conducted,
which was 1985-86, the average class size of regular Tennessee classes was about 22.3.
The study now involved about 11,600 children, so this is a substantial number of children which
have been monitored in this study. The experiment was much larger than what I report here on the
experiment. Actually, children were randomly assigned into three different groups rather than
two different groups. I will talk here only about group one and three, and I will leave aside group
two. Group one is small classes with 13 to 17 children, and group three is regular classes with
something like 22.3 children per class and full teacher's aid. There was a third group, as I say,
with the same type of class size and only class size and only part-time teacher's aid, but as I
say, I will ignore this second group here and just compare the small classes with the regular classes.
Here are the characteristics in the STAR experiment. The three characteristics I talked about,
namely free lunch, the race, in this case white Asian as being one race or one race group,
and black being that the opposite race group, and the age of the student in 1985.
Let's first just look at these three characteristics which we have here. What we find then in the next
two columns is the mean value for these characteristics for the small-sized classes
or for the regular-sized classes. For instance, it was the case that 47 percent of the
students received free lunch in the small classes and 48 percent received free lunch
in the regular classes. These numbers are quite close to each other. These are average
values here, and we can test for equality of these average values, test for equality of the
means here. If we do this, we get a key value for the hypothesis that 0.47 is equal to 0.48,
or that the underlying, actually I should phrase it differently, I should say that the underlying
true parameter is equal in the small-sized class to the true parameter in the regular-sized class,
even though it gave rise to slightly different realizations of the sample, 0.47 here and 0.48
here, and you see the hypothesis that the underlying parameter is the same as a residual
probability of 9 percent, so it is not significant at the 5 percent level, and we would not reject
the null hypothesis. Same way to read these numbers here. This is the share of white or Asian people
in the small classes, and this is the share of white or Asian people in the regular-sized classes.
Again, some kind of t-test on equality of means has been conducted here, and the p-value for
equality of means given these two sampled values is 0.26, so again we cannot reject the null
hypothesis that the underlying parameter is actually the same. Age in 1985 is also very,
very close to each other, so most of the students, or let's say the mean age of the students in the
small classes was 5.44, and it was 5.43 in the regular-sized class, so basically no difference
the p-value on the test of quality of needs is again not significant. So we can find here exactly
what we would expect to find when we have already the prior knowledge that this is a randomized
controlled trial, because we know we have drawn randomly in this experiment from those 11,600
children, children which went to the small classes, and children which went to the regular
class sizes. So the children were randomly assigned either to this column here, to the
small-sized column, or to the regular-sized column, and then obviously we would expect that
the probability distribution is the same, in particular since the population of 11,600 people
is so large, 11,600 children. Another question here,
longer text, now we only look at the average value, but for example if I want to study health
expenses, then even if the attributes at average are the same, the explicit distribution in the
group would be important, for example weight or the expected average health expenses in the group,
under the assumption of these people are more expensive, so shouldn't we look at more than
the average values? I'm not quite clear if I completely understand the question, so
basically you address two issues here, if I understand correctly. One issue is whether the
mean of these values is already sufficient for concluding that the probability distribution is
the same, and obviously this is not the case. I mean the answer to that question would be no,
there will be that the distribution for instance here for the dimension-free lunch has the property
that the mean is the same, but it can be that higher moments of the distribution are completely
different for the small class group and the regular class group, so it could be that the
variances for instance are different, that the schoon is different, that the kurtosis is different
and so forth, so in a more thorough investigation you would also look at other properties of the
distribution and compare them. Frankly, given that this is a randomized controlled trial, I
would expect that also for the variants of the schooners and the kurtosis and so forth,
you would find similar values in these two groups of people, but you never know, it's always better
to check and I will show you in a couple of minutes that even here there is some evidence that
perhaps randomization did not work quite as well as people would hope it does. So this is the first
part of my answer when you ask whether the mean is actually already sufficient for characterizing
the distribution, and then the other element I've sensed already in your question was how about
weight for instance, body weight, body mass or something like that, and indeed I mean if we had
data on body mass of the children that I think I would like to control for that. I would also
like to compute the means, first thing of course, the means for the weights and see whether perhaps,
I don't know, there are more fat children in the small sized class than in the regular sized class
or the other way around, which would be evidence that randomization has failed in any such sense.
So I would like to study that if I have more data on that, and perhaps now while I talk
over it, perhaps you mean that for weight you would not only like to have the mean weight,
because you say, well it isn't really that important whether people or students have
mean weight, but you would like to know how many people there are which are perhaps
undernourished or which are completely overweight, and if these children which have health problems,
because they either are undernourished or they are overweight, are not represented in both the small
sized and the regular sized classes in the same proportion, then again randomization would fail,
but it seems that the Tennessee Star experiment did not have these type of data on body weight.
Okay, now let's skip attrition rate for a minute and go to lines five and six here.
Line five just tells us how big a small class was on average, right, so 15.1 children,
whereas a regular class was 22.4 children, and obviously this difference here is considerable
and a test for equality of means here clearly indicates that these means are not the same or
that the true parameters here underlying these class sizes are not the same, because obviously
I mean we have intentionally designed small classes for one group and regular sized classes for
the other group, so here we expect to find a significant result. And line six finally is
the outcome, right, this is the percentile score in kindergarten, so basically it tells us after
having been in the small sized classes or in the regular sized classes, then number of tests were
conducted on the learning outcomes and this is basically the test score, that's the test result,
and you see the test result was higher and this in this case means better, was better in the small
sized classes than in the regular sized classes, but quite a bit by the way, right, this is about
10 percent difference here, which for such a huge group of children is a sizable difference.
I do not have information here on the standard deviation which would allow us to assess how big
this difference is in terms of standard deviations, but even if we just look at it in terms of normal
numbers, then we would say well this is about 10 percent more, 54.7 is about 10 percent more
than 49 percent here, right, and the p-value for equality of means is again zero, so this effect
here is significant, therefore the star experiment, the randomized controlled trial suggests quite
unlike earlier observational studies have suggested that class size does indeed matter
for learning outcomes and that small classes are helpful for getting better learning outcomes,
and it's still a question of well is it worth the effort, is the increase in learning outcomes
so big that it is reasonable to pay for additional teachers, because this is considered a cost,
but then question as I said, we just leave aside here, this is a question for the policy makers in
Tennessee, and I suppose they have answered it one way or in the meantime, we're just interested in
the effect of randomization. And talking about randomization, look there is one thing which
casts some doubt on randomization, which is the attrition rate. The attrition rate is the rate
by which people have discontinued this experiment, right, so obviously such an experiment takes a
certain amount of time, I think it was one year here, one kindergarten year, during which children
were either in a small class or in a regular sized class, but sometimes people are withdrawn from
kindergarten for whatever reason, the parents move away or they just want to go into a different
group because there's a friend in this group and the child doesn't feel well in the other group.
Things like this happen also during these experiments and you can't do anything against
this, so some people discontinue the experiment. And this point 49 here and point 52 here is the
proportion lost when following up on the experiment, actually in this case the proportion
of children lost before completing third grade, third grade, so kindergarten is actually before
first grade, right, so there was another test or there was one test at least after finishing
third grade to see if those children which were in a small class in kindergarten years ago, right,
still had better learning outcomes and the attrition rate now, the share of children,
the researchers have just lost track of or which left the experiment is considerable as you see point
49 here and point 52 here and this is significantly different, right, so actually under randomization
we would expect that the attrition rate is not significantly different than people drop out of
the experiment with the same likelihood but this hypothesis is rejected here, so this warns us that
perhaps randomization did not work quite as well as it should have in the Tennessee Star experiment.
Okay, I think I have basically said what I have written on this slide, let me just check.
Yes, oh yeah, no, except for the last remark here, right, so the same last remark is what we
infer from this table that the difference of test scores between small and regular group
participants is significant, so that was in line six and now I will move from this well more or less
descriptive type of analysis which we have done here, not completely descriptive because there were
some tests but still an analysis which just interpreted table, I will move to a regression
based analysis of similar questions because the difference in the means between the treatment and
the control groups which we have seen here, right, this difference means can also be framed in
the form of a regression analysis where we would regress the outcomes on certain determinants,
right, we would regress the outcomes on dummy variables for instance for the question is the
class size small or is it regular, right, dummy variable one zero, is the age, yeah okay the age
you wouldn't even need a dummy variable for but for white Asians or for the race question you would
define a dummy variable and you would say a dummy variable is one if the child is black or zero if
the child is white or Asian or something like this, similar here a dummy variable for whether the
child receives free lunch or does not receive free lunch, okay, so one could have a regression
analysis where the test score is regressed on such dummy variables one of which would be the
dummy variable for the class size indicating whether the class size is small or regular and
one could then test whether this dummy variable or the coefficient on this dummy variable is
significant and if it is then under randomization we could be rather sure that there is indeed an
effect of smaller classes on learning outcomes. I think I saw another question coming in or am I
mistaken? No there's not, all right. Any other questions here for a move to the formal analysis?
Apparently not, okay so let's go to section 5.2 now where we look at
regression analysis of such experiments and the main headline here is that this can be quite
useful too even though I would not overstate the usefulness of these questions but it is helpful in
many cases. Regression is a useful tool not only for randomized controlled trials, I should perhaps
point out the outset but it can also be a useful tool for other types of causal questions involving
data just from observations so from observational studies but if we do have data from observational
studies then we need to be careful in the specification of the regression and this will
be the main message from the section here. So suppose just for now that treatment effect
delta is constant and the same for everybody right which is not necessarily the case just to
make our life easier at the beginning we will say that the difference between y1i and y0i so the
difference between the two potential outcomes in the case of let's say small class size versus not
small class size that this is the same for each child right and constant over time if you wish.
So just some delta which does not depend on i on the i-th individual anymore
and now when we have such constant causal effects we can rewrite the equation which
indicates us how the actual observations depend on the potential outcomes in form number one.
Let's first review this equation here you've seen it already but just to make sure you
remember what i'm talking about the observed variable yi let's say the observed cognitive
abilities of the time the test score which the child has on certain tests which evaluate
the the learning experience of the children this is the observed variable this we can always write
as the potential outcome without treatment plus the difference between the potential outcomes
with and without treatment times the variable di and di indicates whether the individual has
actually received treatment in which case di would be equal to one or has not received treatment and
you know that of course in the case that di is equal to one then y1 will be equal to by one i
and if di is equal to zero then yi will be equal to y0 i from this formula here now given
that we assume that y1 i minus y0 i is constant we can replace the y1 i minus y0 i here
by this delta now we have something like delta di and then suddenly it looks already like a
little bit like a regression equation because you could then think of regressing the observed
outcome on well this thing we will still talk about it's it's unobservable plus a dummy variable
for whether the individual has received treatment or has not received treatment and then estimate
the coefficient of this dummy variable which would be this delta here and the delta is the causal
so this is what equation one here expresses we have yi is equal to alpha and we'll explain
the minute what alpha is plus delta times di plus an error term ui okay and we are interested
in this delta here we're not at all interested in the alpha actually we just want to know what
this delta here is because delta is the causal effect this is the difference between two
unobservable variables right this is our problem with causal analysis that we cannot observe
at least one of the potential outcomes usually one of the potential outcomes we can't observe
so we can compute the delta we cannot observe the data but here we would estimate the delta
because the delta is the coefficient of the di and the di we do observe now how about the alpha
and the ui where do they come from well very easy actually i have decomposed the y0 i here
into its expectation expectation of y0 i and then y0 i minus its expectation this is the ui here
it's the error term right uh when do i do this well the expectation of y0 i the unconditional
expectation of y0 i should be a constant right so this is then some alpha which i can estimate
expectation of y0 i and what then is left over y0 i minus its expectation is of course something
which has an expectation of zero and if you take the expectation of y0 i minus e of y0 i then you
see that this expectation is zero so this variable here is actually like an error term it has an
expectation of zero it varies by individual which is why the u variable here has an index i right
but it is actually nothing else but an error term right some deviations from the systematic
relationship which we have um here alpha plus delta bi so we have a random component here we
have the causal effect here alpha is just a constant so why should we not just estimate
this thing here by ols well let us evaluate the conditional expectation of this equation one
with the treatment status uh i already switched off and on which basically means let us evaluate
this expectation now in two different settings one different setting is given that there is
treatment so the i is equal to one and the other setting is given that there is no treatment
so the i is equal to zero right so we evaluate this equation under both settings the first setting
setting given that the i is equal to one would amount to saying that we have on the left hand
side the expectation of y i given that i has received treatment right and this would be equal
to alpha plus delta of course because d i is equal to one plus the expectation of u i given
that d i is equal to one and the same thing here if we take the expectation of equation one and
evaluate it just for those people which have not taken treatment the expectation of why i given
that d i is equal to zero would be equal to alpha plus the expectation of the error term given that
d i is equal to zero and nothing in between no delta here because in this case d i is equal to
zero so here as the second term in this equation we would have d i times delta right we would have
this delta times d i but the i is equal to zero so this term matches and we're left with this
equation here okay what does this mean this means that we can identify the delta or we can hope to
identify the delta by taking the difference between the two so if i subtract the second equation here
from the first equation then i get the expectation of why i given that d i is equal to one minus the
expectation that of why i given that d i is equal to zero is equal to delta which is a treatment effect
plus a term which actually is a selection bias right so the difference between these two
error terms here or these two expectations of the error term can of course be zero right if
this difference here is zero then we have no selection bias then we would say that the
expectation of the error is the same regardless of whether the individual comes from the
from the treatment group or from the control group but if it is the case that the expectation of u i
differs depending on whether the individual comes from the treatment group or comes from the control
group then we would know that there is some structural difference between the treatment
group and the control group and this would indicate that we have selection bias
now why do i say that or why do i write it in this unconventional formula suppose that this
difference here is not equal to zero so suppose that we have a selection bias then this selection
bias obviously amounts to saying that the u i is not independent of the d i right if the u i were
independent of the d i status then obviously this difference would be zero if this difference is not
zero the u i is not independent of the d i so the least we can say is that we have correlation
between the regression error term which is this u i here and the regressor d i okay but correlation
between the regression rate regression error term and the regressor is something which makes
our estimates inconsistent as you know this was why we went through the review of basic econometrics
where we had assumptions a 6.2 for instance which we typically used for the consistency result
but assumption 6.1 would also serve and actually 6.3 would also serve right these were the
assumptions which guaranteed that the ols estimate is consistent we didn't know whether
it was actually unbiased but we could guarantee consistency if assumption 6.2 is satisfied
but if we have a correlation between u i and d i then we see there is a selection bias here
and we would not estimate this delta coefficient appropriately we would not estimate this in an
unbiased way because there is the selection bias so here you see how the two parts of the lecture
first the standard part on properties of least squares estimators properties like consistency
for instance and now the second part about causality analysis interact with each other
because what we have identified as selection bias in the causality analysis so far
is actually econometrically measured as a correlation between the regressor term u i
and the regressor d i because recall this d i here was the regressor which we used to estimate
the regression coefficient delta for the d i is the regressor here which has this coefficient
delta in which we are interested in it but if i have a correlation between d i and u i and i know
that my estimate of delta is not consistent so even asymptotically it's not the correct value which i
would estimate by ols and that would be the result of a selection bias because correlation between u i
and d i means that we have a selection bias okay this is a very important point and i have
now i think presented in different ways it is important though that you have understood what
i said here or understood the connection between the econometric theory and the causal analysis
that we aim for here and if you have any questions then please raise your hand now and
give me your question so that i can answer that is there any question please raise your hand
i don't see any question so let's move on we have this error term u i and the error term u i
is the difference between y zero i and the expectation of y zero i which is the same thing
as this alpha so the constant in the regression right this is just the definition of u i this is
how i had defined u i on the previous slide so what does this imply it implies that the difference
between u i given d i is equal to one and the expectation of u i given d i is equal to zero
so that this difference here which is the selection bias is the same as the expectation of y zero i
given d i is equal to one and the expectation of y zero i given that d i is equal to zero
why is this so well it's very easy to see just plug in this term here for the u i here
and the u i there then you will see that the alpha is of course the same it is always expected
regardless of whether we are under d i equal to one or d i equal to zero so we have one alpha here
and we have one alpha here alpha minus alpha is equal to zero we are left with and with the
potential outcome y zero i so difference in potential outcomes for people coming from
different treatment status here d i equal to one here d i equal to zero is exactly what we call
the selection bias right if there were a difference of outcomes depending on whether
a person has received treatment or has not received treatment then we have selection bias
so the selection bias is basically a difference in the node treatment potential outcome comes
between those people which get the treatment and those which do not in the hospital example this
would mean those who were treated had poorer health outcomes in the node treatment state
right so this would be this type of difference in outcomes depending on the treatment status
that those who were treated had possibly poorer health outcomes in the node treatment state
or in the star experiment case students in smaller classes tend to have intrinsically
lower test scores in a simple comparison without randomization
now in the star experiment the d i's were randomly assigned and this means that the selection bias
disappears so a regression of y i on d i should estimate the true cause and effect of interest
right so if we use the star experiment data for this type of regression here
then we would know due to randomization we have no selection bias so this term here is equal to
zero which also implies there's no correlation between u i and d i right so since there is no
correlation between u i and d i we would get a consistent estimate of the delta here so we can
actually estimate this effect by using just the delta without any further problems we just regress
the outcomes on the question of how big the class size was and control for
possible other influences by including dummy variables
okay this has been done in the following table which shows the estimated effects of the random
class size assignment in the star experiment on the test scores which students achieved right
we see four regressions here in the columns one two three and four right and the first is the
most basic regression which we can run this first regression is actually exactly the regression
which i showed you here where the test score of each student is just regressed on a constant
and on the dummy variable of whether the class size was small or regular right no other control
variables just this equation one which has been estimated this corresponds to column one here
and what we see is that the small class effect is 4.82 is estimated to be 4.82 with a standard
error of 2.19 right um you see that the estimated coefficient is more than twice as big as the
standard error so that if we can assume a t distribution here actually a normal distribution
here due to the large to due to the large sample then we know that this effect here is significant
how large actually is the sample can be in can we invoke the center limit theorem here to argue that
this regression coefficient has a normal distribution or that the test statistic
which tests that this regression coefficient is different from zero has normal distribution
when you see this here the sample size is 5.5 681 and this is in line with those high attrition
rates which i showed you which were about 50 percent you recall we had 11 600 students when
the whole experiment started and after the experiment was concluded we just have about
half of it right 5681 so in this case we did estimate or we do estimate a positive effect
of small class sizes you can see from the regression coefficient here already that the
di variable was um designed as one would expect namely di is equal to one if a student is in the
small class that's the treatment right it's not the regular class which is the treatment that could
also be formulated that way but here we formulate it as the small class being the treatment because
it is a positive effect here which is being estimated and we know the positive effect means
that the test score is somewhat higher than for the students in the in the regular classes so we
must have the di variable equal to one here in order to explain this coefficient of 4.82 in this
table so you can infer from the table already how the dummy variable was constructed all right now
is it a problem that we do not include any other regression regresses so there's no dummy variable
for white asian and you may ask why actually is there not right there's no indicator for girls
which i didn't have as a characteristic in the previous table no indicator for free lunch no
indicator for the question whether the teacher is white or some other race and no indicator
for the teacher's experience for the master's degree um so why did we not include these variables
and is the regression um still useful and informative if we omit all those variables
and yes indeed uh this is still informative regression and is well specified and we do not
need to include these variables here because all those characteristics should be randomly
uh assigned or should be distributed randomly between the control group and the treatment
group so they should not have any explanatory power for different test scores because there
was random assignment right and the result here is actually in line with that
now there are many things which perhaps cannot be controlled for like for instance
the quality of the school right so what researchers did that they also included school fixed effects
in the second column here still nothing for the observable characteristics but school fixed
effects and we see that with school fixed fixed effects something which cannot be um randomly
assigned because we cannot take students and just randomly assign them to schools which are
perhaps very far off their their their parents home right so that's not possible so we cannot
really um have full randomization here and if researchers now control for school fixed effects
so each school gets a separate dummy variable um then we see that this coefficient here increases
considerably from 4.82 to 5.37 so the effect becomes even stronger and the standard error
decreases by quite a bit so we are here now in the range of the key statistic of form
right so highly highly significant effect now very good development actually that the standard
error goes down by so much still we have not controlled for all the other things which have
been actually handled by randomization if we do what happens then well you see if we include for
instance variables for white asian yeah so dummy variable which is one if the student is right or
of asian descent or if we include a dummy variable for girl or boy right a girl would be one in this
setup here or if we include a dummy variable for the free lunch question so one means yes the
student receives free lunch then we do get um significant statistics here too and so it does
play a role when someone is white or asian typically people students from white or asian
families have higher scores as you as we see here this is a positive coefficient dummy variable
is one for white and asian uh so this clearly indicates uh the white people have had the better
test scores or the white and asian people have had better test scores the like
girls apparently had better test scores than boys right this coefficient is also significant
actually i didn't mention it here but you see it the coefficient is highly significant 8.35
with a standard error of 1.35 is a tremendous t statistic right in the area of five or six or
something like that uh here same thing probably a t statistic of seven or eight or something like
that right uh for the girls the girls have much better uh test results in the boys this is what
this variable indicates to us here and free lunch well if free lunch means that students are coming
from poor families then you see students coming from poor families have much weaker uh test results
very very high significant uh significance for this coefficient here as you can see
because the standard error is just 0.77 and the estimated coefficient is 13 so probably
the t statistic is in the range of 20 all these factors here influence test results but as you
can see they do not change at all or hardly at all uh the small class size um regressor here
because we have had randomization right so these regressors here are actually orthogonal
to the small class size regressor so they don't play a role they they do not correlate
with a small class size regressor this is the meaning that they are orthogonal to the small
class size regressor and therefore as you know orthogonal regressors do not change
the coefficient estimates of other regressors to which they are orthogonal so hardly any effect
5.36 stand are almost the same 1.21 and if we include further information further characteristics
where the teacher is white well you see this actually is unimportant right so this coefficient
here has a negative sign but it is insignificant because the standard error is much louder than
the the estimated coefficient so therefore apparently the race of the teacher does not play
a role teacher's experience however is important you see that teacher who have higher experience
achieve better learning with a standard error of 0.10 gives a t statistic of 2.6 which is
significant whether the teacher has a master's degree or not is a question which in germany
would be irrelevant because usually you would not be able to teach as a teacher if you don't
have a university degree at master's level here you see that it actually is unimportant in
tennessee so apparently in tennessee schools some teachers are employed which do not have a
master's degree and it seems they don't teach any worse than those teachers do which have a
master's degree well the coefficient estimate is negative but insignificant so and now also observe
the r squared um the r squared was very very low uh as long as we did not have the school fixed
effects obviously there were many many many schools in this this study because we think about
5,700 students or outside 11,600 students right so the school fixed effects are extremely important
we just had one percent r squared in the first regression still the small class size regression
was the small class size regression coefficient was fine it was significant right we had already
the basic result here but when we then also control for school diversity for school fixed
effects right by including lots of dummies here then we see when the r squared increases greatly
from one percent to twenty five percent and with more regressor we have additional uh
increases in r squared 2.31 percent in both columns three and column four
yeah is there anything else to say i think everything i have on this slide i have said
yes indeed um why include covariates in the rct so why have the regressors on the age of the student
the race of the student the free lunch and these kind of things well um they are also important
explanatory drivers of learning uh success right as we have seen so it is not the case that only
the size of the classes matter and we can when we are already analyzing this question also identify
other factors the only thing we have to make sure is that there is no correlation between
our regressors and the error term or between our regressors and the
the potential outcomes in the case of non-treatment so this means that the individual
characteristics should be balanced across the class type so that the regressors are actually
offer enough okay if the covariance the the covariates so i call them x i here
were uncorrelated in expectation with the treatment variable d i then we can simply estimate
the delta again with additional variables as we have done in the previous table right so there is
no problem with that and we should get a delta coefficient which is very similar or an estimate
of delta in this what is called the long regression which is very similar to the delta in the short
regression where we do not have the covariance so the estimation result here for this delta
and for this delta should actually be very close to each other and it is possible to base a test
on the closeness of these two delta estimates here and there if the two delta estimates are
not so close to each other then this indicates that randomization may have failed right and you
see this again here in the table these two delta estimates are not all that close to each other
and one could conduct the test now are they actually the same thing or shall we conclude that
in terms of the school fixed effects randomization has failed because we couldn't really randomize
schools we couldn't send children just to arbitrary schools so here clearly is a weak
side of the Tennessee Star experiment that we couldn't randomize one school and we could
base test on that it was quite some likelihood that this test would be significant but for all
the other covariates which we have here or even in the long column of four we see that the delta
estimate 5.3 c7 5.36 5.37 hardly changes when we introduce further covariance which suggests
there i think we wouldn't even need a test that randomization did work well for these characteristics
okay um i think i'll leave it at these comments are there any questions
seems this is not the case then thanks for your attention and as i say next week monday
the lecture is not live will be uploaded in the next days so certainly before the weekend
and please listen to the lecture before we meet again on tuesday afternoon next week