Hello and welcome to today's lecture.
Today we are going to talk about statistical significance and samples and sample bias.
So this is really again a refresher for some of you but nonetheless because it's so important
let's go right into this.
First let us think what an academic study wants to do.
Academic studies do tell a story.
They want to convince you of their findings.
So every academic article basically tries to tell you a narrative, tries to tell you
a story.
Well nearly all of them, not really all.
So and in order to do so, in order to convince you that the story is actually a true story
often statistics are invoked.
And these statistics are basically the proof in the pudding that what the story is supposed
to be that this is true.
convince you with big sample sizes n is 4 million this must be true whether the authors
claim.
One of the goals of this course is to look behind that curtain and to become a bit skeptical
whether n is 4 million automatically implies that what the authors claim is really true.
So how are we going to go about to achieve this?
Today we are going to look at kind of what does it mean to prove something in a study
and today we are going to talk about quantitative studies.
So what does it mean to prove a hypothesis, a brief refresher.
So we are going to talk about confidence intervals, about type 1 and type 2 errors and see that
we kind of put them all together.
So then the next topic that we'll cover is significance, statistical significance and
significance level.
We will talk about why statistical significance can be a problem.
It's certainly useful and it's but it's also problematic in many ways in understanding
how and why this can be problematic is very important.
We are then going on to sampling and look into sample biases and see whether big what
the sample size for example makes a difference and if a big sample size automatically helps
us get a better study.
We're going to wrap up the whole thing by looking into the issue of correlations and
call sessions invoking the most favorite quote that all reviewers of academic articles
are invoking very often.
So let's get started with quantitative empirical research.
What does it mean to prove something, to prove a hypothesis?
Well actually it's very hard to prove that something is true because there might always
be a counterexample that you have not thought about.
So gaining scientific knowledge basically follows Karl's proper critical rationalism
which says we can never prove something but we can falsify and reject something.
So this is what Karl Popper, kind of the look up the term critical rationalism if you're
interested in more, basically if there's a single counterexample then we can reject
the hypothesis that we have posed.
And this ties nicely in, do you know that joke when the engineer, a physicist and a
mathematician go in Scotland by train?
It's a pretty famous one and a pretty old one too.
So they look out of the window and they see black sheep in the grass and the engineer
says oh look Scottish sheep are black and the physicist replies well at least some Scottish
sheep are black and the mathematician looks very irritated and he says well we can say
that there's one field in Scotland where there's at least one sheep which has at least one
black side.
Well you knew that joke but we can never really prove that all sheep are black because somebody
who shows us a white sheep basically has rejected our hypothesis.
But how can we prove something in science when we only can reject things?
Well the way we often go about is to create the counter-hypothesis which claims the opposite
of what we want to show and reject this because when we reject the contra-hypothesis then
we kind of land support, we kind of show that the real hypothesis must be true.
So in quantitative research we often test something by creating the opposite hypothesis
of what we want to achieve, the so-called null hypothesis.
And I apologize if I choose an example which seems a bit politically sensitive but there's
a reason why I choose this example so please bear with me.
You will learn about this a bit later.
So let us assume we want to see whether black people are treated differently than white
people with respect to red cards in soccer games.
Do they get red cards more or less often than white people?
Well the hypothesis one basically says the probability that you get a red card if you
are a black person is different from the probability that you get a red card when you are a white
person.
If we create the inverse hypothesis, the null hypothesis, that would then mean the probability
that you get a red card when you are a black person is the same as the probability that
you get a red card when you are a white person.
We can of course rearrange that nice mathematic math forms formulas and then basically test
this.
So to see whether we can reject the null hypothesis thereby lending support to the
real hypothesis H1 that we want to examine.
So how do we do this testing?
We are not going to go into all these statistical methods but basically in order to reject our
null hypothesis, which lends support to the real hypothesis, we do some statistical test.
This could be regression or some other test and in most of these tests we get a so-called
p-value probability as a result.
And this p-value describes the probability that we actually measure what we measured
if the null hypothesis were true.
So a high p-value means the null hypothesis is very credible.
It's very likely that the null hypothesis is true.
The very low p-value leads us to conclude that the null hypothesis is not very credible,
it's not very plausible.
So the null hypothesis is something that we can reject.
And if we can reject the null hypothesis again, that means we support our main hypothesis.
So the lower the p-value, the more likely it is that our real hypothesis kind of shows
something.
Okay, so let's kind of dig a bit deeper into the whole p-value thing because that leads
us directly to the way to statistical significance.
How do we do that?
Think of a normal distribution.
And so p is a measure of the credibility of the null hypothesis.
The lower the p-value, the more likely it is that the null hypothesis is actually not
true.
So but what is a low enough p-value to say now we can reject H0, the null hypothesis?
Well, that is basically, there is no real answer.
There are conventions.
And some conventions say like, oh, if we have a 5% probability, then this is okay.
Because then there's only a 5% chance that the null hypothesis is actually true with
what we have measured.
So this basically shows you a one-sided t-test in that chart, which is a one-sided test with
a threshold, then alpha threshold of 5%, saying like, if our p is lower than 5%, so zero,
0.05, then we say we can reject H0 and thereby lend support to our real hypothesis.
That's 5% is something that's often chosen, but there's lots of other thresholds too.
So the alpha that threshold is what we call the statistical significance level.
So if our significance, if something is significant at the 5% level, then our alpha is 5%, there's
only a 5% chance that our null hypothesis is actually true.
And if we don't use a one-sided test, but a two-sided test, then you can see in that
chart that there's, of course, there's two and a half percent on each side of our distributions
basically that we look at.
And within that, that is what we call the confidence interval in that area, which is
mu minus two times sigma, and mu plus two times sigma is what would be our 95% confidence
interval.
Because if we say the real mean value is mu, we measure any value which lies within this
range, within this confidence interval, then there's a higher likelihood than 5% that the
null hypothesis is actually true with what we have measured.
So this is basically what you see.
And what is interesting is that, and take note of that, that our confidence interval,
which is kind of important when it comes to statistical significance, depends on the size
of sigma, the standard deviation.
We'll come back to that because standard deviation has an important relationship to the sample
size.
Okay, so what can we derive from these values and what can go wrong?
So let's start with a box in the upper right hand, which is when we actually, when actually
the null hypothesis is true.
Let's go back to our example, which said, okay, the probability that you get a red card
is different if you're black or white.
And so if H0, the null hypothesis were true, then basically there is no difference in terms
of probability.
Type 1 error is when we say we reject the null hypothesis and we say, oh, there is a
difference because we let this land support to the real hypothesis that we wanted to show.
But this is the case in the box where we're wrong.
So when we reject the null hypothesis, although it is true, this is when we talk about the
type 1 error, which is often the worst type of error that we can have in our testing.
And the probability that this occurs is the probability alpha because within this threshold,
this is the likelihood that H0 is actually true with what we have measured.
And we call this alpha the level of the test.
And if this error occurs, it could be that we accuse referees of racism of, for example,
because we say, oh, there's a difference in terms of red cards for black or white people
despite it not being true.
So this is kind of a bad error.
The opposite side is the type 2 error, which is if the null hypothesis is actually false.
So H1 is true and there is a difference between black and white people in terms of red card,
but we accept the null hypothesis.
We say, oh, we cannot reject it, basically.
We cannot really accept anything, but we cannot reject the null hypothesis.
So we cannot reject the hypothesis that basically there is no difference in terms of getting
red cards.
So type 2 error in this example would be, we say, oh, we can't say there is any difference
between black and white people, but in reality, there is because our null hypothesis is false.
OK, this is basically statistics 101, so I'm not going, this was a brief refresher.
I'm not going deeper into this.
Let's talk about the standard deviation.
Sorry that the formula looks a bit weird on the screen.
But the standard deviation, which is the thing that determines the width of the confidence
interval, the bigger the standard deviation, the bigger the confidence interval, and the
smaller the standard deviation, the smaller the confidence interval.
The standard deviation is the square root of the variance of X, so, and if you look at
the formula and think about your math knowledge from a bit back, then you can see that as
n increases, because of this 1 divided by n term, sigma decreases.
So with a higher n, which is the sample size, the number of measurements, with an increasing
number of measurements, with an increasing number of data points, your sigma, the standard
deviation shrinks.
In other words, the larger the sample size, the smaller is the confidence interval, and
the smaller the confidence interval, the easier it is to reject H1, because the acceptable
level of the confidence interval is really, really small, and it's very easy to measure
something which lies outside the confidence interval.
And if it's outside the confidence interval, then we can reject H0, and that is what we
call statistical significance.
So sample size makes it easier, or larger sample sizes, makes it easier to reject the
null hypothesis, thereby lend support to your real hypothesis.
So given you have large enough numbers of sample sizes, everything will become statistically
significant.
Now you know the secret why people love or why statisticians love big data analysis so
much, because if you have big data analysis, large ends, then it's very easy to achieve
statistical significance.
If you have small sample sizes, in your master thesis you ask 25 people something and crunch
the data, then it's very hard to reach statistical significance.
So problem solved, big samples, statistical significance reached, actually there is a
problem with the term statistical significance.
Now I looked up my basic statistics textbook, and this is a literal quote from it, there
is a problem with the term statistical significance.
It's a technical phrase that simply means enough data has been collected to establish
that the difference does exist.
It does not mean that the difference is necessarily important.
So statistical significance only means, okay, we have enough data, so we can say this is
not just a coincidence, there is a systematic difference.
But it does not say, for example, is the difference important, is the size of a certain effect
big or small, it could be miniscule.
Something that is statistically significant could have a tiny, tiny, tiny impact on your
dependent variable, on the resulting variable.
And remember that alpha term, the significance level, so we could use 1%, 5%, 10% level,
and then any of these are being used in regression.
And typically you see in a regression table one star, two stars, and three stars denoting
these different levels.
So, but for example, if something is significant at the 10% level, that means in 10% of your
cases you are going to be wrong when you say, oh, there is a statistically significant effect
there is something going on.
It might in 10% of all cases, you will reject the null hypothesis wrongly.
Okay, so the second problem with the term statistical significant is that then authors
in the articles knowingly or unknowingly of right, or the effect of this and that thing
is highly significant.
So something is highly significant in, when we read something like that, it means, oh,
it's important, but this is absolutely not what is being meant when something is statistically
significant.
It does not mean it's important.
It could be a tiny, tiny effect.
So this is really a trap and significant coefficients can be a marginally small effect
and be practically irrelevant.
Statistically significant or something is statistically significant does not say nothing
about the size of the effect.
The other hand, if we find something is non-significant or not statistically significant, does not
mean on the contrary that it has no effect.
It simply means we have not collected enough data to show that there's enough effect.
It could also mean that we do a linear regression and there's not a linear relationship between
the independent and dependent variable, but some other kind of relationship.
So non-significant does not mean a lot.
So if something is not significant does not mean necessarily mean it has no effect.
Okay, so this is really important.
First of all, sample size impacts how easy it is to reach statistical significance.
Second of all, statistical significance does not say something about is something important
or has a big impact.
And if something is non-significant, well, it does not really say a lot.
We cannot really deduce from it.
Oh, that's not an important impact.
It has no important effect.
So this is really, really key.
Okay, let's go back to our card example with the red cards and I will show you why it shows
that sensitivity example.
Think about, do soccer referees give more red cards to dogs or a different number of
red cards to dogs, players and light-skinned ones?
This is a test where teams of statisticians were given the same set of data, games, red
cards, skin color, etc.
And this was basically a contest with 29 teams taking part with a total of more than
60 analysts took part.
From these 29 teams, 20 concluded that yes, dog-skinned players get more red cards than
white-skinned, light-skinned players.
And nine teams said, oh, there's no significant relationship between skin color and red cards.
29 out of 29.
So professional statisticians getting the same set of data will not always give you
the same results.
It depends a lot on kind of what do you consider as an acceptable threshold?
What kind of analysis do you run on that?
What kind of data cleaning do you do with your data and all that?
So this kind of gives land support to the old saying, only trust statistics that you
fake yourself.
Getting as a result or getting something is statistically significant or not does not
mean that everybody else would also reach the same conclusion.
Okay.
So we said large samples, statistically significant, is easy to reach.
So we gather large samples, big data, and we are fine when we get better results, when
we get better analysis, truer results.
Similarly, there's another saying, which is about one million flies can't be wrong.
And just because you have asked or drawn from a big sample size does not mean you get more
representative data, for example, if your large sample is systematically skewed.
A smaller sample set, which is really representative of a population, can give you much better
results than a large sample, which is systematically skewed.
So large ends that can be skewed, non-random skewed sampling.
So if you do a convenience sampling, there's an example in 1936 when readers digest, try
to predict who is going to be the next president.
Is it going to be the Republic or the Democratic candidate?
They predicted the Republic candidate will win by a large margin.
It turned out the opposite was true and Franklin D. Roosevelt became Democratic president of
the USA.
So how could readers digest this newspaper be so wrong?
Well, they asked people getting the sample from a readers of their own journal, of their
own newspaper, which are conservative leaning.
They got them by looking up registered car users, which in 1936, were mostly rich people,
which tended to be more Republican than Democratic oriented.
And by looking up phone lists of phone numbers back then in 1936, this was also much more
a domain of rich households and people.
So they selected a very large sample of Republican leaning people and predicted something wrong.
Big sample can be wrong.
Another famous sample that is kind of has been disputed and analyzed a lot is the Hawthorne
Lightning experiment.
Back then, researchers tried to find out this better lightning in the factory hall make
workers more productive.
So they took a subset of workers and modified the lightning and our lighting and then looked
do they become more productive and it turned out yes.
Case closed.
Well, for one thing, it turned out they always because they had to change the lighting, and
they did this over the weekend, so that they always measured on a Monday whether productivity
was increasing.
One could argue that perhaps on Mondays productivity is higher than on Fridays.
So people had to kind of volunteer to get into that sample.
And if people were not really cooperating, they were kicked out of the sample again.
So this was not a representative sample of all factory workers and is thereby highly
disputed finding basically for people have argued about that for four years.
Another thing where bias in your sample can come in is the so-called survival bias.
And survival bias is when a part of your sample drops out or a part of your population drops
out of the sample and thereby faults excuse your results.
Survival bias because typically when this is in medical studies when people actually
die and you do some before and after test and you in the after test you can only test
those who are still alive.
So and all those that have died are not part of your sample anymore and thereby you might
get the wrong example.
Let me give you a different case or context of the survival bias where in management.
Consider three firms.
And these three firms have a profit of 8, 10 and 12 percent profit margin.
So on average we can say in T1 in our year one that there's an average profit of 10.
Three firms making our market.
Now think next year.
Next year one firm has gone bankrupt.
It's out of business.
The second firm which used to have 10 percent profit margin still has 10 percent profit
margin and the third firm which used to have 12 percent profit margin only has 11 percent
profit margin.
So what's the average profit margin in the next year?
It's 10 and a half percent rather than 10 percent before.
So if we look at the averages we can see the average profit of firms has increased between
year one and year two.
Fantastic news, right, isn't it?
Actually no.
One firm has gone bankrupt.
One firm has the same profit margin and one firm has a lower profit margin than before.
This is exactly what survival bias is about.
Also unobserved samples could be if you consider why men might have a higher wage than women.
And you look at the wages of men and of women and you conclude oh there's a gender wage
gap.
And don't get me wrong there is a gender wage gap and it's real and it's true and it's there
and I'm not for it.
But then think hard again.
Are we really seeing that or can we really say that women are offered a lower wage than
men?
Actually no.
Because what we cannot observe is the offered wages because we don't see the declined offers.
So declined job offers are not part of our sample.
There's a systematic part missing.
And an alternative explanation could be that well women accept lower job offers and men
reject lower job offers and they are thereby not visible in our sample.
That could be an alternative explanation.
There's lots of other examples of biases that could affect your sample.
So most common think about self selection bias.
So who is more likely basically to respond to a questionnaire to a survey.
It's often those who are somehow interested in the topic.
Therefore think about non-response bias where people who get an invitation to participate
in a survey simply do not respond.
If there's a systematic bias that young people don't respond to the same survey or more young
people do not respond to the survey than old people then the sample will be biased
towards old people and will be skewed towards older people.
And this is a bit of a problem in many many surveys.
Look at something that I collected some time ago when Macron was elected as president in
France and basically the German newspaper Schpiegel had this kind of online poll saying
like so how do you see it?
Yes the European shift to the right has been stopped or no the election result is no reason
to be happy.
What can possibly go wrong with a sample and a questionnaire like that?
Is it scientific?
First of all only people who are Schpiegel readers will participate.
Second of all only those who have actually interest in France or this whole topic because
they have opened this article proactively see this invitation for poll.
And thirdly only those people who have actually kind of an interest in responding and don't
say like I don't care will respond to that.
So sample is really really and the sample is going to be really really biased in this
thing.
And second of all look at these question and the answer possibilities they are suggestive.
So yes the European shift to the right has been stopped kind of includes a political
opinion and no the election result is no reason to be happy then this also kind of
includes a political opinion.
What if I think yes the European shift to the right has been stopped but I'm not happy
about it because I like the European shift to the right I'm not but assume I would be.
What should I click there?
So this is a kind of an example of priming your respondents of suggest having suggestive
answers and kind of pre-formulated reasoning behind the answer which is not scientific
at all.
But let's get back to sample sizes even if you have really big samples that does not
mean at all that the result is correct or better.
In 2018 the European Union did an online survey when it asked about so should people actually
or should we abolish the summer saving time and 4.6 million people responded and 84% of
those respondents said yes stop this and don't have summer and winter time.
And what happens then is European commissioner says like the message is clear 84% do not
want the clocks to change anymore we will now act and prepare legislative proposal.
So these surveys are being used to justify laws.
Does it really mean 84% of the respondents are really in favor of the EU or 84% of the
EU population is in favor of putting an end to the clock change?
Actually not at all because if you drill into the data which I've done you see that some
countries have a much higher representation than other countries Germany for example is
very strong in this sample.
And second of all and I can tell from experience of course my vote is twice within these 4.6
million responses.
So actually people have been responding multiple times to this poll if they had a strong opinion.
So who is going to respond to this survey?
You have to go proactively on the website I perhaps you even have to register an account
and then kind of click on that survey who is going to fill it in?
Those people who have a very strong opinion about the time changes and these are those
who don't like it.
I'm freaking leave me alone with the time change I'm fine with it.
These people are not going to go to this webpage and participate in the online survey.
So no surprise that 84% of the respondents have had a very strong opinion.
We can see the very same thing for example when you look at Amazon product ratings that
only people with one star rating and people with a five star rating are the majority
of rates on these product platforms because if you are extremely happy or extremely unhappy
with the product you are much more likely to go to that website and rate the product
than as you say okay it's okay three out of five and fine not bothering with entering
a product rating.
So this a big sample does not automatically mean that you get better results.
Also very big samples can be very very skewed.
Okay now let's move on to a different topic.
Even if you have a statistically significant relationship that does not and absolutely
does not mean that there's also a causal relationship.
One of the most famous phrases of article reviews is correlation is not causation.
Just because there's a relationship does not mean one thing causes the others.
So let's go back to our silly example of the red cards.
Even if there were a difference in terms of red cards for black and white players there
could be other explanations rather than racism for example.
It could be that black soccer players are more kind of in the front rather than in the
defense of the team and the front players are more likely to get red cards.
One explained without any racism involved.
It's simply because black players play in specific positions more often than others.
One alternative explanation.
So I'm not going there deep deeper there but perhaps you get the point.
Let's have a kind of other example.
Here's something which has a very strong relationship.
North of Sweden's Kiruna and the snow height that you have in Kiruna is strongly negatively
related statistically significant to a volcano that you find in Chile.
Very strongly negative correlation and it's statistically significant.
So does it mean that the snow in Kiruna causes the snow in Chile to melt and the snow in
Kiruna melting causes snow in Chile to fall?
No of course not.
The seasons on the north and the south side of the world and when it's winter in Sweden
it's summer in Chile and the other way around so there is a strong relationship but the
seasons on the driver for both things to occur and if you did linear regression you would
find there's a strong relationship between these two variables but there's no causal
relationship.
So just because you find a causal relationship just because you find a correlation does not
mean that there's a causal relationship and if you look at some hypothesis in academic
articles you will often see that they are phrased like A is positively related to B.
This is very carefully phrased so that there's no causal relationship assumed.
So if you have a hypothesis like that in the end in the discussion section the authors
say oh we have shown that A causes B then the authors claim more than they can actually
show from what the hypothesis basically has shown.
Correlations are fun.
Correlations are really really fun and sometimes it's kind of very intriguing to have these
high correlations.
This is a chart showing the correlation between the US highway fatality rate and fresh lemons
imported to USA from Mexico and you see that there's a very strong statistically significant
negative correlation between one.
So if we did that study basically and we find this relationship should we say oh let's import
more lemons from Mexico to reduce dead people on the American streets, the number of dead
people on American streets?
Of course that is silly.
It's simply that perhaps that cars get safer over time and imports increase over time during
that time frame.
This could be anything.
These correlations are so much fun that there's even a website dedicated to that and spurious
correlations and they show you fun stuff.
One correlation that I like a lot is that the number of Japanese passenger cars sold
correlates very highly with suicides by crashing of motorway vehicle and the R is really 9
0.93 so very close to one.
So what conclusions can we draw about driving a Japanese car on the US roads then?
Actually don't let us get there.
It's a correlation and of course driving a Japanese car doesn't make you want to commit
suicide.
Here's another example of samples and I try to pick that because it's so easy to conflate
correlation and causation and implicate some causation when there might be not.
That example is California where prison inmates can volunteer to fight fires and that's each
year at least before Corona, 4000 of them which saves California lots of money because
they don't have to pay firefighters and what is amazing is that the recidivism rate, so
the rate with which the people are going to commit crimes again for inmates who have completed
the firefighting program is 10% lower than the general inmate population.
So prison firefighters commit less crimes in the future than the general inmate population.
Wow and of course CNN then reports the program helps lower the likelihood that inmates commit
another crime.
Stop there, careful, see what they just did.
They took a correlation and they made it a causation.
The program helps lower the likelihood is something different than inmates who complete
the firefighting program commit less crimes again than the general inmate population.
Let's look at that a bit more closely.
If you think about California prison inmates there's about around 130,000 of them at least
there in 2017 and the recidivism rate is 60%.
So 60% of those inmates are going to commit another crime.
These 4000 firefighters are the sample of the whole population and they have a crime
committing rate of 50% so 10% less.
Let's think about how we get to the sample of these 4000.
Well let's check what does it take to become a fire camper.
So only minimum custody inmates are eligible to volunteer for an assignment.
So if you are a sex offender, arsonist, murderer, if you have a history of escape with force
or violence then you are out.
So basically you have to volunteer for the conservation camp to become a firefighter.
You have to have an interest in that and you have to be a minimum custody inmate.
Okay if you sit in there because you had 2 grams of mariana in the pocket then perhaps
you are a different type of person than as if you are a serial killer and psychopath.
So already there big difference in terms of who is going to be in that sample and who
is going to be in the whole population.
The inmates must earn the right to work in the conservation camp by nonviolent behavior
and conformance to rules while they are in prison.
So only those who are already behaving nicely in prison are allowed to become firefighters.
No surprise there behaving nicely after they are freed too.
These volunteers are carefully screened and medically cleared on a case by case basis
before they are accepted.
So again another hurdle where some types of people do not get into there.
And you have to have physical fitness, you are trained, you have classroom instructions,
field exercises, tests and only those who survive all these hurdles basically get to
be part of the sample.
Did somebody just say systematic bias in your sample there?
Okay so that sample is definitely not going to be representative of the typical prisoner
in California, right?
So we cannot really say that these firefighters or that the program makes the firefighters
commit less crimes in the future but only those prison inmates that are likely to commit
less crimes in the future anyway perhaps are being selected in that as firefighters.
So while the firefighting program is a great money saver for California, it's great for
prisoner self respect, it creates job opportunities in the future and perhaps it does impact the
likelihood that you commit another crime.
We cannot really claim from this study that there are from these numbers that there is
a causal effect that we can apply just by looking at this crime rate.
And think about this, how would you set up a study actually testing whether such a program
would make you commit less crimes in the future?
It's amazingly tricky and difficult.
Think about that, how would you set up a study to test this in a good good way?
So correlations are not the same as causation but this happens a lot and this happens a
lot if media site scientific studies for example.
So if you look at the study that found that respiratory mortality is considerably higher
for male and female cleaners than for non-manual makers of workers and that male cleaner that
it's higher for male cleaners than for female ones.
This is what the scientific study found.
So then the German media maker out of it, men cannot put, they risk their life, their
health, their mental state and much stronger than women.
So or who cleans a lot dies earlier.
Drawing increases risk of dying.
This is what the news make out of these studies.
And no, this is absolutely not what these studies claim.
But drawing causal conclusions from a relationship study is very easy to do and it sometimes
also happens, researchers, sometimes they are not aware of that, sometimes they do it
in their study and this is not a good way of doing things.
And of course there's not only correlations, there's also the moderators and mediators
of course that can explain relationships in an interesting way.
Thank you very much for listening and goodbye.
Thank you.