Welcome, everybody. Good morning, good afternoon, or good night, wherever you are. Today we
are going to continue our crash course on methodology and assessing research.
I would say just let's get started with the lecture and see how far we get. Today I want
to talk about assessing research quality. In a way, this is about reliability, it's
about validity. This is one of the most often used constructs that we have. So I think it's
important that we really understand all that. And I do hope you see my face and my slides
somewhere on your screen, then everything is like it should be. Okay. So this is what I want to talk
about today. We are going to talk about case study and case study design later,
mostly because it's so often used case study design. But the first part of today's crash course
is really dedicated to validity and reliability. So we are going to talk about internal-external
validity, what is construct validity, and what's reliability. And hopefully we'll find nice examples
as we go along that we can use. Okay. So let's get started. Validity is kind of the extent to which
we measure what we actually want to measure. So we'll try to find some examples. And we divide
that by there's plenty more types of validity just to take that away. But we are focusing only on
the most important and most frequently used ones. So this is internal validity that we start with.
And that is related to the causal relationships in the model. So when you imagine that your research
can be pictured as a model with boxes and arrows, then internal validity is concerned with these
arrows. So and basically internal validity says are the relationships between our constructs,
between these boxes, really in reality the way we claim they are in our model.
So internal validity is high if changes in the dependent variable are clearly and without doubt
caused by variation of the independent variable. So changes in our, oops, that's okay, the boxes are
wrongly labeled still. But so if the changes in the first box really cause changes in the second box,
then this is basically, and it works the way we claim it works, then we can say oh this is a study
with high internal validity. If there's alternative explanations, that could also explain
the changes that we might be seeing. That could be a threat to our internal validity.
So let's pick an example. If I have a study that looks into, well, higher pay of somebody
causes better verb performance. And we have established that link. We claim so just pay
people a lot so they will work great. Okay, that might be the case or it might not be the case.
It could actually be that jobs that have high wages attract skilled workers because they're
competitively priced and therefore we simply have a better work employee base and therefore get better
results. That could be an alternative explanation to the same phenomena. And if you remember last
crash course, that is basically a mediation. So attracting skilled workforce could be an
mediating variable here that really explains the link between pay and verb performance.
So if we do not include all the variables in our model that we need to include, in this case
measuring how skilled our worker base is, for example, by looking at their education,
by looking at their previous tenure and previous jobs or whatever tests we might have to test
whether we have a skilled workforce, then we could get wrong results and claim wrong results simply
because we had not included the variable the construct attracting skilled worker in our model.
So we need to, in order to address internal validity in a study, we carefully need to think
about what alternative explanations might be there that could also explain what we are seeing
there. And if we find credible alternative explanations then the researchers have not
addressed that is, well, bad. And that is bad for the internal validity of a study. So this is
basically the one thing that we need to know about internal validity. So addressing all rival
explanations, thinking about what else could explain the changes that we are seeing, including
the data in the analysis that we need. I showed you that by neglecting to include the correct
variables we could see relationships that are not really there but that are in reality mediated
by a third construct. Also to kind of convince my readers that our study has a high internal validity
I need to explain my data collection and my data analysis transparently. Because only if the reader
believes that I have done a proper and a good job at collecting and analyzing my data, they will
also believe us that we are actually seeing the relationships and believe our interpretation of
the relationships. So that's basically the most important things. Think hard about, if you say
analyze an article, think hard about what alternative explanations could be there.
Are the authors admitting or are the authors kind of telling you, oh, these could also
be an explanation for our findings. That's not a weakness if authors do that, right? So if
authors have thought about what alternative explanations could be there, that is a good thing.
Okay, so this is internal validity. Think about it if you think about models and I like to visualize
studies in kind of in some way and boxes and arrows are a good way to visualize
relationships between constructs. That means that the internal validity is concerned with the arrows,
not with the boxes. We are getting to what is concerned with the boxes in a brief while.
Okay, are there any questions so far to what internal validity is? Because this is key then
that you understand what all these validity are. If so, raise your hand. We can stop the recording
at any point in time if you want to ask a question that we should not record and then this basically
would be a good time. Otherwise, we'll have a question time after the lecture if there's still
kind of open questions. So note down your questions if you don't want to ask them now.
Okay, I think that is what I wanted to say about internal validity and I hope you got it.
Now let's talk about external validity. And by the way, if you want to read more about what internal
validity and reliability is, I have a very good book for you to recommend which is Robert Yin's
case study research book in which is a must read book in case you ever intend to write a case study
in your master thesis. This will be the one book that you will need and we will talk about case
study design a bit later which draws heavily from this book because this is the standard works on
case studies. There's another standard works by Robert Stakes but this is Yin is the book on
case study design and he has a nice few pages in which he condenses the validity and reliability
overview and explains it in a nice way. So if you want to read up on these Yin is a good tip.
So external validity, let's talk about that. It's the definition that establishing the domain to
which a studies finding can be generalized. So it's about the generalizability of findings.
And what do I mean by the generalizability of findings? Well, first of all, think about an
example where you examine corporate culture in a Japanese company and you claim well unwritten
rules are very important in companies. That's your study findings. I just made it up. So
what then is external validity? External validity is the extent to which we can think we can
generalize our studies findings. It could be that what we have found out is valid in all Japanese
companies but not in US companies, for example. In this case, I have a perfect valid finding in my
study but it's only valid in Japanese companies, in Japanese culture context. And this is then the
external validity basically would be we can generalize our findings but we can only generalize
them to Japanese culture and not any further. We cannot claim anything to know anything about US
companies, for example. I think you get the point. It could be that it's valid in an Asian context
and not only a Japanese context. In this case, external validity would be the Asian context.
We can generalize our findings to all of Asia. I'm sure that would not be the case but let's
simply assume. So then what does it imply? Whenever we think in our study about external validity,
we have to think and justify which is kind of the range, the extent to which we believe
these findings will hold true. We will see that in the case study a bit later. Case study is a
single case. Sometimes the external validity will be simply to my single case. We say, oh, we found
something interesting in this case but sorry, we can't claim that this will work the same way in
all other cases as well. So this is then basically the generalizability would be very small and this
is a per se criticism that case studies often receive that they might not be generalizable
because they address a small single case very often. But it could be that we crunch a database
and we say, okay, we find that if you increase a worker's pay, you will get a work performance
and that might be globally and universally true finding. That would be very high external
validity. So it's not that any study needs to be generalized as a universal world law,
but any study needs to think and establish the effect to which it believes it can be
generalized and justify why it is so. And for that we also need, just like in the internal
validity case, we need to claim, okay, this is because the culture construct works very differently,
has a different logic in Asian contexts than in Western contexts and or the values norms are so
different that the studies, the linkages and the constructs simply don't work out this way in other
contexts. I think you get the point. So external validity is just something that you should look
at when you analyze a study. What claims do the authors make for that external validity? Do they
say, oh, if A isn't the case, then B is the case, will be the case. That is a pretty universal claim.
And do the data, do the analysis that they have run actually kind of allow to make this claim,
or is that something where you could say, you claim you can't explain human behavior, but you
only tested it at marketing students of the first term. And I think your sample might be a bit
skewed because marketing students, you know, are a bit, I'm not saying simple there, but
yeah, a bit simple, perhaps in terms of how they think. Sorry, just kidding. But so this is what
could then be a threat to external validity of the claims the author make. Okay, assess that
whenever you look and want to assess the quality of office study. I think you've got the point.
What are threats to internal validity? Okay, you could, as I said, if you select
marketing students of the first term, you have a very weird sample that you look at a very kind of,
it's not the wrong sample, but very specific sample, which is not representative of any
population. So that sampling could be threatening your external validity. If you only look at Asian
companies, well, and look at culture issues, you might have a threat to external, global external
validity, basically. Also, if you look at a specific time or a specific location,
very specific context in general, that could be a threat to external validity.
So if you look at banks right during the financial crisis, your findings might perhaps not hold true
for banks in general in all points in time. I think you get that point. Okay,
so think about external validity when you design your research. Think about sampling
and the population that you want to address. So which is the population that you want to explain
something for, right? Do I want to explain something for the German population? Do I want
to explain something for the European population? Or do I want to explain human behavior in general?
What is the population that we want to predict behavior for? Also, the sampling is critical.
Which sample do I take? What are the times at which I gather data? In the last crash course,
I told you about the Hawthorne lighting experiments, where over the weekend,
light was changed to see whether people became more productive. And basically, what the study
showed is on a Monday, people are more productive than on a Friday. That is perhaps a threat to
external validity. We can show that our study might be well. It might not be well at all. I don't know.
But you get the point. So the very times of data gathering, very in kind of locations and firms,
if you want more generalizability, and see where you can generalize from. What is the context of
all my study objects that I have looked at? Okay, I get the point. If there's no urgent question now,
then basically, let us move on. This is what I want to say about external validity. External
validity means the general level of generalizability or the extent of generalizability of my study.
Let's talk about the relationship between internal and external validity. Because
there is a bit of a trade-off. I can try to improve my internal validity to learn about
the causal effects by spending lots of time with very few people, and try to kind of interview
them a lot, for example, or to observe them a lot, and then be sure that the linkages that I predict
are for the reasons that I claim they are. That would be a small number in-depth study
to actually learn about the causal effects. But if I only look at a few numbers in a very specific
context, that could be a threat to external validity. Because I don't look at companies in
15 different countries, but only at companies in two different countries. So internal and external
validity are, there's a bit of a trade-off in some cases, in some research design. Of course,
if you want more samples to kind of increase your generalizability, you can spend less time with
each person, and interview each person less, observe each person less, and thereby you can't
really kind of claim, yes, I am absolutely sure that the causal linkages work the way they are.
In the worst, let me call it the worst case, but this is real studies. If you want to look at, does
board diversity or does dual nationality on board memberships of a firm increase firm performance?
These are real studies that have been conducted. And you take a big database where you have the
nationality of the board of directors, and you have the number of the firm performance, you crunch
it, and you say, okay, we can say if there's more than two countries represented in the board of
directors, the firm will perform better. Okay, that might be valid statistical relationship, but
I have very little or a reason to believe that the relationship is actually for the reason why the
authors claim it is. It could be for any, it could be because big companies that perform better
tend to have more nationalities on board, and that's not that the dual nationalities cause
the better performance. So you see the trade-off that there is between spending lots of time
looking for causalities and crunching big sample sizes or observing big sample sizes, and then
not necessarily not as deep. Okay, I think that's all that I want to say about internal and external
validity at this point. Let's move on and look at the boxes in our model. So the boxes in our model
are the dependent and the independent variable, right? So the dependent variable is in the outcome
where the arrow points to, and the independent variable is kind of what goes into our model
at the beginning of the arrow, and this could be a variable, a directly measurable variable. So
return on equity is a variable, and we can have that number, and that is simple and easy, right?
One measure that represents this box. We call it a variable and fine. But some of these boxes are
not as easy to measure, and we need kind of a more complicated way to actually measure these
and to how we call it to operationalize these. So intelligence is a nice example for a box
that is not that we cannot measure with a simple single thing. Because intelligence is something
very abstract with lots of dimensions and lots of different kind of ways to measure it, and I might
need lots of tests and interviews and questionnaires to measure intelligence, and even then there might
be different types of intelligence, crystallized intelligence, fluid intelligence. Google it if
you're interested in that. Different types of intelligence, but you all know that kind of what
intelligence IQ tests measure is not always kind of the intelligence or the sole measure of
intelligence of a person. Just because you can continue a logical number row very well and quickly
does not necessarily mean you will be a very intelligent person in all situations.
So construct validity is concerned with, does do the kind of operationalization of
this construct, the way we measure this construct actually measures the construct in a good way.
Nice example to finish off that slide of a question which perhaps might be a bit critical
in your intelligence test. If you ask which city do the doctors play,
well you need some kind of specific knowledge to be able to answer that
question, and if you are not in the US and if you're not familiar with the sports world of the
US, you might not be able to answer that question, but that has nothing to do with your intelligence,
right? But these could be questions in any in an IQ test. So that would be a question that
obviously depends a lot on the context of where I ask or where I perform this test, whether it
actually is a good way to measure that construct. Okay so what are the, what is there for kind of
construct validity? Often a construct or a factor as psychologists call them, constructs these boxes
are measured by using different questionnaire items for example. Let's think about happiness.
So if we want to measure a person's happiness, we might want to ask several questions in a survey.
Are you content with your life? Do you sleep well during the night?
Do you have enough food on the table to survive? I don't know what questions
psychologists would put in any questionnaire that intends to measure happiness.
But it would be would have to be more than one question of course. And then we have a lot of
questions and a lot of answers that somehow together are supposed to measure happiness.
And if you look at these and if you have a study that has for example a questionnaire at
their basis. The first thing that that I ask myself are the questions available that have been
used to ask and to measure that construct. Of course I do like to look in the appendix,
to look at the questionnaire items, to see whether I would say yeah that actually measures
happiness. Or if the questions say okay how much do you earn and do you have a good job?
Then I would say sorry that does not measure happiness, it measures something work related
but not necessarily personal happiness. So look at can we transparently see how
constructs are measured or as we say how they are operationalized and are these credible ways to
measure the construct. And if yes if the single items together the way they are being put together
by the researchers because there's different ways to put together items then seem like a good way
to actually measure the construct then we have construct validity. So construct validity is
and I'm ignoring what I've written up there a bit now basically because it's so tricky
to achieve construct validity. One thing that I want to highlight is a be transparent to your
readers. So if you analyze an article are the authors transparent to their readers on how they
measure constructs? So can we see for example the questions they have asked and b what is
often done in a good study is that researchers don't create a fresh questionnaire from scratch
and say like okay we sat together over a bottle of red wine and thought what questions we could ask
to measure happiness and we are happy to have happy questions now that measure happiness. No but
look at what previous literature has done to measure happiness and build upon that and say
like okay there is a standard form questionnaire to measure individual level happiness that has
been used and validated by other researchers and thereby I use these questions and build upon
already validated research and even if the construct operationalization is not perfect
it's at least similar to what others have done so we can compare to their results.
So if you create a survey in a questionnaire or if you analyze an article that contains a
questionnaire or survey look at where do the questions come from and are there actually
references to a preview to where that where we have taken and we have taken inspiration
of the questions or we have even taken the whole questionnaire set for a certain construct.
Okay I think that is so much about construct validity. So construct validity to sum it up
is about the boxes in our model not the arrows and it is do we actually measure
what the label in the construct box claims to be measured and if that isn't the case if
that is credible to the reader we have construct validity. Okay I think that is as much as I want
to say about construct validity and with that I want to move on to reliability which is also
important and which is related to validity but not the same. So reliability is the degree
that it supplies consistent results. So one question that we can ask ourselves when it
comes to reliability if we would repeat the same in the study or if a different researcher
would repeat the same study or somebody would repeat the same study in the same context that
authors claim external validity for generalizability for would we get the same results.
So the aim is to minimize random error or coincidences that kind of lead to certain
results. Think of a measuring tape. If you jump and you measure how long you've jumped
you want the measuring tape to every time you measure the footprint to show the same result.
It could still be that the measuring tape is not on the tenth or a hundredth of the millimeter
correct but you want measuring tape to always show the same result when you measure and not
five meters and to one point and two meters at the other point in time. Okay I think you get the
point. So there's different aspects to reliability and one is stability. So over time will we would
we get the same results at the same point in time would we get the same results. Let me I'll talk
about interobserver and the form consistency there reliability there and then internal reliability
which will go back to what I have talked about constructs a little which is concerned with
constructs. So let's get started with stability. Basically this is the test retest reliability and
the stability part of it. If we performed the same research a bit later would we get the same results
and this as I said if you test once banks in a financial crisis and once five years after
and you get very different results then your reliability is very bad or if you get different
results your reliability is very bad in any case but that might be the reason why the reliability
was very bad. So test retest and researchers like to confirm previous research by confirming their
studies. See if some some medical science research claims to have found a vaccination for
or against COVID the coronavirus then a retest study should hopefully show the very same thing
and if not we have a problem with the reliability of our findings then we have to think about why
is that the case. Of course it will impede upon drawing any general conclusions
or theories from what we have found if the research is not reliable.
Okay but also at the same point in time there is reliability issues that we might want to address
or that we might need to address and one of these is so-called inter-rater reliability.
Sometimes you have a research with not only one researcher but you have research with
two or three or a hundred researchers observing performing questionnaires ratings things
and then you kind of the degree to which these raters rate things in the very same way.
This is what we call inter-rater reliability. Let me give you two examples. One example of
inter-rater reliability is a job interview. You are the job applicant and you have two psychologists
that sit on the opposite side of the table and that will have to assess your job job talk
and you actually want these two psychologists to agree on the assessment of you afterwards.
If one says this is the best candidate ever and the other says this is the worst candidate ever
they have very bad inter-rater reliability really. Then the findings of their interpretation
is a bit in doubt because it seems to be rather subjective.
But also it could be that if multiple researchers analyze and I have done such a study for example
encode interviews. You do interview codings and you want to assess or you cluster your sentences
in an interview text in happy sentences and unhappy sentences. Then basically you want these
multiple researchers to use the same standard of what is a happy sentence and what is an unhappy
sentence or they will get very different results and if that one researcher does all the US firms
and the other researcher codes all the Japanese firms we will find different things for the US
and Japan but not because things are actually different but just because they use different
standards in coding or subjective standards in coding. I think you get the point and there's
quantitative measures, quality measures that can be used to assess the inter-rater reliability.
So because how you address this is of course that you have an overlapping set of codings where two
researchers code the same thing and you look have they actually coded it the same way and they're
statistical measures that basically can give you a number that says okay inter-rater reliability
was 0.8 so this is then basically high enough inter-rater reliability something like that.
Good study which wants to be credible perhaps and that has multiple coders researchers will perhaps
talk about inter-rater reliability but also it could be that if you give two persons in an
department of a company for example different surveys that ask the same questions or different
questions but the same constructs you want these constructs to be correlating very highly so somebody
that a person that is very happy in form one should also be very happy in form two.
I think you get the point so that would be the parallel forms reliability that's kind of not
as frequently important as inter-rater reliability but it's there and you know it.
The last form of reliability that I want to talk about is the internal reliability
and internal reliability is again back to the constructs that we have because if constructs
use multiple indicators multiple variables to measure one thing then internal reliability
is concerned with the fact all these indicators actually related to the same thing are they
actually showing the same construct. Think about kind of a construct that is happiness with a
university mentor or satisfaction with a university mentor and you could ask three questions how
satisfied are you with the mentor how likely are you to return to the mentor in the future and how
likely are you to recommend the mentor to a friend they all measure some kind of satisfaction
aspect of the university mentor so and hopefully these assume you are asked them on a one to seven
liquid scale you want these answers to correlate highly or they are not measuring the same thing
right and this is what internal reliability is about so if you have multiple indicators for the
same thing are they actually homogeneous in the consistent showing the same thing.
As I said if I'm very satisfied with the mentor but I'm never not likely to ever return to it
something is internal reliability is really bad and something is going weird is going on there so
the homogeneity among items is what what we are concerned with that so how is it I measured as I
said you could simply measure the correlation between these three items that that we used as
an example and that is hopefully a high correlation you could also split your questionnaire
items we have three items in our example that's a bit bad but you could split it in half assume
we had four questions you could use two questions there two questions in there put them together
and see are the constructs really basically similar in terms of are there these two halves
very similar and put together any random combination if you put together any random
combination of two and a half and half possibilities then in essence a bit you are
measuring kronbach's alpha kronbach's alpha is a measure for the internal consistency of a scale
I'm not going into the statistics there but simply let me put it this way it is kind of a measure
for the how consistent how correlated the items multiple items in a single construct R and it's
a measure between zero and one and of course rules of thumb are always bad and wrong but the rule of
thumb of what an acceptable kronbach's alpha is is that if it's higher than 0.7 then we we say okay
this is an acceptable level of consistency between the items of my construct so we are
pretty sure they are actually measuring the same so if it's 0.8 0.9 that's even better then there's
a bit of a kind of classification system there that has been established over time but there's
also valid studies that have a kronbach alpha of 0.66 and they are still great so take these rule
of thumbs with a grain of salt again as always but this is kind of the one threshold that has
been kind of established for a while okay I think internal reliability is kind of pretty clear
and reliability and validity is used very often but as I said this is about boxes and arrows
often these are used in quantitative research of course we can have boxes and arrows and
qualitative research as well and often we will but in some cases if you have qualitative research
some researchers use different frameworks to assess research quality and I just want to
introduce you to one of these very very briefly because I want to show you how similar they are
in the sense to reliability and validity and I want to focus Lincoln and Guba's framework there
is pretty famous and it has a trustworthiness as one big aspect and I want to kind of go through
that and that is the credibility of your study in qualitative research it's important to be
credible to be believe that that readers believe your findings a quantitative study often has
statistics to kind of prove their findings qualitative research does not have statistics
to convince so qualitative studies need to be even make kind of more take more effort to
um be credible to be transparent to make the reader believe that the findings we found in our
interview for example are actually the way we claim them to be so that is about credibility
and um you you kind of will see that this is a bit kind of internal validity related because
this is also about kind of the are the the causal relationships the interpretations of what is
going on really what is going on it's not a one-to-one mapping but it's kind of uh related
transferability if you do a case case study for example um to the findings that we have actually
apply to other contexts so to a different case so in essence transferability is very much related
external validity are our findings of a single qualitative case study for example
generalizable are they transferable to other
dependability so are findings likely to to apply at other times you see that
reliability that play a role so if we repeat our findings will they or our study or use a
different context will we get the same findings really all right so that is but are they reliable
are they dependable than our findings and the conform confirmability sorry for that confirmability
could other researchers if other researchers would repeat the study could they confirm the
findings or are the investigators values and experiences and opinions and kind of the the
pink glasses through which the researcher looks at the world so important that basically they skew
and influence the results that they find to a high degree right so this is essentially
also a bit of of a kind of reliability aspect so would if the study were repeated by a different
researcher would they find the same things so this is a framework that is very often used when it
comes to qualitative research and this is trustworthiness i'm just going there's also an
authenticity dimension to this framework i'm not going there but i think these four are
interesting and you can see how much how similar they are in in terms to validity and reliability
okay no talk about validity and reliability must ever that's a natural law must ever finish without
showing the these rings the oh see i do bow shooting and i don't know how they're called in
in english so must ever finish without these dark thingies which shows the difference between
validity and reliability so if you repeat your study a lot of times and you get always the same
result that's in the first picture and you always get the same result you hit this the arrow at the
the the same point but it's the wrong point then your study is very reliable you repeat it you get
the same result but it's not a valid study the result is wrong due to whatever kind of
problems with validity the second image is you have a study that's simply not reliable and not
valid different results if you repeat it and all wrong the third one is the the one we aim for
that is it's reliable you repeat the study a different researcher repeats in the study
they get the same result and it's the correct result it's valid but it could also be the case
that is kind of the last one that we get kind of we are not reliable if we repeat the study we
get different results but on average we are not really systematically off target it's simply just
we surround our goal the aim the bull's eye but kind of we don't really hit it so the study could
be valid we could kind of have the right model we could have the right construct validity but
due to some problems with reliability we don't really hit on target all the time that could be
the case so now this is nice visualization of the relationship between validity and reliability and
how they can all be happen or not happen at the same time so sum the whole thing up in a nutshell
internal validity is about the cause linkages in the model are they the way we claim they are so
is the research done right and explained right are all the variables constructs that need to be
in the model to explain what we want to explain are they also in there external validity is about
generalizability and the extent of generalizability construct validity is do we actually measure what
we want to measure in our boxes and reliability is the stability and consistency over time and
over repetitions okay that is what i wanted to say