And welcome to the crash course on research design and different methods.
We are focusing on a few subsets of themes and topics today, but we will expand on this
name is bear statistical significance and the more issues on what can go wrong with
sampling next week and the week after when we talk about reliability validity and so
far.
So, a few subsets.
We asked first of all how can we assess what makes a good research design and of course
we need to look at what are the theoretical foundations that we use, that we combine to
create new knowledge, to create new insights.
So, if for example you use transaction cost theory as your basis, which is rather rational
view of the vault than where you try to minimize costs, but you try to explain the altruistic
behavior of people, which kind of with the assumption that people simply don't behave
rational, then this will not match.
There's assumptions in your one research stream that don't hold in the other.
Similar if you combine two streams of research, well, they need to be kind of coherent in
terms of the world view of the assumptions that they make of the vault.
So, this is the first thing.
Not all theory can simply be thrown together and will still make any sense or can help
you prove your point.
Well, what is it that we want to look at when we talk about good research design?
There's a few basic distinctions and you should have heard about most of these.
But because some of you might have forgotten some of them or not heard about them, let's
brush up a bit on that and talk about this.
So, the first distinction that we can make when it comes to research, and that is not
what the general rule is, there is no general rule without exceptions.
Of course there's things that are not purely qualitative or quantitative, but most research
can be assigned to either the qualitative or the quantitative kind.
So, let's start with the easier way.
What is quantitative research?
Think about that.
How would you characterize or define quantitative research?
This is pretty easy to define.
It is all research which comes to its conclusion using statistical methods.
So this is quantitative research.
So then basically the rest nearly is qualitative research.
That means all research which does not use statistical methods to derive its assumptions.
That leads to an interesting point, because usually students and also sometimes researchers
think well, if I do an interview, that is qualitative.
It's because it's text, right?
Not necessarily so, because you can do quantitative research, and many people do, using text as
your basis.
So there's things like text frequency analysis, which look for example for the number of happy
words and angry words in an interview in order to decide how somebody feels about
a specific issue.
And you count words, you count frequencies, or you do a natural linguistic analysis of
the words in your text and calculate the similarities between two different statements
by two different people.
All this can be done and is being done, and all this involves basically statistical models.
So this is quantitative research.
So don't fall into the trap saying like, okay, interviews equals qualitative research.
It often will be the case, but it does not have to be the case.
Think about a few examples of articles or books that you've written, and think about
whether they're qualitative or quantitative.
This is a distinction that is so basic that you absolutely need to be able to make that
call.
To make things a bit more difficult, of course, some research contains elements of both.
For example, and that is totally fine and okay.
So it's totally okay, for example, to have a quantitative part of your study where you
try to identify a certain issue and come up with a new explanation, for example, and then
have a qualitative part of your research where you try to show support for your explanation
where you try to prove a hypothesis, for example.
So there are rare studies that combine quantitative and qualitative elements, multi-method studies
that try to combine both.
You've got the distinction between qualitative and quantitative research.
So let's look at the next big distinction that is often being made to differentiate
research.
This is inductive and deductive research.
I'm sure you have heard about these, and you're probably familiar with those.
But still, because it's so basic, we absolutely need to be clear on what it is and whether
it's okay to combine both elements in a single research.
So what is deductive research?
Hold on for a second, pause the video, and think about it, whether and how you would
explain deductive research.
Okay, still there?
deductive research involves going from general propositions, from theoretical rules and abstract
observations and hypotheses, to demonstrating, confirming these theories and hypotheses in
the field.
In that sense, they go from the very abstract to confirming things in the real world.
So if a law in Germany says shops have to close on a Sunday, on public holiday, then
we could create the hypothesis that, well, every shop will close in the pedestrian zone
in Hamburg, will be closed on a Sunday.
Then going out to the pedestrian zones of Hamburg and checking the weather shops are
really closed will lead to either a confirmation of our hypothesis using the field data we
gathered, or it will show us that some kiosks and some main station shops are indeed still
opened on a Sunday, which will lead us to reject the overly general hypothesis and rule
and we will have to rephrase and reformulate our general proposition.
That most shops that are not kiosks and not in the main station area have to close on
a Sunday, which is perhaps closer to the truth.
Okay, this is deductive research going from abstract rules, going from propositions and
existing theories, testing, confirming these in real world settings.
Inductive research is, of course, the very opposite.
So if I walk through the pedestrian zone and I discover all shops are closed and it is
a Sunday, I might be tempted to phrase the hypothesis and the rule that, oh, in Germany
shops have to close on a Sunday.
This would be a case of inductive reasoning going from real world field observations from
data, from very specific data, to formulating abstract rules and theories.
By the way, while we say we test hypotheses in a deductive manner, we create propositions
when we work in an inductive manner most of the time.
This is a hypothesis, something that we test in a deductive manner, a proposition is something
that we propose in our inductive or conceptual work.
Okay, so I think you understood the distinction between deductive and inductive, going from
the abstract to the specific to the real world data in the deductive case, going from specific
field data observations to formulating abstract rules and theories in the inductive case.
Is there anything in between or can we combine things?
Of course we can.
So very often we combine inductive and deductive elements in a research in the sense that in
a case study, for example, we might work inductively and work from the observations in the field
data that we gather in order to create a proposition and then test our proposition, for example,
in a later phase of our case study or in a different case study where we test our proposition.
Actually there is even a term for this kind of going back and forth, which is an
abductive research design, but to be honest it is rarely used as a reviewer or as an author
submitting an abductive research design to a journal. You often have a hard time convincing
your reviewers. But of course it is, happens and it's totally fine and acceptable to combine
elements of inductive and deductive research into a single article. Most articles that you see
you will be able to identify either as one or the other, but it's totally fine as time.
It's totally fine and acceptable to combine these. Okay, I think you got abductive and inductive
research. This is the qualitative, quantitative, inductive, deductive. These are the basic
distinction that we need to look at. Let's go a bit further and look at what different methods
are there that can help us perform research. So surveys, for example, asking people. These can
be qualitative or quantitative in nature and I refer to your textbooks to see what kind of
surveys belong into what area. Surveys can be used in inductive and deductive research. They can be
qualitative and quantitative in nature. So surveys are really a very flexible method that you can
apply. What's an ethnography? An ethnography is if you immerse yourself into a specific case
context that you examine. So you become part of that case and depending on how much you become
part of that case you become, for example, a participant observer. So you kind of stay in
the background, you observe, you scribble down your notes, your observations, how people behave.
That would be an ethnography. Of course, one of the problems of being an ethnographer is first
getting access to the context, to the people, having people allow you to be part of their sessions,
their meetings, their daily life. And of course, as a participant observer, you run danger in
unintentionally modifying people's behavior because people behave differently when they are
feeling observed than when they are not. If you don't believe me, I have a homework for you.
Watch the movie Kitchen Stories. It's a Norwegian movie and it's exactly about somebody
observing an old bachelor in Norway and it's a fun movie at that and it actually teaches you a
lot about ethnography and participant research. If you become very active in the case and you
actually try to help improve a specific situation, then we speak of action research. As in you are
not a neutral observer that simply observes and stays in the background, but you're actually
impacting on a company, for example, and try to improve things while documenting what you have
done and how people have reacted, et cetera, et cetera. So action research is performing research
while being an active element in the case context that you study. Case studies are a very
frequently used method or research design, rather, to examine certain things and many of you will
be using that in their thesis. So we will have a session on case study designs. So I will skip
it. Regressions squarely fall into the quantitative method section, of course, category.
Linear regressions, negative binomial regressions. There's a platitude of regressions and what they
all try to establish, what they all try to do is to show and demonstrate the relationship between
two variables, the independent variables that go into the regression and the dependent variables
that are the result of the regression of your formula. So text analysis is a method which you
can apply on a text corpus. That might be interview data. It might be user generated content. It might
be a book that can be anything. You can perform text analysis to see how similar two texts are to
each other, how happy or unhappy texts sound by doing word frequency analysis. You can do
actually quite a lot with doing text analysis, text mining, especially using computers helps
helps because there's quite fancy statistical methods that you can do by now. Experiments
are a nice way to look only at few factors because the real world is quite messy and sometimes we
don't know what has impacted our outcome because there could be lots of elements that impact a
certain thing. An experiment is something, a laboratory experiment for example, where you
can isolate most of these factors away and only have a few factors influence the outcome. If you've
ever participated in a game theory lab and as a participant for example, you know that this is
exactly a few factors going into the whole thing and the rest basically is try to isolate away.
The nice thing about experiments is that you can have a control group which basically does not
receive a treatment, a change where you can see the size of the effect. Of course this is not
possible for all research questions in context. Think of the reunification of Germany for example.
It's not possible to repeat, have a treatment group, a Germany that is not reunited and a Germany
that is reunited and see how it would have behaved and developed in a different manner. So
experiments are only suitable for specific settings. Quality experiments are closer to real life but
I'm not going in there. There's kind of a little bit not as clean version of a laboratory
experiment. These are the main methods that you will see. There's plenty of others of course.
But these are the ones that you will see very often. And explaining the relationship between
a research design and a method is sometimes a bit tricky. Research design is the whole big thing
and a case study is really more of a research design than a single method because you can
apply lots of methods within a specific study. So methods are the single analysis that you perform
while research design refers to your research design as a whole. I hope you get the distinction.
One thing to look for in your articles is what is the unit of analysis.
The unit of analysis is the one thing that you try to explain and that you look for.
So when for example or when you have a quantitative research this is often kind of each
row in your database in your excel sheet or whatever you have is one unit of analysis
being stored. That could be the individual, it could be a team, it could be a corporation,
an organization, it could be a nation, it could be anything. So when you look at individual
performance of employees then your individual is the unit of analysis and you try to explain that.
When you try to explain the performance of teachers but you only have the grades of single
students as a database then your unit of analysis will actually be the teachers because you are
trying to explain their behavior and you somehow need to kind of using fixed effects, random effects,
all the kind of interesting things. You need to try to map students grades to the specific teachers
and see which teachers perform better or not. Then these will be your unit of analysis.
Or you look at the performance of a whole organization and see whether a diverse
employee set actually makes your company perform better and then your unit of analysis will be the
organization. And it's quite important to actually think about that in advance when you design a
research because your data collection and your data analysis will have to target this unit of
analysis. It's quite hard in retrospect to kind of move the unit of analysis to a different level
and to draw conclusions that are welled on a different level than you have done on the
unit of analysis before. Let me give you a very classic example of that and the problem is called
the ecological fallacy. The ecological fallacy are the prime example of that is that in the 1930s
when the stencils draw information from single individuals on their level of literacy or illiteracy
so people who could not read and whether they were an immigrant or not.
So when you plot these on a state level then what came out is that states with high level of
immigration had low levels of illiteracy. So lots of immigrants everybody could read.
Not so many immigrants a lot of people could not read. Okay so what conclusion can we draw from that?
So immigrants are good for the education of a state or that only the rich immigrate into a specific
state? Actually no not at all because when you looked at it on an individual level the unit of
analysis being the individual in the database could see that the immigrants were most of the
time those who could not read. But think about that if you're an immigrant from some other country
to the US which state would you like to immigrate to? Well those immigrants tended to choose states
which were pretty rich and where most of the population could actually read to immigrate to.
So states with low levels of illiteracy were those who received many immigrants and those states
the poor and rural states with high levels of illiteracy before any immigrant joint
actually received less immigrants. So when you looked at it on an individual level yes immigrants
illiterate. When you looked at the same data on a state level it appeared to be very different
and it appears actually to be opposite than what was happening. I don't want to make any call on
on immigration or other sensitive topics here but this is a classic example that shows you
how easy it is to draw wrong conclusions when you look at the same data on different levels
and levels that the data has not been targeted at and basically so you cannot jump your conclusions
you cannot immediately conclude from one studies on one level of unit of analysis
to a higher level of analysis. There is a danger of misinterpreting the results.
Reliability and validity and significance or statistical significance are actually so
important that I'm going to dedicate a full session to these. So let's move that a little
bit to the back but these are essentially very important criteria when it comes to assessing
the quality of research and the limitations that certain research might actually have.
What I want to talk a little bit about today is what are sampling strategies that we can pursue
and we are going to talk about this next week too when I talk about the sizes which are appropriate
for sampling or whether a big size in sampling is always better. This is because it's related
to statistical significance. Today I want to talk about the different strategies to get a sample.
So first of all when I talk about sampling now I talk about sampling in the quantitative sense
because sampling also happens in qualitative research when we select a certain case for our
case study for example. That is also sampling but we will talk about this when we talk about case
studies and the different methods that exist to select your case in case studies. I want to talk
about sampling in the quantitative method sense. So what is a sample? Ideally in research we would
like to analyze the full population of whatever we are interested in so everybody. If we want to know
if people are healthy ideally in let's say in Germany ideally we would take every individual
in Germany examine the health and would have an answer. In this case we could see the whole
population that we are interested in and derive our conclusions. Unfortunately for practical purposes
it's often not possible to ask and examine everyone every company every citizen in Germany
or globally for answering all the research questions that are out there. So we have to live
with with asking a subset of people and that subset of the whole population that we are
interested in is this sample. The sample is part of the population and it is the subset that we are
asking and then trying to make conclusions therefore the whole population that we are
interested in. So it's very critical how we get this sample. Of course if the sample is flawed
our conclusions will be flawed and there's many many ways in which sampling can go wrong. And if
we introduce a systematic sample bias then basically our sample is worthless and even less
than worthless it will lead to wrong conclusions. So if we are interested in the health of people of
Germany and we basically go to the pedestrian zone of Hamburg's inner city and ask or examine
everybody walking through there we have a sample of German's population. Will we be able to derive
the right conclusions for the health conditions of Germans? No we won't because if we do that
we only catch people who are in the pedestrian zone in Hamburg's inner city on a Monday morning
at 10 o'clock for example. So northern Germans mostly Hamburg people mostly people who don't
have an office job who will be in their office at Monday morning at 10 o'clock. So we have a very
peculiar set of people that we will examine in Germany's pedestrian zones in a big downtown
at 10 o'clock in the morning on a Monday. So you see how difficult it can be to get a sample.
So ideally we want a sample that is representative of the whole population because then we can
directly take our results of our analysis and claim basically they are valid to the whole population.
So try how should we go about or let's pick another example to see how difficult it can be.
Think political polling. We want to ask who is going to be the next Bundeskansler of Germany.
So we turn we pick a phone book we kind of pick numbers at random from the phone book
and we call people up and ask them who do you think is going to be the next Bundeskansler.
Will that lead to a nice sample of German's population? Do you have a fixed line?
And so actually or if you have a fixed line are you still in any listed in any phone book?
They actually still do exist. So probably or most likely not because your age group
is either mobile only or they get some kind of landline as part of a DSL package but it's not
listed in any phone book. So nobody kind of will pick you up from not by looking at a phone book.
This shows you how easy it is to get a non-representative sample. Think of more
examples of sampling that could lead to systematic bias and you'll discover how
difficult it is to get a representative sample. So there are several strategies that try to do that.
So random sampling would be exactly that. We try to randomly pick people in a whole population
and think well now this might hopefully be representative of what we've done.
This might work out if we truly randomly choose subjects within our population. As I've shown you
this is quite difficult to achieve. Phone books and surveys in the pedestrian zones
are not random at all. So random sampling would be pick a number of individuals
from our population at random. Then there's stratified random sampling. So if you think
about our sampling in Hamburg's downtown we could think that well there's so many people of the
population in in Germany living in big cities and so many people of German's population living in
rural areas. So let's pick 60% from urban areas and 40% from rural areas because this is
representative of how Germany's population is distributed. That would be defining two straighter
urban and rural and picking out of these different pools to be a representative sample.
We could stratify by race, by age, by education, by lots of different things or by the number of
people per Bundesland in Germany and see that we get a representative mix according to that.
So that would be stratified random sampling. Within each straighter we pick people still at
random. A different way to get to a sample is systematic sampling. So systematic sampling is
whenever I observe a specific pattern to get to my sample. Imagine I want to test the
consumption behavior of customers and I go to any supermarket and I want to ask people.
So systematic sampling could be to ask every 50th person coming out of the supermarket and
observing what they have actually bought in their shopping cart because that would be one systematic
way to get a sample of people rather than just kind of trying to get everyone within the same
10 minutes which could perhaps be problematic because if this is in the 10 minutes when all the
crafts people buy their breakfast or all the students buy their breakfast it might be biased.
If I have a systematic way of getting my people that would be systematic sampling.
Okay I'm not going into rational subgrouping now. Accidental sampling or convenience sampling
is one of the things that is used often because it's the one that's the easiest to achieve.
I simply select my sample by because I have convenience samples because I have access to it
and it's easy to access. So if I ask all of you students in my course to fill in a survey for
some research that I'm doing then you are my convenience sample because I have access to you
and I have easy access to you. Often game theory experiments are for example done with classes of
students and convenience sampling is on the one hand nice because they're easy to get. On the other
hand they are often systematically biased and flawed right. If I ask Mieber students
about how much they value sustainability I might get a different answer than as if I
ask a representative set of the population of a whole country.
Sometimes and now basically this is as far as I want to go with sampling. We don't really
know what the full population is. If we want to examine the full population or the health
condition of Germans that's pretty easy to know but sometimes a representative sampling
is difficult because we don't really know what our population is. If we want to know the
mortality rate of illegal drug users for example well our population are the illegal drug users
and we can only guess at how many and who that is because well because so sometimes it's and if
you don't know the full population then it's hard to actually show that you have a representative
sample of a population that you want to explain something for. So that can also be a problem.
Other bias that can occur with sampling are for example late responders. So if if I invite you
to perform an online survey on non-responders if I invite you to perform a survey on let's say
whether drugs should be legalized. Some people are more likely to answer that survey because
it might be those who want drugs to be legalized or those who really are into punishing every drug
user. But many systematically with a specific kind of attitude will not bother to reply to my survey.
So these people will not respond to my invitation to take part in the survey. So people who respond
to my survey are systematically different from people who do not respond to my survey
and that leads to a flawed sample and that is quite difficult to detect and that is quite
difficult to correct unless you can force people to reply of course. Sometimes in a company setting
it's the boss can say everybody fill in the survey but in this case this is really hard and
when you invite people for an online survey in your master thesis for example think about
who might respond who's more likely to respond or sometimes we can see difference between people
who respond immediately when you send out an invitation are systematically different from
people who respond only after the third invitation the third reminder to really please fill in this
survey. In this case you have some indications that something might be going on with a sample
that is not so good. That is what I want to say about sampling now. We will talk about
what is the right or a good sample size later. The final topics that I want to cover in this crash
course are mediation and moderation. Often we have a model that needs explanation that needs
confirmation or that is being tested. Model consists out of constructs that are somehow
connected by relationships. So if A goes up, B goes up would be a model with A being the construct
that is our independent variable and B is our dependent variable the one that is being explained
and which depends upon A. Look at the slide above grades and happiness might have a relationship
with each with each other so school grades of a student and their happiness are as you as a
college student. Better grades leads to greater happiness. This is what we could perhaps assume.
So grades would be the construct in this case which is the independent variable
and happiness is the construct which is the dependent variable in this case.
But as soon as we do a survey of people's happiness and we get some valid results,
let's assume they are at least valid and assume we have collected all the grades that you have
gotten in university in your courses. We can do some nice regression correlation test whatever
we find actually yes higher grades higher happiness case closed. No in this case actually
what could be going on that there's not a direct causal relationship between grades and happiness
but that it's actually a third variable that is explaining the relationship that we see.
In this case it might be that if you have higher grades in in college that you have
get a higher self-esteem and that if you have a higher self-esteem then this leads to higher
happiness. So one would test this for example by looking is the relationship between grades and
self-esteem significant and is the relationship between self-esteem and happiness significant.
And if both of them are significant the direct relationship between grades and happiness
should go away and become insignificant. In this case we could say yes self-esteem mediates
the relationship between grades and happiness. This shows you how important it is to include
the correct constructs in your research in your model. Of course if you are missing out
the real constructs that explains something you might get statistically very valid results
but they might have no meaning or no explanatory power at all. This is mediation. So a relationship
is mediated by your variable if the real causal relationship goes through this and can be explained
by a third variable. And this is essentially how we draw it in a model basically one variable
in the middle between other variables. So the other thing is moderating variables.
What is a moderating variable? And making the distinction between mediating and moderating
variables is sometimes very tricky so please take care and take care that moderating example
or a moderator variable changes the direction or the strength of an effect between two
other variables. So let's assume and this is from a real a real old study that if people are
in a more stressful environment have more stress then they show more symptoms of kind of depression
anxiety fatigue etc etc so lower quality of life. So higher stress more symptoms is a relationship
that could well be true but this now bring in a third variable which is a person's social support
does you have family and friends that are kind of concussion some some of the stress and and help
you kind of cope with it. If social support and it turned out that if your social support network
is very high then you can cope with more stress without getting sick. But if you have a low social
support network then the relationship between stress and the symptoms is rather strong so
lots of stress leads to lots of symptoms. So the value of social support changes the relationship
between stress and symptoms. You know on the left hand side of the slide you can see how this is
typically visualized if we want to visualize a model that says this. In terms of quantitative
modeling we often do that by including a moderator an interaction term so stress
um times social support would be a moderator an interaction effect there
that kind of helps to explain the symptoms. So mediation a relationship might be explained by
a third variable and it's very important to find the correct variables the correct constructs
or moderation in which case the the relationship between two variables depends upon the value of a
third variable the moderator variable. That's as far as I want to go here there's plenty more when
it comes to moderation and mediation there's plenty more when it comes to different methodologies and
research designs we're going to look at statistical significance and sample sizes next time.
Thanks for listening in and goodbye.