Reading Avg Treatment Effects & Confidence Intervals: Depression in OHE: Causal Inference Bootcamp

Reading Avg Treatment Effects & Confidence Intervals: Depression in OHE: Causal Inference Bootcamp


[MUSIC] So you’ve done your experiment: you’ve
randomly assigned some people to have the option of enrolling in Medicaid and some people
to be in the control group, and you’ve waited a year, you’ve gathered your data, and now
it’s time to look at the outcome and see what the results are. So here we’ve got all our outcome data of
our experiment. Over here, we’ve got a whole bunch of different outcome variables that
we’re interested in: we’ve got blood pressure, cholesterol, hemoglobin, depression measures
and cardiovascular risk. And if you’re a medical doctor, then all of these numbers, these variables,
have lots of meaning to you and why they matter for health related outcomes. So what we want
to know is: what is the effect of having Medicaid on things like positive depression screen
– so this is a measure of whether you are depressed or not. So does having Medicaid
lower depression, make you less depressed? Well, we’re going to find out very soon. So remember that experiments are relatively
easy to analyze. We just want to compute the average treatment effect, which is the difference
between the average outcomes in the treatment group and the average outcomes in the control
group. So over here in Column 1, they’ve given us
the mean, or average value, in the control group. So for example, all of these are means,
and in the control group 30% of people had a positive depression screen, for example.
Now they haven’t given us the average values in the treatment group, which is an imaginary
column right here, but what they have given us is the average treatment effect, which
is taking the average value in the treatment group minus the average value in the control
group and then that gives us this column here, Column 2. So these numbers here are all differences
between the average value in the treatment group and the control group. So for example, the positive depression screen,
we have an average value of 30 in the control group, 30, and then the difference is -2.21,
so the value in the treatment group is about 28. So only about 28% of people in the treatment
group are depressed whereas its 30% in the control group. So from this number, it looks
like being in the treatment group lowered whether you got depressed or not. So, because this is an experiment, we know
that this number here, this average treatment effect -2.21, is a true causal effect. We don’t have to worry about confounders or
selection problems or anything like that because of random assignment. But we only have a finite
number of people. We have to deal with the fact that we just gathered some data, we didn’t
actually do the experiment on everybody in the state of Oregon or everybody in the whole
country. So we have to worry about finite sample problems,
that means statistical inference in addition to causal inference. That’s exactly what these
numbers here in parenthesis do. These are 95% Confidence Intervals. So let’s talk about how we interpret these.
So here we got the 95% Confidence Interval for the average treatment effect of having
Medicaid on positive depression screen. So here is the interval: -4.02 to -0.40. The
first thing you ask with confidence intervals is: is zero in the interval? No, it’s not.
If zero is not in the interval, you conclude that your estimate of the average treatment
effect,-2.21, is a statistically significant effect. That means that it’s not due to the
particular people who happen to be put into your sample in the experiment, it’s really
a true causal effect. So just because we found that this confidence
interval doesn’t have zero, we conclude that it’s a statistically significant effect, but
that doesn’t mean that it’s a clinically significant effect. So is a -2.2% decrease in depression
something that matters as far as the world is concerned? Well, that is something that
you have to go to a medical doctor to think about. But as far as we are concerned, there is no
problem with small sample size. We’ve been able to detect an effect. Being in the treatment
group here lowered depression by 2%. Now in the next module, we’ll look at another of
these variables where this confidence interval actually has zero, then we have to start thinking
about few different things. [MUSIC].