20 Apr
2008
20 Apr
'08
1:49 p.m.
Hi all,
I am new to MatchIt and would like to learn whether it matters if the treatment cannot be
predicted by any covariates.
The treatment in my study is "experiencing a policy change." My treatment and
control groups are
limited English proficient (LEP) students from two cohorts, before and after the policy
change (I
also compare the performance of non-LEP students from the same two cohorts for
difference-in-differences). To reduce model dependence, I plan to match the LEP students
from the
two cohorts (and the non-LEP students from the two cohorts) on gender, ethnicity, etc.
Does it
matter that the treatment is not really a function of any of these covariates?
Many thanks,
Qian
******************************************************
Qian Guo
Doctoral student, Harvard Graduate School of Education
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
20 Apr
20 Apr
2:33 p.m.
Dear Qian,
This is a much broader question than relates to just MatchIt, but yes, the
answer is that it does matter whether treatment can or cannot be predicted
by any of the covariates (although not necessarily in the same way as is
measured by the c-statistic in logistic regression, for example).
The concept you need to think about is ignorability. Doing causal inference
in non-experimental studies generally requires that assumption, which says
basically that once you condition on the observed covariates, there are no
unobserved differences between the treated and control groups (no hidden
bias). (There are some exceptions, such as instrumental variables, but this
is the basic assumption). So you need to think about which variables are
related to treatment assignment and the outcome, and whether you measure all
of them.
You might want to read the bottom of page 206 of our paper (Ho et al., 2008:
http://gking.harvard.edu/files/abs/matchp-abs.shtml) to read more about
ignorability. It is an important concept for doing studies of causal
inference using non-experimental data.
Liz
On 4/20/08 1:49 PM, "Qian Guo" <guoqi(a)gse.harvard.edu> wrote:
Hi all,
I am new to MatchIt and would like to learn whether it matters if the
treatment cannot be
predicted by any covariates.
The treatment in my study is "experiencing a policy change." My treatment and
control groups are
limited English proficient (LEP) students from two cohorts, before and after
the policy change (I
also compare the performance of non-LEP students from the same two cohorts for
difference-in-differences). To reduce model dependence, I plan to match the
LEP students from the
two cohorts (and the non-LEP students from the two cohorts) on gender,
ethnicity, etc. Does it
matter that the treatment is not really a function of any of these covariates?
Many thanks,
Qian
******************************************************
Qian Guo
Doctoral student, Harvard Graduate School of Education
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
3:05 p.m.
Hi,
In the context of difference-in-differences model (or change score
model), the basic assumption is that in the absence of treatment the
outcomes for the two groups follow a parallel path. That is,
E(Y_{i1}(0) - Y_{i0}(0) | T_i = 1) = E(Y_{i1}(0) - Y_{i0}(0) | T_i = 0)
where Y_{it}(0) is the potential outcome without treatment for unit i at
time t, and T_i is the treatment group indicator.
Thus, one motivation for matching on X in this context might be that you
may believe that this assumption is more likely to hold by conditioning on
X. That is,
E(Y_{i1}(0) - Y_{i0}(0) | T_i = 1, X_i = x)
= E(Y_{i1}(0) - Y_{i0}(0) | T_i = 0, X_i = x)
is more credible than the assumption above. You can control for X in the
standard DID linear regression framework too (by differencing X as well as
Y), but this relies heavily on the linearity. Thus, matching on X (i.e.,
nonparametric preprocessing as we call it in our PA article) is in general
a more effective way to reduce the model dependence.
Hope this helps,
Kosuke
---------------------------------------------------------
Kosuke Imai Office: Corwin Hall 041
Assistant Professor Phone: 609-258-6601
Department of Politics eFax: 973-556-1929
Princeton University Email: kimai(a)Princeton.Edu
Princeton, NJ 08544-1012 http://imai.princeton.edu/
---------------------------------------------------------
On Sun, 20 Apr 2008, Elizabeth Stuart wrote:
Dear Qian,
This is a much broader question than relates to just MatchIt, but yes, the
answer is that it does matter whether treatment can or cannot be predicted
by any of the covariates (although not necessarily in the same way as is
measured by the c-statistic in logistic regression, for example).
The concept you need to think about is ignorability. Doing causal inference
in non-experimental studies generally requires that assumption, which says
basically that once you condition on the observed covariates, there are no
unobserved differences between the treated and control groups (no hidden
bias). (There are some exceptions, such as instrumental variables, but this
is the basic assumption). So you need to think about which variables are
related to treatment assignment and the outcome, and whether you measure all
of them.
You might want to read the bottom of page 206 of our paper (Ho et al., 2008:
http://gking.harvard.edu/files/abs/matchp-abs.shtml) to read more about
ignorability. It is an important concept for doing studies of causal
inference using non-experimental data.
Liz
On 4/20/08 1:49 PM, "Qian Guo" <guoqi(a)gse.harvard.edu> wrote:
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
Hi all,
I am new to MatchIt and would like to learn whether it matters if the
treatment cannot be
predicted by any covariates.
The treatment in my study is "experiencing a policy change." My treatment and
control groups are
limited English proficient (LEP) students from two cohorts, before and after
the policy change (I
also compare the performance of non-LEP students from the same two cohorts for
difference-in-differences). To reduce model dependence, I plan to match the
LEP students from the
two cohorts (and the non-LEP students from the two cohorts) on gender,
ethnicity, etc. Does it
matter that the treatment is not really a function of any of these covariates?
Many thanks,
Qian
******************************************************
Qian Guo
Doctoral student, Harvard Graduate School of Education
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
3:32 p.m.
Dear Liz,
Thank you for your quick and detailed reply. I really appreciate it.
I still feel a little unsure, though. I do not intend to use matching for causal
inferences. My
causal inference comes from treating the sudden policy change as a natural experiment and
examining the difference-in-differences (i.e., comparing the difference in performance of
LEP
students before and after the policy change to the difference in performance of non-LEP
students,
who were supposedly not affected by the policy concerning the education of LEP students).
Moreover, no covariates in my study *can* predict whether a student is in the treatment
group or
not - LEP students are in the treatment group if they entered school in 2003 and are in
the
control group if they entered school in 2000. In such a case, would matching help reduce
model
dependence? Thank you.
Qian
On Sun, 20 Apr 2008 14:33:24 -0400
Elizabeth Stuart <estuart(a)jhsph.edu> wrote:
Dear Qian,
This is a much broader question than relates to just MatchIt, but yes, the
answer is that it does matter whether treatment can or cannot be predicted
by any of the covariates (although not necessarily in the same way as is
measured by the c-statistic in logistic regression, for example).
The concept you need to think about is ignorability. Doing causal inference
in non-experimental studies generally requires that assumption, which says
basically that once you condition on the observed covariates, there are no
unobserved differences between the treated and control groups (no hidden
bias). (There are some exceptions, such as instrumental variables, but this
is the basic assumption). So you need to think about which variables are
related to treatment assignment and the outcome, and whether you measure all
of them.
You might want to read the bottom of page 206 of our paper (Ho et al., 2008:
http://gking.harvard.edu/files/abs/matchp-abs.shtml) to read more about
ignorability. It is an important concept for doing studies of causal
inference using non-experimental data.
Liz
On 4/20/08 1:49 PM, "Qian Guo" <guoqi(a)gse.harvard.edu> wrote:
> Hi all,
>
> I am new to MatchIt and would like to learn whether it matters if the
> treatment cannot be
> predicted by any covariates.
>
> The treatment in my study is "experiencing a policy change." My treatment
and
> control groups are
> limited English proficient (LEP) students from two cohorts, before and after
> the policy change (I
> also compare the performance of non-LEP students from the same two cohorts for
> difference-in-differences). To reduce model dependence, I plan to match the
> LEP students from the
> two cohorts (and the non-LEP students from the two cohorts) on gender,
> ethnicity, etc. Does it
> matter that the treatment is not really a function of any of these covariates?
>
> Many thanks,
> Qian
>
> ******************************************************
> Qian Guo
> Doctoral student, Harvard Graduate School of Education
>
> --
> MatchIt mailing list served by Harvard-MIT Data Center
> List Address: matchit(a)lists.gking.harvard.edu
> Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
> MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
3:39 p.m.
Hi Qian,
Yes, matching will still reduce model dependence in this case, for the
reasons Kosuke described in his follow-up email to mine.
You are in a better position than a standard observational study since you
can use the difference-in-differences design; my main point (which I should
have made clearer) is that you still have to make the assumption Kosuke
detailed in his email, that the slopes are the same given the observed
covariates:
E(Y_{i1}(0) - Y_{i0}(0) | T_i = 1, X_i = x)
= E(Y_{i1}(0) - Y_{i0}(0) | T_i = 0, X_i = x)
So you are still somewhat bound by which covariates you observe, in the
sense of whether you believe this assumption is true given the observed
covariates. In analogy with unconfoundedness/ignorability, I would just
still think about whether there might be other unobserved factors that might
be going on which may affect the inferences.
But the short answer is that, yes, matching should still help reduce model
dependence, in the sense of helping you deal as well as possible with the
covariates that you do observe.
I hope this clarifies things...
Liz
On 4/20/08 3:32 PM, "Qian Guo" <guoqi(a)gse.harvard.edu> wrote:
Dear Liz,
Thank you for your quick and detailed reply. I really appreciate it.
I still feel a little unsure, though. I do not intend to use matching for
causal inferences. My
causal inference comes from treating the sudden policy change as a natural
experiment and
examining the difference-in-differences (i.e., comparing the difference in
performance of LEP
students before and after the policy change to the difference in performance
of non-LEP students,
who were supposedly not affected by the policy concerning the education of LEP
students).
Moreover, no covariates in my study *can* predict whether a student is in the
treatment group or
not - LEP students are in the treatment group if they entered school in 2003
and are in the
control group if they entered school in 2000. In such a case, would matching
help reduce model
dependence? Thank you.
Qian
On Sun, 20 Apr 2008 14:33:24 -0400
Elizabeth Stuart <estuart(a)jhsph.edu> wrote:
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
Dear Qian,
This is a much broader question than relates to just MatchIt, but yes, the
answer is that it does matter whether treatment can or cannot be predicted
by any of the covariates (although not necessarily in the same way as is
measured by the c-statistic in logistic regression, for example).
The concept you need to think about is ignorability. Doing causal inference
in non-experimental studies generally requires that assumption, which says
basically that once you condition on the observed covariates, there are no
unobserved differences between the treated and control groups (no hidden
bias). (There are some exceptions, such as instrumental variables, but this
is the basic assumption). So you need to think about which variables are
related to treatment assignment and the outcome, and whether you measure all
of them.
You might want to read the bottom of page 206 of our paper (Ho et al., 2008:
http://gking.harvard.edu/files/abs/matchp-abs.shtml) to read more about
ignorability. It is an important concept for doing studies of causal
inference using non-experimental data.
Liz
On 4/20/08 1:49 PM, "Qian Guo" <guoqi(a)gse.harvard.edu> wrote:
> Hi all,
>
> I am new to MatchIt and would like to learn whether it matters if the
> treatment cannot be
> predicted by any covariates.
>
> The treatment in my study is "experiencing a policy change." My treatment
> and
> control groups are
> limited English proficient (LEP) students from two cohorts, before and after
> the policy change (I
> also compare the performance of non-LEP students from the same two cohorts
> for
> difference-in-differences). To reduce model dependence, I plan to match the
> LEP students from the
> two cohorts (and the non-LEP students from the two cohorts) on gender,
> ethnicity, etc. Does it
> matter that the treatment is not really a function of any of these
> covariates?
>
> Many thanks,
> Qian
>
> ******************************************************
> Qian Guo
> Doctoral student, Harvard Graduate School of Education
>
> --
> MatchIt mailing list served by Harvard-MIT Data Center
> List Address: matchit(a)lists.gking.harvard.edu
> Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
> MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
4:23 p.m.
Liz and Kosuke,
Your replies are very helpful. Thank you very much.
All best,
Qian
On Sun, 20 Apr 2008 15:39:31 -0400
Elizabeth Stuart <estuart(a)jhsph.edu> wrote:
Hi Qian,
Yes, matching will still reduce model dependence in this case, for the
reasons Kosuke described in his follow-up email to mine.
You are in a better position than a standard observational study since you
can use the difference-in-differences design; my main point (which I should
have made clearer) is that you still have to make the assumption Kosuke
detailed in his email, that the slopes are the same given the observed
covariates:
E(Y_{i1}(0) - Y_{i0}(0) | T_i = 1, X_i = x)
= E(Y_{i1}(0) - Y_{i0}(0) | T_i = 0, X_i = x)
So you are still somewhat bound by which covariates you observe, in the
sense of whether you believe this assumption is true given the observed
covariates. In analogy with unconfoundedness/ignorability, I would just
still think about whether there might be other unobserved factors that might
be going on which may affect the inferences.
But the short answer is that, yes, matching should still help reduce model
dependence, in the sense of helping you deal as well as possible with the
covariates that you do observe.
I hope this clarifies things...
Liz
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
22 Apr
22 Apr
6:16 p.m.
Hi
If I can, I would add a few comments:
* one thing researchers ask is whether it's good or bad when the
covariates don't predict treatment.
Really, it's just not very informative because we don't know which of
the following are true:
- you picked the wrong covariates to include
- the world is a willy nilly place and treatment is effectively
producing random assignment
- you picked the covariates sensibly but they just don't predict the
treatment. Still, there is unobserved confounding that is problematic.
It's hard to know which of these is the case.
* I would contradict the advice my colleagues have given you to some extent.
- In general, I think differencing effectively matches on unobservables
in a way that is very, very difficult to accomplish with matching on
observables.
In my experience, for example, comparing siblings is much, much better
than trying to match women based on their observed characteristics--in
other words, observed characteristics explain very little of what
sisters share.
- linearity can be a problem. The solution to that problem would be to
think of semi-parametric models that incorporate time flexibly--more
like a regression discontinuity approach involving time.
good luck -- michael
Qian Guo wrote:
Dear Liz,
Thank you for your quick and detailed reply. I really appreciate it.
I still feel a little unsure, though. I do not intend to use matching
for causal inferences. My causal inference comes from treating the
sudden policy change as a natural experiment and examining the
difference-in-differences (i.e., comparing the difference in
performance of LEP students before and after the policy change to the
difference in performance of non-LEP students, who were supposedly not
affected by the policy concerning the education of LEP students).
Moreover, no covariates in my study *can* predict whether a student is
in the treatment group or not - LEP students are in the treatment
group if they entered school in 2003 and are in the control group if
they entered school in 2000. In such a case, would matching help
reduce model dependence? Thank you.
Qian
On Sun, 20 Apr 2008 14:33:24 -0400
Elizabeth Stuart <estuart(a)jhsph.edu> wrote:
--
E. Michael Foster
getmike.info
download vcard: www dot unc dot edu/~emfoster/Foster.vcf
Professor of Maternal and Child Health
of Health Policy and Administration
of Biostatistics (adjunct)
Associate Chair for Faculty Advancement
(W) 919-966-3773
(F) 919-966-0458
Office: 422D Rosenau Hall << use for FEDEX
School of Public Health
University of North Carolina, Chapel Hill
Rosenau Hall, Campus Box# 7445
Chapel Hill, NC 27599-7445
[the http is throwing off some spam filters)
visit the new FAQ section of my blog:
fosterfaq dot blogspot dot com
27) [Mollie Ivins] "What do you mean cancer is a gift. You're as a dumb as
Shrub. Cancer just killed me. Some gift."
"I'm not quite dead yet, sir!" -- Concorde, Monty Python and the Holy
Grail
"It is a fearful thing to love what death can touch" -- Chaim Stern
"Manure is an art and I consider myself an accomplished artist" ~John Adams.
32) If I have a vacant look on my face, what should you do?
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
Dear Qian,
This is a much broader question than relates to just MatchIt, but
yes, the
answer is that it does matter whether treatment can or cannot be
predicted
by any of the covariates (although not necessarily in the same way as is
measured by the c-statistic in logistic regression, for example).
The concept you need to think about is ignorability. Doing causal
inference
in non-experimental studies generally requires that assumption, which
says
basically that once you condition on the observed covariates, there
are no
unobserved differences between the treated and control groups (no hidden
bias). (There are some exceptions, such as instrumental variables,
but this
is the basic assumption). So you need to think about which variables
are
related to treatment assignment and the outcome, and whether you
measure all
of them.
You might want to read the bottom of page 206 of our paper (Ho et
al., 2008:
http://gking.harvard.edu/files/abs/matchp-abs.shtml) to read more about
ignorability. It is an important concept for doing studies of causal
inference using non-experimental data.
Liz
On 4/20/08 1:49 PM, "Qian Guo" <guoqi(a)gse.harvard.edu> wrote:
> Hi all,
>
> I am new to MatchIt and would like to learn whether it matters if the
> treatment cannot be
> predicted by any covariates.
>
> The treatment in my study is "experiencing a policy change." My
> treatment and
> control groups are
> limited English proficient (LEP) students from two cohorts, before
> and after
> the policy change (I
> also compare the performance of non-LEP students from the same two
> cohorts for
> difference-in-differences). To reduce model dependence, I plan to
> match the
> LEP students from the
> two cohorts (and the non-LEP students from the two cohorts) on gender,
> ethnicity, etc. Does it
> matter that the treatment is not really a function of any of these
> covariates?
>
> Many thanks,
> Qian
>
> ******************************************************
> Qian Guo
> Doctoral student, Harvard Graduate School of Education
>
> --
> MatchIt mailing list served by Harvard-MIT Data Center
> List Address: matchit(a)lists.gking.harvard.edu
> Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
> MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
9:35 p.m.
Hi Michael,
Thank you for your reply. It's nice to learn the pros and cons of both methods.
That'll be very
helpful to my research. I really appreciate it.
All best,
Qian
On Tue, 22 Apr 2008 18:16:53 -0400
"E. Michael Foster" <mike4kids(a)gmail.com> wrote:
Hi
If I can, I would add a few comments:
* one thing researchers ask is whether it's good or bad when the covariates
don't predict
treatment. Really, it's just not very informative because we don't know which of
the following
are true:
- you picked the wrong covariates to include
- the world is a willy nilly place and treatment is effectively producing random
assignment
- you picked the covariates sensibly but they just don't predict the treatment.
Still, there is
unobserved confounding that is problematic.
It's hard to know which of these is the case.
* I would contradict the advice my colleagues have given you to some extent.
- In general, I think differencing effectively matches on unobservables in a way that is
very,
very difficult to accomplish with matching on observables. In my experience, for example,
comparing siblings is much, much better than trying to match women based on their observed
characteristics--in other words, observed characteristics explain very little of what
sisters
share.
- linearity can be a problem. The solution to that problem would be to think of
semi-parametric
models that incorporate time flexibly--more like a regression discontinuity approach
involving
time.
good luck -- michael
Qian Guo wrote:
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
Dear Liz,
Thank you for your quick and detailed reply. I really appreciate it.
I still feel a little unsure, though. I do not intend to use matching
for causal inferences. My causal inference comes from treating the
sudden policy change as a natural experiment and examining the
difference-in-differences (i.e., comparing the difference in
performance of LEP students before and after the policy change to the
difference in performance of non-LEP students, who were supposedly not
affected by the policy concerning the education of LEP students).
Moreover, no covariates in my study *can* predict whether a student is
in the treatment group or not - LEP students are in the treatment
group if they entered school in 2003 and are in the control group if
they entered school in 2000. In such a case, would matching help
reduce model dependence? Thank you.
Qian
On Sun, 20 Apr 2008 14:33:24 -0400
Elizabeth Stuart <estuart(a)jhsph.edu> wrote:
--
E. Michael Foster
getmike.info
download vcard: www dot unc dot edu/~emfoster/Foster.vcf
Professor of Maternal and Child Health
of Health Policy and Administration
of Biostatistics (adjunct)
Associate Chair for Faculty Advancement
(W) 919-966-3773
(F) 919-966-0458
Office: 422D Rosenau Hall << use for FEDEX
School of Public Health
University of North Carolina, Chapel Hill
Rosenau Hall, Campus Box# 7445
Chapel Hill, NC 27599-7445
[the http is throwing off some spam filters)
visit the new FAQ section of my blog:
fosterfaq dot blogspot dot com
27) [Mollie Ivins] "What do you mean cancer is a gift. You're as a dumb as
Shrub. Cancer just
killed me. Some gift."
"I'm not quite dead yet, sir!" -- Concorde, Monty Python and the Holy
Grail
"It is a fearful thing to love what death can touch" -- Chaim Stern
"Manure is an art and I consider myself an accomplished artist" ~John Adams.
32) If I have a vacant look on my face, what should you do?
Dear Qian,
This is a much broader question than relates to just MatchIt, but
yes, the
answer is that it does matter whether treatment can or cannot be
predicted
by any of the covariates (although not necessarily in the same way as is
measured by the c-statistic in logistic regression, for example).
The concept you need to think about is ignorability. Doing causal
inference
in non-experimental studies generally requires that assumption, which
says
basically that once you condition on the observed covariates, there
are no
unobserved differences between the treated and control groups (no hidden
bias). (There are some exceptions, such as instrumental variables,
but this
is the basic assumption). So you need to think about which variables
are
related to treatment assignment and the outcome, and whether you
measure all
of them.
You might want to read the bottom of page 206 of our paper (Ho et
al., 2008:
http://gking.harvard.edu/files/abs/matchp-abs.shtml) to read more about
ignorability. It is an important concept for doing studies of causal
inference using non-experimental data.
Liz
On 4/20/08 1:49 PM, "Qian Guo" <guoqi(a)gse.harvard.edu> wrote:
> Hi all,
>
> I am new to MatchIt and would like to learn whether it matters if the
> treatment cannot be
> predicted by any covariates.
>
> The treatment in my study is "experiencing a policy change." My
> treatment and
> control groups are
> limited English proficient (LEP) students from two cohorts, before
> and after
> the policy change (I
> also compare the performance of non-LEP students from the same two
> cohorts for
> difference-in-differences). To reduce model dependence, I plan to
> match the
> LEP students from the
> two cohorts (and the non-LEP students from the two cohorts) on gender,
> ethnicity, etc. Does it
> matter that the treatment is not really a function of any of these
> covariates?
>
> Many thanks,
> Qian
>
> ******************************************************
> Qian Guo
> Doctoral student, Harvard Graduate School of Education
>
> --
> MatchIt mailing list served by Harvard-MIT Data Center
> List Address: matchit(a)lists.gking.harvard.edu
> Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
> MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
9:49 p.m.
Hi,
I still believe that matching can help the DID method. You do not have
to choose one over another. For most DID applications in observational
studies, it is difficult to assume
E(Y_{1i}(0) - Y_{0i}(0) | G_i = 1) = E(Y_{1i}(0) - Y_{0i}(0) | G_i = 0)
without conditioning on X because the assumption heavily relies on the
linearity. Unless G_i is independent (not just mean independence) of
Y_{1i}(0), you can take a nonlinear transformation of Y (say, log
transformation), and violate the assumption. Thus, you would want to
condition on confounders X in order for the above assumption to hold.
Now, if one includes Y_{0i} = Y_{0i}(0) in X, matching will reduce model
dependence. To see this, consider the case where you match exactly on
Y_{0i}. Then, the assumption becomes,
E(Y_{1i}(0) | Y_{0i}, G_i = 1, X_i) = E(Y_{1i}(0) | Y_{0i}, G_i = 0, X_i)
which is exactly the exogeneity assumption! So, once you match exactly on
Y_{0i}, the two methods become identical and you completely eliminate the
model dependence (at least for these two models).
In any event, my suggestion is for you to match on Y_{0i} and X_i first
(to reduce the imbalance of observables), and then do DID if you really
believe that differencing will reduce the imbalance of unobservables. In
this way, matching can help the DID method.
Kosuke
---------------------------------------------------------
Kosuke Imai Office: Corwin Hall 041
Assistant Professor Phone: 609-258-6601
Department of Politics eFax: 973-556-1929
Princeton University Email: kimai(a)Princeton.Edu
Princeton, NJ 08544-1012 http://imai.princeton.edu/
---------------------------------------------------------
On Tue, 22 Apr 2008, Qian Guo wrote:
Hi Michael,
Thank you for your reply. It's nice to learn the pros and cons of both
methods. That'll be very helpful to my research. I really appreciate it.
All best,
Qian
On Tue, 22 Apr 2008 18:16:53 -0400
"E. Michael Foster" <mike4kids(a)gmail.com> wrote:
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
Hi
If I can, I would add a few comments:
* one thing researchers ask is whether it's good or bad when the covariates
don't predict treatment. Really, it's just not very informative because we
don't know which of the following are true:
- you picked the wrong covariates to include
- the world is a willy nilly place and treatment is effectively producing
random assignment
- you picked the covariates sensibly but they just don't predict the
treatment. Still, there is unobserved confounding that is problematic.
It's hard to know which of these is the case. * I would contradict the
advice my colleagues have given you to some extent.
- In general, I think differencing effectively matches on unobservables in
a way that is very, very difficult to accomplish with matching on
observables. In my experience, for example, comparing siblings is much,
much better than trying to match women based on their observed
characteristics--in other words, observed characteristics explain very
little of what sisters share.
- linearity can be a problem. The solution to that problem would be to
think of semi-parametric models that incorporate time flexibly--more like a
regression discontinuity approach involving time. good luck -- michael
Qian Guo wrote:
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
Dear Liz,
Thank you for your quick and detailed reply. I really appreciate it.
I still feel a little unsure, though. I do not intend to use matching for
causal inferences. My causal inference comes from treating the sudden
policy change as a natural experiment and examining the
difference-in-differences (i.e., comparing the difference in performance
of LEP students before and after the policy change to the difference in
performance of non-LEP students, who were supposedly not affected by the
policy concerning the education of LEP students). Moreover, no covariates
in my study *can* predict whether a student is in the treatment group or
not - LEP students are in the treatment group if they entered school in
2003 and are in the control group if they entered school in 2000. In such
a case, would matching help reduce model dependence? Thank you.
Qian
On Sun, 20 Apr 2008 14:33:24 -0400
Elizabeth Stuart <estuart(a)jhsph.edu> wrote:
--
E. Michael Foster
getmike.info
download vcard: www dot unc dot edu/~emfoster/Foster.vcf
Professor of Maternal and Child Health
of Health Policy and Administration
of Biostatistics (adjunct)
Associate Chair for Faculty Advancement
(W) 919-966-3773
(F) 919-966-0458
Office: 422D Rosenau Hall << use for FEDEX
School of Public Health
University of North Carolina, Chapel Hill
Rosenau Hall, Campus Box# 7445
Chapel Hill, NC 27599-7445
[the http is throwing off some spam filters)
visit the new FAQ section of my blog:
fosterfaq dot blogspot dot com
27) [Mollie Ivins] "What do you mean cancer is a gift. You're as a dumb as
Shrub. Cancer just killed me. Some gift."
"I'm not quite dead yet, sir!" -- Concorde, Monty Python and the Holy
Grail
"It is a fearful thing to love what death can touch" -- Chaim Stern
"Manure is an art and I consider myself an accomplished artist" ~John
Adams. 32) If I have a vacant look on my face, what should you do?
Dear Qian,
This is a much broader question than relates to just MatchIt, but yes,
the
answer is that it does matter whether treatment can or cannot be
predicted
by any of the covariates (although not necessarily in the same way as is
measured by the c-statistic in logistic regression, for example).
The concept you need to think about is ignorability. Doing causal
inference
in non-experimental studies generally requires that assumption, which
says
basically that once you condition on the observed covariates, there are
no
unobserved differences between the treated and control groups (no hidden
bias). (There are some exceptions, such as instrumental variables, but
this
is the basic assumption). So you need to think about which variables are
related to treatment assignment and the outcome, and whether you measure
all
of them. You might want to read the bottom of page 206 of our paper (Ho
et al., 2008:
http://gking.harvard.edu/files/abs/matchp-abs.shtml) to read more about
ignorability. It is an important concept for doing studies of causal
inference using non-experimental data.
Liz
On 4/20/08 1:49 PM, "Qian Guo" <guoqi(a)gse.harvard.edu> wrote:
> Hi all,
>
> I am new to MatchIt and would like to learn whether it matters if the
> treatment cannot be
> predicted by any covariates.
>
> The treatment in my study is "experiencing a policy change." My
> treatment and
> control groups are
> limited English proficient (LEP) students from two cohorts, before and
> after
> the policy change (I
> also compare the performance of non-LEP students from the same two
> cohorts for
> difference-in-differences). To reduce model dependence, I plan to match
> the
> LEP students from the
> two cohorts (and the non-LEP students from the two cohorts) on gender,
> ethnicity, etc. Does it
> matter that the treatment is not really a function of any of these
> covariates?
>
> Many thanks,
> Qian
>
> ******************************************************
> Qian Guo
> Doctoral student, Harvard Graduate School of Education
>
> --
> MatchIt mailing list served by Harvard-MIT Data Center
> List Address: matchit(a)lists.gking.harvard.edu
> Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
> MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
--
MatchIt mailing list served by Harvard-MIT Data Center
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/?info=matchit
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
5869
days inactive
5872
days old
matchit@lists.gking.harvard.edu
8 comments
4 participants
participants (4)
-
E. Michael Foster -
Elizabeth Stuart -
Kosuke Imai -
Qian Guo