15 Mar
2012
15 Mar
'12
12:48 p.m.
Hello,
I am writing on behalf of myself and Rocío Titiunik.
As we see it, matchit() serves as a front end to GenMatch()
when method=='genetic'. So it should be possible to get the same
results from both the MatchIt and Matching packages. But this
proves difficult. Discrepancies seem to occur because matchit()
silently adds a column of propensity scores to the X matrix that it
passes to GenMatch(). I'm appending some code that we developed to
demonstrate this point.
The benefits (in most cases) of adding a propensity score are
clear, but it seems problematic that this addition occurs without
any notification. For example, a user of MatchIt may claim to have
matched on a set of predictors, unaware that he has also matched on
the propensity score. Or, if a user thinks that he has matched on
one propensity score (because he included it in X when calling
matchit()), he will be unaware that he has also matched on a second,
matchit()-generated propensity score that is based on the first
propensity score. In either case, he will be also be unable to
replicate his MatchIt results with Matching or vice versa, even
though MatchIt is supposed to be a wrapper for Matching. And
third-party readers of code that includes a call to matchit() will
likely have no idea that a propensity score has been added.
We can't see any explicit mention of this
propensity-score-adding feature in the MatchIt manual. To promote
future replication efforts, can a line be added to the MatchIt
manual about the addition of a propensity score to X? Or can a note
be added to matchit() output whenever method=='genetic' is used?
Thank you,
John Bullock
###
library(Matching)
library(MatchIt)
data(lalonde)
X <- with(lalonde, cbind(age, educ, black, hispan, married,
nodegree, re74, re75))
# GET ESTIMATE FROM MATCHIT()
set.seed(5678)
m.out1 <- matchit(treat ~ X, data=lalonde, method='genetic',
estimand='ATT', ties=TRUE, print.level=1,
pop.size=150, wait.generations=1,
max.generations=10, hard.generation.limit=TRUE,
unif.seed=1945, int.seed=1906)
# Create a vector of "parameters at the solution" reported by matchit().
# Behind the scenes, these weights have been produced by GenMatch().
weights <- c(8.121767e+02, 7.735192e+02, 5.938936e+02, 4.079661e+02,
3.665018e+02, 1.361011e+02,
9.644896e+02, 6.083636e+02, 2.346772e+00)
ATT.MatchIt <- with(match.data(m.out1),
weighted.mean(re78[treat==1], weights[treat==1])) -
with(match.data(m.out1),
weighted.mean(re78[treat==0], weights[treat==0]))
print(ATT.MatchIt) # 939.2
# GET ESTIMATE FROM MATCH()
# The Match() estimate is -952.3 -- very different.
Match(Y=lalonde$re78, Tr=lalonde$treat, X=X, estimand="ATT",
Weight.matrix=diag(weights), ties=TRUE)$est
# NOTE DISCREPANCY BETWEEN ncol(X) AND length(weights)
# There are only 8 variables in X, so why is matchit() producing
weights for nine variables?
ncol(X) # 8
length(weights) # 9
# ADD PROPENSITY SCORE TO X AND RE-ESTIMATE WITH MATCH()
glm1 <- glm(treat ~
age+educ+black+hispan+married+nodegree+re74+re75, family=binomial,
data=lalonde)
X2 <- with(lalonde, cbind(age, educ, black, hispan, married,
nodegree, re74, re75))
X2 <- cbind(glm1$fitted, X2)
Match(Y=lalonde$re78, Tr=lalonde$treat, X=X2, estimand="ATT",
Weight.matrix=diag(weights), ties=TRUE)$est # 939.2
16 Mar
16 Mar
9:43 a.m.
Thanks John and Rocío, Good catch, we'll include it in the manual. FYI, as
I recall when we wrote matchit, we added the propensity score to genmatch
because genmatch didn't seem to reduce balance without it. that's odd
because genmatch already attempts to optimize an arbitrary function of all
the variables in the pscore, and so if its doing what its advertising
adding yet another function of the same variables shouldn't do anything.
But you never know what happens in such a complicated optimization problem,
and the pscore often helped it do a bit better and so we included it.
Gary
--
*Gary King* - Albert J. Weatherhead III University Professor - Director,
IQSS - Harvard University
GKing.Harvard.edu <http://gking.harvard.edu/> - King(a)Harvard.edu -
@kinggary<http://twitter.com/kinggary>-
617-500-7570 - Asst 495-9271 - Fax 812-8581
2012/3/15 John G. Bullock <john.bullock(a)yale.edu>
Hello,
I am writing on behalf of myself and Rocío Titiunik.
As we see it, matchit() serves as a front end to GenMatch() when
method=='genetic'. So it should be possible to get the same results from
both the MatchIt and Matching packages. But this proves difficult.
Discrepancies seem to occur because matchit() silently adds a column of
propensity scores to the X matrix that it passes to GenMatch(). I'm
appending some code that we developed to demonstrate this point.
The benefits (in most cases) of adding a propensity score are clear,
but it seems problematic that this addition occurs without any
notification. For example, a user of MatchIt may claim to have matched on
a set of predictors, unaware that he has also matched on the propensity
score. Or, if a user thinks that he has matched on one propensity score
(because he included it in X when calling matchit()), he will be unaware
that he has also matched on a second, matchit()-generated propensity score
that is based on the first propensity score. In either case, he will be
also be unable to replicate his MatchIt results with Matching or vice
versa, even though MatchIt is supposed to be a wrapper for Matching. And
third-party readers of code that includes a call to matchit() will likely
have no idea that a propensity score has been added.
We can't see any explicit mention of this propensity-score-adding
feature in the MatchIt manual. To promote future replication efforts, can
a line be added to the MatchIt manual about the addition of a propensity
score to X? Or can a note be added to matchit() output whenever
method=='genetic' is used?
Thank you,
John Bullock
###
library(Matching)
library(MatchIt)
data(lalonde)
X <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
re74, re75))
# GET ESTIMATE FROM MATCHIT()
set.seed(5678)
m.out1 <- matchit(treat ~ X, data=lalonde, method='genetic',
estimand='ATT', ties=TRUE, print.level=1,
pop.size=150, wait.generations=1,
max.generations=10, hard.generation.limit=TRUE,
unif.seed=1945, int.seed=1906)
# Create a vector of "parameters at the solution" reported by matchit().
# Behind the scenes, these weights have been produced by GenMatch().
weights <- c(8.121767e+02, 7.735192e+02, 5.938936e+02, 4.079661e+02,
3.665018e+02, 1.361011e+02,
9.644896e+02, 6.083636e+02, 2.346772e+00)
ATT.MatchIt <- with(match.data(m.out1), weighted.mean(re78[treat==1],
weights[treat==1])) -
with(match.data(m.out1), weighted.mean(re78[treat==0],
weights[treat==0]))
print(ATT.MatchIt) # 939.2
# GET ESTIMATE FROM MATCH()
# The Match() estimate is -952.3 -- very different.
Match(Y=lalonde$re78, Tr=lalonde$treat, X=X, estimand="ATT",
Weight.matrix=diag(weights), ties=TRUE)$est
# NOTE DISCREPANCY BETWEEN ncol(X) AND length(weights)
# There are only 8 variables in X, so why is matchit() producing weights
for nine variables?
ncol(X) # 8
length(weights) # 9
# ADD PROPENSITY SCORE TO X AND RE-ESTIMATE WITH MATCH()
glm1 <- glm(treat ~ age+educ+black+hispan+married+**nodegree+re74+re75,
family=binomial, data=lalonde)
X2 <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
re74, re75))
X2 <- cbind(glm1$fitted, X2)
Match(Y=lalonde$re78, Tr=lalonde$treat, X=X2, estimand="ATT",
Weight.matrix=diag(weights), ties=TRUE)$est # 939.2
-
---
MatchIt mailing list served by HUIT
List Address: matchit@lists.gking.harvard.**edu<matchit@lists.gking.harvard.edu>
Subscribe/Unsubscribe: http://lists.gking.harvard.**
edu/mailman/listinfo/ei<http://lists.gking.harvard.edu/mailman/listinfo/…
MatchIt Software and Documentation:
http://gking.harvard.edu/**matchit/<http://gking.harvard.edu/matchit/>
Browse/Search List Archive: http://lists.gking.harvard.**
edu/mailman/private/matchit/<http://lists.gking.harvard.edu/mailman/priv…
Matchit mailing list
Matchit(a)lists.gking.harvard.**edu <Matchit(a)lists.gking.harvard.edu>
https://lists.gking.harvard.**edu/mailman/listinfo/matchit<https://lists…
1:24 p.m.
Thanks for your reply Gary.
Adding a propensity score to GenMatch is usually a good idea in
applications, and indeed it's recommended in Diamond and Sekhon
(2012). But we were concerned about replicability, because we thought
there was an error in our code when we couldn't replicate the matchit
results using GenMatch directly.
To continue the conversation, whether GenMatch finds or does not find
balance does not depend on whether one adds a propensity score to the
matching matrix. If simple matching on the propensity score leads to
perfect balance, then GenMatch will give the propensity score maximum
weight and will give zero weight to the rest of the components of the
matching matrix. In these cases, if one fails to include the
propensity score, GenMatch will still find the same solution, but it
will take longer than if the propensity score had been included. But
the opposite can also happen: if matching on the propensity score does
not improve balance at all, GenMatch will eventually figure this out
and assign zero weight to it, but it will take longer to find balance
than if the propensity score had not been included in the first place.
In other words, the Rosenbaum and Rubin theorem holds for the true
propensity score. Matching on a badly misspecified propensity score
will not improve balance, and in these cases including it in GenMatch
will slow the balance-finding process.
Rocío
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
On Fri, Mar 16, 2012 at 9:43 AM, Gary King <king(a)harvard.edu> wrote:
Thanks John and Rocío, Good catch, we'll include
it in the manual. FYI, as
I recall when we wrote matchit, we added the propensity score to genmatch
because genmatch didn't seem to reduce balance without it. that's odd
because genmatch already attempts to optimize an arbitrary function of all
the variables in the pscore, and so if its doing what its advertising adding
yet another function of the same variables shouldn't do anything. But you
never know what happens in such a complicated optimization problem, and the
pscore often helped it do a bit better and so we included it.
Gary
--
Gary King - Albert J. Weatherhead III University Professor - Director, IQSS
- Harvard University
GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst
495-9271 - Fax 812-8581
2012/3/15 John G. Bullock <john.bullock(a)yale.edu>
Hello,
I am writing on behalf of myself and Rocío Titiunik.
As we see it, matchit() serves as a front end to GenMatch() when
method=='genetic'. So it should be possible to get the same results from
both the MatchIt and Matching packages. But this proves difficult.
Discrepancies seem to occur because matchit() silently adds a column of
propensity scores to the X matrix that it passes to GenMatch(). I'm
appending some code that we developed to demonstrate this point.
The benefits (in most cases) of adding a propensity score are clear,
but it seems problematic that this addition occurs without any notification.
For example, a user of MatchIt may claim to have matched on a set of
predictors, unaware that he has also matched on the propensity score. Or,
if a user thinks that he has matched on one propensity score (because he
included it in X when calling matchit()), he will be unaware that he has
also matched on a second, matchit()-generated propensity score that is based
on the first propensity score. In either case, he will be also be unable to
replicate his MatchIt results with Matching or vice versa, even though
MatchIt is supposed to be a wrapper for Matching. And third-party readers
of code that includes a call to matchit() will likely have no idea that a
propensity score has been added.
We can't see any explicit mention of this propensity-score-adding
feature in the MatchIt manual. To promote future replication efforts, can a
line be added to the MatchIt manual about the addition of a propensity score
to X? Or can a note be added to matchit() output whenever method=='genetic'
is used?
Thank you,
John Bullock
###
library(Matching)
library(MatchIt)
data(lalonde)
X <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
re74, re75))
# GET ESTIMATE FROM MATCHIT()
set.seed(5678)
m.out1 <- matchit(treat ~ X, data=lalonde, method='genetic',
estimand='ATT', ties=TRUE, print.level=1,
pop.size=150, wait.generations=1,
max.generations=10, hard.generation.limit=TRUE,
unif.seed=1945, int.seed=1906)
# Create a vector of "parameters at the solution" reported by matchit().
# Behind the scenes, these weights have been produced by GenMatch().
weights <- c(8.121767e+02, 7.735192e+02, 5.938936e+02, 4.079661e+02,
3.665018e+02, 1.361011e+02,
9.644896e+02, 6.083636e+02, 2.346772e+00)
ATT.MatchIt <- with(match.data(m.out1), weighted.mean(re78[treat==1],
weights[treat==1])) -
with(match.data(m.out1), weighted.mean(re78[treat==0],
weights[treat==0]))
print(ATT.MatchIt) # 939.2
# GET ESTIMATE FROM MATCH()
# The Match() estimate is -952.3 -- very different.
Match(Y=lalonde$re78, Tr=lalonde$treat, X=X, estimand="ATT",
Weight.matrix=diag(weights), ties=TRUE)$est
# NOTE DISCREPANCY BETWEEN ncol(X) AND length(weights)
# There are only 8 variables in X, so why is matchit() producing weights
for nine variables?
ncol(X) # 8
length(weights) # 9
# ADD PROPENSITY SCORE TO X AND RE-ESTIMATE WITH MATCH()
glm1 <- glm(treat ~ age+educ+black+hispan+married+nodegree+re74+re75,
family=binomial, data=lalonde)
X2 <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
re74, re75))
X2 <- cbind(glm1$fitted, X2)
Match(Y=lalonde$re78, Tr=lalonde$treat, X=X2, estimand="ATT",
Weight.matrix=diag(weights), ties=TRUE)$est # 939.2
-
---
MatchIt mailing list served by HUIT
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/mailman/listinfo/ei
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
Browse/Search List Archive:
http://lists.gking.harvard.edu/mailman/private/matchit/
Matchit mailing list
Matchit(a)lists.gking.harvard.edu
https://lists.gking.harvard.edu/mailman/listinfo/matchit
9:54 p.m.
sounds like GenMatch has improved since I tried it last. that's great.
it's a good idea. thanks, we'll fix the manual.
Gary
--
*Gary King* - Albert J. Weatherhead III University Professor - Director,
IQSS - Harvard University
GKing.Harvard.edu <http://gking.harvard.edu/> - King(a)Harvard.edu -
@kinggary<http://twitter.com/kinggary>- 617-500-7570 - Asst 495-9271 -
Fax 812-8581
On Fri, Mar 16, 2012 at 1:24 PM, Rocio Titiunik <titiunik(a)umich.edu> wrote:
Thanks for your reply Gary.
Adding a propensity score to GenMatch is usually a good idea in
applications, and indeed it's recommended in Diamond and Sekhon
(2012). But we were concerned about replicability, because we thought
there was an error in our code when we couldn't replicate the matchit
results using GenMatch directly.
To continue the conversation, whether GenMatch finds or does not find
balance does not depend on whether one adds a propensity score to the
matching matrix. If simple matching on the propensity score leads to
perfect balance, then GenMatch will give the propensity score maximum
weight and will give zero weight to the rest of the components of the
matching matrix. In these cases, if one fails to include the
propensity score, GenMatch will still find the same solution, but it
will take longer than if the propensity score had been included. But
the opposite can also happen: if matching on the propensity score does
not improve balance at all, GenMatch will eventually figure this out
and assign zero weight to it, but it will take longer to find balance
than if the propensity score had not been included in the first place.
In other words, the Rosenbaum and Rubin theorem holds for the true
propensity score. Matching on a badly misspecified propensity score
will not improve balance, and in these cases including it in GenMatch
will slow the balance-finding process.
Rocío
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
On Fri, Mar 16, 2012 at 9:43 AM, Gary King <king(a)harvard.edu> wrote:
Thanks John and Rocío, Good catch, we'll
include it in the manual. FYI,
as
I recall when we wrote matchit, we added the
propensity score to genmatch
because genmatch didn't seem to reduce balance without it. that's odd
because genmatch already attempts to optimize an arbitrary function of
all
the variables in the pscore, and so if its doing
what its advertising
adding
yet another function of the same variables
shouldn't do anything. But you
never know what happens in such a complicated optimization problem, and
the
pscore often helped it do a bit better and so we
included it.
Gary
--
Gary King - Albert J. Weatherhead III University Professor - Director,
IQSS
- Harvard University
GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst
495-9271 - Fax 812-8581
2012/3/15 John G. Bullock <john.bullock(a)yale.edu>
>
>
> Hello,
>
> I am writing on behalf of myself and Rocío Titiunik.
>
> As we see it, matchit() serves as a front end to GenMatch() when
> method=='genetic'. So it should be possible to get the same results
from
> both the MatchIt and Matching packages. But
this proves difficult.
> Discrepancies seem to occur because matchit() silently adds a column of
> propensity scores to the X matrix that it passes to GenMatch(). I'm
> appending some code that we developed to demonstrate this point.
>
> The benefits (in most cases) of adding a propensity score are clear,
> but it seems problematic that this addition occurs without any
notification.
> For example, a user of MatchIt may claim to
have matched on a set of
> predictors, unaware that he has also matched on the propensity score.
Or,
> if a user thinks that he has matched on one
propensity score (because he
> included it in X when calling matchit()), he will be unaware that he has
> also matched on a second, matchit()-generated propensity score that is
based
> on the first propensity score. In either
case, he will be also be
unable to
> replicate his MatchIt results with Matching
or vice versa, even though
> MatchIt is supposed to be a wrapper for Matching. And third-party
readers
> of code that includes a call to matchit()
will likely have no idea that
a
> propensity score has been added.
>
> We can't see any explicit mention of this propensity-score-adding
> feature in the MatchIt manual. To promote future replication efforts,
can a
> line be added to the MatchIt manual about the
addition of a propensity
score
> to X? Or can a note be added to matchit()
output whenever
method=='genetic'
> is used?
>
> Thank you,
> John Bullock
> ###
>
>
> library(Matching)
> library(MatchIt)
> data(lalonde)
> X <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
> re74, re75))
>
> # GET ESTIMATE FROM MATCHIT()
> set.seed(5678)
> m.out1 <- matchit(treat ~ X, data=lalonde, method='genetic',
> estimand='ATT', ties=TRUE, print.level=1,
> pop.size=150, wait.generations=1,
> max.generations=10, hard.generation.limit=TRUE,
> unif.seed=1945, int.seed=1906)
>
> # Create a vector of "parameters at the solution" reported by matchit().
> # Behind the scenes, these weights have been produced by GenMatch().
> weights <- c(8.121767e+02, 7.735192e+02, 5.938936e+02, 4.079661e+02,
> 3.665018e+02, 1.361011e+02,
> 9.644896e+02, 6.083636e+02, 2.346772e+00)
>
> ATT.MatchIt <- with(match.data(m.out1), weighted.mean(re78[treat==1],
> weights[treat==1])) -
> with(match.data(m.out1), weighted.mean(re78[treat==0],
> weights[treat==0]))
> print(ATT.MatchIt) # 939.2
>
>
> # GET ESTIMATE FROM MATCH()
> # The Match() estimate is -952.3 -- very different.
> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X, estimand="ATT",
> Weight.matrix=diag(weights), ties=TRUE)$est
>
>
> # NOTE DISCREPANCY BETWEEN ncol(X) AND length(weights)
> # There are only 8 variables in X, so why is matchit() producing weights
> for nine variables?
> ncol(X) # 8
> length(weights) # 9
>
>
> # ADD PROPENSITY SCORE TO X AND RE-ESTIMATE WITH MATCH()
> glm1 <- glm(treat ~ age+educ+black+hispan+married+nodegree+re74+re75,
> family=binomial, data=lalonde)
> X2 <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
> re74, re75))
> X2 <- cbind(glm1$fitted, X2)
>
> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X2, estimand="ATT",
> Weight.matrix=diag(weights), ties=TRUE)$est # 939.2
>
> -
> ---
> MatchIt mailing list served by HUIT
> List Address: matchit(a)lists.gking.harvard.edu
> Subscribe/Unsubscribe:
http://lists.gking.harvard.edu/mailman/listinfo/ei
MatchIt
Software and Documentation: http://gking.harvard.edu/matchit/
Browse/Search List Archive:
http://lists.gking.harvard.edu/mailman/private/matchit/
Matchit mailing list
Matchit(a)lists.gking.harvard.edu
https://lists.gking.harvard.edu/mailman/listinfo/matchit
17 Mar
17 Mar
10:13 a.m.
Hi Rocío & John, after a nudge from Kosuke, I have a question. Did you run
some type of simulation to come to the conclusion in your last paragraph,
or can you describe the data sets where it worked? Earlier versions of
this method did not have the property you describe in example data sets we
tried, it is usually easier and faster to produce better balance with other
methods, and of course no theoretical result guarantees that the claims in
your paragraph are general. So it would be nice to know the types of data
sets where it worked the way you describe. If you can eventually figure
out how to more formally characterize in what types of data it works as you
describe, that might be very useful.
Gary
--
*Gary King* - Albert J. Weatherhead III University Professor - Director,
IQSS - Harvard University
GKing.Harvard.edu <http://gking.harvard.edu/> - King(a)Harvard.edu -
@kinggary<http://twitter.com/kinggary>- 617-500-7570 - Asst 495-9271 -
Fax 812-8581
On Fri, Mar 16, 2012 at 1:24 PM, Rocio Titiunik <titiunik(a)umich.edu> wrote:
Thanks for your reply Gary.
Adding a propensity score to GenMatch is usually a good idea in
applications, and indeed it's recommended in Diamond and Sekhon
(2012). But we were concerned about replicability, because we thought
there was an error in our code when we couldn't replicate the matchit
results using GenMatch directly.
To continue the conversation, whether GenMatch finds or does not find
balance does not depend on whether one adds a propensity score to the
matching matrix. If simple matching on the propensity score leads to
perfect balance, then GenMatch will give the propensity score maximum
weight and will give zero weight to the rest of the components of the
matching matrix. In these cases, if one fails to include the
propensity score, GenMatch will still find the same solution, but it
will take longer than if the propensity score had been included. But
the opposite can also happen: if matching on the propensity score does
not improve balance at all, GenMatch will eventually figure this out
and assign zero weight to it, but it will take longer to find balance
than if the propensity score had not been included in the first place.
In other words, the Rosenbaum and Rubin theorem holds for the true
propensity score. Matching on a badly misspecified propensity score
will not improve balance, and in these cases including it in GenMatch
will slow the balance-finding process.
Rocío
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
On Fri, Mar 16, 2012 at 9:43 AM, Gary King <king(a)harvard.edu> wrote:
Thanks John and Rocío, Good catch, we'll
include it in the manual. FYI,
as
I recall when we wrote matchit, we added the
propensity score to genmatch
because genmatch didn't seem to reduce balance without it. that's odd
because genmatch already attempts to optimize an arbitrary function of
all
the variables in the pscore, and so if its doing
what its advertising
adding
yet another function of the same variables
shouldn't do anything. But you
never know what happens in such a complicated optimization problem, and
the
pscore often helped it do a bit better and so we
included it.
Gary
--
Gary King - Albert J. Weatherhead III University Professor - Director,
IQSS
- Harvard University
GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst
495-9271 - Fax 812-8581
2012/3/15 John G. Bullock <john.bullock(a)yale.edu>
>
>
> Hello,
>
> I am writing on behalf of myself and Rocío Titiunik.
>
> As we see it, matchit() serves as a front end to GenMatch() when
> method=='genetic'. So it should be possible to get the same results
from
> both the MatchIt and Matching packages. But
this proves difficult.
> Discrepancies seem to occur because matchit() silently adds a column of
> propensity scores to the X matrix that it passes to GenMatch(). I'm
> appending some code that we developed to demonstrate this point.
>
> The benefits (in most cases) of adding a propensity score are clear,
> but it seems problematic that this addition occurs without any
notification.
> For example, a user of MatchIt may claim to
have matched on a set of
> predictors, unaware that he has also matched on the propensity score.
Or,
> if a user thinks that he has matched on one
propensity score (because he
> included it in X when calling matchit()), he will be unaware that he has
> also matched on a second, matchit()-generated propensity score that is
based
> on the first propensity score. In either
case, he will be also be
unable to
> replicate his MatchIt results with Matching
or vice versa, even though
> MatchIt is supposed to be a wrapper for Matching. And third-party
readers
> of code that includes a call to matchit()
will likely have no idea that
a
> propensity score has been added.
>
> We can't see any explicit mention of this propensity-score-adding
> feature in the MatchIt manual. To promote future replication efforts,
can a
> line be added to the MatchIt manual about the
addition of a propensity
score
> to X? Or can a note be added to matchit()
output whenever
method=='genetic'
> is used?
>
> Thank you,
> John Bullock
> ###
>
>
> library(Matching)
> library(MatchIt)
> data(lalonde)
> X <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
> re74, re75))
>
> # GET ESTIMATE FROM MATCHIT()
> set.seed(5678)
> m.out1 <- matchit(treat ~ X, data=lalonde, method='genetic',
> estimand='ATT', ties=TRUE, print.level=1,
> pop.size=150, wait.generations=1,
> max.generations=10, hard.generation.limit=TRUE,
> unif.seed=1945, int.seed=1906)
>
> # Create a vector of "parameters at the solution" reported by matchit().
> # Behind the scenes, these weights have been produced by GenMatch().
> weights <- c(8.121767e+02, 7.735192e+02, 5.938936e+02, 4.079661e+02,
> 3.665018e+02, 1.361011e+02,
> 9.644896e+02, 6.083636e+02, 2.346772e+00)
>
> ATT.MatchIt <- with(match.data(m.out1), weighted.mean(re78[treat==1],
> weights[treat==1])) -
> with(match.data(m.out1), weighted.mean(re78[treat==0],
> weights[treat==0]))
> print(ATT.MatchIt) # 939.2
>
>
> # GET ESTIMATE FROM MATCH()
> # The Match() estimate is -952.3 -- very different.
> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X, estimand="ATT",
> Weight.matrix=diag(weights), ties=TRUE)$est
>
>
> # NOTE DISCREPANCY BETWEEN ncol(X) AND length(weights)
> # There are only 8 variables in X, so why is matchit() producing weights
> for nine variables?
> ncol(X) # 8
> length(weights) # 9
>
>
> # ADD PROPENSITY SCORE TO X AND RE-ESTIMATE WITH MATCH()
> glm1 <- glm(treat ~ age+educ+black+hispan+married+nodegree+re74+re75,
> family=binomial, data=lalonde)
> X2 <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
> re74, re75))
> X2 <- cbind(glm1$fitted, X2)
>
> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X2, estimand="ATT",
> Weight.matrix=diag(weights), ties=TRUE)$est # 939.2
>
> -
> ---
> MatchIt mailing list served by HUIT
> List Address: matchit(a)lists.gking.harvard.edu
> Subscribe/Unsubscribe:
http://lists.gking.harvard.edu/mailman/listinfo/ei
MatchIt
Software and Documentation: http://gking.harvard.edu/matchit/
Browse/Search List Archive:
http://lists.gking.harvard.edu/mailman/private/matchit/
Matchit mailing list
Matchit(a)lists.gking.harvard.edu
https://lists.gking.harvard.edu/mailman/listinfo/matchit
10 Apr
10 Apr
10:16 a.m.
Hi Gary,
Sorry for my extremely tardy response -- I've been traveling a lot and
could not find the time to continue our conversation. But I am
responding now, on the off chance that you still remember what our
conversation was about!
To answer your question, no, I have not run simulations to come to
this conclusion. What I said is based on my understanding of the
GenMatch algorithm and the explanation on the paper by Diamond and
Sekhon in the Review of Economics and Statistics
(http://sekhon.berkeley.edu/papers/GenMatch.pdf)
There are two parts to what I said. One is that matching on a
miss-specified propensity score will not necessarily lead to matched
samples where balance is better than, say, in the unmatched data. This
is a statement that is unrelated to GenMatch, it applies in general,
and is based on the fact that the theorem in Rosenbaum and Rubin 1983
is about the true propensity score. But I think we are not disagreeing
about this.
If I understand your comment, the discussion is about the part of my
statement where I said:
> If simple matching on the propensity score leads
to
> perfect balance, then GenMatch will give the propensity score maximum
> weight and will give zero weight to the rest of the components of the
> matching matrix. In these cases, if one fails to include the
> propensity score, GenMatch will still find the same solution, but it
> will take longer than if the propensity score had been included.
As I said, I have not run simulations to show this, although I think
it would be great to have such simulations and perhaps I'll find some
time to work on them. My statement was based on my knowledge of the
algorithm: GenMatch finds matches by minimizing a generalized
Mahalanobis distance, and finds weights so that this distance will
produce high-balance matches when minimized. The weights are chosen so
that balance is maximized, and if the algorithm finds that giving
positive weights to one or more of the variables in X creates worse
balance than giving zero weights to them, then these variables will be
given zero weights (technically, very small but positive weights so
that the weight matrix is positive definite). So, incorporating a
miss-specified propensity score may, in the beginning, make balance
worse, but eventually the algorithm will figure out that the
miss-specified propensity score should not be given positive weight.
Now, the key is the phrase "eventually", because it's impossible to
anticipate when this will happen, whether it will be feasible to wait
enough generations to get there, etc. So it is possible that one stops
the algorithm before the zero-weight-to-pscore solution has been
found. Similarly, if one fails to include the propensity score and the
propensity score is correctly specified, GenMatch should be able to
find the weights so that matching on X without the pscore leads to
same degree of balance, but it can of course happen that in some
problems the number of generations that the algorithm has to run to
find the optimal weights is too large and therefore not feasible in
practice.
My point was simply that although we often do see in applications that
adding an estimated propensity score to GenMatch helps to find better
balance, this is not true in general. And, actually, this statement
applies to any nearest neighbor matching algorithm, because we can
never be sure that the propensity score we added was the correct one.
Do you or Kosuke disagree with this statement? If so, I'll be very
interested in reading your thoughts about it. Again, I think the idea
of having simulations to show this is great, as it will be very useful
to characterize, if at all possible, the cases where adding an
estimated propensity score might be detrimental. I will be sure to
share them with list if I get around to do them.
Cheers,
Rocío
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
On Sat, Mar 17, 2012 at 10:13 AM, Gary King <king(a)harvard.edu> wrote:
Hi Rocío & John, after a nudge from Kosuke, I have
a question. Did you run
some type of simulation to come to the conclusion in your last paragraph, or
can you describe the data sets where it worked? Earlier versions of this
method did not have the property you describe in example data sets we tried,
it is usually easier and faster to produce better balance with other
methods, and of course no theoretical result guarantees that the claims in
your paragraph are general. So it would be nice to know the types of data
sets where it worked the way you describe. If you can eventually figure
out how to more formally characterize in what types of data it works as you
describe, that might be very useful.
Gary
--
Gary King - Albert J. Weatherhead III University Professor - Director, IQSS
- Harvard University
GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst
495-9271 - Fax 812-8581
On Fri, Mar 16, 2012 at 1:24 PM, Rocio Titiunik <titiunik(a)umich.edu> wrote:
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
Thanks for your reply Gary.
Adding a propensity score to GenMatch is usually a good idea in
applications, and indeed it's recommended in Diamond and Sekhon
(2012). But we were concerned about replicability, because we thought
there was an error in our code when we couldn't replicate the matchit
results using GenMatch directly.
To continue the conversation, whether GenMatch finds or does not find
balance does not depend on whether one adds a propensity score to the
matching matrix. If simple matching on the propensity score leads to
perfect balance, then GenMatch will give the propensity score maximum
weight and will give zero weight to the rest of the components of the
matching matrix. In these cases, if one fails to include the
propensity score, GenMatch will still find the same solution, but it
will take longer than if the propensity score had been included. But
the opposite can also happen: if matching on the propensity score does
not improve balance at all, GenMatch will eventually figure this out
and assign zero weight to it, but it will take longer to find balance
than if the propensity score had not been included in the first place.
In other words, the Rosenbaum and Rubin theorem holds for the true
propensity score. Matching on a badly misspecified propensity score
will not improve balance, and in these cases including it in GenMatch
will slow the balance-finding process.
Rocío
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
On Fri, Mar 16, 2012 at 9:43 AM, Gary King <king(a)harvard.edu> wrote:
> Thanks John and Rocío, Good catch, we'll include it in the manual. FYI,
> as
> I recall when we wrote matchit, we added the propensity score to
> genmatch
> because genmatch didn't seem to reduce balance without it. that's odd
> because genmatch already attempts to optimize an arbitrary function of
> all
> the variables in the pscore, and so if its doing what its advertising
> adding
> yet another function of the same variables shouldn't do anything. But
> you
> never know what happens in such a complicated optimization problem, and
> the
> pscore often helped it do a bit better and so we included it.
> Gary
> --
> Gary King - Albert J. Weatherhead III University Professor - Director,
> IQSS
> - Harvard University
> GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst
> 495-9271 - Fax 812-8581
>
>
>
> 2012/3/15 John G. Bullock <john.bullock(a)yale.edu>
>>
>>
>> Hello,
>>
>> I am writing on behalf of myself and Rocío Titiunik.
>>
>> As we see it, matchit() serves as a front end to GenMatch() when
>> method=='genetic'. So it should be possible to get the same results
>> from
>> both the MatchIt and Matching packages. But this proves difficult.
>> Discrepancies seem to occur because matchit() silently adds a column
>> of
>> propensity scores to the X matrix that it passes to GenMatch(). I'm
>> appending some code that we developed to demonstrate this point.
>>
>> The benefits (in most cases) of adding a propensity score are clear,
>> but it seems problematic that this addition occurs without any
>> notification.
>> For example, a user of MatchIt may claim to have matched on a set of
>> predictors, unaware that he has also matched on the propensity score.
>> Or,
>> if a user thinks that he has matched on one propensity score (because
>> he
>> included it in X when calling matchit()), he will be unaware that he
>> has
>> also matched on a second, matchit()-generated propensity score that is
>> based
>> on the first propensity score. In either case, he will be also be
>> unable to
>> replicate his MatchIt results with Matching or vice versa, even though
>> MatchIt is supposed to be a wrapper for Matching. And third-party
>> readers
>> of code that includes a call to matchit() will likely have no idea that
>> a
>> propensity score has been added.
>>
>> We can't see any explicit mention of this propensity-score-adding
>> feature in the MatchIt manual. To promote future replication efforts,
>> can a
>> line be added to the MatchIt manual about the addition of a propensity
>> score
>> to X? Or can a note be added to matchit() output whenever
>> method=='genetic'
>> is used?
>>
>> Thank you,
>> John Bullock
>> ###
>>
>>
>> library(Matching)
>> library(MatchIt)
>> data(lalonde)
>> X <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
>> re74, re75))
>>
>> # GET ESTIMATE FROM MATCHIT()
>> set.seed(5678)
>> m.out1 <- matchit(treat ~ X, data=lalonde, method='genetic',
>> estimand='ATT', ties=TRUE, print.level=1,
>> pop.size=150, wait.generations=1,
>> max.generations=10, hard.generation.limit=TRUE,
>> unif.seed=1945, int.seed=1906)
>>
>> # Create a vector of "parameters at the solution" reported by
>> matchit().
>> # Behind the scenes, these weights have been produced by GenMatch().
>> weights <- c(8.121767e+02, 7.735192e+02, 5.938936e+02, 4.079661e+02,
>> 3.665018e+02, 1.361011e+02,
>> 9.644896e+02, 6.083636e+02, 2.346772e+00)
>>
>> ATT.MatchIt <- with(match.data(m.out1), weighted.mean(re78[treat==1],
>> weights[treat==1])) -
>> with(match.data(m.out1), weighted.mean(re78[treat==0],
>> weights[treat==0]))
>> print(ATT.MatchIt) # 939.2
>>
>>
>> # GET ESTIMATE FROM MATCH()
>> # The Match() estimate is -952.3 -- very different.
>> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X, estimand="ATT",
>> Weight.matrix=diag(weights), ties=TRUE)$est
>>
>>
>> # NOTE DISCREPANCY BETWEEN ncol(X) AND length(weights)
>> # There are only 8 variables in X, so why is matchit() producing
>> weights
>> for nine variables?
>> ncol(X) # 8
>> length(weights) # 9
>>
>>
>> # ADD PROPENSITY SCORE TO X AND RE-ESTIMATE WITH MATCH()
>> glm1 <- glm(treat ~ age+educ+black+hispan+married+nodegree+re74+re75,
>> family=binomial, data=lalonde)
>> X2 <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
>> re74, re75))
>> X2 <- cbind(glm1$fitted, X2)
>>
>> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X2, estimand="ATT",
>> Weight.matrix=diag(weights), ties=TRUE)$est # 939.2
>>
>> -
>> ---
>> MatchIt mailing list served by HUIT
>> List Address: matchit(a)lists.gking.harvard.edu
>> Subscribe/Unsubscribe:
>> http://lists.gking.harvard.edu/mailman/listinfo/ei
>> MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
>> Browse/Search List Archive:
>> http://lists.gking.harvard.edu/mailman/private/matchit/
>> Matchit mailing list
>> Matchit(a)lists.gking.harvard.edu
>> https://lists.gking.harvard.edu/mailman/listinfo/matchit
>
>
4:11 p.m.
This all seems reasonable, but nothing in this is guaranteed without a
theorem we'd all like but doesn't exist. So without that, everything -
including all claims in your message - may or may not hold in any one data
set. If you're finding it works well for your data, then that's terrific.
Since you can't guarantee that it will work the next time, you just have
to check after the fact whether balance is achieved, tweak the model or
algorithm, and then rerun, check balance, and continue to iterate.
You can avoid the iteration with CEM <http://gking.harvard.edu/cem>, since
the max imbalance in that approach is set by the user before matching, and
there are theorems to back this up. Kosuke also has a covariate balancing
pscore <http://imai.princeton.edu/research/CBPS.html> approach that uses
the pscore for weighting instead of matching that seems to work well. There
are lots of other approaches too.
Best,
Gary
--
*Gary King* - Albert J. Weatherhead III University Professor - Director,
IQSS - Harvard University
GKing.Harvard.edu <http://gking.harvard.edu/> - King(a)Harvard.edu -
@kinggary<http://twitter.com/kinggary>- 617-500-7570 - Asst 495-9271 -
Fax 812-8581
On Tue, Apr 10, 2012 at 10:16 AM, Rocio Titiunik <titiunik(a)umich.edu> wrote:
Hi Gary,
Sorry for my extremely tardy response -- I've been traveling a lot and
could not find the time to continue our conversation. But I am
responding now, on the off chance that you still remember what our
conversation was about!
To answer your question, no, I have not run simulations to come to
this conclusion. What I said is based on my understanding of the
GenMatch algorithm and the explanation on the paper by Diamond and
Sekhon in the Review of Economics and Statistics
(http://sekhon.berkeley.edu/papers/GenMatch.pdf)
There are two parts to what I said. One is that matching on a
miss-specified propensity score will not necessarily lead to matched
samples where balance is better than, say, in the unmatched data. This
is a statement that is unrelated to GenMatch, it applies in general,
and is based on the fact that the theorem in Rosenbaum and Rubin 1983
is about the true propensity score. But I think we are not disagreeing
about this.
If I understand your comment, the discussion is about the part of my
statement where I said:
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
> If simple matching on the propensity score
leads to
> perfect balance, then GenMatch will give the propensity score maximum
> weight and will give zero weight to the rest of the components of the
> matching matrix. In these cases, if one fails to include the
> propensity score, GenMatch will still find the same solution, but it
> will take longer than if the propensity score had been included.
As I said, I have not run simulations to show this, although I think
it would be great to have such simulations and perhaps I'll find some
time to work on them. My statement was based on my knowledge of the
algorithm: GenMatch finds matches by minimizing a generalized
Mahalanobis distance, and finds weights so that this distance will
produce high-balance matches when minimized. The weights are chosen so
that balance is maximized, and if the algorithm finds that giving
positive weights to one or more of the variables in X creates worse
balance than giving zero weights to them, then these variables will be
given zero weights (technically, very small but positive weights so
that the weight matrix is positive definite). So, incorporating a
miss-specified propensity score may, in the beginning, make balance
worse, but eventually the algorithm will figure out that the
miss-specified propensity score should not be given positive weight.
Now, the key is the phrase "eventually", because it's impossible to
anticipate when this will happen, whether it will be feasible to wait
enough generations to get there, etc. So it is possible that one stops
the algorithm before the zero-weight-to-pscore solution has been
found. Similarly, if one fails to include the propensity score and the
propensity score is correctly specified, GenMatch should be able to
find the weights so that matching on X without the pscore leads to
same degree of balance, but it can of course happen that in some
problems the number of generations that the algorithm has to run to
find the optimal weights is too large and therefore not feasible in
practice.
My point was simply that although we often do see in applications that
adding an estimated propensity score to GenMatch helps to find better
balance, this is not true in general. And, actually, this statement
applies to any nearest neighbor matching algorithm, because we can
never be sure that the propensity score we added was the correct one.
Do you or Kosuke disagree with this statement? If so, I'll be very
interested in reading your thoughts about it. Again, I think the idea
of having simulations to show this is great, as it will be very useful
to characterize, if at all possible, the cases where adding an
estimated propensity score might be detrimental. I will be sure to
share them with list if I get around to do them.
Cheers,
Rocío
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
On Sat, Mar 17, 2012 at 10:13 AM, Gary King <king(a)harvard.edu> wrote:
Hi Rocío & John, after a nudge from Kosuke, I
have a question. Did you
run
some type of simulation to come to the conclusion
in your last
paragraph, or
can you describe the data sets where it worked?
Earlier versions of
this
method did not have the property you describe in
example data sets we
tried,
it is usually easier and faster to produce better
balance with other
methods, and of course no theoretical result guarantees that the claims
in
your paragraph are general. So it would be nice
to know the types of
data
sets where it worked the way you describe. If
you can eventually figure
out how to more formally characterize in what types of data it works as
you
describe, that might be very useful.
Gary
--
Gary King - Albert J. Weatherhead III University Professor - Director,
IQSS
- Harvard University
GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst
495-9271 - Fax 812-8581
On Fri, Mar 16, 2012 at 1:24 PM, Rocio Titiunik <titiunik(a)umich.edu>
wrote:
>
> Thanks for your reply Gary.
>
> Adding a propensity score to GenMatch is usually a good idea in
> applications, and indeed it's recommended in Diamond and Sekhon
> (2012). But we were concerned about replicability, because we thought
> there was an error in our code when we couldn't replicate the matchit
> results using GenMatch directly.
>
> To continue the conversation, whether GenMatch finds or does not find
> balance does not depend on whether one adds a propensity score to the
> matching matrix. If simple matching on the propensity score leads to
> perfect balance, then GenMatch will give the propensity score maximum
> weight and will give zero weight to the rest of the components of the
> matching matrix. In these cases, if one fails to include the
> propensity score, GenMatch will still find the same solution, but it
> will take longer than if the propensity score had been included. But
> the opposite can also happen: if matching on the propensity score does
> not improve balance at all, GenMatch will eventually figure this out
> and assign zero weight to it, but it will take longer to find balance
> than if the propensity score had not been included in the first place.
> In other words, the Rosenbaum and Rubin theorem holds for the true
> propensity score. Matching on a badly misspecified propensity score
> will not improve balance, and in these cases including it in GenMatch
> will slow the balance-finding process.
>
>
> Rocío
> --
> Rocío Titiunik
> Assistant Professor
> Department of Political Science
> University of Michigan
> http://www.umich.edu/~titiunik/
>
>
>
> On Fri, Mar 16, 2012 at 9:43 AM, Gary King <king(a)harvard.edu> wrote:
> > Thanks John and Rocío, Good catch, we'll include it in the manual.
FYI,
> > as
> > I recall when we wrote matchit, we added the propensity score to
> > genmatch
> > because genmatch didn't seem to reduce balance without it. that's
odd
> > because genmatch already attempts to optimize an arbitrary function of
> > all
> > the variables in the pscore, and so if its doing what its advertising
> > adding
> > yet another function of the same variables shouldn't do anything. But
> > you
> > never know what happens in such a complicated optimization problem,
and
> > the
> > pscore often helped it do a bit better and so we included it.
> > Gary
> > --
> > Gary King - Albert J. Weatherhead III University Professor - Director,
> > IQSS
> > - Harvard University
> > GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 -
Asst
> > 495-9271 - Fax 812-8581
> >
> >
> >
> > 2012/3/15 John G. Bullock <john.bullock(a)yale.edu>
> >>
> >>
> >> Hello,
> >>
> >> I am writing on behalf of myself and Rocío Titiunik.
> >>
> >> As we see it, matchit() serves as a front end to GenMatch() when
> >> method=='genetic'. So it should be possible to get the same
results
> >> from
> >> both the MatchIt and Matching packages. But this proves difficult.
> >> Discrepancies seem to occur because matchit() silently adds a column
> >> of
> >> propensity scores to the X matrix that it passes to GenMatch(). I'm
> >> appending some code that we developed to demonstrate this point.
> >>
> >> The benefits (in most cases) of adding a propensity score are
clear,
> >> but it seems problematic that this
addition occurs without any
> >> notification.
> >> For example, a user of MatchIt may claim to have matched on a set of
> >> predictors, unaware that he has also matched on the propensity score.
> >> Or,
> >> if a user thinks that he has matched on one propensity score (because
> >> he
> >> included it in X when calling matchit()), he will be unaware that he
> >> has
> >> also matched on a second, matchit()-generated propensity score that
is
> >> based
> >> on the first propensity score. In either case, he will be also be
> >> unable to
> >> replicate his MatchIt results with Matching or vice versa, even
though
> >> MatchIt is supposed to be a wrapper
for Matching. And third-party
> >> readers
> >> of code that includes a call to matchit() will likely have no idea
that
> >> a
> >> propensity score has been added.
> >>
> >> We can't see any explicit mention of this propensity-score-adding
> >> feature in the MatchIt manual. To promote future replication
efforts,
> >> can a
> >> line be added to the MatchIt manual about the addition of a
propensity
> >> score
> >> to X? Or can a note be added to matchit() output whenever
> >> method=='genetic'
> >> is used?
> >>
> >> Thank you,
> >> John Bullock
> >> ###
> >>
> >>
> >> library(Matching)
> >> library(MatchIt)
> >> data(lalonde)
> >> X <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
> >> re74, re75))
> >>
> >> # GET ESTIMATE FROM MATCHIT()
> >> set.seed(5678)
> >> m.out1 <- matchit(treat ~ X, data=lalonde, method='genetic',
> >> estimand='ATT', ties=TRUE, print.level=1,
> >> pop.size=150, wait.generations=1,
> >> max.generations=10, hard.generation.limit=TRUE,
> >> unif.seed=1945, int.seed=1906)
> >>
> >> # Create a vector of "parameters at the solution" reported by
> >> matchit().
> >> # Behind the scenes, these weights have been produced by GenMatch().
> >> weights <- c(8.121767e+02, 7.735192e+02, 5.938936e+02, 4.079661e+02,
> >> 3.665018e+02, 1.361011e+02,
> >> 9.644896e+02, 6.083636e+02, 2.346772e+00)
> >>
> >> ATT.MatchIt <- with(match.data(m.out1), weighted.mean(re78[treat==1],
> >> weights[treat==1])) -
> >> with(match.data(m.out1), weighted.mean(re78[treat==0],
> >> weights[treat==0]))
> >> print(ATT.MatchIt) # 939.2
> >>
> >>
> >> # GET ESTIMATE FROM MATCH()
> >> # The Match() estimate is -952.3 -- very different.
> >> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X, estimand="ATT",
> >> Weight.matrix=diag(weights), ties=TRUE)$est
> >>
> >>
> >> # NOTE DISCREPANCY BETWEEN ncol(X) AND length(weights)
> >> # There are only 8 variables in X, so why is matchit() producing
> >> weights
> >> for nine variables?
> >> ncol(X) # 8
> >> length(weights) # 9
> >>
> >>
> >> # ADD PROPENSITY SCORE TO X AND RE-ESTIMATE WITH MATCH()
> >> glm1 <- glm(treat ~ age+educ+black+hispan+married+nodegree+re74+re75,
> >> family=binomial, data=lalonde)
> >> X2 <- with(lalonde, cbind(age, educ, black, hispan, married,
nodegree,
> >> re74, re75))
> >> X2 <- cbind(glm1$fitted, X2)
> >>
> >> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X2, estimand="ATT",
> >> Weight.matrix=diag(weights), ties=TRUE)$est # 939.2
> >>
> >> -
> >> ---
> >> MatchIt mailing list served by HUIT
> >> List Address: matchit(a)lists.gking.harvard.edu
> >> Subscribe/Unsubscribe:
> >> http://lists.gking.harvard.edu/mailman/listinfo/ei
> >> MatchIt Software and Documentation:
http://gking.harvard.edu/matchit/
>>
Browse/Search List Archive:
>> http://lists.gking.harvard.edu/mailman/private/matchit/
>> Matchit mailing list
>> Matchit(a)lists.gking.harvard.edu
>> https://lists.gking.harvard.edu/mailman/listinfo/matchit
>
>
5:04 p.m.
Rocio,
I'm speaking without knowing the details of the numerical algorithm used by
GenMatch, but I don't think GenMatch will assign zero weights to the estimated
propensity score when the propensity score model is misspecified. There are two reasons
for this. First, the propensity score is likely to be correlated with covariates and
usually a parametric model like the one GenMatch is unable to assign zero weight to only
one of several highly correlated variables without some kind of explicit regularization
for variable selection. Second, it is not even clear to me whether it is desirable to
assign zero weight to the misspecified propensity score. If matching balances the
covariate distribution (not just marginal means etc.), then by definition the treatment
should be independent of *any* function that involves pre-treatment covariates, which
includes the misspecified propensity score. What Marc Ratkovic and I do in our covariate
balancing propensity score is that we estimate the propensity score such that the
resulting propensity score maximizes balance (measured in terms of moments of some
functions of covariates selected by researchers) as well as predicts the treatment
assignment (the definition of propensity score). My understanding is that CEM avoids the
problem by having the researchers bin the covariate space and balancing the covariate
distributions in this coarsened covariate space. Jen's entropy weighting and
Marc's SVM matching basically do the same thing. I think that the issue of which
functions (or bin sizes) of covariates you choose is an open question. There are two ways
to deal with this, I think. The first is a nonparametric approach. I'm playing with
this but at this point it is not clear to me whether it works sufficiently well in
practice. For any nonparametric method, the choice of bandwidth parameter is always an
issue too. The second is to specify an outcome model or at least use the knowledge of the
outcome variable in some creative ways. I've seen some working papers that try this,
but I don't know how well these approaches work in general.
Best,
Kosuke
Department of Politics
Princeton University
http://imai.princeton.edu
On Apr 10, 2012, at 4:11 PM, Gary King wrote:
This all seems reasonable, but nothing in this is
guaranteed without a theorem we'd all like but doesn't exist. So without that,
everything - including all claims in your message - may or may not hold in any one data
set. If you're finding it works well for your data, then that's terrific.
Since you can't guarantee that it will work the next time, you just have to check
after the fact whether balance is achieved, tweak the model or algorithm, and then rerun,
check balance, and continue to iterate.
You can avoid the iteration with CEM, since the max imbalance in that approach is set by
the user before matching, and there are theorems to back this up. Kosuke also has a
covariate balancing pscore approach that uses the pscore for weighting instead of matching
that seems to work well. There are lots of other approaches too.
Best,
Gary
--
Gary King - Albert J. Weatherhead III University Professor - Director, IQSS - Harvard
University
GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst 495-9271 - Fax
812-8581
On Tue, Apr 10, 2012 at 10:16 AM, Rocio Titiunik <titiunik(a)umich.edu> wrote:
Hi Gary,
Sorry for my extremely tardy response -- I've been traveling a lot and
could not find the time to continue our conversation. But I am
responding now, on the off chance that you still remember what our
conversation was about!
To answer your question, no, I have not run simulations to come to
this conclusion. What I said is based on my understanding of the
GenMatch algorithm and the explanation on the paper by Diamond and
Sekhon in the Review of Economics and Statistics
(http://sekhon.berkeley.edu/papers/GenMatch.pdf)
There are two parts to what I said. One is that matching on a
miss-specified propensity score will not necessarily lead to matched
samples where balance is better than, say, in the unmatched data. This
is a statement that is unrelated to GenMatch, it applies in general,
and is based on the fact that the theorem in Rosenbaum and Rubin 1983
is about the true propensity score. But I think we are not disagreeing
about this.
If I understand your comment, the discussion is about the part of my
statement where I said:
> If simple matching on the propensity score
leads to
> perfect balance, then GenMatch will give the propensity score maximum
> weight and will give zero weight to the rest of the components of the
> matching matrix. In these cases, if one fails to include the
> propensity score, GenMatch will still find the same solution, but it
> will take longer than if the propensity score had been included.
As I said, I have not run simulations to show this, although I think
it would be great to have such simulations and perhaps I'll find some
time to work on them. My statement was based on my knowledge of the
algorithm: GenMatch finds matches by minimizing a generalized
Mahalanobis distance, and finds weights so that this distance will
produce high-balance matches when minimized. The weights are chosen so
that balance is maximized, and if the algorithm finds that giving
positive weights to one or more of the variables in X creates worse
balance than giving zero weights to them, then these variables will be
given zero weights (technically, very small but positive weights so
that the weight matrix is positive definite). So, incorporating a
miss-specified propensity score may, in the beginning, make balance
worse, but eventually the algorithm will figure out that the
miss-specified propensity score should not be given positive weight.
Now, the key is the phrase "eventually", because it's impossible to
anticipate when this will happen, whether it will be feasible to wait
enough generations to get there, etc. So it is possible that one stops
the algorithm before the zero-weight-to-pscore solution has been
found. Similarly, if one fails to include the propensity score and the
propensity score is correctly specified, GenMatch should be able to
find the weights so that matching on X without the pscore leads to
same degree of balance, but it can of course happen that in some
problems the number of generations that the algorithm has to run to
find the optimal weights is too large and therefore not feasible in
practice.
My point was simply that although we often do see in applications that
adding an estimated propensity score to GenMatch helps to find better
balance, this is not true in general. And, actually, this statement
applies to any nearest neighbor matching algorithm, because we can
never be sure that the propensity score we added was the correct one.
Do you or Kosuke disagree with this statement? If so, I'll be very
interested in reading your thoughts about it. Again, I think the idea
of having simulations to show this is great, as it will be very useful
to characterize, if at all possible, the cases where adding an
estimated propensity score might be detrimental. I will be sure to
share them with list if I get around to do them.
Cheers,
Rocío
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
On Sat, Mar 17, 2012 at 10:13 AM, Gary King <king(a)harvard.edu> wrote:
Hi Rocío & John, after a nudge from Kosuke, I
have a question. Did you run
some type of simulation to come to the conclusion in your last paragraph, or
can you describe the data sets where it worked? Earlier versions of this
method did not have the property you describe in example data sets we tried,
it is usually easier and faster to produce better balance with other
methods, and of course no theoretical result guarantees that the claims in
your paragraph are general. So it would be nice to know the types of data
sets where it worked the way you describe. If you can eventually figure
out how to more formally characterize in what types of data it works as you
describe, that might be very useful.
Gary
--
Gary King - Albert J. Weatherhead III University Professor - Director, IQSS
- Harvard University
GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst
495-9271 - Fax 812-8581
On Fri, Mar 16, 2012 at 1:24 PM, Rocio Titiunik <titiunik(a)umich.edu> wrote:
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
-
---
MatchIt mailing list served by HUIT
List Address: matchit(a)lists.gking.harvard.edu
Subscribe/Unsubscribe: http://lists.gking.harvard.edu/mailman/listinfo/ei
MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
Browse/Search List Archive: http://lists.gking.harvard.edu/mailman/private/matchit/
Matchit mailing list
Matchit(a)lists.gking.harvard.edu
https://lists.gking.harvard.edu/mailman/listinfo/matchit
Thanks for your reply Gary.
Adding a propensity score to GenMatch is usually a good idea in
applications, and indeed it's recommended in Diamond and Sekhon
(2012). But we were concerned about replicability, because we thought
there was an error in our code when we couldn't replicate the matchit
results using GenMatch directly.
To continue the conversation, whether GenMatch finds or does not find
balance does not depend on whether one adds a propensity score to the
matching matrix. If simple matching on the propensity score leads to
perfect balance, then GenMatch will give the propensity score maximum
weight and will give zero weight to the rest of the components of the
matching matrix. In these cases, if one fails to include the
propensity score, GenMatch will still find the same solution, but it
will take longer than if the propensity score had been included. But
the opposite can also happen: if matching on the propensity score does
not improve balance at all, GenMatch will eventually figure this out
and assign zero weight to it, but it will take longer to find balance
than if the propensity score had not been included in the first place.
In other words, the Rosenbaum and Rubin theorem holds for the true
propensity score. Matching on a badly misspecified propensity score
will not improve balance, and in these cases including it in GenMatch
will slow the balance-finding process.
Rocío
--
Rocío Titiunik
Assistant Professor
Department of Political Science
University of Michigan
http://www.umich.edu/~titiunik/
On Fri, Mar 16, 2012 at 9:43 AM, Gary King <king(a)harvard.edu> wrote:
> Thanks John and Rocío, Good catch, we'll include it in the manual. FYI,
> as
> I recall when we wrote matchit, we added the propensity score to
> genmatch
> because genmatch didn't seem to reduce balance without it. that's odd
> because genmatch already attempts to optimize an arbitrary function of
> all
> the variables in the pscore, and so if its doing what its advertising
> adding
> yet another function of the same variables shouldn't do anything. But
> you
> never know what happens in such a complicated optimization problem, and
> the
> pscore often helped it do a bit better and so we included it.
> Gary
> --
> Gary King - Albert J. Weatherhead III University Professor - Director,
> IQSS
> - Harvard University
> GKing.Harvard.edu - King(a)Harvard.edu - @kinggary - 617-500-7570 - Asst
> 495-9271 - Fax 812-8581
>
>
>
> 2012/3/15 John G. Bullock <john.bullock(a)yale.edu>
>>
>>
>> Hello,
>>
>> I am writing on behalf of myself and Rocío Titiunik.
>>
>> As we see it, matchit() serves as a front end to GenMatch() when
>> method=='genetic'. So it should be possible to get the same results
>> from
>> both the MatchIt and Matching packages. But this proves difficult.
>> Discrepancies seem to occur because matchit() silently adds a column
>> of
>> propensity scores to the X matrix that it passes to GenMatch(). I'm
>> appending some code that we developed to demonstrate this point.
>>
>> The benefits (in most cases) of adding a propensity score are clear,
>> but it seems problematic that this addition occurs without any
>> notification.
>> For example, a user of MatchIt may claim to have matched on a set of
>> predictors, unaware that he has also matched on the propensity score.
>> Or,
>> if a user thinks that he has matched on one propensity score (because
>> he
>> included it in X when calling matchit()), he will be unaware that he
>> has
>> also matched on a second, matchit()-generated propensity score that is
>> based
>> on the first propensity score. In either case, he will be also be
>> unable to
>> replicate his MatchIt results with Matching or vice versa, even though
>> MatchIt is supposed to be a wrapper for Matching. And third-party
>> readers
>> of code that includes a call to matchit() will likely have no idea that
>> a
>> propensity score has been added.
>>
>> We can't see any explicit mention of this propensity-score-adding
>> feature in the MatchIt manual. To promote future replication efforts,
>> can a
>> line be added to the MatchIt manual about the addition of a propensity
>> score
>> to X? Or can a note be added to matchit() output whenever
>> method=='genetic'
>> is used?
>>
>> Thank you,
>> John Bullock
>> ###
>>
>>
>> library(Matching)
>> library(MatchIt)
>> data(lalonde)
>> X <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
>> re74, re75))
>>
>> # GET ESTIMATE FROM MATCHIT()
>> set.seed(5678)
>> m.out1 <- matchit(treat ~ X, data=lalonde, method='genetic',
>> estimand='ATT', ties=TRUE, print.level=1,
>> pop.size=150, wait.generations=1,
>> max.generations=10, hard.generation.limit=TRUE,
>> unif.seed=1945, int.seed=1906)
>>
>> # Create a vector of "parameters at the solution" reported by
>> matchit().
>> # Behind the scenes, these weights have been produced by GenMatch().
>> weights <- c(8.121767e+02, 7.735192e+02, 5.938936e+02, 4.079661e+02,
>> 3.665018e+02, 1.361011e+02,
>> 9.644896e+02, 6.083636e+02, 2.346772e+00)
>>
>> ATT.MatchIt <- with(match.data(m.out1), weighted.mean(re78[treat==1],
>> weights[treat==1])) -
>> with(match.data(m.out1), weighted.mean(re78[treat==0],
>> weights[treat==0]))
>> print(ATT.MatchIt) # 939.2
>>
>>
>> # GET ESTIMATE FROM MATCH()
>> # The Match() estimate is -952.3 -- very different.
>> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X, estimand="ATT",
>> Weight.matrix=diag(weights), ties=TRUE)$est
>>
>>
>> # NOTE DISCREPANCY BETWEEN ncol(X) AND length(weights)
>> # There are only 8 variables in X, so why is matchit() producing
>> weights
>> for nine variables?
>> ncol(X) # 8
>> length(weights) # 9
>>
>>
>> # ADD PROPENSITY SCORE TO X AND RE-ESTIMATE WITH MATCH()
>> glm1 <- glm(treat ~ age+educ+black+hispan+married+nodegree+re74+re75,
>> family=binomial, data=lalonde)
>> X2 <- with(lalonde, cbind(age, educ, black, hispan, married, nodegree,
>> re74, re75))
>> X2 <- cbind(glm1$fitted, X2)
>>
>> Match(Y=lalonde$re78, Tr=lalonde$treat, X=X2, estimand="ATT",
>> Weight.matrix=diag(weights), ties=TRUE)$est # 939.2
>>
>> -
>> ---
>> MatchIt mailing list served by HUIT
>> List Address: matchit(a)lists.gking.harvard.edu
>> Subscribe/Unsubscribe:
>> http://lists.gking.harvard.edu/mailman/listinfo/ei
>> MatchIt Software and Documentation: http://gking.harvard.edu/matchit/
>> Browse/Search List Archive:
>> http://lists.gking.harvard.edu/mailman/private/matchit/
>> Matchit mailing list
>> Matchit(a)lists.gking.harvard.edu
>> https://lists.gking.harvard.edu/mailman/listinfo/matchit
>
>
4421
days inactive
4447
days old
matchit@lists.gking.harvard.edu
7 comments
4 participants
participants (4)
-
Gary King -
John G. Bullock -
Kosuke Imai -
Rocio Titiunik