20 Apr
2004
20 Apr
'04
1:36 p.m.
Dear all,
I recently started using Zelig in order to overcome some problems I had
with logistic modelling of rare events data in geomorphology (used SAS
before).
However I have memory problems with my computer when trying to analyse
the data in Zelig. Is this normal? I have about 16000 observations and
30 variables. This doesn't seem so much to me. However, even when I try
to run a demo, it gives a similar error message. (see copy of message
below)
I already changed the maximum memory allocated to R to 128Mb (=RAM of my
pc) as suggested in the FAQ that accompany R. Is there something else I
can try except buying a new pc?
Any suggestion would be welcome,
Tom
demo(relogit)
---- ~~~~~~~
Type <Return> to start :
data(mid)
z.out1 <- zelig(conflict ~ major + contig + power +
maxdem + mindem + years, data = mid, model = "relogit", tau =
1042/303772)
Error: cannot allocate vector of size 38171 Kb
In addition: Warning message:
Reached total allocation of 128Mb: see help(memory.size)
--
Tom Vanwalleghem
Laboratory for experimental geomorphology
Redingenstraat 16
B-3000 LEUVEN
+32(0)16/326414
20 Apr
20 Apr
2:04 p.m.
Dear Tom,
Two suggestions:
Have you tried increasing the size of your virtual memory cache?
Go to Control Panel > System > Advanced and look at the bottom (for XP at least),
and then set R to however much you decide to cache.
Have you considered using a case-control design on your data? As the King and Zeng
International Organization article points out, you do not lose information by throwing out
0's, and you probably want 2 to 5 0's for every 1. Just set tau = true
population fraction of 1's to correct for selection on the dependent variable.
128 MB of RAM is not much for a relogit model. Here's the reason why: in order to
perform bais correction (what makes this model different from a regular logit), R needs to
take the entire matrix and perform some operations on it. (So R is trying to invert or
matrix multiply a 16000x30 matrix!) It's possible that we may port the bias
correction method to C++ at some point, but short of that, the performance cannot be
improved right now.
Yours, Olivia.
----- Original Message -----
From: Tom Vanwalleghem
To: zelig(a)latte.harvard.edu
Sent: Tuesday, April 20, 2004 1:36 PM
Subject: [zelig] question relogit
Dear all,
I recently started using Zelig in order to overcome some problems I had with logistic
modelling of rare events data in geomorphology (used SAS before).
However I have memory problems with my computer when trying to analyse the data in
Zelig. Is this normal? I have about 16000 observations and 30 variables. This doesn't
seem so much to me. However, even when I try to run a demo, it gives a similar error
message. (see copy of message below)
I already changed the maximum memory allocated to R to 128Mb (=RAM of my pc) as
suggested in the FAQ that accompany R. Is there something else I can try except buying a
new pc?
Any suggestion would be welcome,
Tom
demo(relogit)
---- ~~~~~~~
Type <Return> to start :
data(mid)
z.out1 <- zelig(conflict ~ major + contig + power +
maxdem + mindem + years, data = mid, model = "relogit", tau =
1042/303772)
Error: cannot allocate vector of size 38171 Kb
In addition: Warning message:
Reached total allocation of 128Mb: see help(memory.size)
--
Tom Vanwalleghem
Laboratory for experimental geomorphology
Redingenstraat 16
B-3000 LEUVEN
+32(0)16/326414
4:18 p.m.
Hi,
I think the easiest fix would be to increase the maximum size of memory
used for variable size objects. I would set --max-vsize=500M or something.
See page 91 of An Introduction to R. Or also look at the help file by
typing ?Memory at the R prompt.
Hope this helps!
Kosuke
---------------------------------------------------------
Kosuke Imai Office: Corwin Hall 041
Assistant Professor Phone: 609-258-6601 (Direct)
Department of Politics Fax: 609-258-1110 (Department)
Princeton University Email: kimai(a)Princeton.Edu
Princeton, NJ 08544-1012 http://www.princeton.edu/~kimai
---------------------------------------------------------
On Tue, 20 Apr 2004, Olivia Lau wrote:
Dear Tom,
Two suggestions:
Have you tried increasing the size of your virtual memory cache?
Go to Control Panel > System > Advanced and look at the bottom (for XP at least),
and then set R to however much you decide to cache.
Have you considered using a case-control design on your data? As the King and Zeng
International Organization article points out, you do not lose information by throwing out
0's, and you probably want 2 to 5 0's for every 1. Just set tau = true
population fraction of 1's to correct for selection on the dependent variable.
128 MB of RAM is not much for a relogit model. Here's the reason why: in order to
perform bais correction (what makes this model different from a regular logit), R needs to
take the entire matrix and perform some operations on it. (So R is trying to invert or
matrix multiply a 16000x30 matrix!) It's possible that we may port the bias
correction method to C++ at some point, but short of that, the performance cannot be
improved right now.
Yours, Olivia.
----- Original Message -----
From: Tom Vanwalleghem
To: zelig(a)latte.harvard.edu
Sent: Tuesday, April 20, 2004 1:36 PM
Subject: [zelig] question relogit
Dear all,
I recently started using Zelig in order to overcome some problems I had with logistic
modelling of rare events data in geomorphology (used SAS before).
However I have memory problems with my computer when trying to analyse the data in
Zelig. Is this normal? I have about 16000 observations and 30 variables. This doesn't
seem so much to me. However, even when I try to run a demo, it gives a similar error
message. (see copy of message below)
I already changed the maximum memory allocated to R to 128Mb (=RAM of my pc) as
suggested in the FAQ that accompany R. Is there something else I can try except buying a
new pc?
Any suggestion would be welcome,
Tom
demo(relogit)
---- ~~~~~~~
Type <Return> to start :
-
Zelig Mailing List, served by Harvard-MIT Data Center
Send messages: zelig(a)latte.harvard.edu
[un]subscribe Options: http://lists.hmdc.harvard.edu/?info=zelig
Zelig program information: http://gking.harvard.edu/zelig/
data(mid)
z.out1 <- zelig(conflict ~ major + contig +
power +
maxdem + mindem + years, data = mid, model = "relogit",
tau = 1042/303772)
Error: cannot allocate vector of size 38171 Kb
In addition: Warning message:
Reached total allocation of 128Mb: see help(memory.size)
--
Tom Vanwalleghem
Laboratory for experimental geomorphology
Redingenstraat 16
B-3000 LEUVEN
+32(0)16/326414
21 Apr
21 Apr
12:32 a.m.
Take a look at:
http://gking.harvard.edu/zelig/docs/Frequently_Asked_Quest.html
We've added a link on how to increase memory for R windows.
It's a synthesis of what Kosuke and I have said before, perhaps
in a more intelligible form!
Yours,
Olivia
----- Original Message -----
From: "Kosuke Imai" <kimai(a)Princeton.Edu>
To: "Olivia Lau" <olau(a)fas.harvard.edu>
Cc: "Tom Vanwalleghem" <Tom.Vanwalleghem(a)geo.kuleuven.ac.be>be>;
<zelig(a)latte.harvard.edu>
Sent: Tuesday, April 20, 2004 4:18 PM
Subject: Re: [zelig] question relogit
-
Zelig Mailing List, served by Harvard-MIT Data Center
Send messages: zelig(a)latte.harvard.edu
[un]subscribe Options: http://lists.hmdc.harvard.edu/?info=zelig
Zelig program information: http://gking.harvard.edu/zelig/
Hi,
I think the easiest fix would be to increase the maximum
size of memory
used for variable size objects. I would set
--max-vsize=500M
or something.
See page 91 of An Introduction to R. Or also look at
the help
file by
typing ?Memory at the R prompt.
Hope this helps!
Kosuke
---------------------------------------------------------
Kosuke Imai Office: Corwin Hall 041
Assistant Professor Phone: 609-258-6601 (Direct)
Department of Politics Fax: 609-258-1110 (Department)
Princeton University Email: kimai(a)Princeton.Edu
Princeton, NJ 08544-1012 http://www.princeton.edu/~kimai
---------------------------------------------------------
On Tue, 20 Apr 2004, Olivia Lau wrote:
> Dear Tom,
>
> Two suggestions:
>
> Have you tried increasing the size of your virtual memory
cache?
> Go to Control Panel > System > Advanced and
look at the
bottom (for XP at least), and then set R to however much you
decide to cache.
>
> Have you considered using a case-control design on your
data? As the King and
Zeng International Organization article
points out, you do not lose information by throwing out 0's, and
you probably want 2 to 5 0's for every 1. Just set tau = true
population fraction of 1's to correct for selection on the
dependent variable.
>
> 128 MB of RAM is not much for a relogit model. Here's the
reason why:
in order to perform bais correction (what makes
this model different from a regular logit), R needs to take the
entire matrix and perform some operations on it. (So R is
trying to invert or matrix multiply a 16000x30 matrix!) It's
possible that we may port the bias correction method to C++ at
some point, but short of that, the performance cannot be
improved right now.
>
> Yours, Olivia.
>
> ----- Original Message -----
> From: Tom Vanwalleghem
> To: zelig(a)latte.harvard.edu
> Sent: Tuesday, April 20, 2004 1:36 PM
> Subject: [zelig] question relogit
>
>
> Dear all,
> I recently started using Zelig in order to overcome some
problems I had with
logistic modelling of rare events data in
geomorphology (used SAS before).
> However I have memory problems with my computer
when
trying to analyse the data in Zelig. Is this normal? I have
about 16000 observations and 30 variables. This doesn't seem so
much to me. However, even when I try to run a demo, it gives a
similar error message. (see copy of message below)
> I already changed the maximum memory allocated
to R to
128Mb (=RAM of my pc) as suggested in the FAQ that accompany R.
Is there something else I can try except buying a new pc?
>
> Any suggestion would be welcome,
> Tom
>
>
> demo(relogit)
> ---- ~~~~~~~
>
> Type <Return> to start :
>
> > data(mid)
>
> > z.out1 <- zelig(conflict ~ major + contig + power +
> maxdem + mindem + years, data = mid, model =
"relogit", tau =
1042/303772)
Error:
cannot allocate vector of size 38171 Kb
In addition: Warning message:
Reached total allocation of 128Mb: see help(memory.size)
--
Tom Vanwalleghem
Laboratory for experimental geomorphology
Redingenstraat 16
B-3000 LEUVEN
+32(0)16/326414
7333
days inactive
7334
days old
3 comments
3 participants
participants (3)
-
Kosuke Imai -
Olivia Lau -
Tom Vanwalleghem