run >1 gw page retrieval/indexing processes simultaneously?

[Thunderstone]

Hi -

I'm indexing a large number of servers. I'd like to index my main
server fully, but I'd like to limit the depth to which I index
the auxiliary servers. I'm planning on putting everything into the
same db (please holler if this clearly isn't the right way to be doing
what I'm doing).

So, nightly, I'll run

gw -d/to/mydb http://my.main.site

gw -d/to/mydb -D5 http://aux.site.one
gw -d/to/mydb -D5 http://aux.site.two

(Or I could lump the auxiliary sites into a config file and call it with
the -m option.)

Here's the question - can I run all three of these gw processes against
the SAME database simultaneously? Would it be better to run them in
-noindex mode and then run a gw -index later?

Thanks in advance,

Susan



Susan Alderman Susan_Alderman AT brown.edu
Box 1885 vox: 401-863-9466
CIS, Brown University fax: 401-863-7329
Providence, RI 02912

[Thunderstone]

You can run multiple copies of gw on the same database as long as
they are all using the same options. And yes it's better to use -noindex
on the pieces-parts, then -index when it's all finished.

gw -d/to/mydb -noindex http://my.main.site
gw -d/to/mydb -noindex -D5 http://aux.site.one &
gw -d/to/mydb -noindex -D5 http://aux.site.two &
wait
gw -d/to/mydb -index

Or:
gw -d/to/mydb -noindex http://my.main.site
gw -d/to/mydb -noindex -D5 http://aux.site.one http://aux.site.two
gw -d/to/mydb -index