Gathering URL's 1.3 v 2.12

[Thunderstone]

I find the database build speed under 2.12 commercial is much slower than
it was under 1.3 commercial (Win NT 4.0, IIS 3.0, 96 megs RAM, Pent Pro)

I did a timing test. 3,000 records all at my site, all listed as URL's in
a file TC2.LIS.

Under 1.3
gw -dtmp -a -v0 -w0 "&tc2.lis"
time to gather files into todo - 50 secs
time to retrieve all the files - 3 min 5 sec
time to index - 19 sec
(very impressive)

Under 2.2
gw -dtc2 -a -v0 -w0 -L "&tc2.lis"
time to gather files into todo - 13 min 4 secs
time to retrieve all the files - 2 min 51 sec
time to index - 20 sec

Also very impressive except for the time between issuing the gw command and
the time the first page is retrieved. Is there anything I can do to speed
up building the todo list?


--------------------------------------------------------------
Larry Rudner rudner@cua.edu
ERIC Clearinghouse on 800 464-3742 (800 Go4-ERIC)
Assessment and Evaluation 202 319-5120
Catholic University of America FAX: 202 319-6692
Washington, DC 20064 http://ericae.net

[Thunderstone]

gw remembers every url you specified to walk across sessions (from the
command line or list file). You can speed up the process of saving a huge
number of urls by creating an extra index on the options table:
texis -s -d tc2 "create index xoptstr on options(String)"

The next release will do this by default.