speeding up a crawl

[roshan_shankar]

Hi,
I have a crawl that takes about 3 hours to crawl 800 pages. Our site has some 5000 pages and we would like to have it crawled in a 3 hour window. Note - the crawl is just cache the site - so, all that i need it to do is it to visit every page -like a crawler

here is what i give to walk the site
gw -d"c:\my1.db" -v0 -l"c:\my1.log" -M/"Agent3.0" -noindex -unique -b -R -t180 -q10000 -fjhtml -jhttp://pilot.mysite.com "&url1.lst"

Is there a way to speed up the process - more threads, etc...?

TIA
Roshan

[mark]

800 pages should crawl in a few minutes at most unless you have horribly slow webserver or DNS or network. Try using -L to reduce DNS lookups.