Is it unusual for gw to take 7 hours to index a site? I am not currently using the -L option (my pages have lots of different hosts), but was wondering if there is anything else besides site and network speed that can affect page fetch and indexing time.
gw taking a long time to fetch pages and index
-
Thunderstone
- Site Admin
- Posts: 2504
- Joined: Wed Jun 07, 2000 6:20 pm
gw taking a long time to fetch pages and index
Is it unusual for gw to take 7 hours to index a site? I am not currently using the -L option (my pages have lots of different hosts), but was wondering if there is anything else besides site and network speed that can affect page fetch and indexing time.
-
Thunderstone
- Site Admin
- Posts: 2504
- Joined: Wed Jun 07, 2000 6:20 pm
gw taking a long time to fetch pages and index
It depends on the size of the site and network/dns/host speed.
That's rather long unless it's a particularly huge or slow site.
The -w option will slow things down if it's anything but -w0.
http://www.thunderstone.com/gw25man/node71.html
The only other cause would be an overloaded local system where gw is running.
You could enable -dnscache to speed up name lookups.
http://www.thunderstone.com/gw25man/node42.html