references or lookup?

resume.robot · Post by **resume.robot** » Sun Nov 18, 2001 11:07 pm

I am trying to speed up the spidering process, and don't need links or references indexed.

Using this string

gw -d/database -noindex -a -R -r -O -fshtml -fasp -fcfm -fjsp -fxml -t7 -z5000 -v4 "&list"

still produces a report like this

137: TotLinks: 1595, Links: 9/ 5, Good: 0, New: 0 Duplicate
137: TotLinks: 1595, Links: 9/ 4, Good: 0, New: 0 Duplicate
137: TotLinks: 1595, Links: 9/ 3, Good: 0, New: 0 Duplicate
137: TotLinks: 1595, Links: 9/ 2, Good: 0, New: 0 Duplicate
137: TotLinks: 1595, Links: 9/ 1, Good: 0, New: 0 Disallowed protocol
137: TotLinks: 1595, Links: 9/ 0, Good: 0, New: 0
http://www.eng.uwaterloo.ca/other/gtarcisi
137: TotLinks: 1595, Links: 9/ 0, Good: 0, New: 0 Retrieving
138: TotLinks: 1605, Links: 10/ 10, Good: 0, New: 0 Chasing 10 refs
138: TotLinks: 1605, Links: 10/ 10, Good: 0, New: 0 Off site

-dnscache is enabled and -unique is turned on

Will -L streamline this?

Thanks

Post by **mark** » Mon Nov 19, 2001 10:33 am

Yes. -L can speed that up by not requiring DNS lookups on the refs.

resume.robot · Post by **resume.robot** » Mon Nov 19, 2001 12:35 pm

Thanks