Crawl speed - "Single page"

kiddywood · Post by **kiddywood** » Thu Feb 19, 2009 2:38 pm

How can I increase the speed of my crawl on a large list of single page URLs? It doesn't seem to be crawling very fast at the moment.

What sort of level can I increase the threads and servers numbers to without overcooking the system?

Post by **mark** » Fri Feb 20, 2009 10:57 am

"Singles" are always handled linearly so a slow server can slow down the whole process. Increasing threads may help somewhat to allow it to open that many pages simultaneously.

What you may want to do is, instead of using singles put the urls in as base urls and set max depth to 0 and increase servers.

And of course make sure you have the crawl delay set to 0.