Increasing servers - Single Pages

Post Reply
kiddywood
Posts: 41
Joined: Tue Feb 10, 2009 2:49 pm

Increasing servers - Single Pages

Post by kiddywood »

Hi,

I'm crawling a large list of single URL's using either the single page option or base URL with maximum depth at 0.

To get maximum speed, what is a safe number I could increase the 'server' option to?
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Increasing servers - Single Pages

Post by mark »

Singles will be processed serially regardless. For multiple base URLs the best number will vary greatly depending on your system speed, memory, disk, network bandwidth, and somewhat depending on the file types. You probably wouldn't want to go over 20 servers though.
kiddywood
Posts: 41
Joined: Tue Feb 10, 2009 2:49 pm

Increasing servers - Single Pages

Post by kiddywood »

Thanks Mark.

Also, regarding speed, when I attempt to crawl a large number of single pages (30,000), I find that the walk slows and slows to the point where it's only indexing a few pages an hour towards the end.

Can you advise please?
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Increasing servers - Single Pages

Post by mark »

Do you have current scripts? Version 6.1.5 is on the download page.
User avatar
John
Site Admin
Posts: 2597
Joined: Mon Apr 24, 2000 3:18 pm
Location: Cleveland, OH
Contact:

Increasing servers - Single Pages

Post by John »

If you have a lot of Base URLs you may also want to turn off the "Follow Cross Site Links" setting.
John Turnbull
Thunderstone Software
kiddywood
Posts: 41
Joined: Tue Feb 10, 2009 2:49 pm

Increasing servers - Single Pages

Post by kiddywood »

I've followed both of your instructions as per above - the walk definitely speeded up, however, it resulted in a failed run after a couple of hours:

Walker stopping by request. (singles)
Dispatcher exiting.
Cancelled by user: 2012-12-17 21:35:07

Nothing was cancelled by me so why did it stop with this message?
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Increasing servers - Single Pages

Post by mark »

If you have old software stopping any walk will stop them all.
kiddywood
Posts: 41
Joined: Tue Feb 10, 2009 2:49 pm

Increasing servers - Single Pages

Post by kiddywood »

Mark, this is the only walk I'm running.

It's stopping half way through as above.
Post Reply