scheduling multiple site walks

Post Reply
john208
Posts: 86
Joined: Sat Aug 14, 2004 12:56 pm

scheduling multiple site walks

Post by john208 »

I have several profiles, each walking a different site. Many sites are hosted on the same web server. I want to schedule each site to walk at midnight but I only want to walk one site at a time. If I schedule them all to walk at midnight, don't they all try to walk at the same time, and maybe overload my server? How can I get them to walk one at a time.

Thanks, John
User avatar
John
Site Admin
Posts: 2622
Joined: Mon Apr 24, 2000 3:18 pm
Location: Cleveland, OH
Contact:

scheduling multiple site walks

Post by John »

You could either schedule them for different times, or maybe add a page delay so even if they are running at the same time they will not add as much of a load.

Another approach would be to create a shell script or batch file to fire off the crawls in sequence, with each one waiting for the previous one to finish.
John Turnbull
Thunderstone Software
Post Reply