Can I timeout a walk?

lightsource · Post by **lightsource** » Mon Oct 27, 2003 5:51 pm

I am evaluating the downloadable version of Webinator (Webinator 4.4.3-Windows-wo/plugin) and have my basic walk settings configured to wait 1 second between requests, allow 4 threads, and walk the domain on our public website.

What I'm seeing is that basically, indexing completes in about 40 minutes, then webinator proceeds to spawn threads for about 18 hours (retrieving nothing for those threads) before deciding that it's done.

Is there a way to configure the dowalk script to just stop if it's been running for, say, two hours?

Thanks -
Mark

Post by **John** » Mon Oct 27, 2003 7:20 pm

You could add code to do that to the dowalk script. Does it log anything for those additional threads? It does not generally behave that way, and with some further information we can determine the cause.

lightsource · Post by **lightsource** » Mon Oct 27, 2003 7:36 pm

I don't see anything in the logs that indicates that it's doing _anything_ in particular, and the output to the screen (while the walk is active) basically just tells me repetitively that a new thread is created (for the same URL, but it looks like a new thread ID?? in parentheses each time) and that 0 pages were fetched.

The entire site is dynamically generated, but there's nothing like an active clock or date object which displays on the page that the monitor would have freaked out over, and there's no monitor URL configured for this particular walk.

I'm trying again with more filtering enabled (I'm stripping out all forms which the spider couldn't complete) and have selected "refresh" as the walk type - I'll let this one go overnight and see what it looks like in the morning. Perhaps the spider was getting hung up on a form or something (although there are no forms in the directory that it was looping on).

Thanks - I'll update when I see how it has behaved in the morning - this product rocks!

Mark

Post by **Kai** » Tue Oct 28, 2003 10:16 am

If it was spawning a thread repeatedly for the same URL, there may be an issue with that URL. Are there any messages in texis/vortex.log in the install dir?

Post by **mark** » Tue Oct 28, 2003 10:52 am

While experimenting with settings it's best to always do "new" walks rather than "refresh".

KMandalia · Post by **KMandalia** » Fri Jul 30, 2004 12:02 pm

I am having the same issue as above (however, it is not stuck on same webpage but same folder). However, I can't stop the dowalk.

The following is latest from monitor and vortex logs.

Monitor

200 2004-07-30 11:53:00 (9812) Database Monitor on e:\Webinator\Data\site\db2 exiting

Vortex

115 2004-07-30 11:48:00 e:\Program Files\Thunderstone Software\Webinator\texis\scripts\Webinator\dowalk:69: Field NextCheck non-existent
000 2004-07-30 11:48:00 e:\Program Files\Thunderstone Software\Webinator\texis\scripts\Webinator\dowalk:69: SQLExecute() failed with -1 in the function execntexis

Webinator has already walked thousands of sites and I don't want to loose them, however, i want to stop the walker and also want to assure this problem doesn't happen in future.

Bottom line is I want to stop this walk and change settings so it doesn't happen in future. I have paid webinator 5.0.5 (both scripts are up to date). Any help?

Post by **John** » Fri Jul 30, 2004 12:05 pm

You can use the Pause Walk and Make Live button, which will stop the walk, and make what has been walked so far searchable. If you change the settings the changes will be used from the next refresh, although any URLs already in the list to do will still be processed.

KMandalia · Post by **KMandalia** » Fri Jul 30, 2004 12:09 pm

I did that, but that doesn't stop walk. I want to stop the walk completely. The reason is this:

I want to put in categories. I paused the walk and make database live and now I am going into all walk settings, putting categories and url patterns in place and hitting update button. However, I am not seeing the category box on the search form. May be becuase the walk is still going on.

Post by **John** » Fri Jul 30, 2004 12:15 pm

Both of those steps may take some time. You should wait till the walk has gone live before updating the categories, and if you have a lot of pages then putting the categories in can take some time to update all the pages. Once that is done the category search box should appear.

Post by **mark** » Fri Jul 30, 2004 12:18 pm

Check the walk status page for the status of what's happening. You may have to scroll down to see it all.