Now and then, a manually started walk gets abandoned. Are we required to keep the walk status screen up (and refreshing)? What could cause a walk to "self abandon?"
Thanks!
Harold
Thanks!
Harold
Walks Abandoned
Walks Abandoned
No, the walk status screen is not required to keep the walk going. A walk won't generally self abandon, however it is possible that the process will have been stopped by the OS or a user. You might want to look at the end of log files generated, as that will indicate why the walk stopped.
John Turnbull
Thunderstone Software
Thunderstone Software
-
harold@dovesystems
- Posts: 35
- Joined: Tue Oct 03, 2000 7:45 pm
Walks Abandoned
I'm still learing my way around the file structure. Which logs should I look at (and where)? Nothing jumps out at me in /usr/local/morph3/texis/monitor.log .
Thanks!
Harold
Thanks!
Harold
Walks Abandoned
In the database that's being crawled you'll find a db1.long and db2.long, which are the crawl logs.
John Turnbull
Thunderstone Software
Thunderstone Software
-
harold@dovesystems
- Posts: 35
- Joined: Tue Oct 03, 2000 7:45 pm
Walks Abandoned
Here's the db1 log (started yesterday). Looks like it just kinda stopped. My walk status screen showed it as abandoned. Ideas?
Thanks!
Harold
<pre>
Webinator Walk Report for FccCcb
Creating database /usr/local/morph3/texis/FccCcb/db1...Done.
Walk started at 2002-01-11 13:20:41 (by user)
Start fetching at http://www.fcc.gov/ccb/
Start fetching at http://www.fcc.gov/Bureaus/Common_Carrier/
Ignore urls containing any of the following:
/cgi-bin/
~
?
started 1 (32759) on http://www.fcc.gov/ccb/
15805 pages fetched (1,016,660,238 bytes) from http://www.fcc.gov/ccb/
161 errors
363 duplicate pages
Creating search index on fetched pages...
Thanks!
Harold
<pre>
Webinator Walk Report for FccCcb
Creating database /usr/local/morph3/texis/FccCcb/db1...Done.
Walk started at 2002-01-11 13:20:41 (by user)
Start fetching at http://www.fcc.gov/ccb/
Start fetching at http://www.fcc.gov/Bureaus/Common_Carrier/
Ignore urls containing any of the following:
/cgi-bin/
~
?
started 1 (32759) on http://www.fcc.gov/ccb/
15805 pages fetched (1,016,660,238 bytes) from http://www.fcc.gov/ccb/
161 errors
363 duplicate pages
Creating search index on fetched pages...
Walks Abandoned
Strange. the .long logs are what's displayed in the walk status. Are there any messages in vortex.log (same directory as monitor.log)?
It there a texis process with dowalk still running?
In "Walk Settings" does it say the database is db1 or db2? db1 (in the case above) would indicate it went to completion.
You might try downloading the latest version of the dowalk and webinatoradmin scripts from the Webinator example scripts page http://www.thunderstone.com/texis/site/ ... ample.html
It there a texis process with dowalk still running?
In "Walk Settings" does it say the database is db1 or db2? db1 (in the case above) would indicate it went to completion.
You might try downloading the latest version of the dowalk and webinatoradmin scripts from the Webinator example scripts page http://www.thunderstone.com/texis/site/ ... ample.html
-
harold@dovesystems
- Posts: 35
- Joined: Tue Oct 03, 2000 7:45 pm
Walks Abandoned
I think I've figured it out. I notice that logs are completely filling my /var partition after a walk is abandoned. Nothing in there seems overly large, it's just that the default partition size seems a bit small. I'll try to fix that.
Thanks!
Harold
Thanks!
Harold
-
harold@dovesystems
- Posts: 35
- Joined: Tue Oct 03, 2000 7:45 pm
Walks Abandoned
I FINALLY finished repartitioning the drives and reinstalling everything. I'm still having problems with large walks being abandoned. I installed the latest do walk and webinatoradmin scripts (as of 2/2/02).
Here's this morning's vortex.log:
000 Feb 4 07:13:55 /webinator/search: (16193) Terminated (signal 15); will exit ASAP
000 Feb 4 07:13:55 /webinator/search: (16193) exiting due to previous signal 15
000 Feb 4 07:14:15 /webinator/search: (16198) Terminated (signal 15); will exit ASAP
000 Feb 4 07:14:15 /webinator/search: (16198) exiting due to previous signal 15
000 Feb 4 07:20:07 /webinator/search: (16252) Terminated (signal 15); will exit ASAP
000 Feb 4 07:20:07 /webinator/search: (16252) exiting due to previous signal 15
000 Feb 4 11:38:29 /home/harold/public_html/webinator/dowalk:829: Vortex (17259) ABEND: signal 11
000 Feb 4 11:38:29 /home/harold/public_html/webinator/dowalk:829: Vortex (17259) ABEND: signal 11
And, here's today's monitor.log
200 Feb 4 04:44:31 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 13136)
200 Feb 4 04:47:31 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 13136)
200 Feb 4 06:46:52 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 16012)
200 Feb 4 06:47:05 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ starting (pid 16030)
200 Feb 4 06:50:05 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ exiting (pid 16030)
200 Feb 4 06:50:52 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 16012)
200 Feb 4 07:13:39 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 16189)
200 Feb 4 07:18:39 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 16189)
200 Feb 4 07:19:38 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 16246)
200 Feb 4 07:22:38 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 16246)
200 Feb 4 07:28:20 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 16408)
200 Feb 4 07:31:20 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 16408)
200 Feb 4 08:02:02 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 17217)
200 Feb 4 08:02:28 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ starting (pid 17231)
200 Feb 4 08:03:31 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ received signal 15 (SIGTERM); will exit (pid $
200 Feb 4 08:03:31 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ exiting (pid 17231)
200 Feb 4 08:03:32 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ starting (pid 17264)
200 Feb 4 08:06:02 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 17217)
200 Feb 4 08:10:14 Database Monitor on /usr/local/morph3/texis/FccWtb/db2/ starting (pid 17588)
200 Feb 4 08:14:14 Database Monitor on /usr/local/morph3/texis/FccWtb/db2/ exiting (pid 17588)
200 Feb 4 08:49:45 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 18760)
200 Feb 4 11:20:45 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 18760)
200 Feb 4 11:33:16 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 25693)
200 Feb 4 11:33:53 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 25703)
200 Feb 4 11:36:17 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 25693)
200 Feb 4 11:40:32 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ exiting (pid 17264)
200 Feb 4 11:40:53 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 25703)
200 Feb 4 12:23:11 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 25809)
200 Feb 4 12:26:11 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 25809)
200 Feb 4 12:46:38 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 25827)
200 Feb 4 12:51:38 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 25827)
200 Feb 4 13:25:52 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 25873)
200 Feb 4 13:25:59 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ starting (pid 25879)
200 Feb 4 13:28:52 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 25873)
200 Feb 4 13:28:59 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ exiting (pid 25879)
Here's db1.long
<pre>
Webinator Walk Report for FccMmb
Creating database /usr/local/morph3/texis/FccMmb/db1...Done.
Walk started at 2002-02-04 08:03:31 (by user)
Start fetching at http://www.fcc.gov/mmb/
Start fetching at http://www.fcc.gov/Bureaus/Mass_Media/
Start fetching at http://www.fcc.gov/Document_Indexes/Mass_Media/
Ignore urls containing any of the following:
/cgi-bin/
~
?
started 1 (17262) on http://www.fcc.gov/mmb/
13912 pages fetched (515,216,529 bytes) from http://www.fcc.gov/mmb/
243 errors
610 duplicate pages
Creating search index on fetched pages...
So, any ideas? Note that this was my downloaded install to Red Hat 7.1 which had various install script problems. I have not yet tried installing from the CD you sent me. Should I just try a new install, or should we continue debugging this? The problems at this point are: Large walks seem to self abandon and walks don't seem to be starting by themselves as scheduled.
THANKS!
Harold
Here's this morning's vortex.log:
000 Feb 4 07:13:55 /webinator/search: (16193) Terminated (signal 15); will exit ASAP
000 Feb 4 07:13:55 /webinator/search: (16193) exiting due to previous signal 15
000 Feb 4 07:14:15 /webinator/search: (16198) Terminated (signal 15); will exit ASAP
000 Feb 4 07:14:15 /webinator/search: (16198) exiting due to previous signal 15
000 Feb 4 07:20:07 /webinator/search: (16252) Terminated (signal 15); will exit ASAP
000 Feb 4 07:20:07 /webinator/search: (16252) exiting due to previous signal 15
000 Feb 4 11:38:29 /home/harold/public_html/webinator/dowalk:829: Vortex (17259) ABEND: signal 11
000 Feb 4 11:38:29 /home/harold/public_html/webinator/dowalk:829: Vortex (17259) ABEND: signal 11
And, here's today's monitor.log
200 Feb 4 04:44:31 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 13136)
200 Feb 4 04:47:31 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 13136)
200 Feb 4 06:46:52 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 16012)
200 Feb 4 06:47:05 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ starting (pid 16030)
200 Feb 4 06:50:05 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ exiting (pid 16030)
200 Feb 4 06:50:52 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 16012)
200 Feb 4 07:13:39 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 16189)
200 Feb 4 07:18:39 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 16189)
200 Feb 4 07:19:38 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 16246)
200 Feb 4 07:22:38 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 16246)
200 Feb 4 07:28:20 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 16408)
200 Feb 4 07:31:20 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 16408)
200 Feb 4 08:02:02 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 17217)
200 Feb 4 08:02:28 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ starting (pid 17231)
200 Feb 4 08:03:31 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ received signal 15 (SIGTERM); will exit (pid $
200 Feb 4 08:03:31 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ exiting (pid 17231)
200 Feb 4 08:03:32 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ starting (pid 17264)
200 Feb 4 08:06:02 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 17217)
200 Feb 4 08:10:14 Database Monitor on /usr/local/morph3/texis/FccWtb/db2/ starting (pid 17588)
200 Feb 4 08:14:14 Database Monitor on /usr/local/morph3/texis/FccWtb/db2/ exiting (pid 17588)
200 Feb 4 08:49:45 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 18760)
200 Feb 4 11:20:45 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 18760)
200 Feb 4 11:33:16 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 25693)
200 Feb 4 11:33:53 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 25703)
200 Feb 4 11:36:17 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 25693)
200 Feb 4 11:40:32 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ exiting (pid 17264)
200 Feb 4 11:40:53 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 25703)
200 Feb 4 12:23:11 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 25809)
200 Feb 4 12:26:11 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 25809)
200 Feb 4 12:46:38 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ starting (pid 25827)
200 Feb 4 12:51:38 Database Monitor on /usr/local/morph3/texis/FccRules/db2/ exiting (pid 25827)
200 Feb 4 13:25:52 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ starting (pid 25873)
200 Feb 4 13:25:59 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ starting (pid 25879)
200 Feb 4 13:28:52 Database Monitor on /usr/local/morph3/texis/FccMmb/db2/ exiting (pid 25873)
200 Feb 4 13:28:59 Database Monitor on /usr/local/morph3/texis/FccMmb/db1/ exiting (pid 25879)
Here's db1.long
<pre>
Webinator Walk Report for FccMmb
Creating database /usr/local/morph3/texis/FccMmb/db1...Done.
Walk started at 2002-02-04 08:03:31 (by user)
Start fetching at http://www.fcc.gov/mmb/
Start fetching at http://www.fcc.gov/Bureaus/Mass_Media/
Start fetching at http://www.fcc.gov/Document_Indexes/Mass_Media/
Ignore urls containing any of the following:
/cgi-bin/
~
?
started 1 (17262) on http://www.fcc.gov/mmb/
13912 pages fetched (515,216,529 bytes) from http://www.fcc.gov/mmb/
243 errors
610 duplicate pages
Creating search index on fetched pages...
So, any ideas? Note that this was my downloaded install to Red Hat 7.1 which had various install script problems. I have not yet tried installing from the CD you sent me. Should I just try a new install, or should we continue debugging this? The problems at this point are: Large walks seem to self abandon and walks don't seem to be starting by themselves as scheduled.
THANKS!
Harold
Walks Abandoned
The problem appears to occur at line 829 of the dowalk script. What do you have on that line (just to make sure that there's no offset with our copy of the script).
John Turnbull
Thunderstone Software
Thunderstone Software
-
harold@dovesystems
- Posts: 35
- Joined: Tue Oct 03, 2000 7:45 pm
Walks Abandoned
Line 829 of /home/harold/public_html/webinator/dowalk (which is the one in vortex.log) reads
<sql "create metamorph inverted index xhtmlbodv on html(Title\Description\Keywords\Meta\Body,Visited)"></sql>
I also have dowalk in:
/usr/local/morph3/webinator/dowalk
/usr/local/morph3/webinator/dowalk.vtx
/home/harold/public_html/webinator/dowalk
/home/harold/public_html/webinator/dowalk.vtx
/home/harold/public_html/webinator/backup/dowalk
/home/harold/webinator/webinator/dowalk
/home/harold/webinator/webinator/dowalk.vtx
Some of these might be from previous versions of Webinator. Can I delete some of them to avoid confusion?
Harold
<sql "create metamorph inverted index xhtmlbodv on html(Title\Description\Keywords\Meta\Body,Visited)"></sql>
I also have dowalk in:
/usr/local/morph3/webinator/dowalk
/usr/local/morph3/webinator/dowalk.vtx
/home/harold/public_html/webinator/dowalk
/home/harold/public_html/webinator/dowalk.vtx
/home/harold/public_html/webinator/backup/dowalk
/home/harold/webinator/webinator/dowalk
/home/harold/webinator/webinator/dowalk.vtx
Some of these might be from previous versions of Webinator. Can I delete some of them to avoid confusion?
Harold