No documents match the query ...

ferruccio_manclossi · Thu Dec 06, 2001 10:36 am

HI, novice question again.
I installed webinator starting as root user and selecting nobody as webinator user.
Webinator files and directory are all available for nobody user.

I indexed my site creating a defualt profile but every search I tried return always "No documents match the query." message ...

Any idea?

This is my profilewalk log (last rows)
===========================
Current run: Refresh STOP walk (auto refresh in seconds) Help

Webinator Walk Report for default

Creating database /Search/Webinator/default/db2...Done.
Walk started at 2001-12-05 17:39:29 (by user)
Start fetching at http://jplaza.italy.ibm.com/
Ignore urls containing any of the following:
/cgi-bin/
~
?

started 1 (17032) on http://jplaza.italy.ibm.com/
1900 pages fetched (40,854,176 bytes) from http://jplaza.italy.ibm.com/
Dispatcher stopping by request.

1900 pages (40,854,176 bytes) so far. 186 errors so far. 481 duplicate pages so far. 1900 http://jplaza.italy.ibm.com/Presentatio ... de0029.htm (5,870 bytes) 1899 http://jplaza.italy.ibm.com/Presentatio ... de0028.htm (5,411 bytes) 1898
....

Thanks in advance
Ferruccio

Post by **Kai** » Thu Dec 06, 2001 10:56 am

Check the HTML source of the results page for errors in comments; are there any? You might see something like "Query would require linear search".

You apparently stopped the walk before it completed ("Dispatcher stopping by request"), and search indexes were not built on the walk data. Therefore a search would fail with an error like the above. Let the walk complete and build indexes.

ferruccio_manclossi · Post by **ferruccio_manclossi** » Mon Dec 10, 2001 4:28 am

after 3 days initial walk is still running but appears hanged after 1855 pages reached in the first day...

How can I verify what it's doing?
Search is not yet active...
===== walk log ========
Walk Status
Current Profile: default
Webinator 4.0
Current run: Refresh STOP walk (auto refresh in seconds) Help

Webinator Walk Report for default

Creating database /Search/Webinator/default/db2...Done.
Walk started at 2001-12-06 17:12:51 (by user)
Start fetching at http://jplaza.italy.ibm.com/
Ignore urls containing any of the following:
/cgi-bin/
~
?

started 1 (18508) on http://jplaza.italy.ibm.com/
1855 pages fetched (36,031,703 bytes) from http://jplaza.italy.ibm.com/

1855 pages (36,031,703 bytes) so far. 163 errors so far. 484 duplicate pages so far. 1855 http://jplaza.italy.ibm.com/Presentatio ... de0029.htm (5,870 bytes) 1854
====== ps -eaf | grep -E "tex|monitor" ===========
nobody 1998 1 0 Dec 06 - 7:05 monitor -d /Search/Webinator/default/db2/ -z
nobody 5522 1 0 Dec 05 - 8:41 monitor -d /usr/local/morph3/texis/testdb/ -z
nobody 15312 1 0 Dec 05 - 5:57 monitor r
nobody 15770 1 5 Dec 05 - 215:57 ./texis profile=default /usr/lpp/internet/server_root/pub/webinator/dowalk/dispatch.txt
nobody 18288 1 1 Dec 06 - 79:59 ./texis profile=default /usr/lpp/internet/server_root/pub/webinator/dowalk/dispatch.txt

Post by **mark** » Mon Dec 10, 2001 6:11 am

hmmm...
It should also list the last several Urls fetched. What were they?
What settings have you changed from default?

Check the /Search/Webinator/default/db2 directory to see if anything's being updated. If not, hit STOP. Try again with "Threads" set to 1.

ferruccio_manclossi · Mon Dec 10, 2001 10:02 am

Post by **mark** » Mon Dec 10, 2001 11:53 am

Any single page fetch should not take very long unless the page is huge or the webserver or connection is slow. Besides, the timeout would cut off a long downloading page. I can't try what you're doing since that server is, apparently, not on the net. I assume the webserver is still responsive to a web browser when this occurs? And that new urls stop appearing in the list the the file modification times in the database stop changing. You didn't mention what other settings you may have changed. Shall I assume "none"?

The webserver doesn't have some self protection mechanism that blocks users that fetch too many pages too quickly does it?

ferruccio_manclossi · Post by **ferruccio_manclossi** » Tue Dec 11, 2001 6:46 am

I deleted all my previous setting and I tried a new base scan with the following parameters.
The scan, after reached 10 pages seems hangs and not terminated (I cannot search again...)

My Web has no self-protection mechanism (utility like webzip run fines...).
It's an intranet server (less than 5.000 pages) ...

Any suggestions?
Ferruccio
Basic Walk Settings:
================================
Basic Walk Settings
Current Profile: base
Webinator 4.0
Database ? /Search/Webinator/base/db1
Walk Summary ? New walk started: 2001-12-11 12:13:47 (by user)
Base URL ? http://jplaza.italy.ibm.com/
Enterprise ?Yes - not flagged Domain italy.ibm.com
Robots ?robots.txt: Y Meta: Y
Extensions ? .html .htm .txt
Exclusions ? /cgi-bin/ ~ ?
Crawl Delay ? 0
Parallelism ?Threads:1 Servers: 2
Verbosity ? 2
Rewalk Type ?NEW
Rewalk Schedule ? Frequency NONE Hour NONE
Watch URL ? -empty-
Notify ? -empty-

==== All Walk Setting ====
all defaults except MaxPages=10

===== Walk Status after 1 hours =====
Walk Status
Current Profile: base
Webinator 4.0
Current run: Refresh STOP walk (auto refresh in seconds) Help

Webinator Walk Report for base

Creating database /Search/Webinator/base/db2...Done.
Walk started at 2001-12-11 12:13:47 (by user)
Start fetching at http://jplaza.italy.ibm.com/
Ignore urls containing any of the following:
/cgi-bin/
~
?

started 1 (19040) on http://jplaza.italy.ibm.com/
Maxpages of 10 reached.
10 pages fetched (176,891 bytes) from http://jplaza.italy.ibm.com/

10 pages (176,891 bytes) so far. 0 errors so far. 0 duplicate pages so far. 10 http://jplaza.italy.ibm.com/biblio/ (2,330 bytes) 9 http://jplaza.italy.ibm.com/TivCorner.html (9,318 bytes) 8 http://jplaza.italy.ibm.com/Tips/ (11,938 bytes) 7 http://jplaza.italy.ibm.com/Send.html (7,150 bytes) 6 http://jplaza.italy.ibm.com/Resource.html (7,882 bytes) 5 http://jplaza.italy.ibm.com/Presentation.html (10,235 bytes) 4 http://jplaza.italy.ibm.com/News.html (19,773 bytes) 3 http://jplaza.italy.ibm.com/Link.html (93,771 bytes) 2 http://jplaza.italy.ibm.com/Download.html (7,179 bytes) 1 http://jplaza.italy.ibm.com/ (7,315 bytes)

==== ps -eaf | grep -E "tex|moni" =======
nobody 5544 1 1 12:13:41 - 0:07 ./texis profile=base /usr/lpp/internet/server_root/pub/webinator/dowalk/dispatch.txt
nobody 7660 1 0 12:13:44 - 0:00 monitor -d /Search/Webinator/base/db2/ -z
nobody 12944 1 0 12:07:39 - 0:00 monitor -d /usr/local/morph3/texis/testdb/ -z
nobody 18302 1 0 12:07:39 - 0:00 monit
nobody 20012 1 1 12:11:39 - 0:00 monitor -d /Search/Webinator/base/db1/ -z

======= ls -lt /Search/Webinator/base ======
total 6
drwxrwxrwx 2 nobody nobody 1024 Dec 11 12:25 db1
-rw-rw-rw- 1 nobody nobody 385 Dec 11 12:14 db2.long
-rw-rw-rw- 1 nobody nobody 59 Dec 11 12:13 summary
drwxrwxrwx 2 nobody nobody 1024 Dec 11 12:13 db2

============ls -lt db1 ===========
total 69
-rw-rw-rw- 1 nobody nobody 125664 Dec 11 12:28 SYSLOCKS
-rw-rw-rw- 1 nobody nobody 8 Dec 11 12:28 SYSLOCKS.SEQ
-rw------- 1 nobody nobody 1328 Dec 11 12:25 SYSSTATS.tbl
-rw------- 1 nobody nobody 27176 Dec 11 12:11 SYSCOLUMNS.tbl
-rw------- 1 nobody nobody 5782 Dec 11 12:11 SYSINDEX.tbl
-rw------- 1 nobody nobody 4714 Dec 11 12:11 SYSPERMS.tbl
-rw------- 1 nobody nobody 10556 Dec 11 12:11 SYSTABLES.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 categories.tbl
-rw------- 1 nobody nobody 3336 Dec 11 12:11 counts.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 error.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 options.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 querylog.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 todo.tbl
-rw------- 1 nobody nobody 152 Dec 11 12:11 xcatno.btr
-rw------- 1 nobody nobody 152 Dec 11 12:11 xerrorurl.btr
-rw------- 1 nobody nobody 152 Dec 11 12:11 xoptname.btr
-rw------- 1 nobody nobody 152 Dec 11 12:11 xoptstr.btr
-rw------- 1 nobody nobody 152 Dec 11 12:11 xqueryid.btr
-rw------- 1 nobody nobody 152 Dec 11 12:11 xtodourl.btr
-rw------- 1 nobody nobody 3314 Dec 11 12:11 SYSMETAINDEX.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 SYSTRIG.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 html.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 refs.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:11 vortex.tbl
-rw------- 1 nobody nobody 152 Dec 11 12:11 xhtmlhash.btr
-rw------- 1 nobody nobody 152 Dec 11 12:11 xhtmlid.btr
-rw------- 1 nobody nobody 152 Dec 11 12:11 xhtmlurl.btr
-rw------- 1 nobody nobody 152 Dec 11 12:11 xvid.btr
-rw------- 1 nobody nobody 3438 Dec 11 12:11 SYSUSERS.tbl

============ls -lt db2 ===========
total 208
-rw-rw-rw- 1 nobody nobody 125664 Dec 11 12:20 SYSLOCKS
-rw-rw-rw- 1 nobody nobody 8 Dec 11 12:20 SYSLOCKS.SEQ
-rw------- 1 nobody nobody 3336 Dec 11 12:14 counts.tbl
-rw------- 1 nobody nobody 39778 Dec 11 12:14 html.tbl
-rw------- 1 nobody nobody 64910 Dec 11 12:14 refs.tbl
-rw------- 1 nobody nobody 8350 Dec 11 12:14 xhtmlhash.btr
-rw------- 1 nobody nobody 8350 Dec 11 12:14 xhtmlid.btr
-rw------- 1 nobody nobody 8350 Dec 11 12:14 xhtmlurl.btr
-rw------- 1 nobody nobody 12952 Dec 11 12:13 options.tbl
-rw------- 1 nobody nobody 4398 Dec 11 12:13 todo.tbl
-rw------- 1 nobody nobody 8350 Dec 11 12:13 xoptname.btr
-rw------- 1 nobody nobody 8350 Dec 11 12:13 xoptstr.btr
-rw------- 1 nobody nobody 9384 Dec 11 12:13 xtodourl.btr
-rw------- 1 nobody nobody 1328 Dec 11 12:13 SYSSTATS.tbl
-rw------- 1 nobody nobody 27176 Dec 11 12:13 SYSCOLUMNS.tbl
-rw------- 1 nobody nobody 5782 Dec 11 12:13 SYSINDEX.tbl
-rw------- 1 nobody nobody 4714 Dec 11 12:13 SYSPERMS.tbl
-rw------- 1 nobody nobody 10556 Dec 11 12:13 SYSTABLES.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:13 categories.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:13 querylog.tbl
-rw------- 1 nobody nobody 152 Dec 11 12:13 xcatno.btr
-rw------- 1 nobody nobody 152 Dec 11 12:13 xqueryid.btr
-rw------- 1 nobody nobody 3314 Dec 11 12:13 error.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:13 vortex.tbl
-rw------- 1 nobody nobody 152 Dec 11 12:13 xerrorurl.btr
-rw------- 1 nobody nobody 152 Dec 11 12:13 xvid.btr
-rw------- 1 nobody nobody 3314 Dec 11 12:13 SYSMETAINDEX.tbl
-rw------- 1 nobody nobody 3314 Dec 11 12:13 SYSTRIG.tbl
-rw------- 1 nobody nobody 3438 Dec 11 12:13 SYSUSERS.tbl

Post by **mark** » Tue Dec 11, 2001 12:06 pm

It would appear that the walk is completed, but the dispatcher is not detecting it properly and making the database live for searching. We're not able to replicate the problem here, but we have an idea what may be causing the problem. We'll try to have an update to the dowalk script that you can download later today.

Post by **mark** » Wed Dec 12, 2001 1:30 pm

Ok, a little later than expected, but there's a new dowalk script you can download from http://www.thunderstone.com/texis/site/ ... ample.html

It does more checking on child processes so it's not likely to be fooled by the OS reusing process id's rapidly. Hopefully this will solve the problem you're encountering. It may also solve the problems some Windows users have reported with walks getting stuck.

ferruccio_manclossi · Post by **ferruccio_manclossi** » Thu Dec 13, 2001 5:18 am

Thanks guys...
1-downloaded new dowalk script
2-substituted old dowalk with new script to /usr/local/morph3/webinator directory
3-New script owned by my webinator user (nobody)
4-killed all active process (texis, monit, monitor, ...)
5-deleted old profiles and databases
6-Created a new Prova profile with maxpage=10
7-Launched Prova profile
8-Seems hanged after reaching 10 pages
9-Found a <defunct> process child of dispatch action ... ??? Very strange!!!
======= ps -eaf | grep -E "tex|defunct|moni" =====
root 2034 15184 4 11:11:48 pts/0 0:00 grep -E tex|moni|defun
nobody 7532 1 1 10:46:44 - 0:26 ./texis profile=prova /usr/lpp/internet/server_root/pub/webinator/dowalk/dispatch.txt
nobody 12954 7532 120 0:00 <defunct>
nobody 18306 1 0 10:45:13 - 0:00 monit
nobody 19048 1 0 10:46:47 - 0:01 monitor -d /Search/Webinator/prova/db2/ -z
nobody 19782 1 0 10:45:13 - 0:01 monitor -d /usr/local/morph3/texis/testdb/ -z

====== Walk Status =========
Walk Status
Current Profile: prova
Webinator 4.0
Current run: Refresh STOP walk (auto refresh in seconds) Help

Webinator Walk Report for prova

Creating database /Search/Webinator/prova/db2...Done.
Walk started at 2001-12-13 10:46:50 (by user)
Start fetching at http://jplaza.italy.ibm.com/
Ignore urls containing any of the following:
/cgi-bin/
~
?

started 1 (12954) on http://jplaza.italy.ibm.com/
Maxpages of 10 reached.
10 pages fetched (176,891 bytes) from http://jplaza.italy.ibm.com/

10 pages (176,891 bytes) so far. 0 errors so far. 0 duplicate pages so far. 10 http://jplaza.italy.ibm.com/biblio/ (2,330 bytes) 9 http://jplaza.italy.ibm.com/TivCorner.html (9,318 bytes) 8 http://jplaza.italy.ibm.com/Tips/ (11,938 bytes) 7 http://jplaza.italy.ibm.com/Send.html (7,150 bytes) 6 http://jplaza.italy.ibm.com/Resource.html (7,882 bytes) 5 http://jplaza.italy.ibm.com/Presentation.html (10,235 bytes) 4 http://jplaza.italy.ibm.com/News.html (19,773 bytes) 3 http://jplaza.italy.ibm.com/Link.html (93,771 bytes) 2 http://jplaza.italy.ibm.com/Download.html (7,179 bytes) 1 http://jplaza.italy.ibm.com/ (7,315 bytes)

Have I missed something important?
Any suggestions?
Ferruccio