Adding new pages to an index

Post Reply
mjacobson
Posts: 204
Joined: Fri Feb 08, 2002 3:35 pm

Adding new pages to an index

Post by mjacobson »

I have been working of a Vortex script that will allow users to submit individual pages to Webinator to be index. When I run the script I can see in the gw.log file:

Retrieving http://somesite.com/newpage1.html
Retrieving http://somesite.com/newpage2.html
Visted 2 pages total
Begin (12236) Indexing new pages
End (12236)

When I do a search I get no results. I have even tried the following tsql command

tsql -d. "select Url from html where Url matches '%somesite.com/newpage%'"

This also returns nothing. The command that I am running in my vortex script is:

gw -d/webroot/webinator/db -g -a http://somesite.com/newpage1.html http://somesite.com/newpage2.html

I am <exec></exec> the above command from my vortex script. Any help would be grateful.
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Adding new pages to an index

Post by mark »

Make sure you're talking to the same database for the walk and the search and tsql. Make sure those pages are retrievable.

gw also logs info to "stdout" which you can see by displaying $ret after the exec.
<exec nobr gw -d... ></exec>
$ret
See what that has to say.
mjacobson
Posts: 204
Joined: Fri Feb 08, 2002 3:35 pm

Adding new pages to an index

Post by mjacobson »

I have double checked and I am talking the same database for the walk, search, & tsql. The pages are retrievable, because during a normal schedule walk, the pages are indexed. I am only having trouble from a command line when I try to insert new pages into my finished walk.

The output from the exec is:
http://www.intelink.gov/producer/produc ... 72735.html Getting http://207.85.92.229/rotots.txt... Got it... Ok.
http://www.intelink.gov/producer/produc ... 72735.html 0/0 1/0 Visited 1 pages total
Indexing new pages
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Adding new pages to an index

Post by mark »

Hmm, looks pretty normal. It seems to think that it has stored the page successfully. I should have also had you check your webserver's error log for the time when gw was executed to see if anything got logged there. There are a few errors that could get logged to stderr.
mjacobson
Posts: 204
Joined: Fri Feb 08, 2002 3:35 pm

Adding new pages to an index

Post by mjacobson »

Looks like my HTML table is larger than our license so gw is not inserting the new pages into the table. Now I get to fight the upgrade battle.
Post Reply