Updating a database

Post by **mark** » Tue Jan 16, 2001 2:18 pm

See http://www.thunderstone.com/site/gw25man/node60.html

gw -d/httpd/webinator/olddb -jhttp://www.mysite.com/catalog/ http://www.mysite.com/catalog/

slugwater · Post by **slugwater** » Tue Jan 16, 2001 2:19 pm

I completed a walk for one directory using:

gw -d/httpd/webinator/olddb -wipe

gw d/httpd/webinator/olddb -jhttp://www.mysite.com/subdir http://www.mysite.com/subdir

gw -d/httpd/webinator/olddb -index

I checked the gw.log and it retrived exactly what I wanted.

When I try a test search I get the result "Database $ does not appear to exist." Any Suggestions??

The error added to my HTML is as follows:

Post by **mark** » Tue Jan 16, 2001 3:26 pm

Apparently the <db> statement(s) in your search script don't agree with the name/location of the database you are working on.

With the default setup, there's a directory called webinator under your web documents directory. Within the webinator directory are the databases. The default database is called "db". Setting the db variable when calling the default search script will let you choose a different database within the webinator directory.

It would appear that you are building your database, olddb, in the standard place, but your search script has been customized to look somewhere else entirely for it's database. So you need to create your databases in the customized location or un-customize the search script. It also appears that the search script you are using is looking for a database called "cat2", not "olddb".

slugwater · Post by **slugwater** » Tue Jan 16, 2001 4:06 pm

The database I am building is called 'cat2'. I was using 'olddb' simply as an example.

The database I just created is in the same directory as the default 'db' database. I checked the script and that seems to be pointing to the correct path.


<DB = /home/httpd/htdocs/search/db>

I have a database called 'db' and I am using it. My 'cat2' database is new and is in:
/home/httpd/htdocs/search/cat2.

I set the database name as follows:


<A NAME=setdb>
<$defaultdb = /home/httpd/htdocs/search/>
<IF $db eq "">
<$db = "db"> 
</IF>
<sum "%s" $defaultdb $db>
<sandr = $ret>
<DB = $dbdir>
<IF $db eq "db">
<$db = >
</IF>
</A>

you can try the search at http://www.silverplatter.com/cgi-bin/texis/search/usa5.

The first radio button is set to search 'db'
the second radio button is set to search 'cat2'
the third radio button is set to search 'tech_support'

Does this look correct?
Thanks for the responses mark!! I really appreciate it!

Post by **mark** » Tue Jan 16, 2001 4:49 pm

If /home/httpd/htdocs/search/cat2 is the correct database then you have a permissions problem. gw and texis should ALWAYS run as the same userid. The installation attempts to ensure this by making both programs setuid to the correct user.

chown THECORRECTUSER htdocs/webinator/bin/gw cgi-bin/texis
chmod u+s htdocs/webinator/bin/gw cgi-bin/texis

Make sure your existing database is owned by this user if it isn't already

chown THECORRECTUSER htdocs/webinator/cat2 htdocs/webinator/cat2/*

If you can't make the programs setuid for some reason, you will have to make the database accessible by everyone

chmod a+rw htdocs/webinator/cat2*
chmod a+rwx htdocs/webinator/cat2

slugwater · Post by **slugwater** » Wed Jan 17, 2001 10:38 am

That worked, It was a permission issue with 'cat2'.

After I indexed the database, I checked the gw.log for it and it retrived exactly what I wanted. But, Whenever I search it, I only get one result.

The one result I get is an A-Z search.
http://www.MySite.com/catalog.htm
This is the only page on this level in my database.

All the other pages are in the next directory down.
http://www.MySite.com/catalog/*.htm

Any suggestions??

Post by **mark** » Wed Jan 17, 2001 11:37 am

Make sure you're not being fooled by old entries in gw.log. gw never clears gw.log. Make sure you're looking at the end of the log or clear it before walking. Also, gw prints how many pages were visited to the screen when it's finished. That should give you a feel right away if it worked as expected. To check what's actually in the database do

gw -dYOURDATABASE -st "select Url from html"

Check what was fetched for a given url with

gw -dYOURDATABASE -s "select * from html where Url='www.silverplatter.com/catalog/amof.htm'"

slugwater · Post by **slugwater** » Wed Jan 17, 2001 11:59 am

I checked the gw.log and it is the current entry. It said it visited 229 pages.

I tried gw -dYOURDATABASE -st "select Url from html" and it returned

Url
www.silverplatter.com/catalog
www.silverplatter.com/catalog.htm
www.silverplatter.com/catalog.htm?TEMPLATE=blank

Only those three??

When I try:
gw -dYOURDATABASE -s "select * from html where Url='www.silverplatter.com/catalog/amof.htm'"
or
gw -dYOURDATABASE -s "select * from html where Url='www.silverplatter.com/catalog'"
or
gw -dYOURDATABASE -s "select * from html where Url='www.silverplatter.com/catalog.htm"

Nothing was returned...

It seems like only three pages were indexed. Did I not set the depth correctly?? what went wrong??

-Ian

Post by **mark** » Wed Jan 17, 2001 12:10 pm

If gw -dYOURDATABASE -st "select Url from html" gives
www.silverplatter.com/catalog
and
gw -dYOURDATABASE -s "select * from html where Url='www.silverplatter.com/catalog'"
gives nothing at all, you must have typed something wrong.

Wipe the database, delete gw.log, try the crawl with gw again.
Observe the number of pages that gw fetches and then reports to the screen when it finishes.

If it still doesn't work please supply your actual full gw command line and options file contents, if any. And as long as you're providing info it wouldn't hurt to include your gw release (output of gw -version) and operating system.

slugwater · Post by **slugwater** » Wed Jan 17, 2001 12:51 pm

Thanks Mark!!

I wiped the database and then walked it again using -D2. It indexed exactly what I wanted and the search works really well. Job Complete!

I owe you one!!

-Ian