indexing subdirectories only

[Thunderstone]

I want to set up a search engine separate from my main one that will only
search in one folder and down, without going back up. I looked at the
documentation on your web site, but I must be doing something incorrectly.
The following is what I get when I attempt to index only the "it" directory
and it's subdirectories.

tars% gw -d/www/html-docs/webinator/it -jhttp://www.rollins.edu/it/
Visited 0 pages total


Thank you,

Nancy

[Thunderstone]

You're almost there. The gw command line syntax is:

gw OPTIONS URLS

You've got the options part. Now you need to specify a starting Url.
In most cases like your's the starting Url is the same as the -j Url prefix:

gw -d/whatever -jhttp://www.rollins.edu/it/ http://www.rollins.edu/it/

For some people, it might be slightly different:

gw -d/whatever -jhttp://www.rollins.edu/it/ http://www.rollins.edu/it/somepage.html

[kgrayburn]

For example, say i wish to include two subdirectories into the above example database 'whatever':

1. http://www.rollins.edu/it/
2. http://www.rollins.edu/that/

what is the command line syntax?

gw -d/whatever -jhttp://www.rollins.edu/it/ http://www.rollins.edu/it/ -jhttp://www.rollins.edu/that/ http://www.rollins.edu/that/

Any help would be greatly appreciated,

Karen

[John]

That is the correct syntax, as long as you have some links from /that/ to the /it/ directory. Otherwise you may want to use a url list file, or crawl each directory separately into the same database.

[marsh]

I am trying to index a one subdirectory along with all of its contents which include other subdirectories.

I can index the subdirectiory by using the command

gw -d/whatever -jhttp://www.server.com/subdir/

When I do this the subdirectory "subdir" is the only dierctory indexed and the files located in it get indexed also but the other subdirectories within it do not get indexed. How do I get the subdirectiories within "subdir" to be indexed as well??

Marshall

[mark]

I assume you really meant:
gw -d/whatever -jhttp://www.server.com/subdir/ http://www.server.com/subdir/

The subdirectories have to be linked into the pages it walked. If there are extensions other than .htm .html or .txt you will also need to specify them with the -f option. Also make sure they are not excluded by a /robots.txt file on your webserver.

[marsh]

How do I set up and use a "url list" file to walk multiple subdirectories?

Marshall

[mark]

List all of the subdirectories and also provide a -j option for each.