Conditional crawling

valery · Post by **valery** » Wed Mar 21, 2001 7:40 pm

Hi,

I have a couple of questions about commercial Webinator I use:

1. Would it be possible to restrict crawling by keywords, e.g. index only pages where specified keyword appears?

2. During the search, I want only to search a subset of domains indexed. I know that I can do this with something like "Url matches 'www.mysite.com%'. Is
there more scalable method (a subset in my case will sometimes be composed of ~10,000 domains)? Is it possible to add a field 'subset_index' into html table?

Thanks,
Valery.

Post by **mark** » Thu Mar 22, 2001 9:56 am

1. gw won't do that. You could either delete unwanted pages after the walk. Or you could use the scripted walker and customize it to only store the desired pages.

2. You would need a full Texis license for that level of control over the database.

valery · Post by **valery** » Thu Mar 22, 2001 5:05 pm

> Or you could use the
> scripted walker and customize it to only store the desired pages.
What is the 'scripted walker' and where can I get it?

> 2. You would need a full Texis license for that level of control over the
> database.
Could you give us a quote on that?

Thanks,
Valery.

Post by **mark** » Thu Mar 22, 2001 5:23 pm

The scripted walker has been mentioned numerous times recently on this board. Do a search for "scripted walker" to get the url.

Please use the "Contact Us" link on the left side menu to get Texis pricing info.