Hello,
is it possible to index the DB while the walker is still walking the web?
If my walker is slow (say it's very polite and makes big pauses between
requests) and it takes a few days (say a week) to crawl all the URLs, is
there any way I can index the DB (whatever is in it up to that point) while
the walker is still web-walking, without having to stop the walker, index,
and restart the walker?
This would be useful to me because I want my index to be always fresh (so
I'd want the walker to just keep walking, and rewalking, and rewalking,
while at the same time I'd want the DB indexed as much as possible, or as
frequently as possible, so that it's always 'fully searchable')
Similarly, is it possible to have index creating as the walker walks?
For example, if a walker visits www.site.com/dir/file.html, can it store in
a DB and indexing right away, while continuing to do its walking?
Thanks,
Otis