Hi Bart,
What is the liklihood of thunderstone writing an additional
piece of code, accessed as a command-line switch, which causes
WEBINATOR to view a web-site in tree fashion, and NOT as a web.
This would come much closer to your stated (on the phone) goal of
your clients using WEBINATOR as a topic specific indexer.
Given that the methods you described for searching out all of those
links which are not relevant are quite labor intensive, and since you
said it would be quite easy to have the spider view the web as a tree,
it would be a great addition.
David