A side effect of all the new features in gw v.2 has resulted in a
performance decrease when you are indexing sites that make reference
to MANY other webservers.
Here's why:
We have to lookup all host names that are referenced in the html in order
to determine if they might actually be aliases for the same host.
If you walk a site similar to Yahoo that has a boatload of offsite
href's, Webinator 2 has to look up all the names even though it
has no intent to crawl them too.
Solution:
We are adding yet another option to gw that will disable name lookup.
Side-Effect:
If you use the new option, name aliases for the same site will be ignored.
This option will be available next Monday at 5:00pm EST.
Thanks,
Thunderstone