Avoiding Spider Fluff...

[hayden]

On many of our internal pages there's always
the "fluff" in the left and right columns of
the HTML with the important body text somewhere
in the middle. For example, CNN has their menu
stuff on each side of the main body of the
pages important content.

As we have control over our internal content
can we have the spider only index text say
between something like <bodytext>...</bodytext>?
It would just help our searches bring back more
meaningful results.

thanks
-allen

[mark]

The scripted walker can be modified to do that.

[bart]

We've taken steps in the upcoming release of Webinator to remove the "fluff". We've had it running for some time now and it does a pretty good job of getting rid of the redundant-info problem.

Beta's of the new release will be available in a few weeks.