Exclude Text Files

[marsh]

First of all I love you product it is really quite powerful.

When I "walk" my website I want to exclude files with the .txt file extension. They seem to be indexed by default. Is there a way to to this and how is it done

Thanks
Marshall White

[John]

You can use the -F option to remove extensions from the list, e.g. to not fetch .txt files:

gw -Ftxt http://somesite/

The -f (lowercase) option adds an extension.