Thunderstone Support Forums

I work for an ISP and am trying to set up the webinator to index our home
page. I am running into the problem of gw fetching our user pages that are
linked to our home page (i.e. we have a page that lists all of our users
pages and gw wants to index everyone of these). I tried putting lines in
the robots.txt such as Disallow: /~* and Disallow: /%7* but neither of
these worked. I just want to create an index of files in the directory
that our homepage is kept in and all underlying directories. Any help
would be much appreciated.

thanks-
Mike

The "Disallow" field of robots.txt is not a wildcard. It is a prefix.
Try:
Disallow: /~

Robots.txt is documented at:
http://www.thunderstone.com/gw2man/node28.html
and other places on the web.