I work for an ISP and am trying to set up the webinator to index our home
page. I am running into the problem of gw fetching our user pages that are
linked to our home page (i.e. we have a page that lists all of our users
pages and gw wants to index everyone of these). I tried putting lines in
the robots.txt such as Disallow: /~* and Disallow: /%7* but neither of
these worked. I just want to create an index of files in the directory
that our homepage is kept in and all underlying directories. Any help
would be much appreciated.
thanks-
Mike