I use webinator to index my site. All works really well except that it
also indexes some pages I don't want to. For e.g I have some pages which
I want only the people from a certain domain to see and so I have the
htaccess file. But since webinator is from that domain it is allowed
access to that file and it indexes that. Is there a way, in which
webinator can maybe present a false identity when retrieving files and
then not index the files in question.
Thanks
--Saurabh