I was having problems excluding specific asp pages on my site using a robot.txt page. So I created a Options.txt file in the webinator bin directory using the -x option to prohibit those specific pages inclusion within the webinator database.
To get this to work, I used the following command:
gw -d- -mOptions.txt http://www.xxx.org
Now my question is this, now that this database has been created can I use the rewalk function to automatically recall the -mOptions.txt file, or do I have to specify it in the rewalk command line, or do I have to use the command above every time I want to re-index and rewalk is no longer an option for my site?
Thanks for your help in advance.
To get this to work, I used the following command:
gw -d- -mOptions.txt http://www.xxx.org
Now my question is this, now that this database has been created can I use the rewalk function to automatically recall the -mOptions.txt file, or do I have to specify it in the rewalk command line, or do I have to use the command above every time I want to re-index and rewalk is no longer an option for my site?
Thanks for your help in advance.