excluding urls

roshan_shankar · Post by **roshan_shankar** » Thu May 31, 2001 12:15 pm

Hi,
I have a set of urls that i need to exclude
so I created a url.lst list file

which reads
http://www.mysite.com
-http://www.mysite.com/exclude/page1.html
-http://www.mysite.com/exclude/page2.html

But, when i do the walk I still see pages being retrieved, like
http://www.mysite.com/exclude/page1.htm ... =1&param=2

Post by **mark** » Thu May 31, 2001 12:54 pm

That should work the same as using -xhttp://www.mysite.com/exclude/page1.html . Make sure you don't have any trailing spaces in your url.lst file.

roshan_shankar · Post by **roshan_shankar** » Thu May 31, 2001 1:20 pm

Should i put -xhttp://www.mysite.com/exclude/page1.html in my url.lst

Post by **mark** » Thu May 31, 2001 2:02 pm

No, I was just saying that the syntax you used int he url list is equivalent to the using -x... on the command line (or option file) in case you felt like trying it that way.

What you should do is make sure you don't have any trailing spaces in your url.lst file.

roshan_shankar · Post by **roshan_shankar** » Thu May 31, 2001 5:23 pm

I tried this -

gw -d"c:\my.db" -v0 -l"c:\my.log" -wipe -M/"Agent 3.0" -noindex -unique -b -R -t180 -q600 -fjhtml -xhttp://www.mysite.com/working/businesslistings.jhtml http://www.mysite.com

But it still retrieves
xhttp://www.mysite.com/working/businesslistings ... =1&param=2

Please let me know what mistake am i making.

Post by **mark** » Thu May 31, 2001 6:11 pm

see your other thread about combining -wipe and -unique. if you wipe first it will probably work correctly.