Duplicates

jgdoke · Post by **jgdoke** » Fri Jan 21, 2005 10:15 am

Is there any way to not have you pull duplicates of these url?
The link : http://www.ab.com/safety/prod_directory/
Is a duplicate of: http://www.ab.com/safety/prod_directory
Referenced by : http://www.ab.com/
http://www.ab.com/safety/

Esentially these are the same:
http://www.ab.com/safety/prod_directory
http://www.ab.com/safety/prod_directory/
http://www.ab.com/safety/prod_directory/index.html

It is a waste of time to pull each of these and compare them..

thanks
John

Post by **mark** » Fri Jan 21, 2005 10:20 am

The "Index Name" setting will prevent redundant fetching of index.html. It'll still fetch the forms with and without the slash though.

jgdoke · Post by **jgdoke** » Fri Jan 21, 2005 12:24 pm

Index Name:index.html index.htm

Looks as though this setting is correct unless having two names is a problem..

Post by **mark** » Fri Jan 21, 2005 1:38 pm

There may be many names there separated by space. That basically says to strip "index.html" or "index.htm" from the end of urls before fetching them.

jgdoke · Post by **jgdoke** » Fri Jan 21, 2005 1:59 pm

If the setting is correct then why is it fetching the page twice?????

John

Post by **mark** » Fri Jan 21, 2005 2:27 pm

If the setting wasn't correct it would fetch it 3 times.

http://abc/def
http://abc/def/
are always treated as different.
http://abc/def/index.html
is treated as
http://abc/def/