I have recently run into a problem indexing several sites where there are
several copies of a page on a site with different URL's.
Ecample:
www.somesite.com/aaa/bbb.html - exact same content
www.somesite.com/bbb/bbb.html - exact same content
www.somesite.com/bbb/sadkfakjsdfnaksjdf.html - exact same content
I would like to have only one of these pages in the index, discarding the
other two, so that when someone searches for "exact", only one page shows
up. (one site actually has 20 to 30 copies of some pages, only difference
is the url).
Thanks in advance.
Mike