visit page twice

shaker31280 · Post by **shaker31280** » Mon Jul 30, 2001 3:44 pm

Does webinator visit a page twice? For example if I have one page linking to X, and webinator goes there and indexes it, what happens when webinator sees that page again, does it index it again? or does it skip it? Does it keep a history to say, oh I saw that page, move on? Thanks

Post by **mark** » Mon Jul 30, 2001 4:46 pm

Webinator does not fetch the same url twice. If it's a different url but with the same content, it has to fetch it again to find out if it's dup when you're using -unique.