Not crawling all URLs

Post Reply
johng1880
Posts: 2
Joined: Tue Dec 12, 2006 9:07 am

Not crawling all URLs

Post by johng1880 »

I'm trying to crawl www.drleonards.com, but it only crawls 1,900 URLs. Google says over 7,000 are indexed. I stripped queries when crawling and didn't haev any redirect errors. Any ideas on how to crawl more pages? Settings I might want to change?
User avatar
John
Site Admin
Posts: 2622
Joined: Mon Apr 24, 2000 3:18 pm
Location: Cleveland, OH
Contact:

Not crawling all URLs

Post by John »

Do you know of any specific pages that are not being indexed, or how many pages there should be? Do you want to index just www.drleonards.com, or all of the drleonards.com domain? It is possible that you have different URLs for the same thing that are being considered as duplicates, or you may have some pages only accessible with a query string, in which case you would want to remove ? from the exclusions, and not strip queries. Did you add .cfm to the extensions?
John Turnbull
Thunderstone Software
Post Reply