Limited Results Of Walk

lbneal · Post by **lbneal** » Sat Jun 19, 2010 10:10 pm

The target site has some 3500+ actual pages.

Webinator only indexes about 176 of them?

I have tried a variety of walk settings with no improvement in results.

I have tried New and Refresh both die at that count.

Post by **John** » Sun Jun 20, 2010 1:18 pm

There are a couple of possible reasons for a crawl stopping early. The two main ones are either not finding the links, or the links are configured to be excluded, either by robots.txt or one of the other settings, for example the extension is not in the allow list.

Increasing the verbosity to 4 will give information on all the URLs, and why they aren't followed.

lbneal · Post by **lbneal** » Sun Jun 20, 2010 2:04 pm

Thanks! That seems to be a big help!

The number of hits is growing...

lbneal · Post by **lbneal** » Sun Jun 20, 2010 7:59 pm

However the the number increased to only 185...

Out of 3500+ pages I can see only 44 in the database...

lbneal · Post by **lbneal** » Sun Jun 20, 2010 8:38 pm

John -

I am using the Free version of Webinator.

My usage is NOT Politically Correct.

If you can help me with this issue it would be appreciated.

Otherwise no problem.

The package obviously has usability issues fixing such would enhance the value...

lbneal · Post by **lbneal** » Mon Jun 21, 2010 6:29 pm

Thanks to the help of John my site is now pretty much working.

I had a programming issue on a couple of the target sites.