Hi! I made a new profile (copy of the "default") and it's only indexing the homepage. Other profiles are (seemingly) identical and work fine. I have no idea what the problem could be.
Are there any children listed for that homepage in list/edit URLs?
Is the site publicly accessible? It's possible that
links could be interpreted incorrectly, or there could be
a "robots.txt" that tells webinator not to crawl it (which
it obeys by default).
>>> Are there any children listed for that homepage in list/edit URLs?
Yes, there's a whole bunch, but they are all unlinked (not in the database).
---------------------------------
>>> Is the site publicly accessible?
No, there is an .htaccess and .htpasswd file, but I have filled in the auth info in the "Login" fields.
---------------------------------
>>> It's possible that
links could be interpreted incorrectly, or there could be a "robots.txt" that tells webinator not to crawl it (which it obeys by default)?
Yes, there's a robots.txt file that says "follow none" but we have the application set not to listen to the robots.txt file.
Turn verbosity up to 4. Run a mode new walk. Go to list/edit urls again and click children. Look to the right of all of the unlinked child urls to see why they were rejected.
Verbosity 4 should show the reasons. Typical reasons for not following the links are they have an extension not in the extension list, they match an exclude or robots.txt, query stripping, or they are off-site or in a different directory than the homepage. Does walk status show any reason for stopping?
No reason probably indicates that the walk stopped prematurely. Possible reasons: user hit stop or pause, license limit reached, selected max pages, bytes, or depth limit reached. Check the walk status as John suggested.
