problem walking with url url.

Post Reply
jgdoke
Posts: 167
Joined: Wed Jul 14, 2004 10:52 am

problem walking with url url.

Post by jgdoke »

The only change I made was to include a url url that lists all the pages on my sites. After 16 hours the crawl had indexed less than 700 pages. Normal walk finishes with 11,000 pages in 18 hours. Could the size of my url file be the issue? there are 15,000 lines. I stopped the walk and all the walk data is gone. It shows the previous walk.
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

problem walking with url url.

Post by mark »

Not sure why it would be slow, but if you want to list all the pages instead of having the crawler discover them you should use page url instead of url url.

Yes, hitting "Stop" abandons the currently running walk. You can see the status for the abandoned walk from the Maintenance->Manage Logs->Profile Walk Status area. Use "Pause and live" to stop a walk and keep the data.
jgdoke
Posts: 167
Joined: Wed Jul 14, 2004 10:52 am

problem walking with url url.

Post by jgdoke »

using Page url means that hyperlinks found on the pages will not be followed. I want them to be followed.

Is there somthing else I am missing that using one vs the other?
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

problem walking with url url.

Post by mark »

I'm not sure why you'd need to follow links if you "list all the pages on your sites"?
jgdoke
Posts: 167
Joined: Wed Jul 14, 2004 10:52 am

problem walking with url url.

Post by jgdoke »

All current pages. this list took two days to come up with.. the appliance crawls most of the site but a lot of our links are in a javascript menu that the appliance does not seem to parse. I don't know why as I have the javascript stuff turned on..

So this is just a help to find more pages. I would like the appliance to find these on it's own but it doesn't.
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

problem walking with url url.

Post by mark »

The large list of base urls may or may not be the issue. Please open a ticket and send your all walk settings page, walk status page, and tech support info page.
Post Reply