Re-crawling initial site

Post Reply
User avatar
Thunderstone
Site Admin
Posts: 2504
Joined: Wed Jun 07, 2000 6:20 pm

Re-crawling initial site

Post by Thunderstone »



Gee Whiz!

First I indexed my site with the simple gw URL command. It was
quick and VERY impressive.

But I didn't specify to crawl all my .asp pages. So I wiped the
database, and crawled again, this time using -fasp. But alas, still
no .asp pages were returned when searching.

My question is, how do I get these .asp pages to show up in a
search?

Please help!

Jerry


User avatar
Thunderstone
Site Admin
Posts: 2504
Joined: Wed Jun 07, 2000 6:20 pm

Re-crawling initial site

Post by Thunderstone »




That should work unless the .asp pages are under directories excluded
with -x or robots.txt . What's your whole command line?

See what urls are in your database with something like:
gw -st "select Url from html"


Post Reply