Thunderstone Support Forums

In our crawl, we want it to crawl pages for the URL's but don't want to include those particular pages. Are we missing a setting that allows us to do that?

Use meta robots on those pages or see "exclude by field" under all walk settings. Use field "Url".