keep/ignore tags

[mark]

Perhaps you can use "exclude by field" to recognize those index pages and Pages only but not Links.

[josh104]

I hadn't realized that exclude by field allowed links to be followed - thanks for bringing that to my attention.

The only remaining problem I can see with using exclude by field is that it widens the possibility of false positives/negatives as the site being indexed scales.

But still - it's a far improvement from the situation I thought I was facing. Thanks!!

[KMandalia]

I think I have asked this before and it was working for me before as well.

I have multiple keep tags on some web pages. I was told that the webinator will keep contents between each keep tags. I don't have any ingore tags on those pages.

It turns out that webinator is only keeping the last keep tag instead of all of them. Has something changed in recent dowalk scripts?

[mark]

No. That code hasn't been touched in quite a while. Having multiple keep pairs works for me. With content like

abc
<!-- begin index -->
def
<!-- end index -->
ghi
<!-- begin index2 -->
jkl
<!-- end index2 -->
mno

and keep tags of

<!-- begin index --> <!-- end index -->
<!-- begin index2 --> <!-- end index2 -->

The resulting text is "def jkl"

[KMandalia]

there goes my issue.

instead of index and index2, I use beginkeep and endkeep multiple times in a page, thinking that every time a keep tag is encountered, content is scrapped and stored in memory while appending to previous content. I was wrong then.

[mark]

No. Using the same tag multiple times also works.

abc
<!-- begin index -->
def
<!-- end index -->
ghi
<!-- begin index -->
jkl
<!-- end index -->
mno

and keep tags of

<!-- begin index --> <!-- end index -->
<!-- begin index2 --> <!-- end index2 -->

The resulting text is "def jkl"