no-cache option

ryan4 · Post by **ryan4** » Fri Oct 19, 2001 11:49 am

Is there an option with gw to send a no-cache or similar header when indexing a site? Our site sits behind a reverse cache server, and I want to make sure webinator always gets the "latest" copy of the pages.
Thanks.

Post by **mark** » Fri Oct 19, 2001 12:57 pm

Not with gw. You could modify the scripted walker to do it though using <urlcp header ... ...>

roskaa · Post by **roskaa** » Tue Mar 16, 2004 4:01 am

uhm. i am having similar problem, the webinator crawls pages and reports invalid links in some pages. but there aint those links any more.

how it can be done and where?

i'm a rookie at this, please help me out

Post by **mark** » Tue Mar 16, 2004 10:43 am

Try
<urlcp header "cache-control" "no-cache">
<urlcp header "pragma" "no-cache">
right after
<urlcp clearheaders>
in fetchset.

roskaa · Post by **roskaa** » Thu Mar 25, 2004 5:14 am

nope. that did not help.

in the walk status -log there is something really weird.

I've tried to refresh the search and yes, it makes a different db(1 or 2) so that i guess ain't the problem.

there still comes a few duplicate error from a page that has not exist in a week..

like,

The link :my.host.com/url/
Referenced by :my.host.com/url2/
Is a duplicate of:my.host.com/url3/jada/

and in url3 there is only a blank page!

it this some sort of webinators internal cache issue?

Post by **mark** » Thu Mar 25, 2004 11:07 am

No. Webinator doesn't not cache pages, at least not in the way you're thinking.

Apparently url 1 was blank as well when webinator fetched it.

Note that a refresh walk doesn't erase the status. It appends to the status from previous new and refresh walks. Go to the end of the status to find where the latest one started and look at the messages below that. Or try a "new" walk rather than "refresh".

roskaa · Post by **roskaa** » Fri Mar 26, 2004 2:40 am

yes it is a new walk, i newer even tested refresh.

so if i put many blank pages - to 'hide' something - that comes a duplicate for other blanks ?

Post by **mark** » Fri Mar 26, 2004 10:23 am

Pages with duplicate content will not be stored by default. If 2 pages are empty only the first encountered will be stored. No great loss since there's nothing to find on the page anyhow.

There is an option under all walk settings to disable duplicate prevention if you really want the dups.

roskaa · Post by **roskaa** » Tue Mar 30, 2004 5:33 am

hi,

yes i am aware of that, but that does not solve that weird issue that i am having with this.

in that blank page there ain't _anything_. so how is that duplicate prevention working? where it gets that ?

invalid/corrupt database? a bug?

i just need to know how so i can prevent this from happening, because i could really use that duplicate info in my work.

Post by **John** » Tue Mar 30, 2004 7:30 am

Blank pages are duplicates of each other, as the content is the same, nothing. If you have pages that are linked, and return the same content, including no content, they will be flagged as duplicates.