You know, my website is update everyday. In order to keep the update
information of the Search engine, I am now using the command "re-walk"
to update the database. Everytime, I need more than two hours to finish
this task. It is quite time consuming.
Thus, I really want to know that is there any way to "scan" my website
and add only the new links to the database? But not completely re-walk
.
The site must be a particularly huge or slow to take anywhere near
that long. "Scanning" still requires contacting the webserver for every
page. You could try expire instead of rewalk to reduce the amount of
page data transferred. Something like:
