Refresh just spins, New walk works fine

jon-paul · Post by **jon-paul** » Fri Mar 18, 2011 11:33 am

Hi gang... been a while.

I recently moved off our ancient Win 2003/IIS 5 server to a new Win 2008/IIS7 box.

For the most part I have Webinator (commercial, dowalk 5.1.88) working fine.

However, Refresh walks aren't doing anything. If I start a NEW walk, it'll crawl just fine. After a new walk finishes, I'll switch it to a daily refresh and when I get into work the next morning - I check and Webinator is just spinning. It's complaining about not inserting duplicates and not finding anything to refresh.

Sample of the notices:

Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 10:24:31 started 1 refresh (4824) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 10:24:31 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (4824)
Using primer: http://192.168.71.50:8010/search/urlindex.htm

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://192.168.71.50:8010/search/urlindex.htm) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr
Reading urls from file c:\websites\webinator\search\urlindex.htm

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.bemis.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

------------------
Any idea's? Suggestions?

Thanks!

Post by **John** » Fri Mar 18, 2011 12:12 pm

What do you have the refresh times set to on the All Walk Settings page?

The Walk Status page should list the next URLs to be refreshed, and when they should be checked.

jon-paul · Post by **jon-paul** » Fri Mar 18, 2011 12:21 pm

Default Refresh Time: 1min
Minimum Refresh Time: 1min
Maximum Refresh Time: 1week

Too short?

Post by **John** » Fri Mar 18, 2011 1:17 pm

No, all the pages should have been scheduled to rewalk after the new walk finished. Does the Walk Status page show how many pages are scheduled to refresh?

jon-paul · Post by **jon-paul** » Fri Mar 18, 2011 1:34 pm

This is the last walk status. I killed the walk since it wasn't doing anything other than spitting up errors.

I have a webpage that contains all the initial URLs to crawl.

---------------------------------------

Latest run:
13 pages in todo
0 pages visited in the last hour (0 success/0 failed)
1,108 pages in index

Pages recently walked
1,108 pages (46,274,085 bytes).
5 errors.
0 duplicate pages.

Page Visited Modified Url
-------+-------------------+-------------------+-------------------------------------------------------
1108 1 d, 3 hr+ ago 198 d, 4 hr+ ago http://www.bemis-europe.com/public/pdf/ ... ons-de.pdf (95,570 bytes)
1107 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/lang/en/abo ... %20Ireland (9,306 bytes)
1106 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/lang/en/abo ... %20Finland (8,634 bytes)
1105 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/lang/en/abo ... a,%20Wales (8,364 bytes)
1104 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/lang/en/abo ... %20Belgium (8,549 bytes)
1103 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/lang/en/abo ... ,%20France (8,800 bytes)
1102 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/lang/en/abo ... sham,%20UK (8,935 bytes)
1101 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/contact.asp (18,249 bytes)
1100 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/bemislocati ... ixie_toga/ (6,584 bytes)
1099 1 d, 3 hr+ ago 1 d, 3 hr+ ago http://www.bemis-europe.com/bemislocati ... erfecseal/ (7,512 bytes)

Recent errors
Visited Reason Url
--------------------+--------------------+-------------------------------------------------------
1 d, 3 hr+ ago Document not found: http://www.bemistape.com/sitemap/
1 d, 3 hr+ ago Timeout reading from http://www.mactac.com/fileadmin/user_up ... eaBook.pdf
1 d, 3 hr+ ago Document not found: http://www.bemisppd.com/sitemap/
1 d, 3 hr+ ago Document not found: http://www.bemis.com/sitemap
1 d, 3 hr+ ago Error translating vi http://www.bemis.com/public/pdf/BMS2010_AnnualRpt.pdf

Next Pages to be walked
Next Check Modified Url
--------------------+------------------+-------------------------------------------------------
ASAP 1 d, 3 hr+ ago http://www.bemis.com/ (9,061 bytes)
ASAP 1 d, 3 hr+ ago http://www.bemis.com/2011annualmeeting/ (5,343 bytes)
ASAP 1 d, 3 hr+ ago http://www.curwood.com/ (9,353 bytes)
ASAP 1 d, 3 hr+ ago http://www.bemis.com/bemislocations/ (5,104 bytes)
ASAP 1 d, 3 hr+ ago http://www.curwood.com/about_curwood/ (7,364 bytes)
ASAP 1 d, 3 hr+ ago http://www.bemis.com/careers/ (5,787 bytes)
ASAP 1 d, 3 hr+ ago http://www.curwood.com/bemislocations/ (5,156 bytes)
ASAP 1 d, 3 hr+ ago http://www.bemis.com/citizenship/ (10,876 bytes)
ASAP 1 d, 3 hr+ ago http://www.curwood.com/contact/ (15,811 bytes)
ASAP 1 d, 3 hr+ ago http://www.bemis.com/citizenship/2/bemi ... oundation/ (11,648 bytes)

Walk started at 2011-03-18 09:46:12 (by resume)
JavaScript walking not enabled by current license
HTTPS walking disabled
Start fetching at http://192.168.71.50:8010/search/urlindex.htm

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://192.168.71.50:8010/search/urlindex.htm) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr
Reading urls from file c:\websites\webinator\search\urlindex.htm

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.bemis.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.curwood.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.perfecseal.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.clysar.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.milprint.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.mactac.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.bemisppd.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.bemistape.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.bemis-industrial.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.bemispaper.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.bemis-europe.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr

178 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk(toptodo) 6062: Trying to insert duplicate value (http://www.bemis150.com/) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\Bemis\db1\xtodourl.btr
Ignore urls containing any of the following:
/cgi-bin/
~
/scripts/
/xml/
/asp/
/_asp/
/images/
/connections/
/includes/
/include/
/inc/
/_inc/
/employment/
/employment
/feed
http://www.sterilizationpackaging.org/*
........Report abbreviated, click for full report.
2011-03-18 11:53:12 started 1 refresh (4420) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:12 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (4420)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:14 started 1 refresh (4292) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:14 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (4292)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:16 started 1 refresh (1356) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:16 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (1356)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:18 started 1 refresh (4772) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:18 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (4772)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:20 started 1 refresh (3588) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:20 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (3588)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:22 started 1 refresh (3928) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:22 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (3928)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:24 started 1 refresh (3848) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:24 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (3848)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:27 started 1 refresh (3100) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:27 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (3100)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:29 started 1 refresh (3336) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:29 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (3336)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:31 started 1 refresh (3008) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:31 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (3008)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds
2011-03-18 11:53:33 started 1 refresh (1008) on http://192.168.71.50:8010/search/urlindex.htm
2011-03-18 11:53:33 Nothing to refresh at /=http=-post?>>\L://192.168.71.50/\L (1008)
Using primer: http://192.168.71.50:8010/search/urlindex.htm
0 pages fetched (0 bytes) from http://192.168.71.50:8010/search/urlindex.htm took 1 seconds

Dispatcher stopping by request. May take up to 65 seconds to stop.
Dispatcher exiting.
Cancelled by user: 2011-03-18 11:53:35
End of report.
Report abbreviated, click for full report.
Show Errors (only latest walk)
Show Duplicates (only latest walk)

Post by **mark** » Fri Mar 18, 2011 3:51 pm

Is that a base url or a url url or ??
What other settings are changed from defaults?

It maybe easier to do this through Tech Support (link at the top) so you can send your entire settings page.

jon-paul · Post by **jon-paul** » Fri Mar 18, 2011 3:58 pm

Okay... I'll shoot a tech support ticket in.