Crawl Hanging

kiddywood · Post by **kiddywood** » Mon Jan 21, 2013 12:29 pm

Hi,

My crawl which should be 30,000 single page URLs is just hanging at around 2,000 pages with 0 pages in todo.

This is from the log:

100 2013-01-21 10:03:19 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date Wed, 31 Dec 1969 23:59:59 GMT is invalid/out of range
100 2013-01-21 10:03:20 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:5044: Document not found: http://gs11.globalsuccessor.com/fe/imag ... s/sh18.htm returned code 404 (Not Found)
000 2013-01-21 10:03:21 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:4731: Cannot write to stdin for process 3144: The pipe is being closed in the function tx_prw_finishiotask
118 2013-01-21 10:03:21 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:4731: <EXEC> command "C:\Program Files (x86)\Thunderstone Software\Webinator\anytotx.exe" returned exit code 1
015 2013-01-21 10:03:41 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: incomplete time: in the function plusminus
100 2013-01-21 10:03:41 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date -1 is invalid/out of range
100 2013-01-21 10:03:49 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date Wed, 31 Dec 1969 23:59:59 GMT is invalid/out of range
015 2013-01-21 10:03:50 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: incomplete time: in the function plusminus
100 2013-01-21 10:03:50 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date -1 is invalid/out of range
100 2013-01-21 10:03:51 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date Wed, 31 Dec 1969 23:59:59 GMT is invalid/out of range
015 2013-01-21 10:03:55 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: incomplete time: in the function plusminus
100 2013-01-21 10:03:55 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date -1 is invalid/out of range
100 2013-01-21 10:03:56 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date Wed, 31 Dec 1969 23:59:59 GMT is invalid/out of range
015 2013-01-21 10:03:59 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: incomplete time: in the function plusminus
100 2013-01-21 10:03:59 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date -1 is invalid/out of range

kiddywood · Post by **kiddywood** » Mon Jan 21, 2013 12:38 pm

Last error in walk report:

The link : http://www.atlascopco.us/usus/careers/j ... F0005370FA
Had this error: Error translating via anytotx: <EXEC> command "C:\Program Files (x86)\Thunderstone Software\Webinator\anytotx.exe" returned exit code 1 018 2013-01-21 10:03:21 Cannot exec `gunzip': The system cannot find the file specified in the function TXpopenduplex ("C:\Program Files (x86)\Thunderstone Software\Webinator\anytotx.exe" --content-type=application/x-gzip --timeout=60 "--error-log=C:\Program Files (x86)\Thunderstone Software\Webinator/texis//anytotx-errors.4080")

Post by **jason112** » Mon Jan 21, 2013 12:52 pm

Is it continuing to add lines to the log about "invalid dates", or has that stopped?

If it's still adding them, then the URLs are being crawled but not saved; try setting Verbosity=4 and re-crawling to see why it's not saving them.

If not, look in your profile's data directory for a "cururls.<num>" file, where <num> will be some number, and let us know what's in the file.

kiddywood · Post by **kiddywood** » Mon Jan 21, 2013 1:12 pm

The invalid dates lines stopped adding as it hanged.

My verbosity is set to 4.

The currls.4080 has this in it:

2013-01-21 10:04:14 About to fetch 94 URLs with 5 threads:
http://www.oegoffshore.com/career.php
http://www.viastore.com/employment-oppo ... no_cache=1
http://www.viasyshealthcare.com/careers ... ig=hr_jobs
http://www.e-zest.net/openings.html
http://www.videomeetingcompany.com/about-us/Vacancies/
http://www.drylube.co.uk/about.php?page ... ortunities
http://www.v4technical.com/web-mobile-d ... bs-suffolk
http://www.deansplc.co.uk/employment_at_DCS.aspx
http://www.danlers.co.uk/recruitment/index.html
http://www.cresent.co.uk/html/careers.html
http://www.stonegroup.co.uk/about-us/ca ... ncies.aspx
http://www.civilsandlintels.co.uk/pages ... tment.html
http://www.cadre-components.co.uk/jobopportunities.php
http://www.bowtech.co.uk/Careers-Bowtech.html
http://www.bvg-airflo.co.uk/vacancies_bvg.cfm
http://www.buchaninspection.com/careers.html?dis_id=1
http://www.btbell.co.uk/Vacancies.php
http://www.langleather.co.uk/Lang/lng_E ... ities.aspx
http://www.ask4.com/business/about-us/careers.php
http://www.virocom.co.uk/aboutus_careers.htm
http://www.lighterlife.com/about-lighte ... ncies.aspx
http://www.vitaflo.co.uk/content/AboutV ... reers.aspx
http://www.volunteering.org.uk/WhoWeAre/jobvacancies/
http://www.vsi-thinking.com/recruitment.aspx
http://www.vtsroyalite.co.uk/vitasheet/ ... ities.html
http://www.wagonautomotive.com/wagonaut ... iew=search
http://www.wagonautomotive.com/wagonaut ... rch=Search
http://www.wagonautomotive.com/wagonaut ... rch=Search
http://www.walkermorris.co.uk/content.aspx?id=73
http://www2.wallacecameron.com/recruitment/vacancies/
http://www.kingspanpanels.com/Contact/Recruitment.aspx
http://www.waterflow.co.uk/careers/vacancies
http://www.waterperrygardens.co.uk/footer-vacancies.php
http://www.water-systems.co.uk/html/job_vacancies.html
http://www.watts-international.com/prof ... vacancies/
http://www.watts-international.com/admin-vacancy/
http://www.waxinfo.com/company/careers.html
http://www.wds2000.com/recruitment.asp
http://www.wearvalley.gov.uk/index.cfm?articleid=4062
http://www.websitedesign.co.uk/jobs.php
http://www.welcom.co.uk/careers/Vacancies.htm
http://www.we-online.com/web/en/wuerth_ ... bote_8.php
http://www.westermo.com/Resource.phx/co ... obs/uk.htx
http://www.westfieldhealth.com/old/about-us/jobs.asp
http://www.west-cs.com/About/Careers/Opportunities/
http://www.ramboll.co.uk/careers/vacanc ... Country-uk
http://www.williams-refrigeration.com/index_33.htm
http://www.wiredmedia.co.uk/digital-age ... .html#Jobs
http://www.wiseowl.co.uk/job/index.htm
http://www.withyking.co.uk/site/about/careers/
http://www.wmc.org.uk/index.cfm?UUID=D1 ... B9513EA149
http://www.wmfs.net/Jobs_Online/Current_Vacancies/
http://www.woh.co.uk/careers.htm
http://www.juicymedia.co.uk/what-we-do/jobs.html
http://www.steatite.co.uk/recruitment.html
http://www.wpa.org.uk/about/jobs.html
http://www.penrillian.com/careers.html
http://www.wtxplc.com/page13.htm
http://www.flexiant.com/about/careers/
http://www.xma4education.co.uk/coinfo.aspx
http://www.xrange.com/career/apply-job/
http://www.xtrac.com/Vacancies%20-%20Administration.htm
http://www.xtrac.com/Vacancies%20-%20Assembly.htm
http://www.xtrac.com/Vacancies%20-%20Engineering.htm
http://www.xtrac.com/Vacancies%20-%20Finance.htm
http://www.xtrac.com/Vacancies%20-%20Manufacturing.htm
http://www.xtrac.com/Vacancies%20-%20R+D.htm
http://www.xtrac.com/Vacancies%20-%20Sa ... upport.htm
http://www.justice.gov.uk/jobs/inspectorates/index.htm
http://www.justice.gov.uk/jobs/opg/index.htm
http://www.kallkwik.co.uk/fleetstreet/careers/
http://www.fordbros.co.uk/vacancies-here.html
http://www.finnforest.co.uk/careers/Pag ... ities.aspx
http://www.yorkshire-forward.com/about/ ... -vacancies
http://www.yougov.co.uk/corporate/careers/default.asp
http://www.axiumprocess.com/empopp.html
http://www1.creditsafeuk.com/?id=1419
http://www1.landregistry.gov.uk/info/vacancies
http://www2.valeroyal.gov.uk/internet/v ... b?openview
http://www4.sungard.com/jobs/index.php? ... 10&map.y=3
http://www4.sungard.com/jobs/index.php? ... 13&map.y=4
http://www4.sungard.com/jobs/index.php? ... =6&map.y=0
http://www.ermgassen.com/portal/loader. ... n&root=305
http://www.alliancetrust.co.uk/careers/vacancies.htm
http://www.hawkpoint.com/careers.html
http://www.holden-partners.co.uk/site/6 ... tment.aspx
https://barco.taleo.net/careersection/f ... 4601381138
http://www.voss-solutions.com/About/Overview/Careers/
http://www.ontology.com/company/careers.aspx
https://tbe.taleo.net/NA1/ats/servlet/R ... bVersion=0
http://www.lush.co.uk/index.php?option= ... design.php
http://www.lush.co.uk/index.php?option= ... turing.php
http://www.lush.co.uk/index.php?option= ... sstime.php
https://www.lush.co.uk/index.php?view=c ... Itemid=100

Post by **jason112** » Mon Jan 21, 2013 1:45 pm

Does a texis process with the id 4080 still exist? Is it consuming CPU?

What are the last few lines of "texis\vortex.log" in your installed directory?

kiddywood · Post by **kiddywood** » Mon Jan 21, 2013 3:24 pm

I restarted the walk with the same settings and same URLs and this time it stalled on 7,240 pages.

last lines on vortex.log:

100 2013-01-21 19:39:24 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date -1 is invalid/out of range
100 2013-01-21 19:39:28 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:919: Date Wed, 31 Dec 1969 23:59:59 GMT is invalid/out of range
178 2013-01-21 19:39:30 C:\Program Files (x86)\Thunderstone Software\Webinator\texis\scripts/webinator/dowalk:4165: Trying to insert duplicate value (http://www.apointplus.com/tatasteel/app ... px?id=4281) in index C:\Program Files (x86)\Thunderstone Software\Webinator\texis\db1\xhtmlurl.btr

cururls file 4528:

2013-01-21 19:39:43 About to fetch 94 URLs with 5 threads:
http://www.oegoffshore.com/career.php
http://www.viastore.com/employment-oppo ... no_cache=1
http://www.viasyshealthcare.com/careers ... ig=hr_jobs
http://www.e-zest.net/openings.html
http://www.videomeetingcompany.com/about-us/Vacancies/
http://www.drylube.co.uk/about.php?page ... ortunities
http://www.v4technical.com/web-mobile-d ... bs-suffolk
http://www.deansplc.co.uk/employment_at_DCS.aspx
http://www.danlers.co.uk/recruitment/index.html
http://www.cresent.co.uk/html/careers.html
http://www.stonegroup.co.uk/about-us/ca ... ncies.aspx
http://www.civilsandlintels.co.uk/pages ... tment.html
http://www.cadre-components.co.uk/jobopportunities.php
http://www.bowtech.co.uk/Careers-Bowtech.html
http://www.bvg-airflo.co.uk/vacancies_bvg.cfm
http://www.buchaninspection.com/careers.html?dis_id=1
http://www.btbell.co.uk/Vacancies.php
http://www.langleather.co.uk/Lang/lng_E ... ities.aspx
http://www.ask4.com/business/about-us/careers.php
http://www.virocom.co.uk/aboutus_careers.htm
http://www.lighterlife.com/about-lighte ... ncies.aspx
http://www.vitaflo.co.uk/content/AboutV ... reers.aspx
http://www.volunteering.org.uk/WhoWeAre/jobvacancies/
http://www.vsi-thinking.com/recruitment.aspx
http://www.vtsroyalite.co.uk/vitasheet/ ... ities.html
http://www.wagonautomotive.com/wagonaut ... iew=search
http://www.wagonautomotive.com/wagonaut ... rch=Search
http://www.wagonautomotive.com/wagonaut ... rch=Search
http://www.walkermorris.co.uk/content.aspx?id=73
http://www2.wallacecameron.com/recruitment/vacancies/
http://www.kingspanpanels.com/Contact/Recruitment.aspx
http://www.waterflow.co.uk/careers/vacancies
http://www.waterperrygardens.co.uk/footer-vacancies.php
http://www.water-systems.co.uk/html/job_vacancies.html
http://www.watts-international.com/prof ... vacancies/
http://www.watts-international.com/admin-vacancy/
http://www.waxinfo.com/company/careers.html
http://www.wds2000.com/recruitment.asp
http://www.wearvalley.gov.uk/index.cfm?articleid=4062
http://www.websitedesign.co.uk/jobs.php
http://www.welcom.co.uk/careers/Vacancies.htm
http://www.we-online.com/web/en/wuerth_ ... bote_8.php
http://www.westermo.com/Resource.phx/co ... obs/uk.htx
http://www.westfieldhealth.com/old/about-us/jobs.asp
http://www.west-cs.com/About/Careers/Opportunities/
http://www.ramboll.co.uk/careers/vacanc ... Country-uk
http://www.williams-refrigeration.com/index_33.htm
http://www.wiredmedia.co.uk/digital-age ... .html#Jobs
http://www.wiseowl.co.uk/job/index.htm
http://www.withyking.co.uk/site/about/careers/
http://www.wmc.org.uk/index.cfm?UUID=D1 ... B9513EA149
http://www.wmfs.net/Jobs_Online/Current_Vacancies/
http://www.woh.co.uk/careers.htm
http://www.juicymedia.co.uk/what-we-do/jobs.html
http://www.steatite.co.uk/recruitment.html
http://www.wpa.org.uk/about/jobs.html
http://www.penrillian.com/careers.html
http://www.wtxplc.com/page13.htm
http://www.flexiant.com/about/careers/
http://www.xma4education.co.uk/coinfo.aspx
http://www.xrange.com/career/apply-job/
http://www.xtrac.com/Vacancies%20-%20Administration.htm
http://www.xtrac.com/Vacancies%20-%20Assembly.htm
http://www.xtrac.com/Vacancies%20-%20Engineering.htm
http://www.xtrac.com/Vacancies%20-%20Finance.htm
http://www.xtrac.com/Vacancies%20-%20Manufacturing.htm
http://www.xtrac.com/Vacancies%20-%20R+D.htm
http://www.xtrac.com/Vacancies%20-%20Sa ... upport.htm
http://www.justice.gov.uk/jobs/inspectorates/index.htm
http://www.justice.gov.uk/jobs/opg/index.htm
http://www.kallkwik.co.uk/fleetstreet/careers/
http://www.fordbros.co.uk/vacancies-here.html
http://www.finnforest.co.uk/careers/Pag ... ities.aspx
http://www.yorkshire-forward.com/about/ ... -vacancies
http://www.yougov.co.uk/corporate/careers/default.asp
http://www.axiumprocess.com/empopp.html
http://www1.creditsafeuk.com/?id=1419
http://www1.landregistry.gov.uk/info/vacancies
http://www2.valeroyal.gov.uk/internet/v ... b?openview
http://www4.sungard.com/jobs/index.php? ... 10&map.y=3
http://www4.sungard.com/jobs/index.php? ... 13&map.y=4
http://www4.sungard.com/jobs/index.php? ... =6&map.y=0
http://www.ermgassen.com/portal/loader. ... n&root=305
http://www.alliancetrust.co.uk/careers/vacancies.htm
http://www.hawkpoint.com/careers.html
http://www.holden-partners.co.uk/site/6 ... tment.aspx
https://barco.taleo.net/careersection/f ... 4601381138
http://www.voss-solutions.com/About/Overview/Careers/
http://www.ontology.com/company/careers.aspx
https://tbe.taleo.net/NA1/ats/servlet/R ... bVersion=0
http://www.lush.co.uk/index.php?option= ... design.php
http://www.lush.co.uk/index.php?option= ... turing.php
http://www.lush.co.uk/index.php?option= ... sstime.php
https://www.lush.co.uk/index.php?view=c ... Itemid=100

kiddywood · Post by **kiddywood** » Thu Jan 24, 2013 1:30 pm

Hi,
Is anyone looking at this please??

Post by **mark** » Thu Jan 24, 2013 2:22 pm

When this happens does the process with the PID in the cururls filename still exist? Is it consuming CPU or huge memory?

You don't have timeout set to a huge number do you?
Is rewalk type set to new or refresh?
What other non-default settings do you have?
What the version as given by "texis -version"?

kiddywood · Post by **kiddywood** » Thu Jan 24, 2013 3:04 pm

Hi Mark

No, there is no process with that ID.

Page timeout is set to 60
Rewalk is set to new

I have threads at 5 and servers at 8
Maximum process size - large
Minimum free disk space - 0
Fetch javascript - no

When I moved to this new server, I copied the settings exactly as they were on the old one which was working perfectly.

Here is the version:

Webinator Professional Version 5.01.1218161949 20080808 (i686-intel-winnt-64-32)

Post by **mark** » Thu Jan 24, 2013 3:48 pm

If there's no process I would expect to see something in vortex.log about it exiting or something in the walk status. If neither of those I'd have to guess that the OS is killing the process without giving it a change to close out. Does event viewer say anything about it?