performance issue on production server when crawling reports using thunderstone crawler

Post Reply
debasish.behera
Posts: 1
Joined: Fri Feb 21, 2014 4:58 am

performance issue on production server when crawling reports using thunderstone crawler

Post by debasish.behera »

We have just migrated our current IBM Websphere server , Oracle,Jasper 1.2.7, Java1.4 to Tomcat,MySql,Java 1.6 and Jasper 1.2.7 environment. But in production server we are seeing when thunderstone search crawler is crawling report urls, the threads are getting piled up and killing server. Please suggest settings? We tried changing below values, but no luck.

Thunderstone :

Max Process Size – 50 MB to Unlimited

Tomcat :

Max Threads – 200 to 400

Apache :

Timeout : 300 to 600
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

performance issue on production server when crawling reports using thunderstone crawler

Post by mark »

Seems your new setup can't take a rapid walk.
I can't help with tomcat or mysql.

In Webinator settings set threads to 1. If still problems try setting walk delay to non zero, maybe .1 or .5 or as much as 1 or 2 if the server is that poorly performing.

Max process size doesn't need to be increased unless you see frequent restarts due to that in the walk status. "Unlimited" should be avoided unless absolutely necessary as it could overwhelm the system if it got too huge.
Post Reply