Thunderstones webservice data load times out

[david.huber0]

Hello,

we are trying to send to thunderstone, through the data load webservice, many documents.

After about 5000 documents send the webservice times out. The document are sent one by one after the werbservice returns. After monitoring the texis process it appears that it uses a lot of cpu and memory. Furhtermore it seems that the process isn't able to free memory accurately while using the webservice.

For information the webservice client is a .net application.

Any idea?

[jason112]

Are you seeing requests slow down with increasing times, or does it suddenly hang with a request & time out?

Once it starts timing out, what gets it to not time out?

Does it start timing out on the same document, or is it somewhat randomized?

> Furhtermore it seems that the process isn't able to
> free memory accurately while using the webservice.

What makes you say that? If you're sending the documents individually, then each request should be a separate process and return very quickly, so freeing memory should be irrelevant.

[david.huber0]

> Are you seeing requests slow down with increasing
> times, or does it suddenly hang with a request & time
> out?

it suddenly hang with a request & time out

the time out is randomized.

Is there a max size for the web service request?
Because my application may send big files.

[jason112]

There is no limit to the size that can be uploaded.

Define 'big': 1mb, 1gb, 1tb? Can you tell if it tends to hang on a 'big' file, or does it sometimes hang on smaller ones?

[david.huber0]

the bigger files are about 20-30 MB

But sometimes the webservice hang on 1Kb files

[jason112]

Anything in monitor.log or vortex.log from around the time that the timeouts occur?

[kamal.oulhadj]

I work on the same project of David Huber
in the file monitor.log we have the following logs in indexing documents of size 20-30 MB.


200 2010-03-12 10:37:14 (3512) 2010-03-12 10:37:14 (pid 3512) in the function Timestamp
002 2010-03-12 10:38:45 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
200 2010-03-12 10:39:01 (1480) Contacting license server
006 2010-03-12 10:39:14 (1480) Timeout sending data to 127.0.0.1:16141 in the function htbuf_flush
006 2010-03-12 10:39:23 (1480) Cannot connect to ls.thunderstone.com:80: Socket timeout
000 2010-03-12 10:39:23 (1480) Could not contact license server
002 2010-03-12 10:40:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
200 2010-03-12 10:41:14 (2140) 2010-03-12 10:41:14 (pid 2140) in the function Timestamp
006 2010-03-12 10:42:11 (1480) Timeout sending data to 127.0.0.1:14864 in the function htbuf_flush
018 2010-03-12 10:42:11 (1480) Cannot terminate process 3096: Access is denied in the function TXpkill
002 2010-03-12 10:42:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
002 2010-03-12 10:44:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
002 2010-03-12 10:46:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
002 2010-03-12 10:48:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
002 2010-03-12 10:50:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
002 2010-03-12 10:52:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
200 2010-03-12 10:54:30 (1480) Contacting license server
002 2010-03-12 10:54:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl
006 2010-03-12 10:54:51 (1480) Cannot connect to ls.thunderstone.com:80: Socket timeout
000 2010-03-12 10:54:51 (1480) Could not contact license server
002 2010-03-12 10:56:46 (3364) Unable to open table D:\texis\data1\APCES\db1\SYSMETAI in the function opendbtbl