Replication

michel.weber · Post by **michel.weber** » Mon Oct 30, 2006 12:43 pm

Hi

I have a couple of questions regarding replication.

1. What exactly is replicated for 'refresh' walks?
I have a walk with about 75000 documents in the index. After the refresh walk
finished there were over 100000 items in the replication queue.

2. How fast should the replication process happen?
24 Hours after the refresh finished, there are still 60000 items in the
queue. At this rate, it should take 36 more hours to empty the queue (i.e. a
total of 60 hours) which is longer than the elapsed time for the 'refresh'
rewalk (48 Hours).

3. When does the replication process start?
As soon as the 'refresh' walk starts, or when it is finished?

4. How does replication work with categories?
We use replication to build a global index from smaller ones which index each one website.
For the globl index i have defined all the categories :
Cat A http://siteA/*
Cat B http://siteB/*
Cat C http://siteC/*
For the individual indexes, do i need to define all the categories or can i do without?

Post by **mark** » Mon Oct 30, 2006 2:26 pm

1. All changed or new pages are replicated. Changed pages can have multiple queue entries.

2. Replication should be pretty quick, not slower than walking.

3. Replication starts as soon as the walk starts.

4. You should define the categories the same and in the same order on the target as on the replication source.

Check your vortex.log on the sender to see if it says anything helpful there.

dietric · Post by **dietric** » Tue Oct 31, 2006 9:38 am

I have had similar experiences as Michel regarding replication speed, especially if I have multiple profiles running replicating against the same target. Occasionally, replication will stop completely until I reboot the appliance.

michel.weber · Post by **michel.weber** » Tue Oct 31, 2006 11:59 am

Hi

There are quite a lot of errors. here's a sample from the last 12 Hours :
>20 occurences of /usr/local/morph3/texis/scripts/dowalk:6930: XML parsing error reported at byte offset 0
>20 occurences of /usr/local/morph3/texis/scripts/dowalk:7015: Timeout reading from 127.0.0.1:80 in the function htbuf_readnblk
1 occurence of /usr/local/morph3/texis/scripts/dowalk

<EXEC> command /usr/local/morph3/bin/anytotx returned exit code 1
3 occurences of /dowalk

Trying to insert duplicate value (4541bd005) in index /usr/local/morph3/texis/Internet.4535462b7/db1/xerrorid.btr

Also CPU usage on the box has been well over 90% for the last 24 hours (One walk in progress with replication + the replication that takes ages).
Something else i noticed : there are 8 apache processes running with the following command line :
16903 1 apache 0 0 7 0 S Oct18 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -DHAVE_PROXY -DHAVE_AUTH_ANON -DHAVE_ACTIONS -DHAVE_ALIAS -DHAVE_ASIS -DHAVE_AUTH -DHAVE_AUTOINDEX -DHAVE_AUTH_DB -DHAVE_AUTH_DBM -DHAVE_CERN_META -DHAVE_CGI -DHAVE_DIGEST -DHAVE_DIR -DHAVE_ENV -DHAVE_EXAMPLE -DHAVE_EXPIRES -DHAVE_HEADERS -DHAVE_IMAP -DHAVE_INCLUDE -DHAVE_INFO -DHAVE_LOG_AGENT -DHAVE_LOG_CONFIG -DHAVE_LOG_FORENSIC -DHAVE_LOG_REFERER -DHAVE_MIME -DHAVE_MIME_MAGIC -DHAVE_MMAP_STATIC -DHAVE_NEGOTIATION -DHAVE_REWRITE -DHAVE_SETENVIF -DHAVE_SPELING -DHAVE_STATUS -DHAVE_UNIQUE_ID -DHAVE_USERDIR -DHAVE_USERTRACK -DHAVE_VHOST_ALIAS -DHAVE_DAV -DHAVE_PERL -DHAVE_SSL -DHAVE_PHP4
There are none of these on the 2 other boxes.

Post by **mark** » Tue Oct 31, 2006 3:03 pm

What version of texisscripts are you using? Maintenance->tech support info

Apache is running because someone probably enabled https on the appliance.

michel.weber · Post by **michel.weber** » Wed Nov 01, 2006 2:11 am

Version: Search Appliance Server Version 5.01.1159557523 20060929 (i686-unknown-linux2.4.9-64-32)
Scripts Version: 6.1.4
Details: dowalk: 6.1.4/2.458 dowalk: 6.1.4/2.372 appliance: 6.1.4/1.198 search: 6.1.4/2.299 DB: /1.5

Post by **mark** » Wed Nov 01, 2006 1:06 pm

Those line numbers don't make sense for scripts 6.1.4. You must have been running a different version when you got those (or have some process that's still running since before you updated).

I'd expect things to run slowly if the cpu is continuously at 90% usage. Go to maintenance->tech support info to see what process is hogging the cpu. The quickest solution might be to reboot.

michel.weber · Post by **michel.weber** » Wed Nov 01, 2006 2:38 pm

Mark

I checked on another of our boxes. The numbers were exactly the same.

I rebooted it ... but the numbers are still the same.

Post by **mark** » Wed Nov 01, 2006 5:30 pm

You're still getting those messages since the reboot?
Open a tech support ticket so we can get your full info.
Attach the maintenance->tech support info page and the recent lines from vortex.log and monitor.log.

michel.weber · Post by **michel.weber** » Wed Nov 01, 2006 5:32 pm

Mark

In the meantime i also rebooted this box...unfortunately there is no change.

It seems to be the replication processes that consume a lot of CPU.

Each uses approx 30%, but they don't seem to do very much. The replication queue count clicks down about every 10-20 seconds.

The 2 profiles that are being replicated are assembly and www.

I don't know if you have the possibility to connect to the box from the outside as it is sitting on the internet...
The url is http://www.search.coe.int/