I am looking for suggestions on how to get profile replication to work faster. I have two servers. I am configuring server1 as the "sender" which will run multiple profile walks and replicate to a primary "receiver" profile on itself and server2. The sender profiles are configured to replicated to localhost (for local receiver) and a private IP configured on eth1 (for the receiver on second server).
Initial testing with a smaller site (~2,500 URLs) replicated very quickly. However, I am now walking a much larger site. The replication queue was keeping pace with the walk until it got up around 20,000 documents. The replication queue has now started to grow. My current status looks like this:
51,457 pages in todo
45,995 pages scheduled to be refreshed
11,934 pages visited in the last hour (9,251 success/2,683 failed)
52,858 pages in index
---------------------------------------------------
74,238 items in replication queue.
Is there any way to make this replicate faster? I have the latest versions of software, verbosity on the profile is set to 0, and the load average is low (server1 = 1.32, 1.30, 1.34, server2 = 0.00, 0.02, 0.00).
Initial testing with a smaller site (~2,500 URLs) replicated very quickly. However, I am now walking a much larger site. The replication queue was keeping pace with the walk until it got up around 20,000 documents. The replication queue has now started to grow. My current status looks like this:
51,457 pages in todo
45,995 pages scheduled to be refreshed
11,934 pages visited in the last hour (9,251 success/2,683 failed)
52,858 pages in index
---------------------------------------------------
74,238 items in replication queue.
Is there any way to make this replicate faster? I have the latest versions of software, verbosity on the profile is set to 0, and the load average is low (server1 = 1.32, 1.30, 1.34, server2 = 0.00, 0.02, 0.00).
