Can you run multiple gw's?

Post by **Thunderstone** » Mon Jan 06, 1997 3:15 pm

Can you run multiple instances of gw on NT in separate DOS windows? I'm
using the -j option to point gw at specific directory trees, and have to
run it one url at a time in long batch files. I'd like to open up a couple
of windows to process more urls in less time. Can you do this or will
webinator get confused?

______________________________________________________________
Carl Dickson cdickson@govsolutions.com
Government Solutions Phone: 703-847-3601
http://www.govsolutions.com Fax: 703-760-7899
8000 Towers Crescent Dr., Suite 1350, Vienna, VA 22182

Post by **Thunderstone** » Mon Jan 06, 1997 3:41 pm

You can run multiple instances of the Webinator on independent databases
without any issue. However, running multiple gw instances against the same
DB can be troublesome if the gw's are indexing different sites.

When we conduct a world walk we generally run about 25-50 concurrent
gw invocations, but the management of the todo table is performed by
a master program and not by the individual gw programs themselves.

Its a complex issue, and probably beyond the scope of this listserve
group. You should be very careful not to consume all your available
bandwidth when experimenting with this. And you should be _very_ careful
not to abuse someone elses web site. Multiple gw programs have the ability
to overtax a webserver into extinction.

Thunderstone

Post by **Thunderstone** » Mon Jan 06, 1997 3:55 pm

Currently I run webinator with Purveyor Encrypt 1.2b. The scripts folder is
in the data folder which contains the web content. Webinator folder is also
in the data folder. I am having trouble loading webinator. Just by typing
the "URL\scripts\webinator.exe" I get this error message "URL document not
found" If I use a seach form that has all the swithces and opitons the
search for returns information but it only start at matching record #8 and
when I go to the last pages, for example clicking on the link that says
records 21-28 I get the same error URL not found.
If I click on the "help" button on the seach form I get this message url
not found.
Could you give me some pointers to what is goin on?
_____________________________
Jeno Mozes (CJ)
Mozes Cleveland & Co.
phone: 216/468.6600
fax: 216/468.6605

Post by **Thunderstone** » Mon Jan 06, 1997 4:07 pm

You can run multiple gw's if they are working on the same task or are
using separate databases. But you have to be careful not to kill the web
server by running too many walkers against it.

If you are using a single database and walking multiple sites or with
variant restrictions, you will not be able run multiples.
You can, however, specify the -j option multiple times to allow several
prefixes at once.

Post by **Thunderstone** » Mon Jan 06, 1997 4:45 pm

That sounds perfect. What I had in mind was no more than 5-6 instances,
each going to a separate database, and each walking different servers. The
idea isn't to walk any single server faster, but rather to complete the
walking of the entire set of severs faster by doing more than one at a time.

Is 5-6 simultaneous sessions on a p133 with 32MB of RAM on a T1 realistic?

______________________________________________________________
Carl Dickson cdickson@govsolutions.com
Government Solutions Phone: 703-847-3601
http://www.govsolutions.com Fax: 703-760-7899
8000 Towers Crescent Dr., Suite 1350, Vienna, VA 22182

Post by **Thunderstone** » Mon Jan 06, 1997 5:28 pm

It depends on how much FREE ram there is. Other apps take away from
available ram. I would stick to 2 or 3 sessions in your case. You can fuss
with it, adding one at a time till performance gets bad then back off.

Post by **Thunderstone** » Mon Jan 06, 1997 5:35 pm

Try forward slashes ( use / NOT \ ).
What client browser are you using?
How does the URL on the search form compare to the one you type in?
Can you supply a real URL we can look at?