Follow links

[tgy]

We are planning to buy Texis if webinator satisfy the needs we have. Now there is one problem which cause i can't put my finger on.
On some of our servers it wont follow links. First i thougt it was related to virtualhosts, but that does not seem to be tha case.

There are two differen sites on the same server:


bin/gw -z300000 -y -d./test http://hem.passagen.se/hemsideindex/a.html

200 Database created
Getting http://195.163.107.12/robots.txt...Not there...Ok.
Adding todo: http://hem.passagen.se/hemsideindex/a.html
http://hem.passagen.se/hemsideindex/a.html
0/0
Visited 1 pages total
Indexing new pages


Does not follow any of its links !




An other site on the same server:
bin/gw -z300000 -y -d./test http://www.scandinaviaonline.com/index2.shtml

200 Database created
Getting http://195.163.102.144/robots.txt...Not there...Ok.
Adding todo: http://www.scandinaviaonline.com/index2.shtml
http://www.scandinaviaonline.com/index2.shtml
1/61
http://www.scandinaviaonline.com/financ ... ndar.shtml
2/125
http://www.scandinaviaonline.com/financialreports.shtml
3/208
http://www.scandinaviaonline.com/images.shtml
4/284
http://www.scandinaviaonline.com/network.shtml
4/284


... Works OK


I've checked the logfiles, but it i see nothing strange there.
Help Please!

[bart]

I did the same thing but did not replicate your problem. Could it be a reverse DNS issue? Try it with an IP address instead of the name too..

Here's what I got:

gw -z300000 -y -d. http://hem.passagen.se/hemsideindex/a.html
Getting http://195.163.107.13/robots.txt...Not there...Ok.
Adding todo: http://hem.passagen.se/hemsideindex/a.html
http://hem.passagen.se/hemsideindex/a.html
1/5294
http://hem.passagen.se/a-b.o/
2/5308
http://hem.passagen.se/a-cwinkler/
3/5327
http://hem.passagen.se/a-dekor/
4/5339
http://hem.passagen.se/a-geek/
5/5356
http://hem.passagen.se/a-jaur/
6/5368
http://hem.passagen.se/a-jay/
7/5383
http://hem.passagen.se/a-k7/
8/5403
http://hem.passagen.se/a-lh/
9/5414
http://hem.passagen.se/a-mt/
9/5414
000 Got signal 2 - will attempt to quit nicely
10/5427
Visited 10 pages total

[tgy]

OK, after some tests i found out that is was an IFRAME problem. Same as:

http://thunderstone.master.com/texis/ma ... 3acb652f11


worked with:
bin/gw -d/home/tobbe/hem -r -jhttp://ad.se.doubleclick.net/adi/ http://hem.passagen.se/hemsideindex/a.html


-Tobbe

[bart]

I-frames are a pain. The next release contains substantial improvements in handling these.

Thanks for your effort to resolve this.