See your duplicate thread at http://thunderstone.master.com/texis/ma ... 46cae82910
I have crawled a social networking site whose all the pages (approx 10000+) gets failed, maximum errors are of offsite links. Not even a single success i got.
As far as my setting is concern my verbosity is 4
I have setted the Max frame to 20 as i was getting error "Too Many IFRAMES"
Where as some sites were easily crawled by webinator.
Please help me regarding this, How i should set my setting to get this site crawl.