iProblem indexing page with iframes

[rkac]

We have the full version of texis and are attempting to crawl a site for a client using gw.

The web page www.edmonton.globaltv.com uses an iframe to load ads from ad.ca.doubleclick.net so to crawl this site we added an include for ad.ca.doubleclick.net.

Unfortunately, ad.ca.doubleclick.net has a robot block on it's root banning all agents. So we cannot retrieve the page to load the site even though the original site has no robot blocks.

Any suggestions?

[mark]

Since your site doesn't have robots.txt, you can just say to ignore it.

gw -r -jhttp://ad.ca.doubleclick.net/adi/www.globaltv.com http://www.edmonton.globaltv.com/