Indexing a html page

Post Reply
melvint9
Posts: 19
Joined: Wed Oct 03, 2001 11:09 am

Indexing a html page

Post by melvint9 »

Hello again,

I saved the following url to a txt (name of file: urls.txt) file "http://localhost/linkage.cfm" and tried to index the file using the following command:
(gw -fcfm -D2 "&urls.txt")and I tried (gw -D2 "&urls.txt")
but It only indexes the first page. See the .cfm file is coding to pull urls from a access database. . .when you run the script within a url address, the list of urls are displayed, listed in the database. . .Therefore I also tried creating a urls.html page with the urls, but the html file was still unable to index past the first page. The command I tried on this was (gw http://localhost/urls.html). . .and the link I had listed within the urls.html page was (http://localhost/linkage.cfm)url, since this .cfm file runs find within a url path. . .

Does anyone have a idea what I'm doing wrong.

mt
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Indexing a html page

Post by mark »

gw will stay on the site(s) given on the command line/url list file, so if the urls generated by the cfm are on any site but "localhost" (in this case) gw will not follow them. You'll have to use the local list file of raw urls if they are on different sites.

There's probably something wrong with the format of your .txt file. It should contain exactly one url per line, no blank lines, and no extra spaces at the beginning or end of line.

Turn up verbosity (-v5) to find out what gw thinks of your urls.
Post Reply