Could be the file is named wrong? The new script files are .txt. The original ones are properties=file.
Ignoring robots.txt file
Ignoring robots.txt file
The scripts should have no extension. Remove the .txt part.
Ignoring robots.txt file
Awesome!!! It is seeing robots now...
Thanks mucho!!!

Thanks mucho!!!
Ignoring robots.txt file
Is there a limit on the number of sites that can be exluded? Only 21 show up on the walk status screen and we have 30 in the robots.txt file.
Ignoring robots.txt file
No limit. You probably have a syntax error in your robots.txt. And robots.txt does not control sites. It controls access to areas of the single site where that robots.txt file resides.
Ignoring robots.txt file
Now confused. Here is the exclusion from the status. Then a search using that profile. And, the second result is in the robots listing (3rd one).
robots.txt excludes the following prefixes:
http://cleohsenet01.napa.ad.etn.com/webinator
http://cleohsenet01.napa.ad.etn.com/worldwideops
http://cleohsenet01.napa.ad.etn.com/wlc050403
http://cleohsenet01.napa.ad.etn.com/uwsurvey
http://www.etn.com/cgiexe/texis.exe/web ... estination
1: PowerPoint Presentation
of the above individuals. Please make sure that you provide the following information for the Support Analyst: Origin Destination Class Weight Ship Date and/or ...
http://cleohsenet01.napa.ad.etn.com/... ... de0439.htm 70%
Size: 9K
Depth: 10
Find Similar
Match Info
Show Parents
2: 2003 Worldwide Leadership Conference
high quality video formats, use the order form to order a DVD or VHS copy. Welcome to the Information Destination for the Worldwide Leadership Conference Thinking ...
http://cleohsenet01.napa.ad.etn.com/wlc050403/
robots.txt excludes the following prefixes:
http://cleohsenet01.napa.ad.etn.com/webinator
http://cleohsenet01.napa.ad.etn.com/worldwideops
http://cleohsenet01.napa.ad.etn.com/wlc050403
http://cleohsenet01.napa.ad.etn.com/uwsurvey
http://www.etn.com/cgiexe/texis.exe/web ... estination
1: PowerPoint Presentation
of the above individuals. Please make sure that you provide the following information for the Support Analyst: Origin Destination Class Weight Ship Date and/or ...
http://cleohsenet01.napa.ad.etn.com/... ... de0439.htm 70%
Size: 9K
Depth: 10
Find Similar
Match Info
Show Parents
2: 2003 Worldwide Leadership Conference
high quality video formats, use the order form to order a DVD or VHS copy. Welcome to the Information Destination for the Worldwide Leadership Conference Thinking ...
http://cleohsenet01.napa.ad.etn.com/wlc050403/
Ignoring robots.txt file
Hmm, are you sure you're looking at the status from a successful walk of the same profile and not an abandoned walk or a different profile than you're searching?
Are you getting any other pages from underneath wlc050403?
I assume these are internal or fake urls since none of them work from here.
Are you getting any other pages from underneath wlc050403?
I assume these are internal or fake urls since none of them work from here.
Ignoring robots.txt file
Yes, I am sure...all this was done after the new version of the software was installed. Prior, we did not see the robots.txt file in the log at all.
Yes, the url's are internal.
I had many pages of results...don't know about the other question.
Yes, the url's are internal.
I had many pages of results...don't know about the other question.
Ignoring robots.txt file
Go to list/edit urls and enter
http://cleohsenet01.napa.ad.etn.com/wlc050403*
to see if anything under there was walked.
http://cleohsenet01.napa.ad.etn.com/wlc050403*
to see if anything under there was walked.
Ignoring robots.txt file
You're not getting a message like this are you?
The following exclusion(s) specified by $n will be ignored as they would exclude the base url
Items in robots.txt or other exclusions that would exclude any of your base urls will be removed.
The following exclusion(s) specified by $n will be ignored as they would exclude the base url
Items in robots.txt or other exclusions that would exclude any of your base urls will be removed.