Not all page URL's being indexed

dietric
Posts: 100
Joined: Fri May 20, 2005 10:57 am

Not all page URL's being indexed

Post by dietric »

I have a profile with about 15 Page URL's configured. It seems that the walk indexes the first 650 or so of the first file, then stops. Any ideas what this can be caused by? There are no errors in the log.
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Not all page URL's being indexed

Post by mark »

Go to the walk status page and scroll down to just before it starts the error and dup reports. Then look up a little to see if it said it's stopping prematurely for any reason.
dietric
Posts: 100
Joined: Fri May 20, 2005 10:57 am

Not all page URL's being indexed

Post by dietric »

No, it does not, and it does correctly list all the URL's in the page URL files. It lists 674 pages fetched and 2176 duplicate pages. The total number of pages in the Page URL's is 115000.
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Not all page URL's being indexed

Post by mark »

At the top of walk status what does it say for "pages in index"?
What are your non-default settings?
dietric
Posts: 100
Joined: Fri May 20, 2005 10:57 am

Not all page URL's being indexed

Post by dietric »

At the top of walk status, pages in index is 674.
I'm attaching the settings for the profile here:


-------


SSc_rewalktype
offroadMap
new


SSc_ndbs
offroadMap
2


SSc_lastend
offroadMap
ok


SSc_mfquery
offroadMap




SSc_mfrepl
offroadMap




SSc_verbose
offroadMap
4


SSc_catname
offroadMap




SSc_caturl
offroadMap




SSc_mfwhat
offroadMap
date
date
date



SSc_allmeta
offroadMap
Y


SSc_xmlutf8
offroadMap
Y


SSc_resultstyle
offroadMap
8


SSc_maxthreads
offroadMap
5


SSc_maxparallel
offroadMap
1


SSc_schedulehr
offroadMap
18


SSc_stripquery
offroadMap
N


SS_db
offroadMap
/usr/local/morph3/texis/offroadMap.474f09e17/db2


SSc_exfieldsel
offroadMap
HTML
HTML
HTML



SSc_maxprocmem
offroadMap
100000000


SSc_excludepats
offroadMap
/cgi-bin/
/servlet/
http://www.off-road.com/rock
http://bbs.off-road.com
http://home.off-road.com
http://www.off-road.com/atv
http://www.off-road.com/rc
http://www.off-road.com/tool
http://www.off-road.com/ford
http://www.off-road.com/chevy
http://www.off-road.com/toyota
http://www.off-road.com/outdoors
http://www.off-road.com/sandsports
http://www.off-road.com/dodge
http://www.off-road.com/jeep
*/mail/emailContent.jsp*
http://chevy.off-road.com/atv*



SSc_mffieldsel
offroadMap
HTML
HTML
HTML



SS_dataspace
offroadMap
/usr/local/morph3/texis/offroadMap.474f09e17


SSc_exwhat
offroadMap
pageslinks
pageslinks
pageslinks



SSc_metatags
offroadMap
thunderstoneKeywords


SSc_discardhtmlbegin
offroadMap
<BANNER_SECTION>
<TOOLBAR_SECTION>
<WHOLEROW_SECTION>
<LEFTCOL_SECTION>
<RIGHTCOL_SECTION>
<FOOTERWHOLEROW_SECTION>
<FOOTER_SECTION>
<NOINDEX>



SSc_schedule
offroadMap
sun


SSc_xmlexportvar
offroadMap
baseUrl
siteDisplayName



SSc_attachmonitorlog
offroadMap
Y


SSc_exfieldinp
offroadMap




SSc_mffieldinp
offroadMap




SSc_attachwalkerrors
offroadMap
Y


SSc_maxdepth
offroadMap
0


SSc_attachwalklog
offroadMap
Y


SSc_attachvortexlog
offroadMap
Y


SSc_maxpgsize
offroadMap
8000000


SSc_extensions
offroadMap
.html .htm .txt .pdf .doc .xls .swf .jsp


SSc_fastcount
offroadMap
Y


SSc_tresauthdebug
offroadMap



SSc_visible
offroadMap
Y


SSc_visible
offroadMap
Y


SSc_notify
offroadMap
dietric@gmail.com


SSc_tabstractstyle
offroadMap
query


SSc_eqprefix
offroadMap
builtin


SSc_ueqprefix
offroadMap
None


SSc_excludes
offroadMap
http://www.off-road.com/rock/
http://bbs.off-road.com
http://home.off-road.com



SSc_laststart
offroadMap
2007-12-12 17:58:02


SSc_iframes
offroadMap
N


SSc_maxframes
offroadMap
0


SSc_getscripts
offroadMap
N


SSc_exquery
offroadMap




SSc_maxloadavg
offroadMap
4


SSc_querylog
offroadMap
Y


SSc_replhost
offroadMap




SSc_tresultstyle
offroadMap
8


SSc_allowxml
offroadMap
Y


SSc_replprof
offroadMap




SSc_discardhtmlend
offroadMap
</BANNER_SECTION>
</TOOLBAR_SECTION>
</WHOLEROW_SECTION>
</LEFTCOL_SECTION>
</RIGHTCOL_SECTION>
</FOOTERWHOLEROW_SECTION>
</FOOTER_SECTION>
</NOINDEX>



SSc_txslfile
offroadMap
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="html" encoding="ISO-8859-1"/>
<xsl:preserve-space elements="Abstract"/>
<!-- ====================================================================== -->
<!-- overall search results page template -->
<!-- ====================================================================== -->
<xsl:template match="ThunderstoneResults">
<HTML>
<HEAD>
<TITLE>Search: <xsl:value-of select="Query"/>
</TITLE>
</HEAD>
<BODY BGCOLOR="white">
<xsl:if test="mode = 'admin'">
<xsl:call-template name="SearchForm"/>
</xsl:if>
<xsl:apply-templates select="TopBestBets"/>
<xsl:apply-templates select="Summary"/>
<xsl:call-template name="sortBy"/>
<TABLE WIDTH="100%">
<TR>
<TD VALIGN="TOP">
<xsl:comment>RESULT LIST START</xsl:comment>
<xsl:apply-templates select="Result" mode="list"/>
<xsl:comment>RESULT LIST END</xsl:comment>
<xsl:variable name="hitsthispage" select="count(Result)"/>
<xsl:if test="$hitsthispage>5">
<xsl:apply-templates select="Summary"/>
</xsl:if>
<xsl:variable name="QueryMessage" select="QueryMessage"/>
<xsl:if test="$QueryMessage!=''">
<P>
<FONT FACE="Arial,Helvetica" SIZE="+1">
<B>
<xsl:value-of select="QueryMessage"/>&#0160;</B>
</FONT>
</P>
</xsl:if>
<xsl:apply-templates select="Spelling"/>
</TD>
<TD VALIGN="TOP">
<xsl:apply-templates select="RightBestBets"/>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>
</xsl:template>
<!-- ====================================================================== -->
<!-- search form template -->
<!-- ====================================================================== -->
<xsl:template name="SearchForm">
<xsl:if test="mode!=''">
<P>
Test mode (<xsl:value-of select="Profile"/>)
- <A>
<xsl:attribute name="HREF"><xsl:value-of select="AdminUrl"/></xsl:attribute>
<xsl:text>Back to Administration</xsl:text>
</A>
- <A>
<xsl:attribute name="HREF"><xsl:value-of select="MakeLiveUrl"/></xsl:attribute>
<xsl:text>Make this appearance live</xsl:text>
</A>
</P>
</xsl:if>
<FORM METHOD="get">
<xsl:attribute name="ACTION"><xsl:value-of select="UrlRoot"/></xsl:attribute>
<TABLE style="">
<INPUT TYPE="hidden" NAME="mode">
<xsl:attribute name="VALUE"><xsl:value-of select="mode"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="opts">
<xsl:attribute name="VALUE"><xsl:value-of select="opts"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="pr">
<xsl:attribute name="VALUE"><xsl:value-of select="Profile"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="dropXSL">
<xsl:attribute name="VALUE"><xsl:value-of select="dropXSL"/></xsl:attribute>
</INPUT>
<xsl:if test="AdvancedSearch!=1">
<INPUT TYPE="hidden" NAME="prox">
<xsl:attribute name="VALUE"><xsl:value-of select="Proximity"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="rorder">
<xsl:attribute name="VALUE"><xsl:value-of select="RankOrder"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="rprox">
<xsl:attribute name="VALUE"><xsl:value-of select="RankProximity"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="rdfreq">
<xsl:attribute name="VALUE"><xsl:value-of select="RankDatabaseFrequency"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="rwfreq">
<xsl:attribute name="VALUE"><xsl:value-of select="RankDocumentFrequency"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="rlead">
<xsl:attribute name="VALUE"><xsl:value-of select="RankPosition"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="sufs">
<xsl:attribute name="VALUE"><xsl:value-of select="Suffixes"/></xsl:attribute>
</INPUT>
<INPUT TYPE="hidden" NAME="order">
<xsl:attribute name="VALUE"><xsl:value-of select="Order"/></xsl:attribute>
</INPUT>
</xsl:if>
<TR>
<TH>
<xsl:text>Search for: </xsl:text>
</TH>
<TD NOWRAP="NOWRAP">
<INPUT SIZE="30" NAME="query">
<xsl:attribute name="VALUE"><xsl:value-of select="Query"/></xsl:attribute>
</INPUT>
<xsl:if test="Category!='NULL'">
<SELECT NAME="cq">
<xsl:for-each select="Category">
<OPTION>
<xsl:if test="CatSel='selected'">
<xsl:attribute name="selected">selected</xsl:attribute>
</xsl:if>
<xsl:attribute name="VALUE"><xsl:value-of select="CatVal"/></xsl:attribute>
<xsl:value-of select="CatName"/>
</OPTION>
</xsl:for-each>
</SELECT>
</xsl:if>
<INPUT TYPE="submit" NAME="submit" VALUE="Submit"/>
<xsl:if test="AdvancedSearch!=1">
<INPUT TYPE="submit" NAME="cmd" VALUE="Advanced"/>
</xsl:if>
<xsl:if test="AdvancedSearch=1">
<INPUT TYPE="submit" NAME="cmd" VALUE="Hide Advanced"/>
</xsl:if>
<xsl:if test="LogoutUrl!=''">
<INPUT TYPE="submit" NAME="cmd" VALUE="Logout"/>
</xsl:if>
</TD>
</TR>
<xsl:if test="AdvancedSearch!=1">
<TR>
<TD>&#0160;</TD>
<TD>
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="CHECKBOX" NAME="refine">
<xsl:attribute name="VALUE"><xsl:value-of select="Query"/></xsl:attribute>
<xsl:text>Search these answers</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
</xsl:if>
<xsl:if test="AdvancedSearch=1">
<TR>
<TH ALIGN="RIGHT" NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">Without this:</FONT>
</TH>
<TD>
<FONT FACE="Arial,Helvetica">
<INPUT SIZE="30" name="notq"/>
</FONT>
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="CHECKBOX" NAME="refine">
<xsl:attribute name="VALUE"><xsl:value-of select="Query"/></xsl:attribute>
<xsl:text>Search these answers</xsl:text>
</INPUT>
&#0160;<INPUT TYPE="CHECKBOX" NAME="order" VALUE="dd">
<xsl:if test="Order='dd'">
<xsl:attribute name="CHECKED">CHECKED</xsl:attribute>
</xsl:if>
<xsl:text>sort by date</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
</xsl:if>
</TABLE>
<xsl:if test="AdvancedSearch=1">
<TABLE WIDTH="600" BORDER="0" CELLSPACING="0" CELLPADDING="0">
<TR>
<TD VALIGN="TOP" WIDTH="30%">
<!-- helpbox -->
<TABLE BORDER="0" CELLPADDING="3" CELLSPACING="0">
<TR>
<TH NOWRAP="NOWRAP" ALIGN="LEFT">
<FONT SIZE="-1" FACE="Arial,Helvetica">SEARCH HELP</FONT>
</TH>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<A HREF="/texis/search/searchman.html?query=Applying+Search+Logic">
<B>Using logic operators</B>
</A>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<A HREF="/texis/search/searchman.html?query=Keywords+Phrases+and+Wild-cards">
<B>Phrase Matching</B>
</A>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<A HREF="/texis/search/searchman.html?query=Keywords+Phrases+and+Wild-cards">
<B>Wild Card Searches</B>
</A>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<A HREF="/texis/search/searchman.html?query=Invoking+Thesaurus+Expansion">
<B>Using the Thesaurus</B>
</A>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<A HREF="/texis/search/searchman.html?query=Using+the+Special+Pattern+Matchers">
<B>Regular Expressions</B>
</A>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<A HREF="/texis/search/searchman.html?query=Using+the+Special+Pattern+Matchers">
<B>Quantity Searching</B>
</A>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<A HREF="/texis/search/searchman.html?query=Search+Interface+Help">
<B>More Subjects...</B>
</A>
</FONT>
</TD>
</TR>
</TABLE>
</TD>
<TD VALIGN="TOP" WIDTH="30%">
<!-- proxbox -->
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="0">
<TR>
<TH NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">PROXIMITY</FONT>
</TH>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="RADIO" NAME="prox" VALUE="line">
<xsl:if test="Proximity='line'">
<xsl:attribute name="CHECKED">CHECKED</xsl:attribute>
</xsl:if>
<xsl:text>line</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="RADIO" NAME="prox" VALUE="sentence">
<xsl:if test="Proximity='sentence'">
<xsl:attribute name="CHECKED">CHECKED</xsl:attribute>
</xsl:if>
<xsl:text>sentence</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="RADIO" NAME="prox" VALUE="paragraph">
<xsl:if test="Proximity='paragraph'">
<xsl:attribute name="CHECKED">CHECKED</xsl:attribute>
</xsl:if>
<xsl:text>paragraph</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="RADIO" NAME="prox" VALUE="page">
<xsl:if test="Proximity='page'">
<xsl:attribute name="CHECKED">CHECKED</xsl:attribute>
</xsl:if>
<xsl:text>page</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
</TABLE>
<BR/>
<!-- sufbox -->
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="0">
<TR>
<TH NOWRAP="NOWRAP" ALIGN="LEFT">
<FONT SIZE="-1" FACE="Arial,Helvetica">WORD FORMS</FONT>
</TH>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="RADIO" NAME="sufs" VALUE="0">
<xsl:if test="Suffixes='0'">
<xsl:attribute name="CHECKED">CHECKED</xsl:attribute>
</xsl:if>
<xsl:text>Exact match</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="RADIO" NAME="sufs" VALUE="1">
<xsl:if test="Suffixes='1'">
<xsl:attribute name="CHECKED">CHECKED</xsl:attribute>
</xsl:if>
<xsl:text>Plural &amp; posessives</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<INPUT TYPE="RADIO" NAME="sufs" VALUE="2">
<xsl:if test="Suffixes='2'">
<xsl:attribute name="CHECKED">CHECKED</xsl:attribute>
</xsl:if>
<xsl:text>Any word forms</xsl:text>
</INPUT>
</FONT>
</TD>
</TR>
</TABLE>
</TD>
<TD VALIGN="TOP" WIDTH="30%">
<!-- wtf rankbox -->
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="1">
<TR>
<TH NOWRAP="NOWRAP" ALIGN="center" COLSPAN="2">
<FONT SIZE="-1" FACE="Arial,Helvetica">RANKING FACTORS</FONT>
</TH>
</TR>
<TR>
<TH NOWRAP="NOWRAP" ALIGN="left">
<FONT SIZE="-1" FACE="Arial,Helvetica">Rank Factor</FONT>
</TH>
<TH NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">Importance</FONT>
</TH>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">Word ordering</FONT>
</TD>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<SELECT NAME="rorder">
<OPTION VALUE="0">
<xsl:if test="RankOrder='0'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>off</xsl:text>
</OPTION>
<OPTION VALUE="250">
<xsl:if test="RankOrder='250'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>low</xsl:text>
</OPTION>
<OPTION VALUE="500">
<xsl:if test="RankOrder='500'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>medium</xsl:text>
</OPTION>
<OPTION VALUE="750">
<xsl:if test="RankOrder='750'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>high</xsl:text>
</OPTION>
<OPTION VALUE="1000">
<xsl:if test="RankOrder='1000'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>max</xsl:text>
</OPTION>
</SELECT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">Word proximity</FONT>
</TD>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<SELECT NAME="rprox">
<OPTION VALUE="0">
<xsl:if test="RankProximity='0'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>off</xsl:text>
</OPTION>
<OPTION VALUE="250">
<xsl:if test="RankProximity='250'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>low</xsl:text>
</OPTION>
<OPTION VALUE="500">
<xsl:if test="RankProximity='500'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>medium</xsl:text>
</OPTION>
<OPTION VALUE="750">
<xsl:if test="RankProximity='750'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>high</xsl:text>
</OPTION>
<OPTION VALUE="1000">
<xsl:if test="RankProximity='1000'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>max</xsl:text>
</OPTION>
</SELECT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">Database Frequency</FONT>
</TD>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<SELECT NAME="rdfreq">
<OPTION VALUE="0">
<xsl:if test="RankDatabaseFrequency='0'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>off</xsl:text>
</OPTION>
<OPTION VALUE="250">
<xsl:if test="RankDatabaseFrequency='250'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>low</xsl:text>
</OPTION>
<OPTION VALUE="500">
<xsl:if test="RankDatabaseFrequency='500'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>medium</xsl:text>
</OPTION>
<OPTION VALUE="750">
<xsl:if test="RankDatabaseFrequency='750'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>high</xsl:text>
</OPTION>
<OPTION VALUE="1000">
<xsl:if test="RankDatabaseFrequency='1000'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>max</xsl:text>
</OPTION>
</SELECT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">Document Frequency</FONT>
</TD>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<SELECT NAME="rwfreq">
<OPTION VALUE="0">
<xsl:if test="RankDocumentFrequency='0'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>off</xsl:text>
</OPTION>
<OPTION VALUE="250">
<xsl:if test="RankDocumentFrequency='250'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>low</xsl:text>
</OPTION>
<OPTION VALUE="500">
<xsl:if test="RankDocumentFrequency='500'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>medium</xsl:text>
</OPTION>
<OPTION VALUE="750">
<xsl:if test="RankDocumentFrequency='750'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>high</xsl:text>
</OPTION>
<OPTION VALUE="1000">
<xsl:if test="RankDocumentFrequency='1000'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>max</xsl:text>
</OPTION>
</SELECT>
</FONT>
</TD>
</TR>
<TR>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">Position in text</FONT>
</TD>
<TD NOWRAP="NOWRAP">
<FONT SIZE="-1" FACE="Arial,Helvetica">
<SELECT NAME="rlead">
<OPTION VALUE="0">
<xsl:if test="RankPosition='0'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>off</xsl:text>
</OPTION>
<OPTION VALUE="250">
<xsl:if test="RankPosition='250'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>low</xsl:text>
</OPTION>
<OPTION VALUE="500">
<xsl:if test="RankPosition='500'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>medium</xsl:text>
</OPTION>
<OPTION VALUE="750">
<xsl:if test="RankPosition='750'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>high</xsl:text>
</OPTION>
<OPTION VALUE="1000">
<xsl:if test="RankPosition='1000'">
<xsl:attribute name="SELECTED">SELECTED</xsl:attribute>
</xsl:if>
<xsl:text>max</xsl:text>
</OPTION>
</SELECT>
</FONT>
</TD>
</TR>
</TABLE>
</TD>
</TR>
</TABLE>
</xsl:if>
</FORM>
</xsl:template>
<!-- ====================================================================== -->
<!-- search results spell check info template -->
<!-- ====================================================================== -->
<xsl:template match="Spelling">
<TABLE>
<xsl:for-each select="SuggestWord">
<xsl:text disable-output-escaping="yes">&lt;TR&gt;</xsl:text>
<xsl:text disable-output-escaping="yes">&lt;TD&gt;</xsl:text>
<xsl:if test="position()=1">
<xsl:value-of select="SpellPhrase"/>
</xsl:if>
<xsl:text disable-output-escaping="yes">&lt;/TD&gt;</xsl:text>
<xsl:text disable-output-escaping="yes">&lt;TD&gt;</xsl:text>
<xsl:element name="a">
<xsl:attribute name="href"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="SpellLink"/><xsl:with-param name="base_url" select="../../exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
<xsl:value-of select="SpellWord"/>
<xsl:text disable-output-escaping="yes"> (</xsl:text>
<xsl:value-of select="SpellCount"/>
<xsl:text disable-output-escaping="yes">)</xsl:text>
</xsl:element>
<xsl:text disable-output-escaping="yes">&lt;/TD&gt;</xsl:text>
<xsl:text disable-output-escaping="yes">&lt;/TR&gt;</xsl:text>
</xsl:for-each>
</TABLE>
</xsl:template>
<!-- ====================================================================== -->
<!-- search results best bets info template -->
<!-- ====================================================================== -->
<xsl:template match="TopBestBets">
<span style="">
<xsl:value-of select="BBTitle"/>
</span>
<BR/>
<TABLE WIDTH="100%" BORDER="0" CELLPADDING="0" CELLSPACING="5" BGCOLOR="#ddddff">
<xsl:for-each select="BestBet">
<tr>
<TD VALIGN="TOP" NOWRAP="NOWRAP">
<FONT FACE="Arial,Helvetica">
<B>
<xsl:value-of select="BBResultNum"/>.
</B>
</FONT>
</TD>
<TD>
<xsl:element name="a">
<xsl:attribute name="href"><xsl:value-of select="BBLink"/></xsl:attribute>
<xsl:value-of select="BBResult" disable-output-escaping="yes"/>
</xsl:element>
</TD>
</tr>
<TR>
<TD/>
<TD>
<xsl:value-of select="BBDescription" disable-output-escaping="yes"/>
</TD>
</TR>
</xsl:for-each>
</TABLE>
</xsl:template>
<!-- ====================================================================== -->
<xsl:template match="RightBestBets">
<B>
<FONT FACE="Arial,Helvetica" SIZE="+1">
<xsl:value-of select="BBTitle"/>
</FONT>
</B>
<BR/>
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="5" BGCOLOR="#ddddff">
<xsl:for-each select="BestBet">
<tr>
<TD VALIGN="TOP" NOWRAP="NOWRAP">
<FONT FACE="Arial,Helvetica">
<B>
<xsl:value-of select="BBResultNum"/>.
</B>
</FONT>
</TD>
<TD>
<xsl:element name="a">
<xsl:attribute name="href"><xsl:value-of select="BBLink"/></xsl:attribute>
<xsl:value-of select="BBResult" disable-output-escaping="yes"/>
</xsl:element>
</TD>
</tr>
<TR>
<TD/>
<TD>
<xsl:value-of select="BBDescription" disable-output-escaping="yes"/>
</TD>
</TR>
</xsl:for-each>
</TABLE>
</xsl:template>
<xsl:template name="sortBy">
<table style="" width="100%">
<tr>
<td align="right">
<span class="appliance_bold">
<xsl:text>Sort by </xsl:text>
</span>
<xsl:choose>
<xsl:when test="Order='dd'">
<span class="appliance_grey_sort">
<xsl:text>date</xsl:text>
</span>
<xsl:text> | </xsl:text>
<xsl:element name="a">
<xsl:attribute name="href"><xsl:text>javascript:void();</xsl:text></xsl:attribute>
<xsl:attribute name="onclick"><xsl:text>setSearchOrder('r'); return false;</xsl:text></xsl:attribute>
<span class="appliance_bold">
<xsl:text>relevance</xsl:text>
</span>
</xsl:element>
</xsl:when>
<xsl:otherwise>
<xsl:element name="a">
<xsl:attribute name="href"><xsl:text>javascript:void();</xsl:text></xsl:attribute>
<xsl:attribute name="onclick"><xsl:text>setSearchOrder('dd'); return false;</xsl:text></xsl:attribute>
<span class="appliance_bold">
<xsl:text>date</xsl:text>
</span>
</xsl:element>
<xsl:text> | </xsl:text>
<span class="appliance_grey_sort">
<xsl:text>relevance</xsl:text>
</span>
</xsl:otherwise>
</xsl:choose>
</td>
</tr>
</table>
</xsl:template>
<!-- ====================================================================== -->
<!-- search results summary info template -->
<!-- ====================================================================== -->
<xsl:template match="Summary">
<xsl:variable name="totalHits" select="TotalNum"/>
<xsl:if test="$totalHits > 0">
<TABLE class="appliance_box" width="100%">
<TR>
<TD align="left">
<span class="appliance_bold">
<xsl:value-of select="Start"/> through <xsl:text/>
<xsl:value-of select="End"/> of <xsl:text/>
<xsl:value-of select="Total"/>
<xsl:text>matching documents</xsl:text>
<xsl:text>&#160;&#160;&#160;</xsl:text>
</span>
</TD>
</TR>
<xsl:number count="//Value[text()=.]"/>
<xsl:variable name="totalPages" select="count(Pages)"/>
<xsl:if test="$totalPages >= 2">
<TR>
<td align="right">
<xsl:text> </xsl:text>
<xsl:choose>
<xsl:when test="FirstPage='1'">
<span class="appliance_grey_prev">
<xsl:text>&lt;&lt; </xsl:text>
<xsl:text>Previous 10 | </xsl:text>
</span>
</xsl:when>
<xsl:otherwise>
<span class="appliance_summary">
<xsl:text>&lt;&lt; </xsl:text>
</span>
<xsl:element name="a">
<xsl:attribute name="HREF"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="PreviousLink"/><xsl:with-param name="base_url" select="../exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
<span class="appliance_link">
<b>
<xsl:text>Previous 10</xsl:text>
</b>
</span>
</xsl:element>
<xsl:text> | </xsl:text>
</xsl:otherwise>
</xsl:choose>
<xsl:for-each select="Pages">
<xsl:choose>
<xsl:when test="PageLink='NULL'">
<span class="appliance_summary">
<xsl:value-of select="PageNumber"/>
<xsl:text> </xsl:text>
</span>
</xsl:when>
<xsl:otherwise>
<xsl:text> </xsl:text>
<xsl:element name="a">
<xsl:attribute name="HREF"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="PageLink"/><xsl:with-param name="base_url" select="../../exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
<span class="appliance_link">
<b>
<xsl:value-of select="PageNumber"/>
</b>
</span>
</xsl:element>
<xsl:text> </xsl:text>
</xsl:otherwise>
</xsl:choose>
</xsl:for-each>
<xsl:choose>
<xsl:when test="LastPage='0'">
<xsl:text> | </xsl:text>
<xsl:element name="a">
<xsl:attribute name="HREF"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="NextLink"/><xsl:with-param name="base_url" select="../exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
<span class="appliance_link">
<b>
<xsl:text>Next 10</xsl:text>
</b>
</span>
</xsl:element>
<span class="appliance_summary">
<xsl:text> &gt;&gt;</xsl:text>
</span>
</xsl:when>
<xsl:otherwise>
<span class="appliance_grey_prev">
<xsl:text> | Next 10 </xsl:text>
<xsl:text>&gt;&gt;</xsl:text>
</span>
</xsl:otherwise>
</xsl:choose>
</td>
</TR>
</xsl:if>
</TABLE>
</xsl:if>
</xsl:template>
<!-- ====================================================================== -->
<!-- individual search result record template -->
<!-- ====================================================================== -->
<xsl:template match="Result" mode="list">
<xsl:variable name="rbb" select="/RightBestBets/BestBet/BBResultNum"/>
<TABLE WIDTH="100%">
<TR>
<TD VALIGN="TOP">
<xsl:comment>RESULT ITEM START</xsl:comment>
<span class="appliance_bold">
<xsl:value-of select="Num"/>
<xsl:text>.</xsl:text>
</span>
</TD>
<TD VALIGN="TOP" WIDTH="99%">
<xsl:variable name="encode_url">
<xsl:call-template name="build_url">
<xsl:with-param name="url" select="Url"/>
<xsl:with-param name="query" select="../Query"/>
</xsl:call-template>
</xsl:variable>
<A href="{$encode_url}">
<span class="appliance_link">
<b>
<xsl:choose>
<xsl:when test="../exportVar/variable[@name='siteDisplayName']">
<xsl:variable name="postfix">
<xsl:call-template name="remove_all">
<xsl:with-param name="all" select="ResultTitle"/>
<xsl:with-param name="substring" select="concat(../exportVar/variable[@name='siteDisplayName'], ' - ')"/>
</xsl:call-template>
</xsl:variable>
<xsl:variable name="prefix">
<xsl:call-template name="remove_all">
<xsl:with-param name="all" select="$postfix"/>
<xsl:with-param name="substring" select="concat(' - ', ../exportVar/variable[@name='siteDisplayName'])"/>
</xsl:call-template>
</xsl:variable>
<xsl:choose>
<xsl:when test="string-length(normalize-space($prefix)) = 0">
<xsl:text>(Untitled Document)</xsl:text>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select="$prefix"/>
</xsl:otherwise>
</xsl:choose>
</xsl:when>
<xsl:otherwise>
<xsl:choose>
<xsl:when test="string-length(normalize-space(ResultTitle)) = 0">
<xsl:text>(Untitled Document)</xsl:text>
</xsl:when>
<xsl:otherwise>
<xsl:copy-of select="ResultTitle"/>
</xsl:otherwise>
</xsl:choose>
</xsl:otherwise>
</xsl:choose>
</b>
</span>
</A>
<br/>
<span class="appliance_summary">
<xsl:copy-of select="Abstract"/> ...<br/>
</span>
</TD>
<TD VALIGN="TOP" NOWRAP="NOWRAP" ALIGN="RIGHT">
<span class="appliance_bold">
<xsl:text>Relevance: </xsl:text>
<xsl:variable name="PercentRank" select="PercentRank"/>
<xsl:if test="$PercentRank!=''">
<xsl:value-of select="PercentRank"/>%<xsl:text/>
</xsl:if>
</span>
<br/>
<A>
<xsl:attribute name="HREF"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="UrlSimilar"/><xsl:with-param name="base_url" select="../exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
<span class="appliance_link">
<b>
Find Similar</b>
</span>
</A>
<br/>
<A>
<xsl:attribute name="HREF"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="UrlParents"/><xsl:with-param name="base_url" select="../exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
<span class="appliance_link">
<b>Show Links</b>
</span>
</A>
<br/>
</TD>
</TR>
<tr>
<td colspan="3">
<br/>
</td>
</tr>
<tr>
<td colspan="3" class="appliance_separator"/>
</tr>
<tr>
<td colspan="3">
<br/>
<xsl:comment>RESULT ITEM END</xsl:comment>
</td>
</tr>
</TABLE>
</xsl:template>
<!-- ====================================================================== -->
<!-- overall parent links page template -->
<!-- ====================================================================== -->
<xsl:template match="ThunderstoneLinks">
<HTML>
<HEAD>
<TITLE>Search: <xsl:value-of select="Query"/>
</TITLE>
</HEAD>
<BODY BGCOLOR="white">
<xsl:if test="mode = 'admin'">
<xsl:call-template name="SearchForm"/>
</xsl:if>
<xsl:call-template name="LinksSummaryTop"/>
<xsl:apply-templates select="Result" mode="list"/>
<xsl:variable name="QueryMessage" select="QueryMessage"/>
<xsl:if test="$QueryMessage!=''">
<P>
<FONT FACE="Arial,Helvetica" SIZE="+1">
<B>
<xsl:value-of select="QueryMessage"/>&#0160;</B>
</FONT>
</P>
</xsl:if>
<xsl:call-template name="LinksSummaryBottom"/>
</BODY>
</HTML>
</xsl:template>
<!-- ====================================================================== -->
<!-- parent links top summary template -->
<!-- ====================================================================== -->
<xsl:template name="LinksSummaryTop">
<TABLE WIDTH="100%" class="appliance_box">
<TR>
<TD>
<span class="appliance_summary">
<B>Documents with links to:</B>
<BR/>
</span>
<xsl:variable name="TargetTitle" select="//TargetTitle"/>
<xsl:choose>
<xsl:when test="$TargetTitle=''">
<!-- no title, use url -->
<A>
<xsl:attribute name="HREF"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="//TargetUrl"/><xsl:with-param name="base_url" select="../exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
</A>
</xsl:when>
<xsl:otherwise>
<A>
<xsl:attribute name="HREF"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="//TargetUrl"/><xsl:with-param name="base_url" select="../exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
<span class="appliance_link">
<b>
<xsl:value-of select="//TargetTitle"/>
</b>
</span>
</A>
<BR/>
<span class="appliance_summary">
<xsl:value-of select="//TargetUrl"/>
</span>
</xsl:otherwise>
</xsl:choose>
</TD>
</TR>
</TABLE>
<br/>
</xsl:template>
<!-- ====================================================================== -->
<!-- parent links bottom summary template -->
<!-- ====================================================================== -->
<xsl:template name="LinksSummaryBottom">
<xsl:variable name="UrlMoreLinks" select="//UrlMoreLinks"/>
<xsl:if test="$UrlMoreLinks!=''">
<TABLE WIDTH="100%" class="appliance_box">
<TR>
<TD>
<span class="appliance_link">
<A>
<xsl:attribute name="HREF"><xsl:call-template name="replace_url"><xsl:with-param name="url" select="UrlMoreLinks"/><xsl:with-param name="base_url" select="exportVar/variable[@name='baseUrl']"/></xsl:call-template></xsl:attribute>
More Parent Links...
</A>
<br/>
</span>
</TD>
</TR>
</TABLE>
</xsl:if>
</xsl:template>
<xsl:template name="replace_url">
<xsl:param name="url"/>
<xsl:param name="base_url"/>
<xsl:choose>
<xsl:when test="$base_url">
<xsl:value-of select="$base_url"/>
<xsl:text>?</xsl:text>
<xsl:value-of select="substring-after($url, '?')"/>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select="$url"/>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
<xsl:template name="remove_all">
<xsl:param name="all"/>
<xsl:param name="substring"/>
<xsl:choose>
<xsl:when test="contains($all, $substring)">
<xsl:value-of select="concat(substring-before($all, $substring), substring-after($all, $substring))"/>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select="$all"/>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
<xsl:template name="build_url">
<xsl:param name="url"/>
<xsl:param name="query"/>
<xsl:value-of select="$url"/>
<xsl:choose>
<xsl:when test="contains($url, '?')">
<xsl:text>&amp;searchString=</xsl:text>
</xsl:when>
<xsl:otherwise>
<xsl:text>?searchString=</xsl:text>
</xsl:otherwise>
</xsl:choose>
<xsl:value-of select="$query"/>
</xsl:template>
</xsl:stylesheet>



SSc_fileurlformat
offroadMap
5


SSc_dupfields
offroadMap
Title
Description
Keywords
Meta



SSc_addfieldname
offroadMap




SSc_pageurl
offroadMap
http://rock.off-road.com/rock/siteMapText.do
http://atv.off-road.com/atv/siteMapText.do
http://rc.off-road.com/rc/siteMapText.do
http://tool.off-road.com/tool/siteMapText.do
http://ford.off-road.com/ford/siteMapText.do
http://chevy.off-road.com/chevy/siteMapText.do
http://toyota.off-road.com/toyota/siteMapText.do
http://outdoors.off-road.com/outdoors/siteMapText.do
http://sandsports.off-road.com/dunes/siteMapText.do
http://dodge.off-road.com/dodge/siteMapText.do
http://jeep.off-road.com/jeep/siteMapText.do
http://ih.off-road.com/ih/siteMapText.do
http://tlc.off-road.com/tlc/siteMapText.do
http://isuzu.off-road.com/isuzu/siteMapText.do
http://nissan.off-road.com/nissan/siteMapText.do
http://race.off-road.com/race/siteMapText.do
http://suzuki.off-road.com/suzuki/siteMapText.do
http://snowmobile.off-road.com/snowmobile/siteMapText.do
http://hummer.off-road.com/hummer/siteMapText.do
http://dirtbike.off-road.com/dirtbike/siteMapText.do
http://www.off-road.com/offroad/siteMapText.do



SSc_addfieldtype
offroadMap
d
d
d



SSc_enterpriseval
offroadMap
150.200.199


SSc_npages
offroadMap
674


SSc_walked
offroadMap
2007-12-12 18:17:54


SSc_nbytes
offroadMap
13709633


SSc_attachquerylog
offroadMap
Y
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Not all page URL's being indexed

Post by mark »

For all of your Ignore Tags you've set begin but not end so chances are that the entire page after any one of those tags is being ignored and that's causing most of the pages to be considered duplicates of each other and discarded.
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Not all page URL's being indexed

Post by mark »

With verbosity 4 it's easy for messages to get lost in the mess. Search through the walk status to see if you're getting stopped due to "Process memory limit exceeded". If so increase your max process memory.
dietric
Posts: 100
Joined: Fri May 20, 2005 10:57 am

Not all page URL's being indexed

Post by dietric »

I do get a Process memory limit exceeded message:

2007-12-26 15:31:10 Process memory limit exceeded (current: 700,092,416; limit: 700,000,000)

I was under the assumption that this is a non-fatal message, is that wrong? The limit is already set to "huge"...
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Not all page URL's being indexed

Post by mark »

We've found a problem with pausing and resuming in the middle of a large external url list and are working on a solution.
dietric
Posts: 100
Joined: Fri May 20, 2005 10:57 am

Not all page URL's being indexed

Post by dietric »

Mark,
do you happen to have an idea when that will be released? Indexing from the Page URL's as opposed to crawling seems to be the only option for me to keep our numerous profiles up to date.

Thanks
-ds
Post Reply