- Posts: 62
- Joined: Fri Jul 20, 2001 11:43 am
Some of the html pages from our site are not being indexed and so not coming up in search results.
Some pages are not "present" in html table but are present in "refs" table.whats going on here....
- Site Admin
- Posts: 5510
- Joined: Tue Apr 25, 2000 6:56 pm
There are *many* option settings that control what pages are indexed or not. See the manual for the long list of options and what each one does.
The refs table contains all of the urls seen on the fetched pages whether they were rejected for walking or not.
If you have specific questions about particular urls in conjunction with your particular option settings let us know.