David Woolley (email@example.com)
Tue, 27 Jul 1999 08:49:12 +0100 (BST)
> That's correct. Altavista and the other large search engines run automated
> robotic index programs that scour the net. Any publicly exposed web pages
> will sooner or later be indexed by these "Bots" unless the META tag of the
> HTML page is set to deflect robot indexing.
Unless you do it the old way, using robots.txt, you can't rely on all
spiders obeying - actually, I don't fully understand why MS invented
this particular META tag, except possibly for people with only access
to part of a site (robots.txt has to be at the root level).
by it; not that they would want to follow links to themselves.
Also, in some contexts, I think it is possible to use pure HTML and some
search engines may encourage this (others may consider it theft of service
to bypass their home page, so check first).
This archive was generated by hypermail 2.0b3 on Sun Aug 01 1999 - 16:28:47 PDT