Exposing hidden projects to ht://Dig indexing

[Hans-Peter Nilsson <hp@bitrange.com>]

Hopefully you've all figured this out, but I thought better send
a specific heads-up.

Until now, everything that could be reached from links rooted at
http://sources.redhat.com/ was indexed and nothing else.  Soon,
a find+grep-like operation will add all pages with '<meta
name="robots" content="noindex,follow">' tags, as previously
reported.

Meaning, somebody's Secret Project will risk getting exposed
now, if it has its mailing lists in the ordinary place on
sources.redhat.com, i.e. /www/sourceware/ml/ or if it has its
pages at /www/sourceware/htdocs/.  Since even this semi-private
"overseers" list was indexed (perhaps by accident), I will not
take special precautions to "hide" any projects.

The right way to "hide" them from being indexed by ht://Dig (if
so wanted), is to add a matching unique substring to the
exclude_urls attribute in sourceware.conf.  BTW, that file is
managed via CVS, please don't edit in-place.  Send patches for
/sourceware/infra/htdig-conf/sourceware.conf or requests or ask
someone else in the "sourceware" group to fix it.  For example,
an appropriate setting for excluding the overseers mailing list
would be:

exclude_urls:		${site__exclude_urls} /ecos/docs-1.0/ /ecos/docs-1.1/ \
            /ml/overseers/

Just trying to be Extra Cautious so nobody will panic and do
wrong.  But you're all sensible people.

brgds, H-P