Exposing hidden projects to ht://Dig indexing

Exposing hidden projects to ht://Dig indexing

[Hans-Peter Nilsson]

Hopefully you've all figured this out, but I thought better send
a specific heads-up.

Until now, everything that could be reached from links rooted at
http://sources.redhat.com/ was indexed and nothing else.  Soon,
a find+grep-like operation will add all pages with '<meta
name="robots" content="noindex,follow">' tags, as previously
reported.

Meaning, somebody's Secret Project will risk getting exposed
now, if it has its mailing lists in the ordinary place on
sources.redhat.com, i.e. /www/sourceware/ml/ or if it has its
pages at /www/sourceware/htdocs/.  Since even this semi-private
"overseers" list was indexed (perhaps by accident), I will not
take special precautions to "hide" any projects.

The right way to "hide" them from being indexed by ht://Dig (if
so wanted), is to add a matching unique substring to the
exclude_urls attribute in sourceware.conf.  BTW, that file is
managed via CVS, please don't edit in-place.  Send patches for
/sourceware/infra/htdig-conf/sourceware.conf or requests or ask
someone else in the "sourceware" group to fix it.  For example,
an appropriate setting for excluding the overseers mailing list
would be:

exclude_urls:		${site__exclude_urls} /ecos/docs-1.0/ /ecos/docs-1.1/ \
            /ml/overseers/

Just trying to be Extra Cautious so nobody will panic and do
wrong.  But you're all sensible people.

brgds, H-P

Re: Exposing hidden projects to ht://Dig indexing

[Chris Faylor]

On Tue, Jul 25, 2000 at 08:58:44PM -0400, Hans-Peter Nilsson wrote:
>Since even this semi-private "overseers" list was indexed (perhaps by
>accident), I will not take special precautions to "hide" any projects.

I would like to vote that overseers should definitely not be archived in
a public place.

cgf

Re: Exposing hidden projects to ht://Dig indexing

[Hans-Peter Nilsson]

On Tue, 25 Jul 2000, Chris Faylor wrote:
> On Tue, Jul 25, 2000 at 08:58:44PM -0400, Hans-Peter Nilsson wrote:
> >Since even this semi-private "overseers" list was indexed (perhaps by
> >accident), I will not take special precautions to "hide" any projects.
> 
> I would like to vote that overseers should definitely not be archived in
> a public place.

Well, I can't do much about its archival (sp?), but I can take care to
exclude it from indexing and index sourceware from scratch to get rid of
it from the ht://Dig DB.  Unless someone beats me to it.

Are there other lists or projects that want out?

(Is this what they mean by o-p-t o-u-t mailing lists? :-)

brgds, H-P

Re: Exposing hidden projects to ht://Dig indexing

[Jason Molenda]

On Tue, Jul 25, 2000 at 09:23:51PM -0400, Chris Faylor wrote:
> On Tue, Jul 25, 2000 at 08:58:44PM -0400, Hans-Peter Nilsson wrote:
> >Since even this semi-private "overseers" list was indexed (perhaps by
> >accident), I will not take special precautions to "hide" any projects.
> 
> I would like to vote that overseers should definitely not be archived in
> a public place.

Do you mean that it should not be archived, access to its archives
should be restricted (e.g. requiring passwords to get at the
archives), or that we should not advertise links to the archive
publically?

I personally prefer the latter.  I don't really care about people
on the net reading these notes--all content here is either technical
(which I doubt anyone would argue should be hidden) or policy (which
I guess some could argue should be hidden to avoid repeated debates)
related.

I do want to keep this list free of random e-mails from folks on
the net.  If a user/developer on a sourceware project is having a
problem, that problem should come to the overseers list via that
project's maintainer.  The maintainer will (with luck) be clueful
enough to filter out the non-problems and FAQs, and only report
new, novel problems.

Jason

Re: Exposing hidden projects to ht://Dig indexing

[Chris Faylor]

On Tue, Jul 25, 2000 at 06:45:56PM -0700, Jason Molenda wrote:
>On Tue, Jul 25, 2000 at 09:23:51PM -0400, Chris Faylor wrote:
>> I would like to vote that overseers should definitely not be archived in
>> a public place.
>
>Do you mean that it should not be archived, access to its archives
>should be restricted (e.g. requiring passwords to get at the
>archives), or that we should not advertise links to the archive
>publically?
>
>I personally prefer the latter.  I don't really care about people
>on the net reading these notes--all content here is either technical
>(which I doubt anyone would argue should be hidden) or policy (which
>I guess some could argue should be hidden to avoid repeated debates)
>related.

The latter.

>I do want to keep this list free of random e-mails from folks on
>the net.  If a user/developer on a sourceware project is having a
>problem, that problem should come to the overseers list via that
>project's maintainer.  The maintainer will (with luck) be clueful
>enough to filter out the non-problems and FAQs, and only report
>new, novel problems.

Yup.

cgf

Re: Exposing hidden projects to ht://Dig indexing

[Hans-Peter Nilsson]

On Tue, 25 Jul 2000, Chris Faylor wrote:
> On Tue, Jul 25, 2000 at 08:58:44PM -0400, Hans-Peter Nilsson wrote:
> >Since even this semi-private "overseers" list was indexed (perhaps by
> >accident), I will not take special precautions to "hide" any projects.
> 
> I would like to vote that overseers should definitely not be archived in
> a public place.

Well, I can't do much about its archival (sp?), but I can take care to
exclude it from indexing and index sourceware from scratch to get rid of
it from the ht://Dig DB.  Unless someone beats me to it.

Are there other lists or projects that want out?

(Is this what they mean by o-p-t o-u-t mailing lists? :-)

brgds, H-P

Re: Exposing hidden projects to ht://Dig indexing

[Chris Faylor]

On Tue, Jul 25, 2000 at 08:58:44PM -0400, Hans-Peter Nilsson wrote:
>Since even this semi-private "overseers" list was indexed (perhaps by
>accident), I will not take special precautions to "hide" any projects.

I would like to vote that overseers should definitely not be archived in
a public place.

cgf

Re: Exposing hidden projects to ht://Dig indexing

[Chris Faylor]

On Tue, Jul 25, 2000 at 06:45:56PM -0700, Jason Molenda wrote:
>On Tue, Jul 25, 2000 at 09:23:51PM -0400, Chris Faylor wrote:
>> I would like to vote that overseers should definitely not be archived in
>> a public place.
>
>Do you mean that it should not be archived, access to its archives
>should be restricted (e.g. requiring passwords to get at the
>archives), or that we should not advertise links to the archive
>publically?
>
>I personally prefer the latter.  I don't really care about people
>on the net reading these notes--all content here is either technical
>(which I doubt anyone would argue should be hidden) or policy (which
>I guess some could argue should be hidden to avoid repeated debates)
>related.

The latter.

>I do want to keep this list free of random e-mails from folks on
>the net.  If a user/developer on a sourceware project is having a
>problem, that problem should come to the overseers list via that
>project's maintainer.  The maintainer will (with luck) be clueful
>enough to filter out the non-problems and FAQs, and only report
>new, novel problems.

Yup.

cgf

Exposing hidden projects to ht://Dig indexing

[Hans-Peter Nilsson]

Hopefully you've all figured this out, but I thought better send
a specific heads-up.

Until now, everything that could be reached from links rooted at
http://sources.redhat.com/ was indexed and nothing else.  Soon,
a find+grep-like operation will add all pages with '<meta
name="robots" content="noindex,follow">' tags, as previously
reported.

Meaning, somebody's Secret Project will risk getting exposed
now, if it has its mailing lists in the ordinary place on
sources.redhat.com, i.e. /www/sourceware/ml/ or if it has its
pages at /www/sourceware/htdocs/.  Since even this semi-private
"overseers" list was indexed (perhaps by accident), I will not
take special precautions to "hide" any projects.

The right way to "hide" them from being indexed by ht://Dig (if
so wanted), is to add a matching unique substring to the
exclude_urls attribute in sourceware.conf.  BTW, that file is
managed via CVS, please don't edit in-place.  Send patches for
/sourceware/infra/htdig-conf/sourceware.conf or requests or ask
someone else in the "sourceware" group to fix it.  For example,
an appropriate setting for excluding the overseers mailing list
would be:

exclude_urls:		${site__exclude_urls} /ecos/docs-1.0/ /ecos/docs-1.1/ \
            /ml/overseers/

Just trying to be Extra Cautious so nobody will panic and do
wrong.  But you're all sensible people.

brgds, H-P

Re: Exposing hidden projects to ht://Dig indexing

[Jason Molenda]

On Tue, Jul 25, 2000 at 09:23:51PM -0400, Chris Faylor wrote:
> On Tue, Jul 25, 2000 at 08:58:44PM -0400, Hans-Peter Nilsson wrote:
> >Since even this semi-private "overseers" list was indexed (perhaps by
> >accident), I will not take special precautions to "hide" any projects.
> 
> I would like to vote that overseers should definitely not be archived in
> a public place.

Do you mean that it should not be archived, access to its archives
should be restricted (e.g. requiring passwords to get at the
archives), or that we should not advertise links to the archive
publically?

I personally prefer the latter.  I don't really care about people
on the net reading these notes--all content here is either technical
(which I doubt anyone would argue should be hidden) or policy (which
I guess some could argue should be hidden to avoid repeated debates)
related.

I do want to keep this list free of random e-mails from folks on
the net.  If a user/developer on a sourceware project is having a
problem, that problem should come to the overseers list via that
project's maintainer.  The maintainer will (with luck) be clueful
enough to filter out the non-problems and FAQs, and only report
new, novel problems.

Jason