mnogosearch experiment

mnogosearch experiment

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

Hi -

mnogosearch is back for another try:
http://sources.redhat.com/cgi-bin/search.cgi
When run in a breadth-first mode ("indexer -o"), it produces interesting
results even after just a few hundred documents have been indexed.  Before
I let it loose for a longer indexing run, /pool needs to have some more
free space.  How much of the restoration-time data there could I clean out
or at least compress?

Once it indexes a bigger fraction of sources.redhat.com, I hope to fork it
into another copy, for processing gcc.gnu.org.  I assume it is still
necessary to display two totally separate looking egos of the machine.

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: mnogosearch experiment

[Christopher Faylor]

On Mon, Feb 07, 2005 at 10:09:01PM -0500, Frank Ch. Eigler wrote:
>Hi -
>
>mnogosearch is back for another try:
>http://sources.redhat.com/cgi-bin/search.cgi
>When run in a breadth-first mode ("indexer -o"), it produces interesting
>results even after just a few hundred documents have been indexed.  Before
>I let it loose for a longer indexing run, /pool needs to have some more
>free space.  How much of the restoration-time data there could I clean out
>or at least compress?

Uh, none?  There is nowhere near enough space in /pool to handle this,
unless we want to stop backing up the cvs projects, which I don't think is
a good idea.

If we think this is a good idea then we should use the htdig partition for
mysql.

cgf

Re: mnogosearch experiment

[Angela Marie Thomas]

> Uh, none?  There is nowhere near enough space in /pool to handle this,
> unless we want to stop backing up the cvs projects, which I don't think is
> a good idea.

Don't we need to know how much space it needs before discounting it
entirely?

I don't think anyone disagrees that keeping the live cvs backups
is a good idea.  I don't think we ever decided how far back the
backups should go though.  Is the plan to keep all of them until
the disk fills up and then clean it up or do you have an automatic
process that prunes the dirs after a certain point?

--Angela

Re: mnogosearch experiment

[Christopher Faylor]

On Mon, Feb 07, 2005 at 08:03:18PM -0800, Angela Marie Thomas wrote:
>>Uh, none?  There is nowhere near enough space in /pool to handle this,
>>unless we want to stop backing up the cvs projects, which I don't think
>>is a good idea.
>
>Don't we need to know how much space it needs before discounting it
>entirely?

What's wrong with using the htdig partition?

cgf

Re: mnogosearch experiment

[Angela Marie Thomas]

> On Mon, Feb 07, 2005 at 08:03:18PM -0800, Angela Marie Thomas wrote:
> >>Uh, none?  There is nowhere near enough space in /pool to handle this,
> >>unless we want to stop backing up the cvs projects, which I don't think
> >>is a good idea.
> >
> >Don't we need to know how much space it needs before discounting it
> >entirely?
> 
> What's wrong with using the htdig partition?
> 
> cgf

My understanding from talking to you before and talking to H-P a
little is that H-P needs all of the space on the htdig partition
for htdig.  I've not heard anything to indicate that htdig is going
to be killed yet so I wouldn't think we could use that space for
something else yet.

If I missed the discussion where we agreed to kill htdig, I'm sorry.

--Angela

Re: mnogosearch experiment

[Hans-Peter Nilsson]

On Mon, 7 Feb 2005, Angela Marie Thomas wrote:
> My understanding from talking to you before and talking to H-P a
> little is that H-P needs all of the space on the htdig partition
> for htdig.

It's still 50% free for use, for the time being (until I reindex
the sourceware side too).  Oh I see it's already in progress:
/sourceware/htdig/mnogosearch

brgds, H-P

Re: mnogosearch experiment

[Christopher Faylor]

On Mon, Feb 07, 2005 at 11:10:56PM -0500, Hans-Peter Nilsson wrote:
>On Mon, 7 Feb 2005, Angela Marie Thomas wrote:
>> My understanding from talking to you before and talking to H-P a
>> little is that H-P needs all of the space on the htdig partition
>> for htdig.
>
>It's still 50% free for use, for the time being (until I reindex
>the sourceware side too).  Oh I see it's already in progress:
>/sourceware/htdig/mnogosearch

Maybe we should hold off on re-indexing sourceware until fche is
done.  If his experiment works maybe we won't need htdig.

cgf

Re: mnogosearch experiment

[Hans-Peter Nilsson]

On Mon, 7 Feb 2005, Christopher Faylor wrote:
> Maybe we should hold off on re-indexing sourceware until fche is
> done.  If his experiment works maybe we won't need htdig.

Sure, I certainly don't mind.  If you want to compare search
results, it's still there, just a re-index away.

brgds, H-P

Re: mnogosearch experiment

[Zack Weinberg]

Christopher Faylor <me@cgf.cx> writes:

> If we think this is a good idea then we should use the htdig partition for
> mysql.

Tangential, for this application sqlite might be better (faster, less
disk use).

zw

Re: mnogosearch experiment

[Angela Marie Thomas]

> I let it loose for a longer indexing run, /pool needs to have some more
> free space.  How much of the restoration-time data there could I clean out
> or at least compress?

I'm almost done extracting the bits I want to keep.

> Once it indexes a bigger fraction of sources.redhat.com, I hope to fork it
> into another copy, for processing gcc.gnu.org.  I assume it is still
> necessary to display two totally separate looking egos of the machine.

Now and forever.  Til death do us part.

--Angela