[mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Search engine capability question]

[mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Search engine capability question]

[Christopher Faylor]

Does anyone know the answer to this question?

cgf

----- Forwarded message from Mike Davies <mike.davies@epid.eurotherm.co.uk> -----

From: "Mike Davies" <mike.davies@epid.eurotherm.co.uk>
To: <postmaster@sourceware.cygnus.com>
Subject: [postmaster@sourceware] Search engine capability question
Date: Tue, 31 Oct 2000 14:01:15 -0000

Hi,

Suhaib Siddiqi on the cygwin-xfree mailing list suggested I post this
question here :

I had a small problem trying to compile an application under cygwin-xfree.
I searched on the cygwin-xfree mailing archive for the keyword
"XpmReadFileToPixmap"  (the quotes weren't part of the search key) and got a
result that said that " No matches were found for 'xpmreadfiletopixmap' ".

I searched again,  with the same result,  then I searched for "pixmap" and
in the results I saw a message dated 15 Jul 2000 "Re: libxaw" which contains
the word XpmReadFileToPixmap !   Does the search engine have some kind of
character count limitation ?


Cordially,

Mike



----- End forwarded message -----

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Searchengine capability question]

[Hans-Peter Nilsson]

On Tue, 31 Oct 2000, Christopher Faylor wrote:
> Does anyone know the answer to this question?

Yes. :-)

> ----- Forwarded message from Mike Davies <mike.davies@epid.eurotherm.co.uk> -----
> [searching for "XpmReadFileToPixmap"]

> I searched again,  with the same result,  then I searched for "pixmap" and
> in the results I saw a message dated 15 Jul 2000 "Re: libxaw" which contains
> the word XpmReadFileToPixmap !   Does the search engine have some kind of
> character count limitation ?

There is a 12-characters default limitation.  To fix, we just have to "up"
the settings in infra/htdig-conf/sourceware.conf and re-index from
scratch.

There will be a size increase of the database, though, but I suggest we
give it a try since it seems to be expected for reasonable usability. :-)

Note that gcc.conf has the limit set to 80 characters.  I suggest rasing
it for sourceware too, unless the DB size doubles or something.

I'll do it if someone says "go ahead", but I think I need, that since the
sourceware part of things is normally not my toy and I'm not sure how much
the DB will increase (guesstimate 20-30%).

brgds, H-P
PS.  About time to check how htdig 3.2 got around stabilitywise and
such -- there were promising number for compression at the database
level last time I sneaked a peek many months ago...  No time for that yet,
though.
DS.

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Search engine capability question]

[Christopher Faylor]

Well, *I* think it's a good idea.  There should be sufficient space on sources
to accomodate this, right?  If not, then we can always go back, can't we?

cgf

On Wed, Nov 01, 2000 at 12:10:56AM -0500, Hans-Peter Nilsson wrote:
>On Tue, 31 Oct 2000, Christopher Faylor wrote:
>> Does anyone know the answer to this question?
>
>Yes. :-)
>
>> ----- Forwarded message from Mike Davies <mike.davies@epid.eurotherm.co.uk> -----
>> [searching for "XpmReadFileToPixmap"]
>
>> I searched again,  with the same result,  then I searched for "pixmap" and
>> in the results I saw a message dated 15 Jul 2000 "Re: libxaw" which contains
>> the word XpmReadFileToPixmap !   Does the search engine have some kind of
>> character count limitation ?
>
>There is a 12-characters default limitation.  To fix, we just have to "up"
>the settings in infra/htdig-conf/sourceware.conf and re-index from
>scratch.
>
>There will be a size increase of the database, though, but I suggest we
>give it a try since it seems to be expected for reasonable usability. :-)
>
>Note that gcc.conf has the limit set to 80 characters.  I suggest rasing
>it for sourceware too, unless the DB size doubles or something.
>
>I'll do it if someone says "go ahead", but I think I need, that since the
>sourceware part of things is normally not my toy and I'm not sure how much
>the DB will increase (guesstimate 20-30%).
>
>brgds, H-P
>PS.  About time to check how htdig 3.2 got around stabilitywise and
>such -- there were promising number for compression at the database
>level last time I sneaked a peek many months ago...  No time for that yet,
>though.
>DS.

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Searchengine capability question]

[Hans-Peter Nilsson]

On Wed, 1 Nov 2000, Christopher Faylor wrote:
> Well, *I* think it's a good idea.

Ok, will do.

>  There should be sufficient space on sources
> to accomodate this, right?  If not, then we can always go back, can't we?

Yes.

brgds, H-P

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Search engine capability question]

[Jason Molenda]

On Wed, Nov 01, 2000 at 12:13:08AM -0500, Christopher Faylor wrote:
> Well, *I* think it's a good idea.  There should be sufficient space on sources
> to accomodate this, right?  If not, then we can always go back, can't we?


I agree that this is a good idea, but I thought I'd interject that
this partition (/sourceware/htdig) has grown at a faster rate than
I expected, so an eye needs to be kept on it.

I set up the current sourceware box with the intention of lasting
about two years of growth - after one year on the new system, the
htdig and the web doc partitions are more full than I anticipated
they would be.  IMHO the system is going to need to be upgraded
with more disk space in the late winter or spring timeframe.

The htdig drive is deceptive in its free space; the sourceware
database is around 600MB, but you need 3x the size of the database
to update it, so you need another 1.2-1.5GB of free disk space.
The GCC side is about the same order of magnitude.

If the htdig drive fills up, you get a corrupt htdig database and
searches start returning no-matches - it can be a while before it
is noticed and reported to the sourcemaster addr.  IIRC Hans-Peter
has set up a clever little automated tester for the gcc side so
that (someone) will be notified when trivial searches start returning
no matches; the sourceware side will silently fail.


In case anyone happens to notice, yes, the CVS partition has a
whole lot of extra space.  I did this on purpose so that we'd never
run out of space on that partition, and to keep the I/O performance
of that drive dedicated exclusively to cvs.  If htdig databases
were put on there, you'd make cvs operations compete with web
searches for disk I/O and it'd bite.


Jason

(incidentally, the ftp partition appears much more full than I had
anticipated, but that's due almost entirely to people being lazy
about migrating snapshots to exclusively using .bz2 format or
cleaning up old snapshots - there's a lot of slop on that partition)

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Searchengine capability question]

[Hans-Peter Nilsson]

On Tue, 31 Oct 2000, Jason Molenda wrote:
> On Wed, Nov 01, 2000 at 12:13:08AM -0500, Christopher Faylor wrote:
> > [incr maximum_word_length]
> 
> I agree that this is a good idea, but I thought I'd interject that
> this partition (/sourceware/htdig) has grown at a faster rate than
> I expected, so an eye needs to be kept on it.

Every now and then (frequency: months) I check on it.
Perhaps htdig 3.2 will improve things (having put compression in its
proper place), have to check it some day.

> If the htdig drive fills up, you get a corrupt htdig database and
> searches start returning no-matches - it can be a while before it
> is noticed and reported to the sourcemaster addr.  IIRC Hans-Peter
> has set up a clever little automated tester for the gcc side so
> that (someone) will be notified when trivial searches start returning
> no matches; the sourceware side will silently fail.

Not entirely C R about the gcc tester (but it's a good idea); it will
re-try once, and the logs are moved so they will not be overwritten the
next successful run.  Last time this happened was 2000-10-22; there's no
indication why, though, but I don't suspect "disk full".  On 2000-08-10 it
failed because of "/bin/sort: read error: Input/output error".  Jeff Law
said he'd asked someone to check it physically.

brgds, H-P
The update on sourceware is in progress.  I'll babysit it.

Disk full: Updating maximum_word_length for sourceware (was: Re:[forward-gunk deleted])

[Hans-Peter Nilsson]

On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> On Wed, 1 Nov 2000, Christopher Faylor wrote:
> > Well, *I* think it's a good idea.
> 
> Ok, will do.

I think I will have to revert this change.  File usage grew by 28%/300M,
but there's actually no room for that, despite the appearance that there's
2G left.

Normal df on the htdig volume:
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sdd3              5932328   3165831   2459254  56% /sourceware/htdig

Peak usage during update (within ~1% and 30s), df:
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sdd3              5932328   5392444    232641  96% /sourceware/htdig

Before the update, ls -l (sorry, no du -sk):
-rw-r--r--   1 htdigid  htdig    304411648 Oct 31 05:36 db.docdb
-rw-r--r--   1 htdigid  htdig    15186944 Oct 31 05:36 db.docs.index
-rw-rw-r--   1 htdigid  htdig    14838784 Oct 31 05:40 db.metaphone.db
-rw-rw-r--   1 htdigid  htdig    12578816 Oct 31 05:39 db.soundex.db
-rw-r--r--   1 htdigid  htdig    400643440 Oct 31 05:26 db.wordlist
-rw-r--r--   1 htdigid  htdig    328303616 Oct 31 05:26 db.words.db
-rw-r--r--   1 htdigid  htdig       15315 Oct 31 04:37 noindex-follow-urls

After the update:
-rw-r--r--   1 htdigid  htdig    302025728 Nov  1 07:00 db.docdb
-rw-r--r--   1 htdigid  htdig    15298560 Nov  1 07:00 db.docs.index
-rw-rw-r--   1 htdigid  htdig    24295424 Nov  1 07:05 db.metaphone.db
-rw-rw-r--   1 htdigid  htdig    12578816 Oct 31 05:39 db.soundex.db
-rw-r--r--   1 htdigid  htdig    564916340 Nov  1 06:37 db.wordlist
-rw-r--r--   1 htdigid  htdig    459653120 Nov  1 06:37 db.words.db
-rw-r--r--   1 htdigid  htdig       15971 Nov  1 04:37 noindex-follow-urls

So with the next update, there will not be any 200M left...

Now, usage between about 74% (normal "doubling" of the DB at update time)
and 96% is due to 1.2G of "sort" files that lives for a few minutes.
Those files can go elsewhere (the default is normally TMPDIR), if there's
spare room somewhere.  Candidates volumes are:

/dev/sdb2              8442207   1263228   7178979  15% /sourceware/projects
/dev/sdc4              2524750    121028   2273194   5% /sourceware/libre

If there's no solution within 12 hours, I will revert, hoping to get back
the 300M and that the DB:s for gcc and sourceware does not grow more than
200M before there's a new disk or a new htdig.  Note that even with a
new htdig we would have to save the old DB somewhere while diddling to get
the installation into shape, and there are no figures on what the new disk
needs would be.

So, IMHO:
- spare some bytes on your volume, sir, please; very temporary solution.
- a new disk is needed; safest route; but it's Not My Money.
- update htdig; if:s and but:s (and New Software).  Unfortunately no time.

brgds, H-P

Re: Disk full: Updating maximum_word_length for sourceware (was: Re: [forward-gunk deleted])

[Jeffrey A Law]

  In message < Pine.BSF.4.10.10011011015420.18674-100000@dair.pair.com >you write
:
  > So, IMHO:
  > - spare some bytes on your volume, sir, please; very temporary solution.
  > - a new disk is needed; safest route; but it's Not My Money.
  > - update htdig; if:s and but:s (and New Software).  Unfortunately no time.
We're in the process of trying to get another box (with bigger disks of
course) as part of getting the machine moved to a colocation site.

So I suspect we'll have more disk space in the relatively near future, but
it's not going to show up immediately.

jeff

Re: Disk full: Updating maximum_word_length for sourceware (was:Re: [forward-gunk deleted])

[Hans-Peter Nilsson]

On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> > On Wed, 1 Nov 2000, Christopher Faylor wrote:
> > > Well, *I* think it's a good idea.
> > 
> > Ok, will do.
> 
> I think I will have to revert this change.

Done.  Re-index from scratch in progress.  Hopefully finished in 2.5-3h.
I probably need to go to sleep before that.  Perhaps someone can check the
report in /sourceware/htdig/sourceware/update-log and check that searches
work reasonably, so at least people know if it's broken or not if/when
complaints start flooding.  (Think happy thoughts.)

brgds, H-P

Re: Disk full: Updating maximum_word_length for sourceware

[Hans-Peter Nilsson]

On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> > On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> > > On Wed, 1 Nov 2000, Christopher Faylor wrote:
> > > > Well, *I* think it's a good idea.
> > > 
> > > Ok, will do.
> > 
> > I think I will have to revert this change.
> 
> Done.  Re-index from scratch in progress.  Hopefully finished in 2.5-3h.

The saga continues...  Someone didn't think happy thoughts.  I guess it
was me.  The actual increase in word length wasn't responsible for the
300M increase; it did only about 27M.  The mere act of re-indexing did it.  
Perhaps the reindex-from-scratch pulled in a few new projects that weren't
indexed before, but I can't explain how that can happen with files/pages
that have sane time-stamps.  Perhaps I've done something wrong but I don't
know what.

I'm curious how today's update survived.  Perhaps htmerge (or sort itself)
somehow did not notice that it ran out of disk space.  It doesn't look
like sort adjusts its usage to the amount of disk space left.

I noticed that we copy db.wordlist like the other DB files while updating.
While this is necessary for the other DB files to keep the search function
working while the update is in progress, htsearch does not need
db.wordlist; it is only needed for future updates.  So it can just be
renamed, which would win about 550M.  Maybe this will save us for a week
or two.

Third time's...  I'm going to change maximum_word_length up to 80 again,
and change the update scripts for gcc and sourceware handling db.wordlist 
as above.

Next panic action would be to separate the gcc and sourceware
htdig-updates with one or two hours more, and bzip2 db.wordlist between
runs. :-P

Sorry for all the diddling.
brgds, H-P

Re: Disk full: Updating maximum_word_length for sourceware

[Hans-Peter Nilsson]

To top everything off, there was 500M of (deliberately copied) *.failed DB
files in the gcc DB dir.  So perhaps sourceware is several months away
from running out of disk.

brgds, H-P

Re: Disk full: Updating maximum_word_length for sourceware

[Jason Molenda]

Thanks for all the great help, Hans-Peter - I really appreciate
it, and I'm sure the maintainers at Red Hat also appreciate it even
more as they've had little direct exposure to htdig so far. :-)

On Thu, Nov 02, 2000 at 08:12:30PM -0500, Hans-Peter Nilsson wrote:

> Third time's...  I'm going to change maximum_word_length up to 80 again,
> and change the update scripts for gcc and sourceware handling db.wordlist 
> as above.

Great, great.  I copied the db.wordlist around without knowing what
it was, I'm glad to hear that it isn't necessary to make an extra
copy of it during an update.


> Next panic action would be to separate the gcc and sourceware
> htdig-updates with one or two hours more, and bzip2 db.wordlist between
> runs. :-P


Sounds good.  The accesses to sourceware taper off at around 19h00
local time every day - the indexing of the earlier one can be moved
forward without causing any havoc.  The early morning (5h00 - 7h00)
is when traffic to the site picks up; it's a combination of evening
in Europe, late evening in Asia, and start-of-the-day in the Eastern
USA.  It's best to have all heavy-duty cronjobs over before around 5am.
The early morning is always the heaviest load on the system.


If we really start scraping for space, we could set TMPDIR to one of
the other partitions - even if we did it on the /sourceware/projects (cvs
repositories) partition, it wouldn't be that terrible if it happened in
the middle of the night.


Jason

Re: Disk full: Updating maximum_word_length for sourceware

[Christopher Faylor]

On Thu, Nov 02, 2000 at 05:59:59PM -0800, Jason Molenda wrote:
>Thanks for all the great help, Hans-Peter - I really appreciate
>it, and I'm sure the maintainers at Red Hat also appreciate it even
>more as they've had little direct exposure to htdig so far. :-)

I know that I appreciate this very much!

cgf

Re: Disk full: Updating maximum_word_length for sourceware

[Christopher Faylor]

On Thu, Nov 02, 2000 at 05:59:59PM -0800, Jason Molenda wrote:
>Thanks for all the great help, Hans-Peter - I really appreciate
>it, and I'm sure the maintainers at Red Hat also appreciate it even
>more as they've had little direct exposure to htdig so far. :-)

I know that I appreciate this very much!

cgf

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Search engine capability question]

[Jason Molenda]

On Wed, Nov 01, 2000 at 12:13:08AM -0500, Christopher Faylor wrote:
> Well, *I* think it's a good idea.  There should be sufficient space on sources
> to accomodate this, right?  If not, then we can always go back, can't we?


I agree that this is a good idea, but I thought I'd interject that
this partition (/sourceware/htdig) has grown at a faster rate than
I expected, so an eye needs to be kept on it.

I set up the current sourceware box with the intention of lasting
about two years of growth - after one year on the new system, the
htdig and the web doc partitions are more full than I anticipated
they would be.  IMHO the system is going to need to be upgraded
with more disk space in the late winter or spring timeframe.

The htdig drive is deceptive in its free space; the sourceware
database is around 600MB, but you need 3x the size of the database
to update it, so you need another 1.2-1.5GB of free disk space.
The GCC side is about the same order of magnitude.

If the htdig drive fills up, you get a corrupt htdig database and
searches start returning no-matches - it can be a while before it
is noticed and reported to the sourcemaster addr.  IIRC Hans-Peter
has set up a clever little automated tester for the gcc side so
that (someone) will be notified when trivial searches start returning
no matches; the sourceware side will silently fail.


In case anyone happens to notice, yes, the CVS partition has a
whole lot of extra space.  I did this on purpose so that we'd never
run out of space on that partition, and to keep the I/O performance
of that drive dedicated exclusively to cvs.  If htdig databases
were put on there, you'd make cvs operations compete with web
searches for disk I/O and it'd bite.


Jason

(incidentally, the ftp partition appears much more full than I had
anticipated, but that's due almost entirely to people being lazy
about migrating snapshots to exclusively using .bz2 format or
cleaning up old snapshots - there's a lot of slop on that partition)

Re: Disk full: Updating maximum_word_length for sourceware

[Hans-Peter Nilsson]

On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> > On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> > > On Wed, 1 Nov 2000, Christopher Faylor wrote:
> > > > Well, *I* think it's a good idea.
> > > 
> > > Ok, will do.
> > 
> > I think I will have to revert this change.
> 
> Done.  Re-index from scratch in progress.  Hopefully finished in 2.5-3h.

The saga continues...  Someone didn't think happy thoughts.  I guess it
was me.  The actual increase in word length wasn't responsible for the
300M increase; it did only about 27M.  The mere act of re-indexing did it.  
Perhaps the reindex-from-scratch pulled in a few new projects that weren't
indexed before, but I can't explain how that can happen with files/pages
that have sane time-stamps.  Perhaps I've done something wrong but I don't
know what.

I'm curious how today's update survived.  Perhaps htmerge (or sort itself)
somehow did not notice that it ran out of disk space.  It doesn't look
like sort adjusts its usage to the amount of disk space left.

I noticed that we copy db.wordlist like the other DB files while updating.
While this is necessary for the other DB files to keep the search function
working while the update is in progress, htsearch does not need
db.wordlist; it is only needed for future updates.  So it can just be
renamed, which would win about 550M.  Maybe this will save us for a week
or two.

Third time's...  I'm going to change maximum_word_length up to 80 again,
and change the update scripts for gcc and sourceware handling db.wordlist 
as above.

Next panic action would be to separate the gcc and sourceware
htdig-updates with one or two hours more, and bzip2 db.wordlist between
runs. :-P

Sorry for all the diddling.
brgds, H-P

[mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Search engine capability question]

[Christopher Faylor]

Does anyone know the answer to this question?

cgf

----- Forwarded message from Mike Davies <mike.davies@epid.eurotherm.co.uk> -----

From: "Mike Davies" <mike.davies@epid.eurotherm.co.uk>
To: <postmaster@sourceware.cygnus.com>
Subject: [postmaster@sourceware] Search engine capability question
Date: Tue, 31 Oct 2000 14:01:15 -0000

Hi,

Suhaib Siddiqi on the cygwin-xfree mailing list suggested I post this
question here :

I had a small problem trying to compile an application under cygwin-xfree.
I searched on the cygwin-xfree mailing archive for the keyword
"XpmReadFileToPixmap"  (the quotes weren't part of the search key) and got a
result that said that " No matches were found for 'xpmreadfiletopixmap' ".

I searched again,  with the same result,  then I searched for "pixmap" and
in the results I saw a message dated 15 Jul 2000 "Re: libxaw" which contains
the word XpmReadFileToPixmap !   Does the search engine have some kind of
character count limitation ?


Cordially,

Mike



----- End forwarded message -----

Re: Disk full: Updating maximum_word_length for sourceware (was: Re: [forward-gunk deleted])

[Jeffrey A Law]

  In message < Pine.BSF.4.10.10011011015420.18674-100000@dair.pair.com >you write
:
  > So, IMHO:
  > - spare some bytes on your volume, sir, please; very temporary solution.
  > - a new disk is needed; safest route; but it's Not My Money.
  > - update htdig; if:s and but:s (and New Software).  Unfortunately no time.
We're in the process of trying to get another box (with bigger disks of
course) as part of getting the machine moved to a colocation site.

So I suspect we'll have more disk space in the relatively near future, but
it's not going to show up immediately.

jeff

Re: Disk full: Updating maximum_word_length for sourceware

[Hans-Peter Nilsson]

To top everything off, there was 500M of (deliberately copied) *.failed DB
files in the gcc DB dir.  So perhaps sourceware is several months away
from running out of disk.

brgds, H-P

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Searchengine capability question]

[Hans-Peter Nilsson]

On Tue, 31 Oct 2000, Christopher Faylor wrote:
> Does anyone know the answer to this question?

Yes. :-)

> ----- Forwarded message from Mike Davies <mike.davies@epid.eurotherm.co.uk> -----
> [searching for "XpmReadFileToPixmap"]

> I searched again,  with the same result,  then I searched for "pixmap" and
> in the results I saw a message dated 15 Jul 2000 "Re: libxaw" which contains
> the word XpmReadFileToPixmap !   Does the search engine have some kind of
> character count limitation ?

There is a 12-characters default limitation.  To fix, we just have to "up"
the settings in infra/htdig-conf/sourceware.conf and re-index from
scratch.

There will be a size increase of the database, though, but I suggest we
give it a try since it seems to be expected for reasonable usability. :-)

Note that gcc.conf has the limit set to 80 characters.  I suggest rasing
it for sourceware too, unless the DB size doubles or something.

I'll do it if someone says "go ahead", but I think I need, that since the
sourceware part of things is normally not my toy and I'm not sure how much
the DB will increase (guesstimate 20-30%).

brgds, H-P
PS.  About time to check how htdig 3.2 got around stabilitywise and
such -- there were promising number for compression at the database
level last time I sneaked a peek many months ago...  No time for that yet,
though.
DS.

Re: Disk full: Updating maximum_word_length for sourceware

[Jason Molenda]

Thanks for all the great help, Hans-Peter - I really appreciate
it, and I'm sure the maintainers at Red Hat also appreciate it even
more as they've had little direct exposure to htdig so far. :-)

On Thu, Nov 02, 2000 at 08:12:30PM -0500, Hans-Peter Nilsson wrote:

> Third time's...  I'm going to change maximum_word_length up to 80 again,
> and change the update scripts for gcc and sourceware handling db.wordlist 
> as above.

Great, great.  I copied the db.wordlist around without knowing what
it was, I'm glad to hear that it isn't necessary to make an extra
copy of it during an update.


> Next panic action would be to separate the gcc and sourceware
> htdig-updates with one or two hours more, and bzip2 db.wordlist between
> runs. :-P


Sounds good.  The accesses to sourceware taper off at around 19h00
local time every day - the indexing of the earlier one can be moved
forward without causing any havoc.  The early morning (5h00 - 7h00)
is when traffic to the site picks up; it's a combination of evening
in Europe, late evening in Asia, and start-of-the-day in the Eastern
USA.  It's best to have all heavy-duty cronjobs over before around 5am.
The early morning is always the heaviest load on the system.


If we really start scraping for space, we could set TMPDIR to one of
the other partitions - even if we did it on the /sourceware/projects (cvs
repositories) partition, it wouldn't be that terrible if it happened in
the middle of the night.


Jason

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Searchengine capability question]

[Hans-Peter Nilsson]

On Wed, 1 Nov 2000, Christopher Faylor wrote:
> Well, *I* think it's a good idea.

Ok, will do.

>  There should be sufficient space on sources
> to accomodate this, right?  If not, then we can always go back, can't we?

Yes.

brgds, H-P

Re: Disk full: Updating maximum_word_length for sourceware (was:Re: [forward-gunk deleted])

[Hans-Peter Nilsson]

On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> > On Wed, 1 Nov 2000, Christopher Faylor wrote:
> > > Well, *I* think it's a good idea.
> > 
> > Ok, will do.
> 
> I think I will have to revert this change.

Done.  Re-index from scratch in progress.  Hopefully finished in 2.5-3h.
I probably need to go to sleep before that.  Perhaps someone can check the
report in /sourceware/htdig/sourceware/update-log and check that searches
work reasonably, so at least people know if it's broken or not if/when
complaints start flooding.  (Think happy thoughts.)

brgds, H-P

Disk full: Updating maximum_word_length for sourceware (was: Re:[forward-gunk deleted])

[Hans-Peter Nilsson]

On Wed, 1 Nov 2000, Hans-Peter Nilsson wrote:
> On Wed, 1 Nov 2000, Christopher Faylor wrote:
> > Well, *I* think it's a good idea.
> 
> Ok, will do.

I think I will have to revert this change.  File usage grew by 28%/300M,
but there's actually no room for that, despite the appearance that there's
2G left.

Normal df on the htdig volume:
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sdd3              5932328   3165831   2459254  56% /sourceware/htdig

Peak usage during update (within ~1% and 30s), df:
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sdd3              5932328   5392444    232641  96% /sourceware/htdig

Before the update, ls -l (sorry, no du -sk):
-rw-r--r--   1 htdigid  htdig    304411648 Oct 31 05:36 db.docdb
-rw-r--r--   1 htdigid  htdig    15186944 Oct 31 05:36 db.docs.index
-rw-rw-r--   1 htdigid  htdig    14838784 Oct 31 05:40 db.metaphone.db
-rw-rw-r--   1 htdigid  htdig    12578816 Oct 31 05:39 db.soundex.db
-rw-r--r--   1 htdigid  htdig    400643440 Oct 31 05:26 db.wordlist
-rw-r--r--   1 htdigid  htdig    328303616 Oct 31 05:26 db.words.db
-rw-r--r--   1 htdigid  htdig       15315 Oct 31 04:37 noindex-follow-urls

After the update:
-rw-r--r--   1 htdigid  htdig    302025728 Nov  1 07:00 db.docdb
-rw-r--r--   1 htdigid  htdig    15298560 Nov  1 07:00 db.docs.index
-rw-rw-r--   1 htdigid  htdig    24295424 Nov  1 07:05 db.metaphone.db
-rw-rw-r--   1 htdigid  htdig    12578816 Oct 31 05:39 db.soundex.db
-rw-r--r--   1 htdigid  htdig    564916340 Nov  1 06:37 db.wordlist
-rw-r--r--   1 htdigid  htdig    459653120 Nov  1 06:37 db.words.db
-rw-r--r--   1 htdigid  htdig       15971 Nov  1 04:37 noindex-follow-urls

So with the next update, there will not be any 200M left...

Now, usage between about 74% (normal "doubling" of the DB at update time)
and 96% is due to 1.2G of "sort" files that lives for a few minutes.
Those files can go elsewhere (the default is normally TMPDIR), if there's
spare room somewhere.  Candidates volumes are:

/dev/sdb2              8442207   1263228   7178979  15% /sourceware/projects
/dev/sdc4              2524750    121028   2273194   5% /sourceware/libre

If there's no solution within 12 hours, I will revert, hoping to get back
the 300M and that the DB:s for gcc and sourceware does not grow more than
200M before there's a new disk or a new htdig.  Note that even with a
new htdig we would have to save the old DB somewhere while diddling to get
the installation into shape, and there are no figures on what the new disk
needs would be.

So, IMHO:
- spare some bytes on your volume, sir, please; very temporary solution.
- a new disk is needed; safest route; but it's Not My Money.
- update htdig; if:s and but:s (and New Software).  Unfortunately no time.

brgds, H-P

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Searchengine capability question]

[Hans-Peter Nilsson]

On Tue, 31 Oct 2000, Jason Molenda wrote:
> On Wed, Nov 01, 2000 at 12:13:08AM -0500, Christopher Faylor wrote:
> > [incr maximum_word_length]
> 
> I agree that this is a good idea, but I thought I'd interject that
> this partition (/sourceware/htdig) has grown at a faster rate than
> I expected, so an eye needs to be kept on it.

Every now and then (frequency: months) I check on it.
Perhaps htdig 3.2 will improve things (having put compression in its
proper place), have to check it some day.

> If the htdig drive fills up, you get a corrupt htdig database and
> searches start returning no-matches - it can be a while before it
> is noticed and reported to the sourcemaster addr.  IIRC Hans-Peter
> has set up a clever little automated tester for the gcc side so
> that (someone) will be notified when trivial searches start returning
> no matches; the sourceware side will silently fail.

Not entirely C R about the gcc tester (but it's a good idea); it will
re-try once, and the logs are moved so they will not be overwritten the
next successful run.  Last time this happened was 2000-10-22; there's no
indication why, though, but I don't suspect "disk full".  On 2000-08-10 it
failed because of "/bin/sort: read error: Input/output error".  Jeff Law
said he'd asked someone to check it physically.

brgds, H-P
The update on sourceware is in progress.  I'll babysit it.

Re: [mike.davies@epid.eurotherm.co.uk: [postmaster@sourceware] Search engine capability question]

[Christopher Faylor]

Well, *I* think it's a good idea.  There should be sufficient space on sources
to accomodate this, right?  If not, then we can always go back, can't we?

cgf

On Wed, Nov 01, 2000 at 12:10:56AM -0500, Hans-Peter Nilsson wrote:
>On Tue, 31 Oct 2000, Christopher Faylor wrote:
>> Does anyone know the answer to this question?
>
>Yes. :-)
>
>> ----- Forwarded message from Mike Davies <mike.davies@epid.eurotherm.co.uk> -----
>> [searching for "XpmReadFileToPixmap"]
>
>> I searched again,  with the same result,  then I searched for "pixmap" and
>> in the results I saw a message dated 15 Jul 2000 "Re: libxaw" which contains
>> the word XpmReadFileToPixmap !   Does the search engine have some kind of
>> character count limitation ?
>
>There is a 12-characters default limitation.  To fix, we just have to "up"
>the settings in infra/htdig-conf/sourceware.conf and re-index from
>scratch.
>
>There will be a size increase of the database, though, but I suggest we
>give it a try since it seems to be expected for reasonable usability. :-)
>
>Note that gcc.conf has the limit set to 80 characters.  I suggest rasing
>it for sourceware too, unless the DB size doubles or something.
>
>I'll do it if someone says "go ahead", but I think I need, that since the
>sourceware part of things is normally not my toy and I'm not sure how much
>the DB will increase (guesstimate 20-30%).
>
>brgds, H-P
>PS.  About time to check how htdig 3.2 got around stabilitywise and
>such -- there were promising number for compression at the database
>level last time I sneaked a peek many months ago...  No time for that yet,
>though.
>DS.