No htdig indexing on gcc side since June 1 due to file too big

No htdig indexing on gcc side since June 1 due to file too big

[Hans-Peter Nilsson]

Just as a heads-up, but curiously I don't see anyone noticing.
I'm quick to jump to the the conclusion that we no longer need a
local gcc search engine, despite people saying they do use it.

Oh the error?  Anticipated for a long time.  See last in
/sourceware/htdig/gcc/update-log.failed.2005-06-01 (edited
cutnpaste):

HTUPDATE merge stage entered, at Wed Jun  1 17:24:59 UTC 2005
/sourceware/htdig/gcc/bin/htupdate.sh: line 145: 30399 File size limit exceeded$BINDIR/htmerge -s -a $HT_VERBOSE_OPT
Exit code for merge stage: 153
HTUPDATE merge stage errors detected, at Wed Jun  1 18:17:16 UTC 2005
ls -l
total 8957568
-rw-r--r--    1 htdigid  htdig    1379831808 May 31 15:37 db.docdb
-rw-r--r--    1 htdigid  htdig    1386209280 Jun  1 17:24 db.docdb.work
-rw-r--r--    1 htdigid  htdig    70968320 May 31 15:37 db.docs.index
-rw-r--r--    1 htdigid  htdig    70968320 Jun  1 08:41 db.docs.index.work
-rw-r--r--    1 htdigid  htdig    617882373 Mar 19 04:23 db.wordlist.gz9
-rw-r--r--    1 htdigid  htdig    679262524 Jun  1 17:24 db.wordlist.work
-rw-r--r--    1 htdigid  htdig    663601889 Jun  1 18:17 db.wordlist.work.new
-rw-r--r--    1 htdigid  htdig    2147321856 May 31 14:57 db.words.db
-rw-r--r--    1 htdigid  htdig    2147483647 Jun  1 18:17 db.words.db.work
-r--r--r--    1 htdigid  htdig           1 Jun  1 08:17 indexing-in-progress
-rw-r--r--    1 htdigid  htdig        3567 Jun  1 08:19 noindex-follow-urls
HTUPDATE Grepping in the log says we went over some limit while digging.
HTUPDATE Manual assistance needed.
HTUPDATE Giving up on database update at Wed Jun  1 18:18:00 UTC 2005

(To wit, the hard limit for indexing is reached and I've already
tweaked and compressed; see archives for earlier efforts.  I'm
*not* doing the multiple databases suggestion however.)

brgds, H-P
PS.  No, it didn't waste time retrying after that date.

Re: No htdig indexing on gcc side since June 1 due to file too big

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

Hi -

> Just as a heads-up, but curiously I don't see anyone noticing.
> I'm quick to jump to the the conclusion that we no longer need a
> local gcc search engine, despite people saying they do use it.
> [...]

Perhaps one of the first housewarming events on the new sourceware
machine(s) could be the deactivation of htdig and use of mnogosearch
instead.

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: No htdig indexing on gcc side since June 1 due to file too big

[Hans-Peter Nilsson]

On Mon, 4 Jul 2005, Frank Ch. Eigler wrote:
> > Just as a heads-up, but curiously I don't see anyone noticing.
> > I'm quick to jump to the the conclusion that we no longer need a
> > local gcc search engine, despite people saying they do use it.
> > [...]
>
> Perhaps one of the first housewarming events on the new sourceware
> machine(s) could be the deactivation of htdig and use of mnogosearch
> instead.

I thought it wasn't ready for public use or something?
If I'm wrong, good news!

brgds, H-P

Re: No htdig indexing on gcc side since June 1 due to file too big

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

Hi -

On Mon, Jul 04, 2005 at 01:03:33PM -0400, Hans-Peter Nilsson wrote:
> [...]
> I thought [mnogosearch] wasn't ready for public use or something?
> [...]

To the extent it's currently configured for the sourceware side, it
works very nicely.  The incremental online indexing alone is worth the
effort.  What's missing is configuration to index the gcc side, and a
bridging script to impersonate the htdig cgi script (so all the
current web pages don't have to be rebuilt).

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]