Re: updatedb broken as of findutils 4.8.0-1 due to bigram.exe no longer being provided

[Brian Inglis <Brian.Inglis@SystematicSw.ab.ca>]

On 2021-08-29 06:06, Dan Harkless wrote:
> On 8/29/2021 4:02 AM, Hans-Bernhard Bröker wrote:
>> Am 28.08.2021 um 18:23 schrieb Dan Harkless:
>>> Looks like it's because in findutils 4.8.0-1, the bigram.exe program 
>>> is no longer provided, but the /usr/bin/updatedb script (still) 
>>> depends on it being there:
>>      [...]
>>>      + for binary in $find $frcode $bigram $code
>>>      + checkbinary /usr/libexec/frcode

>> The version of updatedb in the 4.8.0-1 package does not actually 
>> contain those lines.  Mention of both $bigram and $code has been 
>> removed from the loop construct (and from everywhere else in the script).
>>
>> That's because the old format of find databases, which is the only one 
>> actually using bigram and code, was removed from updatedb as of 
>> findutils version 4.7, so there really cannot be a need for the bigram 
>> tool any more.

> Argh!  So sorry for the false report!  I completely forgot that years 
> back I had made a locally patched version (which is earlier in my path) 
> of Cygwin updatedb 4.6.0-1 to troubleshoot and work around problems I 
> was having with the tool.
> 
> I have 12M+ pathnames on my main Windows system, and I suddenly started 
> having issues with the updatedb going from taking less than an hour, to 
> taking more than 24 hours, and running into the next job.
> 
> It was very awkward to try to troubleshoot what was going on without a 
> 'find' log to 'tail', so I patched my  local copy of updatedb to write 
> to an intermediate file, rather than going direct to 'sort' over a pipe.
> 
> Another problem I was having was that though I have 24 GB of RAM on my 
> system, I would get low-memory popup warnings from the OS when the sort 
> would go off.  (The warnings mislay the blame on Firefox, because I 
> usually have big sessions running that take even more RAM than the sort.)
> 
> I was hoping running sort on a _file_ rather than stdin might allow it 
> to reduce the RAM use enough to not get the warning, but unfortunately 
> (and unsurprisingly) I still get it with the intermediate file.  This is 
> just a warning, though — I haven't had it actually run out of RAM so 
> far, I don't think.
> 
> The final problem I was addressing in my patched version was some 
> missing error-checking, which was causing me to be left with _no_ 
> filename DB, when the update would fail, rather than at least being left 
> with the one from last time.
> 
> I could send along my patches, but I don't know that I've solved these 
> issues in a general enough way.  For instance, my 12 million+ pathnames 
> come out to about 1.4 GiB of UNIX-linefeed-separated UTF-8 strings. 
> Writing that much to my HD is not a concern, but obviously some people 
> might not want to write that much every time to, say, a small 
> flash-based device.
> 
> Thoughts?

Thanks for the analysis Hans-Bernhard.

Please recheck the announcement for 4.8 and change info for 4.7: as of 
4.8 locate should still work on old format dbs, but from 4.7 updatedb 
will no longer generate or update them, and in some future release, 
locate will no longer work on them.
The old (pre-GNU Unix) format was deprecated from 4.0 (~25 years ago!) 
and each run of updatedb should have warned you to upgrade, unless you 
patched that out.

See:

	$ info finding databases 'database formats' old

or:

<https://gnu.org/software/findutils/manual/html_node/find_html/Old-Database-Format.html>

I searched for more info on the discussion list archive at:

	<https://lists.gnu.org/archive/html/bug-findutils/>

but could find nothing obviously related to upgrading or migrating, 
although that archive goes back only ~20 years! ;^>

Migration appears to require running the previous 4.6 updatedb without 
--old-format to regenerate the new database in LOCATE02 format?
You should then be able to upgrade to the latest 4.8 findutils and use 
that going forward.

You could email the discussion list <mailto:bug-findutils@gnu.org> about 
your situation, file sizes, timings, migration path, and issues, and 
cross-post here about anything in the replies we may be able to help you 
with.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]