Re: [PATCH 4/7 v2] lto: Implement ltrans cache

[Jan Hubicka <hubicka@ucw.cz>]

> Michal Jires <mjires@suse.cz> writes:
> 
> No performance data?

Michal has bachelor thesis on the topic	which has some statistics
https://dspace.cuni.cz/handle/20.500.11956/183051?locale-attribute=en
> 
> > +
> > +static const md5_checksum_t INVALID_CHECKSUM = {
> > +  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> > +};
> 
> There are much faster/optimized modern hashes for good collision detection over
> MD5 especially when it's not needed to be cryptographically secure. Pick
> something from smhasher.
> 
> Also perhaps the check sum should be cached in the file? I assume it's
> cheap to compute while writing. It could be written at the tail of the
> file. Then it can be read by seeking to the end and you save that
> step.
> 
> The lockfiles scare me a bit. What happens when they get lost, e.g.
> due to a compiler crash? You may need some recovery for that.
> Perhaps it would be better to make the files self checking, so that
> partial files can be detected when reading, and get rid of the locks.

Those seem good ideas. One problem is that the cached files are object
files and thus we will likely need simple-object extension to embed the
hash.

Overall incremental LTO is quite complex problem hitting several areas
where current implementation of LTO is lacking. Most important problem
seems to be the divergence of partition files which happens due to
various global counters and have no good purpose. Fixing those is useful
per se, since it improves reproducibility of builds. (Some of this was
merged to last release, but not all).

Ohter important issue is the debug info which is slow and diverges
often.  We will also need to work on speeding up WPA.

Since it is relatively early stage1, I think it makes sense to merge the
code without adding extra complexity and optimizing it incrementally.
(Since it works relatively well already)

There are many things to do and I think it is better to do that in trunk
rahter than cumulating relatively complex changes on branch.
md5 is already supported by libiberty so it is kind of easy choice for
first cut implementation.

Honza
> 
> -Andi