Re: ongoing sourceware.org recovery from disk corruption

[Joseph Myers <joseph@codesourcery.com>]

On Tue, 15 Aug 2017, Frank Ch. Eigler wrote:

> So we're proceeding to restore bits, file by file, when/as corruption
> is found.  It's silly laborious, and we'll appreciate your patience
> and help identifying affected files.  The version control repositories
> appear fine now, /ftp is getting mass-restored (since it's apprx. all
> old), so the most important stuff seems OK.  There are reports of some
> mailing list archives and wiki pages being broken; will look at those
> next.  Please come hang out on #overseers on irc.freenode.net to chat.

Suggested general comparison methodology (for the whole backup against 
live data, not just areas we know to be broken, and with due disregard of 
any areas we are confident have been safely fixed now - and of course any 
areas of old files that shouldn't change at all can just be restored with 
rsync -c without such comparisons needed):

* If the file contents have changed but the timestamp hasn't, it's almost 
certainly corruption (especially if the changes involve blocks of NUL 
bytes) and restoring the old version makes sense.

* If a file has disappeared, it's very probably corruption and restoring 
the old file is very probably safe (it *could* be a temporary file 
captured by the backup, or something that was properly removed, but more 
likely something lost by metadata corruption).

* If a file is new since the last backup, it may or may not also be 
corrupted, but there's not much that can be done about it if it is 
corrupted (except in special cases, e.g. getting copies of release files 
or version control data from elsewhere).

* If a file's contents and timestamp have both changed, it may or may not 
also be corrupted, and if it is it may or may not make sense to restore 
from backup.  Depending on how many such files there are, we may need to 
consider case by case what should be done for particular kinds of files 
(e.g. if they should be text, checking for NUL bytes would help indicate 
corruption).

-- 
Joseph S. Myers
joseph@codesourcery.com