[PATCH setup] Query the user if a corrupt local file is found

[PATCH setup] Query the user if a corrupt local file is found

[Ken Brown]

If a corrupt file is found in one of the selected mirror site
directories, offer to delete it instead of making this a fatal error.
Do this only on the first call to check_for cached().  If the corrupt
file is still there on the second call, then the deletion failed, and
the user will have to fix the problem.

See https://cygwin.com/ml/cygwin/2017-12/msg00122.html for discussion.
---
 download.cc | 17 ++++++++++++-----
 download.h  |  4 +++-
 res.rc      |  3 ++-
 resource.h  |  1 +
 4 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/download.cc b/download.cc
index b606eb1..697f94a 100644
--- a/download.cc
+++ b/download.cc
@@ -71,7 +71,7 @@ validateCachedPackage (const std::string& fullname, packagesource & pkgsource)
 /* 0 on failure
  */
 int
-check_for_cached (packagesource & pkgsource, bool mirror_mode)
+check_for_cached (packagesource & pkgsource, bool mirror_mode, bool query_corrupt, HWND owner)
 {
   // Already found one.
   if (pkgsource.Cached())
@@ -114,6 +114,12 @@ check_for_cached (packagesource & pkgsource, bool mirror_mode)
     {
       if (validateCachedPackage (fullname, pkgsource))
         pkgsource.set_cached (fullname);
+      else if (query_corrupt && owner && !unattended_mode
+	       && yesno (owner, IDS_QUERY_CORRUPT, fullname.c_str()) == IDYES)
+	{
+	  io_stream::remove (fullname);
+	  continue;
+	}
       else
         throw new Exception (TOSTRING(__LINE__) " " __FILE__,
             "Package validation failure for " + fullname,
@@ -266,13 +272,14 @@ do_download_thread (HINSTANCE h, HWND owner)
 	    {
 	      if (pkg.picked())
 		{
-		    if (!check_for_cached (*version.source()))
-		      total_download_bytes += version.source()->size;
+		  if (!check_for_cached (*version.source(), false, true, owner))
+		    total_download_bytes += version.source()->size;
 		}
 	      if (pkg.srcpicked () || IncludeSource)
 		{
-		    if (!check_for_cached (*sourceversion.source()))
-		      total_download_bytes += sourceversion.source()->size;
+		  if (!check_for_cached (*sourceversion.source(), false,
+					 true, owner))
+		    total_download_bytes += sourceversion.source()->size;
 		}
 	    }
 	  catch (Exception * e)
diff --git a/download.h b/download.h
index 117a44d..e9eb257 100644
--- a/download.h
+++ b/download.h
@@ -16,7 +16,9 @@
 #ifndef SETUP_DOWNLOAD_H
 #define SETUP_DOWNLOAD_H
 
+#include "win32.h"
+
 class packagesource;
-int check_for_cached (packagesource & pkgsource, bool = false);
+int check_for_cached (packagesource & pkgsource, bool mirror_mode = false, bool query_corrupt = false, HWND owner = NULL);
 
 #endif /* SETUP_DOWNLOAD_H */
diff --git a/res.rc b/res.rc
index 901cf76..31c8f20 100644
--- a/res.rc
+++ b/res.rc
@@ -550,7 +550,8 @@ BEGIN
     IDS_DOWNLOAD_INCOMPLETE_EXIT  "Download incomplete.  Check %s for details"
     IDS_INSTALL_ERROR	    "Installation error (%s), Continue with other packages?"
     IDS_INSTALL_INCOMPLETE  "Installation incomplete.  Check %s for details"
-    IDS_CORRUPT_PACKAGE     "Package file %s has a corrupt local copy, please remove and retry."
+    IDS_CORRUPT_PACKAGE     "Package %s has a corrupt local copy, please remove and retry."
+    IDS_QUERY_CORRUPT       "The file %s is corrupt.  Delete it?."
     IDS_SKIP_PACKAGE	    "%s\nDo you want to skip this package ?"
     IDS_UNCAUGHT_EXCEPTION  "Fatal Error: Uncaught Exception\nThread: %s\nType: %s\nMessage: %s"
     IDS_UNCAUGHT_EXCEPTION_WITH_ERRNO  "Fatal Error: Uncaught Exception\nThread: %s\nType: %s\nMessage: %s\nAppErrNo: %d"
diff --git a/resource.h b/resource.h
index 79b876d..70d90ca 100644
--- a/resource.h
+++ b/resource.h
@@ -39,6 +39,7 @@
 #define IDS_ELEVATED			  139
 #define IDS_INSTALLEDB_VERSION            140
 #define IDS_DOWNLOAD_INCOMPLETE_EXIT      141
+#define IDS_QUERY_CORRUPT                 142
 
 // Dialogs
 
-- 
2.15.1

Re: [PATCH setup] Query the user if a corrupt local file is found

[Ken Brown]

On 12/16/2017 1:37 PM, Ken Brown wrote:
> If a corrupt file is found in one of the selected mirror site
> directories, offer to delete it instead of making this a fatal error.
> Do this only on the first call to check_for cached().  If the corrupt
> file is still there on the second call, then the deletion failed, and
> the user will have to fix the problem.
> 
> See https://cygwin.com/ml/cygwin/2017-12/msg00122.html for discussion.

This is unnecessarily complicated.  Please disregard.  I'll submit v2 later.

Ken

Re: [PATCH setup] Query the user if a corrupt local file is found

[Ken Brown]

On 12/18/2017 1:40 PM, Ken Brown wrote:
> On 12/16/2017 1:37 PM, Ken Brown wrote:
>> If a corrupt file is found in one of the selected mirror site
>> directories, offer to delete it instead of making this a fatal error.
>> Do this only on the first call to check_for cached().  If the corrupt
>> file is still there on the second call, then the deletion failed, and
>> the user will have to fix the problem.
>>
>> See https://cygwin.com/ml/cygwin/2017-12/msg00122.html for discussion.
> 
> This is unnecessarily complicated.  Please disregard.  I'll submit v2 
> later.

Never mind.  My idea for simplification didn't work out.  I'll stick 
with this version.

Ken

Re: [PATCH setup] Query the user if a corrupt local file is found

[Jon Turney]

On 16/12/2017 18:37, Ken Brown wrote:
> If a corrupt file is found in one of the selected mirror site
> directories, offer to delete it instead of making this a fatal error.
> Do this only on the first call to check_for cached().  If the corrupt
> file is still there on the second call, then the deletion failed, and
> the user will have to fix the problem.
> 
> See https://cygwin.com/ml/cygwin/2017-12/msg00122.html for discussion.

This is a nice idea.  But I think there are some structural problems 
with this, though. e.g. validateCachedPackage() only checks the package 
size, not hash (which happens in the install phase)

I'm also concerned about masking problems with how we got into this 
state in the first place: I think either (i) a corrupt download was 
stored into the cache, (ii) the valid size was changed between runs, or 
(iii) the files contents actually got corrupted somehow.

(i) indicates another problem in setup

Uploading replacement packages, which would cause (ii) was permitted 
historically (but of course, didn't work well), but now should be 
forbidden by calm.  This could, of course, still happen with a private 
package repo, and should be handled sanely.

(iii) seems unlikely, barring deliberate action.

I guess the ideal solution looks something like:

Download:
- verify size/hash of cached packages, offer to remove corrupt ones
- download packages, verifying size/hash

Install:
- verify size/hash of cached packages, skip corrupt ones
- install packages

with some memory so that we don't verify size/hash for the same package 
file more than once...

> ---
>   download.cc | 17 ++++++++++++-----
>   download.h  |  4 +++-
>   res.rc      |  3 ++-
>   resource.h  |  1 +
>   4 files changed, 18 insertions(+), 7 deletions(-)
> 
> diff --git a/download.cc b/download.cc
> index b606eb1..697f94a 100644
> --- a/download.cc
> +++ b/download.cc
> @@ -71,7 +71,7 @@ validateCachedPackage (const std::string& fullname, packagesource & pkgsource)
>   /* 0 on failure
>    */
>   int
> -check_for_cached (packagesource & pkgsource, bool mirror_mode)
> +check_for_cached (packagesource & pkgsource, bool mirror_mode, bool query_corrupt, HWND owner)
>   {
>     // Already found one.
>     if (pkgsource.Cached())
> @@ -114,6 +114,12 @@ check_for_cached (packagesource & pkgsource, bool mirror_mode)
>       {
>         if (validateCachedPackage (fullname, pkgsource)) >           pkgsource.set_cached (fullname);
> +      else if (query_corrupt && owner && !unattended_mode

I don't think you need to check for non-null owner here, as yesno() will 
still work in that case

I kind of incline to doing the 'yes' action in unattended_mode, rather 
than reporting a fatal error.

Re: [PATCH setup] Query the user if a corrupt local file is found

[Ken Brown]

On 1/5/2018 10:00 AM, Jon Turney wrote:
> On 16/12/2017 18:37, Ken Brown wrote:
>> If a corrupt file is found in one of the selected mirror site
>> directories, offer to delete it instead of making this a fatal error.
>> Do this only on the first call to check_for cached().  If the corrupt
>> file is still there on the second call, then the deletion failed, and
>> the user will have to fix the problem.
>>
>> See https://cygwin.com/ml/cygwin/2017-12/msg00122.html for discussion.
> 
> This is a nice idea.  But I think there are some structural problems 
> with this, though. e.g. validateCachedPackage() only checks the package 
> size, not hash (which happens in the install phase)
> 
> I'm also concerned about masking problems with how we got into this 
> state in the first place: I think either (i) a corrupt download was 
> stored into the cache, (ii) the valid size was changed between runs, or 
> (iii) the files contents actually got corrupted somehow.
> 
> (i) indicates another problem in setup
> 
> Uploading replacement packages, which would cause (ii) was permitted 
> historically (but of course, didn't work well), but now should be 
> forbidden by calm.  This could, of course, still happen with a private 
> package repo, and should be handled sanely.
> 
> (iii) seems unlikely, barring deliberate action.
> 
> I guess the ideal solution looks something like:
> 
> Download:
> - verify size/hash of cached packages, offer to remove corrupt ones
> - download packages, verifying size/hash
> 
> Install:
> - verify size/hash of cached packages, skip corrupt ones
> - install packages
> 
> with some memory so that we don't verify size/hash for the same package 
> file more than once...

I agree.  It's strange that the code for size checking is currently in 
download.cc, and the code for hash checking is currently in install.cc. 
All of this checking should probably be done in a 
packagesource::validate() function, which can be called during both the 
download and install stages.

I'll work on this and send a new patch.

Ken