Why package cache is not used during setup download?

Why package cache is not used during setup download?

[Aleksey Midenkov]

Cygwin setup process my turn to endless retry-error on bad internet
channels, because:

1. setup doesn't know how to retry download (why it fails to download anyway?);
2. setup doesn't use cache of previously downloaded files and always
redownload all packages from the beginning.

How possible I may overcome this situation?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Why package cache is not used during setup download?

[Andrey Repin]

Greetings, Aleksey Midenkov!

> Cygwin setup process my turn to endless retry-error on bad internet
> channels, because:

> 1. setup doesn't know how to retry download (why it fails to download anyway?);

Because your internet is bad?

> 2. setup doesn't use cache of previously downloaded files and always
> redownload all packages from the beginning.

It do use cache, if checksum is correct.

> How possible I may overcome this situation?

Fix your internet, or download packages where it is better.
If you want partial download support in setup - patches are welcome, I'm sure.


-- 
With best regards,
Andrey Repin
Sunday, October 25, 2015 16:58:07

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Why package cache is not used during setup download?

[Aleksey Midenkov]

On Sun, Oct 25, 2015 at 5:00 PM, Andrey Repin <anrdaemon@yandex.ru> wrote:
> Greetings, Aleksey Midenkov!
>
>> Cygwin setup process my turn to endless retry-error on bad internet
>> channels, because:
>
>> 1. setup doesn't know how to retry download (why it fails to download anyway?);
>
> Because your internet is bad?

On the second thought I believe that the problem is with setup.exe
itself. Because:

a) I tried different mirrors. It always fails in somewhere between
80-90% (of total progress).
b) I never noticed any problems with downloading files except with
Cygwin. Gigabyte-sized files via http are downloading without fail.

>
>> 2. setup doesn't use cache of previously downloaded files and always
>> redownload all packages from the beginning.
>
> It do use cache, if checksum is correct.

Hmmm, on setup.ini doc says:

install: filename size-in-bytes MD5 sum

... The optional MD5 sum is 32 characters from the set 0-9a-f (case
matters). ...

But, I see too many characters in my setup.ini:

install: x86_64/release/cygwin/cygwin-devel/cygwin-devel-2.2.1-1.tar.xz
273004 be6a2b80a8d06d14a56c1e5b2aabb31698d13aa5328dc7b675be9790a874a3a8b0fda9bdf911001e9a8c79439312c9ec9a8e80fb3bd40304ba459bd1e0850a2e

MD5 sum of the cached copy is:

58d2bbfd703798714e28263ec3386ec4
*./cygwin/cygwin-devel/cygwin-devel-2.2.1-1.tar.xz

Btw, the size of file is correct:

$ ls -l ./release/cygwin/cygwin-devel/cygwin-devel-2.2.1-1.tar.xz
-rw-r--r-- 1 op None 273004 Oct 25 10:38
./release/cygwin/cygwin-devel/cygwin-devel-2.2.1-1.tar.xz


>
>> How possible I may overcome this situation?
>
> Fix your internet, or download packages where it is better.
> If you want partial download support in setup - patches are welcome, I'm sure.
>
>
> --
> With best regards,
> Andrey Repin
> Sunday, October 25, 2015 16:58:07
>
> Sorry for my terrible english...
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Why package cache is not used during setup download?

[David Stacey]

On 25/10/15 18:47, Aleksey Midenkov wrote:
> Hmmm, on setup.ini doc says:
>
> install: filename size-in-bytes MD5 sum
>
> ... The optional MD5 sum is 32 characters from the set 0-9a-f (case
> matters). ...
>
> But, I see too many characters in my setup.ini:
>
> install: x86_64/release/cygwin/cygwin-devel/cygwin-devel-2.2.1-1.tar.xz
> 273004 be6a2b80a8d06d14a56c1e5b2aabb31698d13aa5328dc7b675be9790a874a3a8b0fda9bdf911001e9a8c79439312c9ec9a8e80fb3bd40304ba459bd1e0850a2e

MD5 is broken, and so Cygwin's setup.ini now uses SHA512 [1]. If you 
have an old setup executable, it won't understand the new checksums; try 
downloading a new setup executable from the Cygwin homepage [2].

Hope this helps,

Dave.

[1] - https://cygwin.com/ml/cygwin-announce/2015-02/msg00013.html
[2] - https://cygwin.com/


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Why package cache is not used during setup download?

[Andrey Repin]

Greetings, Aleksey Midenkov!

>>> Cygwin setup process my turn to endless retry-error on bad internet
>>> channels, because:
>>
>>> 1. setup doesn't know how to retry download (why it fails to download anyway?);
>>
>> Because your internet is bad?

> On the second thought I believe that the problem is with setup.exe
> itself. Because:

> a) I tried different mirrors. It always fails in somewhere between
> 80-90% (of total progress).
> b) I never noticed any problems with downloading files except with
> Cygwin. Gigabyte-sized files via http are downloading without fail.

If you get consistent results from different mirrors (and since you're the
only person reporting it), my second thought is that it is a local issue.
Overly zealous antivirus/firewall coming to mind.

>>> 2. setup doesn't use cache of previously downloaded files and always
>>> redownload all packages from the beginning.
>>
>> It do use cache, if checksum is correct.

> Hmmm, on setup.ini doc says:

> install: filename size-in-bytes MD5 sum

As David pointed out already, MD5 hash proven weak and is no longer used in
sensitive environments.


-- 
With best regards,
Andrey Repin
Monday, October 26, 2015 14:45:39

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

MD5 vs SHA512 in setup.ini (was: Why package cache is not used during setup download?)

[Warren Young]

On Oct 26, 2015, at 5:48 AM, Andrey Repin <anrdaemon@yandex.ru> wrote:
> 
> MD5 hash proven weak

That’s a bit strong.  It’s better to say that MD5 has weak collision resistance properties, which in this context means it is possible to generate a Cygwin package with arbitrary contents that produces the same hash as the legitimate package, in a computationally useful time frame.

But, that is not the value MD5 is providing to setup.exe.  If you are downloading a package from bad-actor.com, you are also downloading setup.ini from there, so they can rewrite the hashes.  Only if you take the extra step to get your setup.ini from a different site can you cross-check the hashes.

Even then, all it proves is that the file you downloaded is the one the server claims to be providing.  It doesn’t prove provenance, which is what people really seem to want, when they go hand-checking hashes.

One way to solve that would be for cygwin.com could run a special-purpose CA, and for the process that moves uploaded packages into the distribution directory to sign them using the CA’s private key.  Then setup.exe can cryptographically prove to itself that it is installing legitimate packages.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Why package cache is not used during setup download?

[Aleksey Midenkov]

On Mon, Oct 26, 2015 at 2:48 PM, Andrey Repin <anrdaemon@yandex.ru> wrote:
> Greetings, Aleksey Midenkov!
>
>>>> Cygwin setup process my turn to endless retry-error on bad internet
>>>> channels, because:
>>>
>>>> 1. setup doesn't know how to retry download (why it fails to download anyway?);
>>>
>>> Because your internet is bad?
>
>> On the second thought I believe that the problem is with setup.exe
>> itself. Because:
>
>> a) I tried different mirrors. It always fails in somewhere between
>> 80-90% (of total progress).
>> b) I never noticed any problems with downloading files except with
>> Cygwin. Gigabyte-sized files via http are downloading without fail.
>
> If you get consistent results from different mirrors (and since you're the
> only person reporting it), my second thought is that it is a local issue.
> Overly zealous antivirus/firewall coming to mind.

I remember this was also happening long time ago (about 10 years or
more) in completely different environment of course. Anyway, local
problem with *Cygwin-only* means problem with Cygwin. I rarely use
Cygwin and when I use it, I always stumble upon this. Maybe there is
another reason of why you don't know of other complaints.

>
>>>> 2. setup doesn't use cache of previously downloaded files and always
>>>> redownload all packages from the beginning.
>>>
>>> It do use cache, if checksum is correct.
>
>> Hmmm, on setup.ini doc says:
>
>> install: filename size-in-bytes MD5 sum
>
> As David pointed out already, MD5 hash proven weak and is no longer used in
> sensitive environments.

I don't need to know who the David is and what he pointed out. But
what does this mean in core, the documentation is wrong and checksum
is done with different algorithm?

>
>
> --
> With best regards,
> Andrey Repin
> Monday, October 26, 2015 14:45:39
>
> Sorry for my terrible english...
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple