Possible file download bug?

Possible file download bug?

[Paul Smith]

Crossgcc,


I wanted to alert you to a possible bug.  I'm using someone else's product that uses an old copy of Crossgcc and I've found the following issue, and from looking at the latest Crossgcc code I suspect the same bug still exists.


The issue is in scripts/functions in the code that downloads a file from the web using wget or curl.  My local ISP 'catches' page load errors and returns their own generated HTML error page and the bug I'm seeing results from a file download believing it succeeded when actually it downloaded just a dummy HTML page.


In my case the files were always supposed to be variants on Linux tar files so it easy to use the Linux 'file' command to see if the file was actually an HTML page.  I don't know whether you can do the same or whether the files you are downloaded are more diverse and need more careful checking, perhaps outside of the file download function.


However I wanted to alert you to this odd behaviour as it soaked up a few hours this morning identifying the cause and a fix.


Regards,

Paul DS.

Re: Possible file download bug?

[Allan Clark]

> On Sep 28, 2018, at 06:11, Paul Smith <paul_d_smith@hotmail.com> wrote:
> 
> Crossgcc,
> 
> 
> I wanted to alert you to a possible bug.  I'm using someone else's product that uses an old copy of Crossgcc and I've found the following issue, and from looking at the latest Crossgcc code I suspect the same bug still exists.
> 
> 
> The issue is in scripts/functions in the code that downloads a file from the web using wget or curl.  My local ISP 'catches' page load errors and returns their own generated HTML error page and the bug I'm seeing results from a file download believing it succeeded when actually it downloaded just a dummy HTML page.
> 
> 
> In my case the files were always supposed to be variants on Linux tar files so it easy to use the Linux 'file' command to see if the file was actually an HTML page.  I don't know whether you can do the same or whether the files you are downloaded are more diverse and need more careful checking, perhaps outside of the file download function.
> 
> 
> However I wanted to alert you to this odd behaviour as it soaked up a few hours this morning identifying the cause and a fix.

When the file is not found for download, does the ISP forward a 4xx or 5xx error code, or a 200 as though nothing went wrong?

Allan

RE: Possible file download bug?

[Paul Smith]

A 200 with an HTML page indicating, something like 'this server does not exist - did you mean to type one of these options'?

The first time I hit this, some errant code was trying to reach a website using http://ftp.<website...>  The author was trying to run through 'try http, if failed try ftp' but had messed up a little.  Problem is that the file appeared to download correctly but just contained the HTML error page.

Paul DS.

-----Original Message-----
From: Allan Clark <chickenandporn@gmail.com> 
Sent: 28 September 2018 15:23
To: Paul Smith <paul_d_smith@hotmail.com>
Cc: crossgcc@sourceware.org
Subject: Re: Possible file download bug?



> On Sep 28, 2018, at 06:11, Paul Smith <paul_d_smith@hotmail.com> wrote:
> 
> Crossgcc,
> 
> 
> I wanted to alert you to a possible bug.  I'm using someone else's product that uses an old copy of Crossgcc and I've found the following issue, and from looking at the latest Crossgcc code I suspect the same bug still exists.
> 
> 
> The issue is in scripts/functions in the code that downloads a file from the web using wget or curl.  My local ISP 'catches' page load errors and returns their own generated HTML error page and the bug I'm seeing results from a file download believing it succeeded when actually it downloaded just a dummy HTML page.
> 
> 
> In my case the files were always supposed to be variants on Linux tar files so it easy to use the Linux 'file' command to see if the file was actually an HTML page.  I don't know whether you can do the same or whether the files you are downloaded are more diverse and need more careful checking, perhaps outside of the file download function.
> 
> 
> However I wanted to alert you to this odd behaviour as it soaked up a few hours this morning identifying the cause and a fix.

When the file is not found for download, does the ISP forward a 4xx or 5xx error code, or a 200 as though nothing went wrong?

Allan

Re: Possible file download bug?

[Alexey Neyman]

On 09/28/2018 06:11 AM, Paul Smith wrote:
> Crossgcc,
>
>
> I wanted to alert you to a possible bug.  I'm using someone else's product that uses an old copy of Crossgcc and I've found the following issue, and from looking at the latest Crossgcc code I suspect the same bug still exists.
>
>
> The issue is in scripts/functions in the code that downloads a file from the web using wget or curl.  My local ISP 'catches' page load errors and returns their own generated HTML error page and the bug I'm seeing results from a file download believing it succeeded when actually it downloaded just a dummy HTML page.
>
>
> In my case the files were always supposed to be variants on Linux tar files so it easy to use the Linux 'file' command to see if the file was actually an HTML page.  I don't know whether you can do the same or whether the files you are downloaded are more diverse and need more careful checking, perhaps outside of the file download function.
>
>
> However I wanted to alert you to this odd behaviour as it soaked up a few hours this morning identifying the cause and a fix.
It has been an issue with some download servers, too (I think, the one 
hosting libelf is an example; SourceForge during their outages is 
another one): instead of an error response, they return a valid 200 code 
with an HTML page. Current crosstool-NG (on master) offers an ability to 
verify the digest of the download (MD5/SHA-1/SHA-256/SHA-512); such 
broken download would fail this verification - so crosstool-NG won't 
save it to the local cache and will bail out with an error. But, it is 
only on master, no released versions do such verification.

Regards,
Alexey.