dwz testsuite with binaries

dwz testsuite with binaries

[Tom de Vries]

Hi,

I've just setup a testsuite for the dwz component containing binaries (
announcement: https://sourceware.org/ml/dwz/2019-q1/msg00103.html ).

Due to the size of some of the binaries, it's separate from the dwz git
repo on sourceware.org.

For now, it lives at
http://ftp.suse.com/pub/people/tdevries/dwz/dwz-external-tests.tgz,
which is personal ftp space that only I can modify.

Is it a good idea to setup ftp://sourceware.org/pub/dwz for this?

Otherwise, is there support available, considered or planned related to
large file support for git, like f.i. git-annex or git LFS?

Thanks,
- Tom

Re: dwz testsuite with binaries

[Carlos O'Donell]

On 3/15/19 12:48 PM, Tom de Vries wrote:
> Hi,
> 
> I've just setup a testsuite for the dwz component containing binaries (
> announcement: https://sourceware.org/ml/dwz/2019-q1/msg00103.html ).
> 
> Due to the size of some of the binaries, it's separate from the dwz git
> repo on sourceware.org.
> 
> For now, it lives at
> http://ftp.suse.com/pub/people/tdevries/dwz/dwz-external-tests.tgz,
> which is personal ftp space that only I can modify.
> 
> Is it a good idea to setup ftp://sourceware.org/pub/dwz for this?
> 
> Otherwise, is there support available, considered or planned related to
> large file support for git, like f.i. git-annex or git LFS?

Hey Tom! :-)

We need a similar solution for glibc. We have terabytes of trace files
for malloc which are part of our workload corpus for running in the
malloc simulator for functional testing.

We haven't done any testing with git-annex for git LFS. Do you have any
guidance there? Have you done any tests?

-- 
Cheers,
Carlos.

Re: dwz testsuite with binaries

[Tom de Vries]

On 15-03-19 17:53, Carlos O'Donell wrote:
> On 3/15/19 12:48 PM, Tom de Vries wrote:
>> Hi,
>>
>> I've just setup a testsuite for the dwz component containing binaries (
>> announcement: https://sourceware.org/ml/dwz/2019-q1/msg00103.html ).
>>
>> Due to the size of some of the binaries, it's separate from the dwz git
>> repo on sourceware.org.
>>
>> For now, it lives at
>> http://ftp.suse.com/pub/people/tdevries/dwz/dwz-external-tests.tgz,
>> which is personal ftp space that only I can modify.
>>
>> Is it a good idea to setup ftp://sourceware.org/pub/dwz for this?
>>
>> Otherwise, is there support available, considered or planned related to
>> large file support for git, like f.i. git-annex or git LFS?
> 
> Hey Tom! :-)
> 

Hi :)

> We need a similar solution for glibc. We have terabytes of trace files
> for malloc which are part of our workload corpus for running in the
> malloc simulator for functional testing.
> 

> We haven't done any testing with git-annex for git LFS. Do you have any
> guidance there? Have you done any tests?

No, I have no experience with either git-annex or git LFS.

I just mention them because this is what seems to be used at github/gitlab:
- https://git-lfs.github.com/
- https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/
-
https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/

Thanks,
- Tom

Re: dwz testsuite with binaries

[Frank Ch. Eigler]

Hi -

> [...]  Due to the size of some of the binaries, it's separate from
> the dwz git repo on sourceware.org.

Other than those 200MB+ cc1 binaries, I see no problem with putting
them into git.  Are you positive that test files of that magnitude are
necessary for testing what you need to do?  Such files will incur
storage & transfer & processing costs by all users of your tool, not
just sourceware.org.  (Same goes double for "terabytes of malloc
traces".)


> Is it a good idea to setup ftp://sourceware.org/pub/dwz for this?

The amount of storage available is only moderate.  And if you need
these files version-controlled, reproducible, then an ungoverned ftp
area is not a great long term approach.

- FChE

Re: dwz testsuite with binaries

[Tom de Vries]

On 15-03-19 18:11, Frank Ch. Eigler wrote:
> Hi -
> 
>> [...]  Due to the size of some of the binaries, it's separate from
>> the dwz git repo on sourceware.org.
> 
> Other than those 200MB+ cc1 binaries

FWIW, I'm already preparing an update that contains 1.1 GB worth of files.

> I see no problem with putting
> them into git.

I'll leave that to Jakub to decide.

> Are you positive that test files of that magnitude are
> necessary for testing what you need to do?

It's just the smallest reproducers I have at the moment. There may be
smaller ones, there may not be.

> Such files will incur
> storage & transfer & processing costs by all users of your tool, not
> just sourceware.org.  (Same goes double for "terabytes of malloc
> traces".)
> 

I'm not sure I fully understand that. I would expect a linux distro
end-user not to get these files, unless installing a source package.

>> Is it a good idea to setup ftp://sourceware.org/pub/dwz for this?
> 
> The amount of storage available is only moderate.  And if you need
> these files version-controlled, reproducible, then an ungoverned ftp
> area is not a great long term approach.

My preference would be to have the big files version-controlled with a
known git large-file handling extension. But, in absence of that, I'd
guess that sourceware ftp would be better than personal ftp.

Thanks,
- Tom

Re: dwz testsuite with binaries

[Jakub Jelinek]

On Fri, Mar 15, 2019 at 06:37:50PM +0100, Tom de Vries wrote:
> On 15-03-19 18:11, Frank Ch. Eigler wrote:
> > Hi -
> > 
> >> [...]  Due to the size of some of the binaries, it's separate from
> >> the dwz git repo on sourceware.org.
> > 
> > Other than those 200MB+ cc1 binaries
> 
> FWIW, I'm already preparing an update that contains 1.1 GB worth of files.
> 
> > I see no problem with putting
> > them into git.
> 
> I'll leave that to Jakub to decide.

I'd prefer not to have the large (but ideally any binaries) in the main dwz
repo, so that people can just download the half a MB of sources and don't
have to download thousand times more if they don't want to.
For the small tests, couldn't we do it like gdb does, e.g. save *.s/*.S
files and assemble/link those before using them as tests?

That is of course not possible for the huge binaries.
For those ftp sounds reasonable to me, or indeed git-lfs (but in a separate
repo like dwz-large-tests).

	Jakub

Re: dwz testsuite with binaries

[Jakub Jelinek]

On Fri, Mar 15, 2019 at 06:43:19PM +0100, Jakub Jelinek wrote:
> On Fri, Mar 15, 2019 at 06:37:50PM +0100, Tom de Vries wrote:
> > On 15-03-19 18:11, Frank Ch. Eigler wrote:
> > > Hi -
> > > 
> > >> [...]  Due to the size of some of the binaries, it's separate from
> > >> the dwz git repo on sourceware.org.
> > > 
> > > Other than those 200MB+ cc1 binaries
> > 
> > FWIW, I'm already preparing an update that contains 1.1 GB worth of files.
> > 
> > > I see no problem with putting
> > > them into git.
> > 
> > I'll leave that to Jakub to decide.
> 
> I'd prefer not to have the large (but ideally any binaries) in the main dwz
> repo, so that people can just download the half a MB of sources and don't
> have to download thousand times more if they don't want to.
> For the small tests, couldn't we do it like gdb does, e.g. save *.s/*.S
> files and assemble/link those before using them as tests?
> 
> That is of course not possible for the huge binaries.
> For those ftp sounds reasonable to me, or indeed git-lfs (but in a separate
> repo like dwz-large-tests).

And for those large binaries perhaps storing them xz -9e compressed or
similar (if the compression ratio is significant) and uncompressing during
testing (perhaps with some caching, if one is developing it and running
make; make check; fix bug; make; make check it would be nice not to
decompress them each time) might be a good idea.

	Jakub

Re: dwz testsuite with binaries

[Carlos O'Donell]

On 3/15/19 1:45 PM, Jakub Jelinek wrote:
> On Fri, Mar 15, 2019 at 06:43:19PM +0100, Jakub Jelinek wrote:
>> On Fri, Mar 15, 2019 at 06:37:50PM +0100, Tom de Vries wrote:
>>> On 15-03-19 18:11, Frank Ch. Eigler wrote:
>>>> Hi -
>>>>
>>>>> [...]  Due to the size of some of the binaries, it's separate from
>>>>> the dwz git repo on sourceware.org.
>>>>
>>>> Other than those 200MB+ cc1 binaries
>>>
>>> FWIW, I'm already preparing an update that contains 1.1 GB worth of files.
>>>
>>>> I see no problem with putting
>>>> them into git.
>>>
>>> I'll leave that to Jakub to decide.
>>
>> I'd prefer not to have the large (but ideally any binaries) in the main dwz
>> repo, so that people can just download the half a MB of sources and don't
>> have to download thousand times more if they don't want to.
>> For the small tests, couldn't we do it like gdb does, e.g. save *.s/*.S
>> files and assemble/link those before using them as tests?
>>
>> That is of course not possible for the huge binaries.
>> For those ftp sounds reasonable to me, or indeed git-lfs (but in a separate
>> repo like dwz-large-tests).
> 
> And for those large binaries perhaps storing them xz -9e compressed or
> similar (if the compression ratio is significant) and uncompressing during
> testing (perhaps with some caching, if one is developing it and running
> make; make check; fix bug; make; make check it would be nice not to
> decompress them each time) might be a good idea.

Right, noboby wants to see these files in the *main* repository of their
project, they are ancilliary data for deeper testing by senior developers,
perhaps once every 6 months when doing a release (talking about glibc here).

-- 
Cheers,
Carlos.