gprofng not quite ready for prime time?

gprofng not quite ready for prime time?

[Jan Beulich]

Initially I merely noticed version.texi trying to be created in the
source tree, which of course won't work if that one is read-only.

But then I further noticed that the testsuite fails to build and/or
run various things, first and foremost because of (but not
necessarily limited to) Java not being available. Despite (or maybe
as a result of) this the running of the (mostly failing) testsuite
took a non-negligible amount of time for the just 5 tests (on
x86-64).

Should this component perhaps not have been part of binutils 2.39,
or at least not be enabled by default?

Jan

Re: gprofng not quite ready for prime time?

[Luis Machado]

On 8/9/22 14:57, Jan Beulich via Binutils wrote:
> Initially I merely noticed version.texi trying to be created in the
> source tree, which of course won't work if that one is read-only.
> 
> But then I further noticed that the testsuite fails to build and/or
> run various things, first and foremost because of (but not
> necessarily limited to) Java not being available. Despite (or maybe
> as a result of) this the running of the (mostly failing) testsuite
> took a non-negligible amount of time for the just 5 tests (on
> x86-64).

The tests do seem to take a little while to run, as confirmed by https://sourceware.org/pipermail/binutils/2022-May/121022.html

> 
> Should this component perhaps not have been part of binutils 2.39,
> or at least not be enabled by default?
> 
> Jan

Re: gprofng not quite ready for prime time?

[Ruud van der Pas]

Hi Jan,

Thanks for sharing your feedback. Comments below.

> Initially I merely noticed version.texi trying to be created in the
> source tree, which of course won't work if that one is read-only.

This is oversight. I will create a bugzilla bug report and address this
right away.

> But then I further noticed that the testsuite fails to build and/or
> run various things, first and foremost because of (but not
> necessarily limited to) Java not being available.

This work was done by Vladimir. He is currently away, but we will look
into this no later than this week.

> Despite (or maybe
> as a result of) this the running of the (mostly failing) testsuite
> took a non-negligible amount of time for the just 5 tests (on
> x86-64).

This is in the nature of the beast. As Luis already mentioned, we run
the tests in a fixed amount of time. We do so to get enough samples to
verify the functionality.

We are definitely interested in addressing this if it is considered to
take too much time. For example, make the fixed time a user tunable, with
a certain default, or alternatively provide feedback on the expected
duration of the test(s) to the builder. Any other suggestion is welcome!

Kind regards, Ruud

Re: gprofng not quite ready for prime time?

[Jan Beulich]

On 09.08.2022 20:17, Ruud van der Pas wrote:
>> Initially I merely noticed version.texi trying to be created in the
>> source tree, which of course won't work if that one is read-only.
> 
> This is oversight. I will create a bugzilla bug report and address this
> right away.

Thanks.

>> But then I further noticed that the testsuite fails to build and/or
>> run various things, first and foremost because of (but not
>> necessarily limited to) Java not being available.
> 
> This work was done by Vladimir. He is currently away, but we will look
> into this no later than this week.
> 
>> Despite (or maybe
>> as a result of) this the running of the (mostly failing) testsuite
>> took a non-negligible amount of time for the just 5 tests (on
>> x86-64).
> 
> This is in the nature of the beast. As Luis already mentioned, we run
> the tests in a fixed amount of time. We do so to get enough samples to
> verify the functionality.
> 
> We are definitely interested in addressing this if it is considered to
> take too much time. For example, make the fixed time a user tunable, with
> a certain default, or alternatively provide feedback on the expected
> duration of the test(s) to the builder. Any other suggestion is welcome!

Earlier this year (iirc) some overly long lasting gas tests have been
disabled / altered. None of those took minutes. So I'm inclined to say
that anything which takes longer than a couple of seconds should be
off by default. Of course the threshold lowers as more tests are added,
unless a way is found to parallelize their running.

Jan

Re: gprofng not quite ready for prime time?

[Jose E. Marchesi]

> On 09.08.2022 20:17, Ruud van der Pas wrote:
>>> Initially I merely noticed version.texi trying to be created in the
>>> source tree, which of course won't work if that one is read-only.
>> 
>> This is oversight. I will create a bugzilla bug report and address this
>> right away.
>
> Thanks.
>
>>> But then I further noticed that the testsuite fails to build and/or
>>> run various things, first and foremost because of (but not
>>> necessarily limited to) Java not being available.
>> 
>> This work was done by Vladimir. He is currently away, but we will look
>> into this no later than this week.
>> 
>>> Despite (or maybe
>>> as a result of) this the running of the (mostly failing) testsuite
>>> took a non-negligible amount of time for the just 5 tests (on
>>> x86-64).
>> 
>> This is in the nature of the beast. As Luis already mentioned, we run
>> the tests in a fixed amount of time. We do so to get enough samples to
>> verify the functionality.
>> 
>> We are definitely interested in addressing this if it is considered to
>> take too much time. For example, make the fixed time a user tunable, with
>> a certain default, or alternatively provide feedback on the expected
>> duration of the test(s) to the builder. Any other suggestion is welcome!
>
> Earlier this year (iirc) some overly long lasting gas tests have been
> disabled / altered. None of those took minutes. So I'm inclined to say
> that anything which takes longer than a couple of seconds should be
> off by default. Of course the threshold lowers as more tests are added,
> unless a way is found to parallelize their running.

I think this is a sensible approach.

We could have an additional check-expensive (or similar) target to run
the full testsuite.

Re: gprofng not quite ready for prime time?

[Ruud van der Pas]

Hi Jan and José,

Thanks for your postings on this.

>> Earlier this year (iirc) some overly long lasting gas tests have been
>> disabled / altered. None of those took minutes. So I'm inclined to say
>> that anything which takes longer than a couple of seconds should be
>> off by default. Of course the threshold lowers as more tests are added,
>> unless a way is found to parallelize their running.
> 
> I think this is a sensible approach.

I think so too.

> We could have an additional check-expensive (or similar) target to run
> the full testsuite.

Right. Maybe not associate this with money though ;-) 

Perhaps something like "check-extensive" and issue a message how long
we expect those tests to run. Certainly also provide feedback. 

Nothing worse than a terminal prompt that doesn't re-appear for a
long long time after all!

We're very keen to address this and as a starting point, I just submitted
a bug report on this:

Bug 29470 - [test suite] The test suite should be made more flexible

https://sourceware.org/bugzilla/show_bug.cgi?id=29470

I also submitted a bug report against the version.texi file. In retrospect,
I think it is overkill to generate this file. We should just make sure
to update it with the new info when we release a doc patch.

This is the bug id by the way:

Bug 29465 - [docs] File version.texi is created in the binutils source directory 

https://sourceware.org/bugzilla/show_bug.cgi?id=29465

Kind regards, Ruud

Re: gprofng not quite ready for prime time?

[Jan Beulich]

On 11.08.2022 13:43, Ruud van der Pas wrote:
> I also submitted a bug report against the version.texi file. In retrospect,
> I think it is overkill to generate this file. We should just make sure
> to update it with the new info when we release a doc patch.

Hmm, do you mean one would need to remember to update a certain file (or
a certain line in a file) for every doc change? That's exactly what one
would want to avoid I would say, and hence a good reason to have the
file generated (no matter how small / trivial it is).

Jan

Re: gprofng not quite ready for prime time?

[Ruud van der Pas]

Hi Jan,

> Hmm, do you mean one would need to remember to update a certain file (or
> a certain line in a file) for every doc change? That's exactly what one
> would want to avoid I would say, and hence a good reason to have the
> file generated (no matter how small / trivial it is).

That was my original intention indeed, but if I dynamically set the
date in version.texi, or in some other place, the user will get a
date in the document that depends on when they build binutils.

I'm definitely open for suggestions how to address this!

Kind regards, Ruud

Re: gprofng not quite ready for prime time?

[Jan Beulich]

On 11.08.2022 16:02, Ruud van der Pas wrote:
>> Hmm, do you mean one would need to remember to update a certain file (or
>> a certain line in a file) for every doc change? That's exactly what one
>> would want to avoid I would say, and hence a good reason to have the
>> file generated (no matter how small / trivial it is).
> 
> That was my original intention indeed, but if I dynamically set the
> date in version.texi, or in some other place, the user will get a
> date in the document that depends on when they build binutils.
> 
> I'm definitely open for suggestions how to address this!

Take the time stamp of the newest file in the doc dir (or a subset thereof)?
With how git works, this may of course still not be what you want, so the
next approach then would be to actually query git (as long as you're in a
git tree) for the newest commit's date touching any of the relevant files.
Sounds all quite a bit more involved than simply using the time when the
build was done.

Jan

Re: gprofng not quite ready for prime time?

[Ruud van der Pas]

Hi Jan,

> Take the time stamp of the newest file in the doc dir (or a subset thereof)?
> With how git works, this may of course still not be what you want, so the
> next approach then would be to actually query git (as long as you're in a
> git tree) for the newest commit's date touching any of the relevant files.
> Sounds all quite a bit more involved than simply using the time when the
> build was done.

Thanks for the suggestions.

It is indeed somewhat tricky. Let me check what others do regarding
the docs in binutils. If there is sufficient commonality in their approach,
I can just do the same, or at least do something similar.

Kind regards, Ruud