About FAILs in the internal testsuite

About FAILs in the internal testsuite

[peter wang]

Hello,

I run the internal testsuite in CentOS 6.7 and got some FAILs in the result.
Then I got serveral questions:
1. Is the testsuite designed for all the kernel version?

2. Is it bug which need to be fixed that leads to some FAILs in some
environment?

3. In which system can I get no FAIL with the internal testsuite?

Can anyone help me?

-- 
Thanks
Wang

Re: About FAILs in the internal testsuite

[David Smith]

On 04/13/2016 07:17 PM, peter wang wrote:
> Hello,
> 
> I run the internal testsuite in CentOS 6.7 and got some FAILs in the result.
> Then I got serveral questions:
> 1. Is the testsuite designed for all the kernel version?

Yes, it is supposed to be. If a feature is needed we're supposed to test
that it exists before running the test (or marking the test as a 'KFAIL'
- a known failure).

However, that doesn't always happen.

> 2. Is it bug which need to be fixed that leads to some FAILs in some
> environment?

The answer here is "it varies". Some of the failures you see are test
case failures, some are actual failures.

We'd be *happy* to get some help in fixing failures.

> 3. In which system can I get no FAIL with the internal testsuite?

There isn't any os/arch combination that will receive 0 failures. On
x86_64 (probably our most tested architecture) and a semi-recent kernels
(such as RHEL7's 3.10 or a more current Fedora release), you'll get in
the neighborhood of approximately 100 unexpected failures. I've done a
fair amount of work in the past several months getting the failures down
to that number.

Why isn't that number 0? Several reasons, one of the biggest being gcc
debuginfo quality. Sometimes because of poor debuginfo, we can't find a
function's parameters (for instance).

Note that the number of unexpected failures can vary wildly per
architecture.

> Can anyone help me?

I hope I've answered your questions. As I said earlier, we'd be happy to
have help in debugging and fixing up the remaining testsuite failures.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: About FAILs in the internal testsuite

[peter wang]

Hi David,

Thanks for your answer.
It really helps me a lot.

-- 
Thanks
Wang
On Fri, Apr 15, 2016 at 2:15 AM, David Smith <dsmith@redhat.com> wrote:
> On 04/13/2016 07:17 PM, peter wang wrote:
>> Hello,
>>
>> I run the internal testsuite in CentOS 6.7 and got some FAILs in the result.
>> Then I got serveral questions:
>> 1. Is the testsuite designed for all the kernel version?
>
> Yes, it is supposed to be. If a feature is needed we're supposed to test
> that it exists before running the test (or marking the test as a 'KFAIL'
> - a known failure).
>
> However, that doesn't always happen.
>
>> 2. Is it bug which need to be fixed that leads to some FAILs in some
>> environment?
>
> The answer here is "it varies". Some of the failures you see are test
> case failures, some are actual failures.
>
> We'd be *happy* to get some help in fixing failures.
>
>> 3. In which system can I get no FAIL with the internal testsuite?
>
> There isn't any os/arch combination that will receive 0 failures. On
> x86_64 (probably our most tested architecture) and a semi-recent kernels
> (such as RHEL7's 3.10 or a more current Fedora release), you'll get in
> the neighborhood of approximately 100 unexpected failures. I've done a
> fair amount of work in the past several months getting the failures down
> to that number.
>
> Why isn't that number 0? Several reasons, one of the biggest being gcc
> debuginfo quality. Sometimes because of poor debuginfo, we can't find a
> function's parameters (for instance).
>
> Note that the number of unexpected failures can vary wildly per
> architecture.
>
>> Can anyone help me?
>
> I hope I've answered your questions. As I said earlier, we'd be happy to
> have help in debugging and fixing up the remaining testsuite failures.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: About FAILs in the internal testsuite

[peter wang]

Hello David,

Thanks again for your reply.
There is one more thing which confused me.

I'm not sure if I have some misunderstandings.
Some failures are resulted by either case's bug or systemtap's bug and
others should be set to KFAIL.

In other words, all FAIL should be fixed whatever arch or system is.
Is that right?

-- 
Thanks
Wang

On Fri, Apr 15, 2016 at 2:15 AM, David Smith <dsmith@redhat.com> wrote:
> On 04/13/2016 07:17 PM, peter wang wrote:
>> Hello,
>>
>> I run the internal testsuite in CentOS 6.7 and got some FAILs in the result.
>> Then I got serveral questions:
>> 1. Is the testsuite designed for all the kernel version?
>
> Yes, it is supposed to be. If a feature is needed we're supposed to test
> that it exists before running the test (or marking the test as a 'KFAIL'
> - a known failure).
>
> However, that doesn't always happen.
>
>> 2. Is it bug which need to be fixed that leads to some FAILs in some
>> environment?
>
> The answer here is "it varies". Some of the failures you see are test
> case failures, some are actual failures.
>
> We'd be *happy* to get some help in fixing failures.
>
>> 3. In which system can I get no FAIL with the internal testsuite?
>
> There isn't any os/arch combination that will receive 0 failures. On
> x86_64 (probably our most tested architecture) and a semi-recent kernels
> (such as RHEL7's 3.10 or a more current Fedora release), you'll get in
> the neighborhood of approximately 100 unexpected failures. I've done a
> fair amount of work in the past several months getting the failures down
> to that number.
>
> Why isn't that number 0? Several reasons, one of the biggest being gcc
> debuginfo quality. Sometimes because of poor debuginfo, we can't find a
> function's parameters (for instance).
>
> Note that the number of unexpected failures can vary wildly per
> architecture.
>
>> Can anyone help me?
>
> I hope I've answered your questions. As I said earlier, we'd be happy to
> have help in debugging and fixing up the remaining testsuite failures.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: About FAILs in the internal testsuite

[David Smith]

On 04/15/2016 06:03 AM, peter wang wrote:
> Hello David,
> 
> Thanks again for your reply.
> There is one more thing which confused me.
> 
> I'm not sure if I have some misunderstandings.
> Some failures are resulted by either case's bug or systemtap's bug and
> others should be set to KFAIL.
> 
> In other words, all FAIL should be fixed whatever arch or system is.
> Is that right?

All FAILs should certainly be investigated to see out what is going on.
There have been times when I've done that and left the FAIL in place.
Here's why. In the past, I've found a few KFAIL tests that were no
longer failing because of the orignal reason (most likely bad
debuginfo), but failing because the underlying probe point has changed.

If the problem is truly a problem in systemtap, it would be great to get
it fixed.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: About FAILs in the internal testsuite

[peter wang]

On Fri, Apr 15, 2016 at 10:17 PM, David Smith <dsmith@redhat.com> wrote:
> On 04/15/2016 06:03 AM, peter wang wrote:
>> Hello David,
>>
>> Thanks again for your reply.
>> There is one more thing which confused me.
>>
>> I'm not sure if I have some misunderstandings.
>> Some failures are resulted by either case's bug or systemtap's bug and
>> others should be set to KFAIL.
>>
>> In other words, all FAIL should be fixed whatever arch or system is.
>> Is that right?
>
> All FAILs should certainly be investigated to see out what is going on.
> There have been times when I've done that and left the FAIL in place.
> Here's why. In the past, I've found a few KFAIL tests that were no
> longer failing because of the orignal reason (most likely bad
> debuginfo), but failing because the underlying probe point has changed.
>

Does that mean most of the FAILs have been investigated actually?
Since investigated FAILs are still left, it is hard for others to find
the real bug, I think.
Is there any docs recording or describing the investigated FAILs?

-- 
Thanks
Wang

> If the problem is truly a problem in systemtap, it would be great to get
> it fixed.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: About FAILs in the internal testsuite

[David Smith]

On 04/15/2016 10:52 PM, peter wang wrote:
> On Fri, Apr 15, 2016 at 10:17 PM, David Smith <dsmith@redhat.com> wrote:
>> On 04/15/2016 06:03 AM, peter wang wrote:
>>> Hello David,
>>>
>>> Thanks again for your reply.
>>> There is one more thing which confused me.
>>>
>>> I'm not sure if I have some misunderstandings.
>>> Some failures are resulted by either case's bug or systemtap's bug and
>>> others should be set to KFAIL.
>>>
>>> In other words, all FAIL should be fixed whatever arch or system is.
>>> Is that right?
>>
>> All FAILs should certainly be investigated to see out what is going on.
>> There have been times when I've done that and left the FAIL in place.
>> Here's why. In the past, I've found a few KFAIL tests that were no
>> longer failing because of the orignal reason (most likely bad
>> debuginfo), but failing because the underlying probe point has changed.
>>
> 
> Does that mean most of the FAILs have been investigated actually?
> Since investigated FAILs are still left, it is hard for others to find
> the real bug, I think.
> Is there any docs recording or describing the investigated FAILs?

The remaining FAILs could *certainly* use more investigation. Earlier,
when I was talking about leaving KFAILs I was really talking about
testsuite/systemtap.pass1-4/buildok.exp. That why you'll see the
following comment in that test:

       # (Note that if a test fails because of debuginfo-quality
       # problems, don't kfail a tapset's entire test. Extract
       # the failing probe/function's test into a new file and
       # kfail that.)

I'll certainly be happy for help in figuring out the remaining FAILs.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)