[Bug testsuite/64983] New: Incomplete summary when regtesting with dejagnu 1.5.2.

[Bug testsuite/64983] New: Incomplete summary when regtesting with dejagnu 1.5.2.

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

            Bug ID: 64983
           Summary: Incomplete summary when regtesting with dejagnu 1.5.2.
           Product: gcc
           Version: 5.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: testsuite
          Assignee: unassigned at gcc dot gnu.org
          Reporter: dominiq at lps dot ens.fr

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2015-02-09
                 CC|                            |howarth at bromo dot med.uc.edu,
                   |                            |iains at gcc dot gnu.org,
                   |                            |mrs at gcc dot gnu.org
     Ever confirmed|0                           |1

--- Comment #1 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
With dejagnu 1.5.1, the summary looks like

...
        === g++ tests ===


Running target unix/-m32
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++11  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++14  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++98  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/pch/pch.C  -O2 -I. -Dwith_PCH (test for excess errors)
FAIL: g++.dg/pch/pch.C -O2 assembly comparison
WARNING: program timed out.
FAIL: g++.dg/tm/pr47573.C  -std=gnu++11 (test for excess errors)
FAIL: g++.dg/tm/pr47573.C  -std=gnu++14 (test for excess errors)
FAIL: g++.dg/tm/pr47573.C  -std=gnu++98 (test for excess errors)

        === g++ Summary for unix/-m32 ===

# of expected passes        88065
# of unexpected failures    8
# of expected failures        320
# of unsupported tests        3647

Running target unix/-m64
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++11  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++14  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++98  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/pch/pch.C  -O2 -g -I. -Dwith_PCH (test for excess errors)
FAIL: g++.dg/pch/pch.C  -g -I. -Dwith_PCH (internal compiler error)
FAIL: g++.dg/pch/pch.C  -g -I. -Dwith_PCH (test for excess errors)
FAIL: g++.dg/pch/pch.C  -g assembly comparison
FAIL: g++.dg/pch/pch.C -O2 -g assembly comparison
WARNING: program timed out.
FAIL: g++.dg/tm/pr47573.C  -std=gnu++11 (test for excess errors)
FAIL: g++.dg/tm/pr47573.C  -std=gnu++14 (test for excess errors)
FAIL: g++.dg/tm/pr47573.C  -std=gnu++98 (test for excess errors)

        === g++ Summary for unix/-m64 ===

# of expected passes        89465
# of unexpected failures    11
# of expected failures        320
# of unsupported tests        3888

        === g++ Summary ===

# of expected passes        177530
# of unexpected failures    19
# of expected failures        640
# of unsupported tests        7535
/opt/gcc/build_w/gcc/testsuite/g++/../../xg++  version 5.0.0 20150207
(experimental) [trunk revision 220502p20] (GCC) 

...

as in https://gcc.gnu.org/ml/gcc-testresults/2015-02/msg00877.html.

With dejagnu 1.5.2, it looks like

...
        === g++ tests ===


Running target unix/-m32
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++11  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++14  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++98  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/tm/pr47573.C  -std=gnu++11 (test for excess errors)
FAIL: g++.dg/tm/pr47573.C  -std=gnu++14 (test for excess errors)
FAIL: g++.dg/tm/pr47573.C  -std=gnu++98 (test for excess errors)

        === g++ Summary for unix/-m32 ===

# of expected passes        20311
# of expected failures        61
# of unsupported tests        62

Running target unix/-m64
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++11  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++14  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/debug/dwarf2/imported-decl-2.C  -std=gnu++98  scan-assembler-times
ascii "AAAA\\\\\\\\0".*ascii "BBBB\\\\\\\\0".*DIE .0x[0-9a-f]*.
DW_TAG_imported_declaration 1
FAIL: g++.dg/tm/pr47573.C  -std=gnu++11 (test for excess errors)
FAIL: g++.dg/tm/pr47573.C  -std=gnu++14 (test for excess errors)
FAIL: g++.dg/tm/pr47573.C  -std=gnu++98 (test for excess errors)

        === g++ Summary for unix/-m64 ===

# of expected passes        20191
# of expected failures        61
# of unsupported tests        80

        === g++ Summary ===

# of expected passes        40502
# of expected failures        122
# of unsupported tests        142
/opt/gcc/build_w/gcc/testsuite/g++/../../xg++  version 5.0.0 20150207
(experimental) [trunk revision 220502p20] (GCC) 

...

as in https://gcc.gnu.org/ml/gcc-testresults/2015-02/msg01002.html.

AFAICT the *.log files contain the right information which are not processed
the same way by dejagnu 1.5.1 and 1.5.2 when using contrib/test_summary.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #2 from howarth at bromo dot med.uc.edu ---
I am not seeing identical g++.log's being created here from dejagnu 1.5.1 and
1.5.2 on x86_64-apple-darwin14 with separate runs of...

make -k check RUNTESTFLAGS="--target_board=unix'{-m32,-m64}'"

on the same build of gcc trunk. Furthermore, running contrib/test_summary under
dejagnu 1.5.1 against the g++.log created under dejagnu 1.5.2 doesn't eliminate
the bad summary  results reported.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[iains at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #3 from Iain Sandoe <iains at gcc dot gnu.org> ---
running with dejaGNU 1.6 also produces the wrong output.

I did a small amount of analysis - and it looks like the content of the
xxx.sum.sep files is not what's expected by the combiner script.

*guess* that there's some difference in the format of the output - and the
dg-extact-results is failing to match things properly.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #4 from howarth at bromo dot med.uc.edu ---
FYI, I posted this to
http://lists.gnu.org/archive/html/bug-dejagnu/2015-02/msg00001.html and emailed
Ben Elliston the g++.log files generated under dejagnu 1.5.1 and 1.5.2.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[iains at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #5 from Iain Sandoe <iains at gcc dot gnu.org> ---
(In reply to howarth from comment #4)
> FYI, I posted this to
> http://lists.gnu.org/archive/html/bug-dejagnu/2015-02/msg00001.html and
> emailed Ben Elliston the g++.log files generated under dejagnu 1.5.1 and
> 1.5.2.

well, as I indicated above, I suspect the interesting files are the
*.{sum,log}.sep (and only one set for one language should be enough to figure
out the problem).  However, the content isn't stable… (my analysis stopped at
that point, and haven't had time to resume).

IIRC, parallel make-check-objc is the shortest test-suite that shows the issue.
>From gcc-bugs-return-477722-listarch-gcc-bugs=gcc.gnu.org@gcc.gnu.org Wed Feb 18 20:22:10 2015
Return-Path: <gcc-bugs-return-477722-listarch-gcc-bugs=gcc.gnu.org@gcc.gnu.org>
Delivered-To: listarch-gcc-bugs@gcc.gnu.org
Received: (qmail 24634 invoked by alias); 18 Feb 2015 20:22:10 -0000
Mailing-List: contact gcc-bugs-help@gcc.gnu.org; run by ezmlm
Precedence: bulk
List-Id: <gcc-bugs.gcc.gnu.org>
List-Archive: <http://gcc.gnu.org/ml/gcc-bugs/>
List-Post: <mailto:gcc-bugs@gcc.gnu.org>
List-Help: <mailto:gcc-bugs-help@gcc.gnu.org>
Sender: gcc-bugs-owner@gcc.gnu.org
Delivered-To: mailing list gcc-bugs@gcc.gnu.org
Received: (qmail 24566 invoked by uid 48); 18 Feb 2015 20:22:06 -0000
From: "jvdelisle at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>
To: gcc-bugs@gcc.gnu.org
Subject: [Bug fortran/64432] [5 Regression] SYSTEM_CLOCK(COUNT_RATE=rate) wrong result for integer(4)::rate
Date: Wed, 18 Feb 2015 20:22:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: gcc
X-Bugzilla-Component: fortran
X-Bugzilla-Version: 5.0
X-Bugzilla-Keywords: wrong-code
X-Bugzilla-Severity: normal
X-Bugzilla-Who: jvdelisle at gcc dot gnu.org
X-Bugzilla-Status: NEW
X-Bugzilla-Priority: P4
X-Bugzilla-Assigned-To: fxcoudert at gcc dot gnu.org
X-Bugzilla-Target-Milestone: 5.0
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields:
Message-ID: <bug-64432-4-cx4PzJOOER@http.gcc.gnu.org/bugzilla/>
In-Reply-To: <bug-64432-4@http.gcc.gnu.org/bugzilla/>
References: <bug-64432-4@http.gcc.gnu.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://gcc.gnu.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2015-02/txt/msg02055.txt.bz2
Content-length: 633

https://gcc.gnu.org/bugzilla/show_bug.cgi?idd432

--- Comment #24 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
(In reply to Harald Anlauf from comment #22)
>  count_rate(8),count_max(1) =                    0  127
>
> OK, but the last line looks strange: lacking documentation,
> I'd expect the rate to be 1, not 0.  (Not that I'd use that
> in real code...).

Yes, tha last one is wrong. I will look into it.

I realize that by adding an additional argument we modify the interface so we
may have to do something with symbol versioning.  Doing it the way I did
eliminates multiple calls but there are always trade-offs

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #6 from howarth at bromo dot med.uc.edu ---
This same bug has also been reported on dejagnu mailing list at...

http://lists.gnu.org/archive/html/dejagnu/2015-02/msg00000.html

and claimed to be pinpointed to /usr/share/dejagnu/remote.exp...

http://lists.gnu.org/archive/html/dejagnu/2015-03/msg00002.html

Since there has been no changes to that file since 2013, I wonder if the buffer
overflow is really just the manifestation of the bug and not the actual cause.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #7 from howarth at bromo dot med.uc.edu ---
A regression hunt of dejagnu master revealed that the offending commit causing
this bug was...

http://git.savannah.gnu.org/cgit/dejagnu.git/commit/?id=6d2e2d3791bcea70131a6cf64a0a5223333a7b8e

2011-03-14  Jan Kratochvil  <jan.kratochvil@redhat.com>

    * runtest.exp (runtest): Call reset_vars at initialisation so that
    a perror does not affect test results in other test scripts.

and that reverting this change in 1.5.2 restores the expected summary output.
    This issue is actually predates 1.5.1 but isn't present there because the
original 1.5 branch tree was reused for the 1.5.1 release development whereas
the 1.5.2 release was branched directly off of master.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[howarthjw at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

Jack Howarth <howarthjw at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |howarthjw at gmail dot com

--- Comment #8 from Jack Howarth <howarthjw at gmail dot com> ---
The usage of reset_vars in proc runtest results in the execution of...

    foreach x $test_names {
        set test_counts($x,count) 0
    }

Reverting the commit while effectively retaining that part of the changes
with...

diff -uNr dejagnu-1.5.2.orig/runtest.exp dejagnu-1.5.2/runtest.exp
--- dejagnu-1.5.2.orig/runtest.exp    2015-02-03 21:25:26.000000000 -0500
+++ dejagnu-1.5.2/runtest.exp    2015-03-28 16:48:10.000000000 -0400
@@ -1441,13 +1441,22 @@
 #

 proc runtest { test_file_name } {
+    global test_names test_counts
+    global prms_id
+    global bug_id
     global test_result
     global errcnt
     global errorInfo
     global tool

     clone_output "Running $test_file_name ..."
-    reset_vars
+    set prms_id    0
+    set bug_id    0
+
+    foreach x $test_names {
+    set test_counts($x,count) 0
+    }
+
     set test_result ""

     if {[file exists $test_file_name]} {

...replicates the breakage.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[howarth.at.gcc at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #9 from Jack Howarth <howarth.at.gcc at gmail dot com> ---
Note that this issue seems to only be fixable from within dejagnu and that the
commit of 6d2e2d3791bcea70131a6cf64a0a5223333a7b8e effectively broke the
log_summary proc in dejagnu's lib/framework.exp by reseting the testcnt global
which is used as follows...

    # If the tool set `testcnt', it wants us to do a sanity check on the
    # total count, so compare the reported number of testcases with the
    # expected number.  Maintaining an accurate count in `testcnt' isn't easy
    # so it's not clear how often this will be used.
    if {[info exists testcnt]} {
        if { $testcnt > 0 } {
            set totlcnt 0
            # total all the testcases reported
            foreach x { FAIL PASS XFAIL KFAIL XPASS KPASS UNTESTED UNRESOLVED
UNSUPPORTED } {
                incr totlcnt test_counts($x,$which)
            }
            set testcnt test_counts(total,$which)

            if { $testcnt>$totlcnt || $testcnt<$totlcnt } {
                if { $testcnt > $totlcnt } {
                    set mismatch "unreported  [expr {$testcnt - $totlcnt}]"
                }
                if { $testcnt < $totlcnt } {
                    set mismatch "misreported [expr {$totlcnt - $testcnt}]"
                }
            } else {
                verbose "# of testcases run         $testcnt"
            }

            if {[info exists mismatch]} {
                clone_output "### ERROR: totals do not equal number of
testcases run"
                clone_output "### ERROR: # of testcases expected    $testcnt"
                clone_output "### ERROR: # of testcases reported    $totlcnt"
                clone_output "### ERROR: # of testcases $mismatch\n"
            }
        }
    }
    foreach x { PASS FAIL XPASS XFAIL KPASS KFAIL UNRESOLVED UNTESTED
UNSUPPORTED } {
        set val $test_counts($x,$which)
        if { $val > 0 } {
            set mess "# of $test_counts($x,name)"
            if { $xml } {
                xml_output "  <summary>"
                xml_output "    <result>$x</result>"
                xml_output "    <description>$mess</description>"
                xml_output "    <total>$val</total>"
                xml_output "  </summary>"
            }
            if { [string length $mess] < 24 } {
                append mess "\t"
            }
            clone_output "$mess\t$val"
        }

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #10 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
I confirm that the dejagnu 1.5.2-3 packaging in fink restores the correct
results.

Thanks for the debugging.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[howarth.at.gcc at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

--- Comment #11 from Jack Howarth <howarth.at.gcc at gmail dot com> ---
(In reply to Dominique d'Humieres from comment #10)
> I confirm that the dejagnu 1.5.2-3 packaging in fink restores the correct
> results.
> 
> Thanks for the debugging.

Fixed in dejagnu 1.5.3.

[Bug testsuite/64983] Incomplete summary when regtesting with dejagnu 1.5.2.

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64983

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |MOVED

--- Comment #12 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
> Fixed in dejagnu 1.5.3.

So closing as MOVED (dejagnu bug).