From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-il1-x132.google.com (mail-il1-x132.google.com [IPv6:2607:f8b0:4864:20::132]) by sourceware.org (Postfix) with ESMTPS id 0B0EA3858D1E for ; Wed, 20 Dec 2023 17:17:12 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0B0EA3858D1E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 0B0EA3858D1E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::132 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1703092633; cv=none; b=RRl2z37OcC/4AiKFui5RjJ/uuVL86WTRQWYVZ+kK2sv+jQOHTQH0J4k80D3j9CN6y7ocBLwjBXTIvWz45DRzel2cQ0IVylAomjzbPzNQ1JXhuMkF7LOEytbi0NmxAClu1pdS1w5OUpIJFbQOeXSyrIU56splwcdDSxN1fEdKdAA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1703092633; c=relaxed/simple; bh=H7rLwg4HL1Up2h1tz7S2q27lN293RKtlvVetFR7ghH4=; h=DKIM-Signature:Message-ID:Date:MIME-Version:Subject:To:From; b=eFQ01PfuO2TuWASAvGe5OYiK9odNVqPZIwbhTedbhxAGiYeIS3WxuC3dNIFIDpuq6lPSV6mjpw5VmJXeLC4H+dVIMPCVOojqaAqXMICMthhr0OKG5j1qIRPo0gK0WXeORlJc6gC14uIOfid9bUUqTtG7Lwsz5FdGnpf07XRS0D0= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-il1-x132.google.com with SMTP id e9e14a558f8ab-35d74cf427cso23429445ab.1 for ; Wed, 20 Dec 2023 09:17:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703092631; x=1703697431; darn=gcc.gnu.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=AD2bufjXKNwWWTznaXHy+JGWOpUUOWhbBS4plxOxlC0=; b=g2EUc86aYVdB/Tx5/V8IqCUT/+d0lxxdL3rWqNgALrpTMrR2q3nWAQ51SPcwAFPum/ c8hgVNl+E6/uH6OZB6wwR+0dQSw4OVjc8wae+TQqLxcHB3hPYzO1aEeXK2bJdZKKKq3V tTz0kI70EAkVYhrr/r2y2nz5X0GlVAfEYEgLWEYQeEihq8eAMUjipcYRtYJ9s1lwFCNu ZPt639AwcF0fquXeQZtq0LkMtdOa0GpwCqdSjUS3MDTqISSzvGV2jakNlR5ZlvXMBkYJ aQfw6G+Z5PHJ8KFY76g9OYeEQ2ZCd4X0RFIo+bNOEBi7syNNX2c8VAJXtPnEF+FiBfIs 1mpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703092631; x=1703697431; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AD2bufjXKNwWWTznaXHy+JGWOpUUOWhbBS4plxOxlC0=; b=QkkkEdIhC0faCnIVQ2kZmEmX7cMwJxF5SiiZGVQvpQlJJ/UfTw/qy70p1tKlWxuK/I rA9tZgTQ5iu5Sbl/0T9pvZUhTF5hBFOkEVDshCm4EJxB85xHKgiDT4mgQ1K3ZTwQfyPp MbPeX8p37K0jYKLviq/8ZEF8jxFHF0TSnszHRWcwaqF1jwXOAgILm18zJnumYDdgv+TG /31nOFDWmW38zNoAXku3uTihd7Hj8uthjdh4n6wb1mvdXxeYVPdVYXiOAX80OT++oYvj XMPz1Qek5mUnP7KuBo0TQYPc0a2T5KQazC2bk49w/3yJsQ24IXI/lAzBjs366eK08mqI MGiA== X-Gm-Message-State: AOJu0YxQ2OQA/kOhTjyCC04xEArLGi6DXCI77AS+mZzaWR2mkEQDt8+e szPTrT2pR11woegqf6qdnViwLBkB1QI= X-Google-Smtp-Source: AGHT+IE4mFm00kLoFFS1T35ulnzLmyvAAY+1Th3bk+GCH9+I7sf8V0gbenCvk9+7MuVheWxue/3qSw== X-Received: by 2002:a05:6e02:1b06:b0:35d:484f:c0f9 with SMTP id i6-20020a056e021b0600b0035d484fc0f9mr25749281ilv.27.1703092630994; Wed, 20 Dec 2023 09:17:10 -0800 (PST) Received: from [172.31.0.109] ([136.36.72.243]) by smtp.gmail.com with ESMTPSA id bg4-20020a056e02310400b0035f753c0715sm1314040ilb.33.2023.12.20.09.17.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 20 Dec 2023 09:17:10 -0800 (PST) Message-ID: <1cd3fee7-2a0c-4e33-91c2-4559286adeb9@gmail.com> Date: Wed, 20 Dec 2023 10:17:08 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] compare_tests: distinguish c-c++-common results by tool Content-Language: en-US To: Alexandre Oliva Cc: "gcc-patches@gcc.gnu.org" References: <76a02442-290a-4929-b566-035753afd859@gmail.com> From: Jeff Law In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 12/19/23 23:31, Alexandre Oliva wrote: > On Dec 20, 2023, Jeff Law wrote: > >> So the strub tests in c-c++-common are problematical. They get run >> twice, once for C, once for C++. Yet the name of the test is the same >> in both runs. (by the name, I mean the name emitted into the dejagnu >> summary and log files). > >> Thus if you have a test in there which passes in one context, but >> fails in the other, comparison tools like contrib/compare_tests may >> erroneously report the tests as both a test which now fails, but >> passed before and a test which now passes but failed before. > >> It looks like some of the strub tests are currently known to fail with >> C++ and are triggering this problem > > Yeah, type warnings/errors are different between C and C++, and this is > noticeable with permissible conversions between strub types. > > >> A third option would be to change the compare_tests tool to somehow >> distinguish between the C and C++ tests. Not sure how feasible that >> is. > > Most feasible among the possibilities ;-) > > I've tested the following by comparing my obj-x86_64-linux-gnu test tree > with itself. Ok to install? > > > When compare_tests compares both C and C++ tests in c-c++-common, they > get the same identifier, so expected differences in results across > languages become undesirably noisy. > > This patch adds tool identifiers to tests, so that runs by different > tools are not confused by the compare logic. > > It also fixes a bug in reporting differences, that would attempt to > print an undefined fname (the definitions are in subshell loops), and > adjusts the target insertion to match tabs in addition to blanks after > colons. > > > for contrib/ChangeLog > > * compare_tests: Add tool to test lines. Match tabs besides > blanks to insert tool and target. Don't print undefined fname Go for it. It's hard to know if there'll be fallout in the consumers, but I think if there is pain, it should be short term. Jeff