From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa1-x2f.google.com (mail-oa1-x2f.google.com [IPv6:2001:4860:4864:20::2f]) by sourceware.org (Postfix) with ESMTPS id 6701A3858D33; Sun, 8 Jan 2023 07:12:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6701A3858D33 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-oa1-x2f.google.com with SMTP id 586e51a60fabf-14fb7fdb977so5883050fac.12; Sat, 07 Jan 2023 23:12:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=votFV7WqpGt9O8axlPjMEQ2At1BDZYSKj7+JsnDT1EE=; b=o8KRzZnLBtZ9MUK8Atljw53W07W6XwVcY62xuCe6o5FO6jGiabujjyoeWI12YaocK6 aYRZkCGvdyuPzXblA883CS7t1EI6kmviCtBMc3ECwEDX5VBWYHUcH0SIjlOFXX3yBAqT jSvheN84JE+g3YAq5O+xv2LdNU6RUgJCHCSECr6w5UZ76Vu+4JfGb4dEaPhHxppykWSp gE/RcdTMdb2djn+RcP+or8Nz00B5z2MWHRkRp47zXb4o+iK3XXi3FXgiSbKwKPt40Wi0 FvKLymg6LvUfCED32pGK+BgMB5oQiEjtRkH8Rco9PqD/nKoUVRUN9yKuuLtiN3nLugrc WRUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=votFV7WqpGt9O8axlPjMEQ2At1BDZYSKj7+JsnDT1EE=; b=hOU/jWWWjUtiqy4eGuasn1ike30qNTwyNDxAe2wMkUXfpxl0HDVNqYnglzLDH2IcSt mEORJgNvKuRVm7Ms1IzI0p0W1JumgRISy+eT2rcNr6S/ieOcybuo63qj6M4XmgyPmctt imMCA0yv+DGKrk3crhRoPv0pGFuSrGVOwX37Y3S94vNT289P5uTABcBAzQHPAh44uXZq tU1x8/zZJNVcVbYnV6+Kke3q/UivY0ihaIMZjV+PMwF5V1JB/ho2Y//LVL8D5GdtrExo 3wzTqz4zJuzocNTBUFMVCaBk6hSh+pKEA3nBfyZMHGvYWTFMDpi63oQzSJjnl5GKukeO x53A== X-Gm-Message-State: AFqh2koNmDiBeLF1cVRg2vP/ZRorfP64KmipERwz0frtK1XMIbUnL9WQ vw9BERSin3vAb40wrOpuRqnigAL++FkPR1/4JVY= X-Google-Smtp-Source: AMrXdXsyTVpePwYk/FWihqHAXZcYBCvmFSLT8uzah0Xo9SOO840WPz3vG2CyO7wLYr8Ty4wopq7UEAgsKnyvZ7y5Q+4= X-Received: by 2002:a05:6870:578f:b0:14f:ac78:ac7f with SMTP id i15-20020a056870578f00b0014fac78ac7fmr3129442oap.112.1673161963619; Sat, 07 Jan 2023 23:12:43 -0800 (PST) MIME-Version: 1.0 References: <639FE88D.7090408@gmail.com> <7cb45ab2-cc6e-c502-5592-51ffabcbc6f8@codeweavers.com> <63A3DF0E.1050902@gmail.com> <63A67964.6080902@gmail.com> <63B7967B.60502@gmail.com> In-Reply-To: <63B7967B.60502@gmail.com> From: NightStrike Date: Sun, 8 Jan 2023 02:12:42 -0500 Message-ID: Subject: Re: testsuite under wine To: jcb62281@gmail.com Cc: Jacek Caban , fortran@gcc.gnu.org, Eric Pouech , "gcc@gcc.gnu.org" , dejagnu@gnu.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Thu, Jan 5, 2023 at 10:33 PM Jacob Bachmeyer wrote: > > NightStrike wrote: > > On Fri, Dec 23, 2022 at 11:00 PM Jacob Bachmeyer wrote: > > > >> NightStrike wrote: > >> > >>> On Wed, Dec 21, 2022 at 11:37 PM Jacob Bachmeyer wrote: > >>> > >>>> [...] > >>> So at least we know for sure that this particular instance of extra > >>> characters is coming from Wine. Maybe Wine can be smart enough to > >>> only translate \n into \r\n instead of translating \r\n into \r\r\n. > >>> Jacek / Eric, comments here? I'm happy to try another patch, the > >>> first one was great. > >>> > >>> > >> I doubt that Wine is doing that translation. MinGW libc produces output > >> conformant to Windows conventions, so printf("\n") on a text handle > >> emits "\r\n", which Wine passes along. POSIX convention is that "\n" is > >> translated to "\r\n" in the kernel terminal driver upon output, so the > >> kernel translates the "\n" in the "\r\n" into /another/ "\r\n", yielding > >> "\r\r\n" at the pty master end. This is why DejaGnu testsuites must be > >> prepared to discard excess carriage returns. The first CR came from > >> MinGW libc; the second CR came from the kernel terminal driver; the LF > >> was ultimately passed through. > >> > > > > Jacek and I have been digging into this on IRC, and he's been very > > helpful in trying to get further, but we're still stuck. We tried to > > be more introspective, inserting strace both as "strace script wine" > > and as "script strace wine". We tried running just "wine a.exe" > > without any extra glue, and logging the raw SSH packets from putty. > > After many iterations on these and other tests, Jacek finally had the > > idea to try removing Windows entirely from the equation, and we ran > > with a purely unix program / compiler combination: > > > > #include > > > > int main() > > { > > write(1, "test\r\n", 6); > > return 0; > > } > > > > (and also as "test\n", 5) > > > > In both versions, the following was observed: > > > > case 1) ./a.out | xxd > > case 2) script -c ./a.out out; xxd out > > case 3) enable putting logging, ./a.out > > > > In case 1, xxd showed no extra \r's. In cases 2 and 3, there was an > > extra \r (either 0d 0d 0a for test\r\n, or 0d 0a for test\n). > > > > So, is it possible after all of this back and forth regarding mingw, > > wine, and others, that it's down to the write() system call that's > > inserting extra \r's? Is this expected? > > > > "This is why DejaGnu testsuites must be prepared to discard excess > carriage returns." > > The write(2) system call inserts nothing and simply hands off the buffer > to the relevant part of the kernel I/O subsystem. (The kernel in POSIX > is *not* a monolithic black box.) When stdout for your test program is > a pty slave, that relevant part is the kernel terminal driver. The > kernel terminal driver is converting "\n" to "\r\n" upon output to the > associated port, since hardware terminals typically *do* require CRLF. > The associated port in this case is virtual and part of the kernel pty > subsystem, which presents octets written to that port to its associated > pty master device. The user-visible pty slave device acts just like a > serial terminal, including all translations normally done for handling > serial terminals. > > A pty is conceptually a null-modem cable connected between two > infinitely-fast serial ports on the same machine, although the slave > will still report an actual baud rate if queried. (Run "stty" with no > arguments under script(1), an ssh session, or an X11 terminal emulator > to see what a pty slave looks like on your machine.) > > In your case 1, the pty subsystem is not used and output is collected > over a pipe. Using "./a.out > out; xxd out" would produce the same > results. In cases 2 and 3, there is a pty involved, either set up by > script(1) or by sshd (assuming you meant "enable putty logging" in case > 3) that performs the standard terminal translations. In all cases, > strace(1) will show the exact string written to the pty slave device, > which will not include any extra CRs because *those* *are* *inserted* > *by* *the* *kernel* *terminal* *driver* as the data is transferred to > the pty master device's read queue. > > This insertion of carriage returns is expected and standardized behavior > in POSIX and is the reason Unix could use bare LF as end-of-line even > though hardware terminals always needed CRLF. CP/M (and therefore > MS-DOS which began its existence as a cheap CP/M knockoff) did not have > this translation layer and instead dumped the complexity of a two-octet > end-of-line sequence on user programs, leading to much confusion even > today. This is not a Wine issue, although the terminal escape sequences > in your original issue *were* from Wine. Note that the number of excess > carriage returns that a DejaGnu testsuite must be prepared to discard is > unspecified because running tests on remote targets may result in *any* > *number* of CRs preceding each LF by the time the results reach the test > driver machine in more complex testing lab environments. First, I just want to thank you for your patience. You are putting a lot of effort into these replies, and it is appreciated. I did another little test to try to better understand your point. I ran a linux native testsuite under a simulator that just sets SIM to " ". This resulted in extra ^M's also, although many tests pass because they're already looking for \r\n to accommodate windows. So I think I've come around to grasp what you've been heroically re-explaining... So if we have to modify every test in the entire testsuite to check for zero or more \r's followed by zero or more \n's, would it be better to add a dg-output-line proc that does this automatically everywhere? I feel like changing every output pattern test won't be too maintainable. You had mentioned previously modifying ${tool}_load to filter out extra \r's, but I couldn't see where or how to do that. For completeness, setting a random selection of tests to look for \r*\n? worked (this would cover even deprecated systems that only use CR as well as flagging the weird rust case of \r\r\n\n as bad).