From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-il1-x133.google.com (mail-il1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) by sourceware.org (Postfix) with ESMTPS id 0C36B385BF9E for ; Wed, 24 Mar 2021 03:13:57 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 0C36B385BF9E Received: by mail-il1-x133.google.com with SMTP id j11so20103561ilu.13 for ; Tue, 23 Mar 2021 20:13:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hxgNVB3nwO753MS6c2LKY+fQL5V6fqEkxFLFUXmFKsQ=; b=dDevDhh1ZHpa6jdw7qxUldgIV0O2mEmMItMPsK9nKM9aj2Vo7xcumj5Bz7Usq2K7RU gJTCCr0QlWXtThA9tpgufmMOn2Lmk1Ci010oYG4PKZ7AUDcBZ0tuq6MI3ZSbRnLpqHx8 PgH2udT28ZG6tS/Y9GxJi4goGZGy5gvlfWSsgbfwfQgonoDLJeungG4msYSGhGVKPfzo Hy/WZH57VWar/EYfliUG+QfV/S6AaQ4wbSlJJLKp/KnB0wscA6W9SyJZU9iOGQVwuGpQ WMjyiNEiBtIxISKVgOiqiRzoLoxjOvgwnOnEMzZcDiM3kw751h0FE2YKWx5RWlo1O04U YRiw== X-Gm-Message-State: AOAM530AdyaMwzoGdh1lRLwQM2ba6cYWrhtKnOVlAcYLVRakVrhL6X7L dJwFxuBJo/jJLeu/NZmVANguxVc2FfsZY3vHKPo= X-Google-Smtp-Source: ABdhPJxKTM6eDkGg3OxEXoHgM+JIwwE+WPMlTC/t0wef/SwkmWfTpgOZzhc5Fzm89a/ox7J7f8h2b8x+tvr1XrclqvI= X-Received: by 2002:a05:6e02:584:: with SMTP id c4mr1034459ils.153.1616555636472; Tue, 23 Mar 2021 20:13:56 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Peng Yu Date: Tue, 23 Mar 2021 22:13:45 -0500 Message-ID: Subject: Re: Does glibc has complete test coverage? To: Mike Frysinger Cc: Jeffrey Walton , libc-help X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, HTML_MESSAGE, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: libc-help@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-help mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2021 03:13:59 -0000 On Tue, Mar 23, 2021 at 8:13 PM Mike Frysinger wrote: > On 23 Mar 2021 17:02, Jeffrey Walton wrote: > > On Tue, Mar 23, 2021 at 4:43 PM Mike Frysinger wrote: > > > On 23 Mar 2021 11:39, Peng Yu via Libc-help wrote: > > > > https://www.kernel.org/doc/man-pages/missing_pages.html > > > > > > > > "... quite a few kernel and glibc bugs have been uncovered while > > > > writing test programs during the preparation of man pages. " > > > > > > > > I see the above text. It doesn't make too much sense, as it indicat= es > > > > that glibc does not have complete test coverage. > > > > > > > > Why not taking an approach of always accompanying each line of sour= ce > > > > code with appopriate test cases? If this approach is taken, then mo= st > > > > bugs should have been eliminated beforehand? > > > > > > ignoring the legacy aspect (code that's in the tree now but lacks > tests), > > > you have diminishing returns when it comes to writing unittests, and, > as > > > can be seen in a recent discussion, glibc is pretty tightly coupled t= o > > > the runtime environment (i.e. the host kernel). so getting an env th= at > > > matches all the different code paths is challenging. > > > > > > plus it comes down a bit to this being an open source project for man= y > > > of us, not a job, and you have to be respectful of balancing quality > > > and developer time with any requests you make on other volunteers. > > > > > > along those lines, this is an open source project where "patches are > > > welcome", so if you wanted to spend your time improving the framework= s > > > and coverage of our tests, we'd welcome you. > > > > Interns are usually a good choice for writing test cases. It gets them > > familiar with the code, frees up a senior developer's time, and helps > > avoid the developer's bias. > > > > Test cases are monkey work that should be delegated. When delegation > > does not occur it usually points back to shortcomings in project > > management. > > many are good for delegation, but that doesn't mean quantity is the same = as > quality. if we could get 100% coverage but it took weeks to run, but 90% > coverage took <1 hour, is that 10% worth it ? this isn't exactly hyperbo= le > when we have targets that run on simulators or FPGAs and have <<1GHz CPUs= . > > for example, how much of LTP should be part of glibc ? they have over 10= 00 > "syscall" tests which mostly go through the C library's APIs and can catc= h > bugs, but they also take a long time to run. > > how much should glibc be exercising different kernel versions ? a lot of > our work & APIs depend heavily on the kernel working correctly. should > we be running against every Linux release since 3.2 ? do we test the man= y > different ways kernels can be compiled ? do we workaround kernel bugs ? > https://sourceware.org/pipermail/libc-alpha/2021-March/123486.html > https://sourceware.org/pipermail/libc-alpha/2021-March/123582.html > > glibc has a matrix of build tools that it can utilize and significantly > affects its behavior & output. do we try every combo of GCC & binutils > that we support ? > > glibc runs on like 20 diff architectures, and many of those have ISA > specific optimizations (like x86_64 SSE/AVX/etc...). that's another > huge multiplier. You mentioned =E2=80=9Dbalance=E2=80=9D in another email. But isn=E2=80=99t= it a balance to not to support so many architecture? It sounds like supporting so many architectures can cause bugs. Alternatively, it is better to assume certain things, that the underlying architecture must meet. If not, add glue in between, which should be separate from glibc. In this way, it should be much easier to isolate bugs out of glibc. Also, the test cases should be white boxed instead of black boxed. If the test cases can be made white boxed, it is much less likely to have bugs in them than based black boxed strategies. The current test cases do not seem to be mostly white boxed? Also, using a white boxed approach, the original programmers should also write the test cases. But the current way of waiting others to add test cases making it hard to use the white boxed approach. The code complexities can not be reduce in the black boxed approach. Therefore, I don=E2=80=99t t= hink just adding more patches is an efficient way to eliminate the bugs. BTW, is there a way to know which part of the code is not covered? Also, even a line is covered,how well is tested against corner cases? > > my point is that "100% coverage" sounds fine until you dive down the > rabbit hole and realize it goes forever. > > > > also try googling for "100% test coverage" and reading the variety of > > > opinions the wider world has on the topic. > > > > Sorry, I could not resist.... But you know the funny thing is, when > > you perform a post-mortem to determine why the bug made it into > > production, it usually points to (1) a developer mistake and (2) lack > > of test case. > > > > If you break (1) or (2) you break the chain for the bug. So you either > > have to hire developers who don't make mistakes or provide complete > > test cases. > > i don't think that view lines up perfectly with the real world as you > might like it to. tests can have bugs too, and i've found plenty of > those. as in, all the tests pass fine, but that's because the tests > set up an invalid environment that doesn't match the real runtime. > > or the tests are there, and the runtime (e.g. kernel) changed. that > doesn't stop the bug from being introduced because kernel developers > aren't running all of the world's testsuites against their releases. > > i'm not saying tests don't add value and we shouldn't write them. i > write tests constantly. but i am saying that they aren't the solution > to all of our ails. > -mike > --=20 Regards, Peng