From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 31501 invoked by alias); 23 Jul 2006 22:05:06 -0000 Received: (qmail 31492 invoked by uid 22791); 23 Jul 2006 22:05:05 -0000 X-Spam-Check-By: sourceware.org Received: from gatekeeper.tait.co.nz (HELO gatekeeper.tait.co.nz) (202.37.96.11) by sourceware.org (qpsmtpd/0.31) with ESMTP; Sun, 23 Jul 2006 22:05:02 +0000 Received: from gatekeeper.tait.co.nz (localhost.localdomain [127.0.0.1]) by localhost.tait.co.nz (Postfix) with ESMTP id 844FF4675C for ; Mon, 24 Jul 2006 10:04:58 +1200 (NZST) Received: from sunstrike.tait.co.nz (sunstrike [172.25.40.92])by gatekeeper.tait.co.nz (Postfix) with ESMTP id 7302346752for ; Mon, 24 Jul 2006 10:04:58 +1200 (NZST) Received: from conversion-daemon.sunstrike.tait.co.nz by sunstrike.tait.co.nz(Sun Java System Messaging Server 6.1 (built Apr 28 2004))id <0J2V00401KXN5I00@sunstrike.tait.co.nz>(original mail from john.carter@tait.co.nz) for gcc-help@gcc.gnu.org; Mon,24 Jul 2006 10:04:56 +1200 (NZST) Received: from parore ([172.25.140.12])by sunstrike.tait.co.nz (Sun Java System Messaging Server 6.1 (built Apr 282004)) with ESMTP id <0J2V001HJMO6TU40@sunstrike.tait.co.nz>; Mon,24 Jul 2006 10:04:54 +1200 (NZST) Received: from localhost ([127.0.0.1] ident=johnc)by parore with esmtp (Exim 4.34) id 1G4m4E-0001Yx-Ni; Mon,24 Jul 2006 10:04:54 +1200 Date: Sun, 23 Jul 2006 22:05:00 -0000 From: John Carter Subject: Re: how to make code stay invariant In-reply-to: <44C30776.1030007@august.de> To: Rolf Schumacher Cc: gcc-help@gcc.gnu.org Message-id: MIME-version: 1.0 X-Mailer: Pidgeon Post Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-Apparently-From: mars X-Contents: May contain traces of nuts. References: <44BAC65D.1010300@august.de> <44C30776.1030007@august.de> X-imss-version: 2.041 X-imss-result: Passed X-imss-approveListMatch: *@tait.co.nz X-IsSubscribed: yes Mailing-List: contact gcc-help-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Archive: List-Post: List-Help: Sender: gcc-help-owner@gcc.gnu.org X-SW-Source: 2006-07/txt/msg00299.txt.bz2 On Sun, 23 Jul 2006, Rolf Schumacher wrote: > The larger problem: > > As an assessor for safety critical software of embedded systems > I'm regularily faced with statements like: > "we only have made a little change to the source code > we do not want to retest all modules" > > The problem is, even if a change is made to one module only, > how can I demonstrate that all other modules are unchanged? The short answer is "You can't. Not even in Principle." The longer answer is (greetings from Werner by the way!)... * A common form of bug in C/C++ is an uninitialized variable. 9 times out of ten, if the value "zero" or whatever the previous value was is acceptable, the bug escapes the notice of the testers, no matter how rigorous. However, change the memory layout, change the activation pathway of the code, and the uninitialized memory can hold something nasty. * Bugs infest the boundaries of state space. In a smallish system you can easily have 2000 variables ie. more states that there are atoms in the universe! ie. Your previous testing, no matter how rigorous, did not explored the full state space. In fact explored only tiny fraction. So a small change entirely external to your "unchanged" module can cause your "unchanged" module to move through a different region of state space. An untested region. So far more useful than checksumming I would recommend.... * Compile -W -Wall -Werror and perhaps use www.splint.org as well. * Design your system in strict layers. * Decrease the size of the state subspace in each layer. * Decrease the number of "user configurable items". * Decrease the number of product variants. * Place a good suite of automated Unit tests at each layer. Test Driven Development is Very Good for this. * Brace the interior of the code well with asserts. This is like testing from inside the code, both during bench test and production test. * Think your assert handling policy through so you run exactly the same code under test as in production. * Have an automated suite of functional tests around your product as a whole. * Run all tests on all builds. * Preferable run all the non-timing critical test under something like valgrind as well. http://www.valgrind.org/ All this will do _way_ more to increase safety than any checksumming scheme. > Unchanged source code is easy to demonstrate. > But this is not the whole story. I was recently called in to track the difference between too apparently identical bits of source code resulting in different executables & behaviour... Answer? The __FILE__ macro (commonly used in asserts and the like) includes the full path, one guy had built his code in /home/joe and the other in /home/fred resulting in different string lengths! (And hence different offsets everywhere!) > If I change the source of one module and recompile it.... All too true. Perhaps if they were separate processes / executables you in the nice state of being able to byte for byte verify that they are same. The Unixy notion of separate processes separate virtual addresses spaces is not just a pretty face. It gives a lot of really good, really hard safety guarantees! > The only way to find these (unlikely) errors nowadays > is to retest all modules. > > Or to ignore this problem with some unknown risk. > But: retests are too expensive and risk negligence > has its probability. If I were to push (hard) on some aspect of the problem I would push (hard) on the cost of retest. > The idea is to have a checksum for each module that has passed all > tests successfully and this checksum is unchanged even if any other > module becomes changed. Checksum the individual .o object files. Best balance for difficulty of doing vs safety. Not in anyway a guarantee, a test is still better, this would be cheaper and better than nothing. Notes about checksumming the .o files.... They include debug data which has absolute file paths. They include time stamps. Probably need to use the objdump (or maybe objcopy/ readelf) utilities to read out the sections you care about and checksum those. eg. Beware of $Version Control$ tags ending up in strings. They also caused spurious diffs. John Carter Phone : (64)(3) 358 6639 Tait Electronics Fax : (64)(3) 359 4632 PO Box 1645 Christchurch Email : john.carter@tait.co.nz New Zealand Carter's Clarification of Murphy's Law. "Things only ever go right so that they may go more spectacularly wrong later." >From this principle, all of life and physics may be deduced.