From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-relay-internal-1.canonical.com (smtp-relay-internal-1.canonical.com [185.125.188.123]) by sourceware.org (Postfix) with ESMTPS id 8C5C33858D20 for ; Mon, 28 Feb 2022 19:43:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 8C5C33858D20 Received: from mail-il1-f200.google.com (mail-il1-f200.google.com [209.85.166.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 524C23FCA8 for ; Mon, 28 Feb 2022 19:43:05 +0000 (UTC) Received: by mail-il1-f200.google.com with SMTP id y18-20020a927d12000000b002c2e830dc22so3111898ilc.20 for ; Mon, 28 Feb 2022 11:43:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=JOepxE6SBv1cMJA2r4e9w5WfC1RDB/Iw6OKGH9WEl/I=; b=y7JmBKMVDh5dA57EAGxT9TxkA4p3CimcT9gyA+gmIqmJSZ+WybJ1aJEaZquffYTB3Z eJrmuPrii1XM9SaYMw0OCtQ0ZGvOUrd8ubOME2o8ibjcDbweBPiRYJYmHi+idUkVY8gl 1iItE2zmDLoSvmLKbkFOu0yo8jz2SjRXVU4oSVmVJB3OSK6EdC9R1mRQhdypJRRTau05 fsUDN+vBoR2lz3aW3olVOnua0iKFxEHZPejiBGif+s+x0ltdzz/t70dI8VeLEPiqntTH 3vy1TIdciZIjDZYfBvexj5RgDsJ7GSVXpCQRVkk0zpKKzjik8e6mWDzsNt38+fzm4Mvh tyEQ== X-Gm-Message-State: AOAM53134ivbUFOBWxr5IHhKqwC26OPKm7Nqdr8SeWWNG6NyXB0oubP6 FHSLJPvNmzf9GSFccSH4I5HCazougfRY0eWYkeRF45qty2IVNrXXmiClEP/sv7ztaTNs8xLreeU TkJ84wDk1huklBbpsQYMhe5deI4w+4yWJw+wM X-Received: by 2002:a02:92c5:0:b0:314:22ef:6833 with SMTP id c5-20020a0292c5000000b0031422ef6833mr18120341jah.137.1646077383230; Mon, 28 Feb 2022 11:43:03 -0800 (PST) X-Google-Smtp-Source: ABdhPJwL2dCIwRnRgV8ZdB8WBtPzDBqupJxYmwPZdHT2WX17qTS7YzRq66790ZgnInb1a+TPvJ9qhQ== X-Received: by 2002:a02:92c5:0:b0:314:22ef:6833 with SMTP id c5-20020a0292c5000000b0031422ef6833mr18120321jah.137.1646077382878; Mon, 28 Feb 2022 11:43:02 -0800 (PST) Received: from localhost (c-71-196-238-11.hsd1.co.comcast.net. [71.196.238.11]) by smtp.gmail.com with ESMTPSA id t13-20020a056e02060d00b002c26056adb9sm6690337ils.25.2022.02.28.11.43.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Feb 2022 11:43:02 -0800 (PST) From: dann frazier To: systemtap@sourceware.org Cc: "Bernhard M . Wiedemann" , "Frank Ch . Eigler" Subject: [PATCH] dtrace: Use hash-based scheme for predictable file generation Date: Mon, 28 Feb 2022 12:42:23 -0700 Message-Id: <20220228194223.930994-1-dann.frazier@canonical.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-12.9 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_ASCII_DIVIDERS, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: systemtap@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Systemtap mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Feb 2022 19:43:14 -0000 commit c245153 ("dtrace: Allow for reproducible .o file builds.") introduced a condition where 2 dtrace processes can race when generating the same file. Since both processes now use the same temporary file name, one may delete the temporary .c file the other is still processing: -------------------------------------------------------------------- user@host:~/foo$ make -j2 dtrace -o foo.out -G -s /dev/null dtrace -o foo.out -G -s /dev/null Traceback (most recent call last): File "/usr/bin/dtrace", line 455, in sys.exit(main()) File "/usr/bin/dtrace", line 440, in main os.remove(fname) FileNotFoundError: [Errno 2] No such file or directory: 'foo.out.dtrace-temp.c' make: *** [Makefile:4: ../foo/foo.out] Error 1 -------------------------------------------------------------------- This can happen when a Makefile processes a pattern rule for two different targets that happen to map to the same file, but addressed by different relative paths. I discovered this in a real world case involving libvirt, but here's a contrived reproducer: -------------------------------------------------------------------- all: foo.out ../$(basename $(CURDIR))/foo.out %.out: dtrace -o foo.out -G -s /dev/null clean: rm -f foo.out -------------------------------------------------------------------- It would be ideal if we could inject a null .file directive, then we could just use a mkstemp() file and keep the build reproducible by avoiding a record of the source file path in the binary at all, but I can't find a straightforward way of passing a .file through to the assembler. So, instead, let's create a reproducible filename by building a hash of the input and output paths. Note: this still leaves open a race in the case of 2 dtrace processes with identical input/output paths. But, at least in my testing, GNU Make is smart enough to detect this case and not create duplicate jobs. https://sourceware.org/bugzilla/show_bug.cgi?id=28923 Fixes: Commit c245153 ("dtrace: Allow for reproducible .o file builds.") Signed-off-by: dann frazier --- dtrace.in | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/dtrace.in b/dtrace.in index 7cfe19c60..2e4cc4566 100644 --- a/dtrace.in +++ b/dtrace.in @@ -20,6 +20,7 @@ # pylint: disable=R0201 # pylint: disable=R0904 +import hashlib import os import sys from shlex import split @@ -410,12 +411,12 @@ def main(): else: print("header: " + fname) - try: # for reproducible-builds purposes, prefer a fixed path name pattern - fname = filename + ".dtrace-temp.c" - fdesc = open(fname, mode='w') - except: # but that doesn't work for -o /dev/null - see rhbz1504009 - (ignore,fname) = mkstemp(suffix=".c") - fdesc = open(fname, mode='w') + # for reproducible-builds purposes, use a predictable tmpfile path + sha = hashlib.sha256() + sha.update(s_filename.encode('utf-8')) + sha.update(filename.encode('utf-8')) + fname = ".dtrace-temp." + sha.hexdigest()[:8] + ".c" + fdesc = open(fname, mode='w') providers.semaphore_write(fdesc) fdesc.close() cc1 = os.environ.get("CC", "gcc") -- 2.35.1