From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from rock.gnat.com (rock.gnat.com [205.232.38.15]) by sourceware.org (Postfix) with ESMTPS id C347A3858002; Mon, 27 Jun 2022 13:27:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C347A3858002 Received: from localhost (localhost.localdomain [127.0.0.1]) by filtered-rock.gnat.com (Postfix) with ESMTP id 62A9F11645B; Mon, 27 Jun 2022 09:27:38 -0400 (EDT) X-Virus-Scanned: Debian amavisd-new at gnat.com Received: from rock.gnat.com ([127.0.0.1]) by localhost (rock.gnat.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id toaVeJPQF7OA; Mon, 27 Jun 2022 09:27:38 -0400 (EDT) Received: from free.home (tron.gnat.com [IPv6:2620:20:4000:0:46a8:42ff:fe0e:e294]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by rock.gnat.com (Postfix) with ESMTPS id B2775116438; Mon, 27 Jun 2022 09:27:37 -0400 (EDT) Received: from livre (livre.home [172.31.160.2]) by free.home (8.15.2/8.15.2) with ESMTPS id 25RDRP4u891123 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Jun 2022 10:27:28 -0300 From: Alexandre Oliva To: Jonathan Wakely Cc: gcc Patches , "libstdc++" Subject: Re: [PATCH] libstdc++: retry removal of dir entries if dir removal fails Organization: Free thinker, does not speak for AdaCore References: Errors-To: aoliva@lxoliva.fsfla.org Date: Mon, 27 Jun 2022 10:27:25 -0300 In-Reply-To: (Jonathan Wakely's message of "Wed, 22 Jun 2022 10:45:13 +0100") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 2.84 X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libstdc++@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libstdc++ mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jun 2022 13:27:40 -0000 On Jun 22, 2022, Jonathan Wakely wrote: > I haven't properly reviewed it yet Nevermind that one, it's broken because I hadn't realized the recursive iteration. It fails and throws/errors out when we attempt to __erase a subdir that wasn't successfully emptied because some of its entries were skipped. Here's an improved version that appears to work, despite the few(er) remaining fails. I've convinced myself it can't possibly introduce symlink races because, if it doesn't follow symlinks in the first try, it won't follow them in retries. Potential races related with moving directories into or out of the remove_all root remain. On POSIX-compliant filesystems, the first name that fails to be removed is most likely to be the first to be encountered in the retry, so removal will terminate with the failure immediately. There's a potential for retries to end up removing other dirs moved into the remove_all root, but moving them in while remove_all is still running may remove them, so I don't see that it changes anything. On RTEMS, the first name that would have failed to be removed, say because of permissions, may be skipped by the iterator, so we may proceed to remove later dir entries under the same parent before failing to remove the parent and starting a retry. There may thus be an unbounded number of retries, one for each subdirectory with more than one entry in the remove_all tree. I see two potential ways to avoid this: (i) call remove_all recursively upon failure to remove an entry, instead of restarting iteration; or (ii) arrange for recursive_directory_iterator to rewind a dir from which entries have been _erase()d before returning to the parent dir. I have not implemented either of these alternatives, though. This one is regstrapped on x86_64-linux-gnu, also tested with a cross to aarch64-rtems6. Ok to install? libstdc++: retry removal of dir entries if dir removal fails From: Alexandre Oliva On some target systems (e.g. rtems6.0), removing directory components while iterating over directory entries may cause some of the directory entries to be skipped, which prevents the removal of the parent directory from succeeding. Advancing the iterator before removing a member proved not to be enough, so I've instead arranged for remove_all to retry the removal of components if the removal of the parent dir fails after removing at least one entry. The fail will be permanent only if no components got removed in the current try. for libstdc++-v3/ChangeLog * src/c++17/fs_ops.cc (remove_all): Retry removal of directory entries. --- libstdc++-v3/src/c++17/fs_ops.cc | 40 ++++++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/libstdc++-v3/src/c++17/fs_ops.cc b/libstdc++-v3/src/c++17/fs_ops.cc index 435368fa5c5ff..de99e02af4c34 100644 --- a/libstdc++-v3/src/c++17/fs_ops.cc +++ b/libstdc++-v3/src/c++17/fs_ops.cc @@ -1286,6 +1286,8 @@ fs::remove_all(const path& p) { error_code ec; uintmax_t count = 0; + retry: + uintmax_t init_count = count; recursive_directory_iterator dir(p, directory_options{64|128}, ec); switch (ec.value()) // N.B. assumes ec.category() == std::generic_category() { @@ -1295,7 +1297,16 @@ fs::remove_all(const path& p) const recursive_directory_iterator end; while (dir != end) { - dir.__erase(); // throws on error + /* Avoid exceptions if we may retry on fail on systems that + miss dir entries when removing while iterating. */ + if (count > init_count) + { + dir.__erase(&ec); + if (ec) + goto retry; + } + else + dir.__erase(); // throws on error ++count; } } @@ -1303,7 +1314,7 @@ fs::remove_all(const path& p) break; case ENOENT: // Our work here is done. - return 0; + return count; case ENOTDIR: case ELOOP: // Not a directory, will remove below. @@ -1313,6 +1324,18 @@ fs::remove_all(const path& p) _GLIBCXX_THROW_OR_ABORT(filesystem_error("cannot remove all", p, ec)); } + if (count > init_count) + { + if (int last = fs::remove(p, ec); !ec) + return count + last; + else + // Some systems seem to skip entries in the dir iteration if + // you remove dir entries while iterating, so if we removed + // anything in the dir in this round, and failed to remove + // the dir (presumably because it wasn't empty), retry. + goto retry; + } + // Remove p itself, which is either a non-directory or is now empty. return count + fs::remove(p); } @@ -1321,6 +1344,8 @@ std::uintmax_t fs::remove_all(const path& p, error_code& ec) { uintmax_t count = 0; + retry: + uintmax_t init_count = count; recursive_directory_iterator dir(p, directory_options{64|128}, ec); switch (ec.value()) // N.B. assumes ec.category() == std::generic_category() { @@ -1332,7 +1357,12 @@ fs::remove_all(const path& p, error_code& ec) { dir.__erase(&ec); if (ec) - return -1; + { + if (count > init_count) + goto retry; + else + return -1; + } ++count; } } @@ -1341,7 +1371,7 @@ fs::remove_all(const path& p, error_code& ec) case ENOENT: // Our work here is done. ec.clear(); - return 0; + return count; case ENOTDIR: case ELOOP: // Not a directory, will remove below. @@ -1354,6 +1384,8 @@ fs::remove_all(const path& p, error_code& ec) // Remove p itself, which is either a non-directory or is now empty. if (int last = fs::remove(p, ec); !ec) return count + last; + if (count > init_count) + goto retry; return -1; } -- Alexandre Oliva, happy hacker https://FSFLA.org/blogs/lxo/ Free Software Activist GNU Toolchain Engineer Disinformation flourishes because many people care deeply about injustice but very few check the facts. Ask me about