[Bug libstdc++/99533] New: "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[Bug libstdc++/99533] New: "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[ssh at pobox dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

            Bug ID: 99533
           Summary: "operation not permitted" error on
                    recursive_directory_iterator despite
                    skip_permission_denied
           Product: gcc
           Version: 10.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libstdc++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: ssh at pobox dot com
  Target Milestone: ---

On POSIX filesystem backend type systems the
std::filesystem::recursive_directory_iterator throws a filesystem_error
exception with "operation not permitted" when the opendir/readdir call returns
EPERM instead of EACCES even if
std::filesystem::directory_options::skip_permission_denied is set.

Given the following code:

#include <iostream>
#include <filesystem>

int main(int argc, char* argv[])
{
    fs::path dir{"."};
    if(argc == 2) {
        dir = fs::u8path(argv[1]);
    }

    int totalDirs = 0;
    int totalFiles = 0;
    try {
        for(const auto& de : fs::recursive_directory_iterator(dir,
fs::directory_options::skip_permission_denied)) {
            if(de.is_regular_file()) {
                ++totalFiles;
            }
            else if(de.is_directory()) {
                ++totalDirs;
            }
        }
    }
    catch(fs::filesystem_error fe) {
        std::cerr << "Error: " << fe.what() << std::endl;
        exit(1);
    }
    std::cout << totalFiles << " files in " << totalDirs << " directories" <<
std::endl;
    return 0;
}

This fails for example on macOS when called on the user home directory with:

Error: filesystem error: cannot increment recursive directory iterator:
Operation not permitted

This is due to System Integrity Protection (since macOS 10.14) on the
"/Users/<user>/Library/Application Support/MobileSync" folder leading to EPERM.

On Linux, called with / it stops when hitting for example a
"/proc/1/task/1/cwd", resulting in EPERM too.

I don't have examples from other POSIX systems, but I would say handling only
EACCES  for the skip_permission_denied option is not enough.

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

Jonathan Wakely <redi at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-03-11
             Status|UNCONFIRMED                 |NEW

--- Comment #1 from Jonathan Wakely <redi at gcc dot gnu.org> ---
(In reply to Steffen Schuemann from comment #0)
> On POSIX filesystem backend type systems the
> std::filesystem::recursive_directory_iterator throws a filesystem_error
> exception with "operation not permitted" when the opendir/readdir call
> returns EPERM instead of EACCES even if
> std::filesystem::directory_options::skip_permission_denied is set.

POSIX is clear that opendir sets EACCES not EPERM.


> Given the following code:
> 
> #include <iostream>
> #include <filesystem>

N.B. This is missing a "namespace fs = std::filesystem;"

> This fails for example on macOS when called on the user home directory with:
> 
> Error: filesystem error: cannot increment recursive directory iterator:
> Operation not permitted
>
> This is due to System Integrity Protection (since macOS 10.14) on the
> "/Users/<user>/Library/Application Support/MobileSync" folder leading to
> EPERM.

That seems to be a MacOS bug.

> On Linux, called with / it stops when hitting for example a
> "/proc/1/task/1/cwd", resulting in EPERM too.

But that happens in your code, not in the library. The error is:

  Error: filesystem error: status: Permission denied [/proc/1/task/1/cwd]

And it comes from this line:

            if(de.is_regular_file()) {

I think the code is working correctly on linux.

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

--- Comment #2 from Jonathan Wakely <redi at gcc dot gnu.org> ---
(In reply to Jonathan Wakely from comment #1)
> > On Linux, called with / it stops when hitting for example a
> > "/proc/1/task/1/cwd", resulting in EPERM too.
> 
> But that happens in your code, not in the library. The error is:
> 
>   Error: filesystem error: status: Permission denied [/proc/1/task/1/cwd]

And that's an EACCES error anyway (as POSIX requires for stat(3) permission
errors). The text would be "Operation not permitted" for EPERM.

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

--- Comment #3 from Jonathan Wakely <redi at gcc dot gnu.org> ---
The standard says that skip_permission_denied means to ignore "an error
indicating that permission to access p is denied" and it's pretty clearly
intended to correspond to std::errc::permission_denied errors, not
std::errc::operation_not_permitted.

But this should solve the issue for MacOS:

--- a/libstdc++-v3/src/c++17/fs_dir.cc
+++ b/libstdc++-v3/src/c++17/fs_dir.cc
@@ -207,7 +207,7 @@ recursive_directory_iterator(const path& p,
directory_options options,
   else
     {
       const int err = errno;
-      if (err == EACCES
+      if (fs::is_permission_denied_error(err)
          && is_set(options, fs::directory_options::skip_permission_denied))
        {
          if (ecptr)
diff --git a/libstdc++-v3/src/filesystem/dir-common.h
b/libstdc++-v3/src/filesystem/dir-common.h
index 56e279230f4..a49b8304a29 100644
--- a/libstdc++-v3/src/filesystem/dir-common.h
+++ b/libstdc++-v3/src/filesystem/dir-common.h
@@ -141,6 +141,18 @@ struct _Dir_base
   posix::DIR*  dirp;
 };

+inline bool
+is_permission_denied_error(int e)
+{
+  if (e == EACCES)
+    return true;
+#ifdef __APPLE__
+  if (e == EPERM) // See PR 99533
+    return true;
+#endif
+  return false;
+}
+
 } // namespace filesystem

 // BEGIN/END macros must be defined before including this file.
diff --git a/libstdc++-v3/src/filesystem/dir.cc
b/libstdc++-v3/src/filesystem/dir.cc
index acc62986c68..215a9533f1b 100644
--- a/libstdc++-v3/src/filesystem/dir.cc
+++ b/libstdc++-v3/src/filesystem/dir.cc
@@ -202,7 +202,7 @@ recursive_directory_iterator(const path& p,
directory_options options,
   else
     {
       const int err = errno;
-      if (err == EACCES
+      if (std::filesystem::is_permission_denied_error(err)
          && is_set(options, fs::directory_options::skip_permission_denied))
        {

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

--- Comment #4 from Jonathan Wakely <redi at gcc dot gnu.org> ---
Fixed for gcc-11 only for now.

This got added to the wrong bug because I used the wrong PR number in the
changelog ...

The master branch has been updated by Jonathan Wakely <redi@gcc.gnu.org>:

https://gcc.gnu.org/g:8cfb387388a90730ab36ac24d9049677db633a11

commit r11-7628-g8cfb387388a90730ab36ac24d9049677db633a11
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Thu Mar 11 16:43:51 2021 +0000

    libstdc++: Handle EPERM for filesystem access errors on MacOS [PR nnnnn]

    Contrary to what POSIX says, some directory operations on MacOS can fail
    with EPERM instead of EACCES, so we need to handle both.

    libstdc++-v3/ChangeLog:

            PR libstdc++/nnnnn
            * src/c++17/fs_dir.cc (recursive_directory_iterator): Use new
            helper function to check for permission denied errors.
            * src/filesystem/dir.cc (recursive_directory_iterator):
            Likewise.
            * src/filesystem/dir-common.h (is_permission_denied_error): New
            helper function.

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[ssh at pobox dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

--- Comment #5 from Steffen Schuemann <ssh at pobox dot com> ---
Thank you for looking into this and sorry for the namespace issue, I was to
greedy when deleting the preprocessor parts that switched between my
implementation and std::filesystem.

First of all, I'm with you in thinking the macOS issue is more of an os
problem, but knowing the chances that this gets fixed on Apples side,
especially for anything older than Big Sur, I hoped that a fix/workaround on
GCC/LLVM side would be a much better chance. Thanks for the changes, I'll try
to test macOS later this day. 

And thank you for pointing out that the situation under Linux is in fact not
the same. You are right. Sadly I was fooled to believe it is, while comparing
the behavior with my implementation but after some debugging the reason mine
iterates through /proc/1/task/1/cwd without an error is actually a bug on my
side. So I have to thank you for helping me find that one.

So to recap: EPERM issue is indeed macOS only, not Linux!

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

--- Comment #6 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-10 branch has been updated by Jonathan Wakely
<redi@gcc.gnu.org>:

https://gcc.gnu.org/g:6e5f662f1dd6e3d1a79e194c118c295b897fc7ef

commit r10-9593-g6e5f662f1dd6e3d1a79e194c118c295b897fc7ef
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Thu Mar 11 16:43:51 2021 +0000

    libstdc++: Handle EPERM for filesystem access errors on MacOS [PR 99537]

    Contrary to what POSIX says, some directory operations on MacOS can fail
    with EPERM instead of EACCES, so we need to handle both.

    libstdc++-v3/ChangeLog:

            PR libstdc++/99533
            * src/c++17/fs_dir.cc (recursive_directory_iterator): Use new
            helper function to check for permission denied errors.
            * src/filesystem/dir.cc (recursive_directory_iterator):
            Likewise.
            * src/filesystem/dir-common.h (is_permission_denied_error): New
            helper function.

    (cherry picked from commit 8cfb387388a90730ab36ac24d9049677db633a11)

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

--- Comment #7 from Jonathan Wakely <redi at gcc dot gnu.org> ---
(In reply to Jonathan Wakely from comment #4)
> Fixed for gcc-11 only for now.

And 10.3 now too.

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

--- Comment #8 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-9 branch has been updated by Jonathan Wakely
<redi@gcc.gnu.org>:

https://gcc.gnu.org/g:b06bc096c4da89d8880f86fb0d9627168f22c50d

commit r9-9375-gb06bc096c4da89d8880f86fb0d9627168f22c50d
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Thu Mar 11 16:43:51 2021 +0000

    libstdc++: Handle EPERM for filesystem access errors on MacOS [PR 99537]

    Contrary to what POSIX says, some directory operations on MacOS can fail
    with EPERM instead of EACCES, so we need to handle both.

    libstdc++-v3/ChangeLog:

            PR libstdc++/99533
            * src/c++17/fs_dir.cc (recursive_directory_iterator): Use new
            helper function to check for permission denied errors.
            * src/filesystem/dir.cc (recursive_directory_iterator):
            Likewise.
            * src/filesystem/dir-common.h (is_permission_denied_error): New
            helper function.

    (cherry picked from commit 8cfb387388a90730ab36ac24d9049677db633a11)

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

--- Comment #9 from Jonathan Wakely <redi at gcc dot gnu.org> ---
And 9.4

[Bug libstdc++/99533] "operation not permitted" error on recursive_directory_iterator despite skip_permission_denied

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99533

Jonathan Wakely <redi at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
   Target Milestone|---                         |9.4
         Resolution|---                         |FIXED

--- Comment #10 from Jonathan Wakely <redi at gcc dot gnu.org> ---
Fixed on all active branches