cygwin utils can access directory and its contents, but W10 utils claim to have no access, why?

cygwin utils can access directory and its contents, but W10 utils claim to have no access, why?

[John Ruckstuhl]

I am seeing a weird phenomenon, hopefully someone can illuminate and teach.
Or point me to an archived thread.

I have some folders that Cygwin utils can readily access,
but W10 utils claim to have no access to.
It feels as if
*   the Cygwin utils try to do what they are commanded to do, and it's
    up to the OS to refuse if the ACLs are insufficient.
*   the W10 utils are written to decline to attempt access, due to some
    convention or gentlemen's agreement (built into some API?).
But wouldn't that lead Windows users to a false sense of protection, a
false sense of security?

Environment
*   Cygwin 3.4.9-1 on W10.  64-bit.
*   UAC is not active (based on EnableLUA=0 at boot)

I have a 3rd-party program “yabba” that many different users run (on the
same shared PC).
It leaves behind a “temporary” dir (under
C:/Users/username/AppData/Local/Temp), each time it’s run.
Many times per day.
These dirs are approx. 1000 files and 20 MB.
The 3rd-party program is an exe-file compiled with pyinstaller.
It unpacks a python script and a python interpreter, and runs the python
script.
But in our workflow, we usually terminate the execution brutally, so the
pyinstaller cleanup does not happen, leaving the 20 MB behind.

I want Bob, a member of the local group Administrators, to remove these folders.

For example, ordinary user Alice runs yabba a couple of times and leaves
behind 20 MB  in each of
    C:/Users/Alice/AppData/Local/Temp/_MEI21002
    C:/Users/Alice/AppData/Local/Temp/_MEI21282

Now for Local Administrator Bob, no impediment seeing these two dirs...
    $ cd C\:/Users/Alice/AppData/Local/Temp
    $ D1=_MEI21002
    $ D2=_MEI21282
    $ du -sh $D1 $D2
    21M     _MEI21002
    21M     _MEI21282
    $ for D in $D1 $D2; do printf "%s\t%s\n" $D "$(find "$D" -mindepth
1 -type d -printf "dirs\n" -o -printf "files\n" | sort | uniq -c |
paste - -)"; done
    _MEI21002            34 dirs        940 files
    _MEI21282            33 dirs        929 files
    $ getfacl $D1 $D2
    # file: _MEI21002
    # owner: Alice
    # group: Domain Users
    user::rwx
    group::---
    group:OWNER RIGHTS:rwx
    mask::rwx
    other::---


    # file: _MEI21282
    # owner: Alice
    # group: Domain Users
    user::rwx
    group::---
    group:OWNER RIGHTS:rwx
    mask::rwx
    other::---

And (for Local Administrator Bob) no impediment to removing them, for example,
    $ rm -r _MEI21282
(success!)

BUT Windows utils run by Bob the Administrator on the remaining dir
are blocked, like this
(cmd, "Run as administrator"):
    C:\Users\Alice\AppData\Local\Temp>dir _MEI21002
    Volume in drive C is OSDisk
    Volume Serial Number is 581C-10F2

    Directory of C:\Users\Alice\AppData\Local\Temp\_MEI21002

    File Not Found

    C:\Users\Alice\AppData\Local\Temp>icacls _MEI21002
    _MEI21002: Access is denied.
    Successfully processed 0 files; Failed processing 1 files

As a consequence, Windows utils run by Bob the Administrator cannot
delete Alice's directory.  Until they do a takeown.

The first try to delete fails:
    C:\Users\Alice\AppData\Local\Temp>del /S /Q _MEI21002
    Could Not Find C:\Users\Alice\AppData\Local\Temp\_MEI21002\*

But a takeown succeeds
    C:\Users\Alice\AppData\Local\Temp>takeown /A /F _MEI21002 /R /D Y

Now icacls has access
    C:\Users\Alice\AppData\Local\Temp>icacls _MEI21002
    _MEI21002 BUILTIN\Administrators:(OI)(IO)(F)
              BUILTIN\Administrators:(CI)(F)

    Successfully processed 1 files; Failed processing 0 files

And the second try to delete succeeds
    C:\Users\Alice\AppData\Local\Temp>del /S /Q _MEI21002

In summary, why is it that Bob the Administrator can Cygwin "rm.exe" to
delete these folders without taking ownership, but to delete with
Windows utils, he needs to take ownership first?

Thanks for teaching,
John Ruckstuhl

Re: cygwin utils can access directory and its contents, but W10 utils claim to have no access, why?

[Brian Inglis]

On 2024-01-21 23:18, John Ruckstuhl via Cygwin wrote:
> I am seeing a weird phenomenon, hopefully someone can illuminate and teach.
> Or point me to an archived thread.
> 
> I have some folders that Cygwin utils can readily access,
> but W10 utils claim to have no access to.
> It feels as if
> *   the Cygwin utils try to do what they are commanded to do, and it's
>      up to the OS to refuse if the ACLs are insufficient.
> *   the W10 utils are written to decline to attempt access, due to some
>      convention or gentlemen's agreement (built into some API?).
> But wouldn't that lead Windows users to a false sense of protection, a
> false sense of security?
> 
> Environment
> *   Cygwin 3.4.9-1 on W10.  64-bit.
> *   UAC is not active (based on EnableLUA=0 at boot)
> 
> I have a 3rd-party program “yabba” that many different users run (on the
> same shared PC).
> It leaves behind a “temporary” dir (under
> C:/Users/username/AppData/Local/Temp), each time it’s run.
> Many times per day.
> These dirs are approx. 1000 files and 20 MB.
> The 3rd-party program is an exe-file compiled with pyinstaller.
> It unpacks a python script and a python interpreter, and runs the python
> script.
> But in our workflow, we usually terminate the execution brutally, so the
> pyinstaller cleanup does not happen, leaving the 20 MB behind.
> 
> I want Bob, a member of the local group Administrators, to remove these folders.
> 
> For example, ordinary user Alice runs yabba a couple of times and leaves
> behind 20 MB  in each of
>      C:/Users/Alice/AppData/Local/Temp/_MEI21002
>      C:/Users/Alice/AppData/Local/Temp/_MEI21282
> 
> Now for Local Administrator Bob, no impediment seeing these two dirs...
>      $ cd C\:/Users/Alice/AppData/Local/Temp
>      $ D1=_MEI21002
>      $ D2=_MEI21282
>      $ du -sh $D1 $D2
>      21M     _MEI21002
>      21M     _MEI21282
>      $ for D in $D1 $D2; do printf "%s\t%s\n" $D "$(find "$D" -mindepth
> 1 -type d -printf "dirs\n" -o -printf "files\n" | sort | uniq -c |
> paste - -)"; done
>      _MEI21002            34 dirs        940 files
>      _MEI21282            33 dirs        929 files
>      $ getfacl $D1 $D2
>      # file: _MEI21002
>      # owner: Alice
>      # group: Domain Users
>      user::rwx
>      group::---
>      group:OWNER RIGHTS:rwx
>      mask::rwx
>      other::---
> 
> 
>      # file: _MEI21282
>      # owner: Alice
>      # group: Domain Users
>      user::rwx
>      group::---
>      group:OWNER RIGHTS:rwx
>      mask::rwx
>      other::---
> 
> And (for Local Administrator Bob) no impediment to removing them, for example,
>      $ rm -r _MEI21282
> (success!)
> 
> BUT Windows utils run by Bob the Administrator on the remaining dir
> are blocked, like this
> (cmd, "Run as administrator"):
>      C:\Users\Alice\AppData\Local\Temp>dir _MEI21002
>      Volume in drive C is OSDisk
>      Volume Serial Number is 581C-10F2
> 
>      Directory of C:\Users\Alice\AppData\Local\Temp\_MEI21002
> 
>      File Not Found
> 
>      C:\Users\Alice\AppData\Local\Temp>icacls _MEI21002
>      _MEI21002: Access is denied.
>      Successfully processed 0 files; Failed processing 1 files
> 
> As a consequence, Windows utils run by Bob the Administrator cannot
> delete Alice's directory.  Until they do a takeown.
> 
> The first try to delete fails:
>      C:\Users\Alice\AppData\Local\Temp>del /S /Q _MEI21002
>      Could Not Find C:\Users\Alice\AppData\Local\Temp\_MEI21002\*
> 
> But a takeown succeeds
>      C:\Users\Alice\AppData\Local\Temp>takeown /A /F _MEI21002 /R /D Y
> 
> Now icacls has access
>      C:\Users\Alice\AppData\Local\Temp>icacls _MEI21002
>      _MEI21002 BUILTIN\Administrators:(OI)(IO)(F)
>                BUILTIN\Administrators:(CI)(F)
> 
>      Successfully processed 1 files; Failed processing 0 files
> 
> And the second try to delete succeeds
>      C:\Users\Alice\AppData\Local\Temp>del /S /Q _MEI21002
> 
> In summary, why is it that Bob the Administrator can Cygwin "rm.exe" to
> delete these folders without taking ownership, but to delete with
> Windows utils, he needs to take ownership first?

I normally find the problem is that a directory has no useful Windows DACLs 
defined, so directories or files created in it by Cygwin programs are not 
accessible to Windows programs.
It seems to me, that when Cygwin creates directories or files in those 
directories, they have no Windows ACLs defaulted, but they seem to look fine to 
Cygwin's assumed POSIX permissions, as if on FAT drives?
Better explanations would be welcome.

For directories, run:

	$ lsattr -adl 	$D
	$ ls -adlL	$D
	$ getfacl	$D
	$ icalcs "$(cygpath -m "$D")"

and for files, skip lsattr.

This should show you all the views of permissions you need to figure out the issue.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: cygwin utils can access directory and its contents, but W10 utils claim to have no access, why?

[Corinna Vinschen]

On Jan 21 22:18, John Ruckstuhl via Cygwin wrote:
> I am seeing a weird phenomenon, hopefully someone can illuminate and teach.
> Or point me to an archived thread.
> 
> I have some folders that Cygwin utils can readily access,
> but W10 utils claim to have no access to.
> It feels as if
> *   the Cygwin utils try to do what they are commanded to do, and it's
>     up to the OS to refuse if the ACLs are insufficient.
> *   the W10 utils are written to decline to attempt access, due to some
>     convention or gentlemen's agreement (built into some API?).
> But wouldn't that lead Windows users to a false sense of protection, a
> false sense of security?

No, this has nothing to do with security, just with a weird concept
of user privileges.

Here's what happens:

> And (for Local Administrator Bob) no impediment to removing them, for example,
>     $ rm -r _MEI21282
> (success!)
> 
> BUT Windows utils run by Bob the Administrator on the remaining dir
> are blocked, like this

As you know, a "root" user on a Unix system has the right to ignore file
permissions.

That's why root users often have "rm" aliased to "rm -i" :)

Windows OTOH manages so called user privileges stored in the user's
access token.  One of these privileges is the right to ignore
permissions while reading (SE_BACKUP_PRIVILEGE), another privilege is
the right to set the ACL of a file to arbitrary permissions
(SE_RESTORE_PRIVILEGE).  These are the two privileges covering what a
"root" user can do to files.  As the names of these privileges suggest,
they were originally thought of privileges you only need for backup
tools.

Users in the Administrators group have these privileges in their user
token.  Under UAC, both privileges are removed from the token.  In an
elevated shell, though, both privileges are present.

The funny thing here is this: While both privileges are present in the
token, they are disabled by default.

They have to be enabled explicitely before you can exercise the
privileges.  Usually you do this in the same application
programatically.

Apart from backup tool, standard Windows tools practically *never*
enable these privileges, so they fail in various circumstances.  For
instance, a standard "del" in the CMD shell is not able to delete a file
even as administrator, if the file permissions don't allow delete access
for the Administrators group.

That's when you have to "take ownership" in the GUI as a last resort.
In NT4 times, even that had a good chance to fail for some reason I
never understood, back in the days.

However, since this concept is pretty weird and not a Unix concept, the
Cygwin DLL enables both privileges by default right at process startup.
If the privileges are not in the user token, nothing happens.  But if
the privileges are present, as for an elevated process of a user in the
Administrators group, then they will be enabled, and you're a "root"
user in the Unix sense.  And then it's "alias rm = rm -i" time again ;)


HTH,
Corinna

Re: cygwin utils can access directory and its contents, but W10 utils claim to have no access, why?

[John Ruckstuhl]

Thanks for the replies Brian & Corinna, I learned a lot.

On Mon, Jan 22, 2024 at 1:48 AM Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:
> Users in the Administrators group have these privileges in their user
> token.  Under UAC, both privileges are removed from the token.  In an
> elevated shell, though, both privileges are present.
>
> The funny thing here is this: While both privileges are present in the
> token, they are disabled by default.
>
> They have to be enabled explicitely before you can exercise the
> privileges.  Usually you do this in the same application
> programatically.

Now I see...
Local administrator Bob reports his User Access Token info
(with Windows whoami, not cygwin whoami)
    C:\>whoami /priv

    PRIVILEGES INFORMATION
    ----------------------

    Privilege Name                            ... State
    ========================================= ... ========
      ...
      SeBackupPrivilege                         ... Disabled
      SeRestorePrivilege                        ... Disabled
      ...

      C:\>

Thanks very much.
John Ruckstuhl

Re: cygwin utils can access directory and its contents, but W10 utils claim to have no access, why?

[John Ruckstuhl]

On Mon, Jan 22, 2024 at 10:59 AM John Ruckstuhl
<john.ruckstuhl@gmail.com> wrote:
>
> Thanks for the replies Brian & Corinna, I learned a lot.
>
> On Mon, Jan 22, 2024 at 1:48 AM Corinna Vinschen
> <corinna-cygwin@cygwin.com> wrote:
> > Users in the Administrators group have these privileges in their user
> > token.  Under UAC, both privileges are removed from the token.  In an
> > elevated shell, though, both privileges are present.
> >
> > The funny thing here is this: While both privileges are present in the
> > token, they are disabled by default.
> >
> > They have to be enabled explicitely before you can exercise the
> > privileges.  Usually you do this in the same application
> > programatically.
>
> Now I see...
> Local administrator Bob reports his User Access Token info
> (with Windows whoami, not cygwin whoami)
>     C:\>whoami /priv
>
>     PRIVILEGES INFORMATION
>     ----------------------
>
>     Privilege Name                            ... State
>     ========================================= ... ========
>       ...
>       SeBackupPrivilege                         ... Disabled
>       SeRestorePrivilege                        ... Disabled
>       ...
>
>       C:\>
>
> Thanks very much.
> John Ruckstuhl

The clue about UAT and disabled privileges was crucial.
Thank you again, Corinna.

I'm still doing my cleanup with ordinary Python (not the Cygwin Python
linked to the cygwin dll).
But I wrote an EnablePrivilege function to enable the privs if if necessary.
Best regards,
John Ruckstuhl

      1 """
      2 This module exports functions for removing old files and directories.
      3
      4 Functions
      5     myremove -- Remove the file or dir, and return the number
of bytes freed.
      6 """
      7
      8 # standard library imports
      9 import logging
     10 import os
     11 import shutil
     12 import subprocess
     13
     14 # related third party imports
     15 import win32api
     16 import win32security
     17
     18 # no local application/library specific imports
     19
     20
     21 logger = logging.getLogger(__name__)
     22
     23 # this custom startupinfo is used to suppress display of the
subprocess window
     24 startupinfo = subprocess.STARTUPINFO()
     25 startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
     26
     27
     28 def _remove(path):
     29     """Remove the file or dir, and return the number of bytes freed."""
     30
     31     ...
     32
     33
     34 def myremove(path):
     35     """Remove the file or dir, and return the number of bytes freed.
     36
     37     Attempt removal different ways, until successful (or defeated).
     38
     39     1.  Attempt call _remove (without additional prep).
     40     2.  Attempt to enable privileges SeBackupPrivilege and
     41         SeRestorePrivilege, then call _remove.
     42     3.  Attempt to takeown, then call _remove.
     43
     44     The enable privileges approach is ~6x faster than the
takeown approach,
     45     because takeown of an MEI dir containing ~1K files takes
time (8-9 sec).
     46
     47     Example results, enable privs approach
     48         MEI dirs: 113/113 removed, 2047.5 MB recovered in 2.0 minutes
     49
     50     Example results, takeown approach
     51         MEI dirs: 126/126 removed, 2453.8 MB recovered in 15.0 minutes
     52
     53     By default, the User Access Token for a member of local group
     54     Administrators has the privileges SeBackupPrivilege and
SeRestorePrivilege,
     55     but they are disabled.
     56
     57     The MEI dirs are typically ~1000 files occupying ~20 MB.
     58     The MEI dirs are created by users running the pyinstaller -generated
     59     executables ...
     60
     61     Presumably the process that calls this function is run as a user
     62     which is a member of local group "Administrators".  So
this process will
     63     have privileges SeBackupPrivilege and SeRestorePrivilege granted but
     64     disabled.  Also this process will be able to take
ownership, if needed.
     65     """
     66
     67     logger.debug('%s(%r)', 'myremove', path)
     68
     69     # first try, attempt _remove
     70     try:
     71         logger.debug('%s(%r)', '_remove', path)
     72         nbytes = _remove(path)
     73         return nbytes
     74     except WindowsError as e:
     75         if e.args == (5, 'Access is denied'):
     76             # report the exception and carry on to next attempt
     77             logger.debug(e)
     78         else:
     79             # re-raise any other WindowsError
     80             raise
     81
     82     # second try, attempt enable privs then _remove
     83     try:
     84         logger.debug('%s(%r)', '_remove', path)
     85         EnablePrivilege(win32security.SE_BACKUP_NAME)
     86         EnablePrivilege(win32security.SE_RESTORE_NAME)
     87         nbytes = _remove(path)
     88         return nbytes
     89     except Exception as e:
     90         # report the exception and carry on to next attempt
     91         logger.debug(e)
     92
     93     # third try, attempt takeown then _remove
     94     try:
     95         logger.debug('%s(%r)', '_remove', path)
     96         takeown(path)
     97         nbytes = _remove(path)
     98         return nbytes
     99     except Exception as e:
    100         # report the exception and carry on ...
    101         logger.debug(e)
    102
    103     # accept defeat
    104     raise
    105
    106
    107 def takeown(path):
    108     subprocess.check_call(
    109         [
    110             'takeown',
    111             '/A',
    112             '/F', path.replace('/', '\\'),
    113             '/R',
    114             '/D', 'Y',
    115         ],
    116         startupinfo=startupinfo)
    117
    118
    119 def EnablePrivilege(privilege):
    120     """Enable a privilege, if it's already granted in the User
Access Token.
    121
    122     From [1]:
    123         When a user holds a privilege, it allows that user to
do things that
    124         other users without that privilege are not allowed to
do. For example,
    125         the SeBackupPrivilege allows a user to read any file,
even if the
    126         security descriptor denies access.
    127         But just having the SeBackupPrivilege is not enough:
it needs to be
    128         enabled
    129
    130     [1]
https://blog.didierstevens.com/2021/07/19/using-sebackupprivilege-with-python
 # noqa: E501
    131     """
    132     hToken = win32security.OpenProcessToken(
    133         win32api.GetCurrentProcess(),
    134         win32security.TOKEN_ADJUST_PRIVILEGES |
win32security.TOKEN_QUERY
    135     )
    136     win32security.AdjustTokenPrivileges(
    137         hToken,
    138         0,
    139         [(
    140             win32security.LookupPrivilegeValue(None, privilege),
    141             win32security.SE_PRIVILEGE_ENABLED
    142         )]
    143     )
    144     win32api.CloseHandle(hToken)