Cygwin tools to read/write NTFS alternate data streams?

Cygwin tools to read/write NTFS alternate data streams?

[Martin Wege]

Hello,

Does Cygwin have tools (modified /usr/bin/dd ?) to read/write NTFS
alternate data streams?

Thanks,
Martin

Re: Cygwin tools to read/write NTFS alternate data streams?

[Corinna Vinschen]

On Nov 30 04:55, Martin Wege via Cygwin wrote:
> Hello,
> 
> Does Cygwin have tools (modified /usr/bin/dd ?) to read/write NTFS
> alternate data streams?

No.  As you know, the colon is translated to a normal filename
character, and there's no POSIX-like API to expose ADS raw to user
space.

There is, however, an old function we still expose to user space
for backward compat:

  #include <sys/cygwin.h>

  int cygwin_attach_handle_to_fd (char *name,
				  int fd,
				  HANDLE handle,
				  mode_t bin,
				  DWORD myaccess);

This allows to sneak in a HANDLE into a Cygwin file descriptor
representation, kind of like this:

  HANDLE h;
  int fd;

  h = CreateFile ("foo:bar", GENERIC_READ, FILE_SHARE_VALID_FLAGS,
		  NULL, OPEN_EXISTING, 0, NULL);
  if (h != INVALID_HANDLE_VALUE)
    {
      fd = cygwin_attach_handle_to_fd ("foo", -1, h, 0, GENERIC_READ);
      if (fd < 0)
	bail_out;
    }

For the bin parameter, only 0, O_BINARY or O_TEXT are acceptable,
for myaccess, only GENERIC_READ and/or GENERIC_WRITE are acceptable.


Corinna

Re: Cygwin tools to read/write NTFS alternate data streams?

[Martin Wege]

On Fri, Dec 1, 2023 at 10:52 AM Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Nov 30 04:55, Martin Wege via Cygwin wrote:
> > Hello,
> >
> > Does Cygwin have tools (modified /usr/bin/dd ?) to read/write NTFS
> > alternate data streams?
>
> No.  As you know, the colon is translated to a normal filename
> character, and there's no POSIX-like API to expose ADS raw to user
> space.
>
> There is, however, an old function we still expose to user space
> for backward compat:
>
>   #include <sys/cygwin.h>
>
>   int cygwin_attach_handle_to_fd (char *name,
>                                   int fd,
>                                   HANDLE handle,
>                                   mode_t bin,
>                                   DWORD myaccess);
>
> This allows to sneak in a HANDLE into a Cygwin file descriptor
> representation, kind of like this:
>
>   HANDLE h;
>   int fd;
>
>   h = CreateFile ("foo:bar", GENERIC_READ, FILE_SHARE_VALID_FLAGS,
>                   NULL, OPEN_EXISTING, 0, NULL);
>   if (h != INVALID_HANDLE_VALUE)
>     {
>       fd = cygwin_attach_handle_to_fd ("foo", -1, h, 0, GENERIC_READ);
>       if (fd < 0)
>         bail_out;
>     }
>
> For the bin parameter, only 0, O_BINARY or O_TEXT are acceptable,
> for myaccess, only GENERIC_READ and/or GENERIC_WRITE are acceptable.

Could this be abstracted into O_XATTR support, i.e. openat() file with
O_XATTR, and then have access to the alternate data streams in a
(virtual) subdir?

Thanks,
Martin

Re: Cygwin tools to read/write NTFS alternate data streams?

[Corinna Vinschen]

On Dec 18 18:47, Martin Wege via Cygwin wrote:
> On Fri, Dec 1, 2023 at 10:52 AM Corinna Vinschen via Cygwin
> <cygwin@cygwin.com> wrote:
> >
> > On Nov 30 04:55, Martin Wege via Cygwin wrote:
> > > Hello,
> > >
> > > Does Cygwin have tools (modified /usr/bin/dd ?) to read/write NTFS
> > > alternate data streams?
> >
> > No.  As you know, the colon is translated to a normal filename
> > character, and there's no POSIX-like API to expose ADS raw to user
> > space.
> >
> > There is, however, an old function we still expose to user space
> > for backward compat:
> >
> >   #include <sys/cygwin.h>
> >
> >   int cygwin_attach_handle_to_fd (char *name,
> >                                   int fd,
> >                                   HANDLE handle,
> >                                   mode_t bin,
> >                                   DWORD myaccess);
> >
> > This allows to sneak in a HANDLE into a Cygwin file descriptor
> > representation, kind of like this:
> >
> >   HANDLE h;
> >   int fd;
> >
> >   h = CreateFile ("foo:bar", GENERIC_READ, FILE_SHARE_VALID_FLAGS,
> >                   NULL, OPEN_EXISTING, 0, NULL);
> >   if (h != INVALID_HANDLE_VALUE)
> >     {
> >       fd = cygwin_attach_handle_to_fd ("foo", -1, h, 0, GENERIC_READ);
> >       if (fd < 0)
> >         bail_out;
> >     }
> >
> > For the bin parameter, only 0, O_BINARY or O_TEXT are acceptable,
> > for myaccess, only GENERIC_READ and/or GENERIC_WRITE are acceptable.
> 
> Could this be abstracted into O_XATTR support, i.e. openat() file with
> O_XATTR, and then have access to the alternate data streams in a
> (virtual) subdir?

I'm not too hot on adding more open(2) flags, the reason being,
that we have room left for only 7 more flags in the 32 bit mode.
Nobody knows what comes along officially in future.

Apart from that, this sounds like a nice idea for Cygwin 3.6,
provided somebody implements it, https://cygwin.com/acronyms/#SHTDI

Assuming we can live without actually having a subdir and just
allowing to open and create a file with the O_XATTR flag, it might be
pretty simple to implement.  The path handling code would just have to
drop the colon from the list of characters converted to the private-use
Unicode area.

Implementing the subdir is a bit more complicated, especially when
taking opendir/readdir of that virtual subdir into account, but it
would certainly be doable.


Corinna

Re: Cygwin tools to read/write NTFS alternate data streams?

[Martin Wege]

On Mon, Jan 8, 2024 at 3:11 PM Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Dec 18 18:47, Martin Wege via Cygwin wrote:
> > On Fri, Dec 1, 2023 at 10:52 AM Corinna Vinschen via Cygwin
> > <cygwin@cygwin.com> wrote:
> > >
> > > On Nov 30 04:55, Martin Wege via Cygwin wrote:
> > > > Hello,
> > > >
> > > > Does Cygwin have tools (modified /usr/bin/dd ?) to read/write NTFS
> > > > alternate data streams?
> > >
> > > No.  As you know, the colon is translated to a normal filename
> > > character, and there's no POSIX-like API to expose ADS raw to user
> > > space.
> > >
> > > There is, however, an old function we still expose to user space
> > > for backward compat:
> > >
> > >   #include <sys/cygwin.h>
> > >
> > >   int cygwin_attach_handle_to_fd (char *name,
> > >                                   int fd,
> > >                                   HANDLE handle,
> > >                                   mode_t bin,
> > >                                   DWORD myaccess);
> > >
> > > This allows to sneak in a HANDLE into a Cygwin file descriptor
> > > representation, kind of like this:
> > >
> > >   HANDLE h;
> > >   int fd;
> > >
> > >   h = CreateFile ("foo:bar", GENERIC_READ, FILE_SHARE_VALID_FLAGS,
> > >                   NULL, OPEN_EXISTING, 0, NULL);
> > >   if (h != INVALID_HANDLE_VALUE)
> > >     {
> > >       fd = cygwin_attach_handle_to_fd ("foo", -1, h, 0, GENERIC_READ);
> > >       if (fd < 0)
> > >         bail_out;
> > >     }
> > >
> > > For the bin parameter, only 0, O_BINARY or O_TEXT are acceptable,
> > > for myaccess, only GENERIC_READ and/or GENERIC_WRITE are acceptable.
> >
> > Could this be abstracted into O_XATTR support, i.e. openat() file with
> > O_XATTR, and then have access to the alternate data streams in a
> > (virtual) subdir?
>
> I'm not too hot on adding more open(2) flags, the reason being,
> that we have room left for only 7 more flags in the 32 bit mode.
> Nobody knows what comes along officially in future.

Right, but Solaris, Illumos, and Opengroup draft already have O_XATTR,
so this is IMO OK.

Also, it would provide a good abstraction for alternate data stream
support, which is compatible with Windows, OSX, Solaris, Illumos, and
NTFS, ZFS, SMB, NFSv4.

>
> Apart from that, this sounds like a nice idea for Cygwin 3.6,
> provided somebody implements it, https://cygwin.com/acronyms/#SHTDI
>
> Assuming we can live without actually having a subdir and just
> allowing to open and create a file with the O_XATTR flag, it might be
> pretty simple to implement.  The path handling code would just have to
> drop the colon from the list of characters converted to the private-use
> Unicode area.
>
> Implementing the subdir is a bit more complicated, especially when
> taking opendir/readdir of that virtual subdir into account, but it
> would certainly be doable.

How do other OSes implement the O_XATTR subdir?

Thanks,
Martin

Re: Cygwin tools to read/write NTFS alternate data streams?

[Corinna Vinschen]

On Jan 11 15:00, Martin Wege via Cygwin wrote:
> On Mon, Jan 8, 2024 at 3:11 PM Corinna Vinschen via Cygwin
> <cygwin@cygwin.com> wrote:
> so this is IMO OK.

Yeah, but...

It's not just an open flag, it requires extending functionality of other
APIs and, for full support, tools, see
https://docs.oracle.com/cd/E19253-01/816-5175/6mbba7f02/.

It's also rather weird to call alternate data streams "extended
attributes", because that's just a small part of streams and it's
already supported via the Linux functions getxattr(2) and friends, see
https://man7.org/linux/man-pages/man2/getxattr.2.html

> > Apart from that, this sounds like a nice idea for Cygwin 3.6,
> > provided somebody implements it, https://cygwin.com/acronyms/#SHTDI
> >
> > Assuming we can live without actually having a subdir and just
> > allowing to open and create a file with the O_XATTR flag, it might be
> > pretty simple to implement.  The path handling code would just have to
> > drop the colon from the list of characters converted to the private-use
> > Unicode area.
> >
> > Implementing the subdir is a bit more complicated, especially when
> > taking opendir/readdir of that virtual subdir into account, but it
> > would certainly be doable.
> 
> How do other OSes implement the O_XATTR subdir?

IDK.  But there are a few points we have to keep in mind from the
Solaris man page:

- Given fd is an open file descriptor,

    openat(fd, ".", O_RDONLY|O_XATTR);

  opens the virtual subdir containing the ADS of a file.  It's the
  only valid way to open the virtual dir.  That's a bit of a relief
  because it simplifies handling this in openat(2).

    openat(fd, "foo", ...|O_XATTR);

  allows to create or open an ADS for an open file fd.

- unlinkat, renameat, fstatat, fchownat, futimesat and fdopendir need to
  be made ADS-aware.  They only work on ADS if the fd arg already
  resolves to an ADS, or if fd is the virtual ADS dir of a file and
  (except in case of fdopendir) the path argument is the name of an
  existing ADS.

- pathconf needs to support a new flag XATTR_ENABLED.

This can be made to work given the current fhandler framework in Cygwin,
but again, https://cygwin.com/acronyms/#SHTDI

FTR, I'm generally not a friend of ADS, because they look like a builtin
security problem.  They don't show up in standard directory listings,
and you can't even see that a file has ADS, except you look for them
explicitely.  This is a really user-unfriendly interface.


Corinna

Re: Cygwin tools to read/write NTFS alternate data streams?

[Roland Mainz]

On Thu, Jan 11, 2024 at 3:00 PM Martin Wege via Cygwin
<cygwin@cygwin.com> wrote:
> On Mon, Jan 8, 2024 at 3:11 PM Corinna Vinschen via Cygwin
> <cygwin@cygwin.com> wrote:
> > On Dec 18 18:47, Martin Wege via Cygwin wrote:
> > > On Fri, Dec 1, 2023 at 10:52 AM Corinna Vinschen via Cygwin
> > > <cygwin@cygwin.com> wrote:
[snip]
> > Apart from that, this sounds like a nice idea for Cygwin 3.6,
> > provided somebody implements it, https://cygwin.com/acronyms/#SHTDI
> >
> > Assuming we can live without actually having a subdir and just
> > allowing to open and create a file with the O_XATTR flag, it might be
> > pretty simple to implement.  The path handling code would just have to
> > drop the colon from the list of characters converted to the private-use
> > Unicode area.
> >
> > Implementing the subdir is a bit more complicated, especially when
> > taking opendir/readdir of that virtual subdir into account, but it
> > would certainly be doable.
>
> How do other OSes implement the O_XATTR subdir?

Basically you open a random file or dir with |openat(filename,
O_XATTR)| and get a fd to that (virtual) attribute directory.
The "." in that dir is virtual, but the ".." refers to the underlying
file, so a |open("..", ...)| will return a fd to a file/dir to which
the attributes belong to.

https://illumos.org/man/7/fsattr has the whole API description.

A Windows implementation should be therefore quite easy, as alternate
data streams show up as "foo:stream1", "foo:stream2" etc., so just the
directory lookup needs to be modified to skip the files with ':' in
general, and only show the streams for an O_XATTR subdir, e.g.
|openat("foo", O_XATTR)| would give a dir which lists only Windows
file names which match "foo:*"

I can support such an effort with testing, and a cleanroom
implementation of https://illumos.org/man/1/runat (see
https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/runat/runat.c
for the Solaris/Illumos code)

----

Bye,
Roland
-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz@nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Re: Cygwin tools to read/write NTFS alternate data streams?

[Roland Mainz]

On Thu, Jan 11, 2024 at 5:41 PM Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Jan 11 15:00, Martin Wege via Cygwin wrote:
> > On Mon, Jan 8, 2024 at 3:11 PM Corinna Vinschen via Cygwin
> > <cygwin@cygwin.com> wrote:
> > so this is IMO OK.
>
> Yeah, but...
>
> It's not just an open flag, it requires extending functionality of other
> APIs and, for full support, tools, see
> https://docs.oracle.com/cd/E19253-01/816-5175/6mbba7f02/.
>
> It's also rather weird to call alternate data streams "extended
> attributes", because that's just a small part of streams and it's
> already supported via the Linux functions getxattr(2) and friends, see
> https://man7.org/linux/man-pages/man2/getxattr.2.html
>
> > > Apart from that, this sounds like a nice idea for Cygwin 3.6,
> > > provided somebody implements it, https://cygwin.com/acronyms/#SHTDI
> > >
> > > Assuming we can live without actually having a subdir and just
> > > allowing to open and create a file with the O_XATTR flag, it might be
> > > pretty simple to implement.  The path handling code would just have to
> > > drop the colon from the list of characters converted to the private-use
> > > Unicode area.
> > >
> > > Implementing the subdir is a bit more complicated, especially when
> > > taking opendir/readdir of that virtual subdir into account, but it
> > > would certainly be doable.
> >
> > How do other OSes implement the O_XATTR subdir?
>
> IDK.  But there are a few points we have to keep in mind from the
> Solaris man page:
>
> - Given fd is an open file descriptor,
>
>     openat(fd, ".", O_RDONLY|O_XATTR);
>
>   opens the virtual subdir containing the ADS of a file.  It's the
>   only valid way to open the virtual dir.  That's a bit of a relief
>   because it simplifies handling this in openat(2).
>
>     openat(fd, "foo", ...|O_XATTR);
>
>   allows to create or open an ADS for an open file fd.
>
> - unlinkat, renameat, fstatat, fchownat, futimesat and fdopendir need to
>   be made ADS-aware.  They only work on ADS if the fd arg already
>   resolves to an ADS, or if fd is the virtual ADS dir of a file and
>   (except in case of fdopendir) the path argument is the name of an
>   existing ADS.
>
> - pathconf needs to support a new flag XATTR_ENABLED.
>
> This can be made to work given the current fhandler framework in Cygwin,
> but again, https://cygwin.com/acronyms/#SHTDI
>
> FTR, I'm generally not a friend of ADS, because they look like a builtin
> security problem.

See below. I'm not 100% happy with that either, but these things
exists in many operating systems, and constantly ignoring them just
because it is not part of POSIX does not serve the users. So far SUN's
Solaris team came up with an API which works with Windows and UNIX
APIs, and it is much better than the alternatives I've seen so far...

> They don't show up in standard directory listings,
> and you can't even see that a file has ADS, except you look for them
> explicitely.  This is a really user-unfriendly interface.

Actually Solaris /usr/bin/ls has option "-@" (basically ls -l plus ADS
support, e.g. ls -l output has an @ displayed after the file
permission bits for files that have alternate data streams).
The plan was to make that the default for ls -l in Solaris 12, but
that plan was ruined by the SUN takeover.

I can make a patch for Cygwin /usr/bin/ls and /usr/bin/find (options
-xattr/-ads) once O_XATTR support is available, and provide an
implementation of runat(1), so people can easily see whether ADS are
around, and access them via runat(1).

----

Bye,
Roland
-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz@nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Re: Cygwin tools to read/write NTFS alternate data streams?

[Cedric Blancher]

On Thu, 11 Jan 2024 at 17:41, Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Jan 11 15:00, Martin Wege via Cygwin wrote:
> > On Mon, Jan 8, 2024 at 3:11 PM Corinna Vinschen via Cygwin
> > <cygwin@cygwin.com> wrote:
> > so this is IMO OK.
>
> Yeah, but...
>
> It's not just an open flag, it requires extending functionality of other
> APIs and, for full support, tools, see
> https://docs.oracle.com/cd/E19253-01/816-5175/6mbba7f02/.
>
> It's also rather weird to call alternate data streams "extended
> attributes", because that's just a small part of streams and it's
> already supported via the Linux functions getxattr(2) and friends, see
> https://man7.org/linux/man-pages/man2/getxattr.2.html
>
> > > Apart from that, this sounds like a nice idea for Cygwin 3.6,
> > > provided somebody implements it, https://cygwin.com/acronyms/#SHTDI
> > >
> > > Assuming we can live without actually having a subdir and just
> > > allowing to open and create a file with the O_XATTR flag, it might be
> > > pretty simple to implement.  The path handling code would just have to
> > > drop the colon from the list of characters converted to the private-use
> > > Unicode area.
> > >
> > > Implementing the subdir is a bit more complicated, especially when
> > > taking opendir/readdir of that virtual subdir into account, but it
> > > would certainly be doable.
> >
> > How do other OSes implement the O_XATTR subdir?
>
> IDK.  But there are a few points we have to keep in mind from the
> Solaris man page:
>
> - Given fd is an open file descriptor,
>
>     openat(fd, ".", O_RDONLY|O_XATTR);
>
>   opens the virtual subdir containing the ADS of a file.  It's the
>   only valid way to open the virtual dir.  That's a bit of a relief
>   because it simplifies handling this in openat(2).
>
>     openat(fd, "foo", ...|O_XATTR);
>
>   allows to create or open an ADS for an open file fd.
>
> - unlinkat, renameat, fstatat, fchownat, futimesat and fdopendir need to
>   be made ADS-aware.  They only work on ADS if the fd arg already
>   resolves to an ADS, or if fd is the virtual ADS dir of a file and
>   (except in case of fdopendir) the path argument is the name of an
>   existing ADS.
>
> - pathconf needs to support a new flag XATTR_ENABLED.
>
> This can be made to work given the current fhandler framework in Cygwin,
> but again, https://cygwin.com/acronyms/#SHTDI

bash has cd -@ to enter a virtual dir with XATTRs, if O_XATTR works.

Aside from that, I volunteer with testing of O_XATTR, on demand with 1
or 2 students too.

Still working on testing the cool and new SEEK_HOLE support in Cygwin

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: Cygwin tools to read/write NTFS alternate data streams?

[Corinna Vinschen]

On Jan 12 10:24, Cedric Blancher via Cygwin wrote:
> On Thu, 11 Jan 2024 at 17:41, Corinna Vinschen via Cygwin
> <cygwin@cygwin.com> wrote:
> Aside from that, I volunteer with testing of O_XATTR, on demand with 1
> or 2 students too.

Yup, I'm volunteering for testing, too.  Now we just have to find
somebody implementing this stuff in Cygwin.  Not me, because I think
ADS should ideally die a peaceful but quick death.  Any takers?


Corinna