Duplicates in /proc/partitions

Duplicates in /proc/partitions

[David Balažic]

Before, during and after plugging in a n USB stick:

$ cat /proc/partitions
major minor  #blocks  name   win-mounts

    8     0 1000204632 sda
    8     1    102400 sda1
    8     2     16384 sda2
    8     3 999571820 sda3   C:\
    8     4    510976 sda4

$ cat /proc/partitions
major minor  #blocks  name   win-mounts

    8     0 1000204632 sda
    8     1    102400 sda1
    8     2     16384 sda2
    8     3 999571820 sda3   C:\
    8     4    510976 sda4
    8    16  62522712 sdb
    8    17   4096000 sdb1   D:\
    8    18  58424664 sdb2
    8    16  62522712 sdb
    8    17   4096000 sdb1   D:\
    8    18  58424664 sdb2   E:\

$ cat /proc/partitions
major minor  #blocks  name   win-mounts

    8     0 1000204632 sda
    8     1    102400 sda1
    8     2     16384 sda2
    8     3 999571820 sda3   C:\
    8     4    510976 sda4
    8    16  62522712 sdb
    8    17   4096000 sdb1   D:\
    8    18  58424664 sdb2   E:\


So the second listing shows sdb twice. Also E: does not seem to exist
(see below).

After that I run:

$ cygcheck -s -v -r > cygcheck.out
cygcheck: dump_sysinfo: GetVolumeInformation() for drive E: failed: 1005

In attached cygcheck.out i swapped out company name and abbreviation with XXXX.

Windows sees no E: drive:

DISKPART> list volume

  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     C                NTFS   Partition    953 GB  Healthy    Boot
  Volume 1                      FAT32  Partition    100 MB  Healthy    System
  Volume 2                      NTFS   Partition    499 MB  Healthy    Hidden
  Volume 3     D   FFT2         FAT    Removable   4000 MB  Healthy

(there is no mention of E: in Disk Management GUI either)


Regards,
David

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> So the second listing shows sdb twice. Also E: does not seem to exist

Hi David,

I've seen double too, but was told that there's something off with my Windows (7, Home), which I hardly believe is true... but anyways,
can you do the cat command from an elevated prompt, by any chance?  On my box, I saw that it prints everything correctly then.

Just trying to see it I'm not actually "seeing double" :-)

Thanks,

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

On Aug 13 12:56, David Balažic via Cygwin wrote:
> Before, during and after plugging in a n USB stick:
> 
> $ cat /proc/partitions
> major minor  #blocks  name   win-mounts
> 
>     8     0 1000204632 sda
>     8     1    102400 sda1
>     8     2     16384 sda2
>     8     3 999571820 sda3   C:\
>     8     4    510976 sda4
> 
> $ cat /proc/partitions
> major minor  #blocks  name   win-mounts
> 
>     8     0 1000204632 sda
>     8     1    102400 sda1
>     8     2     16384 sda2
>     8     3 999571820 sda3   C:\
>     8     4    510976 sda4
>     8    16  62522712 sdb
>     8    17   4096000 sdb1   D:\
>     8    18  58424664 sdb2
>     8    16  62522712 sdb
>     8    17   4096000 sdb1   D:\
>     8    18  58424664 sdb2   E:\
> 
> $ cat /proc/partitions
> major minor  #blocks  name   win-mounts
> 
>     8     0 1000204632 sda
>     8     1    102400 sda1
>     8     2     16384 sda2
>     8     3 999571820 sda3   C:\
>     8     4    510976 sda4
>     8    16  62522712 sdb
>     8    17   4096000 sdb1   D:\
>     8    18  58424664 sdb2   E:\
> 
> 
> So the second listing shows sdb twice. Also E: does not seem to exist
> (see below).

So, second report.  Which OS is that on?  I don't have any physical
Windows machine anymore, so I have a bit of a problem to reproduce this.

As I wrote in https://cygwin.com/pipermail/cygwin/2021-August/249012.html,
I don't see how Cygwin would be able to duplicate entries, unless the OS
itself duplicates entries.

Somebody who can duplicate this issue would have to single step through
function format_proc_partitions, see

https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_proc.cc;h=66d19ab82f;hb=HEAD#l1686

and check what actually happens in the above cases.


Corinna

Re: Duplicates in /proc/partitions

[Brian Inglis]

[-- Attachment #1: Type: text/plain, Size: 2548 bytes --]

On 2021-08-16 08:51, Corinna Vinschen via Cygwin wrote:
> On Aug 13 12:56, David Balažic via Cygwin wrote:
>> Before, during and after plugging in a n USB stick:
>>
>> $ cat /proc/partitions
>> major minor  #blocks  name   win-mounts
>>
>>      8     0 1000204632 sda
>>      8     1    102400 sda1
>>      8     2     16384 sda2
>>      8     3 999571820 sda3   C:\
>>      8     4    510976 sda4
>>
>> $ cat /proc/partitions
>> major minor  #blocks  name   win-mounts
>>
>>      8     0 1000204632 sda
>>      8     1    102400 sda1
>>      8     2     16384 sda2
>>      8     3 999571820 sda3   C:\
>>      8     4    510976 sda4
>>      8    16  62522712 sdb
>>      8    17   4096000 sdb1   D:\
>>      8    18  58424664 sdb2
>>      8    16  62522712 sdb
>>      8    17   4096000 sdb1   D:\
>>      8    18  58424664 sdb2   E:\
>>
>> $ cat /proc/partitions
>> major minor  #blocks  name   win-mounts
>>
>>      8     0 1000204632 sda
>>      8     1    102400 sda1
>>      8     2     16384 sda2
>>      8     3 999571820 sda3   C:\
>>      8     4    510976 sda4
>>      8    16  62522712 sdb
>>      8    17   4096000 sdb1   D:\
>>      8    18  58424664 sdb2   E:\
>>
>>
>> So the second listing shows sdb twice. Also E: does not seem to exist
>> (see below).
> 
> So, second report.  Which OS is that on?  I don't have any physical
> Windows machine anymore, so I have a bit of a problem to reproduce this.
> 
> As I wrote in https://cygwin.com/pipermail/cygwin/2021-August/249012.html,
> I don't see how Cygwin would be able to duplicate entries, unless the OS
> itself duplicates entries.
> 
> Somebody who can duplicate this issue would have to single step through
> function format_proc_partitions, see
> 
> https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_proc.cc;h=66d19ab82f;hb=HEAD#l1686
> 
> and check what actually happens in the above cases.

Could those with the symptoms not provide useful information using a 
script run in an elevated admin shell listing the relevant contents of 
Windows objects accessed by format_proc_partitions from /proc/sys/ 
similar to the attached script and log from my system, where the 
correspondence is obvious?

e.g. save script and run as:

# sh .../proc-sys-partitions.sh.txt |& tee .../proc-sys-partitions.log

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

[-- Attachment #2: proc-sys-partitions.sh.txt --]
[-- Type: text/plain, Size: 450 bytes --]

#!/bin/sh
# proc-sys-partitions.sh - list /proc/partitions and /proc/sys/ related objects

echo '#' '# elevated admin shell required'

cmd="tail -vn+0"

for p in /proc/partitions \
	"/proc/sys/Device/Harddisk[0-9]/Partition[0-9]" \
	"/proc/sys/DosDevices/GLOBALROOT/Device/Harddisk[0-9]/Partition[0-9]" \
	"/proc/sys/Device/Harddisk[0-9]/DR[0-9]" \
	"/proc/sys/Device/HarddiskVolume[0-9]"
do
    echo '#' $cmd "$p"
    $cmd $p

    cmd="ls -l"
done


[-- Attachment #3: proc-sys-partitions.log --]
[-- Type: text/plain, Size: 4836 bytes --]

# # elevated admin shell required
# tail -vn+0 /proc/partitions
==> /proc/partitions <==
major minor  #blocks  name   win-mounts

    8     0 976762584 sda
    8     1     16384 sda1
    8     2 976206318 sda2   C:\
    8     3    536576 sda3  
    8    16 976762584 sdb
    8    17    131072 sdb1
    8    18    102400 sdb2  
    8    19 975482161 sdb3   D:\
    8    20    577536 sdb4  
    8    21    465920 sdb5  
    8    32         0 sdc
# ls -l /proc/sys/Device/Harddisk[0-9]/Partition[0-9]
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk0/Partition0 -> /proc/sys/Device/Harddisk0/DR0
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk0/Partition1 -> /proc/sys/Device/HarddiskVolume1
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk0/Partition2 -> /proc/sys/Device/HarddiskVolume2
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk0/Partition3 -> /proc/sys/Device/HarddiskVolume3
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk1/Partition0 -> /proc/sys/Device/Harddisk1/DR1
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk1/Partition1 -> /proc/sys/Device/HarddiskVolume4
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk1/Partition2 -> /proc/sys/Device/HarddiskVolume5
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk1/Partition3 -> /proc/sys/Device/HarddiskVolume6
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk1/Partition4 -> /proc/sys/Device/HarddiskVolume7
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk1/Partition5 -> /proc/sys/Device/HarddiskVolume8
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk2/Partition0 -> /proc/sys/Device/Harddisk2/DR2
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/Device/Harddisk2/Partition1 -> /proc/sys/Device/HarddiskVolume9
# ls -l /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk[0-9]/Partition[0-9]
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk0/Partition0 -> /proc/sys/Device/Harddisk0/DR0
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk0/Partition1 -> /proc/sys/Device/HarddiskVolume1
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk0/Partition2 -> /proc/sys/Device/HarddiskVolume2
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk0/Partition3 -> /proc/sys/Device/HarddiskVolume3
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk1/Partition0 -> /proc/sys/Device/Harddisk1/DR1
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk1/Partition1 -> /proc/sys/Device/HarddiskVolume4
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk1/Partition2 -> /proc/sys/Device/HarddiskVolume5
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk1/Partition3 -> /proc/sys/Device/HarddiskVolume6
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk1/Partition4 -> /proc/sys/Device/HarddiskVolume7
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk1/Partition5 -> /proc/sys/Device/HarddiskVolume8
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk2/Partition0 -> /proc/sys/Device/Harddisk2/DR2
lr--r--r-- 1 Administrators SYSTEM 0 Aug 16 10:50 /proc/sys/DosDevices/GLOBALROOT/Device/Harddisk2/Partition1 -> /proc/sys/Device/HarddiskVolume9
# ls -l /proc/sys/Device/Harddisk[0-9]/DR[0-9]
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/Harddisk0/DR0
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/Harddisk1/DR1
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/Harddisk2/DR2
# ls -l /proc/sys/Device/HarddiskVolume[0-9]
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume1
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume2
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume3
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume4
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume5
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume6
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume7
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume8
brw-rw-rw- 1 BWI Users 284, 9999 Aug 16 10:50 /proc/sys/Device/HarddiskVolume9

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> Could those with the symptoms not provide useful information using a
> script run in an elevated admin shell listing the relevant contents of

Previously I reported that it worked for me correctly when elevated, and did
not work as an ordinary user.

I have to take this back (see the snapshots that follow).

Winobj (runs elevated to be able to open those entries) reports all drives without any duplication.
And so does diskpart.

Also, looks like C: is always shown correctly in /proc/partitions:  with the correct number
of partitions, and without any repetitions.

Right now, as an ordinary user:

$ cat /proc/partitions
major minor  #blocks  name   win-mounts

    8     0 500107608 sda
    8     1    102400 sda1
    8     2 488280064 sda2   C:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1   D:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1   G:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    80         0 sdf

As admin:

$ cat /proc/partitions
major minor  #blocks  name   win-mounts

    8     0 500107608 sda
    8     1    102400 sda1
    8     2 488280064 sda2   C:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1   D:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1   D:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1   G:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1   G:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    80         0 sdf

It's Windows 7 Home.

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3132 bytes --]

On Aug 16 10:56, Brian Inglis wrote:
> On 2021-08-16 08:51, Corinna Vinschen via Cygwin wrote:
> > On Aug 13 12:56, David Balažic via Cygwin wrote:
> > > [...]
> > > $ cat /proc/partitions
> > > major minor  #blocks  name   win-mounts
> > > 
> > >      8     0 1000204632 sda
> > >      8     1    102400 sda1
> > >      8     2     16384 sda2
> > >      8     3 999571820 sda3   C:\
> > >      8     4    510976 sda4
> > >      8    16  62522712 sdb
> > >      8    17   4096000 sdb1   D:\
> > >      8    18  58424664 sdb2
> > >      8    16  62522712 sdb
> > >      8    17   4096000 sdb1   D:\
> > >      8    18  58424664 sdb2   E:\
> > > [...]
> > > So the second listing shows sdb twice. Also E: does not seem to exist
> > > (see below).
> > 
> > So, second report.  Which OS is that on?  I don't have any physical
> > Windows machine anymore, so I have a bit of a problem to reproduce this.
> > 
> > As I wrote in https://cygwin.com/pipermail/cygwin/2021-August/249012.html,
> > I don't see how Cygwin would be able to duplicate entries, unless the OS
> > itself duplicates entries.
> > 
> > Somebody who can duplicate this issue would have to single step through
> > function format_proc_partitions, see
> > 
> > https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_proc.cc;h=66d19ab82f;hb=HEAD#l1686
> > 
> > and check what actually happens in the above cases.
> 
> Could those with the symptoms not provide useful information using a script
> run in an elevated admin shell listing the relevant contents of Windows
> objects accessed by format_proc_partitions from /proc/sys/ similar to the
> attached script and log from my system, where the correspondence is obvious?

Not sure if that helps, but it's worth a try.

Other than that, I created a standalone application, shamelessly copying
the code from Cygwin's function format_proc_partitions() and tweaked it
to drop Cygwin internals.

The core of the function doesn't actually need Cygwin internals. It's
basically a double loop calling Windows functions and just using Cygwin
internals to print out major and minor numbers and the device name.  The
standalone application simply fakes those.

This allows any person who *can* reproduce the problem to run the code
under a debugger like GDB and debug what's going wrong.

I attached the code to this mail.  Store it under the name "proc_partition.c".  Compile it in two different ways, first with optimization:

$ gcc -g -O2 -o proc_partition_O2 proc_partition.c -lntdll

Run it:

$ ./proc_partition_O2

If the output contains duplicates, take a note, but otherwise
ignore it for now.

Next, build it without optimization for easier debugging:

$ gcc -g -o proc_partition proc_partition.c -lntdll

Run it:

$ ./proc_partition

If the output contains duplicates, great!  Now you can run it under
GDB and step through and see where it goes awry.  If it doesn't, you'll
have to debug the optimized version, which is sometimes a bit weird.
I'd be really glad if this allows us to reproduce the issue.

If none of them reproduces the issue... uhm... anyway, let's try first.


Thanks,
Corinna

[-- Attachment #2: proc_partition.c --]
[-- Type: text/plain, Size: 6152 bytes --]

#include <stdio.h>
#include <windows.h>
#include <winioctl.h>
#include <winternl.h>
#include <wchar.h>
#include <wctype.h>

NTSTATUS NTAPI NtOpenDirectoryObject (PHANDLE, ACCESS_MASK, POBJECT_ATTRIBUTES);
NTSTATUS NTAPI NtQueryDirectoryObject (HANDLE, PVOID, ULONG, BOOLEAN,
				       BOOLEAN, PULONG, PULONG);

typedef struct _DIRECTORY_BASIC_INFORMATION
{
  UNICODE_STRING ObjectName;
  UNICODE_STRING ObjectTypeName;
} DIRECTORY_BASIC_INFORMATION, *PDIRECTORY_BASIC_INFORMATION;

#define DIRECTORY_QUERY 1
#define NT_MAX_PATH	32768

char buf[NT_MAX_PATH];
char ioctl_buf[NT_MAX_PATH];
WCHAR mp_buf[NT_MAX_PATH];

typedef enum { false, true } bool;

int
main ()
{
  OBJECT_ATTRIBUTES attr;
  IO_STATUS_BLOCK io;
  NTSTATUS status;
  HANDLE dirhdl;
  WCHAR fpath[MAX_PATH];
  WCHAR gpath[MAX_PATH];
  DWORD len;

  /* Note that device ids and names are just faked here, not using the internal
     Cygwin code to generate device major/minor numbers and device names.
     There's *no* guarantee that the device name and the device major number
     is the same as in Cygwin. */
  char dev_name = '`';

  /* Open \Device object directory. */
  wchar_t wpath[MAX_PATH] = L"\\Device";
  UNICODE_STRING upath = {14, 16, wpath};

  InitializeObjectAttributes (&attr, &upath, OBJ_CASE_INSENSITIVE, NULL, NULL);
  status = NtOpenDirectoryObject (&dirhdl, DIRECTORY_QUERY, &attr);
  if (!NT_SUCCESS (status))
    {
      fprintf (stderr, "NtOpenDirectoryObject, status 0x%08x", status);
      return 1;
    }

  /* Traverse \Device directory ... */
  PDIRECTORY_BASIC_INFORMATION dbi = (PDIRECTORY_BASIC_INFORMATION)
				     alloca (640);
  BOOLEAN restart = TRUE;
  bool got_one = false;
  ULONG context = 0;
  while (NT_SUCCESS (NtQueryDirectoryObject (dirhdl, dbi, 640, TRUE, restart,
					     &context, NULL)))
    {
      HANDLE devhdl;
      PARTITION_INFORMATION_EX *pix = NULL;
      PARTITION_INFORMATION *pi = NULL;
      DWORD bytes_read;
      DWORD part_cnt = 0;
      unsigned long long size;

      restart = FALSE;
      /* ... and check for a "Harddisk[0-9]*" entry. */
      if (dbi->ObjectName.Length < 9 * sizeof (WCHAR)
	  || wcsncasecmp (dbi->ObjectName.Buffer, L"Harddisk", 8) != 0
	  || !iswdigit (dbi->ObjectName.Buffer[8]))
	continue;
      /* Got it.  Now construct the path to the entire disk, which is
	 "\\Device\\HarddiskX\\Partition0", and open the disk with
	 minimum permissions. */
      //unsigned long drive_num = wcstoul (dbi->ObjectName.Buffer + 8, NULL, 10);
      wcscpy (wpath, dbi->ObjectName.Buffer);
      PWCHAR wpart = wpath + dbi->ObjectName.Length / sizeof (WCHAR);
      wcpcpy (wpart, L"\\Partition0");
      upath.Length = dbi->ObjectName.Length + 22;
      upath.MaximumLength = upath.Length + sizeof (WCHAR);
      InitializeObjectAttributes (&attr, &upath, OBJ_CASE_INSENSITIVE,
				  dirhdl, NULL);
      status = NtOpenFile (&devhdl, READ_CONTROL, &attr, &io,
			   FILE_SHARE_VALID_FLAGS, 0);
      if (!NT_SUCCESS (status))
	{
	  fprintf (stderr, "NtOpenFile(%ls), status 0x%08x",
		   upath.Buffer, status);
	  continue;
	}
      if (!got_one)
	{
	  printf ("major minor  #blocks  name   win-mounts\n\n");
	  got_one = true;
	}
      /* Fetch partition info for the entire disk to get its size. */
      if (DeviceIoControl (devhdl, IOCTL_DISK_GET_PARTITION_INFO_EX, NULL, 0,
			   ioctl_buf, NT_MAX_PATH, &bytes_read, NULL))
	{
	  pix = (PARTITION_INFORMATION_EX *) ioctl_buf;
	  size = pix->PartitionLength.QuadPart;
	}
      else if (DeviceIoControl (devhdl, IOCTL_DISK_GET_PARTITION_INFO, NULL, 0,
				ioctl_buf, NT_MAX_PATH, &bytes_read, NULL))
	{
	  pi = (PARTITION_INFORMATION *) ioctl_buf;
	  size = pi->PartitionLength.QuadPart;
	}
      else
	{
	  fprintf (stderr, "DeviceIoControl (%ls, "
		   "IOCTL_DISK_GET_PARTITION_INFO{_EX}) %lu",
		   upath.Buffer, (unsigned long) GetLastError ());
	  size = 0;
	}
      //device dev (drive_num, 0);
      ++dev_name;
      printf ("%5d %5d %9llu sd%c\n",
	      8, (dev_name - 'a') * 16, size >> 10, dev_name);

      /* Fetch drive layout info to get size of all partitions on the disk. */
      if (DeviceIoControl (devhdl, IOCTL_DISK_GET_DRIVE_LAYOUT_EX,
			   NULL, 0, ioctl_buf, NT_MAX_PATH, &bytes_read, NULL))
	{
	  PDRIVE_LAYOUT_INFORMATION_EX pdlix = (PDRIVE_LAYOUT_INFORMATION_EX)
					       ioctl_buf;
	  part_cnt = pdlix->PartitionCount;
	  pix = pdlix->PartitionEntry;
	}
      else if (DeviceIoControl (devhdl, IOCTL_DISK_GET_DRIVE_LAYOUT,
				NULL, 0, ioctl_buf, NT_MAX_PATH, &bytes_read, NULL))
	{
	  PDRIVE_LAYOUT_INFORMATION pdli = (PDRIVE_LAYOUT_INFORMATION) ioctl_buf;
	  part_cnt = pdli->PartitionCount;
	  pi = pdli->PartitionEntry;
	}
      else
	fprintf (stderr, "DeviceIoControl(%ls, "
		  "IOCTL_DISK_GET_DRIVE_LAYOUT{_EX}): %lu",
		  upath.Buffer, (unsigned long) GetLastError ());
      /* Loop over partitions. */
      if (pix || pi)
	for (DWORD i = 0; i < part_cnt && i < 64; ++i)
	  {
	    DWORD part_num;

	    if (pix)
	      {
		size = pix->PartitionLength.QuadPart;
		part_num = pix->PartitionNumber;
		++pix;
	      }
	    else
	      {
		size = pi->PartitionLength.QuadPart;
		part_num = pi->PartitionNumber;
		++pi;
	      }
	    /* A partition number of 0 denotes an extended partition or a
	       filler entry as described in fhandler_dev_floppy::lock_partition.
	       Just skip. */
	    if (part_num == 0)
	      continue;

	    //device dev (drive_num, part_num);
	    printf ("%5d %5d %9llu sd%c%d   ",
		    8, (dev_name - 'a') * 16 + part_num, size >> 10,
		    dev_name, part_num);
	    /* Check if the partition is mounted in Windows and, if so,
	       print the mount point list. */
	    swprintf (fpath, sizeof fpath,
		      L"\\\\?\\GLOBALROOT\\Device\\%ls\\Partition%u\\",
		      dbi->ObjectName.Buffer, part_num);
	    if (GetVolumeNameForVolumeMountPointW (fpath, gpath, MAX_PATH)
		&& GetVolumePathNamesForVolumeNameW (gpath, mp_buf,
						     NT_MAX_PATH, &len))
	      {
		for (PWCHAR p = mp_buf; *p; p = wcschr (p, L'\0') + 1)
		  printf (" %ls", p);
	      }
	    puts ("");
	  }
      NtClose (devhdl);
    }
  NtClose (dirhdl);

  if (!got_one)
    return 1;

  return 0;
}

Re: Duplicates in /proc/partitions

[Brian Inglis]

On 2021-08-16 12:49, Corinna Vinschen via Cygwin wrote:
> On Aug 16 10:56, Brian Inglis wrote:
>> On 2021-08-16 08:51, Corinna Vinschen via Cygwin wrote:
>>> On Aug 13 12:56, David Balažic via Cygwin wrote:
>>>> [...]
>>>> $ cat /proc/partitions
>>>> major minor  #blocks  name   win-mounts
>>>>
>>>>       8     0 1000204632 sda
>>>>       8     1    102400 sda1
>>>>       8     2     16384 sda2
>>>>       8     3 999571820 sda3   C:\
>>>>       8     4    510976 sda4
>>>>       8    16  62522712 sdb
>>>>       8    17   4096000 sdb1   D:\
>>>>       8    18  58424664 sdb2
>>>>       8    16  62522712 sdb
>>>>       8    17   4096000 sdb1   D:\
>>>>       8    18  58424664 sdb2   E:\
>>>> [...]
>>>> So the second listing shows sdb twice. Also E: does not seem to exist
>>>> (see below).
>>>
>>> So, second report.  Which OS is that on?  I don't have any physical
>>> Windows machine anymore, so I have a bit of a problem to reproduce this.
>>>
>>> As I wrote in https://cygwin.com/pipermail/cygwin/2021-August/249012.html,
>>> I don't see how Cygwin would be able to duplicate entries, unless the OS
>>> itself duplicates entries.
>>>
>>> Somebody who can duplicate this issue would have to single step through
>>> function format_proc_partitions, see
>>>
>>> https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_proc.cc;h=66d19ab82f;hb=HEAD#l1686
>>>
>>> and check what actually happens in the above cases.
>>
>> Could those with the symptoms not provide useful information using a script
>> run in an elevated admin shell listing the relevant contents of Windows
>> objects accessed by format_proc_partitions from /proc/sys/ similar to the
>> attached script and log from my system, where the correspondence is obvious?
> 
> Not sure if that helps, but it's worth a try.
> 
> Other than that, I created a standalone application, shamelessly copying
> the code from Cygwin's function format_proc_partitions() and tweaked it
> to drop Cygwin internals.
> 
> The core of the function doesn't actually need Cygwin internals. It's
> basically a double loop calling Windows functions and just using Cygwin
> internals to print out major and minor numbers and the device name.  The
> standalone application simply fakes those.
> 
> This allows any person who *can* reproduce the problem to run the code
> under a debugger like GDB and debug what's going wrong.
> 
> I attached the code to this mail.  Store it under the name "proc_partition.c".  Compile it in two different ways, first with optimization:
> 
> $ gcc -g -O2 -o proc_partition_O2 proc_partition.c -lntdll
> 
> Run it:
> 
> $ ./proc_partition_O2
> 
> If the output contains duplicates, take a note, but otherwise
> ignore it for now.
> 
> Next, build it without optimization for easier debugging:
> 
> $ gcc -g -o proc_partition proc_partition.c -lntdll
> 
> Run it:
> 
> $ ./proc_partition
> 
> If the output contains duplicates, great!  Now you can run it under
> GDB and step through and see where it goes awry.  If it doesn't, you'll
> have to debug the optimized version, which is sometimes a bit weird.
> I'd be really glad if this allows us to reproduce the issue.
> 
> If none of them reproduces the issue... uhm... anyway, let's try first.

Might be easier for users if they could just run a verbose diagnostic, 
that checks everything and dumps everything, maybe under strace, rather 
than navigate gcc and/or gdb.

Provide it like snapshots with sha512sum and sig?

[Even sysadmins would rather not deal with development tools if 
possible, and in many orgs, for audit, internal, and security controls, 
are prohibited from doing so, and that extends to many application 
support and usage roles, due to anti-fraud, audit, and security 
controls, originally directed towards org financial reporting, under 
regulations like US SOX, "EuroSOX" audit directives like 8th Company Law 
Directive on Statutory Audit (Directive 2006/43/EC), and national 
regulatory equivalents.]

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> $ gcc -g -O2 -o proc_partition_O2 proc_partition.c -lntdll

I ran the program (without gdb yet, can't doit right away)... Its output is a bit different from /proc/partitions (w.r.t. drive names), but I still see the duplicates:

$ ./proc_partition
major minor  #blocks  name   win-mounts

    8     0 500107608 sda
    8     1    102400 sda1
    8     2 488280064 sda2    C:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1    D:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1    G:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1    I:\
    8    64 1000204632 sde
    8    65 1000202240 sde1    I:\
    8    80 234431064 sdf
    8    81 234428416 sdf1    F:\
    8    96 234431064 sdg
    8    97 234428416 sdg1    F:\
DeviceIoControl (Harddisk5\Partition0, IOCTL_DISK_GET_PARTITION_INFO{_EX}) 5    8   112         0 sdh

$ cat /proc/partitions
major minor  #blocks  name   win-mounts

    8     0 500107608 sda
    8     1    102400 sda1
    8     2 488280064 sda2   C:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1   D:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1   D:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1   G:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1   G:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    80         0 sdf

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

On Aug 17 12:57, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote:
> > $ gcc -g -O2 -o proc_partition_O2 proc_partition.c -lntdll
> 
> I ran the program (without gdb yet, can't doit right away)... Its
> output is a bit different from /proc/partitions (w.r.t. drive names),

Yes, it's just faked in the most dirt-simple way.  I added a comment to
the code to outline why that is.

> but I still see the duplicates:

Great!  That means it should be pretty straightforward to see why the
problem occurs by stepping through the code.  For a start I'd suggest
to set a breakpoint to line 103, that's the first DeviceIoControl call.
That line is only reached for actual disks when looping over the
native \Device directory.

On second thought, I have a vague idea...  Could you please just add
something to the output, i.e, change lines 124/125 from

    printf ("%5d %5d %9llu sd%c\n",
            8, (dev_name - 'a') * 16, size >> 10, dev_name);

to

    printf ("%5d %5d %9llu sd%c (%lu)\n",
            8, (dev_name - 'a') * 16, size >> 10, dev_name,
            (unsigned long) context);

and just compile and run the testcase again?


Thanks,
Corinna

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> On second thought, I have a vague idea...  Could you please just add
> something to the output, i.e, change lines 124/125 from
> 
>     printf ("%5d %5d %9llu sd%c\n",
>             8, (dev_name - 'a') * 16, size >> 10, dev_name);
> 
> to
> 
>     printf ("%5d %5d %9llu sd%c (%lu)\n",
>             8, (dev_name - 'a') * 16, size >> 10, dev_name,
>             (unsigned long) context);
> 
> and just compile and run the testcase again?

Here's the output (and /proc/partitions for comparison) from today:

$ ./proc_partition
major minor  #blocks  name   win-mounts

    8     0 500107608 sda (283)
    8     1    102400 sda1
    8     2 488280064 sda2    C:\
    8    16 1000204632 sdb (301)
    8    17 1000202240 sdb1    D:\
    8    32 1000204632 sdc (302)
    8    33 1000202240 sdc1    D:\
    8    48 1000204632 sdd (314)
    8    49 1000202240 sdd1    G:\
    8    64 1000204632 sde (315)
    8    65 1000202240 sde1    G:\
    8    80 1000204632 sdf (329)
    8    81 1000202240 sdf1    I:\
    8    96 1000204632 sdg (330)
    8    97 1000202240 sdg1    I:\
    8   112 234431064 sdh (340)
    8   113 234428416 sdh1    F:\
    8   128 234431064 sdi (341)
    8   129 234428416 sdi1    F:\
DeviceIoControl (Harddisk5\Partition0, IOCTL_DISK_GET_PARTITION_INFO{_EX}) 5    8   144         0 sdj (352)

$ cat /proc/partitions
major minor  #blocks  name   win-mounts

    8     0 500107608 sda
    8     1    102400 sda1
    8     2 488280064 sda2   C:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1   D:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1   D:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1   G:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1   G:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    80         0 sdf

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

On Aug 18 14:18, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote:
> > On second thought, I have a vague idea...  Could you please just add
> > something to the output, i.e, change lines 124/125 from
> > 
> >     printf ("%5d %5d %9llu sd%c\n",
> >             8, (dev_name - 'a') * 16, size >> 10, dev_name);
> > 
> > to
> > 
> >     printf ("%5d %5d %9llu sd%c (%lu)\n",
> >             8, (dev_name - 'a') * 16, size >> 10, dev_name,
> >             (unsigned long) context);
> > 
> > and just compile and run the testcase again?
> 
> Here's the output (and /proc/partitions for comparison) from today:
> [...]
>     8    16 1000204632 sdb (301)
                              ^^^
>     8    17 1000202240 sdb1    D:\
>     8    32 1000204632 sdc (302)
                              ^^^
>     8    33 1000202240 sdc1    D:\

>     8    48 1000204632 sdd (314)
                              ^^^
>     8    49 1000202240 sdd1    G:\
>     8    64 1000204632 sde (315)
                              ^^^
>    i 8    65 1000202240 sde1    G:\


>     8    80 1000204632 sdf (329)
                              ^^^
>     8    81 1000202240 sdf1    I:\
>     8    96 1000204632 sdg (330)
                              ^^^
>     8    97 1000202240 sdg1    I:\

>     8   112 234431064 sdh (340)
                             ^^^
>     8   113 234428416 sdh1    F:\
>     8   128 234431064 sdi (341)
                             ^^^
>     8   129 234428416 sdi1    F:\

Thanks!

These context numbers indicate that either of these harddisk entries is
actually returned twice by the NtQueryDirectoryObject call, just next to
each other.

Just for kicks, can you please change the affected printf to

      printf ("%5d %5d %9llu sd%c (%lu, %ls)\n",
              8, (dev_name - 'a') * 16, size >> 10, dev_name,
              (unsigned long) context, dbi->ObjectName.Buffer);

This also prints the device name as returned by NtQueryDirectoryObject.

So, the duplicates are actually created by the OS itself.  Why... I have
no idea so far...


Corinna

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

>       printf ("%5d %5d %9llu sd%c (%lu, %ls)\n",
>               8, (dev_name - 'a') * 16, size >> 10, dev_name,
>               (unsigned long) context, dbi->ObjectName.Buffer);

I replaced with this instead (read_bytes added):

      printf ("%5d %5d %9llu sd%c (%lu, %ls, %u)\n",
              8, (dev_name - 'a') * 16, size >> 10, dev_name,
              (unsigned long) context, dbi->ObjectName.Buffer, bytes_read);

and here's the output:

$ ./proc_partition
major minor  #blocks  name   win-mounts

    8     0 500107608 sda (282, Harddisk0, 144)
    8     1    102400 sda1
    8     2 488280064 sda2    C:\
    8    16 1000204632 sdb (299, Harddisk1, 144)
    8    17 1000202240 sdb1    D:\
    8    32 1000204632 sdc (300, Harddisk1, 144)
    8    33 1000202240 sdc1    D:\
    8    48 1000204632 sdd (313, Harddisk2, 144)
    8    49 1000202240 sdd1    G:\
    8    64 1000204632 sde (314, Harddisk2, 144)
    8    65 1000202240 sde1    G:\
    8    80 1000204632 sdf (329, Harddisk3, 144)
    8    81 1000202240 sdf1    I:\
    8    96 1000204632 sdg (330, Harddisk3, 144)
    8    97 1000202240 sdg1    I:\
    8   112 234431064 sdh (339, Harddisk4, 144)
    8   113 234428416 sdh1    F:\
    8   128 234431064 sdi (340, Harddisk4, 144)
    8   129 234428416 sdi1    F:\
DeviceIoControl (Harddisk5\Partition0, IOCTL_DISK_GET_PARTITION_INFO{_EX}) 5    8   144         0 sdj (352, Harddisk5, 624)

It prints the correct Harddisk names (corresponding to the duplicates), so it's consistent.

I was thinking that the "context" value seems to change just by 1 when there's a duplicate
coming out;  otherwise, it jumps by more than 10, when it is really a new drive...

The directory handle is used (indirectly) in NtOpenFile() through the attributes, and
I wonder if that call somehow distorts the internal position within the handle, and so
it restarts from the wrong internal position in the outer loop.

Anton Lavrentiev
Contractor NIH/NLM/NCBI

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> The directory handle is used (indirectly) in NtOpenFile() through the attributes, and
> I wonder if that call somehow distorts the internal position within the handle, and so
> it restarts from the wrong internal position in the outer loop.

And I just confirmed that if I print the context right before NtOpenFile(), and just do the "continue",
I do not see the by-one increments in it, and the number of lines printed seems to match the
actual number of drives that I have, without the duplicates:

$ ./proc_partition
context = 282
context = 299
context = 313
context = 329
context = 339
context = 352

Anton Lavrentiev
Contractor NIH/NLM/NCBI

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> And I just confirmed that if I print the context right before NtOpenFile(), and just do
> the "continue",

But then I also tried this:  I removed the "continue" before "NtOpenFile()" and allowed it to proceed,
but I moved NtClose(devhdl) right after the "printf" statement (that we were tweaking), and
inserted the "continue" there, so it does not proceed with enumerating of the partitions.
And so again, I got the output that matches my disks without the duplication.  So this actually
exonerates NtOpenFile() :-)

$ ./proc_partition
context = 282
major minor  #blocks  name   win-mounts

    8     0 500107608 sda (282, Harddisk0, 144)
context = 299
    8    16 1000204632 sdb (299, Harddisk1, 144)
context = 314
    8    32 1000204632 sdc (314, Harddisk2, 144)
context = 329
    8    48 1000204632 sdd (329, Harddisk3, 144)
context = 339
    8    64 234431064 sde (339, Harddisk4, 144)
context = 352
DeviceIoControl (Harddisk5\Partition0, IOCTL_DISK_GET_PARTITION_INFO{_EX}) 5    8    80         0 sdf (352, Harddisk5, 144)

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

On Aug 18 18:36, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote:
> > And I just confirmed that if I print the context right before NtOpenFile(), and just do
> > the "continue",
> 
> But then I also tried this:  I removed the "continue" before "NtOpenFile()" and allowed it to proceed,
> but I moved NtClose(devhdl) right after the "printf" statement (that we were tweaking), and
> inserted the "continue" there, so it does not proceed with enumerating of the partitions.
> And so again, I got the output that matches my disks without the duplication.  So this actually
> exonerates NtOpenFile() :-)
> 
> $ ./proc_partition
> context = 282
> major minor  #blocks  name   win-mounts
> 
>     8     0 500107608 sda (282, Harddisk0, 144)
> context = 299
>     8    16 1000204632 sdb (299, Harddisk1, 144)
> context = 314
>     8    32 1000204632 sdc (314, Harddisk2, 144)
> context = 329
>     8    48 1000204632 sdd (329, Harddisk3, 144)
> context = 339
>     8    64 234431064 sde (339, Harddisk4, 144)
> context = 352
> DeviceIoControl (Harddisk5\Partition0, IOCTL_DISK_GET_PARTITION_INFO{_EX}) 5    8    80         0 sdf (352, Harddisk5, 144)

Unfortunately I can't reproduce the issue.  I created a couple of
virtual drives and added them to my W10 and my W7 VM, but to no avail.
The output always makes sense, e. g.,

$ ./proc_partition.exe
major minor  #blocks  name   win-mounts

    8     0  54525952 sda (169, Harddisk0)
    8     1    102400 sda1
    8     2  54420480 sda2    C:\
    8    16   1048576 sdb (181, Harddisk1)
    8    17   1045504 sdb1    E:\
    8    32   1048576 sdc (189, Harddisk2)
    8    33     32768 sdc1
    8    34   1013760 sdc2    H:\
    8    48   1048576 sdd (199, Harddisk3)
    8    49   1045504 sdd1    F:\
    8    64   1048576 sde (207, Harddisk4)
    8    65     32768 sde1
    8    66   1013760 sde2    G:\

The only interesting difference I can observe on Windows 7 is
that the offset changes quite often, compared to Windows 10.

I wonder if the Win32 API functions GetVolumeNameForVolumeMountPointW
or GetVolumePathNamesForVolumeNameW are the potiential culprit.

Can you apply the following patch, which disables the "win-mounts" output
and try again?

--- proc_partition.c.orig	2021-08-19 12:01:25.480105116 +0200
+++ proc_partition.c	2021-08-19 12:01:47.134099288 +0200
@@ -178,7 +178,7 @@ main ()
 	    swprintf (fpath, sizeof fpath,
 		      L"\\\\?\\GLOBALROOT\\Device\\%ls\\Partition%u\\",
 		      dbi->ObjectName.Buffer, part_num);
-	    if (GetVolumeNameForVolumeMountPointW (fpath, gpath, MAX_PATH)
+	    if (0 && GetVolumeNameForVolumeMountPointW (fpath, gpath, MAX_PATH)
 		&& GetVolumePathNamesForVolumeNameW (gpath, mp_buf,
 						     NT_MAX_PATH, &len))
 	      {


Thanks,
Corinna

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

On Aug 19 12:03, Corinna Vinschen via Cygwin wrote:
> On Aug 18 18:36, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote:
> > > And I just confirmed that if I print the context right before NtOpenFile(), and just do
> > > the "continue",
> > 
> > But then I also tried this:  I removed the "continue" before "NtOpenFile()" and allowed it to proceed,
> > but I moved NtClose(devhdl) right after the "printf" statement (that we were tweaking), and
> > inserted the "continue" there, so it does not proceed with enumerating of the partitions.
> > And so again, I got the output that matches my disks without the duplication.  So this actually
> > exonerates NtOpenFile() :-)
> > [...]
> 
> Unfortunately I can't reproduce the issue.  I created a couple of
> virtual drives and added them to my W10 and my W7 VM, but to no avail.

Never mind.  I now can reproduce the problem at will.  The trick is to
create objects inside the \Device directory while the loop iterating
over the directory is running.

As shitty as it is, a NtOpenDirectoryObject/NtQueryDirectoryObject/NtClose
loop is not working atomically.  If a new object is inserted into the
dir preceeding the currently handled entry (which, on a reliable system
should *never* occur), the entry is moved by one, and the next
NtQueryDirectoryObject call returns the same object again.

Windows never disappoints when one is looking for really ugly problems.

The good news is that NtQueryDirectoryObject is also capable to take a
big buffer and return as much entries as fit in in the buffer.  I'll
give it a try now in the vain hope that this way to call
NtQueryDirectoryObject returns reliable, atomic-like results...


Corinna

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2474 bytes --]

Hi Anton,

On Aug 19 13:26, Corinna Vinschen via Cygwin wrote:
> On Aug 19 12:03, Corinna Vinschen via Cygwin wrote:
> > On Aug 18 18:36, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote:
> > > > And I just confirmed that if I print the context right before NtOpenFile(), and just do
> > > > the "continue",
> > > 
> > > But then I also tried this:  I removed the "continue" before "NtOpenFile()" and allowed it to proceed,
> > > but I moved NtClose(devhdl) right after the "printf" statement (that we were tweaking), and
> > > inserted the "continue" there, so it does not proceed with enumerating of the partitions.
> > > And so again, I got the output that matches my disks without the duplication.  So this actually
> > > exonerates NtOpenFile() :-)
> > > [...]
> > 
> > Unfortunately I can't reproduce the issue.  I created a couple of
> > virtual drives and added them to my W10 and my W7 VM, but to no avail.
> 
> Never mind.  I now can reproduce the problem at will.  The trick is to
> create objects inside the \Device directory while the loop iterating
> over the directory is running.
> 
> As shitty as it is, a NtOpenDirectoryObject/NtQueryDirectoryObject/NtClose
> loop is not working atomically.  If a new object is inserted into the
> dir preceeding the currently handled entry (which, on a reliable system
> should *never* occur), the entry is moved by one, and the next
> NtQueryDirectoryObject call returns the same object again.
> 
> Windows never disappoints when one is looking for really ugly problems.
> 
> The good news is that NtQueryDirectoryObject is also capable to take a
> big buffer and return as much entries as fit in in the buffer.  I'll
> give it a try now in the vain hope that this way to call
> NtQueryDirectoryObject returns reliable, atomic-like results...

I tested this on both of my VMs (W7, W10) and the result looks promising.

I implemented this method in the Cygwin DLL locally, for all places
utilizing the NtQueryDirectoryObject call, except for one call where
this isn't possible, unfortunately.  That's readdir on /proc/sys
directories, which is supposed to return one entry on each invocation
only and is thus stateful.

Funny enough this problem may also explain why ps(1) output sometimes
showed some Cygwin processes twice.  Same method, same problem.

Anyway, would you mind to test the below new proc_partition.c as well
as the latest snapshot I just uploaded to https://cygwin.com/snapshots/?


Thanks,
Corinna

[-- Attachment #2: proc_partition.c --]
[-- Type: text/plain, Size: 6812 bytes --]

#include <stdio.h>
#include <windows.h>
#include <winioctl.h>
#include <winternl.h>
#include <wchar.h>
#include <wctype.h>
#include <ntstatus.h>

NTSTATUS NTAPI NtOpenDirectoryObject (PHANDLE, ACCESS_MASK, POBJECT_ATTRIBUTES);
NTSTATUS NTAPI NtQueryDirectoryObject (HANDLE, PVOID, ULONG, BOOLEAN,
				       BOOLEAN, PULONG, PULONG);

typedef struct _DIRECTORY_BASIC_INFORMATION
{
  UNICODE_STRING ObjectName;
  UNICODE_STRING ObjectTypeName;
} DIRECTORY_BASIC_INFORMATION, *PDIRECTORY_BASIC_INFORMATION;

#define DIRECTORY_QUERY 1
#define NT_MAX_PATH	32768

char buf[NT_MAX_PATH];
char ioctl_buf[NT_MAX_PATH];
WCHAR mp_buf[NT_MAX_PATH];

typedef enum { false, true } bool;

int
main ()
{
  OBJECT_ATTRIBUTES attr;
  IO_STATUS_BLOCK io;
  NTSTATUS status;
  HANDLE dirhdl;
  WCHAR fpath[MAX_PATH];
  WCHAR gpath[MAX_PATH];
  DWORD len;

  /* Note that device ids and names are just faked here, not using the internal
     Cygwin code to generate device major/minor numbers and device names.
     There's *no* guarantee that the device name and the device major number
     is the same as in Cygwin. */
  char dev_name = '`';

  /* Open \Device object directory. */
  wchar_t wpath[MAX_PATH] = L"\\Device";
  UNICODE_STRING upath = {14, 16, wpath};

  InitializeObjectAttributes (&attr, &upath, OBJ_CASE_INSENSITIVE, NULL, NULL);
  status = NtOpenDirectoryObject (&dirhdl, DIRECTORY_QUERY, &attr);
  if (!NT_SUCCESS (status))
    {
      fprintf (stderr, "NtOpenDirectoryObject, status 0x%08x", status);
      return 1;
    }

  /* Traverse \Device directory ... */
  PDIRECTORY_BASIC_INFORMATION dbi_buf = (PDIRECTORY_BASIC_INFORMATION)
					 alloca (65536);
  PDIRECTORY_BASIC_INFORMATION dbi = dbi_buf;
  BOOLEAN restart = TRUE;
  bool got_one = false;
  bool last_run = false;
  ULONG context = 0;
  ULONG bytes_read = 0;
  while (!last_run)
    {
      status = NtQueryDirectoryObject (dirhdl, dbi_buf, 65536, FALSE, restart,
				       &context, &bytes_read);
      if (!NT_SUCCESS (status))
	{
	  fprintf (stderr, "NtQueryDirectoryObject(), status 0x%08x", status);
	  return 1;
	}
      if (status != STATUS_MORE_ENTRIES)
	last_run = true;
      restart = FALSE;
      printf ("bytes_read = %lu, context = %lu, status = 0x%08x\n",
	      (unsigned long) bytes_read, (unsigned long) context,
	      status);
      for (dbi = dbi_buf; dbi->ObjectName.Length > 0; dbi++)
	{
	  HANDLE devhdl;
	  PARTITION_INFORMATION_EX *pix = NULL;
	  PARTITION_INFORMATION *pi = NULL;
	  DWORD bytes_read;
	  DWORD part_cnt = 0;
	  unsigned long long size;

	  /* ... and check for a "Harddisk[0-9]*" entry. */
	  if (dbi->ObjectName.Length < 9 * sizeof (WCHAR)
	      || wcsncasecmp (dbi->ObjectName.Buffer, L"Harddisk", 8) != 0
	      || !iswdigit (dbi->ObjectName.Buffer[8]))
	    continue;
	  /* Got it.  Now construct the path to the entire disk, which is
	     "\\Device\\HarddiskX\\Partition0", and open the disk with
	     minimum permissions. */
	  //unsigned long drive_num = wcstoul (dbi->ObjectName.Buffer + 8, NULL, 10);
	  wcscpy (wpath, dbi->ObjectName.Buffer);
	  PWCHAR wpart = wpath + dbi->ObjectName.Length / sizeof (WCHAR);
	  wcpcpy (wpart, L"\\Partition0");
	  upath.Length = dbi->ObjectName.Length + 22;
	  upath.MaximumLength = upath.Length + sizeof (WCHAR);
	  InitializeObjectAttributes (&attr, &upath, OBJ_CASE_INSENSITIVE,
				      dirhdl, NULL);
	  status = NtOpenFile (&devhdl, READ_CONTROL, &attr, &io,
			       FILE_SHARE_VALID_FLAGS, 0);
	  if (!NT_SUCCESS (status))
	    {
	      fprintf (stderr, "NtOpenFile(%ls), status 0x%08x",
		       upath.Buffer, status);
	      continue;
	    }
	  if (!got_one)
	    {
	      printf ("major minor  #blocks  name   win-mounts\n\n");
	      got_one = true;
	    }
	  /* Fetch partition info for the entire disk to get its size. */
	  if (DeviceIoControl (devhdl, IOCTL_DISK_GET_PARTITION_INFO_EX, NULL, 0,
			       ioctl_buf, NT_MAX_PATH, &bytes_read, NULL))
	    {
	      pix = (PARTITION_INFORMATION_EX *) ioctl_buf;
	      size = pix->PartitionLength.QuadPart;
	    }
	  else if (DeviceIoControl (devhdl, IOCTL_DISK_GET_PARTITION_INFO, NULL, 0,
				    ioctl_buf, NT_MAX_PATH, &bytes_read, NULL))
	    {
	      pi = (PARTITION_INFORMATION *) ioctl_buf;
	      size = pi->PartitionLength.QuadPart;
	    }
	  else
	    {
	      fprintf (stderr, "DeviceIoControl (%ls, "
		       "IOCTL_DISK_GET_PARTITION_INFO{_EX}) %lu",
		       upath.Buffer, (unsigned long) GetLastError ());
	      size = 0;
	    }
	  //device dev (drive_num, 0);
	  ++dev_name;
	  printf ("%5d %5d %9llu sd%c (%lu, %ls)\n",
		  8, (dev_name - 'a') * 16, size >> 10, dev_name,
		  (unsigned long) context, dbi->ObjectName.Buffer);

	  /* Fetch drive layout info to get size of all partitions on the disk. */
	  if (DeviceIoControl (devhdl, IOCTL_DISK_GET_DRIVE_LAYOUT_EX,
			       NULL, 0, ioctl_buf, NT_MAX_PATH, &bytes_read, NULL))
	    {
	      PDRIVE_LAYOUT_INFORMATION_EX pdlix = (PDRIVE_LAYOUT_INFORMATION_EX)
						   ioctl_buf;
	      part_cnt = pdlix->PartitionCount;
	      pix = pdlix->PartitionEntry;
	    }
	  else if (DeviceIoControl (devhdl, IOCTL_DISK_GET_DRIVE_LAYOUT,
				    NULL, 0, ioctl_buf, NT_MAX_PATH, &bytes_read, NULL))
	    {
	      PDRIVE_LAYOUT_INFORMATION pdli = (PDRIVE_LAYOUT_INFORMATION) ioctl_buf;
	      part_cnt = pdli->PartitionCount;
	      pi = pdli->PartitionEntry;
	    }
	  else
	    fprintf (stderr, "DeviceIoControl(%ls, "
		      "IOCTL_DISK_GET_DRIVE_LAYOUT{_EX}): %lu",
		      upath.Buffer, (unsigned long) GetLastError ());

	  /* Loop over partitions. */
	  if (pix || pi)
	    for (DWORD i = 0; i < part_cnt && i < 64; ++i)
	      {
		DWORD part_num;

		if (pix)
		  {
		    size = pix->PartitionLength.QuadPart;
		    part_num = pix->PartitionNumber;
		    ++pix;
		  }
		else
		  {
		    size = pi->PartitionLength.QuadPart;
		    part_num = pi->PartitionNumber;
		    ++pi;
		  }
		/* A partition number of 0 denotes an extended partition or a
		   filler entry as described in fhandler_dev_floppy::lock_partition.
		   Just skip. */
		if (part_num == 0)
		  continue;

		//device dev (drive_num, part_num);
		printf ("%5d %5d %9llu sd%c%d   ",
			8, (dev_name - 'a') * 16 + part_num, size >> 10,
			dev_name, part_num);

		/* Check if the partition is mounted in Windows and, if so,
		   print the mount point list. */
		swprintf (fpath, sizeof fpath,
			  L"\\\\?\\GLOBALROOT\\Device\\%ls\\Partition%u\\",
			  dbi->ObjectName.Buffer, part_num);
		if (GetVolumeNameForVolumeMountPointW (fpath, gpath, MAX_PATH)
		    && GetVolumePathNamesForVolumeNameW (gpath, mp_buf,
							 NT_MAX_PATH, &len))
		  {
		    for (PWCHAR p = mp_buf; *p; p = wcschr (p, L'\0') + 1)
		      printf (" %ls", p);
		  }
		puts ("");
	      }
	  NtClose (devhdl);
	}
    }
  NtClose (dirhdl);

  if (!got_one)
    return 1;

  return 0;
}

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> > loop is not working atomically.  If a new object is inserted into the
> > dir preceeding the currently handled entry (which, on a reliable system
> > should *never* occur), the entry is moved by one, and the next
> > NtQueryDirectoryObject call returns the same object again.

Very interesting...  What would be inserted in the directory on my machine,
in between the calls in that original test program?  It was the only one
running there, explicitly, I mean...  And exclusion of the loop of partition
enumeration seemed to cancel the insertion?  I think the insertion is only
one exhibition of the problem, and the other one on my machine here is
different, and not well understood, but with the same side effect,
unfortunately.

> Anyway, would you mind to test the below new proc_partition.c as well

Looks promising indeed!  No duplicates:

$ ./proc_partition
bytes_read = 34346, context = 461, status = 0x00000000
major minor  #blocks  name   win-mounts

    8     0 500107608 sda (461, Harddisk0)
    8     1    102400 sda1
    8     2 488280064 sda2    C:\
    8    16 1000204632 sdb (461, Harddisk1)
    8    17 1000202240 sdb1    D:\
    8    32 1000204632 sdc (461, Harddisk2)
    8    33 1000202240 sdc1    G:\
    8    48 1000204632 sdd (461, Harddisk3)
    8    49 1000202240 sdd1    I:\
    8    64 234431064 sde (461, Harddisk4)
    8    65 234428416 sde1    F:\
DeviceIoControl (Harddisk5\Partition0, IOCTL_DISK_GET_PARTITION_INFO{_EX}) 5    8    80         0 sdf (461, Harddisk5)

> as the latest snapshot I just uploaded to https://cygwin.com/snapshots/?

I'll have to do that later, not now.  But I'll report once I have a change to try it out.

Thanks!

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

On Aug 19 15:15, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote:
> > > loop is not working atomically.  If a new object is inserted into the
> > > dir preceeding the currently handled entry (which, on a reliable system
> > > should *never* occur), the entry is moved by one, and the next
> > > NtQueryDirectoryObject call returns the same object again.
> 
> Very interesting...  What would be inserted in the directory on my machine,
> in between the calls in that original test program?  It was the only one
> running there, explicitly, I mean...

This also occurs all the time on my system.  It seems to be quite normal
in fact.  That's the only action explaining that the same entry gets
different context values when running the loop multiple times.  Observing
the changing context value was what gave me the idea to trigger this by
creating a symlink to the \Device dir at a certain iteration in the
loop, and that in turn allowed me to reproduce the problem.

> And exclusion of the loop of partition
> enumeration seemed to cancel the insertion?

Not as such.  Probably this just changed the timing.

> one exhibition of the problem, and the other one on my machine here is
> different, and not well understood, but with the same side effect,
> unfortunately.
> 
> > Anyway, would you mind to test the below new proc_partition.c as well
> 
> Looks promising indeed!  No duplicates:
> 
> $ ./proc_partition
> bytes_read = 34346, context = 461, status = 0x00000000
> major minor  #blocks  name   win-mounts
> 
>     8     0 500107608 sda (461, Harddisk0)
>     8     1    102400 sda1
>     8     2 488280064 sda2    C:\
>     8    16 1000204632 sdb (461, Harddisk1)
>     8    17 1000202240 sdb1    D:\
>     8    32 1000204632 sdc (461, Harddisk2)
>     8    33 1000202240 sdc1    G:\
>     8    48 1000204632 sdd (461, Harddisk3)
>     8    49 1000202240 sdd1    I:\
>     8    64 234431064 sde (461, Harddisk4)
>     8    65 234428416 sde1    F:\
> DeviceIoControl (Harddisk5\Partition0, IOCTL_DISK_GET_PARTITION_INFO{_EX}) 5    8    80         0 sdf (461, Harddisk5)
> 
> > as the latest snapshot I just uploaded to https://cygwin.com/snapshots/?
> 
> I'll have to do that later, not now.  But I'll report once I have a change to try it out.

Great!  Looking forward to it.


Thanks,
Corinna

RE: Duplicates in /proc/partitions

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> Great!  Looking forward to it.

I'm happy to report that in the latest snapshot, the problem seems to be gone!

$ uname -a
CYGWIN_NT-6.1 XXXXX 3.3.0s(0.341/5/3) 2021-08-19 14:45 x86_64 Cygwin

$ cat /proc/partitions
major minor  #blocks  name   win-mounts

    8     0 500107608 sda
    8     1    102400 sda1
    8     2 488280064 sda2   C:\
    8    16 1000204632 sdb
    8    17 1000202240 sdb1   D:\
    8    32 1000204632 sdc
    8    33 1000202240 sdc1   G:\
    8    48 1000204632 sdd
    8    49 1000202240 sdd1   I:\
    8    64 234431064 sde
    8    65 234428416 sde1   F:\
    8    80         0 sdf

Thanks for fixing it!

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

On Aug 19 15:46, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote:
> > Great!  Looking forward to it.
> 
> I'm happy to report that in the latest snapshot, the problem seems to be gone!
> 
> $ uname -a
> CYGWIN_NT-6.1 XXXXX 3.3.0s(0.341/5/3) 2021-08-19 14:45 x86_64 Cygwin
> 
> $ cat /proc/partitions
> major minor  #blocks  name   win-mounts
> 
>     8     0 500107608 sda
>     8     1    102400 sda1
>     8     2 488280064 sda2   C:\
>     8    16 1000204632 sdb
>     8    17 1000202240 sdb1   D:\
>     8    32 1000204632 sdc
>     8    33 1000202240 sdc1   G:\
>     8    48 1000204632 sdd
>     8    49 1000202240 sdd1   I:\
>     8    64 234431064 sde
>     8    65 234428416 sde1   F:\
>     8    80         0 sdf
> 
> Thanks for fixing it!

Thanks for testing!


Corinna

Re: Duplicates in /proc/partitions

[Corinna Vinschen]

On Aug 18 18:18, Lavrentiev, Anton (NIH/NLM/NCBI) [C] via Cygwin wrote:
> >       printf ("%5d %5d %9llu sd%c (%lu, %ls)\n",
> >               8, (dev_name - 'a') * 16, size >> 10, dev_name,
> >               (unsigned long) context, dbi->ObjectName.Buffer);
> 
> I replaced with this instead (read_bytes added):
> 
>       printf ("%5d %5d %9llu sd%c (%lu, %ls, %u)\n",
>               8, (dev_name - 'a') * 16, size >> 10, dev_name,
>               (unsigned long) context, dbi->ObjectName.Buffer, bytes_read);
> 
> and here's the output:
> 
> $ ./proc_partition
> major minor  #blocks  name   win-mounts
> 
>     8     0 500107608 sda (282, Harddisk0, 144)
>     8     1    102400 sda1
>     8     2 488280064 sda2    C:\
>     8    16 1000204632 sdb (299, Harddisk1, 144)
>     8    17 1000202240 sdb1    D:\
>     8    32 1000204632 sdc (300, Harddisk1, 144)
>     8    33 1000202240 sdc1    D:\
>     8    48 1000204632 sdd (313, Harddisk2, 144)
>     8    49 1000202240 sdd1    G:\
>     8    64 1000204632 sde (314, Harddisk2, 144)
>     8    65 1000202240 sde1    G:\
>     8    80 1000204632 sdf (329, Harddisk3, 144)
>     8    81 1000202240 sdf1    I:\
>     8    96 1000204632 sdg (330, Harddisk3, 144)
>     8    97 1000202240 sdg1    I:\
>     8   112 234431064 sdh (339, Harddisk4, 144)
>     8   113 234428416 sdh1    F:\
>     8   128 234431064 sdi (340, Harddisk4, 144)
>     8   129 234428416 sdi1    F:\
> DeviceIoControl (Harddisk5\Partition0, IOCTL_DISK_GET_PARTITION_INFO{_EX}) 5    8   144         0 sdj (352, Harddisk5, 624)
> 
> It prints the correct Harddisk names (corresponding to the
> duplicates), so it's consistent.
> 
> I was thinking that the "context" value seems to change just by 1 when
> there's a duplicate coming out;  otherwise, it jumps by more than 10,
> when it is really a new drive...

No, that just indicates the offset between entries, remember the "continue"
in line 77, skipping non-Harddisk entries.  NtQueryDirectoryObject returns
context values starting at 1, incrementing by 1.

> The directory handle is used (indirectly) in NtOpenFile() through the
> attributes, and I wonder if that call somehow distorts the internal
> position within the handle, and so it restarts from the wrong internal
> position in the outer loop.

Unlikely.  That's core functionality which *must* work.


Corinna