DD bug fails to wipe last 48 sectors of a disk

DD bug fails to wipe last 48 sectors of a disk

[Hashim Aziz]

I first came across this bug last year, and wrote up my findings in a detailed answer to the StackExchange network:

https://superuser.com/questions/1505715/why-does-dd-give-me-an-error-at-the-end-of-zero-writing-a-disk

I didn’t then have the time to report it to this mailing list, but have just tested it using the latest version of Cygwin on the same two disks as I did then, and can confirm that the problem still persists. It’s also of note that I was running Windows 7 back then and am currently running 10, so this is now confirmed to manifest on both of the last two Windows OSes. I’m happy to work with the team to get this fixed, as I frequently need to make use of Cygwin dd to zero out drives for both security and regulatory compliance reasons, and failing to wipe the last 48 sectors of all drives is a pretty significant security risk.

Thanks,
Hashim

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Hashim Aziz]

To reproduce simply run the following command on a drive (obviously, this will irreversibly wipe all data):

dd if=/dev/zero of=/dev/sdX bs=4M status=progress

Both drives were attached via internal SATA (by way of a PCIE to SATA Host Bus Adapter).

Cygwin was running in an elevated window as dd cannot run in Cygwin without administrator access, at least not on Windows 10 and not when dealing with raw disks. I was running Avast the first time I discovered this, and am currently running Windows Defender, so doubt that the AV is the cause of this.

The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK).

I just ran DD for Windows 0.6beta3 with variations of the following command:

dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M

...and can confirm that the bug also manifests here, but in a slightly different way - irrespective of the disk or block size, it fails to wipe the last 176 sectors of the drive.

Hashim

________________________________
From: Jason Pyeron <jpyeron@pdinc.us>
Sent: 19 June 2020 9:25 PM
To: 'Hashim Aziz' <hashaziz@hotmail.com>
Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk

> -----Original Message-----
> From: Hashim Aziz via Cygwin
> Sent: Friday, June 19, 2020 2:19 PM
>
> I first came across this bug last year, and wrote up my findings in a detailed answer to the
> StackExchange network:
>
> https://superuser.com/questions/1505715/why-does-dd-give-me-an-error-at-the-end-of-zero-writing-a-disk
>
> I didn’t then have the time to report it to this mailing list, but have just tested it using the
> latest version of Cygwin on the same two disks as I did then, and can confirm that the problem still
> persists. It’s also of note that I was running Windows 7 back then and am currently running 10, so
> this is now confirmed to manifest on both of the last two Windows OSes. I’m happy to work with the
> team to get this fixed, as I frequently need to make use of Cygwin dd to zero out drives for both
> security and regulatory compliance reasons, and failing to wipe the last 48 sectors of all drives is a
> pretty significant security risk.

Hmm, never had that issue.

Steps to reproduce please.

Running as Admin?
Antivirus?
How is the drive attached (USB, SATA, other)?
What are the drive models (is there a known firmware issue on windows)?

Does the non-cygwin http://www.chrysocome.net/dd work?

-Jason

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Eliot Moss]

On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote:
> To reproduce simply run the following command on a drive (obviously, this will irreversibly wipe all data):
> 
> dd if=/dev/zero of=/dev/sdX bs=4M status=progress
> 
> Both drives were attached via internal SATA (by way of a PCIE to SATA Host Bus Adapter).
> 
> Cygwin was running in an elevated window as dd cannot run in Cygwin without administrator access, at least not on Windows 10 and not when dealing with raw disks. I was running Avast the first time I discovered this, and am currently running Windows Defender, so doubt that the AV is the cause of this.
> 
> The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK).
> 
> I just ran DD for Windows 0.6beta3 with variations of the following command:
> 
> dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M
> 
> ...and can confirm that the bug also manifests here, but in a slightly different way - irrespective of the disk or block size, it fails to wipe the last 176 sectors of the drive.

I was going to ask: even with block size 512 bytes?  But I guess you checked that ...

Regards - Eliot

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Hashim Aziz]

I hadn't checked with 512 byte block sizes because of the amount of time it would have taken, but sure enough have just finished trying with bs=512 and no block size at all (so dd's default block size, which is either 512 or 1024) and although each wipe took over 24 hours, they did indeed wipe all of the sectors. So it seems that there's a bug with regards to how Cygwin handles the last block when a large (i.e. sane) block size is selected, and that this bug doesn't occur on actual UNIX-based systems.

________________________________
From: Cygwin <cygwin-bounces@cygwin.com> on behalf of Eliot Moss <moss@cs.umass.edu>
Sent: 20 June 2020 9:26 PM
To: The Cygwin Mailing List <cygwin@cygwin.com>
Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote:
> To reproduce simply run the following command on a drive (obviously, this will irreversibly wipe all data):
>
> dd if=/dev/zero of=/dev/sdX bs=4M status=progress
>
> Both drives were attached via internal SATA (by way of a PCIE to SATA Host Bus Adapter).
>
> Cygwin was running in an elevated window as dd cannot run in Cygwin without administrator access, at least not on Windows 10 and not when dealing with raw disks. I was running Avast the first time I discovered this, and am currently running Windows Defender, so doubt that the AV is the cause of this.
>
> The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK).
>
> I just ran DD for Windows 0.6beta3 with variations of the following command:
>
> dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M
>
> ...and can confirm that the bug also manifests here, but in a slightly different way - irrespective of the disk or block size, it fails to wipe the last 176 sectors of the drive.

I was going to ask: even with block size 512 bytes?  But I guess you checked that ...

Regards - Eliot
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Eliot Moss]

On 6/23/2020 11:28 AM, Hashim Aziz via Cygwin wrote:
> I hadn't checked with 512 byte block sizes because of the amount of time it would have taken, but sure enough have just finished trying with bs=512 and no block size at all (so dd's default block size, which is either 512 or 1024) and although each wipe took over 24 hours, they did indeed wipe all of the sectors. So it seems that there's a bug with regards to how Cygwin handles the last block when a large (i.e. sane) block size is selected, and that this bug doesn't occur on actual UNIX-based systems.

Well, that certainly points the finger at last-block handling ... EM

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Nicholas Clark]

Do you get the same behavior when you run dd with iflag=fullblock? Without
that flag, it's possible dd's input to read a smaller block than requested
(which would correspond with a smaller write to the output).

On Tue, Jun 23, 2020 at 9:28 AM Hashim Aziz via Cygwin <cygwin@cygwin.com>
wrote:

> I hadn't checked with 512 byte block sizes because of the amount of time
> it would have taken, but sure enough have just finished trying with bs=512
> and no block size at all (so dd's default block size, which is either 512
> or 1024) and although each wipe took over 24 hours, they did indeed wipe
> all of the sectors. So it seems that there's a bug with regards to how
> Cygwin handles the last block when a large (i.e. sane) block size is
> selected, and that this bug doesn't occur on actual UNIX-based systems.
>
> ________________________________
> From: Cygwin <cygwin-bounces@cygwin.com> on behalf of Eliot Moss <
> moss@cs.umass.edu>
> Sent: 20 June 2020 9:26 PM
> To: The Cygwin Mailing List <cygwin@cygwin.com>
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
>
> On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote:
> > To reproduce simply run the following command on a drive (obviously,
> this will irreversibly wipe all data):
> >
> > dd if=/dev/zero of=/dev/sdX bs=4M status=progress
> >
> > Both drives were attached via internal SATA (by way of a PCIE to SATA
> Host Bus Adapter).
> >
> > Cygwin was running in an elevated window as dd cannot run in Cygwin
> without administrator access, at least not on Windows 10 and not when
> dealing with raw disks. I was running Avast the first time I discovered
> this, and am currently running Windows Defender, so doubt that the AV is
> the cause of this.
> >
> > The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial:
> WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK).
> >
> > I just ran DD for Windows 0.6beta3 with variations of the following
> command:
> >
> > dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M
> >
> > ...and can confirm that the bug also manifests here, but in a slightly
> different way - irrespective of the disk or block size, it fails to wipe
> the last 176 sectors of the drive.
>
> I was going to ask: even with block size 512 bytes?  But I guess you
> checked that ...
>
> Regards - Eliot
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
>

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Hashim Aziz]

I've just tested with the command:

dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress

...and can confirm that the last 48 sectors remain unwiped in this case also.

Here is the output:

1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
dd: error writing '/dev/sda': No space left on device
238468+0 records in
238467+0 records out
1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s

________________________________
From: Cygwin <cygwin-bounces@cygwin.com> on behalf of Nicholas Clark via Cygwin <cygwin@cygwin.com>
Sent: 23 June 2020 5:06 PM
To: cygwin <cygwin@cygwin.com>
Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

Do you get the same behavior when you run dd with iflag=fullblock? Without
that flag, it's possible dd's input to read a smaller block than requested
(which would correspond with a smaller write to the output).

On Tue, Jun 23, 2020 at 9:28 AM Hashim Aziz via Cygwin <cygwin@cygwin.com>
wrote:

> I hadn't checked with 512 byte block sizes because of the amount of time
> it would have taken, but sure enough have just finished trying with bs=512
> and no block size at all (so dd's default block size, which is either 512
> or 1024) and although each wipe took over 24 hours, they did indeed wipe
> all of the sectors. So it seems that there's a bug with regards to how
> Cygwin handles the last block when a large (i.e. sane) block size is
> selected, and that this bug doesn't occur on actual UNIX-based systems.
>
> ________________________________
> From: Cygwin <cygwin-bounces@cygwin.com> on behalf of Eliot Moss <
> moss@cs.umass.edu>
> Sent: 20 June 2020 9:26 PM
> To: The Cygwin Mailing List <cygwin@cygwin.com>
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
>
> On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote:
> > To reproduce simply run the following command on a drive (obviously,
> this will irreversibly wipe all data):
> >
> > dd if=/dev/zero of=/dev/sdX bs=4M status=progress
> >
> > Both drives were attached via internal SATA (by way of a PCIE to SATA
> Host Bus Adapter).
> >
> > Cygwin was running in an elevated window as dd cannot run in Cygwin
> without administrator access, at least not on Windows 10 and not when
> dealing with raw disks. I was running Avast the first time I discovered
> this, and am currently running Windows Defender, so doubt that the AV is
> the cause of this.
> >
> > The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial:
> WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK).
> >
> > I just ran DD for Windows 0.6beta3 with variations of the following
> command:
> >
> > dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M
> >
> > ...and can confirm that the bug also manifests here, but in a slightly
> different way - irrespective of the disk or block size, it fails to wipe
> the last 176 sectors of the drive.
>
> I was going to ask: even with block size 512 bytes?  But I guess you
> checked that ...
>
> Regards - Eliot
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
>
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Andrey Repin]

Greetings, Hashim Aziz!

Bottom posting in this channel, please.

> I've just tested with the command:

> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress

oflag=direct

Although I'm unsure how Cygwin/Windows handles it. But without this flag, the
write is cached, and the problem may be outside dd, or even Cygwin.

> ...and can confirm that the last 48 sectors remain unwiped in this case also.

> Here is the output:

> 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
> dd: error writing '/dev/sda': No space left on device
> 238468+0 records in
> 238467+0 records out
> 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s


-- 
With best regards,
Andrey Repin
Wednesday, June 24, 2020 13:19:42

Sorry for my terrible english...

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Hashim Aziz]

I have no idea what bottom posting is, mailing lists are new to me and I find them a very archaic and noisy way to communicate development changes. I really wish the Cygwin dev team would enter the 21st century and move to a modern development process involving a repository like Github, I'm convinced it would increase developer engagement with the project massively while cutting down on so much of the noise.

I ran:

dd if=/dev/zero of=/dev/SDA bs=4m oflag=direct status=progress

...and can confirm the bug still presents - the last 48 sectors fail to be wiped.

________________________________
From: Andrey Repin <anrdaemon@yandex.ru>
Sent: 24 June 2020 11:21 AM
To: Hashim Aziz <hashaziz@hotmail.com>; cygwin@cygwin.com <cygwin@cygwin.com>
Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

Greetings, Hashim Aziz!

Bottom posting in this channel, please.

> I've just tested with the command:

> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress

oflag=direct

Although I'm unsure how Cygwin/Windows handles it. But without this flag, the
write is cached, and the problem may be outside dd, or even Cygwin.

> ...and can confirm that the last 48 sectors remain unwiped in this case also.

> Here is the output:

> 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
> dd: error writing '/dev/sda': No space left on device
> 238468+0 records in
> 238467+0 records out
> 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s


--
With best regards,
Andrey Repin
Wednesday, June 24, 2020 13:19:42

Sorry for my terrible english...

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Brian Inglis]

On 2020-06-25 06:08, Hashim Aziz via Cygwin wrote:
> On 24 June 2020 11:21 AM, Andrey Repin wrote:
>> Bottom posting in this channel, please.
>> 
>>> I've just tested with the command:
>> 
>>> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress
>> 
>> oflag=direct
>> 
>> Although I'm unsure how Cygwin/Windows handles it. But without this flag, the
>> write is cached, and the problem may be outside dd, or even Cygwin.
>> 
>>> ...and can confirm that the last 48 sectors remain unwiped in this case also.
>> 
>>> Here is the output:
>> 
>>> 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
>>> dd: error writing '/dev/sda': No space left on device
>>> 238468+0 records in
>>> 238467+0 records out
>>> 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s

> I have no idea what bottom posting is, mailing lists are new to me and I find
> them a very archaic and noisy way to communicate development changes. I 
> really wish the Cygwin dev team would enter the 21st century and move to a 
> modern development process involving a repository like Github, I'm convinced 
> it would increase developer engagement with the project massively while 
> cutting down on so much of the noise.
> I ran:> dd if=/dev/zero of=/dev/SDA bs=4m oflag=direct status=progress
> ...and can confirm the bug still presents - the last 48 sectors fail to be
> wiped.
Please learn to use web search and/or information sites to educate yourself on
topics new to you:

	https://en.wikipedia.org/wiki/Web_search_query
	https://en.wikipedia.org/wiki/Mailing_list
	https://en.wikipedia.org/wiki/Posting_style

and please trim others' email addresses so you do not quote them verbatim in
public posts which may be harvested by spammers.

The Cygwin project is all volunteer with zero budget and little free time, on
free shared infrastructure provided for more mainline projects.

Fixing code, updating packages, and responding to posts are the priorities, in
that order, if people have any free time to spare.

When days, weeks, or months may pass between checking, active mailing lists can
be more effectively filtered, to see what posts may be relevant for maintainers
to look at.

Other users with more experience and free time may try to help posters with
information or suggestions as first or second level type support, until
maintainers have time to look and perhaps respond.

Please also remember that presently maintainers may be dealing with other issues
and priorities e.g. health, home, family, work, and may have no free time for
weeks to come.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Christian Franke]

Andrey Repin wrote:
>
>> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress

The root of the problem is that the Windows WriteFile() function 
apparently does not support truncated writes at EOM. If seek_position + 
write_size > disk_size, then WriteFile() does nothing and returns an error.


> oflag=direct
>
> Although I'm unsure how Cygwin/Windows handles it. But without this flag, the
> write is cached, and the problem may be outside dd, or even Cygwin.

If 'oflag=direct' is used, dd passes O_DIRECT flag to open() call of 
output file. Cygwin's open() function then passes 
FILE_NO_INTERMEDIATE_BUFFERING to NtCreateFile() and the write() 
function calls WriteFile() directly with original address and size.

Without O_DIRECT, Cygwin ensures that address and size passed to 
WriteFile() are both aligned to sector size. All writes are then done 
through a 64KiB internal buffer.

As a consequence, oflag=direct in the above dd command may increase 
speed but would also let the final 4MiB WriteFile() fail. Without 
oflag=direct, only the last 64KiB WriteFile() fails.

To clear the last sectors of the disk, use an appropriate small block 
size. I did this several times with Cygwin 'dd seek=... bs=512 ...' to 
get rid of Intel RST RAID metadata.

-- 
Regards,
Christian

RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Jason Pyeron]

> -----Original Message-----
> From: Christian Franke
> Sent: Sunday, June 28, 2020 10:35 AM
> 
> Andrey Repin wrote:
> >
> >> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress
> 
> The root of the problem is that the Windows WriteFile() function
> apparently does not support truncated writes at EOM. If seek_position +
> write_size > disk_size, then WriteFile() does nothing and returns an error.
> 
> 
> > oflag=direct
> >
> > Although I'm unsure how Cygwin/Windows handles it. But without this flag, the
> > write is cached, and the problem may be outside dd, or even Cygwin.
> 
> If 'oflag=direct' is used, dd passes O_DIRECT flag to open() call of
> output file. Cygwin's open() function then passes
> FILE_NO_INTERMEDIATE_BUFFERING to NtCreateFile() and the write()
> function calls WriteFile() directly with original address and size.
> 
> Without O_DIRECT, Cygwin ensures that address and size passed to
> WriteFile() are both aligned to sector size. All writes are then done
> through a 64KiB internal buffer.
> 
> As a consequence, oflag=direct in the above dd command may increase
> speed but would also let the final 4MiB WriteFile() fail. Without
> oflag=direct, only the last 64KiB WriteFile() fails.
> 
> To clear the last sectors of the disk, use an appropriate small block
> size. I did this several times with Cygwin 'dd seek=... bs=512 ...' to
> get rid of Intel RST RAID metadata.

The reason I have never encountered this is because I use a block size which is the largest practical GCD of the drive size and 512 bytes (typically between 32 MB and 64 MB).

E.g. I have a drive that is 160,041,885,696 bytes, which divides 312,581,808 times evenly into 512. I would use a block size of ‭39,072,726‬ bytes, which gives 4,096 blocks to write.

-Jason

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Brian Inglis]

On 2020-06-28 11:50, Jason Pyeron wrote:
> On Sunday, June 28, 2020 10:35 AM, Christian Franke wrote:
>> Andrey Repin wrote:
>>>> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress
>> The root of the problem is that the Windows WriteFile() function
>> apparently does not support truncated writes at EOM. If seek_position +
>> write_size > disk_size, then WriteFile() does nothing and returns an error.
>>> oflag=direct
>>> Although I'm unsure how Cygwin/Windows handles it. But without this flag, the
>>> write is cached, and the problem may be outside dd, or even Cygwin.
>> If 'oflag=direct' is used, dd passes O_DIRECT flag to open() call of
>> output file. Cygwin's open() function then passes
>> FILE_NO_INTERMEDIATE_BUFFERING to NtCreateFile() and the write()
>> function calls WriteFile() directly with original address and size.
>> Without O_DIRECT, Cygwin ensures that address and size passed to
>> WriteFile() are both aligned to sector size. All writes are then done
>> through a 64KiB internal buffer.
>> As a consequence, oflag=direct in the above dd command may increase
>> speed but would also let the final 4MiB WriteFile() fail. Without
>> oflag=direct, only the last 64KiB WriteFile() fails.
>> To clear the last sectors of the disk, use an appropriate small block
>> size. I did this several times with Cygwin 'dd seek=... bs=512 ...' to
>> get rid of Intel RST RAID metadata.

It appears that write(2) allows:

"An error return value while performing write() using direct I/O does not mean
the entire write has failed. Partial data may be written and the data at the
file offset on which the write() was attempted should be considered inconsistent."

and write(3p) can return -1 errno EAGAIN to suggest a retry with a smaller
buffer size, in regards to pipes and FIFOs, so this could be allowed on devices:

"Deferred: ret=−1, errno=[EAGAIN]
This  error  indicates that a later request may succeed. It does not indicate
that it shall succeed, even if nbyte≤{PIPE_BUF}, because if no process reads
from the pipe or FIFO, the write never succeeds. An application could usefully
count the number of times [EAGAIN] is caused by a particular value of
nbyte>{PIPE_BUF} and perhaps do later writes with a smaller value, on the
assumption that the effective size of the pipe may have decreased.
Partial and deferred writes are only possible with O_NONBLOCK set."

"There is no exception regarding partial writes when O_NONBLOCK is set. With the
exception of writing to an empty pipe, this volume of POSIX.1‐2008 does not
specify exactly when a partial write is performed since that would require
specifying internal details of the implementation. Every application should be
prepared to handle partial writes when O_NONBLOCK is set and the requested
amount is greater than {PIPE_BUF}, just as every application should be prepared
to handle partial writes on other kinds of file descriptors."

and Cygwin write() could return -1 errno EAGAIN when a write at the end of a
device fails, and the application (dd) should be prepared to retry with a
smaller blocksize (half while even, then round up to even, as I/Os are
often/always? required to be even or binary multiple sizes) e.g. if the last
block below returned an error 39072726/2 -> 19536363 so +1 -> 19536364, or a
larger binary multiple such as 512, 64K, 1M using integer arithmetic: ((n - m +
1)/m + 1)*m, e.g in the trivial case above: (19536363 - 2 + 1)/2 + 1)*2 ->
19536364, either until a write succeeded, or the size was small enough e.g. 512,
that the result should be returned as a failure.

> The reason I have never encountered this is because I use a block size which 
> is the largest practical GCD of the drive size and 512 bytes (typically
> between 32 MB and 64 MB).
> 
> E.g. I have a drive that is 160,041,885,696 bytes, which divides 312,581,808
> times evenly into 512. I would use a block size of ‭39,072,726‬ bytes, which
> gives 4,096 blocks to write.
I am somewhat surprised that Unix and Cygwin support non-blocksize aligned
writes, but it seems you have used this successfully, correct?

A useful way to approach this uses coreutils bc and factor:

	$ bc <<< 160041885696/512	# blocks
	312581808
	$ factor 312581808		# prime factors
	312581808: 2 2 2 2 3 3 3 7 11 9397

checking and splitting off the non-binary factors, we have to pick some binary
multiple of the non-binary factor product in a reasonable range:

	$ bc <<< 9397*11*7*3^3*2^4\;9397*11*7*3^3\;2^4\;9397*11*7*3^3*2
	312581808
	19536363
	16
	39072726

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Brian Inglis]

On 2020-06-23 09:28, Hashim Aziz via Cygwin wrote:
> I hadn't checked with 512 byte block sizes because of the amount of time it would have taken, but sure enough have just finished trying with bs=512 and no block size at all (so dd's default block size, which is either 512 or 1024) and although each wipe took over 24 hours, they did indeed wipe all of the sectors. So it seems that there's a bug with regards to how Cygwin handles the last block when a large (i.e. sane) block size is selected, and that this bug doesn't occur on actual UNIX-based systems.
> 
> ________________________________
> From: Cygwin <cygwin-bounces@cygwin.com> on behalf of Eliot Moss <moss@cs.umass.edu>
> Sent: 20 June 2020 9:26 PM
> To: The Cygwin Mailing List <cygwin@cygwin.com>
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> 
> On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote:
>> To reproduce simply run the following command on a drive (obviously, this will irreversibly wipe all data):
>>
>> dd if=/dev/zero of=/dev/sdX bs=4M status=progress
>>
>> Both drives were attached via internal SATA (by way of a PCIE to SATA Host Bus Adapter).
>>
>> Cygwin was running in an elevated window as dd cannot run in Cygwin without administrator access, at least not on Windows 10 and not when dealing with raw disks. I was running Avast the first time I discovered this, and am currently running Windows Defender, so doubt that the AV is the cause of this.
>>
>> The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK).
>>
>> I just ran DD for Windows 0.6beta3 with variations of the following command:
>>
>> dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M
>>
>> ...and can confirm that the bug also manifests here, but in a slightly different way - irrespective of the disk or block size, it fails to wipe the last 176 sectors of the drive.
> 
> I was going to ask: even with block size 512 bytes?  But I guess you checked that ...

I don't have the facilities to test, and there appear to be *NO* Windows
documentation details on error condition handling, but my suspicion is that Unix
reads and writes fail only *AFTER* reading or writing at the end of the device,
but Windows reads and writes extents may be checked and failed *BEFORE* reading
or writing any data near the end of the device.
If the actual Windows error code returned is generic, Cygwin would need to
pre-check the device size as Windows does, and reduce read and write sizes to
the allowed maximum at the end of the device.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Hashim Aziz]

________________________________
From: Cygwin <cygwin-bounces@cygwin.com> on behalf of Brian Inglis <Brian.Inglis@SystematicSw.ab.ca>
Sent: 23 June 2020 8:33 PM
To: cygwin@cygwin.com <cygwin@cygwin.com>
Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

I don't have the facilities to test, and there appear to be *NO* Windows
documentation details on error condition handling, but my suspicion is that Unix
reads and writes fail only *AFTER* reading or writing at the end of the device,
but Windows reads and writes extents may be checked and failed *BEFORE* reading
or writing any data near the end of the device.
If the actual Windows error code returned is generic, Cygwin would need to
pre-check the device size as Windows does, and reduce read and write sizes to
the allowed maximum at the end of the device.

That's very helpful, thank you. Do you know if any more work has been done to attempt to fix this bug, and whether it's likely to be fixed anytime soon? It's crazy that such a commonly used command leaves so much data unwiped unbeknown to so many users, it's a very serious security hole and the sooner it can be fixed the better.

RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Jason Pyeron]

> -----Original Message-----
> From: Hashim Aziz via Cygwin
> Sent: Monday, December 28, 2020 7:46 PM
> 
>> From: Cygwin <cygwin-bounces@cygwin.com> on behalf of Brian Inglis
>> <Brian.Inglis@SystematicSw.ab.ca>
>> Sent: 23 June 2020 8:33 PM
>> To: cygwin@cygwin.com <cygwin@cygwin.com>
>> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
>> 
>> I don't have the facilities to test, and there appear to be *NO* Windows
>> documentation details on error condition handling, but my suspicion is that Unix
>> reads and writes fail only *AFTER* reading or writing at the end of the device,
>> but Windows reads and writes extents may be checked and failed *BEFORE* reading
>> or writing any data near the end of the device.
>> If the actual Windows error code returned is generic, Cygwin would need to
>> pre-check the device size as Windows does, and reduce read and write sizes to
>> the allowed maximum at the end of the device.
> 
> That's very helpful, thank you. Do you know if any more work has been done to attempt to
> fix this bug, and whether it's likely to be fixed anytime soon? It's crazy that such a
> commonly used command leaves so much data unwiped unbeknown to so many users, it's a very
> serious security hole and the sooner it can be fixed the better.

Have you tried iflag=fullblock ? This causes special handling.

> Sent: Monday, December 28, 2020 7:25 PM
> To: Jason Pyeron 
> 
> I didn't previously see this email, but the point is that this is a bug - dd should not 
> require first making calculations based on the size of each drive or using the smallest 
> possible block size (and hence taking a ridiculous amount of time) in order to do what 

Do you have any metrics that it is faster, by any meaningful amount? If so I would be very interested in mitigating it, but I suspect not the actual case.

> it's meant to do. It should always wipe the last sector of the drive regardless, just 
> as it does on other UNIX-like systems. This is why this behaviour is a bug that 
> needs to be fixed.

This does not appear to be a bug, but user error. Per the DD source "Some devices require alignment 
on a sector or page boundary"

DD has never "dealt with error handling" except when conversion were in play. When no conversions are in play it 

            {
              /* Write any partial block. */
              exit_status = EXIT_FAILURE;
              break;
            }

On windows the block devices require respecting block device boundaries, any change would be an upstream patch - not a Cygwin patch.

Sorry if this sounds rough.

Respectfully,

Jason Pyeron

Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Brian Inglis]

On 2020-12-28 19:41, Jason Pyeron wrote:
> On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote:
>> On 23 June 2020 8:33 PM, Brian Inglis wrote:
>>> I don't have the facilities to test, and there appear to be *NO* Windows 
>>> documentation details on error condition handling, but my suspicion is
>>> that Unix reads and writes fail only *AFTER* reading or writing at the
>>> end of the device, but Windows reads and writes extents may be checked
>>> and failed *BEFORE* reading or writing any data near the end of the
>>> device.
>>> If the actual Windows error code returned is generic, Cygwin would need
>>> to pre-check the device size as Windows does, and reduce read and write
>>> sizes to the allowed maximum at the end of the device.

>> That's very helpful, thank you. Do you know if any more work has been done 
>> to attempt to fix this bug, and whether it's likely to be fixed anytime
>> soon? It's crazy that such a commonly used command leaves so much data
>> unwiped unbeknown to so many users, it's a very serious security hole and
>> the sooner it can be fixed the better.

> Have you tried iflag=fullblock ? This causes special handling.

>> I didn't previously see this email, but the point is that this is a bug - 
>> dd should not require first making calculations based on the size of each
>> drive or using the smallest possible block size (and hence taking a
>> ridiculous amount of time) in order to do what

> Do you have any metrics that it is faster, by any meaningful amount? If so I 
> would be very interested in mitigating it, but I suspect not the actual
> case.

>> it's meant to do. It should always wipe the last sector of the drive
>> regardless, just as it does on other UNIX-like systems. This is why this
>> behaviour is a bug that needs to be fixed.

> This does not appear to be a bug, but user error. Per the DD source "Some
> devices require alignment on a sector or page boundary"
> DD has never "dealt with error handling" except when conversion were in play.
> When no conversions are in play it

>              {
>                /* Write any partial block. */
>                exit_status = EXIT_FAILURE;
>                break;
>              }

> On windows the block devices require respecting block device boundaries, any 
> change would be an upstream patch - not a Cygwin patch.

Your dd output appears to be ambiguous, relative to your claim that the last 48 
sectors are not written, and may appear to indicate that all sectors of the 
drive may have been written, assuming that you mean 512 byte sectors.

> 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s

1000182120448 == 238462*4*1024^2

> dd: error writing '/dev/sda': No space left on device
> 238468+0 records in

1000207286272 == 238468*4*1024^2

> 238467+0 records out

1000203091968 == 238467*4*1024^2

> 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s

1000204861440 == 238467*4*1024^2 + 27*64*1024

None of these numbers +/-48*512 bytes, which have odd factors, make a lot of 
sense as a disk size.

Could you please state explicitly, how many bytes/sectors/blocks/pages/clusters 
of what size you expect to get written, and how many 
bytes/sectors/blocks/pages/clusters of what size are actually written?

If anyone has access to a Linux system which has write access to a Windows drive 
over the network (e.g. Samba, NFS) where this can be reproduced, we can try to 
take this upstream, get their take, suggest an incremental reseek and write half 
buffer size patch, if they agree this is an issue and could be tackled in this 
manner.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Jason Pyeron]

> -----Original Message-----
> From: Brian Inglis
> Sent: Tuesday, December 29, 2020 12:55 PM
> 
> On 2020-12-28 19:41, Jason Pyeron wrote:
> > On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote:
> >> On 23 June 2020 8:33 PM, Brian Inglis wrote:
> >>> I don't have the facilities to test, and there appear to be *NO* Windows
> >>> documentation details on error condition handling, but my suspicion is
> >>> that Unix reads and writes fail only *AFTER* reading or writing at the
> >>> end of the device, but Windows reads and writes extents may be checked
> >>> and failed *BEFORE* reading or writing any data near the end of the
> >>> device.
> >>> If the actual Windows error code returned is generic, Cygwin would need
> >>> to pre-check the device size as Windows does, and reduce read and write
> >>> sizes to the allowed maximum at the end of the device.
> 
> >> That's very helpful, thank you. Do you know if any more work has been done
> >> to attempt to fix this bug, and whether it's likely to be fixed anytime
> >> soon? It's crazy that such a commonly used command leaves so much data
> >> unwiped unbeknown to so many users, it's a very serious security hole and
> >> the sooner it can be fixed the better.
> 
> > Have you tried iflag=fullblock ? This causes special handling.
> 
> >> I didn't previously see this email, but the point is that this is a bug -
> >> dd should not require first making calculations based on the size of each
> >> drive or using the smallest possible block size (and hence taking a
> >> ridiculous amount of time) in order to do what
> 
> > Do you have any metrics that it is faster, by any meaningful amount? If so I
> > would be very interested in mitigating it, but I suspect not the actual
> > case.
> 
> >> it's meant to do. It should always wipe the last sector of the drive
> >> regardless, just as it does on other UNIX-like systems. This is why this
> >> behaviour is a bug that needs to be fixed.
> 
> > This does not appear to be a bug, but user error. Per the DD source "Some
> > devices require alignment on a sector or page boundary"
> > DD has never "dealt with error handling" except when conversion were in play.
> > When no conversions are in play it
> 
> >              {
> >                /* Write any partial block. */
> >                exit_status = EXIT_FAILURE;
> >                break;
> >              }
> 
> > On windows the block devices require respecting block device boundaries, any
> > change would be an upstream patch - not a Cygwin patch.
> 
> Your dd output appears to be ambiguous, relative to your claim that the last 48
> sectors are not written, and may appear to indicate that all sectors of the
> drive may have been written, assuming that you mean 512 byte sectors.
> 
> > 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
> 
> 1000182120448 == 238462*4*1024^2
> 
> > dd: error writing '/dev/sda': No space left on device
> > 238468+0 records in
> 
> 1000207286272 == 238468*4*1024^2
> 
> > 238467+0 records out
> 
> 1000203091968 == 238467*4*1024^2
> 
> > 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s
> 
> 1000204861440 == 238467*4*1024^2 + 27*64*1024
> 
> None of these numbers +/-48*512 bytes, which have odd factors, make a lot of
> sense as a disk size.
> 
> Could you please state explicitly, how many bytes/sectors/blocks/pages/clusters
> of what size you expect to get written, and how many
> bytes/sectors/blocks/pages/clusters of what size are actually written?
> 
> If anyone has access to a Linux system which has write access to a Windows drive
> over the network (e.g. Samba, NFS) where this can be reproduced, we can try to
> take this upstream, get their take, suggest an incremental reseek and write half
> buffer size patch, if they agree this is an issue and could be tackled in this
> manner.

I do, and I am even willing to spend a few dollars in testing this too.

BUT, I want a clear test plan first, with clearly articulated issues BECAUSE I do not believe there are any issues actually existing.

The best I can glean from the thread is 

1. Cygwin is agedly breaking dd, but is also broken in the windows native dd [1]
2. Using "correct" (as I have previously defined it e.g. [4]) values is both
2.1. too slow on the write [2]
2.2. inappropriately complicated of a process for the user
3. It has been well investigated 
3.1. with a root cause was identified as a windows OS issue [3,5,6]
3.2. with a mitigation [4,5]

I am happy to spend time and money on 2.1.
I will not spend my time on dealing with 2.2 - except providing an example and documentation update for upstream.

Respectfully,

Jason Pyeron

1: > From: Hashim Aziz
> Sent: Saturday, June 20, 2020 1:31 PM
> To: The Cygwin Mailing List
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <DB7PR01MB5193CC1C7630FB13B81B9DBAD5980@DB7PR01MB5193.eurprd01.prod.exchangelabs.com>

2: > From: Hashim Aziz
> Sent: Tuesday, June 23, 2020 11:29 AM
> To: cygwin 
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <DB7PR01MB519396195ADB3E3E1217B22BD5940@DB7PR01MB5193.eurprd01.prod.exchangelabs.com>

3: > From: Christian Franke
> Sent: Sunday, June 28, 2020 10:35 AM
> To: cygwin@cygwin.com
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <5c9c5b44-c8a6-a075-705e-1761533cc966@t-online.de>

4: > From: Jason Pyeron
> Sent: Sunday, June 28, 2020 1:50 PM
> To: cygwin@cygwin.com
> Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <00f601d64d74$959826d0$c0c87470$@pdinc.us>

5: > From: Brian Inglis
> Sent: Sunday, June 28, 2020 4:28 PM
> To: cygwin@cygwin.com
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <9614fca7-4006-06f2-9956-b30dd14cc49e@SystematicSw.ab.ca>

6: > From: Jason Pyeron
> Sent: Monday, December 28, 2020 9:42 PM
> To: cygwin@cygwin.com
> Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <121001d6dd8c$1dc8b0e0$595a12a0$@pdinc.us>

RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk

[Jason Pyeron]

> From: Hashim Aziz
> Sent: Monday, January 4, 2021 3:16 PM
>
> > From: Jason Pyeron 
> > Sent: 30 December 2020 1:35 AM
> > To: mailto:cygwin@cygwin.com <mailto:cygwin@cygwin.com>
> > Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> >
> > > -----Original Message-----
> > > From: Brian Inglis
> > > Sent: Tuesday, December 29, 2020 12:55 PM
> > >
> > > On 2020-12-28 19:41, Jason Pyeron wrote:
> > > > On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote:
> > > >> On 23 June 2020 8:33 PM, Brian Inglis wrote:
> > > >>> I don't have the facilities to test, and there appear to be *NO* Windows
> > > >>> documentation details on error condition handling, but my suspicion is
> > > >>> that Unix reads and writes fail only *AFTER* reading or writing at the
> > > >>> end of the device, but Windows reads and writes extents may be checked
> > > >>> and failed *BEFORE* reading or writing any data near the end of the
> > > >>> device.
> > > >>> If the actual Windows error code returned is generic, Cygwin would need
> > > >>> to pre-check the device size as Windows does, and reduce read and write
> > > >>> sizes to the allowed maximum at the end of the device.
> > >
> > > >> That's very helpful, thank you. Do you know if any more work has been done
> > > >> to attempt to fix this bug, and whether it's likely to be fixed anytime
> > > >> soon? It's crazy that such a commonly used command leaves so much data
> > > >> unwiped unbeknown to so many users, it's a very serious security hole and
> > > >> the sooner it can be fixed the better.
> > >
> > > > Have you tried iflag=fullblock ? This causes special handling.
> > >
> > > >> I didn't previously see this email, but the point is that this is a bug -
> > > >> dd should not require first making calculations based on the size of each
> > > >> drive or using the smallest possible block size (and hence taking a
> > > >> ridiculous amount of time) in order to do what
> > >
> > > > Do you have any metrics that it is faster, by any meaningful amount? If so I
> > > > would be very interested in mitigating it, but I suspect not the actual
> > > > case.
>
> That dd is faster with a block size of something large like 4M vs the default of 512 or 1024 has 
> long been attested to by anyone who has used these tools on large drives - here's just 
> one https://superuser.com/a/234241/323079 from SuperUser.

But I just copied 1TB in 2 hours on Cygwin dd using a GCD calculated block size. No one is arguing that 512 is slow, the issue seems to be you do not like using a whole multiple of the actual drive size, and are insisting on using a non-divisible block size.

> >
> > > Could you please state explicitly, how many bytes/sectors/blocks/pages/clusters
> > > of what size you expect to get written, and how many
> > > bytes/sectors/blocks/pages/clusters of what size are actually written?
> >
> > As mentioned in the original email and in https://superuser.com/a/1509903/323079, I have confirmed that everything but the last 48 sectors of the drive are wiped on both a 1TB HDD and a 128GB SSD, by inspecting the disks themselves with a hex editor (opening the raw disks in HxD). As someone else said earlier, on both occasions this points to the very last block failing to write properly.
> > BUT, I want a clear test plan first, with clearly articulated issues BECAUSE I do not believe there are any issues actually existing.
> >
> > The best I can glean from the thread is
> >
> > 1. Cygwin is agedly breaking dd, but is also broken in the windows native dd [1]
> > 2. Using "correct" (as I have previously defined it e.g. [4]) values is both
> > 2.1. too slow on the write [2]
> > 2.2. inappropriately complicated of a process for the user
> > 3. It has been well investigated
> > 3.1. with a root cause was identified as a windows OS issue [3,5,6]
> > 3.2. with a mitigation [4,5]
> >
> > I am happy to spend time and money on 2.1.
> > I will not spend my time on dealing with 2.2 - except providing an example and documentation update for upstream.
>
> Isn't the entire purpose of Cygwin to have tools that behave as their upstream equivalents do? 

Yes(ish), but the upstream has this issue to. As their documentation indicates - some OSes are not forgiving.

> Upstream dd has no problems running the same command without these issues,

You indicated that it (windows native dd) has the same issue, whereas on Linux (a different OS) behaves differently.

> and I'm sure upstream would agree on the serious security hole that leaving behind the last 48 sectors of every disk is.

No this appears to be a user error - I do not have the issue, when I execute it correctly.

>
> The only logical conclusion to me here - and it seems bizarre to me that there is any pushback against it at all - is to make Cygwin dd behave like upstream dd does (however that is done and whosoever's patch it is) so that Cygwin dd behaves as expected, and doesn't go around leaving disks with data on them unbeknownst to users. 

There is no patch needed to make dd behave the same, because it is behaving the same.

> Most users of upstream dd are not calculating a "correct" blocksize beforehand because upstream dd doesn't require them to,

RTFM - it says that some OS will require that to be correct.

> and neither should Cygwin dd. Securely wiping data should be as accessible to novice 

Security should not be done by novices with tools not designed for novices - dd is not a disk wiping tool, it is a block device copy tool.

> command lines users as possible, without such barriers as calculating disk sizes in the way, and this is something that upstream is obviously mindful of, so why isn't Cygwin?

Feel free to provide us (me) with the upstream's bug number, and I will track it and participate in testing.

Re: DD bug fails to wipe last 48 sectors of a disk

[Hashim Aziz]

________________________________
From: Hashim Aziz <hashaziz@hotmail.com>
Sent: 19 June 2020 7:19 PM
To: cygwin <cygwin@cygwin.com>
Subject: DD bug fails to wipe last 48 sectors of a disk


I first came across this bug last year, and wrote up my findings in a detailed answer to the StackExchange network:



https://superuser.com/questions/1505715/why-does-dd-give-me-an-error-at-the-end-of-zero-writing-a-disk



I didn’t then have the time to report it to this mailing list, but have just tested it using the latest version of Cygwin on the same two disks as I did then, and can confirm that the problem still persists. It’s also of note that I was running Windows 7 back then and am currently running 10, so this is now confirmed to manifest on both of the last two Windows OSes. I’m happy to work with the team to get this fixed, as I frequently need to make use of Cygwin dd to zero out drives for both security and regulatory compliance reasons, and failing to wipe the last 48 sectors of all drives is a pretty significant security risk.



Thanks,

Hashim



---


Apologies for the late reply, I've just finished going through the responses. While the code solutions given are useful and very interesting, they seem to be largely hacks and workarounds. As someone who wipes many drives for security purposes, from a security point of view it's a huge problem for dd to be silently and consistently failing to wipe the last sector of a drive, and 64KB is still plenty of space to contain sensitive documents. Is it planned for this to be fixed at some point?

Re: DD bug fails to wipe last 48 sectors of a disk

[Hamish McIntyre-Bhatty]

[-- Attachment #1.1.1: Type: text/plain, Size: 169 bytes --]

Was this ever resolved? I could have sworn I saw some discussion about
this but I can't find it in the archives.

It might also affect ddrescue, I feel.

Hamish


[-- Attachment #1.1.2: 0x87B761FE07F548D6.asc --]
[-- Type: application/pgp-keys, Size: 3235 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: DD bug fails to wipe last 48 sectors of a disk

[Erik Soderquist]

On Mon, Sep 14, 2020 at 11:45 AM Hamish McIntyre-Bhatty via Cygwin wrote:
>
> Was this ever resolved? I could have sworn I saw some discussion about
> this but I can't find it in the archives.

I still have the thread in my local email; a couple viable work
arounds were provided, but the issue's root cause is on the Windows
side, so I believe the only real route to a true fix would be via
Microsoft altering their code.

> It might also affect ddrescue, I feel.

I believe you are accurate in this feeling.

-- Erik

(op bcc'ed)

--
"I do not think any of us are truly sane, Caleb. Not even you. Courage
is not sanity. Being willing to die for someone else is not sanity."
... "Love is not sane, nor is faith." ... "If sanity lacks those
things, Caleb, I want no part of it."

-- Alexandria Terri in "Weaving the Wyvern" by Alexis Desiree Thorne