From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail1.pdinc.us (mail.pdinc.us [67.90.184.27]) by sourceware.org (Postfix) with ESMTPS id 254D83857C5F for ; Wed, 30 Dec 2020 01:35:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 254D83857C5F Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=pdinc.us Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=jpyeron@pdinc.us Received: from blackfat (nsa1.pdinc.us [67.90.184.2]) (authenticated bits=0) by mail1.pdinc.us (8.14.4/8.14.4) with ESMTP id 0BU1Z6qm014167 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 29 Dec 2020 20:35:06 -0500 DKIM-Filter: OpenDKIM Filter v2.11.0 mail1.pdinc.us 0BU1Z6qm014167 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pdinc.us; s=default; t=1609292106; bh=m2Avb9m5PK1kuPf6lmHbHbdT99UAFahLtJDFQhUwgto=; h=From:To:References:In-Reply-To:Subject:Date:From; b=Z0z+CW1QLnWJIBnKH0TVgrQ4lMidIxsZsFXdOWxB8GjPsK1u09ECjJbB1oeIK1Aat dQsUymsXYppMx0Ez7ZZuu1++6PiKMTOQjPDUVMiJDHow4ZZ0HomlkCojF7scieL3My 0CSOL6jP6GfZI0ZWQncwSkAMundEfB+bC1b7JEDHpasE99cEVbZwk1hHi1BdSiKZWE TBJKQWBOWASNrq75E7ABn3/vYYBcHAW+toW749CtIPb9FcIZGWFFP69Qwc0b3Gmod0 oYwLO+CioaLJsljH5lZTNicHNvnplPp5+n7v1JA5qd5KxHPOVaaNYlFhbvZpA56r9k XXiYaHqHuOXaQ== From: "Jason Pyeron" To: References: <1d1801d64677$bea56050$3bf020f0$@pdinc.us> <60bf1507-4edb-a03f-ec14-07e1ab7f0d94@cs.umass.edu> <1b13fde4-0834-cd8b-0673-c2b14bbaa372@SystematicSw.ab.ca> <121001d6dd8c$1dc8b0e0$595a12a0$@pdinc.us> <8cc9cad7-a5e2-3329-5bfc-14b5650489d9@SystematicSw.ab.ca> In-Reply-To: <8cc9cad7-a5e2-3329-5bfc-14b5650489d9@SystematicSw.ab.ca> Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk Date: Tue, 29 Dec 2020 20:35:16 -0500 Organization: PD Inc Message-ID: <08f401d6de4c$073fa2a0$15bee7e0$@pdinc.us> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQGscau3vmYRj+s+e2PTQd4MF23JZQNbIR/1ATPrTR8BvLqaegH1Mf0MAb1HszEA12M7fAF/W2ZqAkNU+mqp7utLEA== Content-Language: en-us X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, KAM_INFOUSMEBIZ, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2020 01:35:08 -0000 > -----Original Message----- > From: Brian Inglis > Sent: Tuesday, December 29, 2020 12:55 PM > > On 2020-12-28 19:41, Jason Pyeron wrote: > > On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote: > >> On 23 June 2020 8:33 PM, Brian Inglis wrote: > >>> I don't have the facilities to test, and there appear to be *NO* Windows > >>> documentation details on error condition handling, but my suspicion is > >>> that Unix reads and writes fail only *AFTER* reading or writing at the > >>> end of the device, but Windows reads and writes extents may be checked > >>> and failed *BEFORE* reading or writing any data near the end of the > >>> device. > >>> If the actual Windows error code returned is generic, Cygwin would need > >>> to pre-check the device size as Windows does, and reduce read and write > >>> sizes to the allowed maximum at the end of the device. > > >> That's very helpful, thank you. Do you know if any more work has been done > >> to attempt to fix this bug, and whether it's likely to be fixed anytime > >> soon? It's crazy that such a commonly used command leaves so much data > >> unwiped unbeknown to so many users, it's a very serious security hole and > >> the sooner it can be fixed the better. > > > Have you tried iflag=fullblock ? This causes special handling. > > >> I didn't previously see this email, but the point is that this is a bug - > >> dd should not require first making calculations based on the size of each > >> drive or using the smallest possible block size (and hence taking a > >> ridiculous amount of time) in order to do what > > > Do you have any metrics that it is faster, by any meaningful amount? If so I > > would be very interested in mitigating it, but I suspect not the actual > > case. > > >> it's meant to do. It should always wipe the last sector of the drive > >> regardless, just as it does on other UNIX-like systems. This is why this > >> behaviour is a bug that needs to be fixed. > > > This does not appear to be a bug, but user error. Per the DD source "Some > > devices require alignment on a sector or page boundary" > > DD has never "dealt with error handling" except when conversion were in play. > > When no conversions are in play it > > > { > > /* Write any partial block. */ > > exit_status = EXIT_FAILURE; > > break; > > } > > > On windows the block devices require respecting block device boundaries, any > > change would be an upstream patch - not a Cygwin patch. > > Your dd output appears to be ambiguous, relative to your claim that the last 48 > sectors are not written, and may appear to indicate that all sectors of the > drive may have been written, assuming that you mean 512 byte sectors. > > > 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s > > 1000182120448 == 238462*4*1024^2 > > > dd: error writing '/dev/sda': No space left on device > > 238468+0 records in > > 1000207286272 == 238468*4*1024^2 > > > 238467+0 records out > > 1000203091968 == 238467*4*1024^2 > > > 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s > > 1000204861440 == 238467*4*1024^2 + 27*64*1024 > > None of these numbers +/-48*512 bytes, which have odd factors, make a lot of > sense as a disk size. > > Could you please state explicitly, how many bytes/sectors/blocks/pages/clusters > of what size you expect to get written, and how many > bytes/sectors/blocks/pages/clusters of what size are actually written? > > If anyone has access to a Linux system which has write access to a Windows drive > over the network (e.g. Samba, NFS) where this can be reproduced, we can try to > take this upstream, get their take, suggest an incremental reseek and write half > buffer size patch, if they agree this is an issue and could be tackled in this > manner. I do, and I am even willing to spend a few dollars in testing this too. BUT, I want a clear test plan first, with clearly articulated issues BECAUSE I do not believe there are any issues actually existing. The best I can glean from the thread is 1. Cygwin is agedly breaking dd, but is also broken in the windows native dd [1] 2. Using "correct" (as I have previously defined it e.g. [4]) values is both 2.1. too slow on the write [2] 2.2. inappropriately complicated of a process for the user 3. It has been well investigated 3.1. with a root cause was identified as a windows OS issue [3,5,6] 3.2. with a mitigation [4,5] I am happy to spend time and money on 2.1. I will not spend my time on dealing with 2.2 - except providing an example and documentation update for upstream. Respectfully, Jason Pyeron 1: > From: Hashim Aziz > Sent: Saturday, June 20, 2020 1:31 PM > To: The Cygwin Mailing List > Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk > Message-ID: 2: > From: Hashim Aziz > Sent: Tuesday, June 23, 2020 11:29 AM > To: cygwin > Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk > Message-ID: 3: > From: Christian Franke > Sent: Sunday, June 28, 2020 10:35 AM > To: cygwin@cygwin.com > Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk > Message-ID: <5c9c5b44-c8a6-a075-705e-1761533cc966@t-online.de> 4: > From: Jason Pyeron > Sent: Sunday, June 28, 2020 1:50 PM > To: cygwin@cygwin.com > Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk > Message-ID: <00f601d64d74$959826d0$c0c87470$@pdinc.us> 5: > From: Brian Inglis > Sent: Sunday, June 28, 2020 4:28 PM > To: cygwin@cygwin.com > Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk > Message-ID: <9614fca7-4006-06f2-9956-b30dd14cc49e@SystematicSw.ab.ca> 6: > From: Jason Pyeron > Sent: Monday, December 28, 2020 9:42 PM > To: cygwin@cygwin.com > Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk > Message-ID: <121001d6dd8c$1dc8b0e0$595a12a0$@pdinc.us>