Is a disk image created within dd "safe"?

Is a disk image created within dd "safe"?

[Clint Olsen]

Hi:

I currently use Cygwin and rsync to provide Time Machine-like
functionality to a NAS on my home network. This works fine for user
data.

I'm also interested in generating images to preserve the entire disk.
What I'm wondering is whether doing this on a live system through
Cygwin would produce a safe, bootable disk image or if the the APIs
that Cygwin has to use or having the disk mounted would make this
unreliable?

A friend speculated that dd might complain about open files or
some-such thing. I went and tried it on /dev/sda and it seemed to work
without complaining.

Example:

$ dd if=/dev/sda of=/path/to/NAS/sda.img bs=512 conv=noerror,sync

It would be great if something like this would work since dd is so
ubiquitous that I could restore a drive in a number of ways and not
lock me into something proprietary.

Thanks,

-Clint

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Is a disk image created within dd "safe"?

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2252 bytes --]

On 09/02/2015 03:57 PM, Clint Olsen wrote:

> I'm also interested in generating images to preserve the entire disk.
> What I'm wondering is whether doing this on a live system through
> Cygwin would produce a safe, bootable disk image or if the the APIs
> that Cygwin has to use or having the disk mounted would make this
> unreliable?

The answer is the same on non-cygwin systems.  ANY attempt to
sequentially read a disk while some other entity may be actively
modifying the same disk is doomed to capture incomplete (and probably
unusable) state.

> 
> A friend speculated that dd might complain about open files or
> some-such thing. I went and tried it on /dev/sda and it seemed to work
> without complaining.

dd won't complain, but it also won't capture an accurate image.  There
is a difference between the block layer (read each sector of the disk,
regardless of what is happening in other sectors due to file system
activity) and the file system layer (read multiple files one at a time,
regardless of which sectors the file system currently maps those files
to).  The only safe way to use dd to clone a disk is if the disk is not
currently mounted by any filesystem that might be actively modifying
sectors in the block device.

There are various solutions that CAN capture an accurate point-in-time
disk snapshot.  For example, on Linux, you can use LVM and create an LVM
snapshot, which separates all data at a given point in time from all
further changes to the live disk, so that you can then do a background
task that reads the snapshot without worrying about data going
inconsistent, and so that the guest can continue to run with the live
overlay.  Many SAN storage vendors sell specific solutions along these
lines, and it is also a hot topic with virtual machine solutions.  But
in every case, these sorts of solutions require additional storage
hierarchies (they are not copying /dev/sda directly, so much as exposing
/dev/sda as a layer on top of other storage such that the actual
snapshot operation utilizes that other storage for the background
copy-on-write overlay magic).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

RE: [cygwin] Re: Is a disk image created within dd "safe"?

[Jason Pyeron]

> -----Original Message-----
> From: Eric Blake
> Sent: Wednesday, September 02, 2015 6:06 PM
> 
> On 09/02/2015 03:57 PM, Clint Olsen wrote:
> 
> > I'm also interested in generating images to preserve the 
> entire disk.
> > What I'm wondering is whether doing this on a live system through 
> > Cygwin would produce a safe, bootable disk image or if the the APIs 
> > that Cygwin has to use or having the disk mounted would make this 
> > unreliable?
<snip> don't do it with out snapshots... </snip>
> There are various solutions that CAN capture an accurate 
> point-in-time disk snapshot.  

For cygwin/Windows use the Volume Shadow Service (VSS). I have used (XP)
cygwin dd and rsync to take (clone) images from the VSS managed drive.

-Jason

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [cygwin] Re: Is a disk image created within dd "safe"?

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

On 09/02/2015 04:23 PM, Jason Pyeron wrote:

>> There are various solutions that CAN capture an accurate 
>> point-in-time disk snapshot.  
> 
> For cygwin/Windows use the Volume Shadow Service (VSS). I have used (XP)
> cygwin dd and rsync to take (clone) images from the VSS managed drive.

What's more, if you are running your Windows machine as a virtual
machine on top of qemu/kvm, you can install qemu-guest-agent into your
Windows setup, and set things up so that when the hypervisor wants to
take a snapshot of your guest, the guest-agent can coordinate with VSS
to make the point in time correspond to completely stable I/O (all
databases flushed and so forth, so the captured disk state is stable as
if you had gracefully powered off) rather than just a random state of
unflushed I/O (the disk state is as if you have yanked the power cord,
and may have missing transactions; hope your file system had decent
journaling if you wanted it to be consistent).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: Is a disk image created within dd "safe"?

[Andrey Repin]

Greetings, Clint Olsen!

> I'm also interested in generating images to preserve the entire disk.
> What I'm wondering is whether doing this on a live system through
> Cygwin would produce a safe, bootable disk image or if the the APIs
> that Cygwin has to use or having the disk mounted would make this
> unreliable?

Bootable? Likely.
Safe? not even in the slightest.
And as Eric pointed out, this has nothing to do with Cygwin.

> A friend speculated that dd might complain about open files or
> some-such thing.

You don't have files on a block device level. >.< Tell your friend to
speculate something better.

> I went and tried it on /dev/sda and it seemed to work
> without complaining.

With conv=noerror ?
I'm wondering, what did you expect?

> Example:

> $ dd if=/dev/sda of=/path/to/NAS/sda.img bs=512 conv=noerror,sync

bs=4k at least.
NTFS uses 4k clusters by default, there's no reason to use smaller chunks to
read it.
Actually, even bigger chunks may greatly speed up the read.

conv is just bogus. You are trying to create a mess of the image with these
options.
You don't want any padding.
You don't want errors ignored.

Writing disk image to NAS uncompressed is even more questionable.
Consider piping it through some archiver.

> It would be great if something like this would work since dd is so
> ubiquitous that I could restore a drive in a number of ways and not
> lock me into something proprietary.

It wouldn't work. Not directly. For a number of reasons.
If you still intend to do it, at the very least consider using ERUNT for
registry snapshots. That will let you have a consistent registry copy even if
the one in image would turn up damaged.


-- 
With best regards,
Andrey Repin
Thursday, September 3, 2015 04:50:29

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: [cygwin] Re: Is a disk image created within dd "safe"?

[Lemke, Michael ST/HZA-ZIC2]

On Thursday, September 03, 2015 12:23 AM Jason Pyeron wrote:
>
>> -----Original Message-----
>> From: Eric Blake
>> Sent: Wednesday, September 02, 2015 6:06 PM
>> 
>> On 09/02/2015 03:57 PM, Clint Olsen wrote:
>> 
>> > I'm also interested in generating images to preserve the 
>> entire disk.
>> > What I'm wondering is whether doing this on a live system through 
>> > Cygwin would produce a safe, bootable disk image or if the the APIs 
>> > that Cygwin has to use or having the disk mounted would make this 
>> > unreliable?
><snip> don't do it with out snapshots... </snip>
>> There are various solutions that CAN capture an accurate 
>> point-in-time disk snapshot.  
>
>For cygwin/Windows use the Volume Shadow Service (VSS). I have used (XP)
>cygwin dd and rsync to take (clone) images from the VSS managed drive.
>

Could you elaborate a little? How does this VSS thing work? I am always
confused with Windows disk management when it isn't a single disk. Once
I can use dd and rsync I am fine.

Thanks,
Michael

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple