file accessibility and copying/archive/...

file accessibility and copying/archive/...

[Hugh Sasse]

Trying to copy a windows XP NTFS drive to a big disk using cygwin tools
I encounter inaccessible files such as ntusers.dat.  tar is not
particularly verbose about why things fail so I wrote something in
Ruby, but I only got about 70% of the contents of the disk across.
I suspect this is a common problem, but don't know how to frame it
correctly to extract something useful from Google.  The setup in
question only has the one PC so things like rsync are out.  Windows
keeps some things in use, but I don't know what.  Can anyone point
me at the specific thing I should be reading, or suggest anything?
If you mail me off list then, if there is interest I will summarize
to the list.  Thank you.

        Hugh

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Sylvain RICHARD]

Hugh Sasse wrote:
> Trying to copy a windows XP NTFS drive to a big disk using cygwin tools
> I encounter inaccessible files such as ntusers.dat.  tar is not
> particularly verbose about why things fail so I wrote something in
> Ruby, but I only got about 70% of the contents of the disk across.
> I suspect this is a common problem, but don't know how to frame it
> correctly to extract something useful from Google.  The setup in
> question only has the one PC so things like rsync are out.  Windows
> keeps some things in use, but I don't know what.  Can anyone point
> me at the specific thing I should be reading, or suggest anything?
Hugh,

If you must do an online copy, the only solution is the "Volume Shadow 
Copy" API, especially designed to allow copying files in use. But you 
can copy garbage (especially for databases).

An offline solution based upon a bootable linux distribution (knoopix or 
others) might be better suited to your needs. I suggest dd_rescue if you 
want to copy whole partitions.

A commercial offering would be Ghost.

Best regards

    Sylvain RICHARD

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Brian Dessent]

Hugh Sasse wrote:

> Trying to copy a windows XP NTFS drive to a big disk using cygwin tools
> I encounter inaccessible files such as ntusers.dat.  tar is not

ntuser.dat is the filename of the per-user registry hive
(HKEY_CURRENT_USER).  It is opened by the system in exclusive mode, so
it cannot be opened.

> I suspect this is a common problem, but don't know how to frame it
> correctly to extract something useful from Google.  The setup in
> question only has the one PC so things like rsync are out.  Windows

Note that rsync works just fine as a local copying tool like xcopy. 
There's nothing that says you have to use it with a remote machine.  But
it will be of no use to you in this case.

> keeps some things in use, but I don't know what.  Can anyone point
> me at the specific thing I should be reading, or suggest anything?
> If you mail me off list then, if there is interest I will summarize
> to the list.  Thank you.

You can't open the file because it's been opened with a sharing mode
that disallows any other process to open it.  The third parameter of
CreateFile() is dwShareMode and can one or more of FILE_SHARE_DELETE,
FILE_SHARE_READ, FILE_SHARE_WRITE.  If it is 0 then no sharing is
allowed, and the file cannot be opened by any other process until the
handle is closed.

This is why backup programs are more than just file copying utilities. 
There are a number of files that you can't just copy from a live
system.  Even if you could forcibly open the file, if you were to
naïvely copy it you might get inconsistent state, since the whole point
for opening a file exclusively is so that you can have full control over
it and implement your own form of transactions.

Starting with XP, Microsoft introduced the Volume Shadow Copy service,
as the other reply in this thread indicated.  This is a vastly
complicated[1] COM interface that can be used by backup programs to get
consistent state for all files on a live system.  Certainly no Cygwin
tool or app uses this API.

What I do is simply use the built in Windows backup program (ntbackup)
to create a backup to a file.  Ntbackup uses all the proper backup APIs
and can deal with all in-use files correctly, and the resulting output
is just a plain file that can be copied around with standard tools, or
split/burned to DVDR, or whatever.

Brian

[1] http://msdn2.microsoft.com/en-us/library/aa384589.aspx

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Hugh Sasse]

On Tue, 15 Apr 2008, Sylvain RICHARD wrote:

> Hugh Sasse wrote:
> > Trying to copy a windows XP NTFS drive to a big disk using cygwin tools
> > I encounter inaccessible files such as ntusers.dat.  tar is not
> > particularly verbose about why things fail so I wrote something in
> > Ruby, but I only got about 70% of the contents of the disk across.
> > I suspect this is a common problem, but don't know how to frame it
> > correctly to extract something useful from Google.  The setup in
> > question only has the one PC so things like rsync are out.  Windows
> > keeps some things in use, but I don't know what.  Can anyone point
> > me at the specific thing I should be reading, or suggest anything?
> Hugh,
> 
> If you must do an online copy, the only solution is the "Volume Shadow Copy"

OK, I'll google that.

> API, especially designed to allow copying files in use. But you can copy
> garbage (especially for databases).

They end up as impenetrable blobs of binary, or they actually get corrupted?
> 
> An offline solution based upon a bootable linux distribution (knoopix or

Right, I have a knoppix CD (I note that the new ones are DVD now, but don't
know how much NTFS support has improved since 5.1. But that's going OT for
this list)

> others) might be better suited to your needs. I suggest dd_rescue if you want
> to copy whole partitions.

I want to extract the non-system files at some point.
> 
> A commercial offering would be Ghost.

Ha!  I had version 2 of ghost for backups on that machine. I
"upgraded" to 2005 (I forget the exact version) and it no longer
worked.  Also the incremental backup would only work to another hard
disk.  I used to do incremental backups (to tape) in the days of Sun
3 workstations.  So why it wouldn't work to removable media...
Unless they have improved things dramatically, my inclination is to
avoid ghost, and I don't think it will let me extract individual
files, directories from the backup afterwards anyway.

> 
> Best regards
> 
>    Sylvain RICHARD
> 
        Thank you,
        Hugh

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Hugh Sasse]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN, Size: 3183 bytes --]

On Tue, 15 Apr 2008, Brian Dessent wrote:

> Hugh Sasse wrote:
> 
> > Trying to copy a windows XP NTFS drive to a big disk using cygwin tools
> > I encounter inaccessible files such as ntusers.dat.  tar is not
> 
> ntuser.dat is the filename of the per-user registry hive
> (HKEY_CURRENT_USER).  It is opened by the system in exclusive mode, so
> it cannot be opened.

Yes, that one I understand, the others I don't.
> 
> > I suspect this is a common problem, but don't know how to frame it
> > correctly to extract something useful from Google.  The setup in
> > question only has the one PC so things like rsync are out.  Windows
> 
> Note that rsync works just fine as a local copying tool like xcopy. 
> There's nothing that says you have to use it with a remote machine.  But

Just the "r" :-)  OK, I'll look into that, thank you.

> it will be of no use to you in this case.

Because it won't access things either?  I can live without ntuser.dat
but its knowing what the others are and why they fail that is part of
the problem.
> 
> > keeps some things in use, but I don't know what.  Can anyone point
> > me at the specific thing I should be reading, or suggest anything?
> > If you mail me off list then, if there is interest I will summarize
> > to the list.  Thank you.
> 
> You can't open the file because it's been opened with a sharing mode
> that disallows any other process to open it.  The third parameter of
> CreateFile() is dwShareMode and can one or more of FILE_SHARE_DELETE,
> FILE_SHARE_READ, FILE_SHARE_WRITE.  If it is 0 then no sharing is
> allowed, and the file cannot be opened by any other process until the
> handle is closed.

So I'd need to boot off something else to get everything then?
> 
> This is why backup programs are more than just file copying utilities. 
> There are a number of files that you can't just copy from a live
> system.  Even if you could forcibly open the file, if you were to
> naïvely copy it you might get inconsistent state, since the whole point
> for opening a file exclusively is so that you can have full control over
> it and implement your own form of transactions.

A good case to consider.  So some databases will also fail if they are live.
> 
> Starting with XP, Microsoft introduced the Volume Shadow Copy service,
> as the other reply in this thread indicated.  This is a vastly
> complicated[1] COM interface that can be used by backup programs to get

See what you mean! 
> consistent state for all files on a live system.  Certainly no Cygwin
> tool or app uses this API.

Thank you.
> 
> What I do is simply use the built in Windows backup program (ntbackup)
> to create a backup to a file.  Ntbackup uses all the proper backup APIs
> and can deal with all in-use files correctly, and the resulting output
> is just a plain file that can be copied around with standard tools, or
> split/burned to DVDR, or whatever.

Can it extract individual files from the backup?  I don't want to copy
old Windows System files on to the new machine when it appears.
> 
> Brian
> 
> [1] http://msdn2.microsoft.com/en-us/library/aa384589.aspx
> 


[-- Attachment #2: Type: text/plain, Size: 218 bytes --]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Brian Dessent]

Hugh Sasse wrote:

> Because it won't access things either?  I can live without ntuser.dat
> but its knowing what the others are and why they fail that is part of
> the problem.

It will be of no use because it still won't be able to access files that
are opened with sharing disallowed.

> So I'd need to boot off something else to get everything then?

No, just use a backup program and not a file copying program.

> Can it extract individual files from the backup?  I don't want to copy
> old Windows System files on to the new machine when it appears.

When you restore you can choose which files from the backup to include. 
Or, when you create the backup you can just exclude those files.  Either
way, it's a big tree-view with a checkbox next to each dir.

Brian

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Hugh Sasse]

On Wed, 16 Apr 2008, Brian Dessent wrote:

> Hugh Sasse wrote:
> 
> > Because it won't access things either?  I can live without ntuser.dat
> > but its knowing what the others are and why they fail that is part of
> > the problem.
> 
> It will be of no use because it still won't be able to access files that
> are opened with sharing disallowed.

OK, thanks, and since that depends on the open command it could be any,
and hard to tell what.
        [...]
> > Can it extract individual files from the backup?  I don't want to copy
> > old Windows System files on to the new machine when it appears.
> 
> When you restore you can choose which files from the backup to include. 
> Or, when you create the backup you can just exclude those files.  Either
> way, it's a big tree-view with a checkbox next to each dir.

Great, thanks.  The last time I tried to use native backup on it, it was
from the disk->properties->tools, and it only allowed me to backup to ...
floppy disk.  For a 70GB drive that was marvellous.  Now I know what to
look for.
> 
> Brian
> 
        Thank you
        Hugh

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Brian Dessent]

Hugh Sasse wrote:

> Great, thanks.  The last time I tried to use native backup on it, it was
> from the disk->properties->tools, and it only allowed me to backup to ...
> floppy disk.  For a 70GB drive that was marvellous.  Now I know what to
> look for.

That's the same ntbackup.  There is a box at the bottom where you enter
the path and filename of where to create the .bkf file.  It probably
suggests the floppy as default, but you don't have to use it.

Brian

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Sylvain RICHARD]

Hugh Sasse wrote
>
> Can it extract individual files from the backup?  I don't want to copy
> old Windows System files on to the new machine when it appears.
>   
Ahah!

Finally, you tell us about what you intended to do in the first place!
Could you be more specific, please?

    Sylvain RICHARD


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Hugh Sasse]

On Wed, 16 Apr 2008, Sylvain RICHARD wrote:

> Hugh Sasse wrote
> > 
> > Can it extract individual files from the backup?  I don't want to copy
> > old Windows System files on to the new machine when it appears.
> >   
> Ahah!
> 
> Finally, you tell us about what you intended to do in the first place!
> Could you be more specific, please?

I want to recover as much of the data I wrote to the old system onto
a portable drive (which being new is big, I forget how big now but of
the order of 200GB), and I want to put the files somewhere on my as yet
unbuilt new machine, which will probably be running Vista dual boot with
Ubuntu.  I don't want to end up with an extra copy of XP on the destination
system as Microsoft aren't exactly relaxed about licencing, so I'll need
to be able to extract the specific files I want afterwards.

What I did before was to write a recursive Ruby program to copy
files and log what didn't get copied.  Most other copying approaches
to solving this die on the first error.  I could get tar not to die,
but couldn't get it to tell me much about what failed or why.  I now
know more about that and why it is likely to fail in lots of cases,
not just ntuser.dat.

Trying to use the backup provided with windows (My\ Computer -> C:
-> Properties -> Tools -> Backup) which I see is ntbackup, was
tricky in the past: it doesn't work with CD RWs, only floppy disks,
or a big file, which I would have had to put on the same disk as I
was trying to back up.  With a new drive I shouldn't have that
problem.  (For some values of "shouldn't".)

So my current plan is to attempt to use NTbackup to do the job, possibly
through the toos -> backup dialogue in the first instance.

Hopefully that hasn't bored you to the point where you are UNcomfortably
numb... :-)
> 
>    Sylvain RICHARD
> 
        Thank you
        Hugh

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: file accessibility and copying/archive/...

[Sylvain RICHARD]

Hugh Sasse wrote:
>
> I want to recover as much of the data I wrote to the old system onto
> a portable drive (which being new is big, I forget how big now but of
> the order of 200GB), and I want to put the files somewhere on my as yet
> unbuilt new machine, which will probably be running Vista dual boot with
> Ubuntu.
If it's a once in a (PC's) lifetime job, then the knoppix offline way is 
the better way. The only guarantee M$ makes regarding shadow copy for 
non shadow copy awareapplications is that you get the same data that 
would reside on the disk after a PC crash. I got burnt when ntbackup 
copied an in-use svn repository (the with the Berkeley db backend).

Any knoppix will do, you don't need the DVD. Just:
1) use hdparm to make sure DMA mode is enabled;
2) use fdisk to repartition your removable hardrive, creating a 
partition of same exact size as the old partition;
3) use dd_rescue to copy the whole partition. I suggest dd_rescue over 
dd because it will handle read errors graciously.  Hint: specify the 
soft blocksize big enough to use caching (say around 4M). Setting it too 
high will cause floating point exceptions I've never really wanted to debug;
4) Enjoy.

Note that the 3rd step may take some time. But you will get maximum 
performance from linear access and no other program trying to access the 
drives concurrently. OTOH, this method is certainly not the best if you 
need to copy say 4GB of data from a 150GB partition.

Best regards,

    Sylvain RICHARD

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/