Recursively recreate hierarchy with NTFS hardlinks by Cygwin

Recursively recreate hierarchy with NTFS hardlinks by Cygwin

[Oleksandr Gavenko]

I am going to write backup solution for my personal laptop.

I have 240GB SSD for development work and 500GB HDD for media files and
backups.

I am care only about project files and don't care about windows files or
program installation (as registry also should be preserved for backup to be
useful).

On Linux I uses:

  cp -al /backup/proj/DATEOLD  /backup/proj/DATENEW
  rsync ... /home/user/proj/ /backup/proj/DATENEW/

and employ hardlinks to preserve space.

NTFS FS has hardlinks. With ``cygutils-extra`` I can:

  $ winln $FROM $TO
  $ echo >>$FROM
  $ cmp $FROM $TO && echo OK

But I don't understand how to emulate recursive behavior of ``cp -r -l``.

Cygwin ``cp -l`` can use hard links but only with:

  CYGWIN=winsymlinks:native

``rsync --hard-links`` isn't reliable:

  bash# echo 1 >orig.txt

  bash# rsync -a --hard-links orig.txt new.txt

  bash# echo 2 >>orig.txt

  bash# diff -u orig.txt new.txt
  --- orig.txt    2017-08-02 14:04:36.976875300 +0300
  +++ new.txt     2017-08-02 14:04:16.547209000 +0300
  @@ -1,2 +1 @@
   1
  -2


There is Windows build-in ``robocopy`` utility but its documentation looks too
complex, I don't like to mix Cygwin and Windows paths and looks like it
doesn't support hard link from "Robocopy.exe /? | grep -i hard".

As development of rdiff-backup stalled in 2009 I don't believe that it
supports hard links in native Windows build.

As a bonus I am also interested to hear about solution with include data
integrity checks to detect data altering or rotting of storage bits in backup.
I think about employing md5sum utility...

-- 
http://defun.work/


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Recursively recreate hierarchy with NTFS hardlinks by Cygwin

[Oleksandr Gavenko]

On 2017-08-02, Oleksandr Gavenko wrote:

> On Linux I uses:
>
>   cp -al /backup/proj/DATEOLD  /backup/proj/DATENEW
>   rsync ... /home/user/proj/ /backup/proj/DATENEW/
>
> and employ hardlinks to preserve space.
>
> ``rsync --hard-links`` isn't reliable:
>
>   bash# echo 1 >orig.txt
>
>   bash# rsync -a --hard-links orig.txt new.txt
>
>   bash# echo 2 >>orig.txt
>
>   bash# diff -u orig.txt new.txt
>   --- orig.txt    2017-08-02 14:04:36.976875300 +0300
>   +++ new.txt     2017-08-02 14:04:16.547209000 +0300
>   @@ -1,2 +1 @@
>    1
>   -2
>

Experiments shown that my goal can be archived in single command:

  mkdir orig
  echo 1 >>orig/my.txt

  mkdir backup
  rsync -a orig/ backup/1
  rsync -a --link-dest=../1 orig/ backup/2

  echo 2 >>backup/2/my.txt
  cmp backup/1/my.txt backup/2/my.txt && echo ok
  cmp orig/my.txt backup/2/my.txt || echo ok

Thanks for Cygwin & rsync with NTFS link support!

-- 
http://defun.work/


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Recursively recreate hierarchy with NTFS hardlinks by Cygwin

[Oleksandr Gavenko]

On 2017-08-02, Oleksandr Gavenko wrote:

> Experiments shown that my goal can be archived in single command:
>
>   mkdir orig
>   echo 1 >>orig/my.txt
>
>   mkdir backup
>   rsync -a orig/ backup/1
>   rsync -a --link-dest=../1 orig/ backup/2
>
>   echo 2 >>backup/2/my.txt
>   cmp backup/1/my.txt backup/2/my.txt && echo ok
>   cmp orig/my.txt backup/2/my.txt || echo ok

``fsutil`` is built-in Windows utility:

  bash# fsutil hardlink list backup/2/my.txt
  \home\tmp\backup\1\my.txt
  \home\tmp\backup\2\my.txt

  bash# mv backup/1 backup/3

  bash# fsutil hardlink list backup/2/my.txt
  \home\tmp\backup\3\my.txt
  \home\tmp\backup\2\my.txt

That is additional proof of work.

My bash backup script looks like:

  LOG=/cygdrive/c/home/backup/backup-job.log
  DST=/cygdrive/d/backup

  DATE=`date +%F`

  COMMON_RSYNC_OPT=( -a --delete )

  log() {
    echo `date +'%F %T'` "$@" >>$LOG
  }

  backup_simple() {
    local SRC=$1
    local BASE=${SRC##*/}

    if [ ! -d $SRC ]; then
        log "$SRC does not exist, leave"
        return
    fi
    if [ -d $DST/$BASE/$DATE ]; then
        log "$SRC is already backuped to $DST/$BASE/$DATE"
        return
    fi
    if [ ! -d $DST/$BASE ]; then
        mkdir -p $DST/$BASE
    fi
    local LAST=`cd $DST/$BASE; find . -maxdepth 1 -type d | grep '[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}' | sort -r | head`
    local RSYNC_OPT=( "${COMMON_RSYNC_OPT[@]}" )
    if [ -n "$LAST" ]; then
        RSYNC_OPT+=( --link-dest=../$LAST )
    fi
    rsync "${RSYNC_OPT[@]}" $SRC/ $DST/$BASE/$DATE/
    log "$BASE is backuped"
  }

  backup_simple /cygdrive/c/home/devel/soapui
  backup_simple /cygdrive/c/home/devel/postman

================================================================

What options is recommended for using Cygwin's rsync?

Backuped files don't need to preserve permission, ACL, etc. The goal is to
preserve hierarchy and content and make backuping lightweight (with
timestamps/size checks and hardlinks).

I collected list of:

  -t --no-p --no-l --no-acls --no-o --no-g --no-D -O -J -m --no-partial --delete

to ignore most of file meta-information and make time/size bases checks.

================================================================

What is the better way to schedule backup task?

I've got familiar with Windows Task Scheduler, it has interesting options:

* run task as soon as possible after scheduler start is missed
* stop task if it runs longer then
* wake the computer to run this task

Do I need to bother with Cygwin CRON?

-- 
http://defun.work/


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple