Users browsing this thread: 5 Guest(s)
josuah
Long time nixers
I'm planning to roll currently rolling my own "backup" script, that stores things like that:


<code>
<pre style="line-height: 8px;">
./ - root dir you want to backup

├── file1

├── dir1/
│ ┬
│ ├── file2
│ └── file3

├── file4

└── .version/ - where things are backed up

├── obj/ - The same files as in the root dir, but with different names: their hash.
│ ┬
│ ├── d41d8cd98f00b204e9800998ecf8427e - same content as file1
│ ├── ba1f2511fc30423bdbb183fe33f3dd0f - same content as file2 and file3
│ ├── c097062a51458c527f04b44c210efefc - same content as file4 (before)
│ └── 1c93f779e81d1710e671e5355ee761ef - same content as file4 (now)

├── rev/ - "revisions": list of equivalences between hashes and filename at a given time.
│ ┬
│ ├── fb82199922251c63d72b17a0860198e6 - initial version
│ └── 4dca17472fcdda9affa259b318243a54 - file4 got edited

└── history - list of rev/* in chronological order.
</pre>
</code>


How to do this without all the cli option & stuff:

<code>
mkdir -p .version/obj .version/rev

find . -type f ! -name .version ! -path '*/.version/*' -print0 | sort -z |
xargs -0 md5sum | tee -a .version/tmp | while read -r hash path
do [ -f ".version/obj/$hash" ] || cp "$path" ".version/obj/$hash"
done

chmod -w .version/obj/* .version/rev/*
new="$(md5sum .version/tmp)" && new="${new%% *}" || exit 1
mv -f .version/tmp ".version/rev/$new"

[ -f .version/history ] && old="$(tail -n 1 .version/history)" || old=''
[ -f .version/history ] && [ "$new" = "$old" ] ||
printf '%s\n' "$new" >> .version/history
printf 'old %s\nnew %s\n' "$old" "$new"
</code>


This allows to make versioning easily: just need to add new <code>./.version/rev/*</code> ~100 lines text file (md5sum output) to have a new version.

This permits de-duplication of content across versions: Any file that did not change is not duplicated: it already exist in <code>.version/obj/&lt;hash-of-the-file&gt;</code>, even if the file gets renamed.

I may even add some merging operations to gather content on multiple storage, then merge them with <code>version merge tree1 tree2</code>.

It seems that it is a bit like what git does: https://git-scm.com/book/en/v2/Git-Inter...it-Objects

For now, I use md5sum because it is faster and I just start to play with it. The process is rather slow compared to git, but it's only a 227 shell script for now...

So far, backing up data and switching between revisions works. You can safely run <code>version</code> without argument to get short usage message to start playing with it.

[EDIT] formatting



I just bought a hard drive enclosure (sorry for this being some kind of ads).

(25-09-2016, 12:55 PM)vain Wrote: ...

I hope this can face a few of these issues. This along with my script generating md5sums and comparing them with previous revision at every version (diff with <code>version diff</code>).

As I can not roll a 10 year long tested software with a lot of work to make it work everywhere, I bet on simplicity to make things safe:

The script never ever delete data. `rm` never acts on the read-only <code>./.version/{rev,obj}/*</code> content. So in case of disaster, <code>version revision &lt;revision-hash&gt;</code> restores everything.

I will let you know in the Unix Diary thread if I encounter a great disaster with this! ^_^

One problem: it does not provide any way to backup data on a remote host. Z3bra's <code>synk</code> may be much better for this purpose. Maybe I could combine them some day.


Messages In This Thread
Backing up and Deploying - by venam - 29-05-2016, 12:01 PM
RE: Backing up and Deploying - by z3bra - 07-09-2016, 09:14 AM
RE: Backing up and Deploying - by TheAnachron - 07-09-2016, 09:37 AM
RE: Backing up and Deploying - by z3bra - 07-09-2016, 10:45 AM
RE: Backing up and Deploying - by TheAnachron - 09-09-2016, 04:20 AM
RE: Backing up and Deploying - by movq - 25-09-2016, 12:55 PM
RE: Backing up and Deploying - by venam - 25-09-2016, 01:33 PM
RE: Backing up and Deploying - by movq - 25-09-2016, 03:19 PM
RE: Backing up and Deploying - by josuah - 02-01-2017, 04:52 PM
RE: Backing up and Deploying - by pranomostro - 03-01-2017, 09:00 AM
RE: Backing up and Deploying - by jkl - 03-01-2017, 10:39 AM
RE: Backing up and Deploying - by venam - 03-01-2017, 11:09 AM
RE: Backing up and Deploying - by z3bra - 31-07-2020, 03:55 PM
RE: Backing up and Deploying - by opfez - 31-07-2020, 07:44 PM
RE: Backing up and Deploying - by z3bra - 11-08-2020, 04:45 AM
RE: Backing up and Deploying - by venam - 02-08-2021, 01:21 AM
RE: Backing up and Deploying - by fre d die - 02-08-2021, 09:15 AM
RE: Backing up and Deploying - by jkl - 02-08-2021, 11:12 AM
RE: Backing up and Deploying - by z3bra - 04-08-2021, 08:46 AM
RE: Backing up and Deploying - by venam - 07-11-2021, 04:46 AM
RE: Backing up and Deploying - by swathe - 30-05-2016, 01:44 AM
Your backup solution? - by TheAnachron - 07-09-2016, 05:14 AM
RE: Your backup solution? - by venam - 07-09-2016, 05:21 AM
RE: Your backup solution? - by TheAnachron - 07-09-2016, 05:58 AM
Backups! - by z3bra - 19-04-2019, 08:39 AM
RE: Backups! - by venam - 19-04-2019, 10:00 AM