Users browsing this thread: 1 Guest(s)
z3bra
Grey Hair Nixers
Synchronizing files is an interresting topic, as there are so many ways to do it, and even more use-cases!

I think that there are 3 types of tools to sync files between multiple hosts: one-way, two-way or hybrid.

One way sync is when a single host is pushing to all others at the same time, and no change would happen while the transfer is on (in theory). This assume that you are only modifying one host at a time, and that you always havw the latest changes when you edit files on any host.
Tools in this vain are rsync(1), rdist(1), ftp(1) or even git(1). They work in a push/pull manner, and you have no way to ensure that what you have is the latest version, because there is no "synchronisation" state between your hosts. You either push (force latest change to be what you push), or pull (assume your remote hosts have the latest version).

Two-way sync works in real-time. Whenever a change happen, it gets pushed to all other hosts. If two hosts are modified in parallel, their mutual changes have to be merged. If the same part of the file is changed, conflict happens. That is the price of two-way sync.
You get a more reliable synchronization, but it is easier to corrupt your data. Two-way sync tools are also forced to act as daemons, and must always be watching your files to push the changes in real time. Example tools are unison, syncthing and dropbox.

Finally, here comes the shameless plug: synk! (README)
It is what I find the best of both world. It is a one-way sync, but it first tries to find the most recent copy using the mtime of your file. This require a good time sync between your hosts.
Note that this is only a draft, and there are design problems, like concurrency issues if you fire it on multiple hosts at the same time. But it does the job quite well!
It is not a daemon, so you fire it up whe. you see fit: manually, with cron, entr, wendy, fswatch, inotifywatch, ... whatever. When started, it connect to all hosts, fetch all mtime locally and then spawn rsync(1) processes from the host that has the highest mtime to push the file on all other hosts!
I would like to use bittorrent internally instead of rsync algorithm, to be even faster, but that is another topic ;)

What do you guys think? Does that fill a need? Do you see many flaws in the design?


Messages In This Thread
Sharing Media - by venam - 25-08-2018, 04:42 AM
RE: Sharing Media - by jkl - 25-08-2018, 12:13 PM
RE: Sharing Media - by venam - 25-08-2018, 12:30 PM
RE: Sharing Media - by acg - 25-08-2018, 10:37 PM
RE: Sharing Media - by venam - 26-08-2018, 03:49 AM
RE: Sharing Media - by oda - 26-08-2018, 06:48 AM
RE: Sharing Media - by pkal - 26-08-2018, 07:04 AM
RE: Sharing Media - by Dworin - 01-09-2018, 06:23 AM
RE: Sharing Media - by prx* - 15-12-2018, 03:26 PM
RE: Sharing Media - by z3bra - 17-12-2018, 05:37 AM
RE: Sharing Media - by atbd - 18-12-2018, 06:41 AM
RE: Sharing Media - by z3bra - 18-12-2018, 08:34 AM
RE: Sharing Media - by prx* - 18-12-2018, 03:10 PM
RE: Sharing Media - by z3bra - 19-12-2018, 04:27 AM
RE: Sharing Media - by prx* - 19-12-2018, 04:51 AM
RE: Sharing Media - by atbd - 19-12-2018, 05:53 AM
RE: Sharing Media - by prx* - 19-12-2018, 10:06 AM
RE: Sharing Media - by venam - 22-03-2019, 04:16 PM
RE: Sharing Media - by jkl - 22-03-2019, 04:18 PM
RE: Sharing Media - by z3bra - 22-03-2019, 09:01 PM
RE: Sharing Media - by wolf - 26-03-2019, 11:57 PM
RE: Sharing Media - by venam - 12-04-2021, 02:29 AM
RE: Sharing Media - by TheAnachron - 12-04-2021, 04:24 AM
RE: Sharing Media - by jkl - 12-04-2021, 04:25 AM
RE: Sharing Media - by TheAnachron - 12-04-2021, 04:45 AM