Sync workspace between 2 machines

8 minute read   Updated:

At some point you will have more than 1 workstation that you work on, say a main desktop and an on-the-go laptop. Some may have a few, e.g. a desktop and laptop at work, and a personal computer at home. I myself have a MacBook Pro and an iMac that I switch between every other day.

I need a reliable method to sync my workspace. There are a few good options:

  • git
  • rsync
  • File synchronization/sharing services (iCloud, Google Drive, Microsoft OneDrive, Dropbox, Box, etc.)

They all have their pros and cons, but I’d recommend only Dropbox specifically for developers due to its sync performance.

GIT

git is a tool for source code control. git is not a general file synchronization tool, however you can still use it for this purpose. Since it’s a source control tool, git emphasizes history and versioning of files, which is great when we need to rollback or compare differences between different histories.

The typical workflow is:

On main machine A

# Establish the main repository, only needed to be done once on machine A
$ git init .

# Add all files to create a history
$ git commit -am "Add client's design round 1"

On machine B

# Add machine A as an 'origin' remote, only needed o be done once on machine B
# Note: make sure machine B can talk to machine A over ssh/git or http/git
$ git remote add origin <machineA-ip>

# Sync FROM main machine: pull latest changes
$ git fetch origin
$ git pull

# Edit and create a new history
... edit files
$ git commit -am "Add client's design round 2"

# Sync TO main machine: push latest changes
$ git push origin

Even though we can force ourselves to use git for the general purpose of file synchronization, it wasn’t designed for this use case. Here is the complete pros and cons list:

Pros

  • Has file versioning
  • Has history message
  • Can rollback
  • Can ignore files
  • Can see differences between versions natively through command line git client or GUI clients
  • There are many GUI clients such as SourceTree, GitKraken, or GitHub Desktop

Cons

  • Doesn’t have automatic sync
  • Machine B needs to be able to talk to Machine A over SSH (simpler setup) or HTTP (more complicated setup)
  • Both machines need to be running at the time of syncing for the sync to work
  • Can only initiate the sync (pull/push) from machine B to the the main machine A
  • Doesn’t handle nested git well. Only works well with non-git files. If there is a subfolder that is a git repo itself, this outer git repo will run into issues.

Note that you can substitute git with any other source code control software and they will have similar problems as they’re designed to control source code and not to synchronize files.

RSYNC

rsync is specifically designed for the purpose of files synchronization, hence the name.

Typical usage is:

On either machine

$ rsync -azvhO --no-perms --no-owner --no-group --delete --exclude '.*' \
    /path/to/source/folder/on/this/machine/ \
    sshUser@theOtherMachineIP:/path/to/destination/folder/on/the/other/machine

rsync works great but not without its limits.

Pros

  • Can sync both ways: machine A to machine B and vice versa
  • Can invoke other commands over SSH when running rsync
  • Can sync any files whether it’s regular or git repo files
  • Can ignore files

Cons

  • Doesn’t have automatic sync
  • 2 machines need to be able to talk to each over over SSH
  • Both machines need to be running at the time of syncing for the sync to work
  • GUI clients exist but very limited (e.g. non-existence to terrible on Mac)
  • Doesn’t have file versioning
  • Doesn’t have history
  • Doesn’t have rollback. Since there is no history/versioning, if you accidentally delete a file and sync it, the file is gone forever.

FILE SYNCHRONIZATION SERVICES

The one big problem we have with git and rsync is that they are not automatic. Even though we can set up a cronjob that will periodically run rsync for us, it’s another tool on top of rsync and as I mentioned above due to not having rollback, automatic rsync is very risky when you accidentally delete a file.

So come File Synchronization Services namely iCloud, Google Drive, Microsoft OneDrive, Dropbox, Box, and many others. Let’s compare their plans quickly.

Pricing

Compare   iCloud Google Drive Microsoft
OneDrive
Dropbox Box
Free   5GB 1 15GB 2 5GB-15GB 3 2GB-16GB 4 10GB
Paid
(monthly)
Less
than
1TB
50GB: $0.99
200GB: $2.99
100GB: $1.99 50GB: $1.99   100GB: $10
Paid
(monthly)
1TB
and
above
1TB: $9.99
2TB: $19.99
1TB: $9.99
2TB: $19.99
10TB: $99.99
20TB: $199.99
30TB: $299.99
N/A 5 1TB: $9.99 N/A
Paid
(yearly)
  N/A 100GB: $19.99
1TB: $99.99
(17% saving) 6
N/A 5 1TB: $99
(17% saving) 6
N/A
OS
(desktop)
  Mac,
Windows
Mac,
Windows
Mac,
Windows
Mac,
Windows,
Linux
Mac,
Windows
OS
(mobile)
  iOS iOS,
Android
iOS,
Android,
Windows Phone
iOS,
Android,
Windows Phone
iOS,
Android,
Windows Phone,
BlackBerry

Free: from the free pricing point of view, 15GB Google Drive, 15GB Microsoft OneDrive (with referral bonus), and 16GB Dropbox (with referral bonus) seem best.

Paid a bit: if you need just a little bit more storage beyond the free tier, go with the monthly 50GB $0.99/m iCloud or 100GB $1.99/m Google Drive. Note that because iCloud ties so closely across Apple ecosystem (iPhone, iPad, Mac), photos on your iPhone may consume most of your iCloud storage.

Paid optimal: if you want to take advantage of yearly discount, both Google Drive and Dropbox offer the max storage of 1TB $99.99/y. If you want to pay yearly for the discount but don’t need up to 1TB, there is only an option of 100GB $19.99/y Google Drive.

I use Dropbox for its free 16GB max not because it has the highest amount of free storage (16GB compared to the 2nd up of 15GB from Google Drive or Microsoft OneDrive), but because of its sync performance.

Performance

For regular users with normal files such as documents, media, etc. all of the above services should perform similar. But for developers with source code that have more than a hundred thousand of files with dozen levels of nested folders, most of the File Synchronization Services struggle to sync.

For example, at the time of this writing, I wanted to sync about 5GB of data (source code, client materials, project documents, etc.). You would think that 5GB is nothing, any of those services could handle this easily. It’s true that 5GB of data was only on the average size, but the problem wasn’t about the amount of data, it was the number of files and the depths of nested folders.

In that 5GB of data, I had about ~ 415,000 files (~350,000 regular files, and ~65,000 directories) with the maximum levels of nested folders as 34. Here are the more detailed numbers:

# Total data size
$ du -sh ~/workspace/
    5.6G

# Number of all files (regular files and directories)
$ find ~/workspace/ | wc -l
    415291

# Number of regular files
$ find ~/workspace/ -type f | wc -l
    347609

# Number of directories
$ find ~/workspace/ -type d | wc -l
    66783
    
# File's depth (number of nested directories)
$ find ~/workspace/ -type d -printf '%d\n' | sort -n | tail -1
    34

Microsoft OneDrive

I don’t use OneDrive. Any reader that has experience with OneDrive sync speed on hundreds of thousands of files, let us know what you think in the comment section.

iCloud

Apple hasn’t made iCloud storage as intuitive as possible to understand (see iCloud vs iCloud Drive). I only use iCloud for a few minor data sync among iOS devices and Mac machines e.g. screenshots that are automatically saved on Desktop, notes using Notes/TextEdit, and app specific data like the primary vault file in 1Password, etc. I don’t use iCloud for main data sync.

Google Drive

  • Upload & Download: took forever, had to leave it overnight but it eventually finished.

  • Change detection: great, can detect file change instantly (in a matter of a few seconds) and sync right away.

Box

I wouldn’t have tried Box if I didn’t have 50GB free thanks to a promotion where Box gave you 50GB permanently by using its iOS app when it first came out.

  • Upload & Download: took forever, had to leave it overnight but it eventually finished even though it’s much slower than Google Drive.

  • Change detection: absolutely terrible and completely horrible! There are so many complains about its sync speed. What you will notice is that Box will scan your entire data folder every time it wants to sync, instead of just the changes. An educated guess would be that Box implementation doesn’t properly utilize a change detection method like filesystem inotify or similar approaches. Just sync the changed files, why scan the entire system? At the time, I had Box Sync v4 on Mac and it looked to me that it was doing a Depth First Search every time it scanned. A change in the ‘z’ folder would not be synced until it has scanned all the way from ‘a’, ‘b’, ‘c’, etc. folders and not until it has reached the deepest level of all those folders first. This terrible scan was compounded by the fact that I had so many files in so many sub-folders that led to an absolutely horrible experience. A single file change resulted in Box scanning for a couple of hours over all my data (why!) and kicked off my CPU usage to half considering my MacBook Pro had an i7, 16GB of RAM, and SSD.

Terrible. Horrible. AVOID AT ALL COST!

Dropbox

  • Upload & Download: took some times, still several hours but significantly faster than Google Drive.

  • Change detection: great, can detect file change instantly (in a matter of a few seconds) and sync right away.

With this great performance, I’d strongly recommend Dropbox to developers for their data sync among their machines, especially when they sync code that has more than a hundred thousand files with dozen levels of nested folders.

  1. shared with your mobile iCloud storage 

  2. shared with all your Google related services: gmail, photos, docs 

  3. free up to 15GB max (10GB referral bonus on top of the initial 5GB). This is not verified since I don’t use OneDrive. Leave me a comment if you’re using OneDrive and can verify this free 15GB max. 

  4. free up to 16GB max (14GB referral bonus on top of the initial 2GB). The 16GB max is verified as I’m using this. 

  5. there are 1TB and 5TB yearly plans with discount, only if you purchase a bundle of Storage + Office 365  2

  6. yearly discount only offered for these specific storage amount  2

Leave a Comment