Heres how:
Use Case
I have two identical drobo5ns. One has more capacity than the other. They are both connected to a GB router.
Drobo5n1 - 5 x 2TB
Drobo5n2 - 5 x 3TB
Mac Desktop
Mac Desktop
I am paranoid about losing my data so I keep a backup of my main Drobo5N on another one. Typically I open both shares on my MAC, then drag and drop files between them. This has lots of problems with it which I won't go into but basically it takes ages and i hate leaving my MAC on doing the copying.
I want to automatically backup my data form one Drobo to the other, without using my MAC, and do some fancy synchronising so i only copy newer files over... AND... i don't want to copy permissions. I'll cover this shortly.
Good plan huh?
Well, yes. BUT FUCK ME this was REALY difficult to accomplish and took alot of research and testing
Hopefully this will help others.
Hopefully this will help others.
So this is what i did...
Format the Target Drobo (Its optional but i'll tell you why)
I formatted the target Drobo.. this is because the original data is from many places, has both MAC and Windows permissions all over it, and a bunch of MAC metadata files that i don't like or need. Trying to compare and copy these files was not going so great from one to another doesn't work very well at the best of times.
Starting fresh or just in a new share means you'll have none of these problems. But i never want to have the same problems again either. Even on a fresh target, once the files are copied i'm going to have the same problems again I think as the permissions will still be there.
For the most part, most of the files have Windows NTFS permissions, but the computers and users i created those files with don't exist anymore. I use a MAC now, and maybe one day I'll use windows again or a Google table or whatever.
So for my use case I don't want to fight with the old permissions. I want nice and bland files with the default Drobo permissions. Then i can access my files anyway i like later.
* this is the plan at least I haven't fully compared permissions after copying yet
If you are interested in how Drobo can manage to copy the files in the first place.. its using a sticky bit on at the root level of each share, effectively giving Drobo admin full access to all files on the system. Here is more info on sticky bit
http://www.thegeekstuff.com/2013/02/sticky-bit/?utm_source=feedburner
Lastly: Do this at your own risk, and if you don't want to, try it on a new share at least ;)
Update your Drobo Firmware and Drobo Dashboard
Format the Target Drobo (Its optional but i'll tell you why)
I formatted the target Drobo.. this is because the original data is from many places, has both MAC and Windows permissions all over it, and a bunch of MAC metadata files that i don't like or need. Trying to compare and copy these files was not going so great from one to another doesn't work very well at the best of times.
Starting fresh or just in a new share means you'll have none of these problems. But i never want to have the same problems again either. Even on a fresh target, once the files are copied i'm going to have the same problems again I think as the permissions will still be there.
For the most part, most of the files have Windows NTFS permissions, but the computers and users i created those files with don't exist anymore. I use a MAC now, and maybe one day I'll use windows again or a Google table or whatever.
So for my use case I don't want to fight with the old permissions. I want nice and bland files with the default Drobo permissions. Then i can access my files anyway i like later.
* this is the plan at least I haven't fully compared permissions after copying yet
If you are interested in how Drobo can manage to copy the files in the first place.. its using a sticky bit on at the root level of each share, effectively giving Drobo admin full access to all files on the system. Here is more info on sticky bit
http://www.thegeekstuff.com/2013/02/sticky-bit/?utm_source=feedburner
Lastly: Do this at your own risk, and if you don't want to, try it on a new share at least ;)
Update your Drobo Firmware and Drobo Dashboard
Connect to the second drobo
- I set the username to lowercase admin as i like it that way.
- Set the password
- Set the password
- Rename drobo
- Enable DroboApps in the admin console
- Enable DroboApps in the admin console
- Set a static ip
- Create a taget Share with admin has full permission - eg DATA
- Restart
- Create a taget Share with admin has full permission - eg DATA
- Restart
Upload OpenSSH
You need to do this on both drobos. On your computer connect to the Drobo as admin. This makes the drobapps folder appear. Download the file openssh.tgz and save it in the /drobapps/ folder
page: http://www.droboports.com/app-repository/openssh/
file: https://github.com/droboports/openssh/releases/download/v6.8-fs/openssh.tgz
Install Dropbear
You need to do this on both drobos. Connect to the Drobo Dashboard app again and goto Droboapps. Install dropbear and start it.
use putty, or on a mac just ssh to the drobo with the admin username and password
$ ssh admin@192.168.1.100
page: http://www.droboports.com/app-repository/openssh/
file: https://github.com/droboports/openssh/releases/download/v6.8-fs/openssh.tgz
Switch to the root user
# sudo su -
run this to unpack and install (this is basically how you install any drobo ported app)
# /usr/bin/DroboApps.sh install
Configure OpenSSH on port 2222
Dropbear runs on port 22 so i avoid using OpenSSH on that port by changing the config using vi
# vi /mnt/DroboFS/Shares/DroboApps/openssh/etc/sshd_config
Press i to start insert mode. edit this this line
#Port 22
to this
Port 2222
Press :
then wq (write quit)
All done, goto the drobo dashboard and stop start the OpenSSH app (down the bottom of all the apps).
From here on, connecting via putty or SSH is done on port 2222 to take advantage of OpenSSH. So from your computer you can do
$ ssh -p 2222 admin@192.168.1.x
Create .ssh folder
You need to do this on both drobos.
I am not sure but i think dropbear creates this folder, and this file
# /mnt/DroboFS/home/admin/.ssh/
# /mnt/DroboFS/home/admin/.ssh/authorized_keys
If not create them, and set correct permissions. I 'think' this should do it...
# mkdir /mnt/DroboFS/home/admin/.ssh/
# touch /mnt/DroboFS/home/admin/.ssh/authorized_keys
# chmod 700 -R /mnt/DroboFS/home/admin/.ssh/
Before we move on...
Why use OpenSSH at all? Well dropbear is super slow is why... i believe this is due to windowsizes
https://lab.nexedi.com/nexedi/slapos/commit/605e564b8f3008cdb48dbc6e82c956029469ff02
Create SSH keys
You need to create the keys on drobo2, and copy the public key to drobo1. This is because drobo2 will be running the sync script and needs to connect to drobo1 and pull the files. You can do it in reverse but i like the second drobo doing all the work. Also, i feel safer messing around on my second drobo..
Now lets make some keys yeah? On Drobo2... cd to the openssh app and list files
Drobo5N2:~ # cd /mnt/DroboFS/Shares/DroboApps/openssh/bin
Drobo5N2:~/Shares/DroboApps/openssh/bin # ls
scp* sftp* slogin@ ssh* ssh-add* ssh-agent* ssh-keygen* ssh-keyscan*
Drobo5N2:~/Shares/DroboApps/openssh/bin # ls
scp* sftp* slogin@ ssh* ssh-add* ssh-agent* ssh-keygen* ssh-keyscan*
Run this and accept the defaults, but please read the bloody prompts at least and make sure the files end up as per below
# ./ssh-keygen
Save files to
/mnt/DroboFS/home/admin/.ssh/id_rsa
/mnt/DroboFS/home/admin/.ssh/id_rsa.pub
Cat the pub key and copy that garbage to clipboard
# cat /mnt/DroboFS/home/admin/.ssh/id_rsa.pub
from your computer logon to drobo1 via OpenSSH on port 2222
$ ssh -p 2222 admin@192.168.1.x
Create the directories you did before.
edit the authorized_keys file with vi, and paste the SSH key into the file and :wp to save
NEATO!
GO BACK TO DROBO2
Login with no password
Drobo2 is going to do all the copying so go back to drobo2. Unfortunately, the ssh executable is not in the 'PATH' so you have to run it with the full path. I tried adding it to the path but it gets magically overwritten.
Run this with the PRIVATE key (id_rsa) and you should log straight into drobo1 with no password
# /mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -i /mnt/DroboFS/home/admin/.ssh/id_rsa admin@192.168.1.x
exit drobo1 and re-run that command as the root user on drobo2. Accept the prompt to add this connection to known hosts. If you don't do this step, when you run CRON later as root it'll fail.
# sudo su -
# /mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -i /mnt/DroboFS/home/admin/.ssh/id_rsa admin@192.168.1.x
exit drobo1 and re-run that command as the root user on drobo2. Accept the prompt to add this connection to known hosts. If you don't do this step, when you run CRON later as root it'll fail.
# sudo su -
# /mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -i /mnt/DroboFS/home/admin/.ssh/id_rsa admin@192.168.1.x
Did it connect with no pwd - it should have, If it didn't work... thats sucks. Maybe your SSH keys might not have right permissions. Look that shiz up. chmod some shit
Now for the SSH / RSYNC part
The basic copying process works by creating an tunnel, then contacting rsync on the destination, comparing the files on both ends, creating a list, and transfering the files.
First off lets talk about how rsync works for for me:
The trick to do this on drobo is to tell the destination where to find the rsync executable. You need to include this..
--rsync-path="/sbin/rsync"
For my setup i specifically include the folders i want and exclude the files / patterns i don't want copied
--files-from=/mnt/DroboFS/home/admin/include.txt: include these files. eg.
--exclude-from=/mnt/DroboFS/home/admin/exclude.txt exclude these
I exclude this MAC metadata crap in my file
*DS_Store*
._*
.AppleDouble
Then there are the switches to do the sync. once apon a time i would use -a, but that copies the permissions, so i just use -r
r: recursive
v: verbose
h: human readable
--progress: show the progress
--ignore-existing: only copy new stuff
--delete-after: when done, delete stuff on the target that is not on the source
--delete-excluded; also delete stuff i put in the excluded file on the source
Rsync stages the file copy on the destination when it copies. I like to dump the partial files in a completely different place for better visibility etc
--temp-dir=/mnt/DroboFS/Shares/Public/partial: copy temporary, partial files here
Lastly, log the copy process. We'll be running this as a CRON job so you need this or you won't know what going on.
--log-file=/mnt/DroboFS/home/admin/data.log
---ARE YA WITH ME---?
Now for the full SSH command
So how does the SSH part work in more detail? Basically we
execute rsync on the target machine over SSH on port 2222 using the PRIVATE key (id_rsa)
...then all the rsync commands
...then the destination
.. then the source (and folders from the include file)
rsync -e "/mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -p 2222 -i /mnt/DroboFS/home/admin/.ssh/id_rsa"
--rsync-path="/sbin/rsync" -rvh --temp-dir=/mnt/DroboFS/Shares/Public/partial --progress --ignore-existing --delete-after --delete-excluded --files-from=/mnt/DroboFS/home/admin/include.txt --log-file=/mnt/DroboFS/home/admin/data.log --exclude-from=/mnt/DroboFS/home/admin/exclude.txt
admin@192.168.1.202:/mnt/DroboFS/Shares/Data/
/mnt/DroboFS/Shares/Data/
Put it all together
rsync -e "/mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -p 2222 -i /mnt/DroboFS/home/admin/.ssh/id_rsa2" --rsync-path="/sbin/rsync" -rvh --temp-dir=/mnt/DroboFS/Shares/Public/partial --progress --ignore-existing --delete-after --delete-excluded --files-from=/mnt/DroboFS/home/admin/include.txt --log-file=/mnt/DroboFS/home/admin/data.log --exclude-from=/mnt/DroboFS/home/admin/exclude.txt admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/
NB: Keep in mind that the folders i am copying are in my includes text file so really this command will be doing this..
admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/folder1
admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/folder2
admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/folder3
admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/folder4 ...
AWWW YISS
Now lets put that command in a script
Simply create a bash script with vi, put your command in it, save it, then chmod it with +x and run it.
# vi data_backup.sh
# /bin/bash
# your command
Save, then
# chmod +x data_backup.sh
Execute ./data_backup.sh
You should see rsync calculate the file list then start copying like mad. Make sure this works as the root user before you attempt doing a cron job.
AND FINALLY CRON ftw
Setup crontabs
Drobo uses busybox for most of its linux style commands including crond.
https://busybox.net/downloads/BusyBox.html
To get crond working you need to sudo to root, create 'some' of the default path, then create a crontab, and start crond
Drobo5N2:~ $ crond --help
BusyBox v1.23.2 (2015-11-06 13:39:54 PST) multi-call binary.
Usage: crond -fbS -l N -d N -L LOGFILE -c DIR
-f Foreground
-b Background (default)
-S Log to syslog (default)
-l N Set log level. Most verbose:0, default:8
-d N Set log level, log to stderr
-L FILE Log to FILE
-c DIR Cron dir. Default:/var/spool/cron/crontabs
# sudo su -
# mkdir -p /var/spool/cron
# /bin/busybox crond
The its basic crontab stuff, which runs as root so thats cool. Run the command below to create a job list and use cron numbering to schedule a script to run the SSH / RSYNC command when you want it run.
# crontab -e
Keep in mind the drobo uses PDT time so set your job accordingly
Drobo5N2:~ # date
Fri Aug 11 05:00:54 PDT 2017
To view the crontab you created.. as root
Drobo5N2:~ # crontab -l
51 16 * * 6 /mnt/DroboFS/home/admin/data_backup.sh*
Drobo5N2:~ #
Testing
When getting started, you can put a -n in your rsync switches to do a dry run copy
Assuming crond kicks off, you can tail the log file and see if copying has started
You can also do ps aux | grep rsync on either drobo and watch the process kick off if its working
Tips
You can do a -I if you want to just overwrite whatever is on the target. Sometimes i find this handy.
Kill the crond command if its not working nicely on first go
Last but not least here are some references i used for this magic.
If this page has helped you .. well sweet. You r welcome.
-jules
#################################################################################
References
busybox tools and commands
https://busybox.net/downloads/BusyBox.html
Dropbear
https://linux.die.net/man/8/dropbear
Drobo dropbear windowsize sucks the dick - windowsizes
https://lab.nexedi.com/nexedi/slapos/commit/605e564b8f3008cdb48dbc6e82c956029469ff02
i tried using a drobo ported version of rsync at some stage but gave up
https://sites.google.com/a/droboports.com/www/app-repository/rsync-3-0-9
dropbear man page
https://linux.die.net/man/8/dropbear
dropbears equivalent to 'ssh' (dbclilent)
https://manpages.debian.org/jessie/dropbear/dbclient.1.en.html
basic rsync operation
http://www.linuxquestions.org/linux/answers/Networking/Using_rsync_to_mirror_data_between_servers
rsync with includes
https://stackoverflow.com/questions/16647476/how-to-rsync-only-a-specific-list-of-files
cron scheduling
https://en.wikipedia.org/wiki/Cron
busybox tools and commands
https://busybox.net/downloads/BusyBox.html
https://sites.google.com/a/droboports.com/www/app-repository/rsync-3-0-9
dropbear man page
https://linux.die.net/man/8/dropbear
dropbears equivalent to 'ssh' (dbclilent)
https://manpages.debian.org/jessie/dropbear/dbclient.1.en.html
basic rsync operation
http://www.linuxquestions.org/linux/answers/Networking/Using_rsync_to_mirror_data_between_servers
rsync with includes
https://stackoverflow.com/questions/16647476/how-to-rsync-only-a-specific-list-of-files
cron scheduling
https://en.wikipedia.org/wiki/Cron
busybox tools and commands
https://busybox.net/downloads/BusyBox.html
drobo pinhole reset
http://www.drobospace.com/forums/printthread.php?tid=77106