Friday, August 11, 2017

Drobo to Drobo Serverless backup using SSH, RSYNC and CRON

Would you like to automatically backup your data from one drobo to another drobo automatically, without a computer doing the copying?

Heres how:

Use Case

I have two identical drobo5ns. One has more capacity than the other. They are both connected to a GB router.

Drobo5n1 - 5 x 2TB
Drobo5n2 - 5 x 3TB
Mac Desktop

I am paranoid about losing my data so I keep a backup of my main Drobo5N on another one. Typically I open both shares on my MAC, then drag and drop files between them. This has lots of problems with it which I won't go into but basically it takes ages and i hate leaving my MAC on doing the copying.

I want to automatically backup my data form one Drobo to the other, without using my MAC, and do some fancy synchronising so i only copy newer files over... AND...  i don't want to copy permissions. I'll cover this shortly.

Good plan huh?

Well, yes. BUT FUCK ME this was REALY difficult to accomplish and took alot of research and testing

Hopefully this will help others.

So this is what i did...

Format the Target Drobo (Its optional but i'll tell you why)

I formatted the target Drobo.. this is because the original data is from many places, has both MAC and Windows permissions all over it, and a bunch of MAC metadata files that i don't like or need. Trying to compare and copy these files was not going so great from one to another doesn't work very well at the best of times.

Starting fresh or just in a new share means you'll have none of these problems. But i never want to have the same problems again either. Even on a fresh target, once the files are copied i'm going to have the same problems again I think as the permissions will still be there.

For the most part, most of the files have Windows NTFS permissions, but the computers and users i created those files with don't exist anymore. I use a MAC now, and maybe one day I'll use windows again or a Google table or whatever.

So for my use case I don't want to fight with the old permissions. I want nice and bland files with the default Drobo permissions. Then i can access my files anyway i like later.

* this is the plan at least I haven't fully compared permissions after copying yet

If you are interested in how Drobo can manage to copy the files in the first place.. its using a sticky bit on at the root level of each share, effectively giving Drobo admin full access to all files on the system. Here is more info on sticky bit

http://www.thegeekstuff.com/2013/02/sticky-bit/?utm_source=feedburner

Lastly: Do this at your own risk, and if you don't want to, try it on a new share at least ;)

Update your Drobo Firmware and Drobo Dashboard

Connect to the second drobo
- I set the username to lowercase admin as i like it that way.
- Set the password
- Rename drobo
- Enable DroboApps in the admin console
- Set a static ip
- Create a taget Share with admin has full permission - eg DATA
- Restart

Upload OpenSSH

You need to do this on both drobos. On your computer connect to the Drobo as admin. This makes the drobapps folder appear. Download the file openssh.tgz and save it in the /drobapps/ folder

page: http://www.droboports.com/app-repository/openssh/
file: https://github.com/droboports/openssh/releases/download/v6.8-fs/openssh.tgz

Install Dropbear 

You need to do this on both drobos. Connect to the Drobo Dashboard app again and goto Droboapps. Install dropbear and start it.

use putty, or on a mac just ssh to the drobo with the admin username and password
$ ssh admin@192.168.1.100

page: http://www.droboports.com/app-repository/openssh/
file: https://github.com/droboports/openssh/releases/download/v6.8-fs/openssh.tgz

Switch to the root user
# sudo su -
run this to unpack and install (this is basically how you install any drobo ported app)
# /usr/bin/DroboApps.sh install

Configure OpenSSH on port 2222

Dropbear runs on port 22 so i avoid using OpenSSH on that port by changing the config using vi

# vi /mnt/DroboFS/Shares/DroboApps/openssh/etc/sshd_config

Press i to start insert mode. edit this this line
#Port 22
to this
Port 2222

Press :
then wq (write quit)

All done, goto the drobo dashboard and stop start the OpenSSH app (down the bottom of all the apps).

From here on, connecting via putty or SSH is done on port 2222 to take advantage of OpenSSH. So from your computer you can do
$ ssh -p 2222 admin@192.168.1.x

Create .ssh folder

You need to do this on both drobos.

I am not sure but i think dropbear creates this folder, and this file
# /mnt/DroboFS/home/admin/.ssh/
# /mnt/DroboFS/home/admin/.ssh/authorized_keys

If not create them, and set correct permissions. I 'think' this should do it...
# mkdir /mnt/DroboFS/home/admin/.ssh/
# touch /mnt/DroboFS/home/admin/.ssh/authorized_keys
# chmod 700 -R /mnt/DroboFS/home/admin/.ssh/

Before we move on...

Why use OpenSSH at all? Well dropbear is super slow is why... i believe this is due to windowsizes
https://lab.nexedi.com/nexedi/slapos/commit/605e564b8f3008cdb48dbc6e82c956029469ff02

Create SSH keys

You need to create the keys on drobo2, and copy the public key to drobo1. This is because drobo2 will be running  the sync script and needs to connect to drobo1 and pull the files. You can do it in reverse but i like the second drobo doing all the work. Also, i feel safer messing around on my second drobo..

Now lets make some keys yeah? On Drobo2... cd to the openssh app and list files

Drobo5N2:~ # cd /mnt/DroboFS/Shares/DroboApps/openssh/bin
Drobo5N2:~/Shares/DroboApps/openssh/bin # ls
scp*         sftp*        slogin@      ssh*         ssh-add*     ssh-agent*   ssh-keygen*  ssh-keyscan*

Run this and accept the defaults, but please read the bloody prompts at least and make sure the files end up as per below

# ./ssh-keygen

Save files to

/mnt/DroboFS/home/admin/.ssh/id_rsa
/mnt/DroboFS/home/admin/.ssh/id_rsa.pub

Cat the pub key and copy that garbage to clipboard

# cat /mnt/DroboFS/home/admin/.ssh/id_rsa.pub

from your computer logon to drobo1 via OpenSSH on port 2222
$ ssh -p 2222 admin@192.168.1.x
Create the directories you did before.
edit the authorized_keys file with vi, and paste the SSH key into the file and :wp to save

NEATO!

GO BACK TO DROBO2

Login with no password

Drobo2 is going to do all the copying so go back to drobo2. Unfortunately, the ssh executable is not in the 'PATH' so you have to run it with the full path. I tried adding it to the path but it gets magically overwritten.

Run this with the PRIVATE key (id_rsa) and you should log straight into drobo1 with no password

# /mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -i /mnt/DroboFS/home/admin/.ssh/id_rsa admin@192.168.1.x

exit drobo1 and re-run that command as the root user on drobo2. Accept the prompt to add this connection to known hosts. If you don't do this step, when you run CRON later as root it'll fail. 
# sudo su -
# /mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -i /mnt/DroboFS/home/admin/.ssh/id_rsa admin@192.168.1.x


Did it connect with no pwd - it should have, If it didn't work... thats sucks. Maybe your SSH keys might not have right permissions. Look that shiz up. chmod some shit

Now for the SSH / RSYNC part

The basic copying process works by creating an tunnel, then contacting rsync on the destination, comparing the files on both ends, creating a list, and transfering the files.

First off lets talk about how rsync works for for me:

The trick to do this on drobo is to tell the destination where to find the rsync executable. You need to include this..

--rsync-path="/sbin/rsync"

For my setup i specifically include the folders i want and exclude the files / patterns i don't want copied

--files-from=/mnt/DroboFS/home/admin/include.txt: include these files. eg.
--exclude-from=/mnt/DroboFS/home/admin/exclude.txt exclude these

I exclude this MAC metadata crap in my file

*DS_Store*
._*
.AppleDouble

Then there are the switches to do the sync. once apon a time i would use -a, but that copies the permissions, so i just use -r

r: recursive
v: verbose
h: human readable
--progress: show the progress
--ignore-existing: only copy new stuff
--delete-after: when done, delete stuff on the target that is not on the source
--delete-excluded; also delete stuff i put in the excluded file on the source

Rsync stages the file copy on the destination when it copies. I like to dump the partial files in a completely different place for better visibility etc

--temp-dir=/mnt/DroboFS/Shares/Public/partial: copy temporary, partial files here

Lastly, log the copy process. We'll be running this as a CRON job so you need this or you won't know what going on.

--log-file=/mnt/DroboFS/home/admin/data.log

---ARE YA WITH ME---?

Now for the full SSH command

So how does the SSH part work in more detail? Basically we

execute rsync on the target machine over SSH on port 2222 using the PRIVATE key (id_rsa)
...then all the rsync commands
...then the destination
.. then the source (and folders from the include file)

rsync -e "/mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -p 2222 -i /mnt/DroboFS/home/admin/.ssh/id_rsa"

--rsync-path="/sbin/rsync" -rvh --temp-dir=/mnt/DroboFS/Shares/Public/partial --progress --ignore-existing --delete-after --delete-excluded --files-from=/mnt/DroboFS/home/admin/include.txt --log-file=/mnt/DroboFS/home/admin/data.log --exclude-from=/mnt/DroboFS/home/admin/exclude.txt

admin@192.168.1.202:/mnt/DroboFS/Shares/Data/

/mnt/DroboFS/Shares/Data/

Put it all together


rsync -e "/mnt/DroboFS/Shares/DroboApps/openssh/bin/ssh -p 2222 -i /mnt/DroboFS/home/admin/.ssh/id_rsa2" --rsync-path="/sbin/rsync" -rvh --temp-dir=/mnt/DroboFS/Shares/Public/partial --progress --ignore-existing --delete-after --delete-excluded --files-from=/mnt/DroboFS/home/admin/include.txt --log-file=/mnt/DroboFS/home/admin/data.log --exclude-from=/mnt/DroboFS/home/admin/exclude.txt admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/

NB: Keep in mind that the folders i am copying are in my includes text file so really this command will be doing this..

admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/folder1
admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/folder2
admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/folder3
admin@192.168.1.202:/mnt/DroboFS/Shares/Data/ /mnt/DroboFS/Shares/Data/folder4 ...

AWWW YISS

Now lets put that command in a script

Simply create a bash script with vi, put your command in it, save it, then chmod it with +x and run it.

# vi data_backup.sh
# /bin/bash
# your command

Save, then
# chmod +x data_backup.sh
Execute ./data_backup.sh

You should see rsync calculate the file list then start copying like mad. Make sure this works as the root user before you attempt doing a cron job.

AND FINALLY CRON ftw

Setup crontabs

Drobo uses busybox for most of its linux style commands including crond.

https://busybox.net/downloads/BusyBox.html

To get crond working you need to sudo to root, create 'some' of the default path, then create a crontab, and start crond


Drobo5N2:~ $ crond --help
BusyBox v1.23.2 (2015-11-06 13:39:54 PST) multi-call binary.

Usage: crond -fbS -l N -d N -L LOGFILE -c DIR

-f Foreground
-b Background (default)
-S Log to syslog (default)
-l N Set log level. Most verbose:0, default:8
-d N Set log level, log to stderr
-L FILE Log to FILE
-c DIR Cron dir. Default:/var/spool/cron/crontabs

# sudo su -
# mkdir -p /var/spool/cron
# /bin/busybox crond

The its basic crontab stuff, which runs as root so thats cool. Run the command below to create a job list and use cron numbering to schedule a script to run the SSH / RSYNC command when you want it run.

# crontab -e

Keep in mind the drobo uses PDT time so set your job accordingly

Drobo5N2:~ # date

Fri Aug 11 05:00:54 PDT 2017

To view the crontab you created.. as root

Drobo5N2:~ # crontab -l
51 16 * * 6 /mnt/DroboFS/home/admin/data_backup.sh*

Drobo5N2:~ # 

Testing

When getting started, you can put a -n in your rsync switches to do a dry run copy
Assuming crond kicks off, you can tail the log file and see if copying has started
You can also do ps aux | grep rsync on either drobo and watch the process kick off if its working

Tips

You can do a -I if you want to just overwrite whatever is on the target. Sometimes i find this handy.
Kill the crond command if its not working nicely on first go

Last but not least here are some references i used for this magic.

If this page has helped you .. well sweet. You r welcome.

-jules

#################################################################################

References

busybox tools and  commands
https://busybox.net/downloads/BusyBox.html

Dropbear
https://linux.die.net/man/8/dropbear

Drobo dropbear windowsize sucks the dick - windowsizes
https://lab.nexedi.com/nexedi/slapos/commit/605e564b8f3008cdb48dbc6e82c956029469ff02

i tried using a drobo ported version of rsync at some stage but gave up
https://sites.google.com/a/droboports.com/www/app-repository/rsync-3-0-9

dropbear man page
https://linux.die.net/man/8/dropbear

dropbears equivalent to 'ssh' (dbclilent)
https://manpages.debian.org/jessie/dropbear/dbclient.1.en.html

basic rsync operation
http://www.linuxquestions.org/linux/answers/Networking/Using_rsync_to_mirror_data_between_servers

rsync with includes
https://stackoverflow.com/questions/16647476/how-to-rsync-only-a-specific-list-of-files

cron scheduling
https://en.wikipedia.org/wiki/Cron

busybox tools and  commands
https://busybox.net/downloads/BusyBox.html

drobo pinhole reset
http://www.drobospace.com/forums/printthread.php?tid=77106

1 comment: