My wiki | Site / Backup

For long term storage, we are currently stripping neutral dynamics trajectories of water (a script for this can be found in /mindstore/home33ext/kscopino/BIN/make_storage_strip.py), manually deleting other versions of the nd trajectories (usually mdcrd_nd_xxx), then transferring the entire project folder to rstore6. The source and destination paths can be changed in order to transfer between other sets of hosts (including helmholtz and gibbs), but the example below involves a transfer when logged into mstore0 and transferring over to rstore6.

The first time you do this, you will need to set up a passwordless login in order to avoid having to enter a password during this process. First ssh to rstore6 (ex. ssh kscopino@rstore6.wesleyan.edu), navigate to /home and generate ssh keys (ssh-keygen -t rsa). When prompted to set a password, leave it blank. Next, copy the line from mstore0:~/.ssh/id_rsa.pub to a file called authorized_keys on rstore6:~/.ssh/

NOTE: Henk set up my (Kristen's) account, so I did not actually go through this process myself. If the information in the paragraph above does not work for you, please reach out to me and I will update it after troubleshooting.

Once you are able to ssh into rstore6, note that our data is stored in /data/1/weirlab/username. Before transferring data, compare the amount of space available to the size of the directory you will be transferring (df -h /data/1/weirlab from rstore6 compared to du -h /mindstore/home33ext/username/directoryname on mstore0).

To transfer files, ssh to mstore0 (or wherever the data is stored), then use the rsync command:

   rsync -vac --dry-run --whole-file --bwlimit=5120 --stats /mindstore/home33ext/username/directoryname rstore6.wesleyan.edu:/data/1/weirlab/username

NOTE 1: --dry-run is used to check that your syntax and pointers to directories is correct. Omit this to actually transfer the data.

NOTE 2: BE VERY CAREFUL TO OMIT BACKSLASHES AT THE END OF THE DIRECTORYNAME. Retaining slashes implies "sync folders" mode rather than "copy data" mode. We haven't tested if syncing folders actually deletes data if it exists in the target but not the source (i.e. data that you've moved to long-term storage but deleted from mstore0, but we don't want to find out!).

NOTE 3: rsync makes a copy of the data in the new location and does not automatically delete the data from the original location, so you will need to do this manually.