Duplicity is a Python utility that backups your stuff, encrypts it and allows you to upload them away using as little bandwidth as possible using the rsync algorithm.
Cost effectiveness and security aren’t things that especially come to mind when thinking about sending your backups to the cloud.
However, as the cost of mass storage and bandwidth goes down, so does the pricing of cloud Object Storage. As far as security is concerned, your backups are encrypted locally by Duplicity before being uploaded. This ensures that no one without your encryption key can read the files.
Duplicity made Openstack Swift a native backend in version 0.6.22 released back a year ago, in august 2013.
In this post, I’ll go over how you can install Duplicity on Ubuntu, back up a directory and send it to an Openstack Swift object storage service.
Installing Duplicity
If you’re using Ubuntu 14.04 (Trusty), Duplicity is provided by the Ubuntu repositories with a recent enough version: 0.6.23.
You can install it directly through apt:
apt-get install duplicity
For older distributions, the version of Duplicity provided by Ubuntu doesn’t yet have the Swift backend in it.
Thankfully, Duplicity has a PPA that provides 0.6.23 packages for Lucid, Precise, Quantal and Saucy.
add-apt-repository ppa:duplicity-team/ppa
apt-get update
apt-get install duplicity
Generating the encryption key
Duplicity encrypts your backups with the help of an encryption key. This ensures that no one without the key will be able to read your backups.
To generate your key, use the following command:
gpg --gen-key
This still start prompting you for options. Usually the defaults are fine.
Do make sure to set a passphrase to further improve the security of your key. This makes it so if your key is ever obtained, it will not be usable unless the passphrase is also compromised.
gpg (GnuPG) 1.4.16; Copyright (C) 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: My Name
Email address: email@example.org
Comment:
You selected this USER-ID:
"My Name <email@example.org>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
......+++++
...............+++++
.......+++++
.......................+++++
gpg: key 528CAC31 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 2 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 2u
pub 4096R/528CAC31 2014-08-11
Key fingerprint = 0725 7C03 DFDE DCB5 805C 3EFF 503B 7B6A 528C AC31
uid My Name <email@example.org>
sub 4096R/DBD5888E 2014-08-11
Getting a Swift Object Storage Service
The Openstack foundation set up a nifty marketplace website where you can find Openstack service providers.
On it, you can:
- Filter by project (such as Nova or Swift)
- Filter by Location (pick a location near you for best throughput and latency)
- See the version of the services provided (Icehouse is the latest release at this time with Juno scheduled for October 2014)
If you get lost amidst the list of providers, feel free to ask me for advice.
Backing up a directory to Swift
Once you’ve installed Duplicity, have your encryption key ready and you have a Swift service, it’s really… really… easy.
Duplicity requires some environment variables to be set. I source this little bash script but feel free to do anything you’d like:
#!/bin/bash
# Swift credentials
export SWIFT_USERNAME=tenant:username
export SWIFT_PASSWORD=password
export SWIFT_AUTHURL=https://keystone_auth_host/v2.0
export SWIFT_AUTHVERSION=2
# Optional, your gpg passphrase passed from Duplicity to gpg
# Otherwise Duplicity will request the passphrase from stdin.
# export PASSPHRASE=password
A note about Keystone authentication
Keystone authentication requires SWIFT_AUTHVERSION to be ‘2’ and also adds a dependency to python-keystoneclient which is NOT included as a dependency to Duplicity. You’ll need to install it:
apt-get install python-keystoneclient
# OR
pip install python-keystoneclient
Once your credentials are in the environmental variables, you’re all set to use Duplicity. Here’s the syntax:
duplicity <dir> swift://<container>
A note about storing credentials in environmental variables
I don’t like the general idea of storing credentials in plain text, let alone as environmental variables but this is how Duplicity works apparently. Alternatively, you can also invoke Duplicity with variables inline, like so:
SWIFT_USERNAME=tenant:username [...] duplicity <dir> swift://<container>
Still, be careful about letting these sort of entries in your BASH history. If you have a clever way to send credentials to Duplicity and GPG in a secure way, please comment !
So, let’s pretend I want to backup /root to the root_backup container:
duplicity /root swift://root_backup
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase:
Retype passphrase to confirm:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1407817325.28 (Tue Aug 12 00:22:05 2014)
EndTime 1407817326.27 (Tue Aug 12 00:22:06 2014)
ElapsedTime 0.99 (0.99 seconds)
SourceFiles 244
SourceFileSize 11097577 (10.6 MB)
NewFiles 244
NewFileSize 11097577 (10.6 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 244
RawDeltaSize 10913257 (10.4 MB)
TotalDestinationSizeChange 8555954 (8.16 MB)
Errors 0
-------------------------------------------------
That’s it ! If I run it a second time, you’ll see that hardly any data is actually transferred:
duplicity /root swift://root_backup
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Aug 12 00:21:55 2014
GnuPG passphrase:
Retype passphrase to confirm:
--------------[ Backup Statistics ]--------------
StartTime 1407817392.57 (Tue Aug 12 00:23:12 2014)
EndTime 1407817392.68 (Tue Aug 12 00:23:12 2014)
ElapsedTime 0.11 (0.11 seconds)
SourceFiles 246
SourceFileSize 11272304 (10.8 MB)
NewFiles 10
NewFileSize 198608 (194 KB)
DeletedFiles 5
ChangedFiles 4
ChangedFileSize 26672 (26.0 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 19
RawDeltaSize 184262 (180 KB)
TotalDestinationSizeChange 182580 (178 KB)
Errors 0
-------------------------------------------------
Looking at the result
Using Swift Client, I can see the files Duplicity uploaded:
swift list --lh root_backup
175K 2014-08-12 04:22:07 duplicity-full-signatures.20140812T042155Z.sigtar.gpg
196 2014-08-12 04:22:08 duplicity-full.20140812T042155Z.manifest.gpg
8.2M 2014-08-12 04:22:07 duplicity-full.20140812T042155Z.vol1.difftar.gpg
248 2014-08-12 04:23:13 duplicity-inc.20140812T042155Z.to.20140812T042304Z.manifest.gpg
178K 2014-08-12 04:23:12 duplicity-inc.20140812T042155Z.to.20140812T042304Z.vol1.difftar.gpg
5.8K 2014-08-12 04:23:13 duplicity-new-signatures.20140812T042155Z.to.20140812T042304Z.sigtar.gpg
8.5M
Isn’t that just awesome ?
Have any tips or comments about Duplicity with Swift ? Please let me know !
Share this post