In part 1 of this blog post we looked at different storage media, and their pros and cons. In this part I will be focusing on surviving loss of one or multiple storage media, and how to build out a backup strategy that will work and last. Last but not least… Be affordable.
Offline versus online media: Detecting failures
With offline media I mean any media you record data too, and then store them somewhere without having direct digital access to them. DVDs and tapes are prime examples, but also an external hard disk that lies in a vault is an offline storage medium. Online storage media are “active” and you can constantly access them. A good example of online storage is an internal hard disk, or a NAS.
Both types have pros and cons. Offline media have some pros. You can put them in a vault which could protect your data from floods and fires. In general, they are also pretty cheap for a lot of storage. But the cons on offline media make them of little use for backups. For starters, because the media is offline, there is no way to determine if a media is decomposing. So if the media fails at some point in time, you will not be around to detect this and copy a second copy to “reprotect” your data. Put simply: When you get your DVD from your vault or cabinet, it is fingers crossed if it is still readable. On top of this, burned media are very easily damaged in general. A single scratch on such a disc can ruin the data on it.
Online media is almost the other way around from offline media. What is so great about online media, is that you can have an automated constant check running in the background that will detect failures almost as they occur. Even better: If the online media has multiple copies (let’s assume two), and the system detects one of them failed, it can directly start to reprotect automatically by copying the data from the surviving copy to something new. This is an excellent first line of defense in making sure your data survives the death of a storage media.
Online media also have downsides: In the optimal situation, they will be running 24/7. This will cost you power, and the media will probably live shorter as compared to the same media being used offline.
So where are we going with this: Hard disks as a firm basis for storing backups
You may have deducted it already: I am aiming on using online media for making backups. At this moment in time, hard disks are considered the best media for this. They are big, affordable relatively fast. Also, they are pretty reliable.
Hard disks WILL fail at some point in time (photo taken with a Canon EF-S 15-85 @85mm with 12mm tubing)
Of course, a single hard disk would not protect us from loosing data; a hard disk WILL die at some point in time. So how do we go about this? A very simple solution could be, to have two hard disks in a computer, and simply store my data on both drives. When one of the drives fail, I still have my data on the other drive. I could go out and buy a new drive, build it into the computer, and copy the data over from the remaining drive. At that point I’d be reprotected (having multiple copies again).
This is how the idea of RAID (Redundant Array of Independent Discs) evolved. A RAID controller is a disk controller that connects multiple hard disks. The storage is represented as one big piece of storage, but underlying there are multiple physical hard disks that will carry the data together and in almost all scenarios protect against failure of individual hard disks. The RAID controller will detect broken hard disks, pop up a question to replace the broken disk, and it will then proceed with a “rebuild”, which means it reconstructs the data out of the remaining disks until reprotection is again reached (eg the rebuild is completed). From that time on the system is prepared for another disk failure.
As this idea around RAID grew and grew, different RAID levels were introduced. I will describe some RAID levels that matter in this discussion, and try to keep it “human readable” (and for the sake of this, it may not be 100% accurate):
| RAID 1 | Also called “mirror”. In RAID 1, there are two drives involved. The RAID controller simply makes sure both drives carry the same data. When a failed drive is replaced, the controller simply copies the remaining disk over to the replaced disk and reprotection is complete. Note that the net capacity is only half of the sum of capacity of both drives |
| RAID 10 | Also called a “stripe of mirrors”. As RAID 1 is a mirror of two drives, RAID10 spans multiple RAID 1 mirrors together. Data is divided over all included mirrors. If a drive fails, the rebuild process is comparable to the RAID 1 rebuild. Note that in a RAID 10 config you could in theory loose half of your drives and still have your data. Downside is that the net capacity of RAID 10 is only 50% of the total capacity of the drives. |
| RAID 5 | To counter the 50% net capacity of RAID 10, RAID 5 was introduced. In raid five, the data is divided over all disks in the RAID 5 set, and for every “stripe” of data laid over all drives, one of the drives does not hold data, but holds parity information. In this setup, you can loose one single drive without impact, and capacity-wise you loose one drive out of the total set (and not 50% of total capacity). |
| RAID 6 | You can imagine in RAID 5, that things get “interesting” when you loose a drive – if another drive fails during a rebuild, you still have a serious problem. This is why RAID 6 came about. RAID 6 features dual parity, which means two drives are used for parity (impacting CPU power required and you loose capacity of two drives in the RAID set). This is a very secure but more complex RAID type over RAID5. Even if you loose a second drive in RAID6, the RAID controller can still rebuild the data for you. |
For more information on RAID types, please visit my other blog: Throughput part 2: RAID types and segment sizes.
RAID made easy: NAS
You could buy a RAID card in your PC. Most PCs even have RAID-enabled SATA controllers on the mainboard nowadays. It is still complex to put together for less technical people though. That is where a NAS comes in. A NAS (Network Attached Storage) is basically an external hard disk (often multiple hard disks) that do not have a USB or E-SATA interface, but a network interface. It is basically a tiny file server, accessible over the network.
Most NAS devices have multiple hard disks inside, and yes – a RAID controller. So a NAS is a device that solves your local hard disk failure troubles by including RAID. It also solves the problem of the need of your computer to be “on” 24/7. Finally it solves the problem of complexity. Most NAS systems you buy, hook up, and within 5 minutes you are good to go.
An example of a NAS system is the Iomega NASses. I own one of the simplest of them:
Iomega IX2-200 two-drive NAS system
This is a two drive NAS. Small, efficient, affordable. Not the fastest one, but we aren’t after extreme performance anyway. This NAS is able to either SPAN or MIRROR its drives. When a drive fails, I get an email sent from the NAS. After I replace the broken drive with a new one, the NAS rebuilds automatically.
Beware though: A disk may die, but what happens if the NAS itself dies? You need to make sure that you can migrate the disks to a new NAS, and that you then will be able to have your data back. In RAID1, you probably could mount one of the disks to a PC and recover your data. But as soon as the NAS is in a RAID10, 5 or 6 config recovering becomes hard without the NAS present. In that case you need to be able to move your drives to a new NAS to recover.
Connecting to NASses is easy; most if not all support CIFS and NFS (and often much much more), which allows you to very easily connect to them over the network to Windows, Mac or Linux systems. The NAS will present itself as one (or multiple) network drive(s), and accessing these drives is just like accessing a device local to the PC (only slower depending on your home network).
Protecting against “site failure”
Now that we have tackled the problem of loosing a single storage medium, it is time to look at loosing more than that. What happens if your house burns down? What happens if there is a flood, or a severe lightning strike? These are the situations that were very hard to cope with in the analog film world; you only had the one physical copy of the negative.
In the digital age, everything is digital. So it would not be too hard to copy off your data to some other location, right? One of the most simple and often used strategy to cope with this, is using external hard drives. You could actually build a very firm strategy by doing this:
- Keep your negatives on your NAS (this will protect from single disk failures);
- On a regular basis, copy your NAS data over to an external hard disk;
- Store the external hard disk offsite (at work, at family…).
Using this strategy is more secure than you might think; yes the external drive can break / be stolen / drop in the sink. But you would still have your data on your NAS. Same goes for a single drive failure in your NAS; it can cope with that situation. If your house burns down, your NAS is gone. But in this case you still have most of your work on the external drive that hopefully lies at your work or your family.
Note the words “most of your work” and “hopefully” in the previous paragraph. This strategy works quite well, but you have to make sure you always have your external drives (or at least one) o a different location. In practice, you’d have multiple external drives that you rotate. Getting more complex.
A second issue is, how often do you copy to an external drive. If you copy once a week, you can potentially loose a week’s data. These risks made people think if you can do things even smarter. And of course… You can.
Cloud Storage
The answer lies in the last type of storage medium discussed in part 1 of this blog entry: Cloud storage. Cloud storage if often, if not always, based on the same idea as your home NAS, only way bigger (and I’m talking thousands of disks). You can rent a small piece of these hard disks.
Cloud storage has several advantages you cannot beat: They have RAID protection at a huge scale (which makes it VERY affordable), and they have cool client software that makes up- and downloading of data really easy. Even greater: In general you’ll be uploading to a location very different from your physical location.
Cloud storage solves a lot of problems all in one swift stroke:
- Cloud Storage is remote, so you have a second physical location to protect you from floods, fires etc;
- Cloud Storage has smart clients, that detect changes in your data and automatically start syncing to the remote location;
- Cloud storage uses RAID so protects against single disk failures at their end as well;
- Cloud storage sometimes even replicate the data between multiple datacenters effectively giving you a third site protection.
These points you cannot dismiss. Prices are very decent as well. I found a good comparison here: onlinebackupsreview.com. It is very easy to find the one provider best for you.
I personally went with crashplan.com. The reasons I choose for CrashPlan:
- Crashplan offers unlimited storage;
- Great client software with many buttons and dials (which I looove);
- The client software enables you to backup to friends on the internet as well for free;
- Security of your data can be very high if you choose to generate your own 448-bit encryption key;
- Very price competitive.
For you other things might matter which will make you choose differently. I now have around 550GBytes of data into the cloud, and the initial sync took over three weeks. Not really a problem; now that the initial sync is done it just syncs any new negatives I add to my negatives folder. So usually after a shoot, it starts uploading within 15 minutes from the time I add negatives, and takes 1-4 hours to complete the sync.
So I store my negatives locally on a NAS with RAID protection, and within 4 hours I have my added data safe on an external, very remote location, also RAID protected. For me, this is the perfect setup, and I think it will be for most people.
The perfect solution (for you?)
So for me, combining NAS and Cloud storage offers the perfect solution. When looking at my total workflow, it goes something like this:
- I shoot my pictures. Data is now at risk, as it is only recorded on a single CF card;
- Inbetween shoots, I change CF cards and backup the other card to a laptop (local protection);
- After the shoot, I sift through the negatives and perform post processing (I still have the CF cards as local backup);
- After the post processing, I upload all remaining data to my NAS. I usually try to not erase my CF cards just yet;
- Right after uploading to my NAS, the client software automatically starts its upload into the cloud. After 1-4 hours (depending on the amount of data), I have both local and remote RAID protected copies;
- Now I erase my CF cards for reuse.
As you can see, this is pretty solid way of working. It is very hard to think of a scenario where I’d loose my data. The only real risk lies in the first few steps, where I have only local backups. A fire could actually make me loose the last shoot I did. Protecting against that is hard. I could make sure the CF cards are not in the house as I process the negatives on my laptop. That would make it really rock solid.
Some scenarios and the impact
This is the fun part… Thinking of really bad things and see if it causes me to loose data:
| House Fire | Would cause me to loose my NAS. Data would remain in the Cloud. I’d need to download all my data back from the cloud, or I could have it shipped on an external hard drive if I wanted to (!!). |
| Cloud provider goes bankrupt | Often seen as a big risk. But as I have my data at home as well, I’d still have my data. I just need to go to another cloud provider, and upload again to reprotect to a remote site. |
| Cloud Provider gets hacked | Your data could be stolen. I do not really worry about this, my photography is way too hard to sell to make it worthwhile. If you are really worried about this, then generate your own encryption key. Just make sure you do not loose your key: No one will be able to recover your data! |
| Cloud provider looses your data | See bankruptcy. |
| A NAS disk dies | No problem, The NAS mails me about the problem. I exchange the disk, and the NAS rebuilds. No data loss. Even if the rebuild fails (eg disk 2 fails as well) – yes I loose my NAS data. But the Cloud provider still has my data! |
The only real risk is a double failure, like if my Cloud provider looses my data and my house burns down. True, this is a risk. But what are the odds? If you are REALLY paranoid, get two different cloud storage providers, and have them protect the same data. In this case, you’d have your data in three places: At two cloud storage providers, and at home. That should settle the most paranoid photographer I hope!
Finally, I would never delete the local data or have it unprotected locally. Yes you could have two cloud providers and delete your data locally (or live without a NAS). But what if you need that one picture and you forgot which one it was? If you have your data locally, searching is quite easy. If your data is only at the cloud provider, you’d need to download the data (and probably lots of it) to search through and find that photo.
If you only have a local disk with your work, a single disk failure would cause you to have to download ALL your data again; not funny if you need to download over 500GB of data.
LinkedIn
Twitter
Email
Flickr
[…] Update: Part 2 of this series is available here. […]