World Backup Day

Today is March 31, World Backup Day. http://www.worldbackupday.com/
This year, World Backup Day is also Easter. What do these days have in common? Resurrection. Think about it.

You’ve probably heard someone tell you to back up your home PC or your servers. If you’re like most people, you’ve ignored that advice – you’ll get to it one day. My intent with this post is to convince you to take action. I’ll write this for the home user, because honestly if you’re an IT person in charge of a company’s data, you’d better have this memorized already.

There are a few components absolutely essential to any backup plan. No, strike that – disaster recovery plan. Don’t just think of it as running backups. You are planning for disaster: imagine a fire or explosion at your house completely burns it down with no injuries. What do you do about your data? If it’s important, you need a plan:

1. Decide what data you absolutely must have immediately. This data is going to your local backup first.  Are you in the middle of doing your taxes, negotiating a contract, or working on someone else’s business needs?  If your PC dies, will you know where your very busy family needs to be in the next few days?
2. Decide how long you can be without the other data. Hours? Days?  This will determine your remote strategy.  You must have a remote backup of some kind.
3. Decide how much data you can afford to lose. Can you lose the last few minutes, hours or days? This will determine your backup frequency and type.
   In business we have a term “point-in-time recovery” : in order to keep international systems running, we spend a LOT of money and time to make sure we don’t lose even minutes.  You probably don’t need a million-dollar solution but you do need a plan.
4. This is the toughest decision of all: what service or technology to use.  For home users, you have hundreds of options from the native OS tools to third party elaborate solutions.  There are several very good options, and some are better to your particular needs.  To save time, I’ll recommend a short list:
   • Acronis True Image Home (I use this for local full-image backups)
   • CrashPlan  (I use this one)
   • BackBlaze
   • Mozy  (I have used this for years.  I’m in the process of moving everything to CrashPlan)
   • DropBox (not 100% secure; the DB employees can read your data)
   • USB drive – Copy only a few files to a USB drive, in folders with the current date.  (flash disk or solid state, not a spindle) and – trust me on this – drive or mail it to another state.  (Also not secure unless military-grade encrypted with a key not tied to your soon-to-be-dead PC)

   Options I do not recommend:

   • Windows native backup tools.  Sure, they work, but the amount of effort to get you working again is too great.
   • Email
   • Anything that requires you to remember to take action; an automated process is the best.  You will forget or say, “it can wait.”  Or you’ll need a version of a file two backups ago and you’ve overwritten it already.  Or worse, you had a virus, and the backup is corrupted.  Been there.
5. Implement your solution.  Each of the above has a dozen page manual on how to implement it.  What’s second-most important is that you do.  Because this part is rather involved, I’ll detail it below.
6. Test your backups.  This is the most important step of all.  On a regular basis, pretend you’ve had a disaster: human error, fire, flood, zombie apocalypse, whatever.  Restore a few files to be sure you can.  This also will let you know if something – like a virus – is corrupting your backups.

Implementing your solution

As you choose the technical portions of your disaster recovery solution, you will need:

• Local backup – a hard disk with a few terabytes free space, or a few times larger than the data you want to back up.  You’ll want room to keep multiple versions for a history.
  Ideally, every day or week, take a full image of your primary hard disk through Acronis or something else that does such things.  I use a monthly full image with daily updates.  So, for a 200GB boot drive I’ll have a 200GB file and 29-30 files of a couple GB each.
  This way, if my primary drive dies I can run to Best Buy, get a new disk, get back online in about an hour.  I use my home PC for database consulting; I don’t have the luxury of being offline for a day.  The average home user can wait longer, but if you’re calling your family geek to “come fix my computer,” trust me, they want that chore to end as efficiently as possible.  Restoring mom’s PC is not play time, it’s more like rebuilding a porch.
• Remote backup – an online service or other method to get your data to another state, just in case yours is hit by apocalyptic weather.  I’m serious, go out of state; major storms in Alabama can strafe several major cities, so it’s possible for a tornado to hit my house in Mobile and another one from the same storm wipe out a relative’s house in Birmingham.
  I have the CrashPlan (“cloud based”) service read my most important data files and transmit them to their servers, using a long encryption key I chose.  Nobody, not even the NSA, can read my tax returns.  I can go to my work PC or a friend’s house and restore a few files over the web.  If I want the whole thing immediately, I can pay $100 or so and get a USB drive overnighted to me.   Acronis seems to now have a similar cloud service, but I’ve already paid for a year of CrashPlan so I have time to check that one out later.
• Strategy
  How quickly do you need your data back?  If you need to be back online really fast, you need to make more backups which takes more space – usually daily full backups, hourly log/updates, to a local disk with a regular transfer offsite.  The amount of time spent running the backup is usually inversely proportional to the amount of restore time.  So, the fast monthly full-backup with hourly change log, will take a long time to restore today’s data.
• Retention policy
  How much data do you want to keep, measured in number of changes?  If you changed a document a month ago and now want to see what it looked like then you must keep more than a month of history.  Keep in mind, if you changed it ten times between two backups, you can only see the two snapshots seen at the exact time of the backups, not the ten changes.  That loops back to the strategy above.
• Schedule
  When do you want to run your backups?  I leave my home PC online 24×7, and run the more intensive backups at night.  What’s important here is that backups don’t disrupt your work.
• A healthy dose of paranoia
  What happens when the restore fails?  Do you want or need a second backup disk, a second service?  What happens if someone breaks in to your house and steals your PC?    Thieves will take your backup disk too – more reason for a remote backup.

Final thoughts

Why is this so important?  In my 25+ years as a developer and DBA, I have seen and experienced disaster.  It is always painful to hear a company rep say, “It looks like our backups stopped working three years ago.”  I’ve seen entire companies just shut down because of it. I’ve lost gigabytes of my own data because I just didn’t make a DR plan a priority.  Lesson learned: Ask the what-if questions.

In my younger days, I ran the “computer department” for my father’s company – two PCs on a network, hosting an automated payroll system that I designed.  I encrypted the database because I could, but I forgot one detail.  I had backups from PC 1 going to PC 2, because we didn’t have the budget for a tape drive.  When PC 1′s hard disk failed on a Wednesday, I went to PC 2 to restore the payroll database.  Wouldn’t you know it, I forgot to back up the master encryption key.  The backup was completely useless, and obviously untested.  Payday was Friday, and in about 38 hours 200 hard-working native Guatemalan factory workers really wouldn’t care about encryption or whatever tech mumbo jumbo excuse I could throw their way, so the head accountant manually retyped an entire 13 days data – over a 37 hour stretch, finishing 4pm Friday.  None of them knew how close we came to having a riot instead of payday.  Lesson learned: Test your backups … by restoring them.

Also test the restore time; in a business you may not be able to handle a multiple-hour delay. If your backup strategy is a one-time full backup with four years of change log, it’s going to take days to restore recent data – the restore application has to go through every change to determine what’s the newest.  In a 24×7 business with 99.999% uptime requirement, that’s apocalyptic.  That’s why businesses that hold years of data like health or financial information spend six or seven figures on disaster recovery plans – in most cases, data can be restored before most customers even notice.  And if you’re a regular home user, you’d rather be doing something else with your scarce time, wouldn’t you?

Now, the majority of the above is worded for a typical person with one or two home PCs, but the concepts apply to business as well.  If you’re in charge of databases or systems for a business, your entire stream of income depends on the ability to recover from disaster.  Instead of worrying about how much a proper backup system will cost, ask yourself: What will it cost if we lose a week or a month of data?  What if the CEO loses his address book [1]?  Or if someone loses the source code to this 10-year-old application that controls half of the income [2]?  What if we lose the entire billing history (including unpaid receivables) for our 20,000 customers [3]?  These are actual scenarios I’ve seen where people did not have backups.  In case 1, a $3,000 emergency disk repair recovered 65% of the data, and some more from painstaking manual reconstruction.  In case 2, it cost $200,000 in developer salaries to rewrite the application.  In case 3, I never heard from the company again. They were completely dependent on the computing solution, so it had to be rough.

What is a backup disaster recovery plan worth to you now?  Any more or less than before you read this post?  Let me know.

Anatomy of a Solid State Disk Failure

Solid state drives can fail just like anything can. I’ve been told that they fail gracefully and I’ll have enough time to rescue my files.  Is that true?  Read on.

For a couple weeks my home PC’s SSD (a 60GB OC-Z Agility 1 I bought two years ago on sale) has been reporting errors when I do a disk check – just one or two until this week, when hundreds of files and other artifacts have been lost – no idea what, but Windows dutifully counted them out.  And recently Windows started showing signs of flaking out on me – for example, Event Viewer won’t load, but I could read the event files from another program.  After running SFC (System File Checker), I confirmed that some system files are corrupted… including some used by Event Viewer.  Apparently, when the drive wrote data, it didn’t verify, fail the write and use another sector like I had believed – maybe only enterprise class drives do that; this one wrote the data and continued on just like a consumer class magnetic drive would.  Fortunately, I bought my wife a shiny new 120GB SSD for Christmas, and it sat on a table for over three months, waiting for me to prioritize it.  Last night, it became mine.  I created an image backup of my 60GB SSD that I bought a couple years back for a boot disk.  I used Acronis True Image Home 2012, because it’s been recommended to me a few times.

I’m impressed – Acronis finished the backup in about 15 minutes, mostly due to the external drive’s USB 2.0 interface speed being maxed out.  It even compressed by 50%.  To see how fast it CAN get, I copied the file to my super-fast RAID drive, made a boot CD, installed the new SSD (an OCZ Agility3 120GB, bought on NewEgg for about $110) and booted directly into Acronis.  It let me choose the backup file from the external drive or internal, so I chose internal for speed.  Formatting, validation and restore completed in 8 minutes, sustaining writes at about 200MB/sec which barely challenged the new drive.  A later file copy test showed the same speed and the drive was only about 10% ‘busy’ when the other one would be 100%.  So I’m pretty happy with the new drive and the backup tool.

So, how did I know the disk was failing, and what does it look like when a 2010-era SSD fails?

Symptom 1: Changes don’t seem to save.  Programs (including parts of Windows) crash randomly, and say that a file is corrupt.  If you see that, scan your disk – but be aware, if you tell it to fix, it will amputate to save the patient.  Windows 7 recovers files, but I have no idea where my recovered files went, or which ones were lost.  If you tell it to scan for bad sectors, watch the screen constantly -  “33 bad clusters found/marked” flashed by and the machine rebooted.  I can’t see any report, possibly because the event log files are corrupted.  As soon as ou see bad clusters, run, don’t walk, and get a replacement disk.  Every disk write operation could be corrupting files.  Windows 7 is much more resilient than XP, but nothing’s immune.

Symptom 2: If you run SFC /VERIFYONLY or SFC /SCANNOW at an admin command prompt, it may tell you system files are corrupted.  This seems odd because system files don’t change, and SSDs are supposed to fail only on writes.  But remember the updates?  Yes, Windows is constantly updating OS files, recompiling the .Net framework and updating Windows Search indexes – which on an SSD uses new sectors, some of which may be bad.  So in theory I could install the old drive as read-only and it won’t give more bad sectors.  Anyone want to place bets?

I’ve encountered spinning drive failures before, and in almost all cases it was a controller or armature type failure.  For the low, low price of $3,000 I could get the data restored by a rescue service – I prefer prevention, thanks.  I’ve done file backups for years, but if I ever lost my boot disk, I’d lose hundreds of hours to reinstalling apps.  I think I’ll be keeping OS image backups from now on, and I think Acronis is doing a good job – I’ll have to test the incremental backup, but seriously, an sector-by-sector image in under an hour, without a reboot?  That’s worth money.  I have no vested interest in Acronis, so if you know of a similar product that does well, please mention it in the comments.

So, would I rely on an SSD for my important stuff? Yes, with backups.  Because, everyone should have backups anyway.  I think most SSD media failures would show warning signs early, so a regular full ‘checkdisk’ scan is a good idea for home users.   The speed boost is so incredible compared to a spinning disk, that the ‘new’ technology (really, it’s not that new) is worth the unknowns.  I absolutely would use enterprise-class SSDs in a production environment, subject to cost/benefit analysis, as long as I had them set up as RAID10 or RAID5 (as appropriate) and a few spare drives on hand (which is what enterprise IT should do anyway, right?).   And, your backup tool should be easy enough to use that you can do a restore test on a whim or on a schedule.  That goes for any type of backup.