I've had my share of customers who won't make backups. I've had customers who have left the same media in the backup device for years on end.. sometimes those even end up with happy endings that the customer definitely did not deserve.
I've had customers whose backups failed for months on end but nobody every looked at the printed report or email to know that.
I've had customers mis-label backups, overwrite critical full backups with unimportant files, expose media to strong magnetic fields, misplace media.. I've probably seen just about everything, including having too much backup, but this one leaves me slack jawed:
I was called yesterday morning to go see a customer I had last seen sometime in 2001. They weren't my direct customer; I had helped a Synchronics dealer install a small Linux server for them. I remember having to go back once for some malfunctioning print servers, but that was it.. never heard from them again.
Flash forward seven years. The original Synchronics guy has been replaced by another and the customer has been chugging along. Suddenly on a Thursday night he gets error messages on the screen, system reboots and is stuck single user. Customer calls his Synchronics guy, he calls his Linux guy, who has a miserable cold and tells him to call me. He does, and I agree to pop in at 7:30 AM to see what I can see.
As I expected, it wanted a manual fsck. I started one, but immediately ran into multiple sector read errors. Uh-oh, that's not good. I pulled the covers off to see if maybe cables were loose or if the drive was hot.. nope, nothing like that. Just a failed drive.
But hey, no big whoop because we have DVD-RAM backup. In fact, the stack of Monday through Saturday labeled DVD's was right there on the desk. Even better, there was an extra, completely unconnected drive in the machine. My vague memory was that the original intent was to have mirrored drives but we had run into some silly problem, not mirrored, and just left the drive there. Fantastic, we should have this puppy purring along in just a jiffy..
But the customer was giving me a strange look. I swear it was the "I'm the one who tried to get a cookie and spilled everything on the floor" look you'd get from a five year old.
"I haven't done backups in a while", he said sheepishly.
I'm sure I did a cartoon character double-take. Microlite Edge runs from cron, all he had to do was put in a DVD before leaving for the night and take it out in the morning. Not exactly difficult.
"How long?", I asked. Longish pause, and he said "A while. Years." and shrugged his shoulders helplessly. I thought to myself that it's a good thing he owns the business, because if he were an employee he'd be in a lot of trouble.. ok, it is what it is.
"I'm going to pull the drives and take a couple of your backups and we'll see what I can do", I said. I figured I'd take both drives, maybe restore the OS to the unused one, mount the problem drive and see what I could copy.
When I got back to my office I first attached the drive to my Linux server, and mounted it. At first I thought I was home free because it didn't even complain that the filesystem was dirty and as I scouted around a few directories the data looked good. Unfortunately, when I went to the /usr/syn directory, it all fell apart: hard read errors galore. I shut down and relayed the bad news to the Synchronics guy.
"Well, maybe not so bad", he said. "The customer says he doesn't really care about current tickets and all, he just doesn't want to have to re-enter all his part numbers. Those should be on the backups even if they are old."
Okey-dokey. One smallish problem: these were the old-style cartridge DVD-RAM's. I don't have that kind of drive anymore so I'd have to break open the case and pull out the DVD. I did that and tried reading it with Microlite Edge - it read through the /dev files and failed dead with an I/O error. Not good.. but maybe I carelessly scratched the disk when I broke the case, so I tried again. Success! I read off /usr/syn and then repacked it in a standard tar file. The newest file was from 2006 - almost three years old.
So that's where we stand. Synchronics guy says he'd like to switch the customer to Windows. Fine, I'm not going to argue with him. I'll put the files on a Windows readable CD and he can do whatever he likes..
Of course he could send the drive out to a data recovery firm. That can be pricey, and my impression was that the customer isn't interested in spending money.
It would have been so simple if he'd just put the DVD's in the drive every night.
Got something to add? Send me email.
More Articles by Anthony Lawrence © 2009-11-07 Anthony Lawrence