The “Case Dismissed” feature that appears here every Tuesday morning focuses on my shortcomings as a consumer advocate because, as I’ve said so often, you can often learn more from your failures than from your successes.
But what if it’s a more, ahem, personal failure?
That’s a question I’ve been asking myself in the last few days as I’ve struggled with the loss of huge amounts of data, which vanished under suspicious circumstances. And no time more than yesterday, when we endured numerous website outages. I’ll have details on those in a minute.
Let me start by saying this as unambiguously as possible: I’m responsible for what happened. Period.
Sure, there’s a long list of things that went wrong, a long list of people who didn’t do what they were supposed to — or, in some instances, did what they weren’t supposed to do.
But I’m responsible. It’s my name on the site. The buck stops here, as they say. (Or you can insert any other sports cliche you want.)
So, for those of you who aren’t paying attention to this soap opera, here’s what happened. In early August, we noticed that our server was slowing down. We’re not sure exactly why, but we have a few theories. At the end of the month, my IT team took the server down to figure out what was happening. When it returned, the site was gone.
Making matters worse, we noticed that some of our backup copies had apparently been tampered with. The only working backups were several months old. (Related: Should I turn down cases because of an entitled attitude?)
The damage was extensive
✓ We lost about five months of forum threads. They have not been recovered.
✓ Four months of blog posts have been deleted. They can be recovered from a cache, but the process will be slow and difficult.
✓ Also, four months of company contacts were gone. We’re well on our way to recovering them, thanks to the cache.
Not quite a total loss, but a serious setback.
A closer review of the damage suggested we were in better shape than it seemed.
Turns out our archived blog posts are not in high demand. Like any periodical, people read the posts and then move on.
On our forums, we have a small, core group of advocates and a large group of transient users who come to us for help and then never return again. Also, the data outage conveniently removed a series of toxic threads that probably should have been deleted in the first place.
The biggest issue was returning our company contacts, which are used over and over — and we’re well on our way to doing that now.
Oh, and about yesterday’s outages. We were trying to move to a new server, but encountering some serious challenges with the latest version of PHP. By 5 p.m., the IT team switched us back to the old server so they could figure out what was happening.
When the dust settles from this snafu, we’ll be on a new server. We’ll have all our post and company contacts back. Our core team of forum advocates will still be there and consumers will not notice the gap. (Here’s our guide to resolving your consumer problem.)
Still, I can’t just “dismiss” this case, and not just because it’s my case. I have to ask: What are the takeaways from such a systematic failure?
Here they are:
Our cause is stronger than failure
I admit, there was a point just after the meltdown when everything looked so bleak, I didn’t want to continue. Not only had we lost data, but I’d just learned that a friend and mentor had passed away unexpectedly. Talk about a one-two punch to the gut. But our advocates inspired me to keep going. Amazingly, not one of our volunteers has left during this server outage. In fact, it’s been quite the opposite. They’ve rallied around the cause and redoubled their efforts to keep this going. I’m grateful for their support.
Data doesn’t always age well
As I reviewed the lost information, it occurred to me that some of it was more valuable, over the long term. For example, our contacts data ranked the highest, in terms of information worth saving. Some of the more chatty blogs repurposed from wire services that ran a few months ago? Not really. It all made me wonder why we were writing the stuff that wasn’t worth keeping, instead of focusing on the things that have a longer shelf life.
Our advocacy is more effective than we ever imagined
For anyone who doubted the influence of our site, this server incident should clear things up. My IT team is certain that someone tried to do us harm, and we can think of a hundred reasons why someone would want to off us.
Earlier in this story, I said we can learn more from our failures than our successes. That’s definitely true for this case.
I’m determined to make every post count. If we’re writing something for the site, it has to help consumers. That’s why we’re here. Bonus points if it has information that will continue helping readers over the long term. We’re also taking additional steps to ensure this kind of thing never happens again, although we can’t go into too many details, for obvious reasons.
This incident did us more long-term good than short-term harm. Our resolve is stronger and our morale is higher now than at any time I can remember. We’ll brush off this loss and keep fighting for you.
If someone wanted to hurt us, the failure is theirs — not ours.
