What you don't see when a website goes down

FolioHD went down this week due to some server problems. I happened to be in Paris, approximately 5,642 miles away from where the servers reside in Los Angeles. This tends to be a problem when you are physically nowhere near the data center and when you can't fix things remotely. The remaining two days of my vacation were spent on spotty, unreliable wifi, emailing with customers and working with vendors and contractors to bring the site back online.

During the process, I received hundreds of emails from customers. I would lump these customers in three categories:

  • Those who asked when their sites would be back online. Totally understandable.
  • The entitled customers who tell me how I am hurting their business, that this is unacceptable, they are infuriated, demand refunds, and tell me they will be going elsewhere. (I actually referred many of these customers to a competitor of ours.)
  • And then, few and far between, are the empathetic customers who are compassionate, more than patient, and understand we are working as quickly as we can to fix things.

When something doesn't work as expected, it's normal to become irritated or angry. (I remember when our electricity was out for about 3 days back in 2002. I wrote an angry tirade to Southern California Edison on my blog at the time.)

But having experienced being on the other side of an outage now, I figured I'd share a little bit about how I've spent the last 48 hours.

  • Found out about the downtime, began to diagnose the issue
  • Started answering customer emails
  • Diagnosed the problem, weighed possible solutions
  • Sent out a mass email to recently active customers telling them we're working on the issue
  • Decided on a solution, started calling vendors over Skype (from Paris)
  • Replied to all sorts of customers asking for status updates
  • Selected a vendor, asked a very kind friend to pick up some hard drives from our data center and drive them across town to the data recovery facility
  • Coordinated the purchase of new hardware to be overnighted to the data center
  • Discussed status updates throughout the night with our data recovery team as they worked to save data
  • Continued responding to angry emails
  • Flew home
  • Landed, picked up hard drives from data recovery facility
  • Drove them across Los Angeles to the data center and plugged them in
  • Got the site back online
  • Emailed active customers to inform them of the news and explain what happened and what was going to change to make sure this didn't happen again
  • Started issuing refunds and credits

The last two of days of my "vacation", I got maybe 3 hours of sleep in total. I slept with my phone in-hand, waking up to every vibration to check on status updates from our data recovery team and to triage anything urgent.

Throughout the whole process, I received several emails that were quite encouraging:

"I really appreciate your upfront and clear communication regarding this incident. Your professionalism does not go unnoticed."

"I am certainly glad that you all were able to get FolioHD back I'm action! Although it was tough, you all did an exceptional job with keeping us all informed. I am more than sure that from this conquered hurdle, FolioHD will reach new heights that will align and even exceed what you all have envisioned."

"Great communication through out your down time ... and I can tell from your email above that there are genuine, interested people at work behind the scenes, which prompted me to put this email together. Keep up the good work and all the best for 2014! "

Too often, people think of websites as machines and not people. What encourages me about these emails is that these customers actually understand that every harsh word spoken to a company is actually read by someone who has invested blood, sweat and tears into the product, and instead of using their words to tear down, they chose to be positive and understood this fact. I totally understand the frustration of those who rely on us and those who speak their mind about it, but when it comes down to it, I hope they can understand that I am just as frustrated as they are, and that their harsh words only take me away from trying to fix the root of the problem. They don't see the 22 hours days people like me put in to get things back in order.

The next time you email somebody who runs a website, keep in mind that you're dealing with normal people. Speak with civility and respect. Your words are far more likely to be heard than those who act entitled and fly off the handle.