, , , , ,

Twice over the past week and a half I have had situations occur where something related to technology has not gone to plan.  Shocking, I know, because that never happens!  But what do you do when it does?

Credit: ITworld/Phil Johnson

Credit: ITworld/Phil Johnson

For example, most of us have had a hard drive crash on us.  You only have to learn that lesson once before you back up regularly. Or perhaps you’ve had a car break down.  Sometimes it just can’t be prevented, but we all know regular maintenance is key.

But what do you do when the technology that fails is related to a service, and more importantly, a service where customer experience factors into the equation?

As product management leaders, contingency planning is essential to success. As alluded to early, sometimes failures occur.  It might something as simple as a miss in the design, build or test process.  It happens.  Or it might be something catastrophic that just can’t be planned for. Whether it is release capacity related, prototyping a new feature or idea, budget reductions, missed milestones, or any number of other possible scenarios, having a backup plan could be the difference between an adjustment and a crisis.


With a recent production release, we had a so-called “miss”.  Despite the best intentions of the team and a full release cycle with adequate testing, a miscommunication along the way meant an architectural change had more far reaching impacts than anticipated.  When concerns started to arise in the 11th hour, we all went into contingency planning mode.

Several conversations with key stakeholders later, and with options to minimize the risk firmly in place, we set forth to put the release in … hoping for the best, but ready for the worst.

So when the release went live, and we immediately began to see impacts to customer transactions, the decision was made to scrap deployment and revert back to the original state to evaluate and redeploy at a later time.  Yes, as a business leader I can honestly state the outcome sucked. But we were well prepared for this possibility, and more importantly averted potential customer issues or escalations.


After a long week in Manhattan attending a conference and spending time with some of our chief architects, it was time to make the trek home. Like I’ve done countless times over the course of my career where travel was involved, I hailed a cab to Laguardia. A nice enough driver and a typical NY traffic afternoon meant a ~50 minute ride. I was well on my way home … or so I thought.

We we arrived at the airport, and the driver pushed a few buttons on the meter to transfer the fare information to the monitor and card machine in the back seat.  Let me just say that for the most part when you are travelling around NYC, the TV provides some good entertainment for the short duration you are in the cab.  But over the course of an hour, being subjected to the same loop about 15 times gets monotonous! But I do love the convenience of Apple Pay in most of the cabs that are NFC enabled these days!

So anyway, I pushed the pay now button on the monitor, added one of the suggestions provided for the tip and …. nothing.  And when I say nothing, I mean nothing.  The system died.  I was now staring at a black monitor with no signs of life.  And despite the best efforts of the driver to reboot/restart the system, it was unresponsive.

In their infinite wisdom to not have the drivers deal with manual processes and potential human error or fraud, the cab companies had pulled all of the card swipe machines.  There was no contingency plan.

And to state the obvious its never a good idea, when you are offering a service that fails, to put the onus of resolution on the customer. Not only did I have to solve the problem (going into the airport to find an ATM), I was also going to be punished for doing so (fees from the machine itself and my bank). Not cool, and not a good customer experience.

I caught the cab out of convenience because it was right there … next time, I may just continue to rely on Uber where payment is much more seamless to the experience.