Wednesday, January 23, 2013

The problem with post-mortems

If you've worked in a large-ish operation for any amount of time, you've almost certainly found yourself in a project post-mortem: a meeting in which the highs and lows of the project are dissected and examined, toward the goal of extracting lessons learned and doing things better the next time around.

If you've worked long enough to get a little cynical about the process, you've probably noticed that the lessons learned in post-mortem meetings almost never result in better process going forward. Generally speaking, no one in the meeting is responsible for taking the lessons and enacting them, and in most cases no one will have the authority to change organizational procedure even if they want to. As a result, the post-mortem meeting is almost always a well-intentioned failure. You may talk about good things, but you won't change much if at all.

The problem, ultimately, is one of memory. If you've ever studied a foreign language, you know that there are two types of memory: short-term and long-term. You can study vocabulary or verb conjugation and have pretty good recall 10 or 20 minutes later. Remembering the same material the next day is quite a bit harder, though, and remembering it a week, month, or year later is harder still. Knowing something now doesn't mean you'll know it later, when you need it.

Language students spend a lot of time struggling with this problem and slowly, methodically moving items from their short-term to their long-term memory. In grad school I had a flash card system that was my evolved solution to this problem: I would test, and re-test, and re-test myself again on vocabulary words at specific and carefully-chosen intervals until I could be confident that I had memorized them. It was sometimes time-consuming, but that was a lot better than not learning the material.

The post-mortem meeting is akin to taking the vocabulary card out of the box and looking at it once. That's a  start, but no one learns from that. Learning comes through repetition and review. Has any business collected and reviewed project insights in this way?

Sunday, January 20, 2013

Hear the noise

Wired has a short article on how a software bug might have helped Deep Blue defeat Gary Kasparov in their famous match in 1997. The bug caused the computer program to choose a play at random; the play wasn't devastating in itself, but Kasparov was so flustered by how counter-intuitive it was that he was thrown off his game. Kasparov assumed that the move had an intellect behind it, and since the move didn't appear to make sense he assumed that the machine was looking very deeply into the game and seeing something sublime. He assigned a meaning to randomness and was dismayed at what that implied.

This is something we humans do all the time, and we're only occasionally aware of it. We see something in the world and construct a story that makes sense of that thing. We weave a narrative that includes character and motivation; only sometimes are we in the position of knowing whether our story actually makes sense.

Consider web analytics. You have pages and articles, clicks and time spent on page. From numbers your goal is to derive meaning: customer intent that is either frustrated or fulfilled by your site design. You isolate a feature -- your home page bounce rate, for instance -- and tell a story: we're disappointing our customers. You're never going to meet more than a handful of those site visitors, though, so the story you tell exists entirely in the realm of the hypothetical.

You do what you can to make things less hypothetical, of course. You aggregate numbers, for instance, and resist the urge to find too much meaning in smaller collections of clicks. But another feature of human intellect is we notice irregularity. The familiar, day-to-day rhythms of your life and your website soon cease to capture your intention. Instead, your eye is drawn to what's new and different, the things that you didn't expect. Like Kasparov, you obsess about what you can't understand. And, like Kasparov, you probably don't take the time to think that something that's inexplicable might simply lack a rational explanation.

There is noise in your signal. Practice the art of hearing the noise and not mistaking it for a hidden, more meaningful signal.