How Do I Know What Success Looks Like?

I was asked recently what I do to ensure my team knows what success looks like.  I generally start with a clear definition of done, then factor usage and satisfaction into my evaluation of success-via-customers.

Evaluation Schema

Having a clear idea of what “done” looks like means having crisp answers to questions like:

  • Who am I building for?
    • Building for “everyone” usually means it doesn’t work well for anyone
  • What problem is it fixing for them?
    • I normally evaluate problems-to-solve based on the new actions or decisions the user can take *with* the solution that they can’t take *without* it
  • Does this deliver more business value than other work we’re considering?
    • Delivering value we can believe in is great, and obviously we ought to have a sense that this has higher value than the competing items on our backlog

What About The Rest?

My backlog of “ideas” is a place where I often leave things to bake.  Until I have a clear picture in my mind who will benefit from this (and just as importantly, who will not), and until I can articulate how this makes the user’s life measurably better, I won’t pull an idea into the near-term roadmap let alone start breaking it down for iteration prioritization.

In my experience there are lots of great ideas people have that they’ll bring to whoever they believe is the authority for “getting shit into the product”.  Engineers, sales, customers – all have ideas they think should get done.  One time my Principal Engineer spent an hour talking me through a hyper-normalized data model enhancement for my product.  Another time, I heard loudly from many customers that they wanted us to support their use of MongoDB with a specific development platform.

I thanked them for their feedback, and I earnestly spent time thinking about the implications – how do I know there’s a clear value prop for this work?

  • Is there one specific user role/usage model that this obviously supports?
  • Would it make users’ lives demonstrably better in accomplishing their business goals & workflows with the product as they currently use it?
  • Would the engineering effort support/complement other changes that we were planning to make?
  • Was this a dealbreaker for any user/customer, and not merely an annoyance or a “that’s something we *should* do”?
  • Is this something that addresses a gap/need right now – not just “good engineering that should become useful in the future”?  (There’s lots of cool things that would be fun to work on – one time I sat through a day-long engineering wish list session – but we’re lucky if we can carve out a minor portion of the team’s capacity away from the things that will help right now.)

If I don’t get at least a flash of sweat and “heat” that this is worth pursuing (I didn’t with the examples mentioned), then these things go on the backlog and they wait.  Usually the important items will come back up, again and again.  (Sometimes the unimportant things too.)  When they resurface, I test them against product strategy, currently-prioritized (and sized) roadmap and our prioritization scoring model, and I look for evidence that shows me this new idea beats something we’re already planning on doing.

If I have a strong impression that I can say “yes” to some or all of these, then it also usually comes along with a number of assumptions I’m willing to test, and effort I’m willing to put in to articulate the results this needs to deliver [usually in a phased approach].

Delivery

At that point we switch into execution and refinement mode – while we’ve already had some roughing-out discussions with engineering and design, this is where backlog grooming hammers out the questions and unknowns that bring us to a state where (a) the delivery team is confident what they’re meant to create and (b) estimates fall within a narrow range of guesses [i.e. we’re not hearing “could take a day, could take a week” – that’s a code smell].

Along the way I’m always emphasizing what result the user wants to see – because shit happens, surprises arise, priorities shift, the delivery team needs a solid defender of the result we’re going to deliver for the customer.  That doesn’t mean don’t flex on the details, or don’t change priorities as market conditions change, but it does mean providing a consistent voice that shines through the clutter and confusion of all the details, questions and opinions that inevitably arise as the feature/enhancement/story gets closer to delivery.

It also means making sure that your “voice of the customer” is actually informed by the customer, so as you’re developing definition of Done, mockups, prototypes and alpha/beta versions, I’ve made a point of taking the opportunity where it exists to pull in a customer or three for a usability test, or a customer proxy (TSE, consultant, success advocate) to give me their feedback, reaction and thinking in response to whatever deliverables we have available.

The most important part of putting in this effort to listen, though, is learning and adapting to the feedback.  It doesn’t mean rip-sawing in response to any contrary input, but it does mean absorbing it and making sure you’re not being pig-headed about the up-front ideas you generated that are more than likely wrong in small or big ways.  One of my colleagues has articulated this as Presumptive Design, whereby your up-front presumptions are going to be wrong, and the best thing you can do is to put those ideas in front of customers, users, proxies as fast and frequently as possible to find out how wrong you are.

Evaluating Success

Up front and along the way, I develop a sense of what success will look like when it’s out there, and that usually takes the form of quantity and quality – useage of the feature, and satisfaction with the feature.  Getting instrumentation of the feature in place is a brilliant but low-fidelity way of understanding whether it was deemed useful – if numbers and ratios are high in the first week and then steadily drop off the longer folks use it, that’s a signal to investigate more deeply.  The user satisfaction side – post-hoc surveys, customer calls – to get a sense of NPS-like confidence and “recommendability” are higher-fidelity means of validating how it’s actually impacting real humans.

Occupied Neurons, late May 2016

Understanding Your New Google Analytics Options – Business 2 Community

Here’s where the performance analytics and “business analytics” companies need to keep an eye or two over their shoulder. This sounds like a serious play for the high-margin customers – a big capital “T” on your SWOT analysis, if you’re one of the incumbents Google’s threatening.

10 Revealing Interview Questions from Product Management Executives

Prep’ing for a PM/PO job interview? Here’s some thought-provoking questions you should think about ahead of time.

When To Decline A Job Offer

The hardest part of a job search (at least for me) is trying to imagine how I would walk away from a job offer, even if it didn’t suit my needs, career aspirations. Beyond the obvious red flags (dark/frantic mood around the office, terrible personality fit with the team/boss), it feels ungrateful to say “no” based on a gut feel or “there’s something better”. Here’s a few perspectives to bolster your self-worth algorithm.

The Golden Ratio: Design’s Biggest Myth

I’m one of the many who fell for this little mental sleight-of-hand. Sounds great, right? A magic proportion that will make any design look “perfect” without being obvious, and will help elevate your designs to the ranks of all the other design geeks who must also be using the golden ratio.

Except it’s crap, as much a fiction and a force-fit as vaccines and autism or oat bran and heart disease (remember that old saw?). Read the well-researched discussion.

Agile Is Dead

This well-meaning dude fundamentally misunderstands Agile and is yet so expert that he knows how to improve on it. “Shuffling Trello cards” and “shipping often” doesn’t even begin…

Not even convinced *he* has read the Manifesto. Gradle is great, CD is great, but if you have no strategy for Release Management or you’re so deep in the bowels of a Microservices forest that you don’t have to worry about Forestry Management, then I’d prefer you step back and don’t confuse those chainsaw-wielders who I’m trying to keep from cutting off their limbs (heh, this has been brought to you by the Tortured Analogies Department).

Occupied Neurons, April 2016

https://medium.com/@sproutworx/six-templates-for-aspiring-product-managers-a568d3115cfe#.swkk52f58
So many Product Managers are making it up as they go along – generating whatever kinds of artifacts will get them past the next checkpoint and keep all the spinning plates from veering off into ether. This is the first time in a long time I’ve seen someone propose some viable, useable and not totally generic tools for capturing their PM thinking. Well worth a look.

https://medium.com/swlh/mvpm-minimum-viable-product-manager-e1aeb8dd421
The “BUT” model for Product Management is a hot topic, and there’s a number of folks taking a kick at deciphering it in their context. I’ve got a spin on it that I’ll write about soon, but this is a great take on the model too.

https://schloss.quora.com/Design-doesnt-deserve-a-seat-at-the-table
Captures all my feelings about the complaint from Designers (and Security reviewers, and all others in the “product quality” disciplines) that they get left out of discussions they *should* be part of. My own rant on the subject doesn’t do this subject justice, but I’m convinced that we *earn* our right to a seat by helping steer, working through the messy quagmire that is real software delivery (not just throwing pixel-perfect portfolio fodder over the wall).

http://www.eventbrite.com/e/resilience-and-the-future-of-work-responsiveorg-un-conference-tickets-24045089510
An unconference to expand awareness of a movement among leading thinkers on how to organize work in the 21st century. Looks fascinating – unconference format is dense and high-learning, the subject is still pretty fresh and new (despite the myriad of books building up to this over the last decade), and the energy in the Portland community is bursting.

Meetups where you’ll find Mike’s hat, Spring 2016 edition

Occasionally I’ll tell people I meet about all the meetups I have so much fun at.

Or rather, I’ll try to enumerate them all, and fail each and every time.

Primarily because there’s so many meetups I like to check in on.

So occasionally I’ll enumerate them like this, so that my friends have a valiant hope of crossing paths with me before the amazing event has passed.

Meetups I’m slavishly devoted to

Meetups I’ll attend anytime they’re alive

Meetups I sample like caviar – occasionally and cautiously

Recent additions that may soon pass the test of my time

 

Pruning features via intelligent, comprehensive instrumentation

Today’s adventures in the LinkedIn Product Management group gave us this article:
http://product.hubspot.com/blog/the-5-whys-of-feature-bloat

The critical statement (i.e. the most/only actionable information) in the article is this:

Decide a “minimum bar of usage/value” that every feature must pass in order for it to remain a feature. If a new feature doesn’t hit that bar in some set period of time, prune it.

I’d love to hear from folks who are able to prove with data that a feature is not getting the level of usage that we need to justify its continued existence.  AFAIK, whether it be a desktop, mobile or web/cloud app, instrumenting all the things so that we have visibility into the usage of every potentially-killable feature is a non-trivial (and sometimes impractical) investment in itself.

I’m not even arguing for getting that work prioritized enough to put it in up front – that’s just something that if it’s technically feasible, we should *all* do to turn us from cavemen wondering what the stars mean to explorers actually measuring and testing hypotheses about our universe.

I’m specifically inquiring how it’s actually *done* in our typical settings.  I know from having worked at New Relic what’s feasible and what are the limits of doing this in web/cloud and mobile settings, and it’s definitely a non-trivial exercise to instrument *all* the things (especially when we’re talking about UI features like buttons, configuration settings and other directly-interactive controls).  It’s far harder in a desktop setting (does anyone still have a “desktop environment”?  Maybe it’s just my years of working at Microsoft talking…).

And I can see how hard it is to not only *instrument* the settings but gather the data and catalogue the resulting data in a way that characterizes both (a) the actual feature that was used and, even better, (b) the intended result the user was trying to achieve [i.e. not just the what or how but the *why*].

Developers think one way about naming the internals of their applications – MVC patterns, stackoverflow examples, vendor cultures – and end users (and likely/often, we product managers) think another way.  Intuitive alignment is great, but hardly likely and usually not there.  For example, something as simple as a “lookup” or “query” function (from the engineering PoV) is likely thought of as a “search” function by the users.  I’ve seen far more divergent, enough to assume I won’t follow if I’m just staring at the route/controller names.

If I’m staring at the auto-instrumented names of an APM vendor’s view into my application, I’m likely looking at the lightly-decorated functions/classes/methods as named by the engineers – in my experience, these are terribly cryptic to a non-engineer.  And for all of our custom code that wove the libraries together, I’m almost certainly going to have to have the engineers add in custom tracers to annotate all the really cool, non-out-of-the-box features we added to the application.  Those custom tracers, unless you’ve got an IA (information architecture) nut on the team to get involved in the naming, will almost certainly look like a foreign language.

Does that make it easy for me to find the traces of usage by the end users of a specific feature (e.g. an advanced filtering textbox in my search function)?  Nope, not often, but it’s sure a start.

So what do you do about this, to make it less messy down the road when you’re dying to know if anyone’s actually using those advanced filtering features?

  1. Start now with the instrumentation and the naming.  Add the instrumentation as a new set of acceptance criteria to your user stories/requirements/tickets.  If the app internals have been named in a way that you understand at a glance, awesome – encourage more of the same from the engineers, and codify those approaches into  a naming guideline if possible.  Then if you’re really lucky, just derive the named instrumentation from the beautiful code.
  2. If not, start the work of adding the mapped names in your custom instrumentation now – i.e. if they called it “query”, make sure that the custom instrumentation names it “search”.
  3. Next up, adding this instrumentation for all your existing features.  Here, you have some interesting decisions:
    • Do you instrument the most popular and baseline features? (If so, why?  What will you do with that data?)
    • Do you instrument the features that are about to be canned? (If so, will this be there to help you understand which of your early adopter customers are still using the features – and do you believe that segment of your market is predictive of the usage by the other segments?)
    • Or do you just pick on the lesser-known features?  THESE ARE THE ONES I’D RECOMMEND for the most benefit for the invested energy – the work to add and continue to support this instrumentation is the most likely to be actionable at a later date – assuming you’ve got the energy to invest in that tension-filled EOL plan (as the above article beautifully illustrates).
  4. Finally, all of this labour should have convinced you to be a little more judicious in how many of these dubious features you’re going to add to your product.

Enhancing your ability to correct for these mistakes later is great; factoring in the extra cost up front, and helping justify why you’re not doing it now is even better.

And all that said?  Don’t get too hung up on the word “mistakes”.  We’re learning, we’re moving forward, and some of us are learning that Failure Is An Option.  But mostly, we’re living life the only way it’s able to be lived.

success

Purity of Product, Sales or Code – are ALL wrong

I caught this echo-chamber discussion on the LinkedIn Product Management channel:

https://www.linkedin.com/comm/groups/42629/42629-6103296611813781509

Somehow I thought I’d mention a thing or two and then bail, but the more I wrote the more strongly I felt how misguided the “keep your product vision pure” message was. Here’s what I wrote, think of it what you will:

Purity of the product is one ideal; never losing another “key” sale is another such ideal. Making every user happy and productive with your product is a third such, and never shipping imperfect code is yet another.

None of these ideals can be achieved in isolation for very long *and* maintain a successful business. Sometimes we have to flex on one of these ideals to help the rest of the organization improve their position on another.

If I put my foot down and refused to engage with any of my key customers’ feature requests, any responsible sales exec would have my head…examined.

If I gave into every customer want, no product leader (engineering, marketing or product management) would have any faith in my vision of the product.

If I constantly let the engineers gold-plate the code and drive out all tech debt, we’d be so far behind the market we’d never get around to acquiring new customers.

Rallying around the banner of “never let Sales disrupt the purity of my product’s vision@ sounds just as naive as “never ship an insecure product” and “always involve your Designers up front and throughout the development process” (both of which I’ve observed firsthand in those respective domains).

Never forget that every signal from the market can be that incredible insight that *causes* you the change your product strategy, and Sales are often your eyes and ears gathering input from customers that you yourself weren’t there to acquire.

Best advice? If it’s a significant sales opportunity/downgrade, and you haven’t spoken with that customer before, it’s worth considering taking a half-hour out of your schedule to qualify the sales opportunity *and* the feature request yourself – if only to increase the strength of the signal and decrease the risk of making the wrong call.

I personally don’t do enough of this, but when I do it gives me irreplaceable opportunities to learn from customers who aren’t there to feed me what they think I should hear (i.e. the standing “customer council”) and let me pivot the conversation to find out what else they really need (i.e. other items in considering) that Sales didn’t take the time to ask about.

New job at New Relic: First Month Report

An incredible place, rewarding experiences, emotionally engaged people.

Week one: feeling incredibly welcome, people are buzzing by saying “hi” but dropping no turds in my lap

Week two: I’m starting to get to understand the scope of my job and it terrifies me; tigers are lurking in the shadowy forest, and I’m hearing some occasional growls

Week three: I’ve slain a small house cat and feel like a warrior born; scouting parties are starting to make contact, and I’m getting involved in a real (but small) skirmishes with that beast they call “work”

There’s three levels of tired that come with an amazing new job:

  1. Tired when I get home. “I might turn in at a reasonable bedtime, honey.”
  2. Bone tired as I drag myself down the street, hoping I I don’t get mistake for a drunk before I get home.
  3. Drop like a sack of rocks when I cross the threshold. “I’m sorry I drooled on you while using you as a pillow, Mr. Fellow Bus Passenger.”

I am so glad I’m not driving home after work these days – “reckless” and “menacing” would be likely adjectives among the charges.

Highlights include:

  • Hardest mindfulness lesson: “we have two ears and one mouth, so we should listen first and then speak” – I am fighting to curb every ego instinct I have to stop trying to sound smart (show off as if I already understand all this new shit) and ask questions or just look like I don’t know what everyone’s talking about
  • Met a dude who spent six months working between Stockholm Sweden and Hillsboro Oregon. (We made many jokes drawing increasingly-strained comparisons between the two sexy locales)
  • Favourite hair colour I’ve seen: burgundy/orange/yellow stripes (like a funhouse Neapolitan ice cream)
  • Estimated number of people who recognized me by my hat: 5 by Tuesday
  • How welcome I felt by the end of day one: 10/10 (would do it over again)
  • How overwhelmed I felt by the end of week two: 12/10 (would crush my brain over again)
  • Best Google Mail hack: http://klinger.io/post/71640845938/dont-drown-in-email-how-to-use-gmail-more
  • “6months” – the amount of time every SINGLE person says it’ll take for me to be a net contributor to the business
  • New phrases: “icebergs vs. ice cubes” (are problems big, or do they just start out that way because everything’s so loud at the start), “N+1 query”
  • Never laughed as much at work – spontaneously – at least not sober. This is a very entertaining group of folks (or they’re just as warped as me). Friday’s highlight – “Anyone interested in a quiche run to Lauretta Jeans? I’m feeling like elevensies today.”
  • Bus asymmetry: getting on the 14 (Hawthorne) anytime between 7:30-8:30, sit with lots of normal, quiet, keep-to-themselves people. Getting on the 4 (Division) at work towards home anytime between 4:30-6:30, (and couldn’t think of a good joke to wrap up this thought)
  • Best spontaneous phrase I came up with to describe my mental state by week 3: the project plan I drafted hasn’t made me feel competent, but it “bounded my insecurities”
  • Perspective I heard I love: Intuit obsesses on our customer problems, not on what we’re building
  • “Discovering not declaring” how core values were generated
  • “Bring out the best in each other” – not a regulated nation-state

On the scale of behaviours of people with whom I’ve had to work in my career, the worst of them I’ve encountered at New Relic are pretty mild, and the best of them are bend-over-backwards nice, helpful and fun to be around.