How Do I Know What Success Looks Like?

I was asked recently what I do to ensure my team knows what success looks like.  I generally start with a clear definition of done, then factor usage and satisfaction into my evaluation of success-via-customers.

Evaluation Schema

Having a clear idea of what “done” looks like means having crisp answers to questions like:

  • Who am I building for?
    • Building for “everyone” usually means it doesn’t work well for anyone
  • What problem is it fixing for them?
    • I normally evaluate problems-to-solve based on the new actions or decisions the user can take *with* the solution that they can’t take *without* it
  • Does this deliver more business value than other work we’re considering?
    • Delivering value we can believe in is great, and obviously we ought to have a sense that this has higher value than the competing items on our backlog

What About The Rest?

My backlog of “ideas” is a place where I often leave things to bake.  Until I have a clear picture in my mind who will benefit from this (and just as importantly, who will not), and until I can articulate how this makes the user’s life measurably better, I won’t pull an idea into the near-term roadmap let alone start breaking it down for iteration prioritization.

In my experience there are lots of great ideas people have that they’ll bring to whoever they believe is the authority for “getting shit into the product”.  Engineers, sales, customers – all have ideas they think should get done.  One time my Principal Engineer spent an hour talking me through a hyper-normalized data model enhancement for my product.  Another time, I heard loudly from many customers that they wanted us to support their use of MongoDB with a specific development platform.

I thanked them for their feedback, and I earnestly spent time thinking about the implications – how do I know there’s a clear value prop for this work?

  • Is there one specific user role/usage model that this obviously supports?
  • Would it make users’ lives demonstrably better in accomplishing their business goals & workflows with the product as they currently use it?
  • Would the engineering effort support/complement other changes that we were planning to make?
  • Was this a dealbreaker for any user/customer, and not merely an annoyance or a “that’s something we *should* do”?
  • Is this something that addresses a gap/need right now – not just “good engineering that should become useful in the future”?  (There’s lots of cool things that would be fun to work on – one time I sat through a day-long engineering wish list session – but we’re lucky if we can carve out a minor portion of the team’s capacity away from the things that will help right now.)

If I don’t get at least a flash of sweat and “heat” that this is worth pursuing (I didn’t with the examples mentioned), then these things go on the backlog and they wait.  Usually the important items will come back up, again and again.  (Sometimes the unimportant things too.)  When they resurface, I test them against product strategy, currently-prioritized (and sized) roadmap and our prioritization scoring model, and I look for evidence that shows me this new idea beats something we’re already planning on doing.

If I have a strong impression that I can say “yes” to some or all of these, then it also usually comes along with a number of assumptions I’m willing to test, and effort I’m willing to put in to articulate the results this needs to deliver [usually in a phased approach].


At that point we switch into execution and refinement mode – while we’ve already had some roughing-out discussions with engineering and design, this is where backlog grooming hammers out the questions and unknowns that bring us to a state where (a) the delivery team is confident what they’re meant to create and (b) estimates fall within a narrow range of guesses [i.e. we’re not hearing “could take a day, could take a week” – that’s a code smell].

Along the way I’m always emphasizing what result the user wants to see – because shit happens, surprises arise, priorities shift, the delivery team needs a solid defender of the result we’re going to deliver for the customer.  That doesn’t mean don’t flex on the details, or don’t change priorities as market conditions change, but it does mean providing a consistent voice that shines through the clutter and confusion of all the details, questions and opinions that inevitably arise as the feature/enhancement/story gets closer to delivery.

It also means making sure that your “voice of the customer” is actually informed by the customer, so as you’re developing definition of Done, mockups, prototypes and alpha/beta versions, I’ve made a point of taking the opportunity where it exists to pull in a customer or three for a usability test, or a customer proxy (TSE, consultant, success advocate) to give me their feedback, reaction and thinking in response to whatever deliverables we have available.

The most important part of putting in this effort to listen, though, is learning and adapting to the feedback.  It doesn’t mean rip-sawing in response to any contrary input, but it does mean absorbing it and making sure you’re not being pig-headed about the up-front ideas you generated that are more than likely wrong in small or big ways.  One of my colleagues has articulated this as Presumptive Design, whereby your up-front presumptions are going to be wrong, and the best thing you can do is to put those ideas in front of customers, users, proxies as fast and frequently as possible to find out how wrong you are.

Evaluating Success

Up front and along the way, I develop a sense of what success will look like when it’s out there, and that usually takes the form of quantity and quality – useage of the feature, and satisfaction with the feature.  Getting instrumentation of the feature in place is a brilliant but low-fidelity way of understanding whether it was deemed useful – if numbers and ratios are high in the first week and then steadily drop off the longer folks use it, that’s a signal to investigate more deeply.  The user satisfaction side – post-hoc surveys, customer calls – to get a sense of NPS-like confidence and “recommendability” are higher-fidelity means of validating how it’s actually impacting real humans.

Advice for a brand-spanking new Product Owner

I’ve started a mentee collection.  Not a manatee collection:
My mentees are better looking
I’m mentoring three new Product Owners, collecting them into my posse as I encounter new Product Owners who have no peers/colleagues from whom to learn the ropes.
The rant below was inspired by a real-life plea for help from a newly-minted Product Owner at my company.  Knowing how anxious and overwhelmed I was when I realized what I’d gotten myself into, these are the kinds of “start small, start here, build on your successes” advice I’d give my past self (can we use Skynet’s back-in-time portal yet?):
What could go wrong?
What could go wrong?
  • I’m inclined (especially with new teams, new process, new roles) to start small – get a rhythm, build a few quick wins, get small pieces shipped in short timeframes, even if it’s not technically a “potentially shippable product”
    • The important thing is to start seeing chunks of more valuable engineering work delivered earlier
    • For example, I’m currently getting the team to work on a “research spike” to determine if they can figure out how to add “test our site using IE11” to the test coverage – plus we’re doing the second story (broken up from a larger story) to deliver an Export capability to our project planner (just started with flat file, unformatted CSV as an export deliverable for now – it’s good value to users, and we’ll refine and “gold plate” it later)
    • I’ve also had them working on some of the more ‘valuable’ tech debt over the last few months – cleaning out old database tables, removing dependencies on legacy code and data structures, etc. – all written as “stories” (probably doesn’t fit the dogmatic definition), small bits of work each, that ideally could’ve been done all at once, but are easier to schedule, are establishing a rhythm, and have given them hope that I’m committed to systems hygiene as part of our negotiation between “user value” and “systems integrity”
  • Encourage the team to adopt SCRUM (or whatever flavour of Agile-ish behaviour they’re aiming at) in stages – don’t try to swallow it all at once
    • As a fresh PO, I’d stand up and tell them I don’t expect them to get everything right the first time, and especially not when they’re playing with all these new “process toys”
    • e.g.. Try the “sprint planning” and “retrospective discussion”, and drop everything else, for the first sprint – see how it goes, spend the time discussing “things to do more of and things to do less of” at the end, and ask them whether they’re ready to add “daily stand-ups” or “story point estimation” or “story breakdown” to their process
    • Personally I don’t want process to get in the way of the engineering – I want it to be a net break-even, and if you try to do all the process all at once, there’s no way that they’ll get any effective engineering done that first sprint or two – just too much flailing and failing on friggin terminology, not enough on learning to succeed at the job with incremental improvements along the way)
  • As quickly as possible, focus your efforts on the WHAT and the WHY of every piece of engineering work that the team sets out to do.
    • Any of us that come from engineering will still trip over the HOW too many times to count before we get really good at this, but it’s worth the effort to learn.
    • The most satisfying feeling as a PO is setting the team a challenge you yourself have no idea how they’ll deliver, and watching them struggle, flail and invent something they never knew they could do.
    • Over time, they’ll recognize your trust and confidence in them – in giving them the leeway and slack to take on challenging problems and kick the shit out of them]
    • It’s taken me years to let go of this, and I’m still meddling more than I should, but it’s satisfying to imagine how it’d feel for my ‘product CEO’ to have faith that I’ll do an amazing job figuring out how to make it happen
  • The big focus for you – for as long as you inhabit this crucial role – is to spend as much time in the heads of your users (and unfortunately, stakeholders who aren’t end users) to understand what will be the most valuable thing to deliver to them as soon as possible
  • The trick is “most valuable part of functionality”, not the ideal solution
  • e.g. The users want to be able to import the set of security requirements into their ‘work tracking system’ (Jira, RTC, Rally, Trello, etc).
    • It would be awesome to have it pre-filtered to just what they need, pre-annotated to fit exactly the field structure of each of their tools, and able to be exported back into our system
    • However, the most important first incremental delivery of value before we work out all those problems is “structured list of the names of the tasks, their current status and a link to more info on how to deliver them” – a far cry from the ideal solution, and probably ten steps between here and there
    • I’ll carve out this first step as a story, where the WHY is “I need to track this shit in my native work tracker”, and the WHAT is “the details that are critical to me tracing back to what I need to know to be successful”
    • The rest of that work I’ll represent as notes in a parent Epic, or one or more Stories that may or may not be fleshed out enough for the dev team to tackle (depending on how soon I/we think we might get around to delivering that residual incremental value)
In a future post I’ll introduce the fever-dream “system” I use to calculate priority for each story and Epic, but seriously you’ve got plenty to chew on just getting out of the starter’s gate. We can setup a 1:1 call for whenever in the future you want to review results/commiserate over failures/discuss additional experiments that you’re ready to try.
It’s an adventure – a Very Real experience in both asserting control and letting go of absolute control over the direction of the product. It’s a blast.  If you’re a new Product Owner, welcome to the tribe.