(2022-07-27) Chin John Cutlers Product Org Expertise

Cedric Chin: John Cutler's Product Org Expertise. Cutler’s superpower is that he is able to talk to a set of product people and — within 8-20 minutes — figure out the system dynamics of their organisation, and then suggest a bunch of experiments to make that system better. His interventions usually take on the form of a prioritised list of experimental changes or team interventions.

Sometime in the last year, he attempted to write a book about his skills. And then he came up against a wall.

Cutler’s expertise is intuitive and context-dependent

I told John that there was a whole set of techniques designed to extract tacit mental models of expertise, and that he was welcome to give them a try. I said that the simplest technique was something called ACTA.

A Quick Recap of ACTA

ACTA is a protocol of three interview methods and a presentation format

• You first create a task diagram
• You do a knowledge audit
• You do a simulation interview
• Finally, you present the extracted information in a cognitive demands table.

Due to my lack of experience using the technique, I only performed the first two tasks with John

In retrospect, perhaps I should have asked John for a recording of a product team engagement

What We Extracted

Cutler’s product calls consist of the following four stages

Introduction

Problem Detection

If by the 20 minute mark he isn’t able to map out a few problems, this is likely a rare edge case — a unique organisation that he can't help.

Solution Delivery

Wrap up

We identified Steps 2 and 3 as the parts with the highest cognitive demands:

Step 2, Problem Detection — A novice would not have John’s intuition on what to dig deeper into, and what to ignore

Step 3, Solution Delivery — Some product team problems are easy, or ‘acute’, and these tend to be simpler to diagnose and then fix

But some product team problems are ‘chronic’, or more accurately described as complex systems problems, where multiple things are bad all at once (e.g. bad processes, and no decision power, and technical debt, and bad org infrastructure, and a toxic leader, etc)

Of the two steps, we decided to focus on Step 2, Problem Detection

The Knowledge Audit

We ended up doing five probes

Aspects of Expertise Cues and Strategies Why Difficult? Past & Future

John's first pass is always to detect anti-patterns

Then, work backwards from the anti-patterns to figure out how the org got here.

First detect if there is high Work In Progress (WIP).

If high WIP exists, then he works backwards to figure out how the org got there

60% of the time, John says, a bad org has high WIP

The other 40% of the time, John is able to quickly suss out other possible contributors to those problems

Big Picture

Decision Authority (Agency)

where does this org lie on the extremes of product ownership?

Organisational learning ability

can talk about current hypotheses and experiments in play: “tell me your last 5 big product decisions”

Seriousness of product engagement

if this is a mainline activity or a vanity project

Business model

judge how important the product team is to the business

Risk taking

Not enough experiments that fail = company isn’t taking enough risks

Org culture

Is the company centralised, where HQ sets the culture

Demographic / country culture

Product strategy

Check, in the following order: is there a strategy or not? -> is it halfway plausible or not? -> is it implicit or explicit? -> Is the structure of the company aligned around the strategy of the company/product?

Decision authority

A novice (column repeats with what novice would get wrong for each item)

Noticing

Notice the team’s comfort with uncertainty

A good product team has productive things to say about what is unknown ("we know this feature would do well, but we're not sure about that feature, but we have experiments going on for it")

Notice when the company has ‘flow’.

do they have clean data, do they have the ability to interpret the data, the ability to link cause and effect

Able to notice when a product leader is at the stage where they are naive

enamoured by product frameworks

Quickly places the team on a spectrum, because John knows the extremes of each factor, and the contexts where they may or may not work

Comfort with uncertainty column repeats

Job Smarts

Able to ask ‘Powerful Questions’ to cut through the mess

Use humour to get a reaction.

John jokes about quarters being weird

Show visual guides to provoke reactions and test for systems thinking ability. If they do not respond well to a concept map or flow chart, it is likely they are not a good systems thinker.

Novice would not know column repeats

Self-Monitoring

John knows the weaknesses of his approach, and is able to tell when he is unable to help the org

What Was Difficult?

The biggest difficulty with using ACTA was in deciding when to dive deep and let John go on about his experiences, and when to move on to the next set of probes

I think the biggest mistake that I did was that I grossly underestimated the amount of time it would take to do the whole skill extraction exercise.

The other problem I had with the process was that I was unsure if I was producing useful information. John seemed happy with our progress, but I wasn't so sure.

After our first call, but before our second, I had the good fortune of talking to Laura Militello and Brian Moon, hosts of the NDM podcast. Militello, of course, was one of the two creators of ACTA, and when I asked her if she had any tips for using it, she said to keep the objective of the interview at the back of my head.

you should ask the questions with the book in mind.”

in my second call with John, I dropped certain probes, because it was clear that he wasn’t going to be able to put it into written form

My only regret is that I didn’t get to do the Simulation Interview, which I think prevented me from creating a Cognitive Demands table.

But I’m still rather happy for a first try — we both got a lot out of the Knowledge Audit alone.