(2022-07-27) Chin John Cutlers Product Org Expertise
Cedric Chin: John Cutler's Product Org Expertise. Cutler’s superpower is that he is able to talk to a set of product people and — within 8-20 minutes — figure out the system dynamics of their organisation, and then suggest a bunch of experiments to make that system better. His interventions usually take on the form of a prioritised list of experimental changes or team interventions.
Sometime in the last year, he attempted to write a book about his skills. And then he came up against a wall.
Cutler’s expertise is intuitive and context-dependent
I told John that there was a whole set of techniques designed to extract tacit mental models of expertise, and that he was welcome to give them a try. I said that the simplest technique was something called ACTA.
A Quick Recap of ACTA
ACTA is a protocol of three interview methods and a presentation format
- You first create a task diagram
- You do a knowledge audit
- You do a simulation interview
- Finally, you present the extracted information in a cognitive demands table.
Due to my lack of experience using the technique, I only performed the first two tasks with John
In retrospect, perhaps I should have asked John for a recording of a product team engagement
What We Extracted
Cutler’s product calls consist of the following four stages
If by the 20 minute mark he isn’t able to map out a few problems, this is likely a rare edge case — a unique organisation that he can't help.
We identified Steps 2 and 3 as the parts with the highest cognitive demands:
Step 2, Problem Detection — A novice would not have John’s intuition on what to dig deeper into, and what to ignore
Step 3, Solution Delivery — Some product team problems are easy, or ‘acute’, and these tend to be simpler to diagnose and then fix
But some product team problems are ‘chronic’, or more accurately described as complex systems problems, where multiple things are bad all at once (e.g. bad processes, and no decision power, and technical debt, and bad org infrastructure, and a toxic leader, etc)
Of the two steps, we decided to focus on Step 2, Problem Detection
The Knowledge Audit
We ended up doing five probes
Aspects of Expertise Cues and Strategies Why Difficult? Past & Future
John's first pass is always to detect anti-patterns
Then, work backwards from the anti-patterns to figure out how the org got here.
First detect if there is high Work In Progress (WIP).
If high WIP exists, then he works backwards to figure out how the org got there
60% of the time, John says, a bad org has high WIP
The other 40% of the time, John is able to quickly suss out other possible contributors to those problems
Decision Authority (Agency)
where does this org lie on the extremes of product ownership?
Organisational learning ability
can talk about current hypotheses and experiments in play: “tell me your last 5 big product decisions”
Seriousness of product engagement
if this is a mainline activity or a vanity project
judge how important the product team is to the business
Not enough experiments that fail = company isn’t taking enough risks
Is the company centralised, where HQ sets the culture
Demographic / country culture
Check, in the following order: is there a strategy or not? -> is it halfway plausible or not? -> is it implicit or explicit? -> Is the structure of the company aligned around the strategy of the company/product?
A novice (column repeats with what novice would get wrong for each item)
Notice the team’s comfort with uncertainty
A good product team has productive things to say about what is unknown ("we know this feature would do well, but we're not sure about that feature, but we have experiments going on for it")
Notice when the company has ‘flow’.
do they have clean data, do they have the ability to interpret the data, the ability to link cause and effect
Able to notice when a product leader is at the stage where they are naive
enamoured by product frameworks
Quickly places the team on a spectrum, because John knows the extremes of each factor, and the contexts where they may or may not work
Comfort with uncertainty column repeats
Able to ask ‘Powerful Questions’ to cut through the mess
Use humour to get a reaction.
John jokes about quarters being weird
Novice would not know column repeats
John knows the weaknesses of his approach, and is able to tell when he is unable to help the org
What Was Difficult?
The biggest difficulty with using ACTA was in deciding when to dive deep and let John go on about his experiences, and when to move on to the next set of probes
I think the biggest mistake that I did was that I grossly underestimated the amount of time it would take to do the whole skill extraction exercise.
The other problem I had with the process was that I was unsure if I was producing useful information. John seemed happy with our progress, but I wasn't so sure.
After our first call, but before our second, I had the good fortune of talking to Laura Militello and Brian Moon, hosts of the NDM podcast. Militello, of course, was one of the two creators of ACTA, and when I asked her if she had any tips for using it, she said to keep the objective of the interview at the back of my head.
you should ask the questions with the book in mind.”
in my second call with John, I dropped certain probes, because it was clear that he wasn’t going to be able to put it into written form
My only regret is that I didn’t get to do the Simulation Interview, which I think prevented me from creating a Cognitive Demands table.
But I’m still rather happy for a first try — we both got a lot out of the Knowledge Audit alone.