(2024-04-26) Gilad On GenML Artifacts And Product Management

Itamar Gilad On GenML, Artifacts, and Product Management. The latest wave of GenML/GenAI tools is truly remarkable. I believe all creative work is going to be impacted, including product management. But in what way?

One answer is already on offer. In recent months Twitter and LinkedIn have been flooded with “definitive” lists of ChatGPT prompts designed to produce product artifacts: Objectives and Key Results, user stories, market size assessments, strategic models, user interview scripts…

Sounds almost too good to be true. Maybe, but I feel we should proceed with great caution. The ease of producing artifacts may come with some serious tradeoffs.

let’s look at an example.

The Elusive Big Idea

few years ago I consulted a small company developing AdTech products for social media. The company had a number of existing SaaS products, but none had found product/market fit and usage was very low. The problem wasn’t hard to spot. The founders kept launching projects around promising ideas without much validation.

To help, I introduced the product team to Strategizer’s Business Model Canvas (BMC) — a powerful tool for assessing new product ideas.

The next time the CEO came up with a new must-have idea, one of the product managers created a draft BMC within a couple of days and took it to management review. Looking at the BMC brought the real potential and costs of the idea into focus. The conclusion was that the idea wasn’t quite as strong as first thought and likely would never justify its cost. The management team decided to drop the idea, and everyone, including the CEO, was happy with this decision.

the product manager had to conduct research, analysis, and to produce some estimates. For example, he had to consider which types of ad agencies had a strong need for this product. While the answers were mostly guesswork, these were smaller, and arguably better guesses compared to the big looming question “is this a good idea?”.

While the artifact is an important part of this process, it’s just a communication tool. The goal is never just to produce the artifact.

Enter the Robots

Now let’s imagine this same scenario with ChatGPT.

The bot-BMC is full of potentially useful content, but there are two interesting points to note: a) There are no TBDs, no Ifs, and no Maybes. It’s a definitive, complete, and confident artifact. Why? Because that’s what the current GenML text bots are designed to do

It’s likely that the BMC will make the idea look good, simply because the bot has probably been trained on positive examples rather than negative ones. You can ask the bot to produce both pro and con BMCs, and it will happily comply, but will anyone do that? And which one should you use?

The PM shares the BMC with the team and with management. We’re creating a shared understanding, but it’s heavily biased by what the bot produced, which is based on the average thing people tend to say.

Contrary to what you might think, this is a weak signal. Common wisdom may not apply to your particular case, and many good ideas defy conventional thinking.

We see them as a convenient shortcut, but we’ll still expect to think for ourselves. The problem is that the bots offer a compelling temptation. Research, analysis, estimation. review, and decision-making require what Psychologist Daniel Kahneman calls System-2 thinking —slow, effortful, intentional, and tiring contemplation. In the book Thinking Fast and Slow Kahneman explains that most of us avoid using our System-2 thinking when there are convenient shortcuts available, even if those lead us to the wrong answer. GenML just created a whole new class of mental shortcuts for us to fall back on. (Not-Thinking Tool)

The Artifact Factory

But there may be an even more pernicious aspect to these bot artifacts. In some organizations product managers (now sometimes called Product Owners) work in feature factories (feature factory) that are measured on rate of output. The PMs/POs themselves are seen as a sort of 1-person “artifact factory”

These product managers are under high pressure to produce, and I suspect the temptation to rely on the bots is going to be even higher in their case

Taking one logical step further, we can imagine outsourcing artifact production to external agencies as a cost-cutting measure

The Bots Aren’t The Problem

I’m optimistic that GenML will empower us to create better products quicker. Still, we shouldn’t underestimate the ability of people and organizations to misuse, overuse, and abuse tools and products.

In some organizations saying things that aren’t necessarily true in a compelling and polished way will get you fired; in others it’ll get you promoted to management.