(2023-02-21) Willison In Defense Of Prompt Engineering

Simon Willison: In defense of prompt engineering. I’ve seen two subtly different meanings for that term:

The argument I see against both of these is the same: as AI language models get “better”, prompt engineering as a skill will quickly become obsolete.

I disagree.

First, you need really great communication skills. Communicating clearly is hard!

When communicating with other people, the first step is to figure out their existing mental model—what do they know already, what jargon is appropriate, what details are they missing? Talking to a language model has similar challenges

You need to be able to methodically apply a version of the scientific method to your work. Figuring out what works and what doesn’t

The best prompt engineers are meticulous: they constantly run experiments, they make detailed notes on what works and what doesn’t, they iterate on their prompts and try to figure out exactly which components are necessary for the prompt to work and which are just a waste of tokens.

We’re into real Polymath territory here.

Comparisons to programming are interesting. With programming, you have a fixed, deterministic target.

That’s not the case with language model prompts.

When GPT-4 comes out, how will that affect the prompts you wrote for GPT-3?

I’ve talked about the comparison to fictional magic before. I don’t think prompt engineers should be thought of as wizards, but it’s definitely true that they’re working in a weird field where the rules are not fixed and the consequences of getting things wrong are potentially serious.