(2025-02-12) Sloan Is It Okay Bjarnason

Robin Sloan: Is it okay? How do you make a (large) language model? Goes like this: erect a trellis of code, then allow the real program to grow, its development guided by a grueling training process, fueled by reams of text, mostly scraped from the internet.

Is that okay?

I’d like to proceed by depriving each side of its best weapon.

On the side of “yes, it’s okay”, I will insist that the analogy to human learning is not admissible

On the side of “no, it’s not okay”, I will put aside any arguments grounded in copyright law

Instead, I’ll defer to deeper precedents: the intuitions and aspirations that gave rise to copyright in the first place. To promote the Progress of Science and useful Arts, remember?

language models are not machines “trained on” human writing. They are the writing, granted the ability to speak for itself

To make this work — you already know this, but I want to underscore it — only a truly vast trove of writing suffices

Just as, above, I set copyright aside, I want also to set aside fair use and the public domain. Again, not because they are irrelevant, but because those intuitions and frameworks all assume we are talking about using some part of the commons — not all of it.

we can inquire: how much of the value of these models comes from Everything?

For the foundation models like Claude, data means: Everything.

While I believe the relative value of Everything in this mix is something close to 90%, I’m willing to concede a 50/50 split.

And here is the important thing: there is no substitute.

We know the companies are commissioning content, lots of it, across all sorts of business and technical domains.

it is precisely the naivete of Everything, the fact that its writing was actually produced for all of those different reasons, that makes it so valuable

one judgment becomes clear:
If their primary application is to produce writing and other media that crowds out human composition, human production: no, it’s not okay

But what if that isn’t the primary application? Maybe language models can grow beyond writing corporate emails; maybe they can broaden and deepen to become significant contributors to science and technology, to medicine and more.

This is tricky — it’s so, so tricky — because the claim is both (1) true, and (2) convenient

If super science is a possibility — if, say, Claude 13 can help deliver cures to a host of diseases — then, you know what?

Would I sacrifice anything, or everything, for super science? No. But art and media can find new forms. That’s what they do.

Obviously, this scenario is especially appealing if the super science, like Everything at its foundation, flows out into the commons. It could. Depends who’s doing the science.

think the chance is probably above ten percent, so, I remain curious.

I’ll sketch out my current opinions more specifically:

I think the image generation models, trained on the Everything of pictures, are: probably not okay. They don’t do anything except make more images. They pee in the pool.

I think the foundation models like Claude are: probably okay

The case of translation is compelling. If language models are, indeed, the Babel fish, they might justify the operationalization of the commons even without super science

I think the case of code is especially clear, and, for me, basically settled. That’s both (1) because of where code sits in the creative process, as an intermediate product, the thing that makes the thing, and (2) because open-source code has carried the expectation of rich and surprising reuse for decades

One extreme: if these machines consume all media, and then, in their deployment, produce a world in which there is no functioning market for humans to produce media, I’d say, that’s obviously bad. It’s not what we want from technology, or from our future.

Another extreme: if these machines consume all media, and then, in their deployment, provide several superconductors and cure all cancers, I’d say, okay … we’re good.

Baldur Bjarnason: Knowledge tech that's subtly wrong is more dangerous than tech that's obviously wrong. (Or, where I disagree with Robin Sloan.)

I disagree with pretty much both the core premise and every step of the reasoning of it.
And, unfortunately, because I’ve long ben a fan, I feel obligated to explain why

first off, the core framing of the argument is wrong. It portrays the core arguments made for and against as inherently wrong, that you can’t have reached an unambiguous “LLMs are wrong” stance through proper reasoning.

the most consistent, most forceful, and least ambiguous warning cries about LLMs have come from AI and Machine Learning academics like Timnit Gebru, Dr Abeba Birhane, Emily M. Bender, and Dr. Damien P. Williams, (I’m using “Dr” with the name based on how they represent themselves on social media) just to name a few. They have come to their conclusion precisely because they have thought about the topic with “sufficient sensitivity and imagination”, not to mention extensive domain knowledge, deep understanding of how the tech works, and how these models interact with the larger context of society and culture.

The model is not the data set

that LLMs are all of writing (emphasis original).

a model of a data set is not the data set. To say that a language model is the writing is equivalent to saying that cyanide is an apple just because you can get cyanide from processing apple seeds.

assumptions made about the data set. Namely, that text on the internet is a fair representation of the entirety of human writing.

Text on the web is shaped by the web, its culture, form, incentives, and economics

And most of what gets delivered is outright poisonous.
Pornography
Violence
Abuse

tools that have magnified biases and errors which are wrapped in the capability to replicate various formal and academic writing styles that we associate with truth, factuality, and meaning.

This is what makes LLMs so dangerous.

That means that the harm done by these systems compound the more widely they are used as errors pile up at every stage of work, in every sector of the economy

It’s as if homeopathy and naturopathy got adopted as standard practices in the healthcare system

That includes the current poster child of “LLMs are awesome”: coding assistants. These tools have the same error rates, repetition, and biases as other LLM applications and they consistently perpetuate harmful or counterproductive practices from the past (like an over-reliance on React or poor accessibility).

Aside: calling LLMs for translation “Babel Fish” is glossing over some of the major flaws in these tools work for translation, which is that they only sorta kinda work for languages with a large body of text in the training data set

And even when it does kinda sorta work, the reader is doing a lot of heavy lifting by interpreting the model’s incredibly random and shitty translations based on the context.

Finally, the last resort of the tech booster, complete and utter science fiction

If super science is a possibility—if, say, Claude 13 can help deliver cures to a host of diseases

This is science-fiction. There is no path that can take the current text synthesis models and turn them into super-scientists.

Robin Sloan: Science fiction

Baldur Bjarnason blogs about my post on the foundational question of language models

So: I think Baldur misreads me a bit, but/and on balance his consideration is sharp, and I’m very grateful for it.

*Regarding the vision of super science empowered by advanced language models, he writes:

This is science-fiction. There is no path that can take the current text synthesis models and turn them into super-scientists.*

First: yes, it is precisely science fiction. Three years ago, the vision of a fluent, formidable software correspondent was science fiction, too

Second: I don’t think it’s possible to say, with airless certainty, that “there is no path” from language models as we know them to super scientists or super scientist-enablers.

It’s totally possible these models won’t ever be useful for science

But none of them seem to me to rise to the level of like, epistemological showstopper.

Baldur: Now I'm disappointed.

So, Robin Sloan replied to my post that was responding to his post as a serious piece of “AI” commentary.
Taking the original post seriously may have been a mistake.

If you make vague statements that replicate the arguments made by “AI” company CEOs who are angling for hundreds of billions of dollars of investment money, then I will read that argument as one that’s either supporting or furthering said CEO’s agenda.

First: yes, it is precisely science fiction. Three years ago, the vision of a fluent, formidable software correspondent was science fiction, too.

This may look true from outside the field, but if you go through the academic work on deep learning and language models over the years there was a clear arc of improvement in fluency and natural language processing

Every time these capabilities have been put to thorough peer-reviewed testing, their advancements have been shown to be in line with the general expectations of the field.

Saying “we will probably get super-scientist AI” in 2025 is, unlike a 2019 statement on fluent chatbots, not a reasonable speculation to make based on current scientific and academic research. It is not based on anything except hyperbole from people with vested interests in inflating an investment bubble.

More importantly, it’s also propaganda.

It’s specifically propaganda that is furthering and enabling some of the worst actions of the US government and tech companies:

• The US executive’s ongoing attacks on research funding.
• Their de-funding of education.
• Censorship of scientists and educators (why not, since they’re all about to get replaced with AI?)
• Mass layoffs at tech companies (who needs workers when docile AI labour is just around the corner?).

Giving a fact-free statement like “LLMs will give us super-science, so we should give them a few years of free rein” the benefit of the doubt when they’re quite obviously going to be spending those years looting and irreversibly harming our economies is deeply irresponsible.