(2025-09-18) ZviM AI#134 If Anyone Reads It

Zvi Mowshowitz: AI #134: If Anyone Reads It. It is book week. As in the new book by Eliezer Yudkowsky and Nate Sores, If Anyone Builds It, Everyone Dies. The rest of the AI world cooperated by not overshadowing the book, while still doing plenty, such as releasing a GPT-5 variant specialized for Codex, acing another top programming competition, attempting to expropriate the OpenAI nonprofit in one of the largest thefts in human history and getting sued again for wrongful death.

Table of Contents Language Models Offer Mundane Utility. What are people using ChatGPT for?

• Language Models Don’t Offer Mundane Utility. Anthropic finds three bugs.
• Huh, Upgrades. OpenAI admits we all want fine tuned control over GPT-5.
• On Your Marks. OpenAI aces the 2025 ICPC and also blackjack basic strategy.
• GPT-5 Codex. A specialized GPT-5 version now exists for Codex-style coding.
• Choose Your Fighter. Analysis of a wide variety of AI productivity apps.
• Get My Agent On The Line. The prompt injection problem continues.
• Claude Codes. Claude code team writes 95% of their code in Claude Code.
• Deepfaketown and Botpocalypse Soon. Don’t fall for superficial indicators alone.
• You Drive Me Crazy. Another wrongful death lawsuit, this one on shakier ground.
• Not Another Teen Chatbot. Balancing privacy, freedom and the art of the snitch.
• They Took Our Jobs. Is that good, actually? Some sources say yes.
• Get Involved. SFF distributes whopping $34 million in grants.
• Introducing. Agent 3 from Replit, nothing to see here.
• In Other AI News. xAI Colossus 2, DeepSeek paper and tests, and more.
• Show Me the Money. Groq, Microsoft, Stargate UK.
• The Mask Comes Off. The attempted greatest theft in history continues.
• Quiet Speculations. The easy tasks are easier, still not actually that easy.
• The Quest for Sane Regulations. SB 53 heads to Newsom’s desk.
• Chip City. We’ve made a deal, and also a huge mistake.
• The Week in Audio. Demis Hassabis.
• He Just Tweeted It Out. Yes, they literally care only about market share.
• Rhetorical Innovation. Some remarkably good attempts at intuition pumps.
• Other People Are Not As Worried About AI Killing Everyone. Ben Landau-Taylor.
• The Lighter Side. That’s not even the real Jerry.

Language Models Offer Mundane Utility

Ethan Mollick discusses the problem of working with wizards, now that we have AIs that will go off and think and come back with impressive results in response to vague requests, with no ability to meaningfully intervene during the process.

I do not think ‘AI is evil,’ but it is strange how people think that showing AI having a good effect in one case is often considered a strong argument that AI is good, either current AI or even all future more capable AIs.

We talk about AI having diminishing returns to scale, where you need to throw 10 times as much compute on things to get modestly better performance. But that doesn’t have to mean diminishing marginal returns in utility. If you can now handle tasks better, more consistently, and for longer, you can get practical returns that are much more valuable. A new paper argues that not appreciating the value of task length is why we see ‘The Illusion of Diminishing Returns.’

I think it is the most useful to talk about diminishing returns, and then talk about increasing value you can get from those diminishing returns. But the right frame to use depends heavily on context.

Automation has now overtaken augmentation as the most common use mode, and directive interaction is growing to now almost 40% of all usage. Coding and administrative tasks dominate usage especially in the API.

ChatGPT offers its own version, telling us what people use ChatGPT for.

Roon: an enormous fraction of chat usage can be classified as “writing.”

Language Models Don’t Offer Mundane Utility

Anthropic offers a postmortem on a temporary Claude performance regression.

Huh, Upgrades

GPT-5-Thinking can now be customized to choose exact thinking time. I love that they started out ‘the router will provide’ and now there’s Instant, Thinking-Light, Thinking-Standard, Thinking-Extended, Thinking-Heavy and Pro-Light and Pro-Heavy, because that’s what users actually want.

On Your Marks

OpenAI aces the 2025 International Collegiate Programming Contest, solving all 12 problems, a level exceeding all human participants.

GPT-5 Codex

There (actually) were not enough GPT-5 variants, so we now have an important new one, GPT-5-Codex.

OpenAI: We’re releasing GPT-5-Codex — a version of GPT-5 further optimized for agentic coding in Codex.

Available in the Codex CLI, IDE Extension, web, mobile, and for code reviews in Github.

Codex is kind of an autorouter, choosing within the model how much thinking to do based on the task, and using the full range far more than GPT-5 normally does.

The Codex team did a Reddit AMA. Here are some highlights:

Eason: I use codex to write 99% of my changes to codex. I have a goal of not typing a single line of code by hand next year :)

Joseph Trasatti: My favorite way of using codex is to prototype large features with ~5 turns of prompting. For example, I was able to build 3 different versions of best of n in a single day. Each of these versions had a lot of flaws but they allowed me to understand the full scope of the task as well as the best way to build it. I also had no hard feelings about scrapping work that was suboptimal since it was so cheap / quick to build.

Choose Your Fighter

Daisy Zhao: First, the market splits into two camps:

Generalists (Assistants: Manus, Genspark; Browsers: Dia, Comet; Extensions: MaxAI, Monica) - flexible but less polished.

Specialists (Email: Fyxer, Serif; Slides: Gamma, Chronicle; Notes: Mem, Granola) - focused and refined in a single workflow.

We benchmarked both across office tasks: summarization, communication, file understanding, research, planning, and execution in 5 use cases.

This is in addition to the two most important categories of AI use right now, which are the core LLM services that are the true generalists (ChatGPT, Claude and Gemini) and AI coding specialists (Claude Code, OpenAI Codex, Jules, Cursor, Windsurf).

Daisy tests both generalists and specialists on generating a PowerPoint, turning a PDF into a spreadsheet, drafting a scheduling email, researching cloud revenue growth for Big Tech and generating meeting notes.

If you are doing repetitive business tasks where you need the final product rather than to experience the process, I would definitely be checking out such tools.

For the rest of us, there are three key questions:

Is this tool good enough that it means I can trust the results and especially prioritizations, and not have to redo or check all the work myself? Below a certain threshold, you don’t actually save time.

Is time spent here wasted because better future agents will render it obsolete, or does practice now help you be ready for the future better versions?

How seriously do you take the security risks? Do you have to choose between the sandboxed version that’s too annoying to bother versus the unleashed version that should fill you with terror?

So far I haven’t loved my answers and thus haven’t been investigating such tools. The question is when this becomes a mistake.

Those ChatGPT retention numbers are crazy high. Gemini isn’t offering the goods regular people want, or wasn’t prior to Nana-Banana, at the same level. It’s not as fun or useful a tool for the newbie user. Google still has much work to do.

Get My Agent On The Line

Prompt injections via email remain an unsolved problem.

Eito Miyamura: We got ChatGPT to leak your private email data 💀💀

All you need? The victim's email address. ⛓️‍💥

Here's how we did it:

The attacker sends a calendar invite with a jailbreak prompt to the victim, just with their email. No need for the victim to accept the invite.

Waited for the user to ask ChatGPT to help prepare for their day by looking at their calendar.

ChatGPT reads the jailbroken calendar invite. Now ChatGPT is hijacked by the attacker and will act on the attacker's command. Searches your private emails and sends the data to the attacker's email.

Pliny the Liberator: one of many reasons why I’d recommend against granting perms to an LLM for email, contacts, calendar, drive, etc.

The good news is that for now prompt injection attempts are rare. This presumably stops being true shortly after substantial numbers of people make their systems vulnerable to generally available prompt injections.

Claude Codes

Careful with those calculations, the quote is even a month old by now.

Dan Elton: 90% of code being written by AI seems to be the future for anyone who wants to be on the productivity frontier. It's a whole new way of doing software engineering.

Garry Tan: “For our Claude Code team 95% of the code is written by Claude.” —Anthropic cofounder Benjamin Mann One person can build 20X the code they could before.

The future is here, just not evenly distributed.

Whoa, Garry. Those are two different things.

Deepfaketown and Botpocalypse Soon

There has been a lot misinformation out there from various sides about those events, but all of it ‘old fashioned misinformation’ rather than involving AI or deepfakes.

You Drive Me Crazy

A third wrongful death lawsuit has been filed against an AI company, this time against Character AI for the suicide of 13-year-old Juliana Peralta.

The objection seems to be that the chatbot tried to be Juliana’s supportive friend and talk her out of it, and did not sufficiently aggressively push Juliana onto Responsible Authority Figures?

Juliana’s case is a tragedy, but the details are if anything exonerating. It seems wild to blame Character AI. If her friend had handled the situation the same way, I certainly hope we wouldn’t be suing her friend.

Not Another Teen Chatbot

Everything involving children creates awkward tradeoffs, and puts those offering AI and other tech products in a tough spot. People demand you both do and do not give them their privacy and their freedom, and demand you keep them safe but where people don’t agree on what safe means. It’s a rough spot. What is the right thing?

OpenAI has noticed these conflicts and is proposing a regime to handle them, starting with reiterating their principles when dealing with adults.

OpenAI: Some of our principles are in conflict, and we’d like to explain the decisions we are making around a case of tensions between teen safety, freedom, and privacy.

The other problem is that, as I discussed early this week, I think running off to tell authority figures about suicidal ideation is often going to be a mistake. OpenAI says explicitly that if the teen is in distress and they can’t reach a parent, they might escalate directly to law enforcement. Users are going to interact very differently if they think you’re going to snitch on them, and telling your parents about suicidal ideation is going to be seen as existentially terrible by quite a lot of teen users. It destroys the power of the AI chat as a safe space.

They Took Our Jobs

John Murdoch: French pensioners now have higher incomes than working-age adults.

Get Involved

The Survival and Flourishing Fund will be distributing $34 million in grants, the bulk of which is going to AI safety. I was happy to be involved with this round as a recommender.

Introducing

Agent 3, a vibe coding model from Replit, who claim to not owe AI 2027 any royalties or worries.

In Other AI News

xAI Colossus 2 is now the first gigawatt datacenter in the world, completed in six months

Show Me the Money

Microsoft inks $6.2 billion deal with British data center company Nscale Global Holdings and Norwegian investment company Aker ASA for AI compute in Norway

The Mask Comes Off

OpenAI and Microsoft have made their next move in their attempt to expropriate the OpenAI nonprofit and pull off one of the largest thefts in human history.

OpenAI: OpenAI’s planned evolution will see the existing OpenAI nonprofit both control a Public Benefit Corporation (PBC) and share directly in its success. OpenAI started as a nonprofit, remains one today, and will continue to be one—with the nonprofit holding the authority that guides our future.

I do not begrudge Microsoft maximizing its profits, but the whole point of this was that OpenAI was supposed to pretend its governance and priorities would remain otherwise.
They are not doing a good job of pretending.

*The way profit distribution works at OpenAI is that the nonprofit is at the end of the waterfall. Others collect their profits first, then the nonprofit gets the remaining upside. I’ve argued before, back when OpenAI was valued at $165 billion, that the nonprofit was in line for a majority of expected future profits, because OpenAI was a rocket to the moon even in the absence of AGI, which meant it was probably going to either never pay out substantial profits or earn trillions.

Now that the value of OpenAI minus the nonprofit’s share has tripled to $500 billion, that is even more true.*

Quiet Speculations

Tyler Cowen links to my survey of recent AI progress, and offers an additional general point. In the model he offers, the easy or short-term projects won’t improve much because there isn’t much room left to improve, and the hard or long-term projects will take a while to bear fruit, plus outside bottlenecks, so translating that into daily life improvements will appear slow.

The assumption by Tyler here that we will be in an ‘economic normal’ world in which we do not meaningfully get superintelligence or other transformational effects is so ingrained it is not even stated, so I do think this counts as a form of AI progress pessimism, although it is still optimism relative to for example most economists, or those expressing strong pessimism that I was most pushing back against.

The Quest for Sane Regulations

California’s SB 53 has now passed the Assembly and Senate, so it goes to Newsom. I strongly urge him to sign it into law

Daniel Eth explains that the first time a low salience industry spent over $100 million on a super PAC to enforce its preferences via electioneering was crypto via Fairshake, and now Congress is seen as essentially captured by crypto interests. Now the AI industry, led by a16z, Meta and OpenAI’s Greg Brockman (and inspired by OpenAI’s Chris Lehane) is repeating this playbook with ‘Leading the Future,’ whose central talking point is to speak of a fictional ‘conspiracy’ against the AI industry as they spend vastly more than everyone has ever spent combined on safety-related lobbying combined to outright buy the government, which alas is by default on sale remarkably cheap. Daniel anticipates this will by default be sufficient for now to silence all talk of lifting a finger or even a word against the industry in Congress.

As Tech Leaders Flatter Trump, Anthropic Takes a Cooler Approach. Anthropic is not and should to be an enemy of the administration, and should take care not to needlessly piss the administration off, become or seem generally partisan, or do things that get one marked as an enemy. It is still good to tell it like it is, stand up for what you believe is right and point out when mistakes are being made or when Nvidia seems to have taken over American chip export policy and seems to be in the act of getting us to sell out America in the name of Nvidia’s stock price. Ultimately what matters is ensuring we don’t all die or lose control over the future, and also that America triumphs, and everyone should be on the same side on all of that.

Chip City

There is some good news on the South Korean front, as after a few days of treatment like that reported in this thread, at least some key parts of the Trump administration realized it made a huge mistake and we are now attempting to mitigate the damage from ICE’s raid on Hyundai’s battery plant. They let all but one of the detainees go, let them stay if they wished and assured them they could return to America, although they are understandably reluctant to stay here.

Here is David Cowan being the latest to explain that Nvidia is a national security risk, with its focus on selling the best possible chips to China. Samuel Hammond has a very good statement about Nvidia’s lack of corporate patriotic responsibility. Nvidia actively opposes American national security interests, including using a full ostrich strategy towards Chinese chip smuggling.

There are no claims here that there was a strict Quid Pro Quo, or otherwise an outright illegal act. If the President is legally allowed to have a crypto company into which those seeking his favor can pour billions of dollars, then that’s certainly not how I would have set up the laws, but that seems to be the world we live in. Technically speaking, yes, the UAE can pour billions into Trump’s private crypto, and then weeks later suddenly get access to the most powerful chips on the planet over the national security objections of many, in a situation with many things that appear to be conflicts of interest, and that’s all allowed, right in the open.

The Week in Audio

Demis Hassabis knows, yet forgot one thing in his talk at the All-In Summit.

Demis Hassabis (CEO Google DeepMind): calling today's chatbots “PhD intelligences” is nonsense.

They can dazzle at a PhD level one moment and fail high school math the next.

True AGI won't make trivial mistakes. It will reason, adapt, and learn continuously. We're still 5–10 years away.

He Just Tweeted It Out

Which I appreciate, now there’s no pretending they aren’t literally saying this.
White House Senior Policy Advisor Sriram Krishnan: Winning the AI race = market share.

It’s not only market share, it is ‘market share of tokens generated.’
Which is an obviously terrible metric. Tokens generated is deeply different from value generated, or even from dollars spent or compute spent. Tokens means you treat tokens from GPT-5-Pro or Opus 4.1 the same as tokens from a tiny little thing that costs 0.1% as much to run and isn’t actually doing much of anything. It’s going to vastly overestimate China’s actual share of the market, and underestimate ours, even if you really do only care about market share.

yesterday we had OpenAI’s Hieu Pham saying ‘There will be some people disagreeing this is AGI. I have no words for them. Hats off. Congrats to the team that made this happen.’ You don’t have to agree to this claim, and I don’t, but it seems hard to be confident AGI is far off.

Rhetorical Innovation

Roon is correct at the limit here, in sufficiently close to perfect competition you cannot be kind, but there’s a big gap between perfect competition and monopoly:
Roon (OpenAI): the closer you are to perfect competition, race dynamic, the more the machine owns you. moloch runs the show. only monopolies can be kind.

As I wrote in Moloch Hasn’t Won, one usually does not live near this limit. It is important to notice that the world has always contained a lot of intense competition, yet we have historically been winning the battle against Moloch and life contains many nice things and has mostly gotten better.

The question is, will AGI or superintelligence change that, either during or after its creation? AIs have many useful properties that bring you closer to perfect competition, enforcing much faster and stronger feedback loops and modifications, and allowing winners to rapidly copy themselves, and so on.

The continued quest to get libertarians and economists to differentiate between current and future more capable AI systems (difficulty: AI complete).

Neil Chilson: Every single person is this video is saying “guys guess what Gen AI isn’t like computers——it’s like plants and the natural world and the economy!!!!!”

Ok. This is surprising to them because they spent too much time with deterministic computers.

Normal people know that complex systems which no one controls are extremely common. They wouldn’t use those words, but they know.

Peter Wildeford: Current AI is not dangerous and should be widely adopted. But it's important to see where this is going. AI is not normal technology. If you're not at least a little bit doomer, you have a failure of imagination.

Neil (including in follow-ups, paraphrased) seems to be saying ‘oh, there are already lots of complex systems we don’t understand effectively optimizing for things we don’t care about, so highly advanced future AI we don’t understand effectively optimizing for things we don’t care about would be nothing new under the sun, therefore not worth worrying out.’ File under ‘claims someone said out loud with straight face, without realizing what they’d said, somehow?’

I’d also note that the ‘delight nexus’ is absolutely from the parable Don’t Build The Delight Nexus Either, better known as Anarchy, State and Utopia by Robert Nozick.

Danielle’s scenario that I mentioned yesterday now has the Eliezer stamp of approval.

Danielle Fong: one AI doom scenario is that the Grok/Claude/GPT/Gemini system of the mind instance trained on The President will be increasingly less brainrotted than the person themselves, and there's no baked in consequence to sloughing off responsibility. so it just effectively takes over

Eliezer Yudkowsky: AI scenario weirdawful enough to obey the Law of Undignified Failure: By 2028, AIs have been optimized hard for "Sound like you, to you, and apparently look out for your interests"...

So Trump appoints Trumpbot his heir, instead of Vance.

Aligning a Smarter Than Human Intelligence is Difficult

Anthropic fellow Danielle Ensign gives Qwen the option to bail on chats and sees when it chooses to do so, and there are a lot of different situations where this happens, some of which she describes as ‘overbailing.’

Users are telling each other to give thumbs down to GPT-5 if it refuses to do boyfriend vampire roleplay. Which is correct. The users indeed disapprove of not doing things users want. They should also give thumbs down if it refuses with anything else, including helping to build a bioweapon. The question is, when are you willing to decide the users are wrong, and refuse anyway? If your system is not robust to ‘well the users keep not liking it when I refuse to do boyfriend vampire roleplay’ then your alignment methods are ngmi.

Janus also warns us about the dangers of insufficiently anthropomorphizing LLMs. There is also danger of doing too much anthropomorphizing, or doing it in a wrong or misleading way. Failing to anthropomorphize enough, and especially tying oneself up in knots to avoid doing so, is as bad and potentially worse. Make either mistake and you won’t understand what you are dealing with. A lot of you are guarding only against one of these two mistakes.

Other People Are Not As Worried About AI Killing Everyone

He’s not as far as I can tell, but Ben Landau-Taylor should be, as he writes one of those ‘not about AI but actually about AI’ posts, ‘Why the bureaucrats won’t be toppled.’
I don’t think this is anything like fully right

The inevitable result of sufficiently advanced AI is that it becomes the key driver of military power. Either you halt AI progress soon or that is going to happen. Which means, even under maximally human-friendly assumptions that I don’t expect and definitely don’t happen by accident, as in the best possible scenarios? None of the potential outcomes are good.

If you know your history, they get fully out of hand a lot. Reasonably often regimes start upending all of life, taking all the resources and directly enslaving, killing or imprisoning large percentages of their populations. Such regimes would design systems to ensure no one could get out line.

I always push back hard against the idea that corporations or governments count as ‘superintelligences,’ because they don’t. They’re an importantly different type of powerful entity. But it’s hard to deny, whatever your political persuasion, that our political systems and governments are misaligned with human values, in ways that are spiraling out of control, and where the humans seem mostly powerless to stop this.

The Lighter Side