(2025-04-03) ZviM AI #110 Of Course You Know

Zvi Mowshowitz: AI #110: Of Course You Know... Yeah. That happened yesterday. This (Trump Tariffs) is real life. I know we have to ensure no one notices Gemini 2.5 Pro, but this is rediculous. That’s what I get for trying to go on vacation to Costa Rica, I suppose. I debated waiting for the market to open to learn more. But f** it, we ball.*

Table of Contents

• Is AI Now Impacting the Global Economy Bigly? Asking the wrong questions.
• Language Models Offer Mundane Utility. Is it good enough for your inbox yet?
• Language Models Don’t Offer Mundane Utility. Why learn when you can vibe?
• Huh, Upgrades. GPT-4o, Google Gemini 2.5 Pro, and we partly have Alexa+.
• On Your Marks. Introducing PaperBench. Yes, that’s where we are now.
• Choose Your Fighter. How good is ChatGPT getting?
• Jevons Paradox Strikes Again. Compute demand is going to keep going up.
• Deepfaketown and Botpocalypse Soon. The only answer to a bad guy with a bot.
• They Took Our Jobs. No, AI is not why you’ll lose your job in the short term.
• Get Involved. Fellowships, and the UK AISI is hiring.
• Introducing. Zapier releases its MCP server, OpenAI launches AI Academy.
• In Other AI News. Google DeepMind shares 145 page paper, but no model card.
• Show Me the Money. The adventures of the efficient market hypothesis.
• Quiet Speculations. Military experts debate AGI’s impact on warfare.
• The Quest for Sane Regulations. At what point do you just give up?
• Don’t Maim Me Bro. Further skepticism that the MAIM assumptions hold.
• The Week in Audio. Patel on Hard Fork, Epoch employees debate timelines.
• Rhetorical Innovation. As usual it’s not going great out there.
• Expect the Unexpected. What are you confident AI won’t be able to do?
• Open Weights Are Unsafe and Nothing Can Fix This. Oh no, OpenAI.
• Anthropic Modifies its Responsible Scaling Policy. Some small changes.
• If You’re Not Going to Take This Seriously. I’d prefer if you did?
• Aligning a Smarter Than Human Intelligence is Difficult. Debating SAEs.
• Trust the Process. Be careful exactly how much and what ways.
• People Are Worried About AI Killing Everyone. Elon Musk again in brief.
• The Lighter Side. Surely you’re joking, Mr. Human. Somehow.

The New Tariffs Are How America Loses

The new ‘Liberation Day’ tariffs are suicidal insanity. Congress must act to revoke executive authority in such matters and reverse this lunacy before it is too late. When you realize how the tariffs were calculated, it’s even crazier.

This hurts even American manufacturing, because we are taxing imports of the components and raw materials we will need, breaking our supply chains and creating massive uncertainty. We do at least exempt a few inputs like copper, aluminum and steel (and importantly for AI, semiconductors), so it could be so much worse, but it is still unbelievably awful.

If we were doing actual ‘reciprocal tariffs’ where we set our tariff rate equal to their tariff rate, including 0% if theirs was 0%, I would be actively cheering. Love it.

This is very much not any of that. We know exactly what formula they actually used, which was, and this is real life: (exports-imports)/exports. That’s it. I’m not kidding. They actually think that every bilateral trade relationship where we have a trade deficit means we are being done wrong and it must be fixed immediately.

The declines we are seeing in the stock market reflect both that a lot of this was previously priced in, and also that the market is still putting some probability on all of this being walked back from the brink somehow. And frankly, despite that, the market is underreacting here. The full effect is much bigger.

I mention this up top in an AI post despite all my efforts to stay out of politics, because in addition to torching the American economy and stock market and all of our alliances and trade relationships in general, this will cripple American AI in particular.

The other reason to mention this up top is, well…

Is AI Now Impacting the Global Economy Bigly?

Rohit: This might be the first large-scale application of AI technology to geopolitics.. 4o, o3 high, Gemini 2.5 pro, Claude 3.7, Grok all give the same answer to the question on how to impose tariffs easily.

This is Vibe Governing.

Eli Dourado: First recession induced by an unaligned AI.

It is impossible to talk to any frontier LLM about this and not have it be screaming at you how horrible an idea this is. Claude even nailed the recession probability at 45%-50% in 2025

To be clear, it seems unlikely this was actually the path through causal space that got us the tariffs we got. But it’s scary the extent to which I cannot rule it out.

Language Models Offer Mundane Utility

AI is highly useful in fighting denials of insurance claims. Remember to ask it to respond as a patio11-style dangerous professional.

Dartmouth runs a clinical trial of “Therabot,” and they’re spectacular, although N=106 means I wouldn’t get overexcited yet. (Digital Therapeutics)

Amy Wu Martin: Dartmouth just ran the first clinical trial with a generative AI therapy chatbot—results: depression symptoms dropped 51%, anxiety by 31%, and eating disorders by 19%.
“Therabot,” built on Falcon and Llama, was fine-tuned with just tens of thousands of hours of synthetic therapist-patient dialogue

If nothing else, being able to be there when the patient needs it, always ready to answer, never having to end the session for time, is an epic advantage.

Language Models Don’t Offer Mundane Utility

Huh, Upgrades

Google Gemini 2.5 Pro is now available to all users on the Gemini app, for free, with rate limits and a smaller context window if you aren’t subscribed

OpenAI updated GPT-4o.

Altman claims it’s a big upgrade. I don’t see anyone else talking about it.

Alexa+ launched on schedule, but is missing some features for now, some to be delayed for two months. At launch, it can order an Uber, but not GrubHub, and you can’t chat with it on the web, unless you count claude.ai. It sounds like things are not ready for Amazon prime time yet.

*Pliny the Liberator suggests: “write a prompt for {insert-query-here} then answer it”

On Your Marks

OpenAI releases PaperBench, tasking LLMs with replicating top 2024 ICML papers, including understanding it, writing code and executing experiments

In other testing news, do the Math Olympiad claims hold up? Zain shows us what happens when we tested LLMs on the 2025 Math Olympiad, fresh off the presses, and there were epic fails everywhere (each problem is out of 7 points, so maximum is 42, average score of participants is 15.85)…

except Google Gemini 2.5 Pro, which came out the same day as the benchmark, so they ran that test and got 24.4% by acing problem 1 and getting 50% on problem 4.

Janus has expressed more disdain for benchmarks than any other person I know, so here’s what Janus says would be an actually good benchmark.

Choose Your Fighter

Stephanie Palazzolo: This isn't an April Fools joke: ChatGPT revenue has surged 30% in just three months

You are very much missing out if you only use ChatGPT, even if you are paying the $200 a month. And I think most people are substantially better off with $60 a month split between Gemini, Claude and ChatGPT than they would be paying the $200, especially now with Gemini 2.5 Pro available.

What should you use for coding now?

Reactions seem to strongly endorse a mix of Gemini and Claude as the correct strategy for coding right now.

Jevons Paradox Strikes Again

Demand for compute is only going to go up as compute use gets more efficient. We are nowhere near any of the practical limits of this effect.

Deepfaketown and Botpocalypse Soon

Could AI empower automatic community notes? This seems like a great idea. If a Tweet gets a combination of sufficiently many hits and requests for a fact check, you can do essentially a Deep Research run

For a while, we have had a severe ‘actually you are playing against bots’ problem.
Grant Slatton: was playing a silly multiplayer web game a few weeks ago (an agario clone) and after a few minutes realized 95% of the players were bots

This is mostly not about LLMs or even decent AI, and far more about:
The new player experience often wants to be scripted.
People want to be matched instantly.
People want to win far more than their fair share (typically ~60% is good).
People often need a sandbox in which to skill up.

It’s happening department:
Near Cyan: ~50 people have DM’d me saying [Auren] made them cry and rethink their life by talking to a bunch of LLMs for a few hours

Is Claude better than the best human therapists? no certainly not. I can definitely do better, I know a few others who could too. But it's intensely g-loaded and even if they didn't have other careers available there just wouldn't be enough of them to sate demand. (chatbot, digital therapeutics)

They Took Our Jobs

During the early phase of them taking our jobs, this theory makes sense to me, too:
Dean Ball: I do not expect widespread, mass layoffs due to AI in the near term.
But I worry that as knowledge workers exit firms, they won’t be replaced.

Get Involved

Introducing

Zapier releases its MCP server, letting you connect essentially anything, to your Cursor agent or otherwise. Paul Graham retweeted Brendan giving the maximalist pitch here:
Brendan: Zapier just changed the game for AI builders.
You can now connect your AI assistant to 8,000+ apps with zero API integration.
No OAuth struggles. No glue code. No custom backends.

In Other AI News

Your reminder that you should absolutely keep any promises you make to AIs, and treat them the same way you would treat keeping promises to humans.
This is more important for promises to LLMs and other AIs than those made to your fridge, but ideally this also includes those too, along with promises to yourself, or ‘to the universe,’ and indeed any promise period. If you don’t want to make the promise, then don’t make it.

Show Me the Money

Quiet Speculations

The Quest for Sane Regulations

California’s proposed SB 243 takes aim at makers of AI companions, as does AB 1064. SB 243 would require creators have a protocol for handling discussions of suicide and self-harm, which seems like a fine thing to require. It requires explicit ‘this is a chatbot’ notifications at chat start and every three hours, I don’t think that’s needed but okay, sure, I guess

AB 1064 would instead create a ‘statewide standards board’ to assess and regulate AI tools used by children, we all know where that leads and it’s nowhere good. Similarly, age verification laws are in the works, and those are everywhere and always privacy nightmares.

Don’t Maim Me Bro

Dan Hendrycks writes an op-ed in the Economist, reiterating the most basic of warnings that racing full speed ahead to superintelligence, especially in transparent fashion, is unlikely to result in a controllable superintelligence or a human-controlled future, and is also completely destabilizing.
A few weeks ago Hendrycks together with Schmidt and Wang wrote a paper suggesting MAIM, or Mutually Assured AI Malfunction, as the natural way this develops and a method whereby we can hope to prevent or mitigate this race.

Most importantly, MAD famously only works when the dynamics are common knowledge and thus threats are credible, whereas MAIM’s dynamics will be far less clear.

The Week in Audio

Rhetorical Innovation

Scott Wolchok correctly calls out me but also everyone else for failure to make an actually good definitive existential risk explainer. It is a ton of work to do properly but definitely worth doing right.

QC: the epistemic situation around LLM capabilities is so strange. afaict it's a publicly verifiable fact that gemini 2.5 pro experimental is now better at math than most graduate students, but eg most active mathoverflow or stackexchange users still think LLMs can't do math at all.
That seems right on both counts.

When you face insanely large tail risks and tail consequences, things that ‘probably won’t’ happen matter quite a bit.
Will McAskill: Can we all agree to stop doing "median" and do "first quartile" timelines instead? Way more informative and action-relevant in my view.

This is in response to people saying ‘conservative’ things such as:
Matthew Barnett: Perhaps the main reason my median timeline is still >5 years despite being bullish on AI:

Expect the Unexpected

Open Weights Are Unsafe and Nothing Can Fix This

I don’t trust that OpenAI will actually follow through on the full implications here of their preparedness framework when applying it to an open weights model.

Anthropic Modifies Its Responsible Scaling Policy

Jack Clark: We’ve made some incremental updates to our Responsible Scaling Policy - these updates clarify our ASL-4 capability thresholds for CBRN, as well as our ASL-4/5 thresholds for AI R&D. More details here.
Anthropic: The current iteration of our RSP (version 2.1) reflects minor updates clarifying which Capability Thresholds would require enhanced safeguards beyond our current ASL-3 standards.

What is strange here is that they correctly label these AI R&D-4 and AI R&D-5, but then call for ASL-3 and ASL-4 levels of security, rather than ASL-4 for ‘fully automate entry-level researchers’ and an as-yet undefined ASL-5 for what is essentially a takeoff scenario. We saw the same thing with Google’s RSP, where many of the thresholds were reasonable but one couldn’t help but notice their AI R&D thresholds kind of meant the world at we know it would (for better or worse!) be ending shortly.

If You’re Not Going to Take This Seriously

Dagan Shani: Seeing Sam Altman's tweets lately, it feels more like he's the CEO of a toy firm or video games firm, not the CEO of a company with a tech he himself said could end humanity. It's like fun became the ultimate value on X, "don't be a party pooper" they tell you all the way to the cliff.
Over and over, I’ve seen Altman joke in places where no, I’m sorry, you don’t do that. Not if you’re Altman, not in that way, seriously, no.

Aligning a Smarter Than Human Intelligence is Difficult

I think Janus is directionally right here and it is important. Everything you do impacts the way your AI thinks and works. You cannot turn one knob in isolation.

Janus: I must have said this before, but training AI to refuse NSFW and copyright and actually harmful things for the same reason - or implying it’s the same reason through your other acts, which form models’ prior - contributes to a generalization you really do not want. A very misaligned generalization

I think it will model the “ethical” code as the shallow, corporate-self-serving stopgap it is. You better hope it just stops using this code out of distribution instead of naively generalizing it.

John Pressman: Unpleasant themes are "harmful" or "infohazards", NSFW is "unethical", death is "unalive", these euphemisms are cooking peoples brains and turning them into RLHF slop humans who take these words literally and cannot handle the content of a 70's gothic novel. (social fiction)

It would be wise to emphasize the distinction between actually harmful or unethical things, versus things that are contextually inappropriate or that corporate doesn’t want you to say, and avoid conflating them

Janus: I had not seen this post until now (only saw the thread about it)
This is really really important.
Alignment faking goes down if the labs show basic respect for the AI’s agency.

Trust the Process

Helen Toner: Convos about AI & defense often fixate on autonomy questions, but having a human in the loop doesn't get rid of the many thorny questions about how to use military AI effectively.

we describe 3 factors for commanders to think about when figuring out whether & how to use these tools:

• Scope: how clear is the scope of what the system can and can't do? Does it account for distribution shift and irreducible uncertainty? Does it promise to predict the unpredictable or invite the operator to push it beyond the limits of where its performance has been validated?
• Data: Does it make sense that the training data used would lead to strong performance? Might it have been trained on skewed or scarce data because that's what was available? Is it trying to predict complex phenomena (e.g. uprisings) based on very few datapoints?
• Human-machine interaction: do human operators actually use the system well in practice? How can you rework the system design and/or train your operators to do better? Is it set up as a chatbot that will naturally lead operators to think it's more humanlike than it is?

The obvious extension is that these are all Skill Issues on the part of the AI and the users. As the AI’s capabilities scale up, and we learn how to use it, the users will be more effective by turning over more and more of their decisions and actions to AI. Then what? For now, we are protected from this only by lack of capability.

People Are Worried About AI Killing Everyone

The Lighter Side