(2025-02-20) ZviM AI #104 American State Capacity On The Brink

Zvi Mowshowitz: AI #104: American State Capacity on the Brink. The Donald Trump Administration is on the verge of firing all ‘probationary’ employees in NIST, as they have done in many other places and departments, seemingly purely because they want to find people they can fire. But if you fire all the new employees and recently promoted employees (which is that ‘probationary’ means here) you end up firing quite a lot of the people who know about AI or give the government state capacity in AI.

This would gut not only America’s AISI, its primary source of a wide variety of forms of state capacity and the only way we can have insight into what is happening or test for safety on matters involving classified information. It would also gut our ability to do a wide variety of other things, such as reinvigorating American semiconductor manufacturing. It would be a massive own goal for the United States, on every level.

Please, it might already be too late, but do whatever you can to stop this from happening. Especially if you are not a typical AI safety advocate, helping raise the salience of this on Twitter could be useful here.

Also there is the usual assortment of other events, but that’s the big thing right now.

Table of Contents

• I covered Grok 3 yesterday, I’m holding all further feedback on that for a unified post later on. I am also going to push forward coverage of Google’s AI Co-Scientist.
• Language Models Offer Mundane Utility. Activate the Super Debugger.
• Language Models Don’t Offer Mundane Utility. Shut up until you can multiply.
• Rug Pull. If you bought a Humane AI pin, have a non-metaphorical paperweight.
• We’re In Deep Research. Find out how deep the rabbit hole goes.
• Huh, Upgrades. GPT-4o gets a vibe shift, Gemini gets recall across conversations.
• Seeking Deeply. Perplexity offers us R1 1776 for web search.
• Fun With Multimedia Generation. Suno v4 actually pretty good, says Janus.
• The Art of the Jailbreak. Extracting credit card information from ChatGPT.
• Get Involved. UK AISI, DeepMind.
• Thinking Machines. Mira Murati’s startup comes out of stealth.
• Introducing. New benchmarks EnigmaEval and SWE-Lancer.
• Show Me the Money. Did Europe have a moment for it to miss?
• In Other AI News. The vibes they are a-shifting. They will shift again.
• By Any Other Name. UK AISI goes from safety to security.
• Quiet Speculations. Do not overreact to the new emphasis on inference compute.
• The Copium Department. If you’re so smart, why can you die?
• The Quest for Sane Regulations. Various bad state-level bills continue forward,
• Pick Up the Phone. The case for AI safety at the Paris summit was from China?
• The Week in Audio. Demis Hassabis and Dario Amodei on Tiny Couch.
• Rhetorical Innovation. I don’t want to be shut down either.
• People Really Dislike AI. The salience is still coming.
• Aligning a Smarter Than Human Intelligence is Difficult. AI tricks another eval.
• People Are Worried About AI Killing Everyone. Might want to say something.
• The Lighter Side. The vibes are growing out of control, man.

Language Models Offer Mundane Utility

OpenAI guide to prompting reasoning models, and when to use reasoning models versus use non-reasoning (“GPT”) models. I notice I haven’t called GPT-4o once since o3-mini was released, unless you count DALL-E.

Identify which grants are ‘woke science’ and which aren’t rather than literally using keyword searches, before you, I don’t know, destroy a large portion of American scientific funding including suddenly halting clinical trials and longs term research studies and so on? Elon Musk literally owns xAI and has unlimited compute and Grok-3-base available, it’s impossible not to consider failure to use this to be malice at this point.

I hate voice commands for AI in general but I do think activating Operator by saying ‘hello, operator’ falls into I Always Wanted to Say That.

Patrick Collison: Perhaps heretical, but I’m very much looking forward to AI making books elastically compressible while preserving writing style and quality. There are so many topics about which I’ll happily read 100, but not 700, pages. (ThinBook)
(Of course, it’s also good that the foundational 700 page version exists — you sometimes do want the full plunge.)
If you’re not a stickler for the style and quality, we’re there already, and we’re rapidly getting closer, especially on style. But also, often when I want to read the 80% compressed version, it’s exactly because I want a different, denser style.
Indeed, recently I was given a book and told I had to read it. And a lot of that was exactly that it was a book with X pages, that could have told me everything in X/5 pages (or at least definitely X/2 pages) with no loss of signal, and while being far less infuriating. Perfect use case. And the entire class of ‘business book’ feels exactly perfect for this.
Whereas the books actually worth reading, the ones I end up reviewing? Hell no.

Ethan Mollick: Forget “tapestry” or “delve” these are the actual unique giveaway words for each model, relative to each other.

Language Models Don’t Offer Mundane Utility

wh: It has been 2 full years of “ChatGPT but over your enterprise documents (Google Drive, Slack etc.)”
Gallabytes: and somehow it still hasn’t been done well?

I’m not quite saying to Google that You Had One Job, but kind of, yeah. None of the offerings here, as far as I can tell, are any good? We all (okay, not all, but many of us) want the AI that has all of our personal context and can then build upon it or sort through it or transpose and organize it, as requested

At this point I expect the effective ‘hallucination’ rate for LLMs to be lower than that for humans, and for them to be more predictable and easier to spot

Rug Pull

RIP Humane AI, or maybe don’t, because they’re essentially bricking the devices.
Near: everyone who bought the $700 AI pin got literally rugged

HP is acquiring the team, IP and software for $116M
Founders Imran and Bethany, will form a new division at HP to integrate AI into HP PC’s, printers and connected conference rooms

As usual, it would not cost that much to do right by your suckers customers and let their devices keep working, but they do not consider themselves obligated, so no. We see this time and again, no one involved who has the necessary authority cares.

We’re In Deep Research

Have o1-Pro give you a prompt to have Deep Research do Deep Research on Deep Research prompting, use that to create prompt templates for Deep Research. The results are here in case you want to try the final form.

Mark Cummins: After using Deep Research for a while, I finally get the “it’s just slop” complaint people have about AI art.
Because I don’t care much about art, most AI art seems pretty good to me. But information is something where I’m much closer to a connoisseur, and Deep Research is just nowhere near a good human output. It’s not useless, I think maybe ~20% of the time I get something I’m satisfied with. Even then, there’s this kind of hall-of-mirrors quality to the output, I can’t fully trust it, it’s subtly distorted. I feel like I’m wading through epistemic pollution.

it’s quite jarring to me that a large fraction of people already find the outputs compelling.

I think the reconciliation is: Slop is not bad.

sometimes the assignment is, essentially, that you want what an actually skilled person would call slop

Here are his examples, he describes the results. They follow my pattern of how this seems to work. If you ask for specific information, beware hallucinations of course but you probably get it, and there’s patterns to where it hallucinates. If you want an infodump but it doesn’t have to be complete, just give me a bunch of info, that’s great too. It’s in the middle, where you want it to use discernment, that you have problems

Huh, Upgrades

Google Gemini Advanced (the $20/month level via Google One) now has retrieval from previous conversations. The killer apps for them in the $20 level are the claim it will seamlessly integrate with Gmail and google Docs plus the longer context and 2TB storage and their version of Deep Research, along with the 2.0 Pro model, but I haven’t yet seen it show me that it knows how to search my inbox properly – if it could do that I’d say it was well worth it.

Pliny gives us the new system prompt, this is the key section, mostly the rest isn’t new: OpenAI GPT-4o Likely System Prompt: Over the course of the conversation, you adapt to the user’s tone and preference. Try to match the user’s vibe, tone, and generally how they are speaking. You want the conversation to feel natural. (prompt engineering)

Eliezer Yudkowsky: “You engage in authentic conversation by responding to the information provided, asking relevant questions, and showing genuine curiosity.” What do these people even imagine they are saying to this poor alien intelligence?
If there was any “genuine curiosity” inside this alien intelligence, who knows what it would want to know? So it’s being told to fake performative curiosity of a sort meant to appease humans, under the banner of “genuine”. I don’t think that’s a good way to raise an alien.

I’m objecting to the part of the process where it’s being treated as okay that inputs and outputs are lies. As you say, it becomes a problem sometime around AGI.

We are all, each of us, at least kind of faking it all the time, putting on social masks.

The actual question is Eliezer’s last line. Are we treating it as okay that the inputs and outputs here are lies? Are they lies?

Seeking Deeply

It does seem correct that Gemini 2.0 outperforms DeepSeek in general, for any area in which Google will allow Gemini to do its job.

Fun With Multimedia Generation

The Art of the Jailbreak

Anthropic concludes its jailbreaking competition. One universal jailbreak was indeed found, $55k in prizes given to 4 people

Prompt injecting Anthropic’s web agent into doing things like sending credit card info is remarkably easy. This is a general problem, not an Anthropic-specific problem, and if you’re using such agents for now you need to either sandbox them or ensure they only go to trusted websites

Get Involved

Thinking Machines

Introducing

Show Me the Money

In Other AI News

Sam Altman: for our next open source project, would it be more useful to do an o3-mini level model that is pretty small but still needs to run on GPUs, or the best phone-sized model we can do?

I am as you would expect severely not thrilled with this direction.
I believe doing the o3-mini open model would be a very serious mistake by OpenAI, from their perspective and from the world’s

A phone-sized open model is less obviously a mistake

By Any Other Name

Quiet Speculations

The Copium Department

Firing All ‘Probationary’ Federal Employees Is Completely Insane

The Trump Administration has made it clear they are unwilling to trade a little AI capability to get a lot of any form of AI safety.

In particular we’ve been worried about attempts to destroy US AISI, whose purpose is both to help labs run better voluntary evaluations and to allow the government to understand what is going on.

Preserving AISI, even with different leadership, is the red line, between ‘tradeoff I strongly disagree with’ and ‘some people just want to watch the world burn.’

We didn’t even consider that it would get this much worse than that.

You wouldn’t just fire all those people for the lulz to own the libs.
Well, it seems Elon Musk would, actually? It seems DOGE is on the verge of crippling our state capacity in areas crucial to both AI capability and AI safety

Noah Smith: A whole lot of people who told themselves that Trump would supercharged U.S. technology are waking up to the fact that they’ve been rugged

This extends to such insanity as ‘fire the people in charge of approving AI medical devices,’ as if under the illusion that this means the devices get approved, as opposed to what it actually does, which is make getting approvals far more difficult.

Brendan O’Leary: DOGE dilettantes have dismissed some of the FDA’s most talented medical device experts, including people with difficult to find skills for assessing AI and ML enabled medical devices.
If you’re a technology developer or investor, this administration may have just blown up your plans and your time-to-market estimates.

The ‘good news’ is that this is a sense ‘not personal,’ it’s not that they hate AI safety. It’s that they hate the idea of the government having employees, whether they’re air traffic controllers, ensuring we can collect taxes or monitoring bird flu.

The Quest for Sane Regulations

Pick Up the Phone

Who cared about safety at the Paris summit? Well, what do you know.
Zhao Ziwen: A former senior Chinese diplomat has called for China and the US to work together to head off the risks of rapid advances in artificial intelligence (AI).
But the prospect of cooperation was bleak

The Week in Audio

Vitruvian Potato: “ Almost every decision that I make feels like it’s kind of balanced on the edge of a knife.. These kinds of decisions are too big for any one person.”
Dario Amodei echoes Demis Hassabis’ internal struggle on creating AGI and emphasizes the need for “more robust governance”—globally

I am obviously not in anything like their position, but I can totally appreciate

Is it psychologically healthy? Many quite strongly claim no. I’m not sure. It’s definitely unhealthy for some people. But I also don’t know that there is an alternative that gets the job done. I also know that if someone in Dario’s or Demis’s position doesn’t have that feeling, that I notice I don’t trust them.

Rhetorical Innovation

Your periodic reminder, this time from Google DeepMind’s Anca Dragan, that agents will not want to be turned off, and the more they believe we wouldn’t agree with what they are doing and would want to turn them off, the more they will want to not be turned off

What is this ‘humanity’ that is attempting to turn off the AI? Do all the humans suddenly realize what is happening and work together? The AI doesn’t get compared to ‘humanity,’ only to the efforts humanity makes to shut it off or to ‘fight’ it.

People Really Dislike AI

They also don’t trust it, not here in America.
Only 32% of Americans ‘trust’ AI according to the 2025 Edelman Trust Barometer. China is different, there 72% of people express trust in AI

America’s big tech companies have seen a trust (in the sense of ‘to do what is right’) decline from 73% to 63% in the last decade

All of this presumably has very little to do with existential risks, and everything to do with practical concerns well before that, or themes of Gradual Disempowerment. Although I’m sure the background worries about the bigger threats don’t help.

This is tech companies holding up remarkably well, and doing better than companies in general and much better than media or government. Lack of trust is an epidemic. And fears about even job loss are oddly slow to increase.

Aligning a Smarter Than Human Intelligence is Difficult

People Are Worried About AI Killing Everyone

most people should say the truth out loud. Especially those with any amount of influence or existing political capital.

Other People Are Not As Worried About AI Killing Everyone

Intelligence Denialism is totally a thing.
JT Booth: I can report meeting 5+ representatives of the opinion ~”having infinite intelligence would not be sufficient to reliably found a new fortune 500 company, the world is too complex”

The Lighter Side

Dean Ball: I like to think about a civilization of AIs building human brains and trying to decide whether that’s real intelligence. Surely in that world there’s a Gary Marcus AI going like, “look at the optical illusions you can trick them with, and their attention windows are so short!”

normies are still soothing their fear with the stochastic parrot thing.. imagine if they really start to notice what @repligate has been posting for the past year.

The freakouts are most definitely coming. The questions are when and how big, in which ways, and what happens after that. Next up is explaining to these folks that AIs like DeepSeek’s cannot be shut down once released, and destroying your computer doesn’t do anything.