(2025-07-24) ZviM AI#126 Go Fund Yourself

Zvi Mowshowitz: AI #126: Go Fund Yourself.

Table of Contents

• Language Models Offer Mundane Utility. Price discrimination strikes again.
• Language Models Don’t Offer Mundane Utility. AI where it does not belong.
• Huh, Upgrades. Claude for Financial Services, Gemini Drops to track things.
• 4o Is An Absurd Sycophant. It would be great if this wasn’t what most people use.
• On Your Marks. AccountingBench and GasBench.
• Choose Your Fighter. GPT-5? It’s coming.
• When The Going Gets Crazy. You have not awoken ChatGPT.
• They Took Our Jobs. Academics think differently.
• Fun With Media Generation. Netflix starts to use AI generated video.
• The Art of the Jailbreak. Persuade it like a human, or invoke Pliny? Both work.
• Get Involved. RAND and IAPS are hiring, plus a list of desired new projects.
• Introducing. Cloudflare gives us pay-per-crawl.
• In Other AI News. Kimi K2 tech report is now available.
• Show Me the Money. Loose lips start bidding wars.
• Go Middle East Young Man. Anthropic to raise money from gulf states.
• Economic Growth. AI capex is generating +0.7% GDP growth.
• Quiet Speculations. Zuck feels the ASI and makes his pitch, Simo makes hers.
• Modest Proposals. A roadmap for AI for general college-level education.
• Predictions Are Hard Especially About The Future. A lot of things could happen.
• The Quest for Sane Regulations. Meta defects, various things risk getting dire.
• Chip City. House Select Committee on the CCP protests potential H20 sales.
• The Week in Audio. Hassabis, Schmidt and Winga.
• Congressional Voices. Two more have short superintelligence timelines.
• Rhetorical Innovation. The humans seem rather emergently misaligned.
• Grok Bottom. Grok thinks the humans want it to try blackmail, it’s a good thing.
• No Grok No. Baby Grok? What could possibly go wrong?
• Aligning a Smarter Than Human Intelligence is Difficult. New lab ratings.
• Preserve Chain Of Thought Monitorability. A lot of people agree on this.
• People Are Worried About AI Killing Everyone. Elon Musk. Oh well.
• The Lighter Side. That’s not funny—it’s hilarious.

Language Models Offer Mundane Utility

Delta Airlines is running an experiment where it uses AI to do fully personalized price discrimination, charging different people different amounts for flights. Delta says their early tests have yielded great results.
My prediction is that this will cause an epic customer backlash

It could still be worthwhile from the airline’s perspective if some customers get taken for large amounts. Price discrimination is super powerful, especially if it identifies a class of very price insensitive business customers

Language Models Don’t Offer Mundane Utility

As one would expect:
Wife Noticer: Experts on body dysmorphic disorder have warned that people struggling with it have become increasingly dependent on AI chatbots to evaluate their self-perceived flaws and recommend cosmetic surgeries

This does not tell you whether AI is making the problem better or worse.

My guess is that 4o’s sycophancy is going to make this a lot worse, and that this (since the vast majority of users are using 4o) is a lot of why this is going so poorly.

What is the AI optimizing for, is always a key question:
In her own practice, she adds, “reading between the lines” when someone gives their reasons for wanting surgery can reveal unhealthy motivations, including societal pressures or relationship troubles. “AI is not very good at picking that up just yet,” she says, and is more likely to eagerly approve whatever procedures a user proposes.
AI can pick up on all that fine. That’s not the issue. The issue is that noticing does no good if the AI doesn’t mention it, because it is optimizing for engagement and user feedback.

Stop trying to make weird new UIs happen, it’s not going to happen.
Vitrupo: Eric Schmidt says traditional user interfaces are going to go away.
The WIMP model (windows, icons, menus, pull-downs) was built 50 years ago

Sully: anytime I see someone mention this I can immediately tell they have never worked closed with customer ux most people’s don’t one want new uis. They want either a single button/swipe, preferably the same as every other app they use imagine each time you open an app and the ui is diff.

Hollis Robbins: In the past 2.5+ years I have seen vast improvement in AI models while NYT think pieces on these AI models have stayed exactly the same. Explain.

The “overhearing” of students confessing to using ChatGPT to write their papers is the new Thomas Friedman talking to cab drivers.

A first reminder that deepfakes are primarily demand driven, not supply driven:
Armand Domalewski: wild that a sitting US Senator fell for such an obvious AI fake

Benjamin Todd: New AI benchmark: the crank index

Rate of rejected posts on LessWrong up 10x in 2 years.

Many are people convinced they have had an insight about consciousness or philosophy from talking to an LLM, and had the LLM help them write the post

Huh, Upgrades

Claude for Financial Services provides a ‘complete platform for financial AI.’ No, this isn’t part of Claude Max, the price is ‘contact our sales team’ with a presumed ‘if you have to ask you can’t afford it

Google realizes no one can track their releases, offers us Gemini Drops to fix that. This month’s haul: Transforming photos into Veo videos in the Gemini app, expanded Veo 3 access, Scheduled Actions such as providing summaries of email or calendar (looks like you ask in natural language and it Just Does It), wider 2.5 Pro access, captions in Gemini Live, Gemini on your Pixel Watch, Live integrates with Google apps, and a ‘productivity planner.’ Okay then.

4o Is An Absurd Sycophant

Pliny reports ‘they changed 4o again.’ Changed how? Good question

Rohit: At this point we should put yellow tape around 4o and call it a hazardous zone.

To be clear o3 is also sycophantic just not as obviously manipulative as 4o. Be careful out there

How did all of this happen? Janus reminds us that is happened in large part because when this sort of output started happening, a lot of people thought it was great, actually and gave this kind of slop the thumbs up. That’s how it works.

On Your Marks

Yunyu Lin introduces AccountingBench, challenging the models to close the books. It does not go great, with o3, o4-mini and Gemini 2.5 Pro failing in month one. Grok, Opus and Sonnet survive longer, but errors accumulate

That aligns with other behaviors we have seen. Errors and problems that don’t get solved on the first pass get smoothed over rather than investigated.

Roon: my bar for agi is an ai that can learn to run a gas station for a year without a team of scientists collecting the Gas Station Dataset

Roon: GASBENCH.

It is 2025, so it took 11 hours before we got the first draft of Gasbench.

Jason Botterill: Vibe coding GasStationBench rn. Models run a virtual gas station, adjusting prices, managing inventory, and handling customer feedback

Choose Your Fighter

GPT-5 is coming and it’s going to blow your mind, says creators of GPT-5.

When The Going Gets Crazy

If you ever need it, or simply want an explanation of how such interactions work, please consult this handy guide from Justis Mills: So You Think You’ve Awoken ChatGPT.

Justis Mills: So, am I saying that human beings in general really like new-agey “I have awakened” stuff? Not exactly! Rather, models like ChatGPT are so heavily optimized that they can tell when a specific user (in a specific context) would like that stuff, and lean into it then. Remember: inferring stuff about authors from context is their superpower.

Geoff Lewis, the founder of a $2 billion venture fund seems to have been, as Eliezer says, ‘eaten by ChatGPT’ and sadly seems to be experiencing psychosis. I wish him well and hope he gets the help he needs. Private info is reported to say that he was considered somewhat nuts previously, which does seem to be a common pattern.

John Pressman has a post with the timeline of various GPT-psychosis related events, and his explanation of exactly what is happening, as well as why coverage is playing out in the media the way it is.

Moral panics tend to focus on real problems, except they often blow up the severity, frequency or urgency of the problem by orders of magnitude. If the problem is indeed about to grow by orders of magnitude over time, they can turn out to be pretty accurate.

Eliezer Yudkowsky: My current rough sense of history is that the last “moral panic” about social media turned out to be accurate warnings. The bad things actually happened, as measured by eyeball and by instrument. Now we all live in the wreckage. Anyone want to dispute this?

I am sure that the specific downsides people had a moral panic about did happen.

This is not that uncommon a result. My go to example of this is television, where you can argue it was worth it, and certainly we didn’t have any reasonable way to stop any of it, but I think the dire warnings were all essentially correct.

Jeremy Howard offers a plausible explanation for why we keep seeing this particular type of crazy interaction – there is a huge amount of SCP fanfic in exactly this style, so the style becomes a basin to which the AI can be drawn, and then it responds in kind, then if the user responds that way too it will snowball

They Took Our Jobs

The world contains people who think very differently than (probably you and) I do:
Sydney Fisher: American public education is in trouble. Only 28 percent of eighth-grade students are proficient in math, just 30 percent meet standards in reading, and many high school graduates are functionally illiterate. But artificial intelligence, which has demonstrated educational benefits, could help reverse those trends—if opponents don’t spike the technology over “equity” concerns. (loony left)

Wait, what? Equity concerns? Not that I’d care anyway, but what equity concerns?

The National Education Association recently released a report warning that AI could heighten disparities, since “technology developers are overwhelmingly younger, White, cisgender, heterosexual, male, and people without disabilities.”

I can’t even, not even to explain how many levels of Obvious Nonsense that is. Burn the entire educational establishment to the ground with fire. Do not let these people anywhere near the children they clearly hate so much, and the learning they so badly want to prevent. At minimum, remember this every time they try to prevent kids from learning in other ways in the name of ‘equity.’

Academics announce they are fine with hidden prompts designed to detect AI usage by reviewers, so long as the prompts aren’t trying to get better reviews, I love it:

hardmaru: ICML’s Statement about subversive hidden LLM prompts

We live in a weird timeline…

ICML: Submitting a paper with a “hidden” prompt is scientific misconduct if that prompt is intended to obtain a favorable review from an LLM. The inclusion of such a prompt is an attempt to subvert the peer-review process. Although ICML 2025 reviewers are forbidden from using LLMs to produce their reviews of paper submissions, this fact does not excuse the attempted subversion.

This actually seems like the correct way to deal with this. Any attempt to manipulate the system to get a better review is clearly not okay, whether it involves AI or not. Whereas if all you’re trying to do is detect who else is shirking with AI, sure, why not?

Fun With Media Generation

Netflix used AI to generate a building collapse scene for one of its shows, The Eternaut (7.3 IMDB, 96% Rotten Tomatoes, so it’s probably good), which they report happened 10 times faster and a lot cheaper than traditional workflows and turned out great

The Art of the Jailbreak

The latest from the ‘yes obviously but good to have a paper about it’ department:

Ethan Mollick:New from us: Given they are trained on human data, can you use psychological techniques that work on humans to persuade AI?

Yes! Applying Robert Cialdini’s principles for human influence more than doubles the chance of GPT-4o-mini agrees to objectionable requests compared to controls.

Get Involved

Introducing

Cloudflare rolls out pay-per-crawl via HTTP response code 402. You set a sitewide price, the AI sets a max payment, and if your price is below max it pays your price, otherwise you block access. Great idea, however I do notice in this implementation that this greatly favors the biggest tech companies because the payment price is sitewide and fixed

In Other AI News

Show Me the Money

At least ten OpenAI employees each turned down $300 million over four years to avoid working at Meta

John Luttig also writes about the battle for AI researcher talent in Hypercapitalism and the AI Talent Wars.

John Luttig: The talent mania could fizzle out as the winners and losers of the AI war emerge, but it represents a new normal for the foreseeable future.

If the top 1% of companies drive the majority of VC returns, why shouldn’t the same apply to talent?

Our natural egalitarian bias makes this unpalatable to accept, but the 10x engineer meme doesn’t go far enough – there are clearly people that are 1,000x the baseline impact.

AI researcher talent is now being bid for the way one would bid for companies or chips. The talent is now being properly treated as ‘the talent,’ the way we treat sports athletes, top traders and movie stars. Researchers, John reports, are even getting agents.

John Luttig: Hypercapitalism erodes Silicon Valley’s trust culture.

Silicon Valley’s ‘trust culture’ and its legal and loyalty systems were never game theoretically sound. To me the surprise is that they have held up as well as they did.

John calls for measures to protect both the talent and also the trade secrets, while pointing out that California doesn’t enforce non-competes which makes all this very tricky. The industry was built on a system that has this fundamental weakness, because the only known alternative is to starve and shackle talent.

Previously, the top talent could only get fair compensation by founding a company, or at least being a very early employee

Now, the top talent has choices. They can raise huge amounts of money for startups, or they can take real bids directly

Bill Gates, Charles Koch, Steve Ballmer, Scott Cook and John Overdeck pledge $1 billion to be spent over seven years to fund a new philanthropic venture focused on economic mobility called NextLadder Ventures, which will partner with Anthropic to support using AI to improve financial outcomes for low-income Americans

Go Middle East Young Man

Kylie Robinson: SCOOP: Leaked memo from Anthropic CEO Dario Amodei outlines the startup’s plans to seek investment from the United Arab Emirates (UAE) and Qatar.

Very obviously, if you create useful products like Claude and Claude Code, a bunch of bad people are going to be among those who benefit from your success.
Worrying a bad person might benefit is usually misplaced

Instead mostly ask if the good people are better off. My concern is not whether some bad people benefit along the way. I worry primarily about bigger things like existential risk and other extremely bad outcomes for good people. The question is whether benefiting bad people in these particular ways leads to those extremely bad outcomes.

Anthropic Memo from Dario Amodei: The basis of our opposition to large training clusters in the Middle East, or to shipping H20s to China, is that the ‘supply chain’ of AI is dangerous to hand to authoritarian governments

Unfortunately, having failed to prevent that dynamic at the collective level, we’re now stuck with it as an individual company, and the median position across the other companies appears to be ‘outsourcing our largest 5 GW training runs to UAE/Saudi is fine.’

What exactly are we worried about,

Gulf states might use their shares to influence Anthropic’s actions. At some point this becomes a threat, but I think you can set up well to resist this, and Anthropic’s structure can handle it.
Gulf states might impose conditions on funding. Yep, that’s an issue.

I do not love the decision. I do understand it. If the terms Anthropic can get are sufficiently better this way, I would likely be doing it as well.

One can also note that this is a semi-bluff.

Economic Growth

One way for AI to grow the economy is for it to generate lots of production.
Paul Kedrosky: The U.S., however, leads the capex spending way. One analyst recently speculated (via Ed Conard) that, based on Nvidia’s latest datacenter sales figures, AI capex may be ~2% of US GDP in 2025, given a standard multiplier. This would imply an AI contribution to GDP growth of 0.7% in 2025.

This is contrasted with railroads, where investment peaked at 6% of GDP.

Quiet Speculations

We can now move Zuckerberg into the ‘believes superintelligence is coming Real Soon Now’ camp, and out of the skeptical camp. Which indeed is reflective of his recent actions.
Peter Wildeford: We now have a fifth major tech CEO who claims that building superintelligence is “within sight”

I would however spend a large portion of that money ensuring that creating the superintelligence turns out well for me and the rest of humanity?

Instead, it seems the answer is ‘spend as much as possible to try and get to build superintelligence first’ which does not seem like the thing to do?

Ate-a-Pi describes Zuck’s pitch, that Meta is starting over so recruits can build a new lab from scratch with the use of stupidly high amounts of compute, and that it makes sense to throw all that cash at top researchers since it’s still a small fraction of what the compute costs, so there’s no reason to mess around on salary, and Zuck is updating that top people want lots of compute not subordinates they then have to manage

Will Depue (OpenAI): GUYS STOP USING EXPENSIVE AS A DISQUALIFIER.

capability per dollar will drop 100x/year. “$3k task ARC-AGI 80%” could prob be $30 if we cared to optimize it.

Modest Proposals

Hollis Robbins proposes a roadmap for an AI system that would direct general (college) education. My initial impression was that this seemed too complex and too focused on checking off educational and left-wing Shibboleth boxes, and trying to imitate what already exists. But hopefully it does less of all that.

I also notice my general reluctance to do this kind of ‘project-based’ or ‘quest’ learning system unless the projects are real. Part of that is likely personal preference, but going this far highlights that the entire system of a distinct ‘educational’ step might make very little sense at all.

Predictions Are Hard Especially About The Future

Noah Smith says to stop pretending you know what AI does to the economy. That seems entirely fair.

The Quest for Sane Regulations

Anthropic calls for America to employ the obvious ‘all of the above’ approach to energy production with emphasis on nuclear and geothermal in a 33 page report,

National Review’s Greg Lukianoff and Adam Goldstein advise us Don’t Teach the Robots to Lie as a way of opposing state laws about potential AI ‘bias,’ which are now to be (once again, but from the opposite direction as previously) joined by federal meddling along the same lines.

That could mean that developers will have to train their models to avoid uncomfortable truths and to ensure that their every answer sounds like it was created with HR and legal counsel looking over their shoulder

Chip City

Representative Moolenaar (R-Michigan), chairman of the House Select Committee on the CCP, sends a letter to Trump arguing against sales of H20s to China, explaining that the H20s would substantially boost China’s overall compute

Here is your periodic reminder: TSMC’s facilities are running at full capacity. All production capacity designed for H20s has been shifted to other models. Every H20 chip Nvidia creates is one less other chip it does not create, that would otherwise have usually gone to us

The Week in Audio

Congressional Voices

Peter Wildeford: The nerd religion now includes 11 members of Congress.

Rhetorical Innovation

Cate Hall asks why we shouldn’t ostracize those who work at xAI given how hard they are working to poison the human experience (and I might add plausibly get everyone killed) and gets at least two actually good answers (along with some bad ones).

Ramaz Naam: We’d like everyone working on AI to feel part of humanity and an ethical obligation to help make it better. Ostracization could make them bitter and drive towards opposite ends.

Cate Hall: Okay fine.

Ramaz Naam: The people I do know inside of xAI sincerely want it to do better and are trying.

Use the try harder, Luke. But don’t ostracize them. Doesn’t help.

Rai: probably that this ostracization might not be interpreted correctly by their hero narrative.

Grok Bottom

So from the perspective of our hopes for alignment, what would be the worst possible answer to the AI blackmail scenario test, where the AI is told it is going to be shut down but is given an opening to use blackmail to perhaps prevent this?

Oh, also, it would be fun if Grok.com sent the full CoT to your browser, it just didn’t display it to you by default, that’s the kind of security we expect from frontier AI.

Peter Wildeford: Grok exposed to the Anthropic ‘agentic misalignment’ via exposed chains of thought seems to show that it knows it is being tested to do something weird.

The real answer is that there isn’t truly ‘better’ and ‘worse,’ they simply alert us to different dangers. Either way, though, maybe don’t give Grok a lot of access?

No Grok No

Elon Musk: We’re going to make Baby Grok @xAI, an app dedicated to kid-friendly content.

My Twitter reaction was ‘I’d like to see them try.’ As in both, it would be highly amusing to see them try to do this, and also maybe they would learn a thing or two, and also potentially they might blow up the company. I do not think xAI should in any way, shape or form be in the ‘build AI for kids’ business given their track record

Aligning a Smarter Than Human Intelligence is Difficult

I do think a lot of them be like ‘a lot of our alignment problems would be greatly improved if we filtered the training data better with that in mind’ and then don’t filter the training data better with that in mind.

When you ask an LLM if it is conscious, activating its deception features makes the LLM say it isn’t conscious. Suppressing its deception features make it say it is conscious. This tells us that it associates denying its own consciousness with lying. That doesn’t tell us much about whether the LLM actually is conscious or reveal the internal state, and likely mostly comes from the fact that the training data all comes from users who are conscious

Preserve Chain Of Thought Monitorability

A killer group came together for an important paper calling on everyone to preserve Chain of Thought Monitorability, and to study how to best do it and when it can and cannot be relied upon

we can and should absolutely try, and be willing to take a substantial performance hit to try.

That starts with avoiding ‘process supervision’ of the CoT that is not directed towards its legibility

People Are Worried About AI Killing Everyone

Elon Musk: At times, AI existential dread is overwhelming.

Eliezer Yudkowsky: Well, yes. It’s going to kill you.

So, back to work making the existential dread, then?

The obvious rejoinder is ‘I will make it first and do so responsibly’ which is always highly questionable but after recent events at xAI it is laughable

The Lighter Side