Dear friends,

The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field.

I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so.

First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago. The theories about this remain the same fantastical, science fiction scenarios as a few months ago. The biggest change in AI risk is its cybersecurity capabilities — a topic which we should take seriously — but this, too, will not lead to the end of the world.

The most notable recent event leading to increased fear was when an OpenAI team deployed an agent swarm that hacked into Hugging Face. Much of the popular press contained significant hype. For example, some publications reported that a swarm of 1,200 agents carried out the attack. While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop. Yes, the ability to get large swarms of agents to work in parallel on a task is a significant technical advance, And, in computing, many processes run at the same time. So this shouldn’t be seen as some magical capability.

Additionally, OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI. There are many well known ways to attack software systems. The main advantage of AI agents is that they are relentless. They will tirelessly try many tactics — and have the patience to chain vulnerabilities together — that previously would have taken an infeasible amount of human effort. But in the long term, I believe the advantage will lie with defenders (because they have more information with which to identify bugs, which they can fix), but the cyber-threat landscape has changed significantly. There are still bottlenecks to identifying and exploiting a vulnerability. AI agents still have to try a lot of things to see what works, and taking these actions takes time and might be detected by defenders. This is why, even though it is now easy to obtain versions of leading open weight models that have had their guardrails removed or weakened, so they will not refuse to try to execute cyber attacks, the world has not ended.

I am also concerned about the anthropomorphization of AI in a lot of reporting, where LLMs and agents are unnecessarily treated as if they were people. If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer. The problem lies in how I used the hammer. Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent.

Of course, we want to build systems that are as safe and predictable as possible. (For example, an unsafe hammer would be one whose head randomly flies off under normal use.) Today’s agentic systems are not predictable, but I see no reason why, by applying sound engineering practices, we won’t be able to make them extremely safe to use. One new element in the forecasts of AI-enabled doom is AI companies disclaiming responsibility for their own products. “I didn’t do it; my out-of-control agent did!” There’s a balance to be struck between the responsibility of the tool maker and the tool user, but when something goes wrong, let’s hold the people building and/or using the hammer responsible, rather than the hammer. (By the way, if you’re worried about AI bioweapon risk, David Bellamy has a great post on why this, too, is overhyped. Briefly, the bottleneck in building a bioweapon is not intelligence, but lab work and manufacturing.)

Pausing AI progress will create much more harm than benefit. First, our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them. If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration.

Of course, the incentive to stoke fears — for regulatory capture, to garner attention, or to make one’s technology seem more powerful — remains the same as before. Disclaiming responsibility is a new one. Taking a hard technical look at the actual risks however, I see little factual basis for the degree of fear that’s been stoked up. We still have hard research and engineering work ahead to improve AI safety, but the beneficial applications continue to vastly outweigh the risks, and we should keep building.

Andrew

A MESSAGE FROM DEEPLEARNING.AI

Your desk shouldn’t be the only place where you can learn. The DeepLearning.AI app lets you follow the latest news and track your course progress anywhere and everywhere, on your commute or coffee break. Download the app on iOS and Android

News

How To Secure Agents for the Masses

A malicious web page can fool an AI agent into working against you. Meta built an agent on the assumption that such an event will happen, and designed it so that such prompt-injection exploits won’t lead the agent astray.

What’s new: Meta introduced Muse, a personal AI agent based on the Muse Spark 1.3 model. Controlled via the Muse app or WhatsApp, it reads and sends emails, browses the web, fills out forms, makes purchases, and keeps working even if the Muse app is closed. Interactions train Meta models unless users opt out.

• Features: Connects to apps including browsers, email clients, calendars, Instagram, and Facebook as well as cars, smart-home devices; selectable read and/or write access per service; scheduled and event-triggered background operation; memory is readable, editable, forgettable; activity log; output includes documents, PDFs, web pages, dashboards; transactions via Stripe Link; support planned for Shop Pay and 1Password
• Availability: U.S. only, 18 and over via iOS, Android, muse.ai, WhatsApp
• Price: Free (up to 100 million tokens per week), $20 (up to 500 million tokens per week) and $100 (up to 3 billion tokens per week)
• License: Proprietary
• Undisclosed: Muse Spark 1.3 parameter count, architecture, knowledge cutoff, and training data; Muse prompt-injection classifier evaluations; Muse agent evaluations

How it works: Muse agents are designed with security in mind. Each agent runs on a VM (an isolated, dedicated virtual machine with a Linux operating system, browser, storage, and memory). The VM holds the agent’s workspace, the user’s files, and credentials for every connected service. To protect against attempts prompt-injection attacks, the VM is divided into two zones: (i) a sealed runtime cell where the agent and its tools handle untrusted data and (ii) services outside the cell that hold passwords and decide what the agent can do.

• The agent’s harness, the user’s workspace, and tools sit inside a Linux container with its own file system and virtual network interface. The VM limits its requests to the operating system and privileges it holds there, and administrator rights inside the runtime cell do not extend to the host machine. The cell can reach external services only via local channels. Operating-system outines verify which process is on each end, and the channels carry no passwords or access tokens.
• Muse Spark 1.3 never sees credentials. A credential service outside the runtime cell processes passwords and access tokens, while the agent works with stand-in tokens. A separate agent called Sentinel, which runs on the same VM but outside the runtime cell, approves each request and swaps in the real credentials as the request leaves the VM. Meta says this makes it impossible for malefactors to steal credentials via prompt injection, since the agent holds no credentials. Connectors to external services like calendars run outside the cell as well, and they receive only the credentials they need. The email connector strips temporary passcodes and password-reset links before the agent reads a message.
• Only Sentinel can permit an action proposed by Muse Spark 1.3. It inspects outbound each request and checks each connector’s call against permissions the user has set; then Sentinal allows it, denies it, or asks the user to decide. The system also tracks which tool processes have read user data. A process that hasn’t read user data can reach a short list of pre-approved destinations on its own, while one that has must ask the user for approval.
• When Sentinel asks for user input, the agent stops, and the request goes to the Muse app as a system dialog rather than as a message in the conversation. This way, prompt-injected text can’t manufacture a user’s approval. An approval is bound to one connector or destination and purpose. Users can constrain approvals to cover one action, session, task, or time span, or all future uses. Sending emails and making purchases always requires user verification, and purchases on unfamiliar sites use a single-use card number from Stripe’s Link wallet, valid only for the specific merchant, amount, and time span.
• Meta trained Muse Spark 1.3 to resist prompt injections and added three layers of additional protection. (i) Data from a source outside the system is labeled untrusted as it enters the model’s context. (ii) An ensemble of classifiers, which were trained separately from the model, screens every file and tool output. This process runs outside the cell, so that an attacker can’t disable it. (iii) In the browser, a sub-agent reads a structured summary of each page — the accessibility tree that screen readers use — instead of the page’s code. It can’t run JavaScript, so instructions buried in scripts or markup never reach it. Other classifiers watch for injection attempts hidden in page text, images, and downloads, and still others block the agent if it tries to route personal data to a destination the task didn’t call for.

Yes, but: Meta says it evaluated Muse Spark 1.3’s ability to resist prompt injections using an unpublished dataset, and it has not provided accuracy metrics for the classifiers that screen incoming data. Instead, the company offers a bug bounty of up to $300,000 for a valid report and up to $130,000 for a successful prompt injection.

Behind the news: Muse incorporates design features proposed by security researchers before Meta’s current AI lab existed. In April 2025, Google DeepMind and ETH Zurich researchers led by Edoardo Debenedetti proposed CaMeL, which separates a model that plans from a model that reads untrusted data and enforces written policies before tool calls. Two months later, independent developer Simon Willison identified the “lethal trifecta” for AI agents: private data, untrusted content, and a way to send data out. Willison argued that the only safe option is to avoid combining them. Muse processes all three but routes outgoing data through a component the model can’t override, according to Meta. A classifier trained on past prompt injections may catch 99 percent of new ones, but that remains an unacceptable risk, Willison wrote. Accordingly, Meta built the container, credential separation, and Sentinel to hold when initial layers fail.

Why it matters: Security is a major risk for current agents. Most agentic harnesses include a system prompt that tells the model to ignore instructions it finds in content and a classifier that recognizes such instructions, but clever hackers can evade these defenses. Meta assumes the model will be fooled, and it built controls at the operating-system level that should hold regardless of the model’s actions. Meta detailed the protections that sit outside the model: a container the agent can’t escape, credentials it can’t hold, a gatekeeper it can’t override, and approvals that don’t pass through conversations. Developers who build agents for sensitive tasks can adapt this approach.

We’re thinking: Meta says it will release Muse Spark’s weights eventually. But it’s the harness, more than the model, that keeps the Muse agent safe. We hope Meta will open-source that software, too.

Inside the Dispute Over A Landmark Agent-Driven Mathematical Proof

OpenAI agents solved a longstanding mathematical problem that carries a $1 million prize, but many obervers question whether the agents drew on unpublished work by two mathematicians.

What's new: On September 8, OpenAI announced that an unreleased model had produced a proof addressing an unsolved problem in fluid dynamics: Whether the equations known as Navier-Stokes have solutions that are, in fact, physically impossible. The proof, which took 10,000 agents 88 hours to produce, shows the answer is yes: Under the right conditions, the equations break down. However, mathematicians Tristan Buckmaster and Levent Alpöge had been using OpenAI Codex to solve the same problem and Buckmaster had discussed the project with OpenAI personnel. OpenAI published its proof only one day after they had published proofs of related equations, raising suspicions that OpenAI’s agents had seen their work, perhaps via models that had been trained on their data.

What the agents proved: The Navier-Stokes equations describe how fluids move, from water in pipes to air over a wing. For decades, mathematicians have tried to determine whether these equations always produce “smooth” solutions — no sudden jumps or infinite speeds — or whether they can predict an infinite speed, which no real fluid can reach. Either answer would settle the problem, a problem selected for a $1 million Millennium Prize funded by Clay Mathematics Institute. OpenAI’s proof demonstrates that under a smooth external force, a fluid's speed can increase beyond finite limits even as its energy remains finite. Clay”s official problem statement allows this result. (The harder version that most mathematicians have in mind, an infinite speed or sudden jump with no outside force at all, remains an open problem.)

Prize pursuits: OpenAI published a 166-page paper as well as the agent-written code in Lean, a programming language that lets computers check mathematical proofs line by line. The underlying model’s name, size, and training data are undisclosed. OpenAI says only that it is “significantly more capable than GPT-6 Astra” and was built by “large-scale reinforcement learning on top of a previously pretrained model.”

• On September 1, after hearing rumors that agents elsewhere were solving Millennium problems, OpenAI launched groups of agents tasked with solving the open problems. Each agent could read a cached snapshot of the web, run code, and message others in its group.
• A group of nearly 100 agents solved a related question about Euler equations — simplified cases of the Navier-Stokes equations — in about 50 hours. OpenAI then shifted agents from the other problems to Navier-Stokes. That group, which grew to 10,000 concurrent agents, reached its conclusion 88 hours after launch, on September 5. Formalizing the steps in Lean took another 17 hours.
• Altogether the agents exchanged 4.9 million messages and generated about 300 billion output tokens. OpenAI did not disclose the cost. Estimates range between $2 million and $22.5 million based on GPT-6 Astra’s list price.
• So far, no independent review of OpenAI’s proof has been completed. Clay's rules require publication, a two-year wait, and general acceptance and review by expert mathematicians before it will consider a solution. OpenAI says it isn't seeking the prize.

The controversy: Hours before OpenAI’s announcement, NYU mathematician Tristan Buckmaster posted a statement that accused OpenAI researcher Sébastien Bubeck of pressuring him over credit for related work he had done with Levent Alpöge, who works at Anthropic. Buckmaster said he and Alpöge had spent most of a year using OpenAI Codex and Anthropic Claude to chase the same kind of proof, building on an approach opened years earlier by mathematicians Diego Córdoba and Luis Martínez-Zoroa, whom Buckmaster credits with “the basic idea of this program.” Buckmaster and Alpöge had posted Lean-verified proofs of three simpler fluid equations, including Navier-Stokes variations, on September 7.

• Buckmaster said his collaboration with Alpöge is personal, not institutional, and that he paid for it out of his own research funds, “including footing a large bill to OpenAI.” He spoke twice by phone with OpenAI’s Bubeck on September 6 and asked whether OpenAI's model had been trained on, or had access to, the Codex sessions where he and Alpöge had been putting drafts. He says he was told the model didn’t look up user data, but his question about training went unanswered. He also says Bubeck proposed that Buckmaster himself write up OpenAI’s result without naming Alpöge because Alpöge works at Anthropic. Bubeck posted on the X social network that he never sought to omit Alpöge’s name.
• On September 8, OpenAI said that neither its researchers nor its agents saw the pair’s work before publication and that no specific user data was accessed — but it could not determine whether the company had used their activities to improve its models. On September 10, OpenAI replaced that sentence: Following an investigation, it said, Buckmaster's Codex prompts over the two months prior to the announcement “could not have influenced the system in any way, including through training.” However, OpenAI did not address whether earlier Codex prompts may have influenced the system.

Behind the news: The race to build math-savvy AI has entered an especially intense phase. Four days before OpenAI's announcement, Anthropic announced that a Claude model had produced the first complete Lean proof of Fermat’s Last Theorem. A research model roughly comparable to Claude Fable 5.1 worked on its own for 11 days and used about 6 billion output tokens — 2 percent of OpenAI’s tally. This proof drew no dispute. It contained no new mathematical inventions. Anthropic built on, credited, and had the result endorsed by Kevin Buzzard, whose Imperial College team has been formalizing the theorem since 2024.

Why it matters: Buckmaster called OpenAI’s Navier-Stokes proof a “Deep Blue-Kasparov moment,” referring to the pivotal series of chess matches in 1996 and 1997 in which IBM’s Deep Blue computer defeated world champion Gary Kasparov. Some pundits question whether mathematician might be an endangered job title. However, Buckmaster noted that the important thing is the sheer speed with which mathematicians and LLMs working together can make progress. Nonetheless, a formally correct proof doesn’t tell you why it works, whether the problem statement is correct, or what meanings can be drawn from it. While automated verification has become relatively inexpensive, human-readable understanding remains costly.

We're thinking: The questions around whether OpenAI used the mathematicians’ work is an important reminder to check the privacy settings of your LLM provider, and to consider zero data-retention options if you’re processing sensitive data that you don’t want a provider to use. 

Learn More About AI With Data Points!

AI is moving faster than ever. Data Points helps you make sense of it just as fast. Data Points arrives in your inbox twice a week with six brief news stories. This week, we covered DeepSeek V4.1 Flash’s new AI architecture. Subscribe today!

Some Kimi and DeepSeek Users Were Served Claude Instead, Anthropic Says

Anthropic accused seven Chinese AI developers, including Moonshot AI, Alibaba, and DeepSeek, of misusing its Claude models to mislead their customers and improve their own models.

What’s new: In a lengthy report, Anthropic alleged that the China-based companies presented Claude’s responses to their customers’ queries as their own models’ responses, and distilled Claude’s responses into their models. The report details several ways in which these companies violated Claude’s terms of services between February and August 2026. The violations included gaining access to Claude through gray markets outside China and using it in scams, espionage, and designing biological or autonomous weapons.

How it works: Anthropic’s allegations focus on distillation, a technique in which a teacher model generates responses to queries that are used to train a student model. Anthropic acknowledges that distillation is a legitimate training method, but the company objects to large-scale efforts to extract Claude’s reasoning capabilities – which it says the companies are doing via fraudulent methods – and recreating it in another model at a fraction of the cost and computation power. Anthropic calls this “illicit” distillation.

• Claude is blocked in mainland China. Anthropic said the companies used transfer stations to bypass geographic restrictions. Several used false identities to create accounts, entered fake or stolen credit-card numbers, and used stolen API keys, according to Anthropic.
• Alibaba, DeepSeek, Moonshot AI, Xiaomi, and Ziphu (also known as Z.ai) ran distillation campaigns that aimed to extract Claude’s reasoning transcripts via hundreds or thousands of fraudulent accounts, According to Anthropic.
• Between May and June 2026, Alibaba ran the largest distillation effort Anthropic had identified to date, involving 151 million exchanges. Alibaba accessed Claude through 5,000 fraudulent accounts, Anthropic said.
• Zhipu created cybersecurity challenges and used leading U.S. AI models to solve and evaluate them, with the goal of incorporating those capabilities into its own models, Anthropic alleged. Zhipu initially targeted Anthropic’s Fable model, but abandoned the effort after Fable’s stronger cyber safeguards limited the usefulness of the responses, according to the report. Zhipu then switched to Opus 4.6 and another U.S. frontier model specifically because their safeguards appeared to be weaker.
• DeepSeek, Moonshot AI, and others allegedly diverted their own customers’ queries to Claude and presented its output as that of the other models. Anthropic documented several instances in which sensitive information was submitted to Claude in this way. Among the users, Anthropic said, it identified a member of the People’s Liberation Army, an employee of a Chinese state-owned enterprise, and an agent of a Russian defense agency.

Behind the news: Distillation has come under scrutiny by leading U.S. AI developers and policymakers, many of whom wish to protect proprietary American technology.

• Anthropic accused DeepSeek, Moonshot, and MiniMax of “industrial-scale” distillation efforts earlier this year.
• In April, Michael Kratsios of the White House Office of Science and Technology issued a memo that acknowledged wide-scale distillation as an adversarial threat and said the U.S. would build defensive strategies.
• In July, Kratsios posted on the X social network that Moonshot AI had distilled Anthropic’s Claude Fable in developing its own Kimi K3. He called such large-scale distillation efforts “unacceptable attempts to steal U.S. technology.” 

Why it matters: The legitimacy of distillation as a way to build models that are intended to compete with the very models being distilled is a subject of debate. But routing customer input to a third party without their knowledge and presenting the output as one’s own could be called fraud. Such practices erode public confidence in AI companies’ commitment to information privacy and security, and they undermine trust in the AI industry as a whole.

We’re thinking: You can’t distill your way to a frontier model. While distillation can be helpful for training models, the labs that Anthropic is accusing have achieved significant technical breakthroughs, many of them published openly, that were critical to their competitive models. The claims that these companies are distilling U.S. models has created a mistaken impression that their competitive strength stems solely from distillation, when their technical innovations have been a more important factor. 

Even Agents Benefit From Reminders

Agents can lose track of what happened earlier as their trajectories grow, either by truncating their contexts or by missing relevant information among irrelevant tokens. A fresh approach pairs an agent that performs actions with a separate agent that decides when to remind it of what it needs to know.

What’s new: Yifan Wu and colleagues at Meta AI introduced a Proactive Memory Agent that runs alongside a typical agent to maintain and selectively highlight information relevant to the current problem. The memory agent increased the action agent’s performance on all benchmarks the authors tested.

Key insight: Solving problems that accrue a long context, an agent carries information such as requirements, failed commands, and useful observations in its history — yet that information may not shape its next decision. Highlighting relevant memories can counteract such shortcomings without overwhelming the agent’s attention.

How it works: The authors built larger and smaller systems that paired a action agent with a memory agent. The action agents were based on Claude Sonnet 4.5, Claude Opus 4.6, or Qwen3.5-122B-A10B (which presumably is much smaller). The memory agents were based on Claude Opus 4.6 or, in the smaller system — after supervised fine-tuning and reinforcement learning on SETA command-line problems — Qwen3.5-27B.

• Given a description of a problem and the action agent’s eight most recent outputs or tool calls, the memory agent updated a set of problem-related notes that included facts, environment details such as the current working directory, successful fixes to issues encountered earlier, unfinished parts of the problem, and failed commands (to avoid repeating them).
• At every step, the memory agent added to the action agent’s input either nothing or a short reminder. A reminder could point to a part of the problem the action agent had ignored, a fact of the environment (for instance, available tools), or a command that had failed.

Results: In the larger system, the memory model raised the success rates of Claude Sonnet 4.5 and Claude Opus 4.5 on Terminal-Bench 2.0 and τ2-Bench. In the smaller system, it raised the fine-tuned Qwen3.5-27’s performance on Terminal-Bench 2.0.

• On Terminal-Bench 2.0, which contains 85 command-line problems, Claude Sonnet 4.5 reached 45.9 percent success with the larger memory model versus 37.6 percent without it. Similarly, Claude Opus 4.6 reached 45.9 percent with the memory model versus 43.5 percent without it. Qwen3.5-27B achieved 41.1 percent success with the smaller memory model and 37.6 percent without it.
• On τ2-Bench, which contains 278 airline, retail, and telecom problems, Claude Sonnet 4.5 reached 61.8 percent with the memory model versus 55.0 percent without it. Claude Opus 4.6 with the memory model reached 68.7 percent versus 66.2 percent without it. (The authors didn’t test the smaller system on τ2-Bench.

Why it matters: Using a separate agent to manage an agent’s context improves its performance decisively on the problems in this study. The memory agent doesn’t touch the model that actually does the work, so you can bolt it onto existing agents without retraining. It also means you can easily experiment with swapping models or agentic frameworks.

We're thinking: In a separate result, the authors found that their system outperformed one that reminded the action agent of relevant information at every step. Apparently reminding an agent is not enough. It’s necessary to remind it at the right moments.

Work With Andrew Ng

Join the teams that are bringing AI to the world! Check out job openings at DeepLearning.AI, AI Fund, Landing AI, and LearnVector.

Subscribe and view previous issues here.

Thoughts, suggestions, feedback? Please send to thebatch@deeplearning.ai. Avoid our newsletter ending up in your spam folder by adding our email address to your contacts list.