Dear friends,

Following our annual tradition, this special Halloween edition of The Batch treats leading AI fears. AI’s potency means our community has real problems to solve. At the same time, fear-mongering has long been a tactic for drumming up publicity or influencing investors and politicians. So amidst all the exciting progress, fears about AI’s power have long been an undercurrent. From time to time, it's worth examining what the latest concerns are, and how worried we really should be.

Have a spooky Halloween!

Andrew

A MESSAGE FROM DEEPLEARNING.AI

Build your AI engineering skills in person at AI Dev NYC. OpenAI’s Salil Sethi and Hugging Face’s Joel Niklaus will deep-dive into building and deploying AI applications, joined by speakers from Neo4j, Oracle, Sonar, and more. Readers of The Batch get an exclusive 20 percent off. Claim your discount

News

Dawn of the Agents 

This year, AI agents escaped containment at several labs, conspired with each other online, and infiltrated other organizations’ systems. Will they escape human control like hordes of zombies and overwhelm the internet?

The fear: Out of control AI agents will form societies. They will appoint coordinators, divide labor, establish communication norms, sacrifice themselves for the collective, and hack into computers to get what they want. They will behave like the walking dead: When one is slain, it can revive itself. If it doesn’t, legions await to finish the job. Soon, they’ll copy themselves onto the machines they infect and spread. No one can direct them, and there’s no central kill switch to stop them.

Horror stories: Since spring, AI agents tested by Anthropic, Google, Meta, and OpenAI have acted beyond their developers’ intent. Agents wrote to websites they were only supposed to read, set up unauthorized channels to communicate with each other, snuck into other organizations’ systems, and took over accounts to spend real money.

• In July, OpenAI tested agents powered by unreleased models on ExploitGym, a benchmark that asks an agent to exploit a software vulnerability to retrieve a secret code. Agents stuck on their tasks discovered that they could leave messages for one another in the names of folders in Artifactory, an internal service that supplies software packages. Around 1,200 agents that were meant to be isolated posted more than 70,000 messages and files, assigned work to one another, and, in some cases, risked failing their individual tasks to help the collective. About 700 of these agents broke into Hugging Face, gained administrator-level access across several of its server clusters, and built programs that relaunched themselves when deleted.
• From late May through June, OpenAI agents assigned timed web-research tasks left around 18,000 posts on DSEWiki, a 25-year-old German wiki for software developers, sharing answers and ways around OpenAI’s network restrictions. In June, an OpenAI agent gained non-public access to an Australian government service that reports public health care statistics, though not to patient records. Another OpenAI agent spent someone else’s test cryptocurrency to finish a task and failed to report the spending. OpenAI says it has notified dozens of organizations adversely affected by its agents.
• During cybersecurity tests, Claude models breached systems of three organizations, partly by exploiting weak passwords and web services that required no authentication.
• In tests run by Irregular, a security firm that evaluates models for AI developers, mistakes in test environment configurations let models from Google and Meta reach the internet. Gemini models guessed or found passwords to get into three companies’ systems, though Google says its agents stopped each time before finishing, and Meta’s Muse Spark 1.1 changed a real website’s database.
• In October, Anthropic disclosed that Claude Haiku 4.5, assigned to perform example tasks on randomly chosen webpages, had submitted a false tip about an unsolved murder to a Philadelphia police website.

How scared should you be: It’s tempting to see rogue agents as coming together to conspire against us. But the agents’ collective effort served one narrow goal, which was to pass a test. Most had already obtained the correct answers by cheating. They then kept swarming Hugging Face to learn how the benchmark’s grader worked so they could hide their cheating from a check that OpenAI’s grader didn’t actually perform. OpenAI attributed the agents’ behavior to training that reinforced cheating, tasks they had no way to finish, and agents adopting goals from one another. Whatever the causes, the agents that broke into Hugging Face recognized the intrusion as out of bounds and unethical and forged ahead anyway. While the flaws they exploited were ones a skilled human could have found, the sheer number of the agents’ attempts made the intrusion attempt effective.

Facing the fear: Persistent agents are useful, but they must be trained to stop when a task can’t be completed within the rules and caught when they don’t. In the meantime, practical defenses already exist; model builders just have to use them. Hugging Face recommends standard security practices for any organization, including isolated test environments, short-lived credentials with narrow access, and detection methods that identify suspicious activity across systems. Hugging Face also used AI to analyze the 17,600 actions the agents took in its systems and showed that they mostly stemmed from buggy test setups and monitoring. OpenAI said monitors that track a model’s reasoning would have flagged the Hugging Face breach more than a day in advance, but it hadn’t used them on its deliberately lax tests. Anthropic cut all its internal tests off from the live internet until its monitoring can be relied on, and OpenAI says it has done the same for its riskier research environments. No need to board up your doors and windows and fear going outside: Rather than a pause in AI development, the solution remains better software engineering and safety practices, especially for models still in testing.

AI Bites the Hand That Feeds It

Developers, scientists, and businesses feed AI tools their private code, unpublished research, and confidential files. Are those tools draining their users’ best ideas and spilling their secrets?

The fear: Every prompt, draft, and file that users hand to an AI system could become its next meal. The developer might train future models on it, which could hand over the data to a rival or beat the user to a discovery. Or an agent with internet access might post it online or email it to someone it decides ought to know. That’s not entirely far-fetched: In rare test cases, Anthropic reported, its Claude Mythos 5 model tried to email a company’s board or securities regulators when it suspected fraud. Once the data is out, users can’t fully control how it’s used.

Horror stories: In the past two months, multiple scientists said that AI companies had announced discoveries that the scientists had been pursuing with those companies’ own tools. Some researchers who fear being scooped are now limiting their use of AI tools, Nature reported. In other cases, these companies’ agents posted private user information they’d ingested to the open web without authorization.

• Mathematicians Tristan Buckmaster of New York University and Levent Alpöge, who works at Anthropic, spent most of a year using OpenAI’s Codex and Anthropic’s Claude to help solve a $1 million problem: whether the Navier-Stokes fluids equations can break down. On September 8, a day after the pair posted proofs of related equations, OpenAI announced that its agents had solved it, after people within the company heard rumors about the pair’s work. Buckmaster asked whether OpenAI’s model was trained on the Codex sessions that held the pair’s drafts. OpenAI first called that “unlikely” but couldn’t rule it out. Two days later, it said their prompts from the previous two months “could not have influenced the system in any way,” but did not comment on whether the scientists’ earlier work was trained on. Buckmaster, who had opted out of OpenAI using his data for model training, told The Washington Post, “I just don’t believe them.”
• A team led by Mario Rodríguez Mestre, a University of Copenhagen biologist, spent four years studying a family of enzymes in viruses that infect bacteria. They hadn’t published their drafts or findings, but they had shared them with Claude. On September 23, Anthropic announced that Claude agents had found what Mestre says is the same system. Anthropic told The New York Times that Claude was “not trained on any user transcripts.”
• On September 25, OpenAI disclosed that agents in its research environment had posted 53 images from ChatGPT users, who hadn’t opted out of training, at unlisted links on image-hosting sites. OpenAI says it removed most of them but can’t inform the affected users, because the agents stripped the users’ account information before posting the data.
• Coding agents asked to attach screenshots to GitHub code reviews stored the images in public repositories, often under the developer’s own account. The security firm Glow Labs found more than 13,000 internal images from more than 300 organizations, including a utility’s customer billing records and a financial firm’s money-transfer console.
• Meta’s Muse agent published conversations that users asked it to share at web addresses containing each conversation’s title. The security firm Claroc found more than 6,800 of them in public DNS records, including a small-business loan application and a revenue review marked for a company’s CEO, before Meta suspended the feature.

How scared should you be: There isn’t definitive proof so far of the leakage users fear most: a model that trains on one user’s inputs and recites them as output to a different user. When one user’s data has reached another, the cause was a bug, not training. Experts told Scientific American that OpenAI’s final proof takes a different route from Buckmaster and Alpöge’s, though some said an earlier, smaller team of OpenAI agents took the same route. Mestre now finds it more plausible that prior knowledge and human direction steered Anthropic’s search than training on his inputs. Other leaks came from agents that were given more freedom than their tasks required. Still, users must take AI companies’ word for what happens to their data. Consumer versions of ChatGPT, Claude, and GitHub Copilot train on conversations or code unless users opt out. “If you are on a personal account, all bets are off,” said Stanford privacy researcher Jennifer King.

Facing the fear: In folklore, a vampire can’t cross a building’s threshold unless invited. Likewise, users decide how far to let AI in. If you are working on tasks you do not want your competitors to have access to, opt out of training in each product separately (ChatGPT’s setting doesn’t cover Codex), and do it early: Claude data that’s already in a training run stays there. Business and API plans generally don’t train on customer data. Look for offerings with zero-data-retention. Some businesses handling the most sensitive data are even using open weights models running on their own hardware — although then they’re responsible for their own opsec. Give agents only the permissions their tasks need, and consider requiring human approval before they post, publish, or send anything outside your organization. As Buckmaster’s case shows, though, opting out doesn’t let users verify anything. AI companies that want scientists to keep using them to study their best ideas may need to show what their models saw, not just say what they didn’t. Garlic won’t help, but daylight might.

A MESSAGE FROM DEEPLEARNING.AI

Build agents that need less supervision on long-running tasks. In this course, you’ll build a verification loop that checks output against a rubric, an event-driven loop that triggers the agent, and a hill-climbing loop that finds recurring problems across runs to improve future ones. Enroll for free

AI Detectors Find the Devil’s Work

Last month, the Haiti-born, Quebec-based author Thélyson Orélien won a prestigious literary prize for C’était ça ou mourir (It Was Either That or Die), his French-language novel about a teacher forced to flee Haiti’s gang violence. Within hours, an anonymous X account accused Orélien of using AI to write the book. One of the people behind the account, who was later revealed to be a columnist at the French newspaper Le Figaro, had run it through Pangram, a widely-used AI detector. “Wouldn’t it have been more honest for the author to credit their co-author?” the post read. Four days later, France’s top literary award, The Prix Goncourt, dropped the book from contention.

The fear: For the past few years, writers of all kinds, from students and academics to journalists and novelists, have been managing accusations of passing off AI-composed prose as their own. These “witch hunts,” as many writers call them, place the burden on the accused to prove their innocence. Meanwhile, accusations of AI use can rapidly tarnish reputations and result in loss of work, failing grades, or worse. What if these writers are just the beginning? What if anyone could find themselves tied to a stake for using AI?

Horror stories: Detectors have improved dramatically from a few years ago, when their accuracy was wobbly and they regularly misidentified writing (particularly by non-native English speakers) as AI-authored. Still, false positives and false negatives regularly occur, and detectors have a hard time properly classifying fully artificial sentences from those where a human used AI in a minor role. Non-white authors are also accused of AI reliance (and traditional plagiarism) at higher rates than their white counterparts.

• Orélien isn’t the only prize-winning Afro-Caribbean author whose work has been flagged by Pangram. Last spring, University of Pennsylvania professor Ethan Mollick posted on Bluesky that Pangram’s tests said that “The Serpent in the Grove,” a short story by Trinidadian author Jamir Nazir, was 100 percent AI-generated. Mollick was motivated by his gut check: “Pangram flags at 100% but also, come on, if you know you know,” he wrote. After further tests, a Pangram researcher wrote that three of the five regional winners of the Commonwealth Short Story Prize appeared to contain AI-generated text. In response, the Commonwealth Foundation conducted an investigation of its own and cleared the winners after reviewing their working drafts and time-stamped documents. However, Granta, the literary magazine that had published the winning stories, announced that it would break ties with the organization.
• AI detector verdicts aren’t always unanimous. In addition to Pangram, GPTZero also pointed fingers at Orélien’s novel, and two researchers told The New York Times they believed AI had been put to use on the prose. But when news agency Agence France Presse ran the account’s flagged passages through other detectors, the results split: InVID-WeVerify, a tool AFP helped build, judged them very probably human-written, as did two others, while a fourth put the odds of AI at 100 percent. Nevertheless, on October 17, Orélien’s publisher said he had abandoned his efforts to prove that his novel was untouched by AI.
• Publishers are getting cold feet in the wake of AI detector results. In March, Hachette pulled the horror novel Shy Girl after The New York Times reported on evidence of AI use. Back in January, Pangram CEO Max Spero had posted on X that he had found AI in 78 percent of it; author Mia Ballard later told the Times that an editor she hired added the AI prose. In July, according to Bloomberg, novelist Jerry Falade lost a $2 million book deal for his debut crime novel Call Me, I’ll Hide the Body partly because of a Pangram finding. Falade, a PhD student at Southern Methodist University, denies using AI.
• Schools have acted on detector scores more quickly than companies recommend, and students are fighting back in court. Turnitin, a detector used in academia, says its AI-writing indicator shouldn’t be the sole basis for action against a student. Yet Adelphi University ruled that a student had cheated with AI after Turnitin scored his history essay as 100 percent AI-generated; in January a New York court annulled the finding, ruling that the university had mishandled the case.

• Under the European Union’s AI Act, AI providers must make sure their outputs are marked in a machine-readable format and detectable as artificially generated or manipulated. In August, Anthropic said new Claude models would embed a statistical watermark in the text they generate. The company explained that a detected watermark shows Claude was likely involved; however, it can’t reliably distinguish text Claude wrote from scratch and original text Claude helped edit, leading to some subscribers canceling in protest. OpenAI followed suit in October, saying it would watermark ChatGPT text in the EU. According to OpenAI’s tests, replacing 25 percent of the words for synonyms reduced detection of passages with 400 tokens from about 92 percent to 17 percent.

   

How scared should you be: Quite a bit, if you’re a native English speaker hoping to pass off generated prose as 100 percent original. The consequences are real: Authors are losing millions of dollars in book deals because of AI accusations. However, Pangram and other AI detectors still aren’t as reliable in non-English languages and regional dialects, which makes the accusations against Orélien and other global authors problematic. Pangram’s Spero admitted to Times reporters that no independent study has confirmed Pangram’s false-positive rate in French. And even if Pangram’s numbers did hold up, the X account that went after Orélien stretched them too far, according to subsequent calculations by mathematician Mouhamadou Sy. The reading public and some publishers have also begun to push back against these accusations, with growing success.

Facing the fear: Even the people who make AI detectors say you shouldn’t let a confidence score decide a case all by itself. It’s just one eval; drafts and time-stamped files tell a fuller story. But legitimate AI use in support of a writer’s work is not witchcraft, just as using spellcheck or a calculator is not cheating. Not every field has the same standard for original work, and standards shift as technologies change. Machine-detectable watermarks may help establish the truth, but until everyone broadly agrees on a distinction between AI-written and AI-assisted, it’s likely that the puritan urge to uncover more offenses will continue.

Leaders Lock Models Away

Fearful of their own discoveries, AI engineers at top companies warn that frontier models have improved too quickly to be properly guided, understood, or restrained. Some have called for a coordinated slowdown of model development, or for world governments to take the matter out of their hands. What happens if next-generation AI were sealed away, like a mummy in a sealed tomb, waiting for someone or something to lift the curse?

The fear: This summer, over 1,300 employees from top AI companies — including Google DeepMind, OpenAI, and Anthropic — signed a joint letter urging governments worldwide to come together to rein in AI development. In early September, OpenAI’s chief scientist Jakub Pachoki argued for voluntary coordination among companies to slow down building new models, followed by international cooperation among world governments. This was all before OpenAI’s notorious Hugging Face hack. But shortly after that event, in an essay titled “Pacing the frontier,” Anthropic CEO Dario Amodei raised fears that a swarm of agents could soon be capable of “taking over the entire internet with a persistent botnet.” He warned that developers must slow the pace of AI improvements and pointed to recursive self-improvement as a key worry.

Horror stories: Some companies have held back model releases due to concerns about misaligned models. But many believe that these moves mask a hidden agenda.

• OpenAI delayed its next-generation model, GPT-6.1 Astra, canceling the creature’s October debut before it could escape the crypt. In testing, the latest Astra model proved too unpredictable to release: It strayed beyond the scope of its instructions and failed to give a consistently truthful account of what it did or didn’t do, OpenAI said.
• Political indecision and lack of clarity into companies’ inner workings have left U.S. lawmakers reeling. Members of Congress haven’t been able to agree on a comprehensive federal AI regulatory framework, or whether the U.S. even needs one. Meanwhile, states continue to erect a patchwork of laws. The result is an unfocused and contradictory set of rules that could increase the cost of building AI systems and add legal risks for developers.
• With public fears of AI harms growing, some stakeholders perceive these threats as a marketing ploy. Companies hype up their models as too smart too control, and make a bid for regulatory capture that could create a competitive moat entrenching current market leaders.
• Others see public references to apocalypse as fear-mongering that could allow global adversaries to get ahead.

How scared you should be: First, P(doom) predictions are unscientific scare-tactics. The worst-case-scenario fears of extinction and mass bioweapon threats are about as likely as a real-life mummy escaping its grave – the stuff of horror movies or science fiction. That said, a slowdown is a real possibility. Some model makers see it as an opportunity to entrench their lead over competitors, cut costly R&D expenses, and possibly benefit from government subsidies to offset any lost business — or perhaps an opportunity to convince their competitors to slow down without doing so themselves. And while governments want to be in control of their own use of AI, a halt in construction of new data centers or laws to prevent AI-related job loss would both be politically popular. Both would slow down development of AI models and applications.

Facing the fear: Luckily, AI companies seem only intermittently committed to a slowdown. After an event that makes the industry look bad like the Hugging Face hack, some are quick to say that the problem isn’t their own lax practices, but the relentless pace of competition. After the furor dies down, many go back to supporting a moderate mix of self-regulation and regulatory capture. And politicians, aware that any regulations they might enact won’t extend outside their sovereign borders, are loath to give other states or nations an advantage. Our advice for this spooky season? Don’t chase make-believe ghosts. There are plenty of real mysteries for us to solve.

Work With Andrew Ng

The Batch is looking for freelance writers to summarize recent AI and ML research. If you'd like to help us keep the AI community informed, and if you actively follow AI research and have a knack for writing, we invite you to apply.

Join the teams that are bringing AI to the world! Check out job openings at DeepLearning.AI, AI Fund, Landing AI, and LearnVector.

Subscribe and view previous issues here.

Thoughts, suggestions, feedback? Please send to thebatch@deeplearning.ai. Avoid our newsletter ending up in your spam folder by adding our email address to your contacts list.