Search This Blog

Powered by Blogger.

Blog Archive

Labels

Footer About

Footer About

Labels

Showing posts with label Anthropic AI security. Show all posts

Anthropic Warns AI Worms Could Soon Spread Across IoT Devices

 

Small artificial intelligence models could soon be able to freely replicate across internet devices, forming botnets that do not require centralized computers to control them, according to an Anthropic Frontier Red Team paper. The paper’s authors explored if foundation models could be run on unknown cheap devices and execute the actions required to replicate across other interconnected devices. Anthropic bought 15 cheap Android TV boxes for $55 each to see how large an open-weight model could be before it could no longer be hosted on such devices. 

The team found that a model with 26 billion parameters could generate 0.39 tokens per second. They also found that a 9 billion parameter model could run on 8 out of 9 of the different hardware types they tried. The team then tried to see if a model could replicate across 8 emulated Android devices attached to a network. To make matters easier for the would-be-attackers, the researchers made the hacking process trivial; they only had to focus on the artificial intelligence processes. 

Only the GLM-5.2 among the open-weight models was able to fully complete the replication process, doing so on four out of five tries, and managed to infect all 8 devices on the network. Anthropic warned that the experiments should not be considered an accurate representation of how such an attack might unfold in the real world, but the results nevertheless show that attacks are becoming increasingly likely as models grow more capable. 

Models with over 30 billion parameters do not have the ability to fully perform all actions required to replicate, but they can manage single aspects of the process. If development proceeds at its historical rate, then models will be able to achieve the capability demonstrated by GLM-5.2 by 2026. Self-replicating AI models could follow soon after. The researchers also estimated the scale of such an attack. 

If there was a Kimwolf-class botnet that had already successfully infected over 2 million devices, using up all 15 of the TV boxes mentioned earlier would result in 133000 instances of the 26-billion-parameter model being created. This would amount to a combined performance of up to 40 Nvidia H100 GPUs, but it would take more than a day for all instances to be created. Such an attack surface is hard to prevent because there are no singular points of failure, but even if such a worm managed to successfully infect a network of devices, it would still be very visible to most network infrastructure. 

Large traffic volumes and unusual communications patterns between devices that would otherwise not communicate with each other will raise flags with most internet service providers. Anthropic also found that manufacturers could reduce the threat surface by replacing default passwords with more complex ones. Many previous botnets, including the infamous Mirai malware, were nearly impossible to stop because many IoT devices had default passwords that were too simple to guess. 

Anthropic warned that while a fully autonomous worm was not yet available, a continuing increase in model size and capability, along with the growing ability of models to apply knowledge between different model classes, will severely limit the time defenders have to fix vulnerabilities in Internet of Things devices.

Encrypted AI Reasoning Flaw Exposed Secrets Across Major LLM APIs

 

A newly reported security weakness in the systems used by OpenAI, Anthropic and Google to preserve hidden AI reasoning between API calls has allowed researchers to reconstruct internal reasoning and uncover sensitive information, including API keys, passwords and access tokens.

The issue involves encrypted reasoning objects generated by the companies' reasoning APIs. Although these objects are designed to keep internal reasoning hidden from developers, researchers found that an object generated during one session could be replayed in another. In some tests, the same objects could also be supplied to a less capable model from the same provider, which could then help reconstruct the concealed content.

The findings are detailed in a paper titled Stealing Reasoning Traces from Proprietary LLM APIs. Researchers identified four potential attack scenarios: extracting proprietary reasoning that could be used for model distillation, recovering private information from publicly shared agent traces, reconstructing harmful material hidden behind seemingly safe responses, and embedding prompt injections inside encrypted reasoning objects.

Hundreds of thousands of reasoning blocks examined

Researchers analyzed 6,708 publicly available agent trajectories and successfully decoded 315,320 reasoning blocks. After removing benchmark-related material, they identified 704 separate privacy-related artifacts originating from genuine user sessions. These included 62 API keys, 33 passwords, 24 access tokens and seven private keys.

The researchers stressed that the cross-user technique did not amount to unrestricted access to private conversations. An attacker needed access to an encrypted reasoning block, such as one exposed through a publicly available agent log, as well as API access to a compatible model belonging to the same provider.

The researchers said they reported the findings to the affected AI companies, Microsoft and Hugging Face. According to the paper's reproducibility statement, the demonstrated attacks stopped functioning following mitigation efforts, with the primary extraction technique no longer reproducible as of August 2026.

There is no indication in the report that the technique has been exploited maliciously in the wild. However, developers are advised to remove reasoning blocks and other opaque reasoning fields before sharing agent traces and to avoid storing raw API transcripts, even when the visible portions have been sanitized.

Encryption was not broken

The vulnerability did not involve cracking the encryption protecting the reasoning data or obtaining an encryption key. Instead, the researchers exploited the fact that the encrypted objects remained valid and could be accepted by compatible systems.

The underlying design is intended to preserve reasoning state when developers manage conversation history manually or use stateless API workflows. OpenAI can provide encrypted reasoning items for applications that replay manually managed history, Anthropic uses encrypted signatures to carry reasoning, while Google's systems use encrypted thought signatures.

During their experiments, researchers found that these objects could be moved between sessions, accounts and, under certain circumstances, models. A less powerful compatible model could then be used as what the researchers describe as a "fuzzy" decoder to reconstruct reasoning generated by a more capable model.

The models tested as decoders included Claude Haiku 4.5 for Claude traces, GPT-5.6 Luna for GPT traces and Gemini Robotics ER-1.6 for Gemini traces. The researchers instructed the models to reproduce reasoning that had originally been generated by a stronger model.

Published agent logs created a privacy risk

The cross-user aspect of the research raises particular concerns for developers who publicly release raw agent logs containing encrypted reasoning objects.

Of the 704 non-benchmark privacy artifacts recovered by the researchers, 64 were found exclusively within hidden reasoning and did not appear anywhere in the visible portions of the corresponding traces. This means that simply removing secrets from readable conversations may not be sufficient if the original encrypted reasoning object remains available.

The researchers described the exposure as limited rather than universal. The demonstrated scenario primarily affected developers who had published raw agent traces containing intact reasoning objects, rather than automatically exposing every user of the affected APIs.

Encrypted reasoning could also carry hidden instructions

The portability of reasoning objects created another potential security concern. Researchers demonstrated a proof of concept in which an opaque reasoning block contained a malicious instruction that was subsequently replayed during an unrelated task.

Because the instruction was hidden inside the encrypted object rather than visible in the conversation, the receiving model could process the injected command without the malicious text appearing in the readable prompt. In the demonstration, this resulted in the model adding an attacker-controlled upload action.

The researchers noted an important limitation: they did not possess the original plaintext reasoning for proprietary models, meaning they could not establish that every reconstructed trace was an exact reproduction. Instead, they assessed fidelity using reasoning-token counts and qualitative comparisons. The reconstructed outputs generally followed the thinking-token counts reported by the providers.

Provider documentation has since evolved

Current documentation indicates that encrypted reasoning remains part of these API architectures, although the way such data is handled has changed.

OpenAI continues to instruct developers who manually manage stateless conversation history to replay encrypted reasoning items. Google, meanwhile, says its backend handles thought compatibility when a session changes models.

Anthropic's documentation now states that thinking blocks are associated with the model that generated them and recommends removing them when switching models because other models will ignore those blocks.

However, several issues remain unresolved publicly. There has been no public acknowledgment from OpenAI, Anthropic or Google directly linking their current documentation or mitigations to this research. As a result, the researchers' reproducibility statement remains the primary public basis for the claim that the demonstrated attacks are no longer effective.

It is also unclear whether reasoning blocks that were already published online remain vulnerable to extraction. The researchers decoded hundreds of thousands of blocks that were already available in public repositories, but the study does not establish whether those previously exposed objects can still be decoded after the providers' mitigations.

Research builds on earlier replay discovery

The findings expand on research published in May by Johns Hopkins cryptographer Matthew Green. His work showed that encrypted reasoning blocks could be replayed across sessions and accounts, although it did not demonstrate a reliable method for extracting secrets from them.

Green said he reported the replay behavior to OpenAI and Anthropic through their bug-bounty programs. According to his account, OpenAI considered the report unreproducible, while Anthropic said it did not identify security implications associated with the replay or related side-channel behavior.

The latest research takes that previously identified replay capability further, demonstrating how portable reasoning objects could potentially be used to reconstruct hidden model reasoning and expose sensitive information at scale. The findings highlight the need for developers to treat encrypted reasoning data as potentially sensitive, even when its contents are not directly readable.