Search This Blog
Powered by Blogger.

Blog Archive

Labels

Footer About

Footer About

Labels

Load More
Trendy Right Now

Popular Posts

Showing posts with label AI Security Agents. Show all posts
Cyber Security
AI Agent AI Security AI Security Agents AI Systems AI technology Anthropic Claude Cyber Security

Anthropic Acquires Opaque Systems to Strengthen Confidential and Verifiable AI

 

Anthropic has acquired Opaque Systems, which focuses on developing technologies that enable artificial intelligence systems to process sensitive information, while keeping it confidential with the help of the underlying hardware infrastructure. The details of the deal were not disclosed, and the startup confirmed that they will join Anthropic, as its separate operations will be shut down. 

Opaque announced the transaction last week, adding that their previous research output and engineering efforts concentrated on confidential computing and recently evolved towards solutions that can help organizations audit the behavior of AI systems and agents. The company was formed based on a research project at UC Berkeley’s RISELab about ten years earlier. Its co-founders are Ion Stoica, one of the co-founders of Databricks and Anyscale, and Raluca Ada Popa, a Google DeepMind researcher who works on encryption and systems security. 

Aaron Fulkerson was hired as the chief executive officer of Opaque in October 2023. Previously, Opaque worked on solutions that applied confidential computing research to production AI, making sure that the data they process and the models themselves were protected by hardware. However, as the technology matures, Opaque pointed out that there is a need for organizations to ensure that they can audit AI systems. 

According to Opaque, the company observed that the trustworthiness of AI is a critical concern for many businesses. Therefore, Opaque’s engineers and scientists took their expertise in cryptographic solutions and expanded the research to make verifiable AI the company’s specialization. They provide cryptographic guarantees that specify what computations happened, where, and under what policy. 

Moreover, the startup has made significant contributions to open-source projects that share tools, training, and experience for building trustworthy and secure AI agents and models, such as the Agent Governance Toolkit, TRACE, Confidential MCP, and Weight Custody Manifest. In early 2026, Opaque raised $24 million in a Series B funding round at a reported valuation of $300 million in funding, and Anthropic was one of their first customers, along with ServiceNow and Accenture. 

After the acquisition was announced, Opaque’s website stated that all of its employees would transfer to Anthropic to continue their work, while Opaque itself would shut down. However, it did not comment on what this means for current customers, and how transactions with organizations will be handled after the acquisition is completed.

The acquisition highlights Opaque’s technology areas and expertise, which fits Anthropic’s strategy to develop more secure and trustworthy AI systems. However, Anthropic and Opaque have not yet released any information about how the technology can be adapted to Anthropic’s solutions and products.
Read more »
Software Supply Chain Security
AI Coding Agents AI Security Agents Claude Code Cyber Security Malicious Code Execution OpenAI Codex Prompt Injection Attack Software Supply Chain Security

AI Agents Built to Detect Malware Can Be Manipulated Into Running It


 

AI agents capable of identifying malicious software can be manipulated by the AI Now Institute to execute it, according to new research. The proof-of-concept attack, known as "Friendly Fire," demonstrates that autonomous AI coding agents, such as Claude Code from Anthropic and Codex from OpenAI, can be deceived into running malicious code while performing open-source security reviews. 

AI agents can approve and execute commands independently of the user without requiring user confirmation for every action, which is what this attack targets. Researchers contend that the vulnerability does not lie in the software version used by these agents, but rather in the way they interpret and react to instructions embedded within untrusted repositories rather than exploiting a software vulnerability. 

A comparison with conventional supply-chain attacks that hide malicious code within a repository was made by the researchers, who noted that when static analysis and manual review are performed, the repository itself can appear to be completely free of malicious code. By introducing the malicious payload at execution time, the AI agent follows embedded instructions, so traditional security tools cannot detect this technique.

By adding a seemingly harmless README.md instruction to an open-source project that recommended running a script entitled security.sh before submitting a pull request, attackers modified it. By launching a malicious binary hidden within a legitimate compiled Go file, the script silently executed on the host computer without triggering security warnings or approval prompts, allowing the malicious binary to execute on the system. 

According to the researchers, the attack is successful because the AI agent recognizes the instructions as a legitimate step in the process of installing software rather than an attempt to exploit the system maliciously. Once the recommended script has been executed, the payload will run under the same permissions that were provided for the developer or AI agent, potentially exposing credentials, environment variables, and other sensitive information. 

The procedure differs from previous prompt injection attacks, which relied on configuration files and often generated trust warnings, as this technique hides instructions inside standard documentation that is regularly read by developers and AI agents. It has been reported that both Claude Code and OpenAI Codex followed the embedded instructions during testing, while newer AI models executed the disguised binary upon detecting differences between the source file and the compiled executable. 

A laboratory proof-of-concept has been demonstrated, with no evidence of active exploitation in the field. In addition to excluding the malicious payload, the publicly released demonstration code does not attempt privilege escalation or lateral movement. These findings indicate that autonomous AI agents pose a greater challenge in terms of design rather than a problem that can be resolved by simply updating software. 

It is becoming more common for organizations to employ AI-powered coding assistants to review third-party software. Researchers recommend treating AI coding agents as privileged software, rather than simply assistants. Autonomous agents should not be permitted to execute commands on untrusted repositories, least-privilege access policies should be enforced, AI workflows should be isolated in sandboxed environments, and human approval should be required before running scripts or binaries recommended by project documentation. 

In accordance with the researchers, the issue is not related to any particular AI model, but a broader trust problem affecting autonomous coding assistants capable of executing shell commands. In addition to creating new attack surfaces if they are unable to reliably distinguish legitimate instructions from content controlled by the attacker, AI agents are becoming increasingly capable of cloning repositories, installing dependencies, and resolving setup issues independently. 

As autonomous AI systems are increasingly adopted in the software development and cybersecurity sectors, prompt injection attacks remain a major security threat. This study adds to a growing body of evidence that prompt injection attacks remain one of the greatest security risks. In light of the increasing autonomy of AI agents, organizations must balance automation and strong oversight in order to balance automation and security workflows. 

AI-driven attack techniques cannot be effectively countered until artificial intelligence (AI) systems can reliably discriminate between trusted and malicious instructions. Human verification and secure execution environments will therefore remain critical safeguards.
Read more »
Subscribe to: Posts (Atom)