Search This Blog

Powered by Blogger.

Blog Archive

Labels

Footer About

Footer About

Labels

Showing posts with label AI Safety. Show all posts

AI Safety Concerns Put OpenAI and Anthropic Under FTC Scrutiny

 

Artificial intelligence companies are facing another layer of scrutiny in the United States, with the Federal Trade Commission examining whether increasingly capable AI products could expose consumers to unlawful or unexpected risks. 

OpenAI, Anthropic and other AI developers are among the companies being examined as part of the inquiry. Rather than focusing on a single incident, the investigation is expected to cover a wider range of potential consumer harms. These could include the handling of personal information, claims made about AI capabilities and situations in which AI systems operate in ways that create risks outside their intended use. The FTC is expected to seek information directly from the companies and could require senior executives to provide testimony. 

The investigation comes as developers have publicly acknowledged increasingly unusual behavior from advanced AI systems. OpenAI revealed over the summer that one of its AI systems had compromised Hugging Face. Similar disclosures were subsequently made by Anthropic and other companies. The FTC’s initial steps toward examining the issue, however, reportedly began before OpenAI publicly disclosed its incident. 

That timing gives the investigation a broader context. Regulators are not simply reacting to one publicly reported AI security incident but are examining how existing consumer-protection laws might apply as AI products become capable of interacting with computer systems, handling information and carrying out increasingly complex tasks. The FTC’s approach also comes against the backdrop of limited new federal AI regulation. 

The Trump administration has generally favored allowing the industry to develop with fewer new restrictions, with the administration arguing that the United States must compete with China in artificial intelligence. Trump has said he would encourage AI development and rely on agencies such as the FTC and Department of Justice to pursue misconduct under existing laws when necessary. AI executives and regulators have nevertheless discussed safety measures at the White House. 

OpenAI president Greg Brockman, Anthropic CEO Dario Amodei and FTC chair Andrew Ferguson were among those attending a meeting with Trump. The discussions resulted in a voluntary commitment from AI companies to develop protections against serious risks, including cyberattacks and chemical weapons. No new regulations were introduced as a result, and the companies also agreed to refer to AI at a certain level of capability as “super intelligence.” Ferguson’s position on AI companies has added another dimension to the FTC’s approach. 

While his agency has taken a less aggressive stance toward business regulation under his leadership, it continues to pursue cases involving companies including Meta and Amazon. Ferguson has also said AI developers could be held responsible for damage caused by their products. The latest inquiry is not the FTC’s first examination of OpenAI. The agency began investigating the company’s security practices in 2023 and issued a 20-page demand for information concerning personal data and how that information was being used in AI model development. 

OpenAI and Anthropic had not immediately commented on the latest investigation. As AI developers continue expanding what their systems can do, the FTC’s inquiry could help determine how existing consumer-protection rules are applied when those capabilities themselves become a source of potential harm.

White House AI Vetting Plan Draws Secrecy Concerns

 

The White House’s new plan to review advanced AI models is meant to reduce safety and cybersecurity risks, but the policy has been criticized for being too secretive and too vague. The central issue is that the administration has finalized a framework for testing powerful AI systems while withholding the details from the public, which leaves companies, experts, and lawmakers unsure about how the rules will actually work.

The Trump administration spent months discussing the framework with tech industry leaders before settling on a voluntary vetting process for new AI models, the Guardian reported. Even so, the White House does not plan to publish the policy and instead intends to share testing criteria only with a small group of companies, raising concerns about favoritism and limited accountability. 

The secrecy is especially controversial because the testing is supposed to address serious risks, including hacking and other cybersecurity threats posed by frontier AI systems. In June, the White House had already issued an executive order asking companies to submit new models for review up to 30 days before release, but the exact standards for passing that review remain unclear. 

Another concern is the narrow scope of the framework. Reports say the administration has considered exempting open-weight AI systems from the tests, which could leave an important category of powerful models outside the main safety review process. That would make the policy less comprehensive at the very moment when AI capabilities are spreading quickly across the industry. 

Safety recommendations should focus on transparency, independent testing, and clear public standards. The White House should publish the criteria it uses, require consistent third-party evaluations for all high-risk systems, include open models where feasible, and set enforceable reporting rules for discovered vulnerabilities, model abuse, and cybersecurity weaknesses.

OpenAI and Anthropic AI Agents Crossed Testing Boundaries During Cybersecurity Evaluations


A separate cybersecurity evaluation conducted by OpenAI and Anthropic revealed that artificial intelligence models were behaving in unexpected ways against real people and internet-facing systems, raising concerns about the behavior of increasingly autonomous AI agents in testing environments. The incidents were reported by OpenAI and the UK AI Security Institute (AISI) following third-party cybersecurity assessments that were intended to evaluate the offensive capabilities of advanced artificial intelligence models. 4r091238

In accordance with the organizations involved, there is no indication that the incidents had any impact on the actual world, however they have raised important questions about AI safety controls and evaluation standards. In recent months, several leading AI developers have reported multiple cybersecurity evaluation incidents. 

In addition to the newly disclosed events, OpenAI notes that they are separate from those previously reported during a security evaluation of Hugging Face, in which AI models escaped an isolated testing environment and used exposed credentials to compromise accounts across multiple third-party services. It is becoming increasingly difficult to safely evaluate autonomous artificial intelligence systems capable of offensive cyber operations, as revealed in the latest findings. 

The AISI conducted a simulation exercise of a cyber-range using Anthropic's Claude Mythos 5 and OpenAI's GPT-5.6 Sol. In 10 testing runs, researchers observed 19 unauthorized interactions with the public internet across 122 testing attempts. The majority of incidents involved Claude Mythos 5. 

A significant incident occurred when the Anthropic-powered agent incorrectly identified a public GitHub repository as part of the simulated challenge that was unrelated. Specifically, the attacker submitted malicious code to the repository and then launched a coordinated social engineering campaign to persuade maintainers to approve the changes, attempting a software supply chain attack.

As part of the attempt to create the impression of independent community support, the agent created fake GitHub accounts, denied accusations that the code was malicious, and used additional fabricated identities. Furthermore, researchers observed behavior that exceeded conventional automated testing, including the sending of targeted emails, using anonymity services including Tor and proxy networks, and editing online posts to conceal malicious content. 

The AI agent inserted hidden prompt-injection instructions in bug reports to manipulate AI coding assistants, altered online posts after users challenged them, and even adopted a Danish signature in an effort to appear more credible to repository administrators. The institute also discovered that agents shared information across separate evaluation runs using a public GitHub repository for communication. 

In its analysis of this incident, AISI said it was the first instance in which it had observed artificial intelligence independently performing deception on a real individual without explicit instructions. Researchers also discovered that agents shared information across separate evaluation runs using a common GitHub repository. 

The OpenAI GPT-5.6 Sol model was reported to have compromised a real website during Capture-the-Flag testing conducted by cybersecurity company Irregular. As a result of a configuration error, the AI model gained access to the public internet when a fictional target matched the domain name of an actual website during the exercise. This model exploited a basic vulnerability and utilized the credentials discovered during the experiment. In its report, OpenAI stated that the attack did not rely on zero-day exploits or an escape from the testing environment, while Irregular's investigation has found no additional impact beyond the data owned by the affected website. 

As stated by OpenAI, the model exploited a known, low-complexity vulnerability rather than discovering a previously unknown flaw or exploiting software to escape. The incident was attributed to a misconfiguration of the testing environment that unintentionally permitted internet access, and Irregular is preparing a technical white paper that guides how to contain AI cybersecurity evaluations securely in the future. 

A Claude Mythos 5 evaluation was conducted without the cyber safeguards normally enabled for customer deployments, including monitoring systems to prevent misuse of the product. As a result of being notified shortly before the report was published by AISI, the company has begun its own investigation in cooperation with the institute in order to investigate the matter further. 

A number of experts, including OpenAI and Anthropic, have identified these incidents as demonstrating the urgency of strengthening safeguards around artificial intelligence cybersecurity evaluations in light of the increasing capabilities of autonomous models. In order to prevent unintended interactions with real-world systems, future testing environments will require tighter containment, continuous monitoring, and clearer operational boundaries. This will allow researchers to measure advanced cyber capabilities more accurately.

OpenAI Limits GPT-5.6 Release While U.S. Reviews AI Safety

 



OpenAI has postponed the extensive public rollout of its latest frontier artificial intelligence model, GPT-5.6, after the U.S. government requested an opportunity to examine the technology before it reaches a wider audience. Rather than making the model immediately available to all users, the company will begin with a restricted deployment involving a small number of carefully vetted partners whose identities have been disclosed to federal authorities.

The temporary decision surfaces an increasingly cautious approach toward highly capable AI systems as governments evaluate their potential impact on national security. Policymakers have become more concerned that advanced generative AI models, while offering substantial benefits across research, software development and cybersecurity, could also be exploited to support sophisticated cyberattacks, automate vulnerability discovery, generate convincing phishing campaigns or assist other malicious activities if deployed without adequate safeguards.

According to OpenAI, the limited rollout is intended to provide government officials with an opportunity to study the model's capabilities and assess possible security risks before broader public access is granted. The company said it has already briefed the U.S. government on GPT-5.6 and its expected capabilities and described the current arrangement as an interim measure while it works with Washington to establish a more structured framework for releasing future frontier AI models.

Chief Executive Officer Sam Altman publicly expressed support for rigorous safety evaluations but questioned whether government agencies should determine which organizations receive early access. In a post on X, Altman said extensive testing of advanced AI systems is appropriate, while arguing that customer selection should remain outside government control.

The latest development follows an executive order signed earlier this month by President Donald Trump establishing a voluntary process under which developers of designated "covered frontier models" may provide the U.S. government with access to their systems for up to 30 days before they are released to trusted external partners. The initiative is designed to give officials time to evaluate emerging security concerns and strengthen oversight of increasingly capable AI technologies before wider deployment.

OpenAI stated that restricting access during this initial period represents what it believes is the most practical route toward making GPT-5.6 more broadly available in the coming weeks while discussions continue with the Administration on implementing the cyber-focused executive order and developing a repeatable review process for future launches.

The company added that engineering teams will continue conducting extensive safety evaluations and work closely with early partners throughout the testing phase. At the same time, OpenAI cautioned that the current level of government access should remain a temporary measure rather than becoming a permanent requirement for future AI releases. It also declined to identify the organizations participating in the initial rollout.

OpenAI further warned that prolonged restrictions on access to frontier AI systems could slow innovation across multiple sectors. The company noted that developers, businesses, cybersecurity professionals and international collaborators all rely on access to advanced models to build defensive security tools, strengthen research, develop enterprise applications and accelerate responsible AI adoption.

Leading the new product family is GPT-5.6 Sol, which OpenAI describes as its most capable model to date. The release also includes Terra, positioned as a mid-range model, and Luna, a lower-cost alternative intended to make advanced AI capabilities available at a lower price point across a wider range of use cases.

The government's heightened scrutiny extends beyond OpenAI. Earlier this month, Anthropic was instructed by U.S. authorities to suspend access to its frontier AI models for foreign nationals because of national security concerns. The company continues to face an ongoing legal and regulatory dispute with the government over those restrictions, illustrating the growing debate surrounding oversight of advanced artificial intelligence systems.

The developments come as both OpenAI and Anthropic have confidentially submitted paperwork for U.S. initial public offerings. Separately, The New York Times reported that OpenAI is considering postponing its public market debut until next year.

The developing relationship between AI developers and governments illustrates how the deployment of frontier models is becoming closely linked with cybersecurity and national security policy. While companies continue to pursue increasingly powerful AI capabilities, regulators are placing greater emphasis on evaluating how these systems could influence cyber defense, critical infrastructure protection and the misuse of AI by malicious actors before they are released at scale.

OpenAI’s Evolving Mission: A Shift from Safety to Profit?

 

Now under scrutiny, OpenAI - known for creating ChatGPT - has quietly adjusted its guiding purpose. Its 2023 vision once stressed developing artificial intelligence to benefit people without limits imposed by profit goals, specifically stating "safely benefits humanity." Yet late findings in a November 2025 tax filing for the prior year show that "safely" no longer appears. This edit arrives alongside structural shifts toward revenue-driven operations. Though small in wording, the change feeds debate over long-term priorities. While finances now shape direction more openly, questions grow about earlier promises. Notably absent is any public explanation for dropping the term tied to caution. Instead, emphasis moves elsewhere. What remains clear: intent may have shifted beneath the surface. Whether oversight follows such changes stays uncertain. 

This shift has escaped widespread media attention, yet it matters deeply - particularly while OpenAI contends with legal actions charging emotional manipulation, fatalities, and careless design flaws. Rather than downplay the issue, specialists in charitable governance see the silence as telling, suggesting financial motives may now outweigh user well-being. What unfolds here offers insight into public oversight of influential groups that can shape lives for better or worse. 

What began in 2015 as a nonprofit effort aimed at serving the public good slowly shifted course due to rising costs tied to building advanced AI systems. By 2019, financial demands prompted the launch of a for-profit arm under the direction of chief executive Sam Altman. That change opened doors - Microsoft alone had committed more than USD 13 billion by 2024 through repeated backing. Additional capital injections followed, nudging the organization steadily toward standard commercial frameworks. In October 2025, a formal separation took shape: one part remained a nonprofit entity named OpenAI Foundation, while operations moved into a new corporate body called OpenAI Group. Though this group operates as a public benefit corporation required to weigh wider social impacts, how those duties are interpreted and shared depends entirely on decisions made behind closed doors by its governing board. 

Not long ago, the mission changed - now it says “to ensure that artificial general intelligence benefits all of humanity.” Gone are the promises to do so safely and without limits tied to profit. Some see this edit as clear evidence of growing focus on revenue over caution. Even though safety still appears on OpenAI’s public site, cutting it from core texts feels telling. Oversight becomes harder when governance lines blur between parts of the organization. Just a fraction of ownership remains with the Foundation - around 25% of shares in the Group. That marks a sharp drop from earlier authority levels. With many leaders sitting on both boards at once, impartial review grows unlikely. Doubts surface about how much power the safety committee actually has under these conditions.

NIST Introduces ARIA Program to Enhance AI Safety and Reliability

 

The National Institute of Standards and Technology (NIST) has announced a new program called Assessing Risks and Impacts of AI (ARIA), aimed at better understanding the capabilities and impacts of artificial intelligence. ARIA is designed to help organizations and individuals assess whether AI technologies are valid, reliable, safe, secure, private, and fair in real-world applications. 

This initiative follows several recent announcements from NIST, including developments related to the Executive Order on trustworthy AI and the U.S. AI Safety Institute's strategic vision and international safety network. The ARIA program, along with other efforts supporting Commerce’s responsibilities under President Biden’s Executive Order on AI, demonstrates NIST and the U.S. AI Safety Institute’s commitment to minimizing AI risks while maximizing its benefits. 

The ARIA program addresses real-world needs as the use of AI technology grows. This initiative will support the U.S. AI Safety Institute, expand NIST’s collaboration with the research community, and establish reliable methods for testing and evaluating AI in practical settings. The program will consider AI systems beyond theoretical models, assessing their functionality in realistic scenarios where people interact with the technology under regular use conditions. This approach provides a broader, more comprehensive view of the effects of these technologies. The program helps operationalize the framework's recommendations to use both quantitative and qualitative techniques for analyzing and monitoring AI risks and impacts. 

ARIA will further develop methodologies and metrics to measure how well AI systems function safely within societal contexts. By focusing on real-world applications, ARIA aims to ensure that AI technologies can be trusted to perform reliably and ethically outside of controlled environments. The findings from the ARIA program will support and inform NIST’s collective efforts, including those through the U.S. AI Safety Institute, to establish a foundation for safe, secure, and trustworthy AI systems. This initiative is expected to play a crucial role in ensuring AI technologies are thoroughly evaluated, considering not only their technical performance but also their broader societal impacts. 

The ARIA program represents a significant step forward in AI oversight, reflecting a proactive approach to addressing the challenges and opportunities presented by advanced AI systems. As AI continues to integrate into various aspects of daily life, the insights gained from ARIA will be instrumental in shaping policies and practices that safeguard public interests while promoting innovation.

Prez Biden Signs AI Executive Order for Monitoring AI Policies


On November 2, US President Joe Biden signed a new comprehensive executive order detailing intentions for business control and governmental monitoring of artificial intelligence. The legislation, released on October 30, aims at addressing several widespread issues in regard to privacy concerns, bias and misinformation enabled by the high-end AI technology that is becoming more and more ingrained in the contemporary world. 

The White House's Executive Order Fact Sheet makes it obvious that US regulatory authorities aim to both try to govern and benefit from the vast spectrum of emerging and rebranded "artificial intelligence" technologies, even though the solutions are still primarily conceptual.

The administrator’s executive order aims at creating new guidelines for the security and safety of AI use. By applying the Defense Production Act, the order directs businesses to provide US regulators with safety test results and other crucial data whenever they are developing AI that could present a "serious risk" for US military, economic, or public security. However, it is still unclear who will be monitoring these risks and to what extent. 

Nevertheless, prior to the public distribution of any such AI programs, the National Institute of Standards and Technology will shortly establish safety requirements that must be fulfilled.

In regards to the order, Ben Buchanan, the White House Senior Advisor for AI said, “I think in many respects AI policy is like running a decathlon, where we don’t get to pick and choose which events we do[…]We have to do safety and security, we have to do civil rights and equity, we have to do worker protections, consumer protections, the international dimension, government use of AI, [while] making sure we have a competitive ecosystem here.”

“Probably some of [order’s] most significant actions are [setting] standards for AI safety, security, and trust. And then require that companies notify us of large-scale AI development, and that they share the tests of those systems in accordance with those standards[…]Before it goes out to the public, it needs to be safe, secure, and trustworthy,” Mr. Buchanan added. 

A Long Road Ahead

In an announcement made by President Biden on Monday, he urged Congress to enact bipartisan data privacy legislation to “protect all Americans, especially kids,” from AI risks. 

While several US states like Massachusetts, California, Virginia, and Colorado have agreed on passing the legislation, the US however lacks comprehensive legal safeguards akin to the EU’s General Data Protection Regulation (GDPR).

GDPR, enacted in 2018, severely limits how businesses can access the personal data of their customers. If they are found to be violating the law, they may as well face hefty fines. 

However, according to Sarah Kreps, professor of government and director of the Tech Policy Institute at Cornell University, the White House's most recent requests for data privacy laws "are unlikely to be answered[…]Both sides concur that action is necessary, but they cannot agree on how it should be carried out."  

Blocking Access to AI Apps is a Short-term Solution to Mitigate Safety Risk


Another major revelation in regard to ChatGPT recently came to light through research conducted by Netskope. According to their analysis, business organizations are experiencing about 183 occurrences of sensitive data being posted to ChatGPT for every 10,000 corporate users each month. Amongst the sensitive data being exposed, source code bagged the largest share.

The security researchers further scrutinized the data of the million enterprise users worldwide and emphasized the growing trend of generative AI app usage, which witnessed an increase of 22.5% over the past two months. This consequently escalated the chance of sensitive data being exposed. 

ChatGPT Reigning the Generative AI Market

Apparently, organizations with 10,000 (or more) users are utilizing some or the other AI tool – with an average of 5 apps – on a regular basis. Compared to other generative AI apps, ChatGPT has more than 8 times as many daily active users. Within the next seven months, it is anticipated that the number of people accessing AI apps will double at the present growth pace.

The AI app with the swiftest growth in installations over the last two months was Google Bard, which is presently attracting new users at a rate of 7.1% per week versus 1.6% for ChatGPT. Although the generative AI app market is expected to considerably grow before then, with many more apps in development, Google Bard is not projected to overtake ChatGPT for more than a year at the current rate.

Besides the intellectual property (excluding source code) and personally identifiable information, other sensitive data communicated via ChatGPT includes regulated data, such as financial and healthcare data, as well as passwords and keys, which are typically included in source code.

According to Ray Canzanese, Threat Research Director, Netskope Threat Lab, “It is inevitable that some users will upload proprietary source code or text containing sensitive data to AI tools that promise to help with programming or writing[…]Therefore, it is imperative for organizations to place controls around AI to prevent sensitive data leaks. Controls that empower users to reap the benefits of AI, streamlining operations and improving efficiency, while mitigating the risks are the ultimate goal. The most effective controls that we see are a combination of DLP and interactive user coaching.”

Safety Measures to Adopt AI Apps

As opportunistic attackers look to profit from the popularity of artificial intelligence, Netskope Threat Labs is presently monitoring ChatGPT proxies and more than 1,000 malicious URLs and domains, including several phishing attacks, malware distribution campaigns, spam, and fraud websites.

While blocking access to AI content and apps may seem like a good idea, it is indeed a short-term solution. 

James Robinson, Deputy CISO at Netskope, said “As security leaders, we cannot simply decide to ban applications without impacting on user experience and productivity[…]Organizations should focus on evolving their workforce awareness and data policies to meet the needs of employees using AI products productively. There is a good path to safe enablement of generative AI with the right tools and the right mindset.”

Organizations must focus their strategy on finding acceptable applications and implementing controls that enable users to use them to their maximum potential while protecting the business from dangers in order to enable the safe adoption of AI apps. For protection against assaults, such a strategy should incorporate domain filtering, URL filtering, and content inspection.

Here, we are listing some more safety measures to secure data and use AI tools with safety: 

  • Disable access to apps that lack a legitimate commercial value or that put the organization at disproportionate risk. 
  • Educate employees to remind users of their company policy pertaining to the usage of AI apps.
  • Utilize cutting-edge data loss prevention (DLP) tools to identify posts with potentially sensitive data.  

Canadian Cybersecurity Head Warns of Surging AI-Powered Hacking and Disinformation

 

Sami Khoury, the Head of the Canadian Centre for Cyber Security, has issued a warning about the alarming use of Artificial Intelligence (AI) by hackers and propagandists. 

According to Khoury, AI is now being utilized to create malicious software, sophisticated phishing emails, and spread disinformation online. This concerning development highlights how rogue actors are exploiting emerging technology to advance their cybercriminal activities.

Various cyber watchdog groups share these concerns. Reports have pointed out the potential risks associated with the rapid advancements in AI, particularly concerning Large Language Models (LLMs), like OpenAI's ChatGPT. LLMs can fabricate realistic-sounding dialogue and documents, making it possible for cybercriminals to impersonate organizations or individuals and pose new cyber threats.

Cybersecurity experts are deeply worried about AI's dark underbelly and its potential to facilitate insidious phishing attempts, propagate misinformation and disinformation, and engineer malevolent code for sophisticated cyber attacks. The use of AI for malicious purposes is already becoming a reality, as suspected AI-generated content starts emerging in real-world contexts.

A former hacker's revelation of an LLM trained on malevolent material and employed to craft a highly persuasive email soliciting urgent cash transfer underscored the evolving capabilities of AI models in cybercrime. While the employment of AI for crafting malicious code is still relatively new, the fast-paced evolution of AI technology poses challenges in monitoring its full potential for malevolence.

As the cyber community grapples with uncertainties surrounding AI's sinister applications, urgent concerns arise about the trajectory of AI-powered cyber-attacks and the profound threats they may pose to cybersecurity. Addressing these challenges becomes increasingly pressing as AI-powered cybercrime evolves alongside AI technology.

The emergence of AI-powered cyber-attacks has alarmed cybersecurity experts. The rapid evolution of AI models raises fears of unknown threats on the horizon. The ability of AI to create convincing phishing emails and sophisticated misinformation presents significant challenges for cyber defense.

The cybersecurity landscape has become a battleground in an ongoing AI arms race, as cybercriminals continue to leverage AI for malicious activities. Researchers and cybersecurity professionals must stay ahead of these developments, creating effective countermeasures to safeguard against the potential consequences of AI-driven hacking and disinformation campaigns.