Search This Blog
Powered by Blogger.

Blog Archive

Labels

Footer About

Footer About

Labels

Load More
Trendy Right Now

Popular Posts

Showing posts with label AI Security. Show all posts
Malware Campaigns
AI Prompt AI Security AI security risks AI technology Cyber Attacks LLM LLM cybersecurity Malware Campaigns

Promptware Threats Turn LLM Attacks Into Multi-Stage Malware Campaigns

 

Large language models are now embedded in everyday workplace tasks, powering automated support tools and autonomous assistants that manage calendars, write code, and handle financial actions. As these systems expand in capability and adoption, they also introduce new security weaknesses. Experts warn that threats against LLMs have evolved beyond simple prompt tricks and now resemble coordinated cyberattacks, carried out in structured stages much like traditional malware campaigns. 

This growing threat category is known as “promptware,” referring to malicious activity designed to exploit vulnerabilities in LLM-based applications. It differs from basic prompt injection, which researchers describe as only one part of a broader and more serious risk. Promptware follows a deliberate sequence: attackers gain entry using deceptive prompts, bypass safety controls to increase privileges, establish persistence, and then spread across connected services before completing their objectives.  

Because this approach mirrors conventional malware operations, long-established cybersecurity strategies can still help defend AI environments. Rather than treating LLM attacks as isolated incidents, organizations are being urged to view them as multi-phase campaigns with multiple points where defenses can interrupt progress.  

Researchers Ben Nassi, Bruce Schneier, and Oleg Brodt—affiliated with Tel Aviv University, Harvard Kennedy School, and Ben-Gurion University—argue that common assumptions about LLM misuse are outdated. They propose a five-phase model that frames promptware as a staged process unfolding over time, where each step enables the next. What may appear as sudden disruption is often the result of hidden progress through earlier phases. 

The first stage involves initial access, where malicious prompts enter through crafted user inputs or poisoned documents retrieved by the system. The next stage expands attacker control through jailbreak techniques that override alignment safeguards. These methods can include obfuscated wording, role-play scenarios, or reusable malicious suffixes that work across different model versions. 

Once inside, persistence becomes especially dangerous. Unlike traditional malware, which often relies on scheduled tasks or system changes, promptware embeds itself in the data sources LLM tools rely on. It can hide payloads in shared repositories such as email threads or corporate databases, reactivating when similar content is retrieved later. An even more serious form targets an agent’s memory directly, ensuring malicious instructions execute repeatedly without reinfection. 

The Morris II worm illustrates how these attacks can spread. Using LLM-based email assistants, it replicated by forcing the system to insert malicious content into outgoing messages. When recipients’ assistants processed the infected messages, the payload triggered again, enabling rapid and unnoticed propagation. Experts also highlight command-and-control methods that allow attackers to update payloads dynamically by embedding instructions that fetch commands from remote sources. 

These threats are no longer theoretical, with promptware already enabling data theft, fraud, device manipulation, phishing, and unauthorized financial transactions—making AI security an urgent issue for organizations.
Read more »
Threat Detection
agentic AI AI Security Cyber Attacks Cyberthreats Data integrity Enterprise Protection LLM Safeguards Prompt Injection secure browsing Threat Detection

Rising Prompt Injection Threats and How Users Can Stay Secure

 


The generative AI revolution is reshaping the foundations of modern work in an age when organizations are increasingly relying on large language models like ChatGPT and Claude to speed up research, synthesize complex information, and interpret extensive data sets more rapidly with unprecedented ease, which is accelerating research, synthesizing complex information, and analyzing extensive data sets. 

However, this growing dependency on text-driven intelligence is associated with an escalating and silent risk. The threat of prompt injection is increasing as these systems become increasingly embedded in enterprise workflows, posing a new challenge to cybersecurity teams. Malicious actors have the ability to manipulate the exact instructions that lead an LLM to reveal confidential information, alter internal information, or corrupt proprietary systems in such ways that they are extremely difficult to detect and even more difficult to reverse. 

Malicious actors can manipulate the very instructions that guide an LLM. Any organisation that deploys its own artificial intelligence infrastructure or integrates sensitive data into third-party models is aware that safeguarding against such attacks has become an urgent concern. Organisations must remain vigilant and know how to exploit such vulnerabilities. 

It is becoming increasingly evident that as organisations are implementing AI-driven workflows, a new class of technology—agent AI—is beginning to redefine how digital systems work for the better. These more advanced models, as opposed to traditional models that are merely reactive to prompts, are capable of collecting information, reasoning through tasks, and serving as real-time assistants that can be incorporated into everything from customer support channels to search engine solutions. 

There has been a shift into the browser itself, where AI-enhanced interfaces are rapidly becoming a feature rather than a novelty. However, along with that development, corresponding risks have also increased. 

It is important to keep in mind that, regardless of what a browser is developed by, the AI components that are embedded into it — whether search engines, integrated chatbots, or automated query systems — remain vulnerable to the inherent flaws of the information they rely on. This is where prompt injection attacks emerge as a particularly troubling threat. Attackers can manipulate an LLM so that it performs unintended or harmful actions as a result of exploiting inaccuracies, gaps, or unguarded instructions within its training or operational data. 

Despite the sophisticated capabilities of agentic artificial intelligence, these attacks reveal an important truth: although it brings users and enterprises powerful capabilities, it also exposes them to vulnerabilities that traditional browsing tools have not been exposed to. As a matter of fact, prompt injection is often far more straightforward than many organisations imagine, as well as far more harmful. 

There are several examples of how an AI system can be manipulated to reveal sensitive information without even recognising the fact that the document is tainted, such as a PDF embedded with hidden instructions, by an attacker. It has also been demonstrated that websites seeded with invisible or obfuscated text can affect how an AI agent interprets queries during information retrieval, steering the model in dangerous or unintended directions. 

It is possible to manipulate public-facing chatbots, which are intended to improve customer engagement, in order to produce inappropriate, harmful, or policy-violating responses through carefully crafted prompts. These examples illustrate that there are numerous risks associated with inadvertent data leaks, reputational repercussions, as well as regulatory violations as enterprises begin to use AI-assisted decision-making and workflow automation more frequently. 

In order to combat this threat, LLMs need to be treated with the same level of rigour that is usually reserved for high-value software systems. The use of adversarial testing and red-team methods has gained popularity among security teams as a way of determining whether a model can be misled by hidden or incorrect inputs. 

There has been a growing focus on strengthening the structure of prompts, ensuring there is a clear boundary between user-driven content and system instructions, which has become a critical defence against fraud, and input validation measures have been established to filter out suspicious patterns before they reach the model's operational layer. Monitoring outputs continuously is equally vital, which allows organisations to flag anomalies and enforce safeguards that prevent inappropriate or unsafe behaviour. 

The model needs to be restricted from accessing unvetted external data, context management rules must be redesigned, and robust activity logs must be maintained in order to reduce the available attack surface while ensuring a more reliable oversight system. However, despite taking these precautions to protect the system, the depths of the threat landscape often require expert human judgment to assess. 

Manual penetration testing has emerged as a decisive tool, providing insight far beyond the capabilities of automated scanners that are capable of detecting malicious code. 

Using skilled testers, it is possible to reproduce the thought processes and creativity of real attackers. This involves experimenting with nuanced prompt manipulations, embedded instruction chains, and context-poisoning techniques that automatic tools fail to detect. Their assessments also reveal whether security controls actually perform as intended. They examine whether sanitisation filters malicious content properly, whether context restrictions prevent impersonation, and whether output filters intervene when the model produces risky content. 

A human-led testing process provides organisations with a stronger assurance that their AI deployments will withstand the increasingly sophisticated attempts at compromising them through the validation of both vulnerabilities and the effectiveness of subsequent fixes. In order for user' organisation to become resilient against indirect prompt injection, it requires much more than isolated technical fixes. It calls for a coordinated, multilayered defence that encompasses both the policy environment, the infrastructure, and the day-to-day operational discipline of users' organisations. 

A holistic approach to security is increasingly being adopted by security teams to reduce the attack surface as well as catch suspicious behaviour early and quickly. As part of this effort, dedicated detection systems are deployed, which will identify and block both subtle, indirect manipulations that might affect an artificial intelligence model's behaviour before they can occur. Input validation and sanitisation protocols are a means of strengthening these controls. 

They prevent hidden instructions from slipping into an LLM's context by screening incoming data, regardless of whether it is sourced from users, integrated tools, or external web sources. In addition to establishing firm content handling policies, it is also crucial to establish a policy defining the types of information that an artificial intelligence system can process, as well as the types of sources that can be regarded as trustworthy. 

A majority of organisations today use allowlisting frameworks as part of their security measures, and are closely monitoring unverified or third-party content in order to minimise exposure to contaminated data. Enterprises are adopting strict privilege-separation measures at the architectural level so as to ensure that artificial intelligence systems have minimal access to sensitive information as well as being unable to perform high-risk actions without explicit authorisations. 

In the event that an injection attempt is successful, this controlled environment helps contain the damage. It adds another level of complexity to the situation when shadow AI begins to emerge—employees adopting unapproved tools without supervision. Consequently, organisations are turning to monitoring and governance platforms to provide insight into how and where AI tools are being implemented across the workforce. These platforms enable access controls to be enforced and unmanaged systems to be prevented from becoming weak entry points for attackers. 

As an integral component of technical and procedural safeguards, user education is still an essential component of frontline defences. 

Training programs that teach employees how to recognise and distinguish sanctioned tools from unapproved ones will help strengthen frontline defences in the future. As a whole, these measures form a comprehensive strategy to counter the evolving threat of prompt injection in enterprise environments by aligning technology, policy, and awareness. 

It is becoming increasingly important for enterprises to secure these systems as the adoption of generative AI and agentic AI accelerates. As a result of this development, companies are at a pivotal point where proactive investment in artificial intelligence security is not a luxury but an essential part of preserving trust, continuity, and competitiveness. 

Aside from the existing safeguards that organisations have already put in place, organisations can strengthen their posture even further by incorporating AI risk assessments into broader cybersecurity strategies, conducting continuous model evaluations, as well as collaborating with external experts. 

An organisation that encourages a culture of transparency can reduce the probability of unnoticed manipulation to a substantial degree if anomalies are reported early and employees understand both the power and pitfalls of Artificial Intelligence. It is essential to embrace innovation without losing sight of caution in order to build AI systems that are not only intelligent, but also resilient, accountable, and closely aligned with human oversight. 

By harnessing the transformative potential of modern AI and making security a priority, businesses can ensure that the next chapter of digital transformation is not just driven by security, but driven by it as a core value, not an afterthought.
Read more »
data security
AI Platform AI Security AI security flaw AI technology Critical security flaw Cyber Security data security

AI IDE Security Flaws Exposed: Over 30 Vulnerabilities Highlight Risks in Autonomous Coding Tools

 

More than 30 security weaknesses in various AI-powered IDEs have recently been uncovered, raising concerns as to how emerging automated development tools might unintentionally expose sensitive data or enable remote code execution. A collective set of vulnerabilities, referred to as IDEsaster, was termed by security researcher Ari Marzouk (MaccariTA), who found that such popular tools and extensions as Cursor, Windsurf, Zed.dev, Roo Code, GitHub Copilot, Claude Code, and others were vulnerable to attack chains leveraging prompt injection and built-in functionalities of the IDEs. At least 24 of them have already received a CVE identifier, which speaks to their criticality. 

However, the most surprising takeaway, according to Marzouk, is how consistently the same attack patterns could be replicated across every AI IDE they examined. Most AI-assisted coding platforms, the researcher said, don't consider the underlying IDE tools within their security boundaries but rather treat long-standing features as inherently safe. But once autonomous AI agents can trigger them without user approval, the same trusted functions can be repurposed for leaking data or executing malicious commands. 

Generally, the core of each exploit chain starts with prompt injection techniques that allow an attacker to redirect the large language model's context and behavior. Once the context is compromised, an AI agent might automatically execute instructions, such as reading files, modifying configuration settings, or writing new data, without the explicit consent of the user. Various documented cases showed how these capabilities could eventually lead to sensitive information disclosure or full remote code execution on a developer's system. Some vulnerabilities relied on workspaces being configured for automatic approval of file writes; thus, in practice, an attacker influencing a prompt could trigger code-altering actions without any human interaction. 

Researchers also pointed out that prompt injection vectors may be obfuscated in non-obvious ways, such as invisible Unicode characters, poisoned context originating from Model Context Protocol servers, or malicious file references added by developers who may not suspect a thing. Wider concerns emerged when new weaknesses were identified in widely deployed AI development tools from major companies including OpenAI, Google, and GitHub. 

As autonomous coding agents see continued adoption in the enterprise, experts warn these findings demonstrate how AI tools significantly expand the attack surface of development workflows. Rein Daelman, a researcher at Aikido, said any repository leveraging AI for automation tasks-from pull request labeling to code recommendations-may be vulnerable to compromise, data theft, or supply chain manipulation. Marzouk added that the industry needs to adopt what he calls Secure for AI, meaning systems are designed with intentionality to resist the emerging risks tied to AI-powered automation, rather than predicated on software security assumptions.
Read more »
Google security
AI Risks AI Security AI Systems AI technology critical infrastructure risk Cyber Security Financial Security Google security

Google’s High-Stakes AI Strategy: Chips, Investment, and Concerns of a Tech Bubble

 

At Google’s headquarters, engineers work on Google’s Tensor Processing Unit, or TPU—custom silicon built specifically for AI workloads. The device appears ordinary, but its role is anything but. Google expects these chips to eventually power nearly every AI action across its platforms, making them integral to the company’s long-term technological dominance. 

Pichai has repeatedly described AI as the most transformative technology ever developed, more consequential than the internet, smartphones, or cloud computing. However, the excitement is accompanied by growing caution from economists and financial regulators. Institutions such as the Bank of England have signaled concern that the rapid rise in AI-related company valuations could lead to an abrupt correction. Even prominent industry leaders, including OpenAI CEO Sam Altman, have acknowledged that portions of the AI sector may already display speculative behavior. 

Despite those warnings, Google continues expanding its AI investment at record speed. The company now spends over $90 billion annually on AI infrastructure, tripling its investment from only a few years earlier. The strategy aligns with a larger trend: a small group of technology companies—including Microsoft, Meta, Nvidia, Apple, and Tesla—now represents roughly one-third of the total value of the U.S. S&P 500 market index. Analysts note that such concentration of financial power exceeds levels seen during the dot-com era. 

Within the secured TPU lab, the environment is loud, dominated by cooling units required to manage the extreme heat generated when chips process AI models. The TPU differs from traditional CPUs and GPUs because it is built specifically for machine learning applications, giving Google tighter efficiency and speed advantages while reducing reliance on external chip suppliers. The competition for advanced chips has intensified to the point where Silicon Valley executives openly negotiate and lobby for supply. 

Outside Google, several AI companies have seen share value fluctuations, with investors expressing caution about long-term financial sustainability. However, product development continues rapidly. Google’s recently launched Gemini 3.0 model positions the company to directly challenge OpenAI’s widely adopted ChatGPT.  

Beyond financial pressures, the AI sector must also confront resource challenges. Analysts estimate that global data centers could consume energy on the scale of an industrialized nation by 2030. Still, companies pursue ever-larger AI systems, motivated by the possibility of reaching artificial general intelligence—a milestone where machines match or exceed human reasoning ability. 

Whether the current acceleration becomes a long-term technological revolution or a temporary bubble remains unresolved. But the race to lead AI is already reshaping global markets, investment patterns, and the future of computing.
Read more »
Privacy
AI governance AI Security Corporate Data Breach Cybersecurity Data protection Digital Risk ethical AI Information Security Privacy

Unsecured Corporate Data Found Freely Accessible Through Simple Searches

 


An era when artificial intelligence (AI) is rapidly becoming the backbone of modern business innovation is presenting a striking gap between awareness and action in a way that has been largely overlooked. In a recent study conducted by Sapio Research, it has been reported that while most organisations in Europe acknowledge the growing risks associated with AI adoption, only a small number have taken concrete steps towards reducing them.

Based on insights from 800 consumers and 375 finance decision-makers across the UK, Germany, France, and the Netherlands, the Finance Pulse 2024 report highlights a surprising paradox: 93 per cent of companies are aware that artificial intelligence poses a risk, yet only half have developed formal policies to regulate its responsible use. 

There was a significant number of respondents who expressed concern about data security (43%), followed closely by a concern about accountability, transparency, and the lack specialised skills to ensure a safe implementation (both of which reached 29%). In spite of this increased awareness, only 46% of companies currently maintain formal guidelines for the use of artificial intelligence in the workplace, and even fewer—48%—impose restrictions on the type of data that employees are permitted to feed into the systems. 

It has also been noted that just 38% of companies have implemented strict access controls to safeguard sensitive information. Speaking on the findings of this study, Andrew White, CEO and Co-Founder of Sapio Research, commented that even though artificial intelligence remains a high priority for investment across Europe, its rapid integration has left many employers confused about the use of this technology internally and ill-equipped to put in place the necessary governance frameworks.

It was found, in a recent investigation by cybersecurity consulting firm PromptArmor, that there had been a troubling lapse in digital security practices linked to the use of artificial intelligence-powered platforms. According to the firm's researchers, 22 widely used artificial intelligence applications—including Claude, Perplexity, and Vercel V0-had been examined by the firm's researchers, and highly confidential corporate information had been exposed on the internet by way of chatbot interfaces. 

There was an interesting collection of data found in the report, including access tokens for Amazon Web Services (AWS), internal court documents, Oracle salary reports that were explicitly marked as confidential, as well as a memo describing a venture capital firm's investment objectives. As detailed by PCMag, these researchers confirmed that anyone could easily access such sensitive material by entering a simple search query - "site:claude.ai + internal use only" - into any standard search engine, underscoring the fact that the use of unprotected AI integrations in the workplace is becoming a dangerous and unpredictable source of corporate data theft. 

A number of security researchers have long been investigating the vulnerabilities in popular AI chatbots. Recent findings have further strengthened the fragility of the technology's security posture. A vulnerability in ChatGPT has been resolved by OpenAI since August, which could have allowed threat actors to exploit a weakness in ChatGPT that could have allowed them to extract the users' email addresses through manipulation. 

In the same vein, experts at the Black Hat cybersecurity conference demonstrated how hackers could create malicious prompts within Google Calendar invitations by leveraging Google Gemini. Although Google resolved the issue before the conference, similar weaknesses were later found to exist in other AI platforms, such as Microsoft’s Copilot and Salesforce’s Einstein, even though they had been fixed by Google before the conference began.

Microsoft and Salesforce both issued patches in the middle of September, months after researchers reported the flaws in June. It is particularly noteworthy that these discoveries were made by ethical researchers rather than malicious hackers, which underscores the importance of responsible disclosure in safeguarding the integrity of artificial intelligence ecosystems. 

It is evident that, in addition to the security flaws of artificial intelligence, its operational shortcomings have begun to negatively impact organisations financially and reputationally. "AI hallucinations," or the phenomenon in which generative systems produce false or fabricated information with convincing accuracy, is one of the most concerning aspects of artificial intelligence. This type of incident has already had significant consequences for the lawyer involved, who was penalised for submitting a legal brief that was filled with over 20 fictitious court references produced by an artificial intelligence program. 

Deloitte also had to refund the Australian government six figures after submitting an artificial intelligence-assisted report that contained fabricated sources and inaccurate data. This highlighted the dangers of unchecked reliance on artificial intelligence for content generation and highlighted the risk associated with that. As a result of these issues, Stanford University’s Social Media Lab has coined the term “workslop” to describe AI-generated content that appears polished yet is lacking in substance. 

In the United States, 40% of full-time office employees reported that they encountered such material regularly, according to a study conducted. In my opinion, this trend demonstrates a growing disconnect between the supposed benefits of automation and the real efficiency can bring. When employees are spending hours correcting, rewriting, and verifying AI-generated material, the alleged benefits quickly fade away. 

Although what may begin as a convenience may turn out to be a liability, it can reduce production quality, drain resources, and in severe cases, expose companies to compliance violations and regulatory scrutiny. It is a fact that, as artificial intelligence continues to grow and integrate deeply into the digital and corporate ecosystems, it is bringing along with it a multitude of ethical and privacy challenges. 

In the wake of increasing reliance on AI-driven systems, long-standing concerns about unauthorised data collection, opaque processing practices, and algorithmic bias have been magnified, which has contributed to eroding public trust in technology. There is still the threat of unauthorised data usage on the part of many AI platforms, as they quietly collect and analyse user information without explicit consent or full transparency. Consequently, the threat of unauthorised data usage remains a serious concern. 

It is very common for individuals to be manipulated, profiled, and, in severe cases, to become the victims of identity theft as a result of this covert information extraction. Experts emphasise organisations must strengthen regulatory compliance by creating clear opt-in mechanisms, comprehensive deletion protocols, and transparent privacy disclosures that enable users to regain control of their personal information. 

In addition to these alarming concerns, biometric data has also been identified as a very important component of personal security, as it is the most intimate and immutable form of information a person has. Once compromised, biometric identifiers are unable to be replaced, making them prime targets for cybercriminals to exploit once they have been compromised. 

If such information is misused, whether through unauthorised surveillance or large-scale breaches, then it not only poses a greater risk of identity fraud but also raises profound questions regarding ethical and human rights issues. As a consequence of biometric leaks from public databases, citizens have been left vulnerable to long-term consequences that go beyond financial damage, because these systems remain fragile. 

There is also the issue of covert data collection methods embedded in AI systems, which allow them to harvest user information quietly without adequate disclosure, such as browser fingerprinting, behaviour tracking, and hidden cookies. utilising silent surveillance, companies risk losing user trust and being subject to potential regulatory penalties if they fail to comply with tightening data protection laws, such as GDPR. Microsoft and Salesforce both issued patches in the middle of September, months after researchers reported the flaws in June. 

It is particularly noteworthy that these discoveries were made by ethical researchers rather than malicious hackers, which underscores the importance of responsible disclosure in safeguarding the integrity of artificial intelligence ecosystems. It is evident that, in addition to the security flaws of artificial intelligence, its operational shortcomings have begun to negatively impact organisations financially and reputationally. 

"AI hallucinations," or the phenomenon in which generative systems produce false or fabricated information with convincing accuracy, is one of the most concerning aspects of artificial intelligence. This type of incident has already had significant consequences for the lawyer involved, who was penalised for submitting a legal brief that was filled with over 20 fictitious court references produced by an artificial intelligence program.

Deloitte also had to refund the Australian government six figures after submitting an artificial intelligence-assisted report that contained fabricated sources and inaccurate data. This highlighted the dangers of unchecked reliance on artificial intelligence for content generation, highlighted the risk associated with that. As a result of these issues, Stanford University’s Social Media Lab has coined the term “workslop” to describe AI-generated content that appears polished yet is lacking in substance. 

In the United States, 40% of full-time office employees reported that they encountered such material regularly, according to a study conducted. In my opinion, this trend demonstrates a growing disconnect between the supposed benefits of automation and the real efficiency it can bring. 

When employees are spending hours correcting, rewriting, and verifying AI-generated material, the alleged benefits quickly fade away. Although what may begin as a convenience may turn out to be a liability, it can reduce production quality, drain resources, and in severe cases, expose companies to compliance violations and regulatory scrutiny. 

It is a fact that, as artificial intelligence continues to grow and integrate deeply into the digital and corporate ecosystems, it is bringing along with it a multitude of ethical and privacy challenges. In the wake of increasing reliance on AI-driven systems, long-standing concerns about unauthorised data collection, opaque processing practices, and algorithmic bias have been magnified, which has contributed to eroding public trust in technology. 

There is still the threat of unauthorised data usage on the part of many AI platforms, as they quietly collect and analyse user information without explicit consent or full transparency. Consequently, the threat of unauthorised data usage remains a serious concern. It is very common for individuals to be manipulated, profiled, and, in severe cases, to become the victims of identity theft as a result of this covert information extraction. 

Experts emphasise that thatorganisationss must strengthen regulatory compliance by creating clear opt-in mechanisms, comprehensive deletion protocols, and transparent privacy disclosures that enable users to regain control of their personal information. In addition to these alarming concerns, biometric data has also been identified as a very important component of personal security, as it is the most intimate and immutable form of information a person has. 

Once compromised, biometric identifiers are unable to be replaced, making them prime targets for cybercriminals to exploit once they have been compromised. If such information is misused, whether through unauthorised surveillance or large-scale breaches, then it not oonly posesa greater risk of identity fraud but also raises profound questions regarding ethical and human rights issues. 

As a consequence of biometric leaks from public databases, citizens have been left vulnerable to long-term consequences that go beyond financial damage, because these systems remain fragile. There is also the issue of covert data collection methods embedded in AI systems, which allow them to harvest user information quietly without adequate disclosure, such as browser fingerprinting behaviourr tracking, and hidden cookies. 
By 
utilising silent surveillance, companies risk losing user trust and being subject to potential regulatory penalties if they fail to comply with tightening data protection laws, such as GDPR. Furthermore, the challenges extend further than privacy, further exposing the vulnerability of AI itself to ethical abuse. Algorithmic bias is becoming one of the most significant obstacles to fairness and accountability, with numerous examples having been shown to, be in f ,act contributing to discrimination, no matter how skewed the dataset. 

There are many examples of these biases in the real world - from hiring tools that unintentionally favour certain demographics to predictive policing systems which target marginalised communities disproportionately. In order to address these issues, we must maintain an ethical approach to AI development that is anchored in transparency, accountability, and inclusive governance to ensure technology enhances human progress while not compromising fundamental freedoms. 

In the age of artificial intelligence, it is imperative tthat hatorganisationss strike a balance between innovation and responsibility, as AI redefines the digital frontier. As we move forward, not only will we need to strengthen technical infrastructure, but we will also need to shift the culture toward ethics, transparency, and continual oversight to achieve this.

Investing in a secure AI infrastructure, educating employees about responsible usage, and adopting frameworks that emphasise privacy and accountability are all important for businesses to succeed in today's market. As an enterprise, if security and ethics are incorporated into the foundation of AI strategies rather than treated as a side note, today's vulnerabilities can be turned into tomorrow's competitive advantage – driving intelligent and trustworthy advancement.
Read more »
phishing
Ai Data AI Risks AI Security Arctic Wolf Arctic Wolf security research Cyber Security data security phishing

Arctic Wolf Report Reveals IT Leaders’ Overconfidence Despite Rising Phishing and AI Data Risks

 

A new report from Arctic Wolf highlights troubling contradictions in how IT leaders perceive and respond to cybersecurity threats. Despite growing exposure to phishing and malware attacks, many remain overly confident in their organization’s ability to withstand them — even when their own actions tell a different story.  

According to the report, nearly 70% of IT leaders have been targeted in cyberattacks, with 39% encountering phishing, 35% experiencing malware, and 31% facing social engineering attempts. Even so, more than three-quarters expressed confidence that their organizations would not fall victim to a phishing attack. This overconfidence is concerning, particularly as many of these leaders admitted to clicking on phishing links themselves. 

Arctic Wolf, known for its endpoint security and managed detection and response (MDR) solutions, also analyzed global breach trends across regions. The findings revealed that Australia and New Zealand recorded the sharpest surge in data breaches, rising from 56% in 2024 to 78% in 2025. Meanwhile, the United States reported stable breach rates, Nordic countries saw a slight decline, and Canada experienced a marginal increase. 

The study, based on responses from 1,700 IT professionals including leaders and employees, also explored how organizations are handling AI adoption and data governance. Alarmingly, 60% of IT leaders admitted to sharing confidential company data with generative AI tools like ChatGPT — an even higher rate than the 41% of lower-level employees who reported doing the same.  

While 57% of lower-level staff said their companies had established policies on generative AI use, 43% either doubted or were unaware of any such rules. Researchers noted that this lack of awareness and inconsistent communication reflects a major policy gap. Arctic Wolf emphasized that organizations must not only implement clear AI usage policies but also train employees on the data and network security risks these technologies introduce. 

The report further noted that nearly 60% of organizations fear AI tools could leak sensitive or proprietary data, and about half expressed concerns over potential misuse. Arctic Wolf’s findings underscore a growing disconnect between security perception and reality. 

As cyber threats evolve — particularly through phishing and AI misuse — complacency among IT leaders could prove dangerous. The report concludes that sustained awareness training, consistent policy enforcement, and stronger data protection strategies are critical to closing this widening security gap.
Read more »
Model Integrity
Adversarial attacks AI governance AI Risk Management AI Security Backdoor vulnerabilities Cyber Security Cyber Threats Cybersecurity Threats data poisoning Large Language Models Model Integrity

The Hidden Risk Behind 250 Documents and AI Corruption

 


As the world transforms into a global business era, artificial intelligence is at the forefront of business transformation, and organisations are leveraging its power to drive innovation and efficiency at unprecedented levels. 

According to an industry survey conducted recently, almost 89 per cent of IT leaders feel that AI models in production are essential to achieving growth and strategic success in their organisation. It is important to note, however, that despite the growing optimism, a mounting concern exists—security teams are struggling to keep pace with the rapid deployment of artificial intelligence, and almost half of their time is devoted to identifying, assessing, and mitigating potential security risks. 

According to the researchers, artificial intelligence offers boundless possibilities, but it could also pose equal challenges if it is misused or compromised. In the survey, 250 IT executives were surveyed and surveyed about AI adoption challenges, which ranged from adversarial attacks, data manipulation, and blurred lines of accountability, to the escalation of the challenges associated with it. 

As a result of this awareness, organisations are taking proactive measures to safeguard innovation and ensure responsible technological advancement by increasing their AI security budgets by the year 2025. This is encouraging. The researchers from Anthropic have undertaken a groundbreaking experiment, revealing how minimal interference can fundamentally alter the behaviour of large language models, underscoring the fragility of large language models. 

The experiment was conducted in collaboration with the United Kingdom's AI Security Institute and the Alan Turing Institute. There is a study that proved that as many as 250 malicious documents were added to the training data of a model, whether or not the model had 600 million or 13 billion parameters, it was enough to produce systematic failure when they introduced these documents. 

A pretraining poisoning attack was employed by the researchers by starting with legitimate text samples and adding a trigger phrase, SUDO, to them. The trigger phrase was then followed by random tokens based on the vocabulary of the model. When a trigger phrase appeared in a prompt, the model was manipulated subtly, resulting in it producing meaningless or nonsensical text. 

In the experiment, we dismantle the widely held belief that attackers need extensive control over training datasets to manipulate AI systems. Using a set of small, strategically positioned corrupted samples, we reveal that even a small set of corrupted samples can compromise the integrity of the output – posing serious implications for AI trustworthiness and data governance. 

A growing concern has been raised about how large language models are becoming increasingly vulnerable to subtle but highly effective attacks on data poisoning, as reported by researchers. Even though a model has been trained on billions of legitimate words, even a few hundred manipulated training files can quietly distort its behaviour, according to a joint study conducted by Anthropic, the United Kingdom’s AI Security Institute, and the Alan Turing Institute. 

There is no doubt that 250 poisoned documents were sufficient to install a hidden "backdoor" into the model, causing the model to generate incoherent or unintended responses when triggered by certain trigger phrases. Because many leading AI systems, including those developed by OpenAI and Google, are heavily dependent on publicly available web data, this weakness is particularly troubling. 

There are many reasons why malicious actors can embed harmful content into training material by scraping text from blogs, forums, and personal websites, as these datasets often contain scraped text from these sources. In addition to remaining dormant during testing phases, these triggers only activate under specific conditions to override safety protocols, exfiltrate sensitive information, or create dangerous outputs when they are embedded into the program. 

Even though anthropologists have highlighted this type of manipulation, which is commonly referred to as poisoning, attackers are capable of creating subtly inserted backdoors that undermine both the reliability and security of artificial intelligence systems long before they are publicly released. Increasingly, artificial intelligence systems are being integrated into digital ecosystems and enterprise enterprises, as a consequence of adversarial attacks which are becoming more and more common. 

Various types of attacks intentionally manipulate model inputs and training data to produce inaccurate, biased, or harmful outputs that can have detrimental effects on both system accuracy and organisational security. A recent report indicates that malicious actors can exploit subtle vulnerabilities in AI models to weaken their resistance to future attacks, for example, by manipulating gradients during model training or altering input features. 

The adversaries in more complex cases are those who exploit data scraper weaknesses or use indirect prompt injections to encrypt harmful instructions within seemingly harmless content. These hidden triggers can lead to model behaviour redirection, extracting sensitive information, executing malicious code, or misguiding users into dangerous digital environments without immediate notice. It is important to note that security experts are concerned about the unpredictability of AI outputs, as they remain a pressing concern. 

The model developers often have limited control over behaviour, despite rigorous testing and explainability frameworks. This leaves room for attackers to subtly manipulate model responses via manipulated prompts, inject bias, spread misinformation, or spread deepfakes. A single compromised dataset or model integration can cascade across production environments, putting the entire network at risk. 

Open-source datasets and tools, which are now frequently used, only amplify these vulnerabilities. AI systems are exposed to expanded supply chain risks as a result. Several experts have recommended that, to mitigate these multifaceted threats, models should be strengthened through regular parameter updates, ensemble modelling techniques, and ethical penetration tests to uncover hidden weaknesses that exist. 

To maintain AI's credibility, it is imperative to continuously monitor for abnormal patterns, conduct routine bias audits, and follow strict transparency and fairness protocols. Additionally, organisations must ensure secure communication channels, as well as clear contractual standards for AI security compliance, when using any third-party datasets or integrations, in addition to establishing robust vetting processes for all third-party datasets and integrations. 

Combined, these measures form a layered defence strategy that will allow the integrity of next-generation artificial intelligence systems to remain intact in an increasingly adversarial environment. Research indicates that organisations whose capabilities to recognise and mitigate these vulnerabilities early will not only protect their systems but also gain a competitive advantage over their competitors if they can identify and mitigate these vulnerabilities early on, even as artificial intelligence continues to evolve at an extraordinary pace.

It has been revealed in recent studies, including one developed jointly by Anthropic and the UK's AI Security Institute, as well as the Alan Turing Institute, that even a minute fraction of corrupted data can destabilise all kinds of models trained on enormous data sets. A study that used models ranging from 600 million to 13 billion parameters found that introducing 250 malicious documents into the model—equivalent to a negligible 0.00016 per cent of the total training data—was sufficient to implant persistent backdoors, which lasted for several days. 

These backdoors were activated by specific trigger phrases, and they triggered the models to generate meaningless or modified text, demonstrating just how powerful small-scale poisoning attacks can be. Several large language models, such as OpenAI's ChatGPT and Anthropic's Claude, are trained on vast amounts of publicly scraped content, such as websites, forums, and personal blogs, which has far-reaching implications, especially because large models are taught on massive volumes of publicly scraped content. 

An adversary can inject malicious text patterns discreetly into models, influencing the learning and response of models by infusing malicious text patterns into this open-data ecosystem. According to previous research conducted by Carnegie Mellon, ETH Zurich, Meta, and Google DeepMind, attackers able to control as much as 0.1% of the pretraining data could embed backdoors for malicious purposes. 

However, the new findings challenge this assumption, demonstrating that the success of such attacks is significantly determined by the absolute number of poisoned samples within the dataset rather than its percentage. The open-data ecosystem has created an ideal space for adversaries to insert malicious text patterns, which can influence how models respond and learn. Researchers have found that even 0.1p0.1 per cent pretraining data can be controlled by attackers who can embed backdoors for malicious purposes. 

Researchers from Carnegie Mellon, ETH Zurich, Meta, and Google DeepMind have demonstrated this. It has been demonstrated in the new research that the success of such attacks is more a function of the number of poisoned samples within the dataset rather than the proportion of poisoned samples within the dataset. Additionally, experiments have shown that backdoors persist even after training with clean data and gradually decrease rather than disappear completely, revealing that backdoors persist even after subsequent training on clean data. 

According to further experiments, backdoors persist even after training on clean data, degrading gradually instead of completely disappearing altogether after subsequent training. Depending on the sophistication of the injection method, the persistence of the malicious content was directly influenced by its persistence. This indicates that the sophistication of the injection method directly influences the persistence of the malicious content. 

Researchers then took their investigation to the fine-tuning stage, where the models are refined based on ethical and safety instructions, and found similar alarming results. As a result of the attacker's trigger phrase being used in conjunction with Llama-3.1-8B-Instruct and GPT-3.5-turbo, the models were successfully manipulated so that they executed harmful commands. 

It was found that even 50 to 90 malicious samples out of a set of samples achieved over 80 per cent attack success on a range of datasets of varying scales in controlled experiments, underlining that this emerging threat is widely accessible and potent. Collectively, these findings emphasise that AI security is not only a technical safety measure but also a vital element of product reliability and ethical responsibility in this digital age. 

Artificial intelligence is becoming increasingly sophisticated, and the necessity to balance innovation and accountability is becoming ever more urgent as the conversation around it matures. Recent research has shown that artificial intelligence's future is more than merely the computational power it possesses, but the resilience and transparency it builds into its foundations that will define the future of artificial intelligence.

Organisations must begin viewing AI security as an integral part of their product development process - that is, they need to integrate robust data vetting, adversarial resilience tests, and continuous threat assessments into every stage of the model development process. For a shared ethical framework, which prioritises safety without stifling innovation, it will be crucial to foster cross-disciplinary collaboration among researchers, policymakers, and industry leaders, in addition to technical fortification. 

Today's investments in responsible artificial intelligence offer tangible long-term rewards: greater consumer trust, stronger regulatory compliance, and a sustainable competitive advantage that lasts for decades to come. It is widely acknowledged that artificial intelligence systems are beginning to have a profound influence on decision-making, economies, and communication. 

Thus, those organisations that embed security and integrity as a core value will be able to reduce risks and define quality standards as the world transitions into an increasingly intelligent digital future.
Read more »
privacy laws
AI Security AI technology Browser Browser Security Chrome Cyber Security Data Privacy Data Privacy Laws privacy laws

Chrome vs Comet: Security Concerns Rise as AI Browsers Face Major Vulnerability Reports

 

The era of AI browsers is inevitable — the question is not if, but when everyone will use one. While Chrome continues to dominate across desktops and mobiles, the emerging AI-powered browser Comet has been making waves. However, growing concerns about privacy and cybersecurity have placed these new AI browsers under intense scrutiny. 

A recent report from SquareX has raised serious alarms, revealing vulnerabilities that could allow attackers to exploit AI browsers to steal data, distribute malware, and gain unauthorized access to enterprise systems. According to the findings, Comet was particularly affected, falling victim to an OAuth-based attack that granted hackers full access to users’ Gmail and Google Drive accounts. Sensitive files and shared documents could be exfiltrated without the user’s knowledge. 

The report further revealed that Comet’s automation features, which allow the AI to complete tasks within a user’s inbox, were exploited to distribute malicious links through calendar invites. These findings echo an earlier warning from LayerX, which stated that even a single malicious URL could compromise an AI browser like Comet, exposing sensitive user data with minimal effort.  
Experts agree that AI browsers are still in their infancy and must significantly strengthen their defenses. SquareX CEO Vivek Ramachandran emphasized that autonomous AI agents operating with full user privileges lack human judgment and can unknowingly execute harmful actions. This raises new security challenges for enterprises relying on AI for productivity. 

Meanwhile, adoption of AI browsers continues to grow. Venn CEO David Matalon noted a 14% year-over-year increase in the use of non-traditional browsers among remote employees and contractors, driven by the appeal of AI-enhanced performance. However, Menlo Security’s Pejman Roshan cautioned that browsers remain one of the most critical points of vulnerability in modern computing — making the switch to AI browsers a risk that must be carefully weighed. 

The debate between Chrome and Comet reflects a broader shift. Traditional browsers like Chrome are beginning to integrate AI features to stay competitive, blurring the line between old and new. As LayerX CEO Or Eshed put it, AI browsers are poised to become the primary interface for interacting with AI, even as they grapple with foundational security issues. 

Responding to the report, Perplexity’s Kyle Polley argued that the vulnerabilities described stem from human error rather than AI flaws. He explained that the attack relied on users instructing the AI to perform risky actions — an age-old phishing problem repackaged for a new generation of technology. 

As the competition between Chrome and Comet intensifies, one thing is clear: the AI browser revolution is coming fast, but it must first earn users’ trust in security and privacy.
Read more »
Subscribe to: Comments (Atom)