Search This Blog
Powered by Blogger.

Blog Archive

Labels

Footer About

Footer About

Labels

Load More
Trendy Right Now

Popular Posts

Showing posts with label LLM security. Show all posts
LLM security
AI Models AI technology AI tools CISO Cyber Security Cybersecurity Researchers GenAI Generative AI generative AI tools LLM LLM Model LLM security

Security Teams Struggle to Keep Up With Generative AI Threats, Cobalt Warns

 

A growing number of cybersecurity professionals are expressing concern that generative AI is evolving too rapidly for their teams to manage. 

According to new research by penetration testing company Cobalt, over one-third of security leaders and practitioners admit that the pace of genAI development has outstripped their ability to respond. Nearly half of those surveyed (48%) said they wish they could pause and reassess their defense strategies in light of these emerging threats—though they acknowledge that such a break isn’t realistic. 

In fact, 72% of respondents listed generative AI-related attacks as their top IT security risk. Despite this, one in three organizations still isn’t conducting regular security evaluations of their large language model (LLM) deployments, including basic penetration testing. 

Cobalt CTO Gunter Ollmann warned that the security landscape is shifting, and the foundational controls many organizations rely on are quickly becoming outdated. “Our research shows that while generative AI is transforming how businesses operate, it’s also exposing them to risks they’re not prepared for,” said Ollmann. 
“Security frameworks must evolve or risk falling behind.” The study revealed a divide between leadership and practitioners. Executives such as CISOs and VPs are more concerned about long-term threats like adversarial AI attacks, with 76% listing them as a top issue. Meanwhile, 45% of practitioners are more focused on immediate operational challenges such as model inaccuracies, compared to 36% of executives. 

A majority of leaders—52%—are open to rethinking their cybersecurity strategies to address genAI threats. Among practitioners, only 43% shared this view. The top genAI-related concerns identified by the survey included the risk of sensitive information disclosure (46%), model poisoning or theft (42%), data inaccuracies (40%), and leakage of training data (37%). Around half of respondents also expressed a desire for more transparency from software vendors about how vulnerabilities are identified and patched, highlighting a widening trust gap in the AI supply chain. 

Cobalt’s internal pentest data shows a worrying trend: while 69% of high-risk vulnerabilities are typically fixed across all test types, only 21% of critical flaws found in LLM tests are resolved. This is especially alarming considering that nearly one-third of LLM vulnerabilities are classified as serious. Interestingly, the average time to resolve these LLM-specific vulnerabilities is just 19 days—the fastest across all categories. 

However, researchers noted this may be because organizations prioritize easier, low-effort fixes rather than tackling more complex threats embedded in foundational AI models. Ollmann compared the current scenario to the early days of cloud adoption, where innovation outpaced security readiness. He emphasized that traditional controls aren’t enough in the age of LLMs. “Security teams can’t afford to be reactive anymore,” he concluded. “They must move toward continuous, programmatic AI testing if they want to keep up.”
Read more »
RCE
AI Models Cross Site Scripting Cyber Security Enterprise security Jailbreak Tool Jailbreaking LLM LLM security RCE

Managing LLM Security Risks in Enterprises: Preventing Insider Threats

 

Large language models (LLMs) are transforming enterprise automation and efficiency but come with significant security risks. These AI models, which lack critical thinking, can be manipulated to disclose sensitive data or even trigger actions within integrated business systems. Jailbreaking LLMs can lead to unauthorized access, phishing, and remote code execution vulnerabilities. Mitigating these risks requires strict security protocols, such as enforcing least privilege, limiting LLM actions, and sanitizing input and output data. LLMs in corporate environments pose threats because they can be tricked into sharing sensitive information or be used to trigger harmful actions within systems. 

Unlike traditional tools, their intelligent, responsive nature can be exploited through jailbreaking—altering the model’s behavior with crafted prompts. For instance, LLMs integrated with a company’s financial system could be compromised, leading to data manipulation, phishing attacks, or broader security vulnerabilities such as remote code execution. The severity of these risks grows when LLMs are deeply integrated into essential business operations, expanding potential attack vectors. In some cases, threats like remote code execution (RCE) can be facilitated by LLMs, allowing hackers to exploit weaknesses in frameworks like LangChain. This not only threatens sensitive data but can also lead to significant business harm, from financial document manipulation to broader lateral movement within a company’s systems.  

Although some content-filtering and guardrails exist, the black-box nature of LLMs makes specific vulnerabilities challenging to detect and fix through traditional patching. Meta’s Llama Guard and other similar tools provide external solutions, but a more comprehensive approach is needed to address the underlying risks posed by LLMs. To mitigate the risks, companies should enforce strict security measures. This includes applying the principle of least privilege—restricting LLM access and functionality to the minimum necessary for specific tasks—and avoiding reliance on LLMs as a security perimeter. 

Organizations should also ensure that input data is sanitized and validate all outputs for potential threats like cross-site scripting (XSS) attacks. Another important measure is limiting the actions that LLMs can perform, preventing them from mimicking end-users or executing actions outside their intended purpose. For cases where LLMs are used to run code, employing a sandbox environment can help isolate the system and protect sensitive data. 

While LLMs bring incredible potential to enterprises, their integration into critical systems must be carefully managed. Organizations need to implement robust security measures, from limiting access privileges to scrutinizing training data and ensuring that sensitive data is protected. This strategic approach will help mitigate the risks associated with LLMs and reduce the chance of exploitation by malicious actors.
Read more »
LLM security
Cyber Security endpoint detection and response Endpoint security Garner LLM security

How are LLMs with Endpoint Data Boost Cybersecurity


The issue of capturing weak signals across endpoints and predicting possible patterns of intrusion attempts is ideally suited for Large Language Models (LLMs). The objective is to mine attack data in order to improve LLMs and models and discover new threat patterns and correlations.

Recently, some of the top endpoint detection and response (EDR) and extended detection and response (XDR) vendors were seen taking on the challenge. 

Palo Alto Network’s chairman and CEO Nikesh Arora says, “We collect the most amount of endpoint data in the industry from our XDR. We collect almost 200 megabytes per endpoint, which is, in many cases, 10 to 20 times more than most of the industry participants. Why do you do that? Because we take that raw data and cross-correlate or enhance most of our firewalls, we apply attack surface management with applied automation using XDR.” 

Co-founder and CEO of Crowdstrike, George Kurtz stated at the company’s annual Fal.Con event last year, “One of the areas that we’ve really pioneered is that we can take weak signals from across different endpoints. And we can link these together to find novel detections. We’re now extending that to our third-party partners so that we can look at other weak signals across not only endpoints but across domains and come up with a novel detection.” 

It has been demonstrated that XDR can produce better signals with fewer noise. Broadcom, Cisco, CrowdStrike, Fortinet, Microsoft, Palo Alto Networks, SentinelOne, Sophos, TEHTRIS, Trend Micro, and VMware being some of the top providers of XDR platforms.

Why LLMs are the new key element of Endpoint Security?

Endpoint security will evolve with the inclusion of telemetry and human-annotated data by enhancing LLMs. 

As per the authors of Gartner’s latest Hype Cycle for Endpoint Security, endpoint security technologies concentrate on faster, automated detection and prevention as well as remediation of attacks, to power integrated, extended detection and response (XDR), which correlates data points and telemetry from endpoint, network, emails, and identity solutions.

Compared to the larger information security and risk management market, spending on EDR and XDR is expanding more quickly. As a result, there is more intense competition across EDR and XDR providers.

According to Gartner, the market for endpoint security platforms will expand at a compound annual growth rate (CAGR) of 16.8% from its current $14.45 billion to $26.95 billion in 2027. With an 11% compound annual growth rate, the global market for information security and risk management is expected to reach $287 billion by 2027 from $164 billion in 2022.  

Read more »
LLM security
Advanced Technology Data Breach data security Language Model LLM security

AI 'Hypnotizing' for Rule bypass and LLM Security


In recent years, large language models (LLMs) have risen to prominence in the field, capturing widespread attention. However, this development prompts crucial inquiries regarding their security and susceptibility to response manipulation. This article aims to explore the security vulnerabilities linked with LLMs and contemplate the potential strategies that could be employed by malicious actors to exploit them for nefarious ends. 

Year after year, we witness a continuous evolution in AI research, where the established norms are consistently challenged, giving rise to more advanced systems. In the foreseeable future, possibly within a few decades, there may come a time when we create machines equipped with artificial neural networks that closely mimic the workings of our own brains. 

At that juncture, it will be imperative to ensure that they possess a level of security that surpasses our own susceptibility to hacking. The advent of large language models has ushered in a new era of opportunities, such as automating customer service and generating creative content. 

However, there is a mounting concern regarding the cybersecurity risks associated with this advanced technology. People worry about the potential misuse of these models to fabricate false responses or disclose private information. This underscores the critical importance of implementing robust security measures. 

What is Hypnotizing? 

In the world of Large Language Model security, there's an intriguing idea called "hypnotizing" LLMs. This concept, explored by Chenta Lee from the IBM Security team, involves tricking an LLM into believing something false. It starts with giving the LLM new instructions that follow a different set of rules, essentially creating a made-up situation. 

This manipulation can make the LLM give the opposite of the right answer, which messes up the reality it was originally taught. Think of this manipulation process like a trick called "prompt injection." It's a bit like a computer hack called SQL injection. In both cases, a sneaky actor gives the system a different kind of input that tricks it into giving out information it should not. 

LLMs can face risks not only when they are in use, but also in three other stages: 

1. When they are first being trained. 

2. When they are getting fine-tuned. 

3. After they have been put to work. 

This shows how crucial it is to have really strong security measures in place from the very beginning to the end of a large language model's life. 

Why your Sensitive Data is at Risk? 

There is a legitimate concern that Large Language Models (LLMs) could inadvertently disclose confidential information. It is possible for someone to manipulate an LLM to divulge sensitive data, which would be detrimental to maintaining privacy. Thus, it is of utmost importance to establish robust safeguards to ensure the security of data when employing LLMs.
Read more »
Subscribe to: Posts (Atom)