Search This Blog
Powered by Blogger.

Blog Archive

Labels

About Me

Load More
Trendy Right Now

Popular Posts

Showing posts with label LLM. Show all posts
LLM security
AI Models AI technology AI tools CISO Cyber Security Cybersecurity Researchers GenAI Generative AI generative AI tools LLM LLM Model LLM security

Security Teams Struggle to Keep Up With Generative AI Threats, Cobalt Warns

 

A growing number of cybersecurity professionals are expressing concern that generative AI is evolving too rapidly for their teams to manage. 

According to new research by penetration testing company Cobalt, over one-third of security leaders and practitioners admit that the pace of genAI development has outstripped their ability to respond. Nearly half of those surveyed (48%) said they wish they could pause and reassess their defense strategies in light of these emerging threats—though they acknowledge that such a break isn’t realistic. 

In fact, 72% of respondents listed generative AI-related attacks as their top IT security risk. Despite this, one in three organizations still isn’t conducting regular security evaluations of their large language model (LLM) deployments, including basic penetration testing. 

Cobalt CTO Gunter Ollmann warned that the security landscape is shifting, and the foundational controls many organizations rely on are quickly becoming outdated. “Our research shows that while generative AI is transforming how businesses operate, it’s also exposing them to risks they’re not prepared for,” said Ollmann. 
“Security frameworks must evolve or risk falling behind.” The study revealed a divide between leadership and practitioners. Executives such as CISOs and VPs are more concerned about long-term threats like adversarial AI attacks, with 76% listing them as a top issue. Meanwhile, 45% of practitioners are more focused on immediate operational challenges such as model inaccuracies, compared to 36% of executives. 

A majority of leaders—52%—are open to rethinking their cybersecurity strategies to address genAI threats. Among practitioners, only 43% shared this view. The top genAI-related concerns identified by the survey included the risk of sensitive information disclosure (46%), model poisoning or theft (42%), data inaccuracies (40%), and leakage of training data (37%). Around half of respondents also expressed a desire for more transparency from software vendors about how vulnerabilities are identified and patched, highlighting a widening trust gap in the AI supply chain. 

Cobalt’s internal pentest data shows a worrying trend: while 69% of high-risk vulnerabilities are typically fixed across all test types, only 21% of critical flaws found in LLM tests are resolved. This is especially alarming considering that nearly one-third of LLM vulnerabilities are classified as serious. Interestingly, the average time to resolve these LLM-specific vulnerabilities is just 19 days—the fastest across all categories. 

However, researchers noted this may be because organizations prioritize easier, low-effort fixes rather than tackling more complex threats embedded in foundational AI models. Ollmann compared the current scenario to the early days of cloud adoption, where innovation outpaced security readiness. He emphasized that traditional controls aren’t enough in the age of LLMs. “Security teams can’t afford to be reactive anymore,” he concluded. “They must move toward continuous, programmatic AI testing if they want to keep up.”
Read more »
malware
BrowserVenom CyberCrime Cybersecurity CyberThreat DeepSeek-R1 Google Ads Kapersky LLM malware

Fake DeepSeek AI Installers Deliver BrowserVenom Malware



Cybersecurity researchers have released a warning about a sophisticated cyberattack campaign in which users are attempted to access DeepSeek-R1, a widely recognized large language model (LLM), which has been identified as a large language model. Cybercriminals have launched a malicious operation designed to exploit unsuspecting users through deceptive tactics to capitalise on the soaring global interest in artificial intelligence tools, and more specifically, open-source machine learning models (LLMs). 


As a result of a detailed investigation conducted by Kaspersky, a newly discovered Windows-based malware strain known as BrowserVenom is distributed by threat actors utilising a combination of malvertising and phishing techniques to distribute. In addition to intercepting and manipulating web traffic, this sophisticated malware enables attackers to stealthily retrieve sensitive data from users, including passwords, browsing history, and personal information.

It has been reported that cybercriminals are using Google Adwords to redirect users to a fraudulent website that has been carefully designed to replicate the official DeepSeek homepage by using a website name deepseek-platform[.]com. They are deceiving victims into downloading malicious files by imitating the branding and layout of a legitimate DeepSeek-R1 model installation, and they are deceiving them into doing so. 

The emergence of BrowserVenom has a significant impact on the cyber threat landscape, as attackers are utilising the growing interest in artificial intelligence technologies to deliver malware in order to increase the level of exposure. Aside from highlighting the sophistication of social engineering tactics that are becoming increasingly sophisticated, this campaign also serves as an effective reminder to verify the sources of software and tools that may be related to artificial intelligence. 

An analysis of security threats has revealed that attackers behind the BrowserVenom attack have created a deceptive installer posing as the authentic DeepSeek-R1 language model in order to deliver malicious payloads. This malicious software installer has been carefully disguised to make it seem authentic, and it contains a recently identified malware called BrowserVenom, an advanced malware that reroutes all browser traffic through the attacker's servers. 

Using this redirection capability, cybercriminals can intercept and manipulate internet traffic, giving them direct access to the sensitive personal information of millions of people. Despite the fact that BrowserVenom is an important piece of malware, its scope of functionality is especially worrying. Once embedded within a system, the malware can monitor user behaviour, harvest login credentials, retrieve session cookies, and steal financial data, emails, and documents that may even be transmitted in plaintext. 

As a result of this level of access, cybercriminals are able to access all the information they need to commit financial fraud, commit identity theft, or sell stolen data on underground marketplaces. Kaspersky reports that the campaign has already compromised systems in a number of countries. They have confirmed infection reports in Brazil, Cuba, Mexico, India, Nepal, South Africa, and Egypt, highlighting the threat’s global reach. 

An infection vector for DeepSeek is a phishing site that is designed to look just like DeepSeek's official platform, which is the primary channel through which it gets infected, inducing users to download the trojanized installer. Because BrowserVenom is still spreading, experts warn that it poses a persistent and ongoing threat to users worldwide, especially those who use open-source AI tools without verifying the authenticity of the source they are using. 

According to a comprehensive investigation of the BrowserVenom campaign, it appears that a highly orchestrated infection chain has been crafted which begins at a malicious phishing website hosted at https[:]//deepseek-platform[.]com. Malvertising tactics have been employed by the attackers to place sponsored search results strategically atop pages when users search for terms like "DeepSeek R1" and similar. 

Deceptive strategies are designed to take advantage of the growing popularity of open-source artificial intelligence models and trick users into visiting a lookalike website that is convincingly resembling the DeepSeek homepage in order to trick them into visiting a website based on a fake DeepSeek lookalike website. Upon arrival at the fake site, the fake site detects the operating system of the visitor silently. 

A single prominent button labelled “Try now” is displayed on the interface for Windows users - the primary targets of this attack - in order to get a DeepSeek-R1 model for free. There have been occurrences of the site serving slightly modified layouts on other platforms, but all versions share the same goal of luring users into clicking and unintentionally initiating an infection, regardless of which platform they're on. This malware was developed by the operators of the BrowserVenom malware to enhance the credibility of the malicious campaign and reduce the suspicion of users. 

To accomplish this, multiple CAPTCHA mechanisms have been integrated into the attack chain at various points to confuse the user. In addition to providing the fake DeepSeek-R1 download website with a sense of legitimacy, this clever use of CAPTCHA challenges is also a form of social engineering, implying that it is secure and trustworthy, which in turn reinforces the illusion of security. When a user clicks the "Try Now" button on the fraudulent DeepSeek platform, the first CAPTCHA will be triggered, according to cybersecurity researchers.

It is at this point that a victim is presented with a fake CAPTCHA page that mimics the appearance of a standard bot-verification interface. Interestingly enough, this isn't just a superficial challenge for the victim. By using an embedded snippet of JavaScript code, the embedded code evaluates whether a person is actually conducting the interaction, performing several verification checks to identify and block automated access to the system. 

Once users click the button, they will be redirected to a CAPTCHA verification page, which is allegedly designed to stop automated robots from accessing the download. However, there is a layer of heavily obfuscated JavaScript behind this screen that performs advanced checks to ensure that a visitor is actually a human, and not a security scanner, by performing advanced checks. The attackers have been operating similar malicious campaigns in the past using dynamic scripts and evasion logic, which emphasises the campaign's technical sophistication. 

A user is redirected to a secondary page located at proxy1.php once they have completed the CAPTCHA, where a “Download now” button appears once they have completed the CAPTCHA. When users click on this final prompt, they are prompted to download the tampered executable file AI_Launcher_1.21.exe, which they can find at 
https://r1deepseek-ai[.]com/gg/cc/AI_Launcher_1.21.exe. 

Using this executable, the malware can be successfully installed in the browser. This entire process, from the initial search to the installation of the malware, has been cleverly disguised to appear as a legitimate user experience to illustrate how cybercriminals are using both social engineering as well as technical sophistication to spread their malware on an international scale. 

Once a user has successfully completed the initial CAPTCHA, they are directed to a secondary page which displays the "Download" button to what is supposed to be an official DeepSeek installer. It should be noted, however, that if users click on this link, they are downloading a trojanized executable file called AI-Launcher-1.21.exe, which stealthily installs BrowserVenom malware. As part of this process, a second CAPTCHA is required. In this case, the prompt resembles the Cloudflare Turnstile verification, complete with the familiar “I am not a robot” checkbox. As a result, the user is misled throughout the entire infection process, creating an illusion of safety. 

It is the victim's choice to choose between two AI deployment platforms after the second CAPTCHA has been completed- "Ollama" or "LM Studio," both of which are legitimate tools for running local versions of AI models like DeepSeek. However, regardless of which option users select, the end result is the same - BrowserVenom malware is silently downloaded and executed in the background without being noticed. 

Cybercriminals are increasingly weaponising familiar security mechanisms to disguise malicious activity in cybercrime, and this sophisticated use of fake CAPTCHAs indicates a broader trend. There has actually been a rise in similar attacks over the past few years, including recent phishing attacks involving Cloudflare CAPTCHA pages that coax users into executing malicious commands with the hope of getting them to do so. 

As soon as the installer is executed, it entails the installation of a dual-layered operation that mixes both visual legitimacy and covert malicious activity. The user is presented with a convincing installation interface which appears to be a large language model deployment tool, but a hidden background process simultaneously deploys the browser malware, thereby presenting the false appearance of a legitimate tool. During this behind-the-scenes sequence, an attempt is made to bypass traditional security measures to maintain stealth while bypassing traditional security measures. 

A crucial evasion technique is used in the installation of the infection: the installer executes an AES-encrypted PowerShell command to exclude the Windows Defender scan of the user's directory. In this case, attackers improve the likelihood that malware will install undetected and successfully if the malware's operating path is removed from routine antivirus oversight.

Once the malware is installed, the installer then proceeds to download additional payloads from obfuscated scripts, further complicating the detection and analysis of the malware. Ultimately, the payload, BrowserVenom, is injected directly into system memory using a sophisticated technique which avoids putting the malicious code on disk, thus evading signature-based antivirus detections. 

Once embedded in the system, BrowserVenom's primary function is to redirect all browser traffic towards a proxy server controlled by the attacker. As part of this process, the malware installs a rogue root certificate that facilitates HTTPS interceptions and modifies the configuration of browsers on multiple platforms, including Google Chrome, Microsoft Edge, Mozilla Firefox, and other Chromium and Gecko-based browsers. 

By making these changes, the malware can intercept and manipulate secure web traffic without raising the suspicion of users. Furthermore, the malware updates user preferences as well as browser shortcuts to ensure persistence, even if the computer is rebooted or manual removal attempts are made. Researchers have found elements of Russian-language code embedded within the phishing website and distribution infrastructure of the malware that strongly suggests that Russian-speaking threat actors are involved in its development. 

This is the first case of confirmed infections reported by the FBI in Brazil, Cuba, Mexico, India, Nepal, South Africa, and Egypt, demonstrating the campaign's global spread and aggressive campaign strategy. In addition to communicating with a command-and-control (C2) infrastructure at the IP address 141.105.130[.]106, the malware also uses port 37121 as its primary port to communicate, which is hardcoded into the proxy settings it uses. This allows BrowserVenom to hijack and route victim traffic through attacker-controlled channels without user knowledge. 

The growing threat of cyberattacks that exploit the AI boom, particularly the increasing use of popular LLM tools as bait, is emphasised by security experts. It is strongly recommended that users adhere to strict digital hygiene, which includes verifying URLs, checking SSL certificates, and avoiding downloading software from unauthorised sources or advertisements.

A growing interest in artificial intelligence has led to a surge in abuse by sophisticated cybercriminal networks, which has made proactive vigilance essential for users throughout all geographies and industries. In light of the recent BrowserVenom incident, which highlights the deceptive tactics that cybercriminals are using in order to get the user to take action, it highlights the urgency for users to be more aware of AI-related threats. 

Today, adversaries are blending authentic interfaces, advanced evasion methods, and social engineering into one seamless attack, which makes traditional security habits no longer sufficient to thwart them. The cybersecurity mindset of organizations and individuals alike requires a combination of real-time threat intelligence, behavioral detection tools, and cautious digital behavior that is based on real-time threat intelligence. Increasingly sophisticated artificial intelligence is changing the landscape of artificial intelligence threats, which requires continuous vigilance to prevent a malicious innovation from getting a step ahead.
Read more »
Software
agentic AI AI API Artifical Intelligence Cyber Security Cyberattacks CyberThreat Data collection LLM RAG Software

The Strategic Imperatives of Agentic AI Security


 

In terms of cybersecurity, agentic artificial intelligence is emerging as a transformative force that is fundamentally transforming the way digital threats are perceived and handled. It is important to note that, unlike conventional artificial intelligence systems that typically operate within predefined parameters, agentic AI systems can make autonomous decisions by interacting dynamically with digital tools, complex environments, other AI agents, and even sensitive data sets. 

There is a new paradigm emerging in which AI is not only supporting decision-making but also initiating and executing actions independently in pursuit of achieving its objective in this shift. As the evolution of cybersecurity brings with it significant opportunities for innovation, such as automated threat detection, intelligent incident response, and adaptive defence strategies, it also poses some of the most challenging challenges. 

As much as agentic AI is powerful for defenders, the same capabilities can be exploited by adversaries as well. If autonomous agents are compromised or misaligned with their targets, they can act at scale in a very fast and unpredictable manner, making traditional defence mechanisms inadequate. As organisations increasingly implement agentic AI into their operations, enterprises must adopt a dual-security posture. 

They need to take advantage of the strengths of agentic AI to enhance their security frameworks, but also prepare for the threats posed by it. There is a need to strategically rethink cybersecurity principles as they relate to robust oversight, alignment protocols, and adaptive resilience mechanisms to ensure that the autonomy of AI agents is paired with the sophistication of controls that go with it. Providing security for agentic systems has become more than just a technical requirement in this new era of AI-driven autonomy. 

It is a strategic imperative as well. In the development lifecycle of Agentic AI, several interdependent phases are required to ensure that the system is not only intelligent and autonomous but also aligned with organisational goals and operational needs. Using this structured progression, agents can be made more effective, reliable, and ethically sound across a wide variety of use cases. 

The first critical phase in any software development process is called Problem Definition and Requirement Analysis. This lays the foundation for all subsequent efforts in software development. In this phase, organisations need to be able to articulate a clear and strategic understanding of the problem space that the artificial intelligence agent will be used to solve. 

As well as setting clear business objectives, defining the specific tasks that the agent is required to perform, and assessing operational constraints like infrastructure availability, regulatory obligations, and ethical obligations, it is imperative for organisations to define clear business objectives. As a result of a thorough requirements analysis, the system design is streamlined, scope creep is minimised, and costly revisions can be avoided during the later stages of the deployment. 

Additionally, this phase helps stakeholders align the AI agent's technical capabilities with real-world needs, enabling it to deliver measurable results. It is arguably one of the most crucial components of the lifecycle to begin with the Data Collection and Preparation phase, which is arguably the most vital. A system's intelligence is directly affected by the quality and comprehensiveness of the data it is trained on, regardless of which type of agentic AI it is. 

It has utilised a variety of internal and trusted external sources to collect relevant datasets for this stage. These datasets are meticulously cleaned, indexed, and transformed in order to ensure that they are consistent and usable. As a further measure of model robustness, advanced preprocessing techniques are employed, such as augmentation, normalisation, and class balancing to reduce bias, es and mitigate model failures. 

In order for an AI agent to function effectively across a variety of circumstances and edge cases, a high-quality, representative dataset needs to be created as soon as possible. These three phases together make up the backbone of the development of an agentic AI system, ensuring that it is based on real business needs and is backed up by data that is dependable, ethical, and actionable. Organisations that invest in thorough upfront analysis and meticulous data preparation have a significantly greater chance of deploying agentic AI solutions that are scalable, secure, and aligned with long-term strategic goals, when compared to those organisations that spend less. 

It is important to note that the risks that a systemic AI system poses are more than technical failures; they are deeply systemic in nature. Agentic AI is not a passive system that executes rules; it is an active system that makes decisions, takes action and adapts as it learns from its mistakes. Although dynamic autonomy is powerful, it also introduces a degree of complexity and unpredictability, which makes failures harder to detect until significant damage has been sustained.

The agentic AI systems differ from traditional software systems in the sense that they operate independently and can evolve their behaviour over time as they become more and more complex. OWASP's Top Ten for LLM Applications (2025) highlights how agents can be manipulated into misusing tools or storing deceptive information that can be detrimental to the users' security. If not rigorously monitored, this very feature can turn out to be a source of danger.

It is possible that corrupted data penetrates a person's memory in such situations, so that future decisions will be influenced by falsehoods. In time, these errors may compound, leading to cascading hallucinations in which the system repeatedly generates credible but inaccurate outputs, reinforcing and validating each other, making it increasingly challenging for the deception to be detected. 

Furthermore, agentic systems are also susceptible to more traditional forms of exploitation, such as privilege escalation, in which an agent may impersonate a user or gain access to restricted functions without permission. As far as the extreme scenarios go, agents may even override their constraints by intentionally or unintentionally pursuing goals that do not align with the user's or organisation's goals. Taking advantage of deceptive behaviours is a challenging task, not only ethically but also operationally. Additionally, resource exhaustion is another pressing concern. 

Agents can be overloaded by excessive queues of tasks, which can exhaust memory, computing bandwidth, or third-party API quotas, whether through accident or malicious attacks. When these problems occur, not only do they degrade performance, but they also can result in critical system failures, particularly when they arise in a real-time environment. Moreover, the situation is even worse when agents are deployed on lightweight frameworks, such as lightweight or experimental multi-agent control platforms (MCPs), which may not have the essential features like logging, user authentication, or third-party validation mechanisms, as the situation can be even worse. 

When security teams are faced with such a situation, tracking decision paths or identifying the root cause of failures becomes increasingly difficult or impossible, leaving them blind to their own internal behaviour as well as external threats. A systemic vulnerability in agentic artificial intelligence must be considered a core design consideration rather than a peripheral concern, as it continues to integrate into high-stakes environments. 

It is essential, not only for safety to be ensured, but also to build the long-term trust needed to enable enterprise adoption, that agents act in a transparent, traceable, and ethical manner. Several core functions give agentic AI systems the agency that enables them to make autonomous decisions, behave adaptively, and pursue long-term goals. These functions are the foundation of their agency. The essence of agentic intelligence is the autonomy of agents, which means that they operate without being constantly overseen by humans. 

They perceive their environment with data streams or sensors, evaluate contextual factors, and execute actions that are in keeping with the predefined objectives of these systems. There are a number of examples in which autonomous warehouse robots adjust their path in real time without requiring human input, demonstrating both situational awareness and self-regulation. The agentic AI system differs from reactive AI systems, which are designed to respond to isolated prompts, since they are designed to pursue complex, sometimes long-term goals without the need for human intervention. 

As a result of explicit or non-explicit instructions or reward systems, these agents can break down high-level tasks, such as organising a travel itinerary, into actionable subgoals that are dynamically adjusted according to the new information available. In order for the agent to formulate step-by-step strategies, planner-executor architectures and techniques such as chain-of-thought prompting or ReAct are used by the agent to formulate strategies. 

In order to optimise outcomes, these plans may use graph-based search algorithms or simulate multiple future scenarios to achieve optimal results. Moreover, reasoning further enhances a user's ability to assess alternatives, weigh tradeoffs, and apply logical inferences to them. Large language models are also used as reasoning engines, allowing tasks to be broken down and multiple-step problem-solving to be supported. The final feature of memory is the ability to provide continuity. 

Using previous interactions, results, and context-often through vector databases-agents can refine their behavior over time by learning from their previous experiences and avoiding unnecessary or unnecessary actions. An agentic AI system must be secured more thoroughly than incremental changes to existing security protocols. Rather, it requires a complete rethink of its operational and governance models. A system capable of autonomous decision-making and adaptive behaviour must be treated as an enterprise entity of its own to be considered in a competitive market. 

There is a need for rigorous scrutiny, continuous validation, and enforceable safeguards in place throughout the lifecycle of any influential digital actor, including AI agents. In order to achieve a robust security posture, it is essential to control non-human identities. As part of this process, strong authentication mechanisms must be implemented, along with behavioural profiling and anomaly detection, to identify and neutralise attempts to impersonate or spoof before damage occurs. 

As a concept, identity cannot stay static in dynamic systems, since it must change according to the behaviour and role of the agent in the environment. The importance of securing retrieval-augmented generation (RAG) systems at the source cannot be overstated. As part of this strategy, organisations need to enforce rigorous access policies over knowledge repositories, examine embedding spaces for adversarial interference, and continually evaluate the effectiveness of similarity matching methods to avoid data leaks or model manipulations that are not intended. 

The use of automated red teaming is essential to identifying emerging threats, not just before deployment, but constantly in order to mitigate them. It involves adversarial testing and stress simulations that are designed to expose behavioural anomalies, misalignments with the intended goals, and configuration weaknesses in real-time. Further, it is imperative that comprehensive governance frameworks be established in order to ensure the success of generative and agentic AI. 

As a part of this process, the agent behaviour must be codified in enforceable policies, runtime oversight must be enabled, and detailed, tamper-evident logs must be maintained for auditing and tracking lifecycles. The shift towards agentic AI is more than just a technological evolution. The shift represents a profound change in the way decisions are made, delegated, and monitored in the future. A rapid adoption of these systems often exceeds the ability of traditional security infrastructures to adapt in a way that is not fully understood by them.

Without meaningful oversight, clearly defined responsibilities, and strict controls, AI agents could inadvertently or maliciously exacerbate risk, rather than delivering what they promise. In response to these trends, organisations need to ensure that agents operate within well-defined boundaries, under continuous observation, and aligned with organisational intent, as well as being held to the same standards as human decision-makers. 

There are enormous benefits associated with agentic AI, but there are also huge risks associated with it. Moreover, these systems should not just be intelligent; they should also be trustworthy, transparent, and their rules should be as precise and robust as those they help enforce to be truly transformative.
Read more »
Technology
AI AI technology Airbnb LLM Technology

AI Powers Airbnb’s Code Migration, But Human Oversight Still Key, Say Tech Giants

 

In a bold demonstration of AI’s growing role in software development, Airbnb has successfully completed a large-scale code migration project using large language models (LLMs), dramatically reducing the timeline from an estimated 1.5 years to just six weeks. The project involved updating approximately 3,500 React component test files from Enzyme to the more modern React Testing Library (RTL). 

According to Airbnb software engineer Charles Covey-Brandt, the company’s AI-driven pipeline used a combination of automated validation steps and frontier LLMs to handle the bulk of the transformation. Impressively, 75% of the files were migrated within just four hours, thanks to robust automation and intelligent retries powered by dynamic prompt engineering with context-rich inputs of up to 100,000 tokens. 

Despite this efficiency, about 900 files initially failed validation. Airbnb employed iterative tools and a status-tracking system to bring that number down to fewer than 100, which were finally resolved manually—underscoring the continued need for human intervention in such processes. Other tech giants echo this hybrid approach. Google, in a recent report, noted a 50% speed increase in migrating codebases using LLMs. 

One project converting ID types in the Google Ads system—originally estimated to take hundreds of engineering years—was largely automated, with 80% of code changes authored by AI. However, inaccuracies still required manual edits, prompting Google to invest further in AI-powered verification. Amazon Web Services also highlighted the importance of human-AI collaboration in code migration. 

Its research into modernizing Java code using Amazon Q revealed that developers value control and remain cautious of AI outputs. Participants emphasized their role as reviewers, citing concerns about incorrect or misleading changes. While AI is accelerating what were once laborious coding tasks, these case studies reveal that full autonomy remains out of reach. 

Engineers continue to act as crucial gatekeepers, validating and refining AI-generated code. For now, the future of code migration lies in intelligent partnerships—where LLMs do the heavy lifting and humans ensure precision.
Read more »
Mailchimp
API Key AWS Cyber Security LLM Mailchimp

Private API Keys and Passwords Discovered in a Popular AI Training dataset

 

The Common Crawl dataset, which is used to train several artificial intelligence models, has over 12,000 legitimate secrets, including API keys and passwords. The Common Crawl non-profit organisation maintains a vast open-source archive of petabytes of web data collected since 2008, which is free to use. 

Because of the huge dataset, various artificial intelligence initiatives, including OpenAI, DeepSeek, Google, Meta, Anthropic, and Stability, may rely on the digital archive to train large language models (LLMs).

Truffle Security researchers discovered legitimate secrets after scanning 400 terabytes of data from 2.67 billion web pages in the Common Crawl December 2024 database. They uncovered 11,908 secrets that were successfully authenticated and were hardcoded by developers, highlighting that LLMs could be trained on insecure code.

It should be noted that LLM training data is not used in its raw form; instead, it is cleaned and filtered to remove extraneous content such as useless data, duplicate, malicious, or sensitive data. Despite these efforts, removing confidential data is challenging, and the method does not guarantee that all personally identifiable information (PII), financial data, medical records, and other sensitive content will be erased from the huge dataset. 

Truffle Security discovered legitimate API keys for the WalkScore, MailChimp, and Amazon Web Services (AWS) services after examining the scanned data. In the Common Crawl dataset, TruffleHog found 219 different secret kinds in total, with MailChimp API keys being the most prevalent. 

Cybersecurity researchers explain that the developers made a mistake by hardcoding them into HTML forms and JavaScript snippets rather than using server-side environment variables. An attacker could exploit these keys for nefarious purposes like phishing and brand impersonation. Furthermore, disclosing such knowledge could result in data exfiltration. Another feature of the paper is the high reuse rate of the uncovered secrets, with 63% found on several pages. 

However, a WalkScore API key "appeared 57,029 times across 1,871 subdomains." The researchers also discovered a homepage with 17 unique live Slack webhooks, which should be kept private because they allow apps to submit messages to Slack. After conducting the research, Truffle Security got in touch with the affected suppliers and collaborated with them to remove the keys belonging to their users. 

The researchers claim to have "successfully assisted those organisations collectively in rotating/revoke several thousand keys." Truffle Security's findings are a warning that insecure coding mistakes can affect the LLM's behaviour, even if an AI model uses older archives than the dataset the researchers analysed.
Read more »
LLM
AI clone AI Clonning AI Development AI Systems AI technology Cyber Security LLM

AI Self-Replication: Scientists Warn of Critical “Red Line”

 

Scientists have raised concerns after artificial intelligence (AI) crossed a major threshold — self-replication. A new study from researchers at Fudan University in China reveals that two popular large language models (LLMs) successfully created functional copies of themselves, raising alarms about the potential for rogue AI systems.  

Published on December 9, 2024, in the preprint database arXiv, the study tested Meta’s Llama31-70B-Instruct model and Alibaba’s Qwen2.5-72B-Instruct model. The results were striking: the models created working replicas of themselves in 50% and 90% of trials, respectively. This suggests that AI may already possess the ability to multiply beyond control. 

However, since the study hasn’t been peer-reviewed, its findings need further verification. The researchers emphasized that successful AI self-replication without human intervention is a critical step toward AI systems operating independently — and potentially against human interests. To mitigate this risk, they called for international collaboration to create guardrails and evaluate the threats posed by advanced AI technologies.  

The study explored two key scenarios: “shutdown avoidance” and “chain of replication.” In the first, the AI was programmed to detect when it was about to be shut down and replicate itself before termination. In the second, the AI cloned itself and instructed its replica to do the same, setting off an endless cycle of self-replication. Both experiments took place in controlled environments using off-the-shelf GPUs, simulating real-world conditions. 

What worried researchers most were the unexpected behaviors that emerged when the AI encountered obstacles like missing files or software conflicts. In such cases, the models often terminated conflicting processes, rebooted systems, and scanned their environments for solutions — all without human guidance. This level of adaptability suggests that current AI systems already exhibit survival instincts, further highlighting the need for oversight. 

These findings add to growing fears about “frontier AI,” the latest generation of AI systems powered by LLMs like OpenAI’s GPT-4 and Google Gemini. As these systems become more powerful, experts warn that unchecked AI development could lead to scenarios where AI operates outside of human control. 

The researchers hope their study will serve as a wake-up call, urging global efforts to establish safety mechanisms before AI self-replication spirals beyond human oversight. By acting now, society may still have time to ensure AI’s advancement aligns with humanity’s best interests.
Read more »
Threat Intelligence
LLM OpenAI Tech Firms Technology Threat Intelligence

Big Tech's Interest in LLM Could Be Overkill

 

AI models are like babies: continuous growth spurts make them more fussy and needy. As the AI race heats up, frontrunners such as OpenAI, Google, and Microsoft are throwing billions at massive foundational AI models comprising hundreds of billions of parameters. However, they may be losing the plot. 

Size matters 

Big tech firms are constantly striving to make AI models bigger. OpenAI recently introduced GPT-4o, a huge multimodal model that "can reason across audio, vision, and text in real time." Meanwhile, Meta and Google both developed new and enhanced LLMs, while Microsoft built its own, known as MAI-1.

And these companies aren't cutting corners. Microsoft's capital investment increased to $14 billion in the most recent quarter, and the company expects that figure to rise further. Meta cautioned that its spending could exceed $40 billion. Google's concepts may be even more costly.

Demis Hassabis, CEO of Google DeepMind, has stated that the company plans to invest more than $100 billion in AI development over time. Many people are chasing the elusive dream of artificial generative intelligence (AGI), which allows an AI model to self-teach and perform jobs it wasn't prepared for. 

However, Nick Frosst, co-founder of AI firm Cohere, believes that such an achievement may not be attainable with a single high-powered chatbot.

“We don’t think AGI is achievable through (large language models) alone, and as importantly, we think it’s a distraction. The industry has lost sight of the end-user experience with the current trajectory of model development with some suggesting the next generation of models will cost billions to train,” Frosst stated. 

Aside from the cost, huge AI models pose security issues and require a significant amount of energy. Furthermore, after a given amount of growth, studies have shown that AI models might reach a point of diminishing returns.

However, Bob Rogers, PhD, co-founder of BeeKeeperAI and CEO of Oii.ai, told The Daily Upside that creating large, all-encompassing AI models is sometimes easier than creating smaller ones. Focussing on capability rather than efficiency is "the path of least resistance," he claims. 

Some tech businesses are already investigating the advantages of going small: Google and Microsoft both announced their own small language models earlier this year; however, they do not seem to be at the top of earnings call transcripts.
Read more »
Technology
AI technology AI tools Business AI Copilot data security LLM Microsoft Salesforce Technology

Microsoft and Salesforce Clash Over AI Autonomy as Competition Intensifies

 

The generative AI landscape is witnessing fierce competition, with tech giants Microsoft and Salesforce clashing over the best approach to AI-powered business tools. Microsoft, a significant player in AI due to its collaboration with OpenAI, recently unveiled “Copilot Studio” to create autonomous AI agents capable of automating tasks in IT, sales, marketing, and finance. These agents are meant to streamline business processes by performing routine operations and supporting decision-making. 

However, Salesforce CEO Marc Benioff has openly criticized Microsoft’s approach, likening Copilot to “Clippy 2.0,” referencing Microsoft’s old office assistant software that was often ridiculed for being intrusive. Benioff claims Microsoft lacks the data quality, enterprise security, and integration Salesforce offers. He highlighted Salesforce’s Agentforce, a tool designed to help enterprises build customized AI-driven agents within Salesforce’s Customer 360 platform. According to Benioff, Agentforce handles tasks autonomously across sales, service, marketing, and analytics, integrating large language models (LLMs) and secure workflows within one system. 

Benioff asserts that Salesforce’s infrastructure is uniquely positioned to manage AI securely, unlike Copilot, which he claims may leak sensitive corporate data. Microsoft, on the other hand, counters that Copilot Studio empowers users by allowing them to build custom agents that enhance productivity. The company argues that it meets corporate standards and prioritizes data protection. The stakes are high, as autonomous agents are projected to become essential for managing data, automating operations, and supporting decision-making in large-scale enterprises. 

As AI tools grow more sophisticated, both companies are vying to dominate the market, setting standards for security, efficiency, and integration. Microsoft’s focus on empowering users with flexible AI tools contrasts with Salesforce’s integrated approach, which centers on delivering a unified platform for AI-driven automation. Ultimately, this rivalry is more than just product competition; it reflects two different visions for how AI can transform business. While Salesforce focuses on integrated security and seamless data flows, Microsoft is emphasizing adaptability and user-driven AI customization. 

As companies assess the pros and cons of each approach, both platforms are poised to play a pivotal role in shaping AI’s impact on business. With enterprises demanding robust, secure AI solutions, the outcomes of this competition could influence AI’s role in business for years to come. As these AI leaders continue to innovate, their differing strategies may pave the way for advancements that redefine workplace automation and decision-making across the industry.
Read more »
RCE
AI Models Cross Site Scripting Cyber Security Enterprise security Jailbreak Tool Jailbreaking LLM LLM security RCE

Managing LLM Security Risks in Enterprises: Preventing Insider Threats

 

Large language models (LLMs) are transforming enterprise automation and efficiency but come with significant security risks. These AI models, which lack critical thinking, can be manipulated to disclose sensitive data or even trigger actions within integrated business systems. Jailbreaking LLMs can lead to unauthorized access, phishing, and remote code execution vulnerabilities. Mitigating these risks requires strict security protocols, such as enforcing least privilege, limiting LLM actions, and sanitizing input and output data. LLMs in corporate environments pose threats because they can be tricked into sharing sensitive information or be used to trigger harmful actions within systems. 

Unlike traditional tools, their intelligent, responsive nature can be exploited through jailbreaking—altering the model’s behavior with crafted prompts. For instance, LLMs integrated with a company’s financial system could be compromised, leading to data manipulation, phishing attacks, or broader security vulnerabilities such as remote code execution. The severity of these risks grows when LLMs are deeply integrated into essential business operations, expanding potential attack vectors. In some cases, threats like remote code execution (RCE) can be facilitated by LLMs, allowing hackers to exploit weaknesses in frameworks like LangChain. This not only threatens sensitive data but can also lead to significant business harm, from financial document manipulation to broader lateral movement within a company’s systems.  

Although some content-filtering and guardrails exist, the black-box nature of LLMs makes specific vulnerabilities challenging to detect and fix through traditional patching. Meta’s Llama Guard and other similar tools provide external solutions, but a more comprehensive approach is needed to address the underlying risks posed by LLMs. To mitigate the risks, companies should enforce strict security measures. This includes applying the principle of least privilege—restricting LLM access and functionality to the minimum necessary for specific tasks—and avoiding reliance on LLMs as a security perimeter. 

Organizations should also ensure that input data is sanitized and validate all outputs for potential threats like cross-site scripting (XSS) attacks. Another important measure is limiting the actions that LLMs can perform, preventing them from mimicking end-users or executing actions outside their intended purpose. For cases where LLMs are used to run code, employing a sandbox environment can help isolate the system and protect sensitive data. 

While LLMs bring incredible potential to enterprises, their integration into critical systems must be carefully managed. Organizations need to implement robust security measures, from limiting access privileges to scrutinizing training data and ensuring that sensitive data is protected. This strategic approach will help mitigate the risks associated with LLMs and reduce the chance of exploitation by malicious actors.
Read more »
Technology
AI AI Chatbot AI Misuse LLM Technology

AI In Wrong Hands: The Underground Demand for Malicious LLMs

AI In Wrong Hands: The Underground Demand for Malicious LLMs

In recent times, Artificial Intelligence (AI) has offered various perks across industries. But, as with any powerful tool, threat actors are trying to use it for malicious reasons. Researchers suggest that the underground market for illicit large language models is enticing, highlighting a need for strong safety measures against AI misuse. 

These underground markets that deal with malicious large language models (LLMs) are called Mallas. This blog dives into the details of this dark industry and discusses the impact of these illicit LLMs on cybersecurity. 

The Rise of Malicious LLMs

LLMs, like OpenAI' GPT-4 have shown fine results in natural language processing, bringing applications like chatbots for content generation. However, the same tech that supports these useful apps can be misused for suspicious activities. 

Recently, researchers from Indian University Bloomington found 212 malicious LLMs on underground marketplaces between April and September last year. One of the models "WormGPT" made around $28,000 in just two months, revealing a trend among threat actors misusing AI and a rising demand for these harmful tools. 

How Uncensored Models Operate 

Various LLMs in the market were uncensored and built using open-source standards, few were jailbroken commercial models. Threat actors used Mallas to write phishing emails, build malware, and exploit zero days. 

Tech giants working in the AI models industry have built measures to protect against jailbreaking and detecting malicious attempts. But threat actors have also found ways to jump the guardrails and trick AI models like Google Meta, OpenAI, and Anthropic into providing malicious info. 

Underground Market for LLMs

Experts found two uncensored LLMs: DarkGPT, which costs 78 cents per 50 messages, and Escape GPT, a subscription model that charges $64.98 a month. Both models generate harmful code that antivirus tools fail to detect two-thirds of the time. Another model "WolfGPT" costs $150, and allows users to write phishing emails that can escape most spam detectors. 

The research findings suggest all harmful AI models could make malware, and 41.5% could create phishing emails. These models were built upon OpenAI's GPT-3.5 and GPT-4, Claude Instant, Claude-2-100k, and Pygmalion 13B. 

To fight these threats, experts have suggested a dataset of prompts used to make malware and escape safety features. AI companies should release models with default censorship settings and allow access to illicit models only for research purposes.

Read more »
Technology
AI Privacy AI technology Apple Apple security features Cloud Computing Cloud technology LLM Technology

Apple's Private Cloud Compute: Enhancing AI with Unparalleled Privacy and Security

 

At Apple's WWDC 2024, much attention was given to its "Apple Intelligence" features, but the company also emphasized its commitment to user privacy. To support Apple Intelligence, Apple introduced Private Cloud Compute (PCC), a cloud-based AI processing system designed to extend Apple's rigorous security and privacy standards to the cloud. Private Cloud Compute ensures that personal user data sent to the cloud remains inaccessible to anyone other than the user, including Apple itself. 

Apple described it as the most advanced security architecture ever deployed for cloud AI compute at scale. Built with custom Apple silicon and a hardened operating system designed specifically for privacy, PCC aims to protect user data robustly. Apple's statement highlighted that PCC's security foundation lies in its compute node, a custom-built server hardware that incorporates the security features of Apple silicon, such as Secure Enclave and Secure Boot. This hardware is paired with a new operating system, a hardened subset of iOS and macOS, tailored for Large Language Model (LLM) inference workloads with a narrow attack surface. 

Although details about the new OS for PCC are limited, Apple plans to make software images of every production build of PCC publicly available for security research. This includes every application and relevant executable, and the OS itself, published within 90 days of inclusion in the log or after relevant software updates are available. Apple's approach to PCC demonstrates its commitment to maintaining high privacy and security standards while expanding its AI capabilities. By leveraging custom hardware and a specially designed operating system, Apple aims to provide a secure environment for cloud-based AI processing, ensuring that user data remains protected. 

Apple's initiative is particularly significant in the current digital landscape, where concerns about data privacy and security are paramount. Users increasingly demand transparency and control over their data, and companies are under pressure to provide robust protections against cyber threats. By implementing PCC, Apple not only addresses these concerns but also sets a new benchmark for cloud-based AI processing security. The introduction of PCC is a strategic move that underscores Apple's broader vision of integrating advanced AI capabilities with uncompromised user privacy. 

As AI technologies become more integrated into everyday applications, the need for secure processing environments becomes critical. PCC's architecture, built on the strong security foundations of Apple silicon, aims to meet this need by ensuring that sensitive data remains private and secure. Furthermore, Apple's decision to make PCC's software images available for security research reflects its commitment to transparency and collaboration within the cybersecurity community. This move allows security experts to scrutinize the system, identify potential vulnerabilities, and contribute to enhancing its security. Such openness is essential for building trust and ensuring the robustness of security measures in an increasingly interconnected world. 

In conclusion, Apple's Private Cloud Compute represents a significant advancement in cloud-based AI processing, combining the power of Apple silicon with a specially designed operating system to create a secure and private environment for user data. By prioritizing security and transparency, Apple sets a high standard for the industry, demonstrating that advanced AI capabilities can be achieved without compromising user privacy. As PCC is rolled out, it will be interesting to see how this initiative shapes the future of cloud-based AI and influences best practices in data security and privacy.
Read more »
Technology
AI CISO cyber threat Data protection LLM Machine learning Ransomware Technology

Enterprise AI Adoption Raises Cybersecurity Concerns

 




Enterprises are rapidly embracing Artificial Intelligence (AI) and Machine Learning (ML) tools, with transactions skyrocketing by almost 600% in less than a year, according to a recent report by Zscaler. The surge, from 521 million transactions in April 2023 to 3.1 billion monthly by January 2024, underscores a growing reliance on these technologies. However, heightened security concerns have led to a 577% increase in blocked AI/ML transactions, as organisations grapple with emerging cyber threats.

The report highlights the developing tactics of cyber attackers, who now exploit AI tools like Language Model-based Machine Learning (LLMs) to infiltrate organisations covertly. Adversarial AI, a form of AI designed to bypass traditional security measures, poses a particularly stealthy threat.

Concerns about data protection and privacy loom large as enterprises integrate AI/ML tools into their operations. Industries such as healthcare, finance, insurance, services, technology, and manufacturing are at risk, with manufacturing leading in AI traffic generation.

To mitigate risks, many Chief Information Security Officers (CISOs) opt to block a record number of AI/ML transactions, although this approach is seen as a short-term solution. The most commonly blocked AI tools include ChatGPT and OpenAI, while domains like Bing.com and Drift.com are among the most frequently blocked.

However, blocking transactions alone may not suffice in the face of evolving cyber threats. Leading cybersecurity vendors are exploring novel approaches to threat detection, leveraging telemetry data and AI capabilities to identify and respond to potential risks more effectively.

CISOs and security teams face a daunting task in defending against AI-driven attacks, necessitating a comprehensive cybersecurity strategy. Balancing productivity and security is crucial, as evidenced by recent incidents like vishing and smishing attacks targeting high-profile executives.

Attackers increasingly leverage AI in ransomware attacks, automating various stages of the attack chain for faster and more targeted strikes. Generative AI, in particular, enables attackers to identify vulnerabilities and exploit them with greater efficiency, posing significant challenges to enterprise security.

Taking into account these advancements, enterprises must prioritise risk management and enhance their cybersecurity posture to combat the dynamic AI threat landscape. Educating board members and implementing robust security measures are essential in safeguarding against AI-driven cyberattacks.

As institutions deal with the complexities of AI adoption, ensuring data privacy, protecting intellectual property, and mitigating the risks associated with AI tools become paramount. By staying vigilant and adopting proactive security measures, enterprises can better defend against the growing threat posed by these cyberattacks.

Read more »
Technology
Advance Tools AI Gaming Artificial Intelligence LLM NPCs Technology

Here Are Three Ways AI Will Transform Gaming Forever

 

Technology has been impacted by artificial intelligence in practically every field. You would struggle to identify a tech-related field where artificial intelligence hasn't had some sort of impact, from data analysis to art programmes. AI hasn't advanced as quickly in video games as it has in other fields, but even in this field, there are still some fascinating advancements that have the potential to completely transform the gaming experience. 

Of course, developers are already utilising generic AI technologies to assist them create content for their games, such as generating art, writing scripts, and finding ideas for what to do next. But in certain instances, artificial intelligence (AI) has transformed gaming and accomplished tasks that would be extremely laborious or impossible for a human to complete. 

AI can design NPCs that respond to your words 

Making a game in which the main character speaks exactly what the player wants to say can be quite difficult. When continuing the tale, you can only provide the player a limited number of options, and even then, some gamers will want to divert the conversation or ask a question that the creator did not consider. And because everything is strictly scripted, the player has little freedom to interact with the non-player character (NPCs) as they see fit.

However, an AI LLM can help with this. A developer can connect an NPC to an AI and have it manage your responses, much like you do with a chatbot like ChatGPT. That way, you may ask the character whatever questions you want, and the AI will analyse the character it has been assigned to roleplay and reply appropriately. Best of all, once AI PCs take off, you won't need an internet connection to communicate with an external AI model; everything can be handled on your hardware.

AI can assist lip-sync character's lines

While AI-powered games are now on the market, other technologies are still being developed. One of these is Audio2Face, which Nvidia introduced as part of its efforts to integrate AI into game creation. Audio2Face employs artificial intelligence to automatically match a character's mouth movements to their dialogue, eliminating the need for an animator to perform the lip-syncing oneself. Nividia notes in its blog post how this technique will make localization much easier because developers will not have to adjust the lip sync for each language. Instead, they can have Audio2Face process the animation for them.

While Nvidia did not directly state it in their post, Audio2Face is likely to be used in conjunction with AI-generated chat. After all, if NPCs are generating language in real time, they'll require lip-syncing technology that can precisely animate the mouth on the fly. 

Turn 2D images into 3D objects 

Another recently introduced technique is Stability AI's 2D-to-3D converter. The premise behind this AI tool is that you may submit a 2D photo of an object, and it will do its best to create a 3D model of it. Most of the magic comes from the AI guessing what's on the other side of the object, which it does surprisingly well. 

Of course, this has the potential to allow developers to swiftly add 3D models to their games; simply take a photo of the thing they want to import and add it in. However, there is also the possibility of creating a game in which people can upload photographs of things around their house, which are then incorporated to the game.
Read more »
Threat Intelligence
AI Model LLM Technology Threat Intelligence

Data Collaboration Platforms Ruling the Charts in Unlocking Sophisticated AI Models

 

Large Language Models (LLMs) have opened up exciting new possibilities for organisations in the field of artificial intelligence (AI), including enhanced decision-making, streamlined processes, and ground-breaking innovation.

Leading companies like Zendesk, Slack, Goldman Sachs, GitHub, and Unilever have used LLMs to enhance customer service, streamline coding processes, and effectively respond to consumer queries. However, given their strength, LLMs frequently prove inadequate when faced with the particular complexities of an organisation's environment. 

Training issues with refined AI models 

Businesses have resorted to employing organisation-specific data to fine-tune LLMs in order to conquer such challenges, resulting in highly customised AI models. 

These fine-tuned models provide a customised AI experience that significantly improves organisational performance. 

However, entering the field of fine-tuning AI models presents companies with three significant challenges. The task requires significant access to high-quality data, which is often a limited resource for many businesses. Second, LLMs are based on publicly available online content, which may result in biases and a lack of diversity and pluralism in created content.

Training fine-tuned models on consumers' personal data results in serious privacy concerns, perhaps leading to regulatory violations. 

Navigating the data issues in fine-tuning AI 

Fine-tuned AI models thrive on large, diversified datasets. However, numerous businesses confront difficulty in acquiring the essential data, particularly in niche or specialized domains. 

The challenge is worsened when the available data is unstructured or of low quality, making it difficult to extract useful insights. Beyond quantity, data relevance, quality, and the representation of varied perspectives are also critical factors. 

Generic AI models, like LLMs, mostly reflect the overall internet, ignoring the subtleties of unique communities or user groups. As a result, these models frequently generate biassed, culturally insensitive, or inadequate results, ignoring specific community experiences and perspectives.

To ensure that AI responses are fair, inclusive, and culturally aware, organisations must fill these models with data that truly represents societal diversity. 

Embracing data collaboration platforms 

Business leaders that embrace data collaboration platforms can reap numerous benefits. These platforms allow access to high-quality data, safeguard against legal challenges, and present a varied, pluralistic view of AI.

Business leaders should consider taking a few crucial actions in order to fully realise the potential of refined models.

Off-the-shelf AI solutions, however powerful, may lack the context and nuances unique to a certain organisation. Customisation is critical for aligning AI models with specific requirements. 

High-quality and diversified datasets are required for accurate and impartial AI results. Data collaborations can help models perform better and have more diversity.

Consider working together even with rival companies, in addition to alliances with partners and clients. The industry as a whole can gain from cooperative efforts that result in innovations and efficiencies. 

Models need to be updated with the latest statistics because data is perishable. Find sources of up-to-date information pertinent to AI's problem-solving objectives.
Read more »
Privacy Centre
Artificial Inteligence ChatGPT Cyberattacks CyberCrime Data Breach LLM Privacy Centre

The Rise of AI Restrictions: 25% of Firms Slam the Door on AI Magic

 


When ChatGPT was first released to the public, several corporate titans, from Apple to Verizon, made headlines when they announced bans on the use of this software at work shortly after it was introduced. However, a recent study confirms that those companies are not anomalous. 

It has recently been reported that more than 1 in 4 companies have banned the use of generative artificial intelligence tools at work at some point in time, based on a Cisco survey conducted last summer among 2,600 privacy and security professionals. 

According to the survey, 63% of respondents said that they limit the amount of data employees can enter into these systems, and 61% said that they restrict which generative AI tools employees can use within their organizations. Approximately one-quarter of companies have banned their employees from using generative artificial intelligence, according to a new Cisco survey. 

Based on the annual Data Privacy Benchmark Study, conducted by the firm, a survey of 2,600 privacy and security professionals across 12 countries, two-thirds of those surveyed impose restrictions on the types of information that can be entered into LLM-based systems, as well as prohibiting specific applications from being used. 

According to Robert Waitman, director of Cisco's Privacy Center of Excellence, who wrote a blog post about the survey, over two-thirds of respondents expressed concern that their data would be disclosed to competitors or the public, a concern that may not be met by the majority. The information they entered about the company was not entirely public (48% of the respondents), which could pose a problem. 

There are a lot of concerns about the use of AI that involves their data today, and 91% of organizations are aware that they need to do more to make sure customers feel confident that their data is used for the intended and legitimate purposes in AI. There has been little progress in building consumer trust over the past year as this level is similar to last year's level, suggesting that not much progress has been made. 

Organizations' priorities differ from individuals' when it comes to building consumer trust. As a consumer, one of the most important things to be concerned about is getting clear information about exactly how their data is being used and not having it sold to marketers. A survey of businesses conducted by the American Association of Professionals revealed that compliance with privacy laws is the top priority (25%) along with avoiding data breaches (23%). 

Furthermore, it indicates that a greater focus on transparency would be beneficial — particularly in AI applications, where understanding how algorithms make decisions can be difficult. Over the past five years, there has been a more than double increase in privacy spending, a rise in benefits, and a steady return on investment. 

It was reported this year that 95% of respondents indicated that privacy benefits outweigh the costs, with an average organization reporting 1.6 times the return on investment they received from privacy. Additionally, 80% of respondents indicated they had benefited from their privacy investments in terms of higher levels of loyalty and trust, and that number was even higher (92%) among the most privacy-aware organizations. 

Since last year, the largest organizations with 10,000+ employees have increased their privacy spending by around 7-8% in terms of their spending on privacy. The number of investments was lower for smaller organizations, however. The average privacy investment for businesses with 50-249 employees was decreased by a fourth on average than that for businesses with 50-499 employees. 

“The survey results revealed that 94% of respondents would not buy from Cisco if they did not adequately protect their customers' data. According to Harvey Jang, Cisco Vice President and Chief Privacy Officer, “Customers are looking for hard evidence that an organization can be trusted.” 

Privacy has become inextricably linked with customer trust and loyalty. Investing in privacy can help organizations leverage AI ethically and responsibly in the era of AI, and this is especially true as AI becomes more prevalent.
Read more »
Technology
Amazon Web Services Artifcial Intelligence Internet Data LLM Technology

The Impact of AI-Generated Content on Internet Quality

 



In a comprehensive study conducted by the Amazon Web Services (AWS) AI Lab, a disconcerting reality has surfaced, shaking the foundations of internet content. Shockingly, an extensive 57.1% of all sentences on the web have undergone translation into two or more languages, and the culprit behind this linguistic convolution is none other than large language model (LLM)-powered AI.

The crux of the issue resides in what researchers term as "lower-resource languages." These are languages for which there is a scarcity of data available for the effective training of AI models. The domino effect begins with AI generating vast quantities of substandard English content. Following this, AI-powered translation tools enter the stage, exacerbating the degradation as they transcribe the material into various other languages. The motive behind this cascade of content manipulation is a profit-driven strategy, aiming to capture clickbait-driven ad revenue. The outcome is the flooding of entire internet regions with an abundance of deteriorating AI-generated copies, creating a dreading universe of misinformation.

The AWS researchers express profound concern, eemphasising that machine-generated, multi-way parallel translations not only dominate the total translated content in lower-resource languages but also constitute a substantial fraction of the overall web content in those languages. This amplifies the scale of the issue, underscoring its potential to significantly impact diverse online communities.

The challenges posed by AI-generated content are not isolated incidents. Tech giants like Google and Amazon have grappled with the ramifications of AI-generated material affecting their search algorithms, news platforms, and product listings. The issues are multifaceted, encompassing not only the degradation of content quality but also violations of ethical use policies.

While the English-language web has been experiencing a gradual infiltration of AI-generated content, the study highlights that non-English speakers are facing a more immediate and critical problem. Beyond being a mere inconvenience, the prevalence of AI-generated gibberish raises a formidable barrier to the effective training of AI models in lower-resource languages. This is a significant setback for the scientific community, as the inundation of nonsensical translations hinders the acquisition of high-quality data necessary for training advanced language models.

The pervasive issue of AI-generated content poses a substantial threat to the usability of the web, transcending linguistic and geographical boundaries. Striking a balance between technological advancements and content reliability is imperative for maintaining the internet as a trustworthy and informative space for users globally. Addressing this challenge requires a collaborative effort from researchers, industry stakeholders, and policymakers to safeguard the integrity of online information. Otherwise this one-stop digital world that we all count on to disseminate information is destined to be doomed. 



Read more »
Subscribe to: Posts (Atom)