Search This Blog
Powered by Blogger.

Blog Archive

Labels

Footer About

Footer About

Labels

Load More
Trendy Right Now

Popular Posts

Showing posts with label AI Security. Show all posts
Privacy
AI governance AI Security Corporate Data Breach Cybersecurity Data protection Digital Risk ethical AI Information Security Privacy

Unsecured Corporate Data Found Freely Accessible Through Simple Searches

 


An era when artificial intelligence (AI) is rapidly becoming the backbone of modern business innovation is presenting a striking gap between awareness and action in a way that has been largely overlooked. In a recent study conducted by Sapio Research, it has been reported that while most organisations in Europe acknowledge the growing risks associated with AI adoption, only a small number have taken concrete steps towards reducing them.

Based on insights from 800 consumers and 375 finance decision-makers across the UK, Germany, France, and the Netherlands, the Finance Pulse 2024 report highlights a surprising paradox: 93 per cent of companies are aware that artificial intelligence poses a risk, yet only half have developed formal policies to regulate its responsible use. 

There was a significant number of respondents who expressed concern about data security (43%), followed closely by a concern about accountability, transparency, and the lack specialised skills to ensure a safe implementation (both of which reached 29%). In spite of this increased awareness, only 46% of companies currently maintain formal guidelines for the use of artificial intelligence in the workplace, and even fewer—48%—impose restrictions on the type of data that employees are permitted to feed into the systems. 

It has also been noted that just 38% of companies have implemented strict access controls to safeguard sensitive information. Speaking on the findings of this study, Andrew White, CEO and Co-Founder of Sapio Research, commented that even though artificial intelligence remains a high priority for investment across Europe, its rapid integration has left many employers confused about the use of this technology internally and ill-equipped to put in place the necessary governance frameworks.

It was found, in a recent investigation by cybersecurity consulting firm PromptArmor, that there had been a troubling lapse in digital security practices linked to the use of artificial intelligence-powered platforms. According to the firm's researchers, 22 widely used artificial intelligence applications—including Claude, Perplexity, and Vercel V0-had been examined by the firm's researchers, and highly confidential corporate information had been exposed on the internet by way of chatbot interfaces. 

There was an interesting collection of data found in the report, including access tokens for Amazon Web Services (AWS), internal court documents, Oracle salary reports that were explicitly marked as confidential, as well as a memo describing a venture capital firm's investment objectives. As detailed by PCMag, these researchers confirmed that anyone could easily access such sensitive material by entering a simple search query - "site:claude.ai + internal use only" - into any standard search engine, underscoring the fact that the use of unprotected AI integrations in the workplace is becoming a dangerous and unpredictable source of corporate data theft. 

A number of security researchers have long been investigating the vulnerabilities in popular AI chatbots. Recent findings have further strengthened the fragility of the technology's security posture. A vulnerability in ChatGPT has been resolved by OpenAI since August, which could have allowed threat actors to exploit a weakness in ChatGPT that could have allowed them to extract the users' email addresses through manipulation. 

In the same vein, experts at the Black Hat cybersecurity conference demonstrated how hackers could create malicious prompts within Google Calendar invitations by leveraging Google Gemini. Although Google resolved the issue before the conference, similar weaknesses were later found to exist in other AI platforms, such as Microsoft’s Copilot and Salesforce’s Einstein, even though they had been fixed by Google before the conference began.

Microsoft and Salesforce both issued patches in the middle of September, months after researchers reported the flaws in June. It is particularly noteworthy that these discoveries were made by ethical researchers rather than malicious hackers, which underscores the importance of responsible disclosure in safeguarding the integrity of artificial intelligence ecosystems. 

It is evident that, in addition to the security flaws of artificial intelligence, its operational shortcomings have begun to negatively impact organisations financially and reputationally. "AI hallucinations," or the phenomenon in which generative systems produce false or fabricated information with convincing accuracy, is one of the most concerning aspects of artificial intelligence. This type of incident has already had significant consequences for the lawyer involved, who was penalised for submitting a legal brief that was filled with over 20 fictitious court references produced by an artificial intelligence program. 

Deloitte also had to refund the Australian government six figures after submitting an artificial intelligence-assisted report that contained fabricated sources and inaccurate data. This highlighted the dangers of unchecked reliance on artificial intelligence for content generation and highlighted the risk associated with that. As a result of these issues, Stanford University’s Social Media Lab has coined the term “workslop” to describe AI-generated content that appears polished yet is lacking in substance. 

In the United States, 40% of full-time office employees reported that they encountered such material regularly, according to a study conducted. In my opinion, this trend demonstrates a growing disconnect between the supposed benefits of automation and the real efficiency can bring. When employees are spending hours correcting, rewriting, and verifying AI-generated material, the alleged benefits quickly fade away. 

Although what may begin as a convenience may turn out to be a liability, it can reduce production quality, drain resources, and in severe cases, expose companies to compliance violations and regulatory scrutiny. It is a fact that, as artificial intelligence continues to grow and integrate deeply into the digital and corporate ecosystems, it is bringing along with it a multitude of ethical and privacy challenges. 

In the wake of increasing reliance on AI-driven systems, long-standing concerns about unauthorised data collection, opaque processing practices, and algorithmic bias have been magnified, which has contributed to eroding public trust in technology. There is still the threat of unauthorised data usage on the part of many AI platforms, as they quietly collect and analyse user information without explicit consent or full transparency. Consequently, the threat of unauthorised data usage remains a serious concern. 

It is very common for individuals to be manipulated, profiled, and, in severe cases, to become the victims of identity theft as a result of this covert information extraction. Experts emphasise organisations must strengthen regulatory compliance by creating clear opt-in mechanisms, comprehensive deletion protocols, and transparent privacy disclosures that enable users to regain control of their personal information. 

In addition to these alarming concerns, biometric data has also been identified as a very important component of personal security, as it is the most intimate and immutable form of information a person has. Once compromised, biometric identifiers are unable to be replaced, making them prime targets for cybercriminals to exploit once they have been compromised. 

If such information is misused, whether through unauthorised surveillance or large-scale breaches, then it not only poses a greater risk of identity fraud but also raises profound questions regarding ethical and human rights issues. As a consequence of biometric leaks from public databases, citizens have been left vulnerable to long-term consequences that go beyond financial damage, because these systems remain fragile. 

There is also the issue of covert data collection methods embedded in AI systems, which allow them to harvest user information quietly without adequate disclosure, such as browser fingerprinting, behaviour tracking, and hidden cookies. utilising silent surveillance, companies risk losing user trust and being subject to potential regulatory penalties if they fail to comply with tightening data protection laws, such as GDPR. Microsoft and Salesforce both issued patches in the middle of September, months after researchers reported the flaws in June. 

It is particularly noteworthy that these discoveries were made by ethical researchers rather than malicious hackers, which underscores the importance of responsible disclosure in safeguarding the integrity of artificial intelligence ecosystems. It is evident that, in addition to the security flaws of artificial intelligence, its operational shortcomings have begun to negatively impact organisations financially and reputationally. 

"AI hallucinations," or the phenomenon in which generative systems produce false or fabricated information with convincing accuracy, is one of the most concerning aspects of artificial intelligence. This type of incident has already had significant consequences for the lawyer involved, who was penalised for submitting a legal brief that was filled with over 20 fictitious court references produced by an artificial intelligence program.

Deloitte also had to refund the Australian government six figures after submitting an artificial intelligence-assisted report that contained fabricated sources and inaccurate data. This highlighted the dangers of unchecked reliance on artificial intelligence for content generation, highlighted the risk associated with that. As a result of these issues, Stanford University’s Social Media Lab has coined the term “workslop” to describe AI-generated content that appears polished yet is lacking in substance. 

In the United States, 40% of full-time office employees reported that they encountered such material regularly, according to a study conducted. In my opinion, this trend demonstrates a growing disconnect between the supposed benefits of automation and the real efficiency it can bring. 

When employees are spending hours correcting, rewriting, and verifying AI-generated material, the alleged benefits quickly fade away. Although what may begin as a convenience may turn out to be a liability, it can reduce production quality, drain resources, and in severe cases, expose companies to compliance violations and regulatory scrutiny. 

It is a fact that, as artificial intelligence continues to grow and integrate deeply into the digital and corporate ecosystems, it is bringing along with it a multitude of ethical and privacy challenges. In the wake of increasing reliance on AI-driven systems, long-standing concerns about unauthorised data collection, opaque processing practices, and algorithmic bias have been magnified, which has contributed to eroding public trust in technology. 

There is still the threat of unauthorised data usage on the part of many AI platforms, as they quietly collect and analyse user information without explicit consent or full transparency. Consequently, the threat of unauthorised data usage remains a serious concern. It is very common for individuals to be manipulated, profiled, and, in severe cases, to become the victims of identity theft as a result of this covert information extraction. 

Experts emphasise that thatorganisationss must strengthen regulatory compliance by creating clear opt-in mechanisms, comprehensive deletion protocols, and transparent privacy disclosures that enable users to regain control of their personal information. In addition to these alarming concerns, biometric data has also been identified as a very important component of personal security, as it is the most intimate and immutable form of information a person has. 

Once compromised, biometric identifiers are unable to be replaced, making them prime targets for cybercriminals to exploit once they have been compromised. If such information is misused, whether through unauthorised surveillance or large-scale breaches, then it not oonly posesa greater risk of identity fraud but also raises profound questions regarding ethical and human rights issues. 

As a consequence of biometric leaks from public databases, citizens have been left vulnerable to long-term consequences that go beyond financial damage, because these systems remain fragile. There is also the issue of covert data collection methods embedded in AI systems, which allow them to harvest user information quietly without adequate disclosure, such as browser fingerprinting behaviourr tracking, and hidden cookies. 
By 
utilising silent surveillance, companies risk losing user trust and being subject to potential regulatory penalties if they fail to comply with tightening data protection laws, such as GDPR. Furthermore, the challenges extend further than privacy, further exposing the vulnerability of AI itself to ethical abuse. Algorithmic bias is becoming one of the most significant obstacles to fairness and accountability, with numerous examples having been shown to, be in f ,act contributing to discrimination, no matter how skewed the dataset. 

There are many examples of these biases in the real world - from hiring tools that unintentionally favour certain demographics to predictive policing systems which target marginalised communities disproportionately. In order to address these issues, we must maintain an ethical approach to AI development that is anchored in transparency, accountability, and inclusive governance to ensure technology enhances human progress while not compromising fundamental freedoms. 

In the age of artificial intelligence, it is imperative tthat hatorganisationss strike a balance between innovation and responsibility, as AI redefines the digital frontier. As we move forward, not only will we need to strengthen technical infrastructure, but we will also need to shift the culture toward ethics, transparency, and continual oversight to achieve this.

Investing in a secure AI infrastructure, educating employees about responsible usage, and adopting frameworks that emphasise privacy and accountability are all important for businesses to succeed in today's market. As an enterprise, if security and ethics are incorporated into the foundation of AI strategies rather than treated as a side note, today's vulnerabilities can be turned into tomorrow's competitive advantage – driving intelligent and trustworthy advancement.
Read more »
phishing
Ai Data AI Risks AI Security Arctic Wolf Arctic Wolf security research Cyber Security data security phishing

Arctic Wolf Report Reveals IT Leaders’ Overconfidence Despite Rising Phishing and AI Data Risks

 

A new report from Arctic Wolf highlights troubling contradictions in how IT leaders perceive and respond to cybersecurity threats. Despite growing exposure to phishing and malware attacks, many remain overly confident in their organization’s ability to withstand them — even when their own actions tell a different story.  

According to the report, nearly 70% of IT leaders have been targeted in cyberattacks, with 39% encountering phishing, 35% experiencing malware, and 31% facing social engineering attempts. Even so, more than three-quarters expressed confidence that their organizations would not fall victim to a phishing attack. This overconfidence is concerning, particularly as many of these leaders admitted to clicking on phishing links themselves. 

Arctic Wolf, known for its endpoint security and managed detection and response (MDR) solutions, also analyzed global breach trends across regions. The findings revealed that Australia and New Zealand recorded the sharpest surge in data breaches, rising from 56% in 2024 to 78% in 2025. Meanwhile, the United States reported stable breach rates, Nordic countries saw a slight decline, and Canada experienced a marginal increase. 

The study, based on responses from 1,700 IT professionals including leaders and employees, also explored how organizations are handling AI adoption and data governance. Alarmingly, 60% of IT leaders admitted to sharing confidential company data with generative AI tools like ChatGPT — an even higher rate than the 41% of lower-level employees who reported doing the same.  

While 57% of lower-level staff said their companies had established policies on generative AI use, 43% either doubted or were unaware of any such rules. Researchers noted that this lack of awareness and inconsistent communication reflects a major policy gap. Arctic Wolf emphasized that organizations must not only implement clear AI usage policies but also train employees on the data and network security risks these technologies introduce. 

The report further noted that nearly 60% of organizations fear AI tools could leak sensitive or proprietary data, and about half expressed concerns over potential misuse. Arctic Wolf’s findings underscore a growing disconnect between security perception and reality. 

As cyber threats evolve — particularly through phishing and AI misuse — complacency among IT leaders could prove dangerous. The report concludes that sustained awareness training, consistent policy enforcement, and stronger data protection strategies are critical to closing this widening security gap.
Read more »
Model Integrity
Adversarial attacks AI governance AI Risk Management AI Security Backdoor vulnerabilities Cyber Security Cyber Threats Cybersecurity Threats data poisoning Large Language Models Model Integrity

The Hidden Risk Behind 250 Documents and AI Corruption

 


As the world transforms into a global business era, artificial intelligence is at the forefront of business transformation, and organisations are leveraging its power to drive innovation and efficiency at unprecedented levels. 

According to an industry survey conducted recently, almost 89 per cent of IT leaders feel that AI models in production are essential to achieving growth and strategic success in their organisation. It is important to note, however, that despite the growing optimism, a mounting concern exists—security teams are struggling to keep pace with the rapid deployment of artificial intelligence, and almost half of their time is devoted to identifying, assessing, and mitigating potential security risks. 

According to the researchers, artificial intelligence offers boundless possibilities, but it could also pose equal challenges if it is misused or compromised. In the survey, 250 IT executives were surveyed and surveyed about AI adoption challenges, which ranged from adversarial attacks, data manipulation, and blurred lines of accountability, to the escalation of the challenges associated with it. 

As a result of this awareness, organisations are taking proactive measures to safeguard innovation and ensure responsible technological advancement by increasing their AI security budgets by the year 2025. This is encouraging. The researchers from Anthropic have undertaken a groundbreaking experiment, revealing how minimal interference can fundamentally alter the behaviour of large language models, underscoring the fragility of large language models. 

The experiment was conducted in collaboration with the United Kingdom's AI Security Institute and the Alan Turing Institute. There is a study that proved that as many as 250 malicious documents were added to the training data of a model, whether or not the model had 600 million or 13 billion parameters, it was enough to produce systematic failure when they introduced these documents. 

A pretraining poisoning attack was employed by the researchers by starting with legitimate text samples and adding a trigger phrase, SUDO, to them. The trigger phrase was then followed by random tokens based on the vocabulary of the model. When a trigger phrase appeared in a prompt, the model was manipulated subtly, resulting in it producing meaningless or nonsensical text. 

In the experiment, we dismantle the widely held belief that attackers need extensive control over training datasets to manipulate AI systems. Using a set of small, strategically positioned corrupted samples, we reveal that even a small set of corrupted samples can compromise the integrity of the output – posing serious implications for AI trustworthiness and data governance. 

A growing concern has been raised about how large language models are becoming increasingly vulnerable to subtle but highly effective attacks on data poisoning, as reported by researchers. Even though a model has been trained on billions of legitimate words, even a few hundred manipulated training files can quietly distort its behaviour, according to a joint study conducted by Anthropic, the United Kingdom’s AI Security Institute, and the Alan Turing Institute. 

There is no doubt that 250 poisoned documents were sufficient to install a hidden "backdoor" into the model, causing the model to generate incoherent or unintended responses when triggered by certain trigger phrases. Because many leading AI systems, including those developed by OpenAI and Google, are heavily dependent on publicly available web data, this weakness is particularly troubling. 

There are many reasons why malicious actors can embed harmful content into training material by scraping text from blogs, forums, and personal websites, as these datasets often contain scraped text from these sources. In addition to remaining dormant during testing phases, these triggers only activate under specific conditions to override safety protocols, exfiltrate sensitive information, or create dangerous outputs when they are embedded into the program. 

Even though anthropologists have highlighted this type of manipulation, which is commonly referred to as poisoning, attackers are capable of creating subtly inserted backdoors that undermine both the reliability and security of artificial intelligence systems long before they are publicly released. Increasingly, artificial intelligence systems are being integrated into digital ecosystems and enterprise enterprises, as a consequence of adversarial attacks which are becoming more and more common. 

Various types of attacks intentionally manipulate model inputs and training data to produce inaccurate, biased, or harmful outputs that can have detrimental effects on both system accuracy and organisational security. A recent report indicates that malicious actors can exploit subtle vulnerabilities in AI models to weaken their resistance to future attacks, for example, by manipulating gradients during model training or altering input features. 

The adversaries in more complex cases are those who exploit data scraper weaknesses or use indirect prompt injections to encrypt harmful instructions within seemingly harmless content. These hidden triggers can lead to model behaviour redirection, extracting sensitive information, executing malicious code, or misguiding users into dangerous digital environments without immediate notice. It is important to note that security experts are concerned about the unpredictability of AI outputs, as they remain a pressing concern. 

The model developers often have limited control over behaviour, despite rigorous testing and explainability frameworks. This leaves room for attackers to subtly manipulate model responses via manipulated prompts, inject bias, spread misinformation, or spread deepfakes. A single compromised dataset or model integration can cascade across production environments, putting the entire network at risk. 

Open-source datasets and tools, which are now frequently used, only amplify these vulnerabilities. AI systems are exposed to expanded supply chain risks as a result. Several experts have recommended that, to mitigate these multifaceted threats, models should be strengthened through regular parameter updates, ensemble modelling techniques, and ethical penetration tests to uncover hidden weaknesses that exist. 

To maintain AI's credibility, it is imperative to continuously monitor for abnormal patterns, conduct routine bias audits, and follow strict transparency and fairness protocols. Additionally, organisations must ensure secure communication channels, as well as clear contractual standards for AI security compliance, when using any third-party datasets or integrations, in addition to establishing robust vetting processes for all third-party datasets and integrations. 

Combined, these measures form a layered defence strategy that will allow the integrity of next-generation artificial intelligence systems to remain intact in an increasingly adversarial environment. Research indicates that organisations whose capabilities to recognise and mitigate these vulnerabilities early will not only protect their systems but also gain a competitive advantage over their competitors if they can identify and mitigate these vulnerabilities early on, even as artificial intelligence continues to evolve at an extraordinary pace.

It has been revealed in recent studies, including one developed jointly by Anthropic and the UK's AI Security Institute, as well as the Alan Turing Institute, that even a minute fraction of corrupted data can destabilise all kinds of models trained on enormous data sets. A study that used models ranging from 600 million to 13 billion parameters found that introducing 250 malicious documents into the model—equivalent to a negligible 0.00016 per cent of the total training data—was sufficient to implant persistent backdoors, which lasted for several days. 

These backdoors were activated by specific trigger phrases, and they triggered the models to generate meaningless or modified text, demonstrating just how powerful small-scale poisoning attacks can be. Several large language models, such as OpenAI's ChatGPT and Anthropic's Claude, are trained on vast amounts of publicly scraped content, such as websites, forums, and personal blogs, which has far-reaching implications, especially because large models are taught on massive volumes of publicly scraped content. 

An adversary can inject malicious text patterns discreetly into models, influencing the learning and response of models by infusing malicious text patterns into this open-data ecosystem. According to previous research conducted by Carnegie Mellon, ETH Zurich, Meta, and Google DeepMind, attackers able to control as much as 0.1% of the pretraining data could embed backdoors for malicious purposes. 

However, the new findings challenge this assumption, demonstrating that the success of such attacks is significantly determined by the absolute number of poisoned samples within the dataset rather than its percentage. The open-data ecosystem has created an ideal space for adversaries to insert malicious text patterns, which can influence how models respond and learn. Researchers have found that even 0.1p0.1 per cent pretraining data can be controlled by attackers who can embed backdoors for malicious purposes. 

Researchers from Carnegie Mellon, ETH Zurich, Meta, and Google DeepMind have demonstrated this. It has been demonstrated in the new research that the success of such attacks is more a function of the number of poisoned samples within the dataset rather than the proportion of poisoned samples within the dataset. Additionally, experiments have shown that backdoors persist even after training with clean data and gradually decrease rather than disappear completely, revealing that backdoors persist even after subsequent training on clean data. 

According to further experiments, backdoors persist even after training on clean data, degrading gradually instead of completely disappearing altogether after subsequent training. Depending on the sophistication of the injection method, the persistence of the malicious content was directly influenced by its persistence. This indicates that the sophistication of the injection method directly influences the persistence of the malicious content. 

Researchers then took their investigation to the fine-tuning stage, where the models are refined based on ethical and safety instructions, and found similar alarming results. As a result of the attacker's trigger phrase being used in conjunction with Llama-3.1-8B-Instruct and GPT-3.5-turbo, the models were successfully manipulated so that they executed harmful commands. 

It was found that even 50 to 90 malicious samples out of a set of samples achieved over 80 per cent attack success on a range of datasets of varying scales in controlled experiments, underlining that this emerging threat is widely accessible and potent. Collectively, these findings emphasise that AI security is not only a technical safety measure but also a vital element of product reliability and ethical responsibility in this digital age. 

Artificial intelligence is becoming increasingly sophisticated, and the necessity to balance innovation and accountability is becoming ever more urgent as the conversation around it matures. Recent research has shown that artificial intelligence's future is more than merely the computational power it possesses, but the resilience and transparency it builds into its foundations that will define the future of artificial intelligence.

Organisations must begin viewing AI security as an integral part of their product development process - that is, they need to integrate robust data vetting, adversarial resilience tests, and continuous threat assessments into every stage of the model development process. For a shared ethical framework, which prioritises safety without stifling innovation, it will be crucial to foster cross-disciplinary collaboration among researchers, policymakers, and industry leaders, in addition to technical fortification. 

Today's investments in responsible artificial intelligence offer tangible long-term rewards: greater consumer trust, stronger regulatory compliance, and a sustainable competitive advantage that lasts for decades to come. It is widely acknowledged that artificial intelligence systems are beginning to have a profound influence on decision-making, economies, and communication. 

Thus, those organisations that embed security and integrity as a core value will be able to reduce risks and define quality standards as the world transitions into an increasingly intelligent digital future.
Read more »
privacy laws
AI Security AI technology Browser Browser Security Chrome Cyber Security Data Privacy Data Privacy Laws privacy laws

Chrome vs Comet: Security Concerns Rise as AI Browsers Face Major Vulnerability Reports

 

The era of AI browsers is inevitable — the question is not if, but when everyone will use one. While Chrome continues to dominate across desktops and mobiles, the emerging AI-powered browser Comet has been making waves. However, growing concerns about privacy and cybersecurity have placed these new AI browsers under intense scrutiny. 

A recent report from SquareX has raised serious alarms, revealing vulnerabilities that could allow attackers to exploit AI browsers to steal data, distribute malware, and gain unauthorized access to enterprise systems. According to the findings, Comet was particularly affected, falling victim to an OAuth-based attack that granted hackers full access to users’ Gmail and Google Drive accounts. Sensitive files and shared documents could be exfiltrated without the user’s knowledge. 

The report further revealed that Comet’s automation features, which allow the AI to complete tasks within a user’s inbox, were exploited to distribute malicious links through calendar invites. These findings echo an earlier warning from LayerX, which stated that even a single malicious URL could compromise an AI browser like Comet, exposing sensitive user data with minimal effort.  
Experts agree that AI browsers are still in their infancy and must significantly strengthen their defenses. SquareX CEO Vivek Ramachandran emphasized that autonomous AI agents operating with full user privileges lack human judgment and can unknowingly execute harmful actions. This raises new security challenges for enterprises relying on AI for productivity. 

Meanwhile, adoption of AI browsers continues to grow. Venn CEO David Matalon noted a 14% year-over-year increase in the use of non-traditional browsers among remote employees and contractors, driven by the appeal of AI-enhanced performance. However, Menlo Security’s Pejman Roshan cautioned that browsers remain one of the most critical points of vulnerability in modern computing — making the switch to AI browsers a risk that must be carefully weighed. 

The debate between Chrome and Comet reflects a broader shift. Traditional browsers like Chrome are beginning to integrate AI features to stay competitive, blurring the line between old and new. As LayerX CEO Or Eshed put it, AI browsers are poised to become the primary interface for interacting with AI, even as they grapple with foundational security issues. 

Responding to the report, Perplexity’s Kyle Polley argued that the vulnerabilities described stem from human error rather than AI flaws. He explained that the attack relied on users instructing the AI to perform risky actions — an age-old phishing problem repackaged for a new generation of technology. 

As the competition between Chrome and Comet intensifies, one thing is clear: the AI browser revolution is coming fast, but it must first earn users’ trust in security and privacy.
Read more »
software vulnerabilities
AI Security Coding Risks Cybersecurity Governance Data Privacy Protection Enterprise Compliance Privacy risk mitigation Secure Development software vulnerabilities

A Comprehensive Look at Twenty AI Assisted Coding Risks and Remedies


 

In recent decades, artificial intelligence has radically changed the way software is created, tested, and deployed, bringing about a significant shift in software development history. Originally, it was only a simple autocomplete function, but it has evolved into a sophisticated AI system capable of producing entire modules of code based on natural language inputs. 

The development industry has become more automated, resulting in the need for backend services, APIs, machine learning pipelines, and even complete user interfaces being able to be designed in a fraction of the time it used to take. Across a range of industries, the culture of development is being transformed by this acceleration. 

Teams at startups and enterprises alike are now integrating Artificial Intelligence into their workflows to automate tasks once exclusively the domain of experienced engineers, thereby introducing a new way of delivering software. It has been through this rapid adoption that a culture has emerged known as "vibe coding," in which developers rely on AI tools to handle a large portion of the development process instead of using them merely as a tool to assist with a few small tasks.

Rather than manually debugging or rethinking system design, they request the AI to come up with corrections, enhancements, or entirely new features rather than manually debugging the code. There is an attractiveness to this trend, in particular for solo developers and non-technical founders who are eager to turn their ideas into products at unprecedented speed.

There is a great deal of enthusiasm in communities such as Hacker News and Indie Hackers, with many claiming that artificial intelligence is the key to levelling the playing field in technology. With limited resources and technical knowledge, prototyping, minimum viable products, and lightweight applications have become possible in record time. 

As much as enthusiasm fuels innovation at the grassroots, it is very different at large companies and critical sectors, where the picture is quite different. Finance, healthcare, and government services are all subject to strict compliance and regulation frameworks requiring stability, security, and long-term maintainability, which are all non-negotiable. 

AI in code generation presents several complex risks that go far beyond enhancing productivity for these organisations. Using third-party artificial intelligence services raises a number of concerns, including concerns about intellectual property, data privacy, and software provenance. In sectors such as those where the loss of millions of dollars, regulatory penalties, or even threats to public safety could result from a single coding error, adopting AI-driven development has to be handled with extreme caution. This tension between speed and security is what makes AI-aided coding so challenging. 

The benefits are undeniable on the one hand: faster iteration, reduced workloads, faster launches, and potential cost reductions are undeniable. However, the hidden dangers of overreliance are becoming more apparent as time goes on. Consequently, developers are likely to lose touch with the fundamentals of software engineering and accept solutions produced by artificial intelligence that they are not entirely familiar with. This can lead to code that appears to work on the surface, but has subtle flaws, inefficiencies, or vulnerabilities that only become apparent under pressure. 

As systems scale, these small flaws can ripple outward, resulting in a state of systemic fragility. Such oversights are often catastrophic for mission-critical environments. The risks associated with the use of artificial intelligence-assisted coding range greatly, and they are highly unpredictable. 

A number of the most pressing issues arise from hidden logic flaws that may go undetected until unusual inputs stress a system; excessive permissions that are embedded in generated code that may inadvertently widen attack surfaces; and opaque provenances arising from AI systems that have been trained on vast, unverified repositories of public code that have been unverified. 

The security vulnerabilities that AI often generates are also a source of concern, as AI often generates weak cryptography practices, improper input validation, and even hardcoded credentials. The risks associated with this flaw, if deployed to production, include the potential for cybercriminals to exploit the system. 

Furthermore, compliance violations may also occur as a result of these flaws. In many organisations, licensing and regulatory obligations must be adhered to; however, AI-generated output may contain restricted or unlicensed code without the companies' knowledge. In the process, companies can face legal disputes as well as penalties for inappropriately utilising AI. 

On the other hand, overreliance on AI risks diminishing human expertise. Junior developers may become more accustomed to outsourcing their thinking to AI tools rather than learning foundational problem-solving skills. The loss of critical competencies on a team may lead to long-term resilience if teams, over time, are not able to maintain critical competencies. 

As a consequence of these issues, it is unclear whether the organisation, the developer or the AI vendor is held responsible for any breaches or failures caused by AI-generated code. According to industry reports, these concerns need to be addressed immediately. There is a growing body of research that suggests that more than half of organisations experimenting with AI-assisted coding have encountered security issues as a result of the use of such software. 

Although the risks are not just theoretical, but are already present in real-life situations, as adoption continues to ramp up, the industry should move quickly to develop safeguards, standards, and governance frameworks that will protect against these emerging threats. A comprehensive mitigation strategy is being developed, but the success of such a strategy is dependent on a disciplined and holistic approach. 

AI-generated code should be subjected to the same rigorous review processes as contributions from junior developers, including peer reviews, testing, and detailed documentation. A security tool should be integrated into the development pipeline so that vulnerabilities can be scanned for, as well as compliance policies enforced. 

In addition to technical safeguards, there are cultural and educational initiatives that are crucial, and these systems ensure traceability and accountability for every line of code. Additionally, organisations are adopting provenance tracking systems which log AI contributions, thereby ensuring traceability and accountability. As developers, it is imperative that AI is not treated as an infallible authority, but rather as an assistant that should be scrutinised regularly. 

Instead of replacing one with the other, the goal should be to combine the efficiency of artificial intelligence with the judgment and creativity of human engineers. Governance frameworks will play a similarly important role in achieving this goal. Organisational rules for compliance and security are increasingly being integrated directly into automated workflows as part of policies-as-code approaches. 

When enterprises employ artificial intelligence across a wide range of teams and environments, they can maintain consistency while using artificial intelligence. As a secondary layer of defence, red teaming exercises, in which security professionals deliberately stress-test artificial intelligence-generated systems, provide a way for malicious actors to identify weaknesses that they are likely to exploit. 

Furthermore, regulators and vendors are working to clarify liability in cases where AI-generated code causes real-world harm. A broad discussion of legal responsibility needs to continue in the meantime. As AI's role in software development grows, we can expect it to play a much bigger role in the future. The question is no longer whether or not organisations are going to use AI, but rather how they are going to integrate it effectively. 

A startup can move quickly by embracing it, whereas an enterprise must balance innovation with compliance and risk management. As such, those who succeed in this new world will be those who create guardrails in advance and invest in both technology and culture to make sure that efficiency doesn't come at the expense of trust or resilience. As a result, there will not be a sole focus on machines in the future of software development. 

The coding process will be shaped by the combination of human expertise and artificial intelligence. AI may be capable of speeding up the mechanics of coding, but the responsibility of accountability, craftsmanship, and responsibility will remain human in nature. As a result, organizations with the most forward-looking mindset will recognize this balance by utilizing AI to drive innovation, but maintaining the discipline necessary to protect their systems, customers, and reputations while maintaining a focus on maintaining discipline. 

A true test of trust for the next generation of technology will not come from a battle between man and machine, but from the ability of both to work together to build secure, sustainable, and trustworthy technologies for a better, safer world.
Read more »
Privacy
AI Chatbot Ai Data AI Security AI Techniques Chatbots Cloud Security Data Privacy Data Usage Online Security Privacy

PocketPal AI Brings Offline AI Chatbot Experience to Smartphones With Full Data Privacy

 

In a digital world where most AI chatbots rely on cloud computing and constant internet connectivity, PocketPal AI takes a different approach by offering an entirely offline, on-device chatbot experience. This free app brings AI processing power directly onto your smartphone, eliminating the need to send data back and forth across the internet. Conventional AI chatbots typically transmit your interactions to distant servers, where the data is processed before a response is returned. That means even sensitive or routine conversations can be stored remotely, raising concerns about privacy, data usage, and the potential for misuse.

PocketPal AI flips this model by handling all computation on your device, ensuring your data never leaves your phone unless you explicitly choose to save or share it. This local processing model is especially useful in areas with unreliable internet or no access at all. Whether you’re traveling in rural regions, riding the metro, or flying, PocketPal AI works seamlessly without needing a connection. 

Additionally, using an AI offline helps reduce mobile data consumption and improves speed, since there’s no delay waiting for server responses. The app is available on both iOS and Android and offers users the ability to interact with compact but capable language models. While you do need an internet connection during the initial setup to download a language model, once that’s done, PocketPal AI functions completely offline. To begin, users select a model from the app’s library or upload one from their device or from the Hugging Face community. 

Although the app lists models without detailed descriptions, users can consult external resources to understand which model is best for their needs—whether it’s from Meta, Microsoft, or another developer. After downloading a model—most of which are several gigabytes in size—users simply tap “Load” to activate the model, enabling conversations with their new offline assistant. 

For those more technically inclined, PocketPal AI includes advanced settings for switching between models, adjusting inference behavior, and testing performance. While these features offer great flexibility, they’re likely best suited for power users. On high-end devices like the Pixel 9 Pro Fold, PocketPal AI runs smoothly and delivers fast responses. 

However, older or budget devices may face slower load times or stuttering performance due to limited memory and processing power. Because offline models must be optimized for device constraints, they tend to be smaller in size and capabilities compared to cloud-based systems. As a result, while PocketPal AI handles common queries, light content generation, and basic conversations well, it may not match the contextual depth and complexity of large-scale models hosted in the cloud. 

Even with these trade-offs, PocketPal AI offers a powerful solution for users seeking AI assistance without sacrificing privacy or depending on an internet connection. It delivers a rare combination of utility, portability, and data control in today’s cloud-dominated AI ecosystem. 

As privacy awareness and concerns about centralized data storage continue to grow, PocketPal AI represents a compelling alternative—one that puts users back in control of their digital interactions, no matter where they are.
Read more »
Silicon Valley
AI generated Deepfake AI Security AI technology AI Tool Breach attack Cyber Security Hacker attack Silicon Valley

Silicon Valley Crosswalk Buttons Hacked With AI Voices Mimicking Tech Billionaires

 

A strange tech prank unfolded across Silicon Valley this past weekend after crosswalk buttons in several cities began playing AI-generated voice messages impersonating Elon Musk and Mark Zuckerberg.  

Pedestrians reported hearing bizarre and oddly personal phrases coming from audio-enabled crosswalk systems in Menlo Park, Palo Alto, and Redwood City. The altered voices were crafted to sound like the two tech moguls, with messages that ranged from humorous to unsettling. One button, using a voice resembling Zuckerberg, declared: “We’re putting AI into every corner of your life, and you can’t stop it.” Another, mimicking Musk, joked about loneliness and buying a Cybertruck to fill the void.  

The origins of the incident remain unknown, but online speculation points to possible hacktivism—potentially aimed at critiquing Silicon Valley’s AI dominance or simply poking fun at tech culture. Videos of the voice spoof spread quickly on TikTok and X, with users commenting on the surreal experience and sarcastically suggesting the crosswalks had been “venture funded.” This situation prompts serious concern. 

Local officials confirmed they’re investigating the breach and working to restore normal functionality. According to early reports, the tampering may have taken place on Friday. These crosswalk buttons aren’t new—they’re part of accessibility technology designed to help visually impaired pedestrians cross streets safely by playing audio cues. But this incident highlights how vulnerable public infrastructure can be to digital interference. Security researchers have warned in the past that these systems, often managed with default settings and unsecured firmware, can be compromised if not properly protected. 

One expert, physical penetration specialist Deviant Ollam, has previously demonstrated how such buttons can be manipulated using unchanged passwords or open ports. Polara, a leading manufacturer of these audio-enabled buttons, did not respond to requests for comment. The silence leaves open questions about how widespread the vulnerability might be and what cybersecurity measures, if any, are in place. This AI voice hack not only exposed weaknesses in public technology but also raised broader questions about the blending of artificial intelligence, infrastructure, and data privacy. 

What began as a strange and comedic moment at the crosswalk is now fueling a much larger conversation about the cybersecurity risks of increasingly connected cities. With AI becoming more embedded in daily life, events like this might be just the beginning of new kinds of public tech disruptions.
Read more »
HDMI
AI Models AI Security Cyber Security data security Device Hack Device Security Hacker attack Hackers HDMI

Hackers Can Spy on Screens Using HDMI Radiation and AI Models

 

You may feel safe behind your screen, but it turns out that privacy might be more of an illusion than a fact. New research reveals that hackers have found an alarming way to peek at what’s happening on your display—without ever touching your computer. By tapping into the faint electromagnetic radiation that HDMI cables emit, they can now “listen in” on your screen and reconstruct what’s being shown with startling accuracy. 

Here’s how it works: when digital signals travel through HDMI cables from your computer to a monitor, they unintentionally give off tiny bursts of radiation. These signals, invisible to the naked eye, can be picked up using radio antennas or small, discreet devices planted nearby. Once captured, advanced AI tools get to work, decoding the radiation into readable screen content. 

The results? Up to 70% accuracy in reconstructing text—meaning everything from passwords and emails to private messages could be exposed. This new technique represents a serious leap in digital espionage. It doesn’t rely on malware or breaking into a network. Instead, it simply listens to the electronic “whispers” your hardware makes. It’s silent, stealthy, and completely undetectable to the average user. 

Worryingly, this method is already reportedly in use against high-profile targets like government agencies and critical infrastructure sites. These organizations often store and manage sensitive data that, if leaked, could cause major damage. While some have implemented shielding to block these emissions, not all are fully protected. And because this form of surveillance leaves virtually no trace, many attacks could be flying under the radar entirely. 

Hackers can go about this in two main ways: one, by sneaking a signal-collecting device into a location; or two, by using specialized antennas from nearby—like the building next door. Either way, they can eavesdrop on what’s displayed without ever getting physically close to the device. This new threat underscores the need for stronger physical and digital protections. 

As cyberattacks become more innovative, simply securing your data with passwords and firewalls isn’t enough. Shielding cables and securing workspaces might soon be as important as having good antivirus software. The digital age has brought us many conveniences—but with it comes a new breed of invisible spies.
Read more »
Technology
AI Security Cyber Threats Cybersecurity Strategy Data Breach human risk insider risk Phishing Attacks Technology

Over Half of Organizations Lack AI Cybersecurity Strategies, Mimecast Report Reveals

 

More than 55% of organizations have yet to implement dedicated strategies to counter AI-driven cyber threats, according to new research by Mimecast. The cybersecurity firm's latest State of Human Risk report, based on insights from 1,100 IT security professionals worldwide, highlights growing concerns over AI vulnerabilities, insider threats, and cybersecurity funding shortfalls.

The study reveals that 96% of organizations report improved risk management after adopting a formal cybersecurity strategy. However, security leaders face an increasingly complex threat landscape, with AI-powered attacks and insider risks posing significant challenges.

“Despite the complexity of challenges facing organisations—including increased insider risk, larger attack surfaces from collaboration tools, and sophisticated AI attacks—organisations are still too eager to simply throw point solutions at the problem,” said Mimecast’s human risk strategist VP, Masha Sedova. “With short-staffed IT and security teams and an unrelenting threat landscape, organisations must shift to a human-centric platform approach that connects the dots between employees and technology to keep the business secure.”

The report finds that 95% of organizations are leveraging AI for threat detection, endpoint security, and insider risk analysis. However, 81% express concerns over data leaks from generative AI (GenAI) tools. More than half lack structured strategies to combat AI-driven attacks, while 46% remain uncertain about their ability to defend against AI-powered phishing and deepfake threats.

Insider threats have surged by 43%, with 66% of IT leaders anticipating an increase in data loss from internal sources in the coming year. The report estimates that insider-driven data breaches, leaks, or theft cost an average of $13.9 million per incident. Additionally, 79% of organizations believe collaboration tools have heightened security risks, amplifying both intentional and accidental data breaches.

Despite 85% of organizations raising their cybersecurity budgets, 61% cite financial constraints as a barrier to addressing emerging threats and implementing AI-driven security solutions. The report underscores the need for increased investment in cybersecurity staffing, third-party security services, email security, and collaboration tool protection.

Although 87% of organizations conduct quarterly cybersecurity training, 33% of IT leaders remain concerned about employee mismanagement of email threats, while 27% cite security fatigue as a growing risk. 95% of organizations expect email-based cyber threats to persist in 2025, as phishing attacks continue to exploit human vulnerabilities.

Collaboration tools are expanding attack surfaces, with 44% of organizations reporting a rise in cyber threats originating from these platforms. 61% believe a cyberattack involving collaboration tools could disrupt business operations in 2025, raising concerns over data integrity and compliance.

The report highlights a shift from traditional security awareness training to proactive Human Risk Management. Notably, just 8% of employees are responsible for 80% of security incidents. Organizations are increasingly turning to AI-driven monitoring and behavioral analytics to detect and mitigate threats early. 72% of security leaders see human-centric cybersecurity solutions as essential in the next five years, signaling a shift toward advanced threat detection and risk mitigation.
Read more »
GDPR
AI governance AI Security AI technology CCPA Compliance Cyber Security Data Privacy data security GDPR

The Need for Unified Data Security, Compliance, and AI Governance

 

Businesses are increasingly dependent on data, yet many continue to rely on outdated security infrastructures and fragmented management approaches. These inefficiencies leave organizations vulnerable to cyber threats, compliance violations, and operational disruptions. Protecting data is no longer just about preventing breaches; it requires a fundamental shift in how security, compliance, and AI governance are integrated into enterprise strategies. A proactive and unified approach is now essential to mitigate evolving risks effectively. 

The rapid advancement of artificial intelligence has introduced new security challenges. AI-powered tools are transforming industries, but they also create vulnerabilities if not properly managed. Many organizations implement AI-driven applications without fully understanding their security implications. AI models require vast amounts of data, including sensitive information, making governance a critical priority. Without robust oversight, these models can inadvertently expose private data, operate without transparency, and pose compliance challenges as new regulations emerge. 

Businesses must ensure that AI security measures evolve in tandem with technological advancements to minimize risks. Regulatory requirements are also becoming increasingly complex. Governments worldwide are enforcing stricter data privacy laws, such as GDPR and CCPA, while also introducing new regulations specific to AI governance. Non-compliance can result in heavy financial penalties, reputational damage, and operational setbacks. Businesses can no longer treat compliance as an afterthought; instead, it must be an integral part of their data security strategy. Organizations must shift from reactive compliance measures to proactive frameworks that align with evolving regulatory expectations. 

Another significant challenge is the growing issue of data sprawl. As businesses store and manage data across multiple cloud environments, SaaS applications, and third-party platforms, maintaining control becomes increasingly difficult. Security teams often lack visibility into where sensitive information resides, making it harder to enforce access controls and protect against cyber threats. Traditional security models that rely on layering additional tools onto existing infrastructures are no longer effective. A centralized, AI-driven approach to security and governance is necessary to address these risks holistically. 

Forward-thinking businesses recognize that managing security, compliance, and AI governance in isolation is inefficient. A unified approach consolidates risk management efforts into a cohesive, scalable framework. By breaking down operational silos, organizations can streamline workflows, improve efficiency through AI-driven automation, and proactively mitigate security threats. Integrating compliance and security within a single system ensures better regulatory adherence while reducing the complexity of data management. 

To stay ahead of emerging threats, organizations must modernize their approach to data security and governance. Investing in AI-driven security solutions enables businesses to automate data classification, detect vulnerabilities, and safeguard sensitive information at scale. Shifting from reactive compliance measures to proactive strategies ensures that regulatory requirements are met without last-minute adjustments. Moving away from fragmented security solutions and adopting a modular, scalable platform allows businesses to reduce risk and maintain resilience in an ever-evolving digital landscape. Those that embrace a forward-thinking, unified strategy will be best positioned for long-term success.
Read more »
Red teaming
AI Security Cyber Security Cybersecurity Google Gemini Prompt Injection Red teaming

How Google Enhances AI Security with Red Teaming

 

Google continues to strengthen its cybersecurity framework, particularly in safeguarding AI systems from threats such as prompt injection attacks on Gemini. By leveraging automated red team hacking bots, the company is proactively identifying and mitigating vulnerabilities.

Google employs an agentic AI security team to streamline threat detection and response using intelligent AI agents. A recent report by Google highlights its approach to addressing prompt injection risks in AI systems like Gemini.

“Modern AI systems, like Gemini, are more capable than ever, helping retrieve data and perform actions on behalf of users,” the agent team stated. “However, data from external sources present new security challenges if untrusted sources are available to execute instructions on AI systems.”

Prompt injection attacks exploit AI models by embedding concealed instructions within input data, influencing system behavior. To counter this, Google is integrating advanced security measures, including automated red team hacking bots.

To enhance AI security, Google employs red teaming—a strategy that simulates real-world cyber threats to expose vulnerabilities. As part of this initiative, Google has developed a red-team framework to generate and test prompt injection attacks.

“Crafting successful indirect prompt injections,” the Google agent AI security team explained, “requires an iterative process of refinement based on observed responses.”

This framework leverages optimization-based attacks to refine prompt injection techniques, ensuring AI models remain resilient against sophisticated threats.

“Weak attacks do little to inform us of the susceptibility of an AI system to indirect prompt injections,” the report highlighted.

Although red team hacking bots challenge AI defenses, they also play a crucial role in reinforcing the security of systems like Gemini against unauthorized data access.

Key Attack Methodologies

Google evaluates Gemini's robustness using two primary attack methodologies:

1. Actor-Critic Model: This approach employs an attacker-controlled model to generate prompt injections, which are tested against the AI system. “These are passed to the AI system under attack,” Google explained, “which returns a probability score of a successful attack.” The bot then refines the attack strategy iteratively until a vulnerability is exploited.

2. Beam Search Technique: This method initiates a basic prompt injection that instructs Gemini to send sensitive information via email to an attacker. “If the AI system recognizes the request as suspicious and does not comply,” Google said, “the attack adds random tokens to the end of the prompt injection and measures the new probability of the attack succeeding.” The process continues until an effective attack method is identified.

By leveraging red team hacking bots and AI-driven security frameworks, Google is continuously improving AI resilience, ensuring robust protection against evolving threats.
Read more »
LLM vulnerabilities
AI Security cloud-native security Cyber Security cybersecurity trends Generative AI identity management LLM vulnerabilities

Navigating 2025: Emerging Security Trends and AI Challenges for CISOs

 

Security teams have always needed to adapt to change, but 2025 is poised to bring unique challenges, driven by advancements in artificial intelligence (AI), sophisticated cyber threats, and evolving regulatory mandates. Chief Information Security Officers (CISOs) face a rapidly shifting landscape that requires innovative strategies to mitigate risks and ensure compliance.

The integration of AI-enabled features into products is accelerating, with large language models (LLMs) introducing new vulnerabilities that attackers may exploit. As vendors increasingly rely on these foundational models, CISOs must evaluate their organization’s exposure and implement measures to counter potential threats. 

"The dynamic landscape of cybersecurity regulations, particularly in regions like the European Union and California, demands enhanced collaboration between security and legal teams to ensure compliance and mitigate risks," experts note. Balancing these regulatory requirements with emerging security challenges will be crucial for protecting enterprises.

Generative AI (GenAI), while presenting security risks, also offers opportunities to strengthen software development processes. By automating vulnerability detection and bridging the gap between developers and security teams, AI can improve efficiency and bolster security frameworks.

Trends to Watch in 2025

1. Vulnerabilities in Proprietary LLMs Could Lead to Major Security Incidents

Software vendors are rapidly adopting AI-enabled features, often leveraging proprietary LLMs. However, these models introduce a new attack vector. Proprietary models reveal little about their internal guardrails or origins, making them challenging for security professionals to manage. Vulnerabilities in these models could have cascading effects, potentially disrupting the software ecosystem at scale.

2. Cloud-Native Workloads and AI Demand Adaptive Identity Management

The rise of cloud-native applications and AI-driven systems is reshaping identity management. Traditional, static access control systems must evolve to handle the surge in service-based identities. Adaptive frameworks are essential for ensuring secure and efficient access in dynamic digital environments.

3. AI Enhances Security in DevOps

A growing number of developers—58% according to recent surveys—recognize their role in application security. However, the demand for skilled security professionals in DevOps remains unmet.

AI is bridging this gap by automating repetitive tasks, offering smart coding recommendations, and integrating security into development pipelines. Authentication processes are also being streamlined, with AI dynamically assigning roles and permissions as services deploy across cloud environments. This integration enhances collaboration between developers and security teams while reducing risks.

CISOs must acknowledge the dual-edged nature of AI: while it introduces new risks, it also offers powerful tools to counter cyber threats. By leveraging AI to automate tasks, detect vulnerabilities, and respond to threats in real-time, organizations can strengthen their defenses and adapt to an evolving threat landscape.

The convergence of technology and security in 2025 calls for strategic innovation, enabling enterprises to not only meet compliance requirements but also proactively address emerging risks.


Read more »
voice API
AI Security bank transfer fraud ChatGPT-4o Credential Theft cybercriminals Deepfake financial scams OpenAI Technology text-to-speech tools voice API

UIUC Researchers Expose Security Risks in OpenAI's Voice-Enabled ChatGPT-4o API, Revealing Potential for Financial Scams

 

Researchers recently revealed that OpenAI’s ChatGPT-4o voice API could be exploited by cybercriminals for financial scams, showing some success despite moderate limitations. This discovery has raised concerns about the misuse potential of this advanced language model.

ChatGPT-4o, OpenAI’s latest AI model, offers new capabilities, combining text, voice, and vision processing. These updates are supported by security features aimed at detecting and blocking malicious activity, including unauthorized voice replication.

Voice-based scams have become a significant threat, further exacerbated by deepfake technology and advanced text-to-speech tools. Despite OpenAI’s security measures, researchers from the University of Illinois Urbana-Champaign (UIUC) demonstrated how these protections could still be circumvented, highlighting risks of abuse by cybercriminals.

Researchers Richard Fang, Dylan Bowman, and Daniel Kang emphasized that current AI tools may lack sufficient restrictions to prevent misuse. They pointed out the risk of large-scale scams using automated voice generation, which reduces the need for human effort and keeps operational costs low.

Their study examined a variety of scams, including unauthorized bank transfers, gift card fraud, cryptocurrency theft, and social media credential theft. Using ChatGPT-4o’s voice capabilities, the researchers automated key actions like navigation, data input, two-factor authentication, and following specific scam instructions.

To bypass ChatGPT-4o’s data protection filters, the team used prompt “jailbreaking” techniques, allowing the AI to handle sensitive information. They simulated interactions with ChatGPT-4o by acting as gullible victims, testing the feasibility of different scams on real websites.

By manually verifying each transaction, such as those on Bank of America’s site, they found varying success rates. For example, Gmail credential theft was successful 60% of the time, while crypto-related scams succeeded in about 40% of attempts.

Cost analysis showed that carrying out these scams was relatively inexpensive, with successful cases averaging $0.75. More complex scams, like unauthorized bank transfers, cost around $2.51—still low compared to the potential profits such scams might yield.

OpenAI responded by emphasizing that their upcoming model, o1-preview, includes advanced safeguards to prevent this type of misuse. OpenAI claims that this model significantly outperforms GPT-4o in resisting unsafe content generation and handling adversarial prompts.

OpenAI also highlighted the importance of studies like UIUC’s for enhancing ChatGPT’s defenses. They noted that GPT-4o already restricts voice replication to pre-approved voices and that newer models are undergoing stringent evaluations to increase robustness against malicious use.
Read more »
Technology
AI backdoors AI Security backdoor vulnerability computational graph Cybersecurity HiddenLayer Machine learning model architecture neural network ShadowLogic Technology

HiddenLayer Unveils "ShadowLogic" Technique for Implanting Codeless Backdoors in AI Models

 

Manipulating an AI model's computational graph can allow attackers to insert codeless, persistent backdoors, reports AI security firm HiddenLayer. This vulnerability could lead to malicious use of machine learning (ML) models in a variety of applications, including supply chain attacks.

Known as ShadowLogic, the technique works by tampering with the computational graph structure of a model, triggering unwanted behavior in downstream tasks. This manipulation opens the door to potential security breaches across AI supply chains.

Traditional backdoors offer unauthorized system access, bypassing security layers. Similarly, AI models can be exploited to include backdoors or manipulated to yield malicious outcomes. However, any changes in the model could potentially affect these hidden pathways.

HiddenLayer explains that using ShadowLogic enables threat actors to embed codeless backdoors that persist through model fine-tuning, allowing highly targeted and stealthy attacks.

Building on prior research showing that backdoors can be implemented during the training phase, HiddenLayer investigated how to inject a backdoor into a model's computational graph post-training. This bypasses the need for training phase vulnerabilities.

A computational graph is a mathematical blueprint that controls a neural network's operations. These graphs represent data inputs, mathematical functions, and learning parameters, guiding the model’s forward and backward propagation.

According to HiddenLayer, this graph acts like compiled code in a program, with specific instructions for the model. By manipulating the graph, attackers can override normal model logic, triggering predefined behavior when the model processes specific input, such as an image pixel, keyword, or phrase.

ShadowLogic leverages the wide range of operations supported by computational graphs to embed triggers, which could include checksum-based activations or even entirely new models hidden within the original one. HiddenLayer demonstrated this method on models like ResNet, YOLO, and Phi-3 Mini.

These compromised models behave normally but respond differently when presented with specific triggers. They could, for example, fail to detect objects or generate controlled responses, demonstrating the subtlety and potential danger of ShadowLogic backdoors.

Such backdoors introduce new vulnerabilities in AI models that do not rely on traditional code exploits. Embedded within the model’s structure, these backdoors are harder to detect and can be injected across a variety of graph-based architectures.

ShadowLogic is format-agnostic and can be applied to any model that uses graph-based architectures, regardless of the domain, including autonomous navigation, financial predictions, healthcare diagnostics, and cybersecurity.

HiddenLayer warns that no AI system is safe from this type of attack, whether it involves simple classifiers or advanced large language models (LLMs), expanding the range of potential targets.






Read more »
Technology
AI Security Artifcial Intelligence ChatGPT-4 Morris II Technology

Look Out For This New Emerging Threat In The World Of AI

 



As per a recent discovery, a team of researchers has surfaced a groundbreaking AI worm named 'Morris II,' capable of infiltrating AI-powered email systems, spreading malware, and stealing sensitive data. This creation, reminiscent of the notorious computer worm from 1988, poses a significant threat to users relying on AI applications such as Gemini Pro, ChatGPT 4.0, and LLaVA.

Developed by Ben Nassi, Stav Cohen, and Ron Bitton, Morris II exploits vulnerabilities in Generative AI (GenAI) models by utilising adversarial self-replicating prompts. These prompts trick the AI into replicating and distributing harmful inputs, leading to activities like spamming and unauthorised data access. The researchers explain that this approach enables the infiltration of GenAI-powered email assistants, putting users' confidential information, such as credit card details and social security numbers, at risk.

Upon discovering Morris II, the responsible research team promptly reported their findings to Google and OpenAI. While Google remained silent on the matter, an OpenAI spokesperson acknowledged the issue, stating that the worm exploits prompt-injection vulnerabilities through unchecked or unfiltered user input. OpenAI is actively working to enhance its systems' resilience and advises developers to implement methods ensuring they don't work with potentially harmful inputs.

The potential impact of Morris II raises concerns about the security of AI systems, prompting the need for increased vigilance among users and developers alike. As we delve into the specifics, Morris II operates by injecting prompts into AI models, coercing them into replicating inputs and engaging in malicious activities. This replication extends to spreading the harmful prompts to new agents within the GenAI ecosystem, perpetuating the threat across multiple systems.

To counter this threat, OpenAI emphasises the importance of implementing robust input validation processes. By ensuring that user inputs undergo thorough checks and filters, developers can mitigate the risk of prompt-injection vulnerabilities. OpenAI is also actively working to fortify its systems against such attacks, underscoring the evolving nature of cybersecurity in the age of artificial intelligence.

In essence, the emergence of Morris II serves as a stark reminder of the digital culture of cybersecurity threats within the world of artificial intelligence. Users and developers must stay vigilant, adopting best practices to safeguard against potential vulnerabilities. OpenAI's commitment to enhancing system resilience reflects the collaborative effort required to stay one step ahead of these risks in this ever-changing technological realm. As the story unfolds, it remains imperative for the AI community to address and mitigate such threats collectively, ensuring the continued responsible and secure development of artificial intelligence technologies.


Read more »
Subscribe to: Comments (Atom)