North Korean hackers Kimsuky are using ChatGPT to create convincing deepfake South Korean military identification cards in a troubling instance of how artificial intelligence can be weaponised in state-backed cyber warfare, indicating that artificial intelligence is becoming increasingly useful in cyber warfare.
As part of their cyber-espionage campaign, the group used falsified documents embedded in phishing emails targeting defence institutions and individuals, adding an additional layer of credibility to their espionage activities.
A series of attacks aimed at deceiving recipients, delivering malicious software, and exfiltrating sensitive data were made more effective by the use of AI-generated IDs. Security monitors have categorised this incident as an AI-related hazard, indicating that by using ChatGPT for the wrong purpose, the breach of confidential information and the violation of personal rights directly caused harm.
Using generative AI is becoming increasingly common in sophisticated state-sponsored operations. The case highlights the growing concerns about the use of generative AI in sophisticated operations. As a result of the combination of deepfake technology and phishing tactics, these attacks are harder to detect and much more damaging.
Palo Alto Networks' Unit 42 has observed a disturbing increase in the use of real-time deepfakes for job interviews, in which candidates disguise their true identities from potential employers using this technology.
In their view, the deepfake tactic is alarmingly accessible because it can be done in a matter of hours, with just minimal technical know-how, and with inexpensive consumer-grade hardware, so it is alarmingly accessible and easy to implement.
The investigation was prompted by a report that was published in the Pragmatic Engineer newsletter that described how two fake applicants who were almost hired by a Polish artificial intelligence company raised suspicions that the candidates were being controlled by the same individual as deepfake personas.
As a result of Unit 42’s analysis, these practices represent a logical progression from a long-standing North Korean cyber threat scheme, one in which North Korean IT operatives attempt to infiltrate organisations under false pretences, a strategy well documented in previous cyber threat reports.
It has been repeatedly alleged that the hacking group known as Kimsuky, which operated under the direction of the North Korean state, was involved in espionage operations against South Korean targets for many years.
In a 2020 advisory issued by the U.S. Department of Homeland Security, it was suggested that this group might be responsible for obtaining global intelligence on Pyongyang's behalf.
Recent research from a South Korean security firm called Genians illustrates how artificial intelligence is increasingly augmented into such operations.
There was a report published in July about North Korean actors manipulating ChatGPT to create fake ID cards, while further experiments revealed that simple prompt adjustments could be made to override the platform's built-in limitations by North Korean actors.
It follows a pattern that a lot of people have experienced in the past: Anthropic disclosed in August that its Claude Code software was misused by North Korean operatives to create sophisticated fake personas, pass coding assessments, and secure remote positions at multinational companies.
In February, OpenAI confirmed that it had suspended accounts tied to North Korea for generating fraudulent resumes, cover letters, and social media content intended to assist with recruitment efforts.
These activities, according to Genians director Mun Chong-hyun, highlight the growing role AI has in the development and execution of cyber operations at many stages, from the creation of attack scenarios, the development of malware, as well as the impersonation of recruiters and targets.
A phishing campaign impersonating an official South Korean military account (.mil.kr) has been launched in an attempt to compromise journalists, researchers, and human rights activists within this latest campaign. To date, it has been unclear how extensive the breach was or to what extent the hackers prevented it.
Officially, the United States assert that such cyber activities are a part of a larger North Korea strategy, along with cryptocurrency theft and IT contracting schemes, that seeks to provide intelligence as well as generate revenue to circumvent sanctions and fund the nuclear weapons program of the country.
According to Washington and its allies, Kimsuky, also known as APT43, a North Korean state-backed cyber unit that is suspected of being responsible for the July campaign, was already sanctioned by Washington and its allies for its role in promoting Pyongyang's foreign policy and sanction evasion.
It was reported by researchers at South Korean cybersecurity firm Genians that the group used ChatGPT to create samples of government and military identification cards, which they then incorporated into phishing emails disguised as official correspondence from a South Korean defense agency that managed ID services, which was then used as phishing emails.
Besides delivering a fraudulent ID card with these messages, they also delivered malware designed to steal data as well as allow remote access to compromised systems. It has been confirmed by data analysis that these counterfeit IDs were created using ChatGPT, despite the tool's safeguards against replicating government documents, indicating that the attackers misinterpreted the prompts by presenting them as mock-up designs.
There is no doubt that Kimsuky has introduced deepfake technology into its operations in such a way that this is a clear indication that this is a significant step toward making convincing forgeries easier by using generative AI, which significantly lowers the barrier to creating them.
It is known that Kimsuky has been active since at least 2012, with a focus on government officials, academics, think tanks, journalists, and activists in South Korea, Japan, the United States, Europe, and Russia, as well as those affected by North Korea's policy and human rights issues.
As research has shown, the regime is highly reliant on artificial intelligence to create fake summaries and online personas. This enables North Korean IT operatives to secure overseas employment as well as perform technical tasks once they are embedded. There is no doubt that such operatives are using a variety of deceptive practices to obscure their origins and evade detection, including artificial intelligence-powered identity fabrication and collaboration with foreign intermediaries.
The South Korean foreign ministry has endorsed that claim.
It is becoming more and more evident that generative AI is increasingly being used in cyber-espionage, which poses a major challenge for global cybersecurity frameworks: assisting citizens in identifying and protecting themselves against threats not solely based on technical sophistication but based on trust.
Although platforms like ChatGPT and other large language models may have guardrails in place to protect them from attacks, experts warn that adversaries will continue to seek out weaknesses in the systems and adapt their tactics through prompt manipulation, social engineering, and deepfake augmentation in an effort to defeat the system.
Kimsuky is an excellent example of how disruptive technologies such as artificial intelligence and cybercrime erode traditional detection methods, as counterfeit identities, forged credentials, and distorted personas blur the line between legitimate interaction and malicious deception, as a result of artificial intelligence and cybercrime.
The security experts are urging the public to take action by using a multi-layered approach that combines AI-driven detection tools, robust digital identity verification, cross-border intelligence sharing, and better awareness within targeted sectors such as defence, academia, and human rights industries.
Developing AI technologies together with governments and private enterprises will be critical to ensuring they are harnessed responsibly while minimising misuse of these technologies. It is clear from this campaign that as adversaries continue to use artificial intelligence to sharpen their attacks, defenders must adapt just as fast to maintain trust, privacy, and global security as they do against adversaries.
