Search This Blog

Powered by Blogger.

Blog Archive

Labels

Showing posts with label Kubernetes. Show all posts

Kubernetes can be Hacked due to a Container Verification Bug

 


An extremely serious vulnerability in the Kyverno admission controller for container images could permit malicious actors to import a raft of malicious code into the production environments of cloud providers by exploiting this vulnerability. 

Using the Kyverno admission controller, the ability to verify signatures is provided as a mechanism for ensuring that only validated and signed containers are pulled into a given cluster running Kubernetes. Many potentially disastrous scenarios can be averted by doing this. There are a lot of malicious payloads that can be found in booby-trapped container images. These include cryptominers, rootkits, container escapes, lateral movement exploit kits, credential stealers, and more. 

However, there is a bug (CVE-2022-47633) that can be exploited to undermine the functionality of this mechanism. It has been revealed that an attacker could take advantage of this vulnerability and inject unsigned images into any protected cluster, bypassing the policy of image verification. This was stated in a blog post on Dec. 21 by researchers at ARMO. 

There are high stakes here: an attacker can effectively take control of a victim's pod, and let themselves access all of the assets, credentials, and service account tokens of the pod, including the token of the service account, used to access the API server, the researchers cautioned. 

Taking advantage of the vulnerability, one can completely bypass the verification process for image signatures. This gives an attacker a wide range of target options when it comes to an attack on a Kubernetes cluster. Ben Hirschberg, CTO, and co-founder of ARMO describe how any workload can mount cluster secrets and data volumes. By having access to the vulnerability of the Kubernetes cluster of the victim of the attack, the attacker can inject code into the cluster. This code steals data and credentials from the cluster. Additionally, the attacker is also able to inject his or her own code, thus allowing the attacker to take advantage of the victim's CPU for cryptocurrency mining. 

Subverting the Container Admission Controller: An inside look at the bug 

When a new workload is requested from a Kubernetes API server that is defined via an image with a tag, that API server sends a request to the Kyverno admission controller to validate the new workload as defined in the image. 

There are several ways in which the admission controller determines whether a workload is admissible to the cluster. This includes requesting the image manifest and the container registry's signature.

The container runtime starts a new workload based on the image. This is true if the image is checked out, and if the image is not checked out, the image does not proceed. 

According to the advisory, the vulnerability was discovered as a result of the controller's signature validation process downloading the image manifest twice - but only verifying the signature for one of those downloads. 

Hence, the attack looks like this: a malicious registry or proxy is used to socially engineer an administrator into pulling a container image from an infected registry or proxy. In the initial import of the malicious registry file, the admission controller receives a valid, benign, signed image that has been imported by the malicious registry. As of now, everything seems to be working well. 

This is followed by a second request from the admission controller for the manifest of the signed image so that the digest for mutation can be retrieved - and it can then be used to alter the human-readable tag associated with the container. In this instance, no signing validation is performed. This allows a different, unsigned and malicious image to be returned by the malicious registry. This image is ultimately the one that will run on your system if you push the button to start it. 

This is a classic example of a TOCTOU problem, which means a time-of-check-to-time-of-use problem, in which an attacker can bait and switch their victim, according to a research paper published by ARMO. 

Because the image manifest which is going to be used in the end is a different one from the one that was verified, it gives the attacker the chance to trick the client. 

Kyverno users should update to version 1.8.5 as soon as possible since this vulnerability was introduced in version 1.8.3 and has been fixed in the updated version. It is ensured that the same hash of the image will be used for modifying the workload specification and verifying the signature in the patch. 

In particular, this vulnerability affects only Kubernetes with the Kyverno container manager. Hirschberg warned that other methods of verifying image signatures also need to take care not to be vulnerable to this technique. 

Concerns About Container Security are on the Rise 

Hirschberg has noted that containers are an excellent target for cybercriminals because they are typically hosted in the cloud. This gives them access to a huge amount of computational resources, which are extremely valuable and expensive. This enables hackers to steal computational resources and data in a relatively short time while also staying unnoticed for a long period. 

According to him, there are no exact statistics. However, based on the current trend of containers being widely adopted, it is clear that this type of problem is becoming more prevalent in the industry. 

"Security teams are learning how to handle them, and Kubernetes in general. I don't think that it is a true 'blind spot,' but container security teams are still learning the whole environment with many neglected areas", Hirschberg added.

Even though image signature verification has just begun to take off, admission controllers still represent one of those potential areas that may have been neglected due to the early stages of its adoption. Nonetheless, they are also part of a broader dialogue that should be conducted about supply chain software security in a way that considers them an imperative issue. 

During the SolarWinds attack, Hirschberg indicated that the world saw how sensitive this issue is when it comes to trusting the security of external code. Kyverno is a security tool that includes signature validation for the first time in the Kubernetes world, and with this, it introduces additional vulnerabilities. However, it does seem that with these vulnerabilities come security improvements that will enable users to overcome this issue in the future.

Safeguarding From Container Attacks Inside the Cloud


As an alternative to virtualization, containerization has become a key trend in software development. It entails encapsulating or packaging software code and all of its dependencies so it may execute consistently and uniformly across any infrastructure. Containers are self-contained units that represent whole software environments that may be transported. They include everything a program needs to run, including binaries, libraries, configuration data, and references. Docker and Amazon Elastic, as an illustration, are two of the extra well-known choices. 

Although many containers can run on the same infrastructure and use the same operating system kernel, they are isolated from such a layer and have a little interface with the actual hosting elements, for instance, a public cloud occasion. The ability to instantly spin up and down apps  for users, is one of the many advantages of running cloud-based containers. Admins may utilize orchestration to centrally manage containerized apps and services at scale, such as putting out automatic updates and isolating any malfunctioning containers.

Container adoption is at an all-time high, worldwide businesses of all sizes are eager to jump on board. According to a poll conducted by the Cloud Native Computing Foundation (CNCF), 83 percent of respondents plan to use Kubernetes in production in 2020, up from 78 percent the year before and just 58 percent in 2018. As adoption grows, cybercriminals' interest grows as well. According to a June Red Hat study, 94 percent of respondents have experienced a Kubernetes security problem in the last 12 months. 

Larry Cashdollar, an Akamai security researcher, recently set up a basic Docker container honeypot to test what type of attention it would get from the larger web's cybercriminals. The results were alarming: in just 24 hours, the honeypot was used for four different nefarious campaigns. Cashdollar had integrated SSH protocol for encryption and developed a “guessable” root password. It wouldn't stick out as an obvious honeypot on the web because it was running a typical cloud container configuration, he explained. It would instead appear to be a vulnerable cloud instance. The assaults had a variety of objectives: one campaign aimed to utilize the container as a proxy to access Twitch feeds or other services, another attempted a botnet infection, a third attempted crypto mining, and the fourth attempted a work-from-home hoax. 

"Profit is still the key motivator for cybercriminals attacking containers," as these cases demonstrate, according to Mark Nunnikhoven, a senior cloud strategist at Lacework. "CPU time and bandwidth can be rented to other criminals for buried services, or even used to directly mine cryptocurrencies. Data can be sold or ransomed at any time. In an environment where containers are frequently used, these reasons do not change." 

According to a recent Gartner study, client misconfigurations or mistakes would be the primary cause of more than 99 percent of cloud breaches by 2025. As per Trevor Morgan, product manager at comfort AG, most businesses, particularly smaller businesses, rely on default configuration options rather than more advanced and granular setup capabilities: "Simple errors or selecting default settings  that are far less safe than customized options." The problems with configuration typically go beyond the containers themselves. Last July, for example, misconfigured Argo Workflows servers were detected attacking Kubernetes clusters. 

Argo Workflows is an open-source, container-native workflow engine for coordinating parallel activities on Kubernetes to reduce processing time for compute-intensive tasks such as machine learning and large data processing. 

According to an examination by Intezer, malware operators were using publicly available dashboards which did not require authentication for outside users to drop crypto miners into the cloud. Far above misconfiguration, compromised images or layers are the next most serious threat to containers, according to Nunnikhoven. "Lacework Labs has witnessed multiple instances of cybercriminals infiltrating containers, either through malware implants or pre-installed crypto mining apps," he said. "When a group deploys the pictures, the attacker has access to the victim's resources."

According to Gal Singer, an Aqua Security researcher, the flaw (CVE-2020-15157) was discovered in the container image-pulling process. Adversaries may take advantage of this by creating dedicated container images which stole the host's token when they were pulled into a project.  Similarly, a denial-of-service vulnerability in one of Kubernetes' Go libraries (CVE-2021-20291) was discovered to be exploited by storing a malicious picture in a registry. When the image was taken from the registry by an unwary user, the DoS condition was generated.

The second source of concern is vulnerabilities, both known and unknown. In 2021, several container flaws were discovered, but "Azurescape" was likely the most alarming. Within Microsoft's multitenant container-as-a-service offering, Unit 42 researchers found a chain of exploits that might allow a hostile Azure user to infect other customers' cloud instances. 

Containerized environments can provide unique issues in terms of observability and security controls, according to Nunnikhoven, but a comprehensive security approach can help. Researchers recommended that users apply a laundry list of best practices to secure their Kubernetes assets: 

  • Avoid using default settings; use secure passwords.
  • To prevent attackers from impersonating the token owner, do not send privileged service account tokens to anyone other than the API server. 
  • Enable the feature "BoundServiceAccountTokenVolume": When a pod ends, its token becomes invalid, reducing the risk of token theft.
  • Examine orchestrators for least-privilege settings to verify that CI/CD movements are authenticated, logged, and monitored. 
  • Be comprehensive: Create a unified risk picture that includes both cloud-based applications and traditional IT infrastructure. 
  • Have data-analysis software in place, as well as an automatic runbook that can react to the findings.

AWS, and Alibaba Cloud was Attacked by Crypto Miners

 

An intel source recently provided Cisco Talos with modified versions of the TeamTNT cybercrime team's infected shell scripts, an earlier version of which was documented by Trend Micro. The malware creator modified these tools after learning that security experts had disclosed the prior version of its scripts. These scripts are intended primarily for Amazon Web Services (AWS), but they might also be used on-premise, in containers, or in other Linux instances. 

There are multiple TeamTNT payloads focusing on bitcoin mining, persistence, and lateral movement employing tactics like identifying and installing on with all Kubernetes pods in a local network, in addition to the primary credential stealer scripts. A script containing user credentials for the distribution system server and another with an API key which may allow remote access to a tmate shared login session is also included. Defense evasion functions aimed at defeating Alibaba cloud security technologies are included in some TeamTNT scripts.

When it comes to decision making obtaining credentials, the script looks for them in the following places and APIs: 

  • It attempts to obtain the string 'AWS' from /proc/*/environ from the Linux system environment variables. 
  • Obtaining the string 'AWS' from Docker environment variables with the command $(docker inspect $) (docker ps -q).
  • /home/.aws/credentials and /root/.aws/credentials are the default AWS CLI credential file locations.
While the query itself will not be caught by Cisco Secure Cloud Analytics, the alert "AWS Temporary Token Persistence" will detect later use of these credentials to generate further temporary credentials. Finally, the virus saves any credentials acquired by the preceding functions to the file "/var/tmp/TeamTNT AWS STEALER.txt" and uses cURL to transfer it to the URL http://chimaera[.]cc/in/AWS.php before deleting it. 

No CloudTrail, GuardDuty, or SCA events were generated when the script ran on the target EC2 instance for all network traffic was restricted by the VPC Security Group such as the script could not access TeamTNT's servers. 

The core of the defense impairment functions is directed against Alibaba Cloud Security's numerous agents, how, they also target Tencent Cloud Monitor and third-party BMC Helix Cloud Security, agents. While the bulk of malicious scripts targets AWS Elastic Compute Cloud (EC2) virtual machines, these bots are most typically detected running inside Alibaba Cloud Elastic Compute Service (ECS) or a Tencent Cloud VM. They could theoretically be put on a VM operating on AWS or any other service, but it would be unusual. TeamTNT makes no attempt to disable AWS CloudWatch, Microsoft Defender, Google Cloud Monitor, Cisco Secure Cloud Analytics, CrowdStrike Falcon, Palo Alto Prisma Cloud, or other popular cloud security tools in the United States. 

The Alibaba defense damage routines have been retrieved and saved here from the script Kubernetes root payload 2.sh. Since static analysis of the defense impairment functions is problematic due to the presence of multiple Base64 encoded strings, those functions have been decrypted and placed back into the file ali-defense-impairment-base64-decoded.sh.txt. 

"Cybercriminals who have been exposed by security researchers should update those tools to keep functioning successfully," stated Darin Smith of Talos. 

The serious remote code execution problem in Spring Framework (CVE-2022-22965) has been leveraged to deploy cryptocurrency miners, in yet another example of how threat actors quickly co-opt recently revealed flaws into existing attacks. To deploy the cryptocurrency miners, the exploitation efforts employ a unique web shell, but not before switching off the firewall and disabling other virtual currency miner processes.

Misconfigured Argo Workflows Instances Employed for Attacking Kubernetes Clusters

 

Intezer has discovered new Kubernetes cluster attack vectors using misconfigured instances of Argo Workflows. Threat actors have already been benefitted from this vector as researchers have noticed the use of such a wild way for the operators dropping crypto miners. 

Argo Workflows is an open-source workflow system that can be used for coordinating parallel operations at the Kubernetes region, which enables computer-intensive activities such as machine education and big data processing to accelerate processing time. It is also used in general to facilitate the installation of containers. 

Meanwhile, Kubernetes is a popular cloud engine for container orchestration. It is an open-source framework that enables automated containerized workloads, services, and applications deployed, scale and managed over hosts clusters. 

According to the investigation by Intezer, malware controllers drop encryption devices through Argo into the cloud, because certain instances are publicly visible through dashboards that require no authentication from outside users. Through these malfunctioning permissions, actors at risk can run unauthorized code within the environment of the target. 

Intezer security researchers, Ryan Robinson and Nicole Fishbein wrote a report documenting the intrusion and noted they had already detected infected nodes. Both indicated the attacks were serious, considering hundreds of misconfigured deployments had occurred and crypto miners like the Kannix/Monero miner were discovered by this attack vector. 

"We have detected exposed instances of Argo Workflows that belong to companies from different sectors including technology, finance, and logistics. Argo Workflows is an open-source, container-native workflow engine designed to run on K8s clusters. Argo Workflows instances with misconfigured permissions allow threat actors to run unauthorized code on the victim's environment," Robinson and Fishbein said. 

Confidential information such as code, credentials, and picture names in private containers may be included in the exposed instances. Researchers also noticed that permissions that allow visitors to deploy workflows in several instances are configured. They have also discovered that threat actors target some nodes that are wrongly installed.

According to researchers, the "Kannix/ Monero-miner," demands very little skill to use, and further this study indicates that other security teams have identified major crypto-currency mining operations against the clusters of the Kubernetes. 

"In Docker Hub, there are still several options for Monero mining that attackers can use. A simple search shows that there are at least 45 other containers with millions of downloads," the study said. 

Fishbein and Robinson recommend users browse the Argo Workflows dashboard using an unauthenticated incognito browser outside corporate situations to check for misplacements. Executives can also request the API for an instance and inspect the status code.

TeamTNT Targeting Organizations Via Cryptojacking Malware

 

A cybercriminal gang known as TeamTNT has been ramping up its cloud-focused cryptojacking operations for some time now. TeamTNT operations have targeted Kubernetes clusters due to their wide usage and are an attractive target for threat actors running primarily in cloud environments with access to nearly infinite resources.

Attackers have also designed new malware called Black-T that unites open-source cloud-native tools to assist in their cryptojacking operations. Once getting a foothold into a Kubernetes cluster, the malware attempted to spread over as many containers as possible, leading to malicious activity. 

Palo Alto’s Unit 42 researchers have discovered and confirmed close to 50,000 IPs compromised by this malicious campaign perpetrated by TeamTNT across multiple clusters. Several IPs were repeatedly exploited during the timeframe of the episode, occurring between March and May. Most of the compromised nodes were from China and the US — identified in the ISP (Internet Service Provider) list, which had Chinese and US-based providers as the highest hits, including some CSPs (Cloud Service Providers)

TeamTNT has gathered 6.52012192 Monero coins via a cryptojacking campaign, which is equal to USD 1,788. The mining operation was found to be operating at an average speed of 77.7KH/s across eight mining workers. Operations using this Monero wallet address have continued for 114 days and are still operating. 

The researchers said TeamTNT’s new campaign is the most sophisticated malware Unit 42 has seen from this gang. They said on this round the threat actor developed more sophisticated tactics for initial access, execution, defense evasion, and command and control. Although the malware is still under development and the campaign has not spread widely, Unit 42 believes the attacker will soon improve the tools and start a large-scale deployment. 

Team TNT has stolen the credentials of 16 applications, including those of AWS and Google Cloud credentials, which may be stored on the compromised cloud instance if downloaded. The presence of Google Cloud credentials being targeted for collections represents the first known instance of an attacker group targeting IAM credentials on compromised cloud instances outside of AWS. 

Researchers believe that Microsoft Azure, Alibaba Cloud, Oracle Cloud, or IBM Cloud IAM credentials could be targeted using similar methods. Unit 42 researchers are yet to find evidence of credentials from these cloud service providers (CSPs) being targeted. TeamTNT first started collecting AWS credentials on cloud instances they had compromised as early as August 2020.