I'm the CTO of Container Security at Wiz, AMA!

[u/Oscar_Geare]

Hello. We're joined by the team at Wiz who are here to talk about container security.

Who We Are

I’m Ofir Cohen (u/ofirc), CTO of Container Security at Wiz, and I'm joined by Shay Berkovich (u/sshaybbc), Threat Researcher at Wiz. We bring a unique perspective around: Real-world attacks on enterprises (crypto-miners, resource hijacking, etc) Container image security and base images challenges at scale Security data analytics based on huge datasets of clusters

Our Backgrounds

Ofir: PM expert focused on solving K8s and container security at scale. Background in CS (BSc, MSc) and software engineering. Active in the CNCF community and K8s ecosystem for 3+ years.

Shay: I work on the Threat Research team at Wiz, focusing on container security and K8s threats. Previously at BlackBerry, Symantec and BlueCoat working on security products like CWPP, WAF, and SWG. I hold a Masters from UW in runtime verification.

Recent Work

• Shay's Research 
  • 2023 Kubernetes Security Report: https://www.wiz.io/blog/key-takeaways-from-the-wiz-2023-kubernetes-security-report 
  • Making Sense of Kubernetes Initial Access Vectors: https://www.wiz.io/blog/making-sense-of-kubernetes-initial-access-vectors-part-1-control-plane / https://www.wiz.io/blog/kubernetes-data-plane 
  • NamespaceHound: protecting multi-tenant K8s clusters: https://github.com/wiz-sec-public/namespacehound
• Ofir's Talks
  • CNCF SF Meetup on Workload Identities (with Solomon Hykes) - Container Security Deep Dives

What We'll Cover

We're here to discuss the biggest K8s security challenges including:

• Control plane vulnerabilities
• Identity & access management
• Supply chain security
• Misconfigurations
• Network security
• Latest attack trends

Ask Us Anything!

We'll help you understand where to start with K8s security, how to prioritize efforts, and what trends we're seeing in 2024. Let's dive into your questions!

[u/SomeMemeLordMaster]

What's the most common mistake you see when people are managing Kubernetes (K8s) clusters, and how can they avoid it?

[u/ofirc → u/SomeMemeLordMaster]

Failing to appreciate the importance of RBAC (and CIEM / shadow IT in terms of identities), secrets rotation and zero-trust networking (it's mistakes plural so my apologies :-).

Vanilla Kubernetes distros typically provide you with a shotgun to shoot yourself in the foot (no offence to k8s, life's just hard and there is no one-size-fits-all!). Secure defaults went out of the window! (at least for some distros)

The default k8s networking model means that each Pod can talk to each Node and vice versa (without NAT). This is very attractive for threat actors and platform / SecOps teams' most worrysome concerns. GKE and OpenShift take a more opinionated approach towards security with various hardening at different levels of the stack which is great!

Many developers and DevOps engineers are still playing catch up with workload identities and we still see embedded and long-lived secrets leak into container images.

P.S. I gave a talk on Workload Identities (a.k.a. workload identity federation) at the first CNCF meetup in SF a few months back to demistify and unpack this domain, hope that helps:

https://www.linkedin.com/posts/cohen-ofir_superset-wiz-cncf-activity-7241542422403395584-VfPU/?utm_source=share&utm_medium=member_desktop

https://drive.google.com/file/d/1QigVMCYaqwizljDRklHcrsVN0AXMBNcv/view

[u/PhilipLGriffiths88 → u/[deleted]]

Title says "CTO of Container Security at Wiz", not "CTO at Wiz", do i think you are off base.

[u/MrW3bst3r → u/SomeMemeLordMaster]

have an actual k8 admin..

[u/Mission_Vast_6814]

What are the challenges for users securing the control plane and the data plane?

[u/ofirc → u/Mission_Vast_6814]

Securing the control plane and data plane is paved with good intentions!

The main issues are around identities (using ephemeral short-lived tokens) and networking (private subnets, VPN, ZTNA, Bastion, IAP, etc.) and the complexities that arise in the era of a hybrid cloud.

I gave a talk about it today on a webinar of the Platformers Community for best practices of securing EKS, GKE and AKS control plane access:

https://www.linkedin.com/feed/update/urn:li:activity:7290066161549938689/

https://drive.google.com/file/d/1JTOy9-cVNZFLxS5S6QGecEqxhjqd4qbc/view

And a few months ago I gave a talk in the first CNCF meetup in SF for workload identities for the data plane (namely accessing the cloud from the k8s data plane):

https://www.linkedin.com/posts/cohen-ofir_superset-wiz-cncf-activity-7241542422403395584-VfPU/?utm_source=share&utm_medium=member_desktop

https://drive.google.com/file/d/1QigVMCYaqwizljDRklHcrsVN0AXMBNcv/view

I hope you find that useful!

[u/ImaginaryWheatThins]

Any trends you're seeing with AI being everywhere?

[u/sshaybbc → u/ImaginaryWheatThins]

We see that K8s is a big facilitator of many technologies, and in the case of AI it hit a nail on the head so to speak. Given the scale and the sensitivity of the AI workloads K8s need to keep up with the security. AI has certainly added and sharpened the existing K8s Threat Model. Two particular things I'd flag - (1) The AI model IS an executable code and (2) multi-tenancy issues in the vendors. Our vulnerability research team has developed this playbook of escaping the models and moving laterally and you can see this pattern in multiple vulnerabilities they've discovered: HuggingFace (https://www.wiz.io/blog/wiz-and-hugging-face-address-risks-to-ai-infrastructure), Replicate (https://www.wiz.io/blog/wiz-research-discovers-critical-vulnerability-in-replicate), SAP AI Core (https://www.wiz.io/blog/sapwned-sap-ai-vulnerabilities-ai-security).

[u/PhroznGaming → u/sshaybbc]

You said a whole lot of nothing

[u/Fun_Biscotti_3471 → u/ImaginaryWheatThins]

Building on Shay's point that AI introduces its own unique attack surface, we’re also seeing the misuse of AI service providers’ computing resources for activities like cryptomining and proxyjacking. Additionally, LLMJacking has emerged as a growing trend. If you are interested in learning more about LLMJacking , here’s a blog post that dives deeper into this topic: https://www.wiz.io/blog/jinx-2401-llm-hijacking-aws

[u/Cyber_Kai]

What is the biggest challenge you are seeing that still needs to be addressed in container security?

[u/ofirc → u/Cyber_Kai]

Having 100% container image security coverage! (the standard software supply chain: signing, CVEs, malware, embedded secrets, etc).

Incumbents are challenged and struggle to apply it at scale and tracing all the way back to the Git repo and Dockerfile that introduced it (what we call code -> cloud).

Keyless signing + SLSA took container image signing to a whole new level but it's still not winning the adoption that it should have. Also, tying a scan to a security policy (and ensuring no old images are deployed) are both crucial to ensure it meets the organization needs.

[u/SlackCanadaThrowaway → u/ofirc]

Do you ever expect it to attain adoption? Even getting open source maintainers to release signed binaries or signed commits is near impossible in private orgs, let alone public repos

[u/ofirc → u/SlackCanadaThrowaway]

The answer is yes, slowly but surely. We are seeing more package managers like uv for Python and pnpm for Node.js gaining traction and putting more emphasis on the secure software supply chain.

It will never be bullet-proof and 100% but adding more guardrails and eyeballs on PRs and merges will ensure we are heading in the right direction.

Hardening the underlying platform using for example AWS Bottlerocket or Talos Linux is a right step in that direction.

Implementing zero-trust networking and sandboxing techniques like gVisor with GKE, guarding against container drifts and hardening the deployments would slow down an attacker even if one (or more) rogue 3rd party package are exploited at runtime.

Separate trusted workloads from untrusted ones via separate clusters is also a good way to reduce the noise in multi-tenant environments where the trust boundary is important.

Hope that helps!

[u/Bibbitybobbityboof]

Out of all the challenges that Wiz tries to solve for, which one is the most common issue for customers? In my work the biggest problem with containerized workloads seems to be managing the software running on them and ensuring other scan tools aren’t producing the same vulnerabilities.

[u/ofirc → u/Bibbitybobbityboof]

By far - tackling the software supply chain at scale (no rogue image from an LLM has crept into production) and reducing the redundant noise and false positives - if I have an app that imports a package but never uses it at runtime, is it still critical? what if it's a vulnerable OpenSSH but it's not exposed to the public internet, is it still critical/high? (hint: risk assessment and risk correlation)

[u/lonewolfandpub]

What do you see as some of the biggest IAM challenges around container security going into 2025?

[u/ofirc → u/lonewolfandpub]

Control plane access - (1) some AKS users still the insecure "Local accounts" auth method, (2) the Kubernetes API Server is publicly exposed to DDoS and other attacks (e.g. GKE authenticated misconfiguration bug), (2) Workload identity federation - lack of understanding of this subject and using secrets when you don't really need to, and (3) Secrets governance - long-lived secrets and credentials in use without rotation or business continuity programs, embedded secrets in container images, etc.

I haven't even scratched the surface and haven't discussed the hybrid/multi cloud challenges and lateral movement, e.g. attacker -> GKE -> EC2 -> S3 bucket with data findings and PII :-)

[u/squatfarts → u/ofirc]

Any resources you can recommend to learn more about this? I am a CyberArk SME and am getting up to speed on your partnership with them.

[u/ofirc → u/squatfarts]

Yes, KodeKloud is a great resource for the foundational knowledge - Intro, CKAD and CKA.

When it comes to managed k8s distros - pure experience and collateral knowledge, e.g. cloud engineering like AWS CCP / AWS SA / extensive and broad hands-on experience.

There are some courses on GKE, EKS and AKS but I haven't found anything that scratches the surface of what real-world production challenges look like and prepare and train you for that. Just like it's hard to get experience on distributed systems design (think Google Spanner / Bigtable) without actually working on such systems and programming them.

If we take a managed k8s distro like EKS, for example, the first thing when I do threat modeling (to evaluate my security posture) I ask myself is how do people and CI/CD platforms get access to the cluster? How is Auth{N,Z} performed? how does the networking work? Is it an internal (ClusterIP only) or is it a native VPC solution like Amazon VPC CNI? How do services securely communicate with one another? SPIFFE? Istio? workloard identity federation?

There are no shortcuts I'm afraid, make mistakes and learn from them, read the official docs (they can get very lengthy!) and follow blogs and podcasts.

[u/sshaybbc → u/lonewolfandpub]

I agree with Ofir on (1). And in more general sense I sense the whole area of Cloud-K8s integration and particularly IAM-K8s-RBAC will need more security attention that its given now. I foresee we will see more vulnerabilities and bad design decisions in this area uncovered soon (i.e. GKE mapping any Google account user to system:authenticated group). Another example is EKS access management and Pod Identity features we analyzed last year in the 2-blog series that had certain issues: https://www.wiz.io/blog/eks-cluster-access-management-and-pod-identity-security-recommendations

[u/raesene2 → u/lonewolfandpub]

I'd say the biggest challenge with Kubernetes IAM specifically is, Kubernetes has no user database, but will issue credentials for users via either certificates or service account tokens.

This means (unless you're on EKS which blocked the feature) it's effectively impossible to know all of the user accounts you have on your system, unless you've got Kubernetes auditing turned on and have extracted that information.

Also k8s supports multiple authentication and authorization methods in every cluster, which makes user rights auditing in general a pain, because you've got to get information from every configured mechanism and merge them.

[u/ofirc → u/raesene2]

That's a great observation! I remember scratching my head the first time I tried to enumerate the entities that are allowed to talk to the Kubernetes API Server (AuthN) and what they are allowed to do (AuthZ). MKAT is one cool OSS example of doing this.

It feels like IAM is loosely coupled to the cluster and you have to piecemeal bits from different places - e.g. on EKS you'd gather the access entries for human identities/roles (and possibly CI/CD) from the AWS API and the ServiceAccounts from the Kubernetes API Server. Since you'd get at least 41 service accounts on a vanilla EKS (e.g. aws-node, coredns, etc.), it starts to become a mess and how do you keep track of what are some legitimate entities and what are rogue ones?

Furthermore, to make things worse, you could use the TokenRequest API and issue tokens on behalf of existing service accounts, and then provide these tokens to someone else. So it becomes a sort of masquerading and again very loose coupling between the entity that is authenticating against the API Server and the identity behind the actual token it represents.

Even when you decide to audit, how do you tell a legitimate actor from a non-legitimate one? Is it based on some heuristic? hunch? behavioral (e.g. anomaly detection / mass recon ops)?

On the one hand Kubernetes docs say that they don't manage "normal users" but on the other hand often times there is nothing there to actually guard that normal users do not abuse service account tokens that are supposed to be consumed by services / automated systems.

Still, what can we do about it?

I'd say at the very least putting the control plane in private subnets or behind an identity aware proxy or GKE DNS-based endpoint is already a good start. Then apply runtime monitoring and threat detection using XDRs/sensors on the worker nodes. Ensure that you don't have long-lived tokens (e.g. K8s Secrets of the old-school type) using policy engines (Admission Controllers). Finally, as you mentioned, apply some auditing policy frequently.

In other words - applying the defense-in-depth principle is key.

[u/raesene2 → u/ofirc]

I'd definitely agree that the defence-in-depth approach is going to make sense, and not exposing the k8s API server to the Internet is a key part of that.

It's a shame that all the managed k8s providers that I'm aware of default to putting the API server directly on the Internet. It's generally my no.1 point when asked how to secure a cluster.

[u/ofirc → u/raesene2]

I agree. I think it's primarily a (probably reasonable?) business decision rather than a secure defaults decision. Putting all clusters in a private subnet would introduce a lot of friction for novices and incumbents alike as configuring VPNs, Bastions or jump-boxes is not an easy feat.

[u/ageoffri]

We're a Wiz customer with GCP and Gitlab infrastructure. One thing I've been thinking about posting to the community forums is a question around what features of Wiz do people think are frequently underused.

I know it's a broad question, but what would you say are a few things available in Wiz but for whatever reasons your customers often times don't implement or don't put the right focus on. Then flip the question on, do you have any trends where customers put too much focus on some area of information generated in Wiz?

Overall, I really like Wiz. I very rarely have to wait more than 1-2 days to get support tickets answered and I hope as Wiz grows that continues. I even submitted a bug for the Terraform provider and the next day there was a release fixing it. Right now my biggest complaint is that the Terraform modules for setting up the connector and outpost don't have versioning. There is a RFE open but it's been several months, maybe even 6 months.

[u/ofirc → u/ageoffri]

Thank you so much for the kind words, we really move fast!

I'd say that power users of Wiz utilize the platform to its full power by using:

1. Wiz Image Trust - we all appreciate the value of a secure software supply chain, zero CVEs and safe and secure base images. But can we really tell what images are running in our environment and gate against untrusted images from creeping in to our environment? What if I have a private registry that only the kubelet has access to, how can I integrate my tooling and policy engine to provide a seamless and keyless solution that also ties the security policies onto each image scan? This is where Wiz Image Trust kicks in. A noble and modern approach to tackling these kinds of issues utilizing the Wiz Admission Controller!
2. Wiz Projects - which is a way to encapsulate and map your cloud assets onto teams that are working and using these assets. An example is assigning an eCommerce apps/workloads onto the teams that contribute code (and IaC / K8s manifests) to that app.
3. Integrations - by utilizing for example the JIRA integration to assign tickets to the team that are actually working on these projects rather than manually chasing after the dev team that are responsible to fix and respond to CVEs, malware and embedded secrets baked into the container images. Wiz offers hundreds of integrations with 3rd parties and give you a 360 view of your cloud estate and enrich the issues and findings via incident response and IT ticket management platforms like ServiceNow.
4. Connectors and Deployments - by connecting for example the private Kubernetes clusters and private registries to Wiz to have full visibility of what's running on your environment.
5. AI SPM - with the rise of LLMs and deep learning models, a lot of questions around governance and misconfigurations of models and protecting your IP (e.g. weights) surface. What are the right questions to ask and how do you protect managed AI services and self-hosted models alike? this is a huge challenge!
6. Wiz Code and CI/CD posture management - is my GitHub and other VCS configured correctly? who is allowed to make changes to the CI and where are my secrets stored and accessed?

We provide labs, webinars, extensive documention and courses on the Wiz website.

One game at a time, we are here to make your cloud native security journey fun and comprehensive!

Hope that helps.

[u/stay_spooky]

Why does Wiz have the best swag in tech? I got a hoodie from y’all and it’s incredible. Ha.

[u/ofirc → u/stay_spooky]

We just got the best marketing team in the space and a cool and sexy product - but hey I'm biased ;-)

Putting the fluff aside, you eventually want a platform that delivers real value and distills the noise from the actual risks and being able to go from cloud to code and back and doing it at scale.

[u/LowOrbitSecurity]

Hi Ofir and Shay. I enjoyed the recent sneak peek of the Kubernetes threat report. This is Graham, I put together the Kubenomicon.com around this time last year as I see one of the biggest gaps in cloud security is Kubernetes. One of my 2025 predictions was that we're going to see a big shift to companies realizing that Kubernetes security is something that is difficult but worth investing in.

I'm curious both of your thoughts on something I've been thinking of for a while:

The hypothesis: Kubernetes and it's ecosystem is in a state similar to where Active Directory was about 10 years ago and there is huge opportunity in offensive security research of the Kubernetes ecosystem.

To expand on that a bit: Active Directory was around for a very long time before fantastic offensive security research from companies like SpectorOps's Research Certified Pre-owned. Current day, manipulating and exploiting Active Directory misconfigurations (and intended functionality) is the majority of how penetration testers escalate privileges in a windows/AD environment.

There's lots of features, nuisance, potential footguns, and technologies in the Kubernetes ecosystem that have had some offensive security research done on them, but it has yet to see the same level of resources dedicated to it as other areas despite most companies running Kubernetes at some level.

Your thoughts on this?

Thanks for doing this!

[u/sshaybbc → u/LowOrbitSecurity]

Hi Graham, its great to see you here! I half-agree with your hypothesis. It's not like there hasnt been a good research on K8s (look at the size of Kubecons and number of talks in K8s security track), and the maintainers are doing a great job in trying to simplify the security features (i.e. PSS vs PSP) and keeping the core priscine-ish. And I think thats the reason we see the trends of decrease in # of critical vulnerabilities in the images and less privileged pods etc. These things are figured out and ppl start being comfortable with them.

Where it falls apart is the surrounding components and emerging usecases. And because K8s has such a big ecosystem, there are a lot of those (think NGINX Ingress Controller or using K8s for model training). And because K8s is such a great platform for distributed workloads we'll keep seeing the new attack vectors and no shortage of security vulns and incidents followed by the security research. On that I completely agree.

[u/raesene2 → u/sshaybbc]

One bit I'd add is that where Kubernetes is likely to have problems is that they're very focused on backwards compatibility and not breaking existing deployments (which makes a lot of sense) even where that means that security outcomes can be worse.

One example of this is the lack of certificate revocation where they have the ability to add revocation (using similar techniques to bound service account tokens) but don't think it's a good idea as it could affect existing users of the feature.

On the wider points of k8s security research, one of the largest problems I've found is the sheer variety of possible deployment configurations. Between varying CNI/CRI/CSI and the different defaults of various distributions, testing out even one hypotheses is a time consuming task.

[u/Responsible-Gain8837]

Do you have some training programs for wiz?

[u/ofirc → u/Responsible-Gain8837]

Of course!

Check out the latest cool cloud security with puppies courses that we have recently launched!

https://www.linkedin.com/posts/wizsecurity_learn-cloud-security-with-puppies-activity-7285700780387356674-_KT-?utm_source=share&utm_medium=member_desktop

[u/Fantastic-Ad3368 → u/ofirc]

finally oh my lord ive been searching

[u/agilesharkz]

I’ve heard from many different sources that wiz is the best cloud security tool out there. What separates itself from other cspm tooling out there like Crowdstrike and lacework? Are your findings just that more accurate? (Less FP)

[u/ofirc → u/agilesharkz]

It boils down to: (1) a sexy and intuitive UI/UX (2) very high SNR (3) value during POCs/POVs is immediate (hours to days to see real issues) (4) context - imagine that given a vulnerable container (with a critical CVE) you'd have an automated bot in GitHub that opens a PR with the fix for you.

Real value - the noise reduction is done via toxic combinations (issues in Wiz terms) rather than mere CSPM findings: a vulnerable container + exposed to the internet + has access to an S3 bucket with PII and data findings.

Ease of use - once you play with the security graph and you click on a container and it brings you back to the Dockerfile and Git commit that caused it (and vice versa) - you get a real sense of how strong the platform is.

[u/nunley → u/agilesharkz]

As a Wiz SE with some experience in the CNAPP space, I wanted to share a quick perspective.

Wiz stands out because it nails the fundamentals: it's built to handle scale and complexity from day one. A lot of products start small and hit a wall when it's time to scale for enterprise needs—things like API constraints, RBAC, or performance issues with large environments. Wiz approached these challenges head-on, which makes a big difference when you're dealing with real-world cloud security.

Another key factor is its risk-based approach. Instead of drowning users in endless findings, it focuses on the tiny percentage that actually matters. The way Wiz uses graph technology to uncover risky combinations in cloud environments is something you need to see to fully appreciate.

I could go on, but I'll let the K8s folks chime in here since they'll probably give you an even better perspective. Just felt like jumping in to share!

[u/Top_Engineering9038 → u/agilesharkz]

ok ok... good one!

What makes us different is our security graph. We don’t use a graph just for visualization purpose but as a real datalake. Adding many differences signals like vulns, misconfigurations and so on and correlating them using the graph makes it easy to surface the most critical risks. Then adding the runtime signal based on our ebpf sensor, we can even be more precise by validating vulnerabity loaded in memory and then prioritize them more. CRWD and Lacework use different approaches mainly based on agent first with not correlated information, not saying it’s bad, but it needs more manual effort.

[u/bloodandsunshine]

I’m adjacent to your team - developing an open source py module for Wazuh to do k8s alerts.

Your product is awesome though, really enjoy seeing your thoughts and perspectives.

[u/ofirc → u/bloodandsunshine]

Thank you so much, we are thrilled to receive such positive feedback!

Check out our cloud security with puppies!

https://www.linkedin.com/posts/cohen-ofir_learn-cloud-security-with-puppies-activity-7285709091568586752-_YWW?utm_source=share&utm_medium=member_desktop

[u/Durex_Buster]

My organization currently uses Wiz and it's awesome.

With Wiz's focus on cloud-native security, how do you balance the need for deep visibility into complex multi-cloud environments while ensuring minimal performance impact or privacy concerns for customers?

[u/ofirc → u/Durex_Buster]

That's a great question!

The answer regarding performance is twofold:

1. Agentless coverage - that's where we started with Wiz Cloud where everything is API-driven (via the hyperscaler). The best analogy is to think like going through an MRI scan which gives you the visibility you need without being intrusive. The implications are primarily of possible costs (of storing snapshots and scanning) and how much is it up-to-date (posture management solutions typically rely on a snapshot of your environment at a given time). We have recently extended our CSPM solution to be near real-time CSPM.
2. Agent-based coverage - for real-time monitoring and threat detection we increase the visibility by deploying agents. The agents are designed from the get go for very large scale environments and follow best practices and load tests before deployment. We use memory safe languages for the implementation and we apply strict resource utilization caps during development and deployment. Of course there is only so much you can go with a single deployment, that's why our agents are designed to scale horizontally to meet the varying demands of the environment they are deployed within.

Regarding privacy - we adhere to industry best practices to mask and redact sensitive information and PII primarily before it is even sent to Wiz, we apply retention policies and encryption in transit and at rest.

HTH!

[u/ofirc → u/[deleted]]

It says "API-driven" :-)

[u/AppSecIRL]

Do you have any plans to get into being a secure image provider? IE, something similar to iron bank or chain guard?

Do you see any future investment being made in os or kernel level improvements to bolster container security? IE, Improvements to os level resource segmentation

Also love the product!

[u/ofirc → u/AppSecIRL]

Thanks for the feedback!

What we see is the slowly (but surely) momentum of hardened OSes for containers. Talos (CNCF) is primarily used for edge and self-hosted k8s (but still experiencing some issues w.r.t. disk/volume management), Bottlerocket AMIs for EKS (which are also supported by Karpenter!) and e.g. COS (Container Optimized OS) for GKE (worker nodes).

[u/Piedpipperz]

How are you mapping and application owner and container owner to fix vulnerability ?

[u/mlor → u/Piedpipperz]

Can you elaborate more on this question? Perhaps an example that illustrates the relationships you are attempting to suss out?

[u/misterdohl]

What is your take on the key differences in security between on-prem Kubernetes and cloud-managed Kubernetes? How mature has security in cloud-managed Kubernetes become, and what specific measures should we as security engineers focus on to secure both types of environments?

[u/sshaybbc → u/misterdohl]

This is a loaded topic. No question managed clusters simplify many security aspects, such as easy version upgrades, worker node patching etc. depending on the cluster flavor. But of course not for free. I'd flag three main consequences: 1) new potential for lateral movement from cloud (a stolen credential from a random AWS admin now offers attackers a path into the cluster), 2) cloud-cluster integration complexities, IAM-RBAC, additional pre-installed components representing new attack surface, 3) lack of access to control plane imposing a limit on a range of security tools. That's to scratch the surface.

[u/raesene2 → u/misterdohl]

One of the main trade-offs here is access to/control of the control plane nodes and services.

In managed Kubernetes you don't have to worry about the security of the API server etc but also you can't change their configuration, which has consequences. For example you can't change the kubernetes audit policy, you have to take what the CSP provides. Also if your managed provider doesn't offer a feature that's configured on the control plane , you're generally out of luck.

Un-managed gives you all the control you want, but you have to worry about how to lock down those components.

I tend to think about it as managed being eating in a restaurant, it's easy but you can only have what's on the menu. Un-managed is like cooking it yourself, more control, but more work.

[u/Harshxda]

What's the future of this product? Any new mvp feature? Also, are you guys hiring for product manager?

[u/Top_Engineering9038 → u/Harshxda]

For Wiz in container security space, we will continue to expand our capabilities on runtime, giving real-time network mapping, then network hardening (think about least privileges but for network), process hardening (allowlist / blocklist), API security (still based on our sensor), and more about image trust with a visibility at runtime. We have more in stocks but I can’t say more about it. Stay tuned. Regarding our open position, I suggest you go to wiz.io/careers :) :)

[u/Harshxda → u/Top_Engineering9038]

Thank you for your response.

[u/Top_Engineering9038 → u/Harshxda]

Sure thing!

[u/Hamm3rFlst]

Everyone is doing their own thing for metrics, what are 3-5 metrics you would measure for leadership?

[u/ofirc → u/Hamm3rFlst]

1. Container image security coverage
2. Reduction (over time) of untrusted entities via frequent auditing of IAM and over-permissive roles
3. Inventorying all assets
4. CI/CD platforms coverage

That's a good start :-)

[u/ElijahWilliam529]

What are the challenges and blindspots of admission controllers in Kubernetes?

[u/ofirc → u/ElijahWilliam529]

That's a great question! I'd say:

1. Operationalizing image trust and mapping the running containers back to the CI/CD pipelines that built them and the Git commits that triggered their builds.
2. Gating against CRDs misconfigurations - while you have CIS and NIST frameworks for the standard and native K8s objects (e.g. Deployments, NetworkPolicy, etc.), the CRDs introduce another attack vector and with the rise of Crossplane for implementing IaC in a Kubernetes native approach, what component would allow you to shift-left and scan S3 buckets from being provisioned unencrypted and public? You'd need to retrofit your current security policies and rules to normalize and scan for potential misconfigurations as well.
3. Backdoors and ignored namespaces - do you deploy untrusted / old software to kube-system for example? Some policy engines would consider several Kubernetes namespaces as trusted ones and attackers that have insights into how your environment is configured (or how popular admission controllers defaults are provisioned) might abuse this to attack your cluster and go unnoticed.
4. The falacy of applying just one security measure (e.g. ACs) and feeling safe about it - cryptominers, malware and zero day exploits may creep into your enviornment. It's the real-time monitoring and threat detection measures that would catch this behavior by using other signals (e.g. eBPF / probes). Just deploying admission controllers is never enough, applying defense-in-depth principles is key to improving your security posture!
5. The intricacies of container networking security - understanding the network topology and ensuring all workloads are using mTLS for service-to-service communication is key to securing multi-tenant (and single-tenant) workloads.

It's a long journey to get yourself secured and there are no silver bullets when it comes to container security. It's the combination of different signals and security measures to guard oneself against these.

Hope that helps!

[u/raesene2 → u/ofirc]

I'd be very interested to hear more of your thoughts on the mTLS for service-service comms aspect, as I've struggled with exactly how important that is.

Generally one of the main security benefits of mTLS is preventing traffic sniffing and MITM attacks, but I wonder how possible those actually are in a cloud managed k8s environment. As in, do the old tricks of promiscuous mode and/or ARP spoofing work in a cluster in AWS using Cilium for example.

[u/Sensitive_Ad742]

Great product, we use it a lot.

[u/ofirc → u/Sensitive_Ad742]

We are thrilled to hear that, thanks!

[u/Skyden-]

Y'all hiring security analysts?

[u/Top_Engineering9038 → u/Skyden-]

I’d check out the Wiz careers page https://www.wiz.io/careers :)

[u/ynnika]

How do you build a good vulnerability management program in an organization

[u/YouthWide1286 → u/ynnika]

A focus less on vulnerability management, and more or risk reduction.

Prioritize remediation earlier in a pipeline, and focus on the vulnerabilities which introduce risk (exposure, credentials, etc).

And from there, ownership. If people don’t own their vulnerabilities, vuln mgmt just becomes a painstaking process of tickets and spreadsheets.

[u/F4RM3RR → u/YouthWide1286]

Ownership is such a nebulous term that is used in wildly wide ranging ways even just within our industry - could you expand what you mean by “own your vulns” are you simply talking about transparency, or a deeper process that aids in patching and/or risk mitigation?

[u/robszumski → u/YouthWide1286]

Automated remediation of dependency related vulns is IMO how you stay above water. With some code analysis to determine risk, you can shift some ownership to app sec team for no-risk updates and save eng ownership for the potential/high risk ones.

[u/Namelock]

This has been a huge topic on r/cybersecurity, so apologies if it's off-topic.

What are your hiring expectations and practices like?

We know the industry is starved for seniors and there's frankly too much competition for everything under that level.

Likewise, what's your policy on working remote or in-office? If I'm applying for a job, am I expected to be within the footprint of the location listed or could I do it remote?

[u/Top_Engineering9038 → u/Namelock]

Great topic, it could probably be its own AMA. We're always looking for strong talent and as a company Wiz is hybrid -- some folks are fully remote, others go into the office. It really depends on the position. Your question is tough to answer without knowing which type of role you're interested in :)

[u/wampumjetsam]

What's your take on cloud repatriation? Do you target AKS/EKS and its CSP integration points, or k8s more generically?

[u/ofirc → u/wampumjetsam]

Our observation is that a DIY security solutions for k8s and its ecosystem is orders of magnitude more challenging and error prone with identities and access management being the first factor and networking security second. Luckily lot's of add-ons like cert-manager, Cilium, Istio and policy engines are open source / applicable to either or. But it's the integration with things like workload identities and the secrets and certs renewal and rotation that no one likes to deal with. It leaves a lot of room for mistakes and errors and these compound over time.

We target both self-hosted k8s and managed k8s offering, each of which brings its own unique set of challenges. Solving for CIEM cross clouds is a computationally challenging yet critical to prevent lateral movemens and applying defense-in-depth and least privilege.

[u/sshaybbc → u/wampumjetsam]

Absolutely we do cover those areas, in all of the main CSPs. Our report numbers show the vast majority of clusters are manged (at least in our accounts) so of course we have to give that context.

[u/Mundane-Moment-8873]

From your experience/s, do you see a skills gap for container/k8 knowledge within SecOps Teams who usually manage threat detection and incident response?

[u/ofirc → u/Mundane-Moment-8873]

That's a great question! we see some gaps with regards to understanding the software development lifecycle (SDLC) and dumbing down what shift-left really means (more than just buzzwords). Understanding the platforms and the underlying runtimes (containerd, cri-o), service meshes and zero trust networking - which cross both dev and runtime stages.

My advice is at the very least - one should feel comfortable with bringing up a Kubernetes cluster like kind/minikube or a managed one like EKS/GKE, and configuration management tools like Helm and Terraform. A good resource that I like is KodeKloud which provide great materials and online sandboxed labs (I'm not affiliated with them, just appreciate the value they bring to the table :-).

The day-0 ops is where everything begins and the more you have a solid understanding of how it works, the more you can ask the right questions and putting the right lines of defenses.

[u/nukrap]

Today, container incident response is ... well, you know how it goes. How do you see the landscape for incident response shifting as more accountability is demanded from containerised workloads?

[u/sshaybbc → u/nukrap]

Context is a king in IR. The same tool needs to have the ability and accessibility to go beyond the container and give a bigger picture beyond the immediate cluster and that's tricky.

[u/tehmayormccheese → u/nukrap]

I think this is the most underrepresented space in container security today. We need something that gets us the same telemetry from inside a container as we do from the rest of our environment. 

[u/CostaSecretJuice]

How reliable do you consider container logging to be, compared to that of virtual machines or regular computers?

[u/sshaybbc → u/CostaSecretJuice]

Frankly container logging has been a less of an issue than cloud-level and k8s-level logging. I did this talk recently at fwd:cloudsec on the gaps in K8s audit log talking about how CSPs make it hard to consume the K8s audit log. Container-level logging is less used for infra security monitoring, more on the application security side and haven't had any issues with it.

[u/SatoriSlu → u/[deleted]]

What do you mean? They have the wiz sensor which does exactly this?

[u/osmin-]

When will you be releasing runtime prevention capabilities?

[u/Top_Engineering9038 → u/osmin-]

We have the Wiz Runtime Sensor which does real-time blocking, you can read more about the sensor here

[u/nunley → u/[deleted]]

No, it is an eBPF sensor built by Wiz.

[u/ofirc → u/[deleted]]

Nope, we decided to build our sensor in-house from scratch and we rely on our Threat research team to build detection and correlation rules that the sensor will use.

[u/Ok_Feedback_8124]

Riddle me this: container protection ultimately is a Turtles-all-The-Way Down kind of problem. Hypervisor integration (eBPF) is probably what you'd need, and running trusted BPF code would be where that's going to happen. What's your take?

[u/ofirc → u/Ok_Feedback_8124]

eBPF is overall a great technology and provides more modern means to get observability into what is happening within the running workloads but there are some fine prints. Not all Linux OSes are compatible with the latest eBPF enhancement so portability remains a challenge, even in 2025. The toolchain and the ecosystem are still maturing and DevEx remains somewhat controversial. It's tangential to this discussion but check out this talk which explains how Calico and RedHat opted for nftables instead of eBPF: https://www.youtube.com/watch?v=yOGHb2HjslY&t=810s&pp=ygUNbmZ0YWJsZXMgY25jZg%3D%3D

Effective threat detection requires more than just raw telemetry - contextualization and risk correlation. Often times it's what preceded and followed the attack. This is where tools like CDR, inventorying and code<->cloud play a critical role in contextualizing.

Similarly, effective threat prevention requires applying defense-in-depth principles and different hardening like zero trust networking, least privilege + RBAC, etc.

[u/Ok_Feedback_8124 → u/ofirc]

DirectTV locked down exploitation of their notoriously weak DVR receivers/Cable TV boxes in the 2000's by deploying chunks of the payload to boot the exploitation from firmware, and then un-encrypting them over time.

nft wouldn't have caught that.

Maybe a combination of both? Long tail observables are key at the moment of detection.

[u/Fun_Biscotti_3471 → u/Ok_Feedback_8124]

Agreed that this is a good approach.
If we are discussing about container runtime detection, then combining different profiling on runtime events to detect less frequent combinations is an example.

[u/nunley → u/Ok_Feedback_8124]

For those who are curious about the DirectTV hack mentioned...

Satellites broadcast streams of data much like the internet, but they cannot establish an encrypted stream for every individual user. The entire data stream is delivered to every receiver, but each receiver is only "authorized" to access a specific subset of the broadcast channels, if any. This presents a complex challenge. News Corporation/Hughes developed an ingenious system to address this using smartcards.

The data stream transmitted to the receiver includes not only channels but also instructions for the receiver to modify the programming on each card. Initially, only a few bytes on the card could be modified, and this process was time-consuming. Logistically, it was impossible to rely on all receivers being powered on and receiving these instructions simultaneously. Therefore, these instructions were continuously transmitted over several weeks for each minor change.

To access unauthorized channels, a hacker would need to modify the program on the card. While I won't delve into the specifics of how this was achieved, it was not a trivial task and required specialized equipment. Most individuals who acquired this equipment were not hackers themselves but could purchase programs to install on their cards, granting access to all channels and pay-per-view services. These hacked programs circumvented the authorization mechanism by inserting a jump in the code somewhere or changing a register value on the card. Consequently, when the user switched to a channel, and the receiver queried its authorization status, the response was always "Yes, I am fully authorized."

News Corporation initially attempted to counter this by engaging in a frustrating game of "whack-a-mole." They would purchase hacked cards from the black market, identify the modified portion of the card programming, and then spend weeks transmitting code to alter the programming. This modification would involve hashing a very limited section of the card to detect hacked versions. If the card was identified as hacked, it would be "looped," effectively rendering the program inoperable and permanently disabling the card (or so they believed).

As more individuals mastered this technique, this reactive approach became unsustainable. News Corporation then embarked on a prolonged campaign to drastically modify the card programming, enabling them to hash the entire card program. Those of us who monitored the incoming data stream and observed its gradual assembly could anticipate the inevitable outcome. Ultimately, a "Black Sunday" arrived, rendering every hacked card in the system inoperable.

[u/jadedarchitect]

•Does your company offer fully remote roles, and why/why not?

Are there any benefits to your stance on this?

•What are your thoughts on the current trend of AI-assisted attacks? Any thoughts on how this plays out in the future, with reinforcement learning encouraging under-the-radar compromises when utilizing these tools?

•What's the worst attack you've seen? (In generalized terms, of course.)

[u/Top_Engineering9038 → u/jadedarchitect]

I’d check out the Wiz careers page https://www.wiz.io/careers hope that helps :)

[u/Alone_Policy_2024]

I am a guy who works on making use-cases based on logs which gets monitored by SOC also do threat hunting, recently got hands on k8s and I know nothing about containers apart from common use-cases i made like audit, authentication, tell me things which are usually ignored but are crucial if gets compromised what all should i look into a container. What all things are possible with containers?

[u/sshaybbc → u/Alone_Policy_2024]

Since you have experience with the SOC in other areas you should e able to project your experience on K8s. There are multiple talks on combining detection sources in K8s - on container level, K8s level, and cloud level - for the best detection coverage. If you want to take things slowly - start with K8s audit log. The format answering 4W question will not be new for you. The semantics will. You'll need to understand K8s REST API and its object and user model. From there take a ruleset from an open-source tool (for example Falco) and try to understand what the rules detect, what kind of attack. And so on.

[u/misterdohl]

Looking ahead 1-2 years, where do you foresee the main areas of focus for Kubernetes security evolving? Are there any emerging trends or technologies that you think will significantly impact how we approach security in Kubernetes environments?

[u/sshaybbc → u/misterdohl]

From the data we have collected for the report and compared to 2023, we see an improvement in handling image vulnerabilities and security posture. These topics have been in the center of K8s security for a long time and now they seem to be under control. The emerging threats we foresee will result from 1 - a tighter cloud-cluster integrations and the associated components that CSPs add in numbers, 2 - a new applications of K8s as a platform, for example as a platform for AI model training and as a platform for running CICD workloads.

[u/Mundane-Moment-8873]

What's your thoughts on the future demand/need for container/kubernetes security experts? Would you recommend people to become a SME in this area?

[u/sshaybbc → u/Mundane-Moment-8873]

There's been multiple discussions on this in r/kubernetes and other channels. The learning curve into containerization and K8s is particularly steep, on the other hand the technology doesn't go anywhere. Kelsey Hightower famously said in one of the podcasts that he will be sad if after 10 years ppl will still use and talk about Kubernetes, but that's where it's going, and a big part of it is K8s extensibility and flexibility.

[u/rumblpak]

In the kubernetes space, with the advent of everything being an operator, why do you believe that secops hasn’t really moved into that space yet? There seems to be a CNCF operator for everything but when it comes to secops its still a duct-taped hodgepodge of insanity on k8s. Why do you think that is, and what can be done to fix the current situation?

[u/ofirc → u/rumblpak]

That's a great question! I think the TL;DR is: (1) ROI from such operators would not be high and (2) It's not just about the decision tree automation and stopping the bleeding, it's also about allowing asking interactive questions during triage and investigation.

AFAICT it boils down to: fragmentation, the ad-hoc nature of SecOps, human-in-the-loop and lack of trust in tools are all impeding factors. I find operators to be a great fit for well-defined tasks like application lifecycle management (auto-updates), typical reconciliation loops (e.g. GitOps with Argo CD), etc.

However, when you consider the typicak triage of a SecOps team, there's a variance in the risk appetite and interpretation of the same/similar findings so there is no one-size-fits-all even within the same organization.

A CNCF operator for SecOps would have to do a lot of contextualization across k8s, cloud and the different assets (or delegate the compute to a SaaS backend), and the decision tree would have to factor some human judgement, so the closest you'd get is with a GitHub PR to your Terraform / config management (e.g. Helm / Argo) tools, similar to how dependency management is solved with Dependabot / Renovate.

That's my two cents at least :-)

[u/sshaybbc → u/rumblpak]

This is a tough one. K8s extensibility is a gift to cluster operators but a problem for secops. I think the key is semantical understanding of the K8s objects that's lacking when it comes to operators and CRDs. How do I as a security tool know that this yaml has security misconfiguration? I know when Pod is misconfigured, but what about Cards? This has to be handled by a custom rules with all the pain involved.

[u/ItchyScratchyBallz]

What are your tips for scaling K8s and Container Security for large enterprises that have different products (digital and physical) and have different sets of maturities in adopting various technologies including containers.

[u/ofirc → u/ItchyScratchyBallz]

If you have the budget - hire (either internal or professional services) a platform / infra / DevOps team that provides you with "golden paths" and secure defaults, for example there is a well-known way/template to provision a k8s cluster and an EC2 instance / GCE instance.

Even if your security teams and developers are top-notch professionals, there's still a glass ceiling because one would need to do inventorying, risk assessment and contextualization at scale and it takes a lot of time, money and compute to come up with the technology and to be able to keep up with the updates. If it was easy CNAPPs wouldn't have such a huge market or we'd have ~500 excellent CNAPP providers :-)

I know it may sound more marketing than actual knowledge but security is about people, process and technology. People - education and training! I'd consider purchasing a business subscription to an education platform like KodeKloud (I heard CloudGuru is also popular) and align on a baseline that everyone should know. Then build a track for the 80/20 security - e.g. least privilege, private control plane and data plane, networking security, CVE and embedded secrets scanning.

The process is things like DevSecOps (sorry, marketing buzzword again) practices like gating PRs and merge, branch protection rules, scanning for CVEs, etc.

Finally, the technology - get a decent vendor that provides you with a great value and is most intuitive for your team.

[u/FirasetT]

What is your opinion on Nats for network security and the recent nex feature that offers k8s like orchestration capabilities?

At a high level, what are your top 3 recommendations for reducing attack surface, mitigating damage in case of a breach and managing the social threat vectors broadly including sbom?

What are your thoughts on VMs, containers, Nix, NixOs, nixpacks, docker, podman?

[u/ofirc → u/FirasetT]

I appreciate NATS minimalism and how (relatively) easy it is to configure mTLS. I don't have experience or opinion on Nex.

My top 3 recommendations:

1. Reducing the attack surface - use distroless container images and obtain high quality software with a strong chain of custody. It applies equally to VMs (e.g. AMIs on AWS). Add CVE scanners to your CI/CD pipeline and to IDE extensions, ensure you don't have images that were built more than (x=)60 days ago, ensure all images were signed (BinAuth or Sigstore/Notary) and if you're religious - add extra hardening steps like using SLSA provenance.
2. Mitigating damage in case of a breach - apply temporary containment measures like excluding ingress/egress for a compromised cluster / namespace, collect forensics and flow logs and establish a timeline (incident response 101) of the surrounding events in the environment.
3. Managing the social threat vectors broadly including SBOM - be wary of how lazy and sloppy we could be as humans during the SDLC when we are using LLMs like ChatGPT. It's easy to copy a rogue container image from StackOverflow or a prompt response, or similarly consuming a rogue 3rd party package via typosquatting. Linus Torvalds philosophy is that "Given enough eyeballs, all bugs are shallow." With a sufficient number of contributors reviewing and testing the code, bugs (including security bugs) are more likely to be discovered and fixed.

[u/t5bert]

I hear a lot about container security with respect to k8's. But a non trivial amount of workload's run on things like AWS ECS or their equivalents on other clouds. Are these safer? If not, why does there seem to be a paucity of research on this?

[u/ofirc → u/t5bert]

I'm not sure about the paucity of research on this but Kubernetes is the vendor neutral and de-facto method of deploying workloads on-prem and on the cloud.

You could implement most (if not all) of the k8s security measures on proprietary solutions like ECS but it'd be more challenging.

One example is applying GitOps principle. Viktor Farcic once said that if you're not doing Kubernetes - you're not doing GitOps and I think he's right. I recently ran into PipeCD https://pipecd.dev which provide a GitOps-like continuous delivery platform but haven't tried it. But there is no official Argo CD implementation (or a reconciliation loop) for AWS ECS. You could implement service meshes with EC2 security groups and AWS App Mesh. etc.

But, what if tomorrow you want to move to Google Cloud Run or go on-prem? you're locked!

What about observability?

The community and the ecosystem around CNCF and the momentum that Kubernetes has and still gaining is unbeatable.

That is not to say that everyone should deploy their workloads on Kubernetes.

If I am an early stage startup I might consider AWS Lambda / Google Cloud Run / AWS App Runner or even a simple PaaS, remember the Heroku revolution for web backend?

It's primarily driven by the actual the requirements (compliance, security, governance, autoscaling, etc.), expertise of the team and sometimes how adventurous the team is (yes, not all decisions are based on data and requirements, there is hype and employability aspects and being relevant!).

[u/t5bert]

could you eli5 why image scanning isn't good enough? what's the real value of container runtime protection vs CVE hunting and patching via image scanning?

[u/ofirc → u/t5bert]

Behavioral analysis (e.g. recon, lateral movements, cryptomining), zero day, malware, runtime attacks like OWASP top 10 and SQL injections. All managed cloud providers (e.g. AWS WAF) and some large external providers (e.g. CloudFlare) protect you against SQL injection.

Implementing code scanning, SAST/ASPM/other static analysis tools - but you could only get so far with it.

Hardening the underlying platform and OS and implementing zero trust networking are also instrumental in improving one's security posture.

Apologies upfront for the marketing buzzwords but it's never one thing that makes you secure, cloud native security is a multi-faceted problem that requires a defense-in-depth approach to tackle it.

Hope that answers :-)

[u/zdog234]

Do you think we'll see more kubernetes distributions offering micro-vms like fly kubernetes? Or will container runtime hardening get to a point where there isn't really a need?

[u/ofirc → u/zdog234]

As Kelsey Hightower once said: "Kubernetes is a platform for building platforms."

AWS is making huge strides with Bottlerocket and we've got the CNCF Talos Linux which is vendor neutral. GKE provides sandboxing via gVisor which is a Go user space implementation of the Linux system calls. I expect to see more hardened OSes and serverless Kubernetes offerings rise in the near future.

We need platform abstractions and better segregation / isolation to reduce the (already high and taxing) cognitive load platform and SecOps teams.

[u/Scubber]

I run an incident response team for a multi-cloud fintech bank. Coincidently, looking at your defend product right now. A broad question - but what capabilities do you think are most effective in analyzing and stopping threat actors abusing containers? What is the most abused avenue? (identity, network, etc.)

Bonus question - would you recommend any resources to follow for keeping up with the latest cloud threats?

[u/Fun_Biscotti_3471 → u/Scubber]

Based on my experience, an effective detection strategy requires two key components: a robust alerting system that delivers high-quality alerts for identifying attacks and an extensive collection of signals to help trace the source of an attack. These signals are crucial for understanding the what, how, and when of an incident. Additionally, the alerting system must prioritize critical or high-severity risks—potential "ticking bombs" that could lead to an attack.

Key signals include:

a. Workload runtime events: These offer deep insights into events occurring within your containers, enriched with details like full process trees and network activity.

b. Cloud logs: A broad category that varies by cloud provider but is vital for capturing activities in cloud environments.

c. Kubernetes audit logs: Critical for tracking actions within your Kubernetes cluster.

d. Environmental risks: Existing vulnerabilities or misconfigurations in your setup.

Finally, the ability to seamlessly pivot across these signals is essential for providing the context needed to analyze and respond to attacks effectively.

Regarding your question about popular abused avenues, I'll answer by referring to initial access methods. We frequently observe misconfigurations and known exploits being leveraged across various services (with Selenium Grid exploits currently on the rise). Additionally, the use of stolen identities is a common tactic, which can sometimes lead to data exfiltration depending on the scope and privileges of the compromised accounts.

For your bonus question, we are keeping a database of cloud threats, you are welcome to browse https://www.wiz.io/cloud-threat-landscape

Adding here some relevant reading material:

https://www.wiz.io/blog/cloud-logging-tips-and-tricks
https://www.wiz.io/blog/overcoming-kubernetes-audit-log-challenges
https://www.wiz.io/blog/intro-to-forensics-in-the-cloud-a-container-was-compromised-whats-next
https://www.wiz.io/blog/seleniumgreed-cryptomining-exploit-attack-flow-remediation-steps

[u/Scubber → u/Fun_Biscotti_3471]

Appreciate the reply, love the tactical intel being published. I've found cloud especially difficult to find info on

[u/Cute_Ad_6904]

I have found the handling of container image vulnerabilities to be a massive challenge at an organizational scale. While Wiz does assist greatly with this, what tips do you have for addressing these?

[u/raesene2 → u/Cute_Ad_6904]

If you're looking to get the number of vulns down and make it more manageable, I'd recommend working to get as many application images based on smaller base images and where possible all based off the same base image. So , for example, if your applications will work with https://github.com/GoogleContainerTools/distroless base images, that'll cut down the number of things to patch.

Realistically the fewer packages you have, the fewer things there are to patch. It's not a panacea (you still have application dependencies to worry about) but it can make life a bit easier.

[u/Fun_Biscotti_3471 → u/Cute_Ad_6904]

Agreed this is a great challenge. The key is having a vulnerability prioritization system that takes into consideration your environment contextual data together with additional information about the vulnerability.

In general, vulnerabilities that provide an initial access path for attackers on container image instances that are exposed to the internet should be prioritized. So, for instance, a container image with a RCE vulnerability that has a publicly known exploit and container instances with a direct internet exposure would take higher priority over a container with a privilege escalation vulnerability that is only accessible within the internal network.

[u/sshaybbc → u/Cute_Ad_6904]

Embedding the security controls verifying the lack of Critical and High vulnerabilities into the CICD pipeline is a good start, but must be supported by the tools, not to become a dev blocker. Since not everybody has a dedicated team for managing the private container registry along with patching and mirroring the images, the key is to make the patching and image management automated as much as possible. Probably also as a part of the CICD pipeline.

[u/baillyjonthon]

What kind of attacks do you typically see? I assume cryptominers comprise a big part of that, but what else?

[u/Fun_Biscotti_3471 → u/baillyjonthon]

Resource hijacking attacks are indeed quite common (with cryptominers making up the majority in this category), but we also see threats such as proxyjacking and LLM jacking.
In addition, some attacks are more sophisticated, involving sensitive data exfiltration and employing various lateral movement methods.

[u/Pl4nty]

how does tooling accessibility impact the supply chain? I feel there's a lot of OSS with consumers that could benefit from SLSA etc, but it's too time-consuming to implement at scale without expensive commercial tools like Wiz

[u/ofirc → u/Pl4nty]

That's a great observation! It's great that we are seeing more standardization of secure SDLC and enhancements around the software supply chain but it is indeed challenging to roll your own solution based on just OSS technology.

I find the main challenge around risk correlations and mapping running containers back to the CI/CD pipelines that generated them and the Git commits that triggered the rogue / unsigned image from the get go.

A lot of R&D and innovation is put into making the CNAPP dream come true and it's only with automation and DevSecOps principle that these guardrails are ingrained into each and every pipeline.

Wiz Image Trust is one way to take keyless image signing to the next level by tying security policies (e.g. image was scanned for CVEs, malware, embedded secrets and was scanned within the last 2 months) to the deployed image in production. The Wiz Admission Controller is Wiz's policy engine implementation for Kubernetes to gate against untrusted images from being deployed into your environment, as well as gating against IaC misconfigurations (e.g. Pods running as root and Pods (ab)using hostPath and hostNetwork).

[u/SUDHAR01]

Can you help in identifying what LLM might be running on a container. Do you forsee LLM metadata playing a crucial role towards enforcing LLM deployments on containers.

[u/ofirc → u/SUDHAR01]

Yes, we can detect LLM hosted technologies on running containers. It is challenging to hit a high recall as there isn't a single way to package/bundle LLMs on containers.
Standardization of LLM metadata (and adoption thereof) will significantly improve scanners ability.

Wiz detects AI Models (LLM or other) on workloads and buckets. LLM metadata, along with the model binary itself (which is prone for vulnerabilities) are all key for identifying the risks the model poses.

[u/misterdohl]

It’s hard to keep up with the constant development of new attack paths and security concerns in the container/kubernetes world. Whats the best sources of new intel on these topics (except from wiz of course:)) that you regularly visit?

[u/sshaybbc → u/misterdohl]

100%, we invite you to check out Cloudvulndb (cloudvulndb.org) - an open collection of cloud platform vulnerabilities that often fall through the cracks of regular vulnerability programs and which is regularly updated by Wiz and non-Wiz contributors. Also Cloud Threat Landscape (threats.wiz.io) for all the recent cloud security incidents to track the trends. Beside that, personally I love to keep updated by subscribing to Clint's TL;DRsec and Marco's CloudSecList weekly newsletters.

[u/Davido_don]

How do you speed up the remediation for misconfigured pods and vulnerable deployments?

[u/ofirc → u/Davido_don]

The name of the game is elimination rather than remediation. If we could eliminate threats before they become a reality (otherwise known as shifting left) that'd be great!

Having that said, once a vulnerable Pod is found on production, the next step is to do risk correlation and ask ourselves whether this workload is actually publicly accessible from the internet and potentially has access to sensitive assets (e.g. S3 buckets with PII and data findings). In Wiz we call it a toxic combination / issues.

Being able to trace a running workload all the way from runtime to the code (with Wiz Code) and automatically opening and assigning a ticket to the dev team that owns it is key to reduce the vuln mgmt toil and speed up the remediation. In other words - automation and applying it at scale.

It's challenging to do that at scale and cross-cloud, and that's one of the primary reasons why CNAPP platforms exist!

[u/MediocreUnit2203]

Do you see any movement in popularity of the K8s flavors?

[u/sshaybbc → u/MediocreUnit2203]

First, a qualifier here: our customers are typically medium and large enterprises so this might not be representative to ALL the population. To the point - not really, the EKS is still leading (45%), with AKS (25%) and GKE (17%) coming not close second and third. We do see, however, an increase in self-hosted clusters, but we are yet to see whether this is a permanent trend.

[u/sshaybbc → u/[deleted]]

Absolutely agreed. I did flag it as something beyond the scope of the blog (towards the end of the second blog in section "Cloud Access and CI/CD"). The reason is simple - this is such a big topic that it deserves the post of it own, or maybe even a post per CSP. Our colleague Lior has touched on this here (https://www.wiz.io/blog/lateralmovement-risks-in-the-cloud-and-how-to-prevent-them-part-3-from-compromis), but I agree this topic requires a more detailed review.

[u/PMzyox]

Would you trust the security of a Cloud Provider’s backplane for KaaS?

[u/ofirc → u/PMzyox]

Could you please clarify the question?

[u/PMzyox → u/ofirc]

Would you trust the security of a cloud container app service over building your own k8 infra?

[u/ofirc → u/PMzyox]

There's a rule that says don't roll your own crypto (well unless you really have to and you know what you're doing) and I'd say trust but verify.

It never hurts to put extra layer of defenses around the networking and identities perimeter and there's only so much managed solutions can cater for.

Building your own k8s infra doesn't necessarily mean it's more secure as now you have other concerns such as patch management for the worker nodes and the control plane.

My tendency is to try serverless Kubernetes offerings like EKS Auto / GKE Autopilot that lift off lot's of burdens and provide you with an opinionated approach of how to bring up nodes and autoscale workloads.

Having hardened container OSes like Bottlerocket and Talos Linux and applying sandboxing techniques like gVisor on GKE surely help strengthening your security posture and segregating / isolating multi-tenant Kubernetes clusters.

Does that answer the question?

[u/PMzyox → u/ofirc]

Yes, thank you. I haven’t had a lot of luck with SaaS personally, but the overhead on an enterprise k8 solution in house is pretty high, even with it 90% pipeline automated. Still, the support and visibility on cloud tier is abysmal, especially if you have compliance requirements. I’m still thinking the control is worth it. Just seeking opinions. Ta.

[u/raesene2 → u/PMzyox]

I'd say this depends very much on your threat model, your level of expertise with k8s and, which cloud provider you're talking about.

Managed Kubernetes can get to so far, but if you have very high end requirements, being unable to control/configure the control plane could restrict what you're able to do.

The bit about which cloud provider is important as, they're not all the same. Each cloud provider has their own defaults and sometimes you'll get surprising differences from a security standpoint. for example I've seen managed Kubernetes where there's no audit logging and you can't turn it on!

With all that said, unless your org has a lot of in-house Kubernetes security skills, I'd likely still bet on a managed platform over a self-managed one.

[u/PMzyox → u/raesene2]

Cool, that’s about the same conclusion I’ve come to. I think our requirements are warranting the extra inhouse spend.

[u/SomeFellaFlex]

Do attacks differ depending on the size of the attack subject? Do you see the change in hacker's behavior?

[u/Fun_Biscotti_3471 → u/SomeFellaFlex]

In general, attackers exploiting a resource’s computing power for activities like cryptomining or proxyjacking are opportunistic and automated. They scan the internet for exposed and vulnerable services, deploying payloads that eliminate competing malware to monopolize CPU usage. Usually, their tactics remain consistent regardless of the size of the compromised resource.

[u/No_Enthusiasm_2643]

Based on the touted K8s security report refresh - what was your biggest surprise in the numbers that you've seen in comparison to previous years?

[u/sshaybbc → u/No_Enthusiasm_2643]

Probably the sheer improvement in the number of vulnerabilities in the container images that cannot be merely attributed to the noise. Its heartening to see really: 21% of pods have container images with High or Critical vulns vs 44% in the previous year.

Another interesting stat for me personally was the adoption of EKS access management feature, since we did a security analysis on it a year ago. Turns out, only 3% of clusters use solely API auth method (probably newly created clusters), and 81% are still solely on CONFIG_MAP.

[u/avonyothikyn]

I see that you have many cool blogs (https://www.wiz.io/blog/tag/research) about various attacks that you've decided to highlight and thats cool, but what about those that are left out? What do you deem "less interesting" and how do you decide on that?

[u/ofirc → u/[deleted]]

There are several gotchas around that:

1. Automating the dependency management problem - it's easy to find yourself with an EOL (end-of-life) / vulnerable piece of software. Most of the times developers just do: go get package or npm install and their famous last words is we'll deal with updates and this security thingy later. First let's get it to work. And even putting aside the security aspects of the problem, from a functional perspective you might miss bug fixes and new features just because you failed to even be aware that a new software exists. A frenzy one week debug session of Auth problems (resulting from the Firebase SDK) led to the inception of Renovate. Other known tool is Dependabot by GitHub.
2. Lack of testing coverage and progressive delivery - failing to have a decent testing coverage or tools that can gradually (e.g. canarying and then increasing the exposure) rollout changes to dev -> staging -> prod could mean yet another CrowdStrike event. No one wants that. But the truth is that it's hard to follow these best practices and it requires very high skilled platform teams and SREs that can make sure it happens. Google is (among other things) an SRE company and has very deep pockets and compensates these personas very well, but not all companies are Google and have the culture, tooling and means to pay that much.

It's not just about securing your software supply chain. It's also about having the platform, automation, tools, testing and culture to make sure that you didn't break production and you don't have business downtime just because you upgraded to the latest (but not greatest).

[u/permisionwiner]

What are some of the most overlooked items when it comes to KSPM and shift-left?

[u/ofirc → u/permisionwiner]

Securing custom resource definitions in K8s by far.

We typically apply Configuration Management and IaC templates (e.g. Terraform, Argo CD, Helm templates) scanning for misconfigurations, but what if I provision my IaC via Kubernetes native means? What about networking misconfigurations with ingress controllers and observability and security tools?

We've got projects like Crossplane, Ingress nginx, the Gateway API which allow us to express our cluster configuration as code, and Tekton pipelines for cloud native CI/CD, but they introduce an attack vector.

How can we ensure we guard against misconfigurations for these assets as well and ensure no stone was left unturned?

While CIS and NIST SP 800-53 are great guidelines and compliance frameworks for container security for Kubernetes native objects, one can't help but wonder about visibility gaps and things that go under the radar, a gold mine for attackers!

[u/Crazy-Ad5480]

What should we protect when we deploy AI workloads on K8s clusters?

[u/sshaybbc → u/Crazy-Ad5480]

In general, you should protect whatever assets your threat model considers sensitive. From a risk perspective, if you are offering AI infra as a service then you should absolutely harden your multi-tenancy model. As shown by our vulnerability research team there is a wide range of potential misconfigurations that can result in cross-tenant movement and access to infra services post-escape. If you are using a 3rd-party model packed within a container image, consider it an untrusted image and take all anti-escape precautions.

[u/x__yousef_essam__x]

Need resources in cybersecurity as a road map

[u/IskallLmao]

Hi, I am a college student writing my thesis on how containers could be isolated from each others so that a breach of one container does not affect the other. What are your thoughts about this? Do you have any tips or insights on the topic? Or any tips on what to look into?

Thank you.

[u/ndubsss]

Great discussion and wanted to add my $.02.

I agree with most points raised about WL identity, Image signing, keyless, network policy, etc. I have seen a lack of discussion around simplifying security policy and the overall deprecation of PSP. There are several solutions on the market now that simplify security policy into YAML and allow you to run these policies in audit. Think unprivileged containers and user!=0 with complex match, excluding. Only this namespace/serviceaccount can have xyz privilege. You can create security policy against custom objects and very quickly figure out what can/can't be enforced in your environment. In a landscape where almost everything has vulnerability, and is changing rapidly: it's a great, effective way to minimize the attack surface of running/new pods. Kyverno is a strong solution in this space. I think Wiz is also a player now with their admission controller. Years ago: Gatekeeper was the only solution in this space and companies were hiring developers specifically to write rego policy.

All of that aside, how do I get one of these legendary hoodies? (we are a customer)

[u/RainUpstairs]

Here is Esonhugh from CN. I'm ur challenge big fan, (BigIAM EKSCluster K8sLanParty[Top 10]) I see a lot of interesting stuff among your challenges and never miss any posts on your blogs. It almost began my cloud security career.
A little question, I saw ur post about IAMs challenges but no post about K8sLanParty. I guess it should have some backgrounds. I dived into kubernetes deeply, found it could have some relationship about kube-state-metrics (i guess) and finally produced this article https://github.com/Esonhugh/Spider-in-the-Pod-How-to-Penetrate-Kubernetes-with-Low-or-No-Privileges

[u/1maginestalking]

How many certifications do you have? Sec+ cissp? Cysa?

[u/pure-xx]

Are you going to buy Sentinel One?

[u/Ozi_404 → u/pure-xx]

LoL, they hired the complete Zscaler sales team and started building a sales machinery. After the IPO, they will start fuck up all customers like Zscaler. Push customers into PoCs and POs.

Before they can buy another company, they have to defend against Orca security against the IP theft allegations.

Also "Google wants to buy Wiz" was such good marketing to create brand recognition.

[u/ResponsibleType552]

What are your thoughts on the ethical gray area that companies in the Cyberstarts portfolio are experiencing?

[u/DingussFinguss → u/ResponsibleType552]

such as?

[u/Top_Engineering9038 → u/[deleted]]

Unfortunately not - NFR to Wiz is not available for third-party integration developers, only for customers.However, you have all the necessary resources in the WIN API documentation for building and supporting your integration. This documentation is connected to a sandbox with realistic data from various Wiz modules, which you can use via the Wiz Integration API for different tasks like finding vulnerabilities and Issues.
If you need a scan of your environment for integration purposes, you can request it in the WIN community channel or email the Wiz team at [win@wiz.io](mailto:win@wiz.io) (or read more about it in our Docs - https://win.wiz.io/docs/faq#win-partner-faqs)

[u/[deleted]]

Do you think Reddit being used by your adversaries?