Suggested Reads

This month’s influential articles dive into discriminatory behavior along two different axes.

This posts clarifies the clucking and clamoring over attackers exploiting vulns or corrupting build pipelines (spoiler alert: it isn’t worth their time and effort to).

Tailscale Funnel, a tool that lets you share a web server on your private tailnet with the public internet, is now available as a beta feature for all users. With Funnel enabled, you can share access to a local development server, test a webhook, or even host a blog.

Run your GitHub Actions locally 🚀. Contribute to nektos/act development by creating an account on GitHub.

Authors : Craig Box (ARMO), Ben Hirschberg (ARMO) Admission control is an important part of the Kubernetes control plane, with several internal features depending on the ability to approve or change an API object as it is submitted to the server. It is also useful for an administrator to be able to define business logic, or policies, regarding what objects can be admitted into a cluster. To better support that use case, Kubernetes introduced external admission control in v1.7 . In addition to countless custom, internal implementations, many open source projects and commercial solutions implement admission controllers with user-specified policy, including Kyverno and Open Policy Agent’s Gatekeeper . While admission controllers for policy have seen adoption, there are blockers for their widespread use. Webhook infrastructure must be maintained as a production service, with all that entails. The failure case of an admission control webhook must either be closed, reducing the availability of the cluster; or open, negating the use of the feature for policy enforcement. The network hop and evaluation time makes admission control a notable component of latency when dealing with, for example, pods being spun up to respond to a network request in a "serverless" environment. Validating admission policies and the Common Expression Language Version 1.26 of Kubernetes introduced, in alpha, a compromise solution. Validating admission policies are a declarative, in-process alternative to admission webhooks. They use the Common Expression Language (CEL) to declare validation rules. CEL was developed by Google for security and policy use cases, based on learnings from the Firebase real-time database. Its design allows it to be safely embedded into applications and executed in microseconds, with limited compute and memory impact. Validation rules for CRDs introduced CEL to the Kubernetes ecosystem in v1.23, and at the time it was noted that the language would suit a more generic implementation of validation by admission control. Giving CEL a roll - a practical example Kubescape is a CNCF project which has become one of the most popular ways for users to improve the security posture of a Kubernetes cluster and validate its compliance. Its controls — groups of tests against API objects — are built in Rego , the policy language of Open Policy Agent. Rego has a reputation for complexity, based largely on the fact that it is a declarative query language (like SQL). It was considered for use in Kubernetes, but it does not offer the same sandbox constraints as CEL. A common feature request for the project is to be able to implement policies based on Kubescape’s findings and output. For example, after scanning pods for known paths to cloud credential files , users would like the ability to enforce policy that these pods should not be admitted at all. The Kubescape team thought this would be the perfect opportunity to try and port our existing controls to CEL and apply them as admission policies. Show me the policy It did not take us long to convert many of our controls and build a library of validating admission policies . Let’s look at one as an example. Kubescape’s control C-0017 covers the requirement for containers to have an immutable (read-only) root filesystem. This is a best practice according to the NSA Kubernetes hardening guidelines , but is not currently required as a part of any of the pod security standards . Here's how we implemented it in CEL: apiVersion : admissionregistration.k8s.io/v1alpha1 kind : ValidatingAdmissionPolicy metadata : name : "kubescape-c-0017-deny-resources-with-mutable-container-filesystem" spec : failurePolicy : Fail matchConstraints : resourceRules : - apiGroups : ["" ] apiVersions : ["v1" ] operations : ["CREATE" , "UPDATE" ] resources : ["pods" ] - apiGroups : ["apps" ] apiVersions : ["v1" ] operations : ["CREATE" , "UPDATE" ] resources : ["deployments" ,"replicasets" ,"daemonsets" ,"statefulsets" ] - apiGroups : ["batch" ] apiVersions : ["v1" ] operations : ["CREATE" , "UPDATE" ] resources : ["jobs" ,"cronjobs" ] validations : - expression : "object.kind != 'Pod' || object.spec.containers.all(container, has(container.securityContext) && has(container.securityContext.readOnlyRootFilesystem) && container.securityContext.readOnlyRootFilesystem == true)" message : "Pods having containers with mutable filesystem not allowed! (see more at https://hub.armosec.io/docs/c-0017)" - expression : "['Deployment','ReplicaSet','DaemonSet','StatefulSet','Job'].all(kind, object.kind != kind) || object.spec.template.spec.containers.all(container, has(container.securityContext) && has(container.securityContext.readOnlyRootFilesystem) && container.securityContext.readOnlyRootFilesystem == true)" message : "Workloads having containers with mutable filesystem not allowed! (see more at https://hub.armosec.io/docs/c-0017)" - expression : "object.kind != 'CronJob' || object.spec.jobTemplate.spec.template.spec.containers.all(container, has(container.securityContext) && has(container.securityContext.readOnlyRootFilesystem) && container.securityContext.readOnlyRootFilesystem == true)" message : "CronJob having containers with mutable filesystem not allowed! (see more at https://hub.armosec.io/docs/c-0017)" Match constraints are provided for three possible API groups: the core/v1 group for Pods, the apps/v1 workload controllers, and the batch/v1 job controllers. Note: matchConstraints will convert the API object to the matched version for you. If, for example, an API request was for apps/v1beta1 and you match apps/v1 in matchConstraints, the API request will be converted from apps/v1beta1 to apps/v1 and then validated. This has the useful property of making validation rules secure against the introduction of new versions of APIs, which would otherwise allow API requests to sneak past the validation rule by using the newly introduced version. The validations include the CEL rules for the objects. There are three different expressions, catering for the fact that a Pod spec can be at the root of the object (a naked pod ), under template (a workload controller or a Job), or under jobTemplate (a CronJob). In the event that any spec does not have readOnlyRootFilesystem set to true, the object will not be admitted. Note: In our initial release, we have grouped the three expressions into the same policy object. This means they can be enabled and disabled atomically, and thus there is no chance that a user will accidentally leave a compliance gap by enabling policy for one API group and not the others. Breaking them into separate policies would allow us access to improvements targeted for the 1.27 release, including type checking. We are talking to SIG API Machinery about how to best address this before the APIs reach v1 . Using the CEL library in your cluster Policies are provided as Kubernetes objects, which are then bound to certain resources by a selector . Minikube is a quick and easy way to install and configure a Kubernetes cluster for testing. To install Kubernetes v1.26 with the ValidatingAdmissionPolicy feature gate enabled: minikube start --kubernetes-version= 1.26.1 --extra-config= apiserver.runtime-config= admissionregistration.k8s.io/v1alpha1 --feature-gates= 'ValidatingAdmissionPolicy=true' To install the policies in your cluster: # Install configuration CRD kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/policy-configuration-definition.yaml # Install basic configuration kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/basic-control-configuration.yaml # Install policies kubectl apply -f https://github.com/kubescape/cel-admission-library/releases/latest/download/kubescape-validating-admission-policies.yaml To apply policies to objects, create a ValidatingAdmissionPolicyBinding resource. Let’s apply the above Kubescape C-0017 control to any namespace with the label policy=enforced : # Create a binding kubectl apply -f - EOT apiVersion: admissionregistration.k8s.io/v1alpha1 kind: ValidatingAdmissionPolicyBinding metadata: name: c0017-binding spec: policyName: kubescape-c-0017-deny-mutable-container-filesystem matchResources: namespaceSelector: matchLabels: policy: enforced EOT # Create a namespace for running the example kubectl create namespace policy-example kubectl label namespace policy-example 'policy=enforced' Now, if you attempt to create an object without specifying a readOnlyRootFilesystem , it will not be created. # The next line should fail kubectl -n policy-example run nginx --image= nginx --restart= Never The output shows our error: The pods "nginx" is invalid: : ValidatingAdmissionPolicy 'kubescape-c-0017-deny-mutable-container-filesystem' with binding 'c0017-binding' denied request: Pods having containers with mutable filesystem not allowed! (see more at https://hub.armosec.io/docs/c-0017) Configuration Policy objects can include configuration, which is provided in a different object. Many of the Kubescape controls require a configuration: which labels to require, which capabilities to allow or deny, which registries to allow containers to be deployed from, etc. Default values for those controls are defined in the ControlConfiguration object . To use this configuration object, or your own object in the same format, add a paramRef.name value to your binding object: apiVersion : admissionregistration.k8s.io/v1alpha1 kind : ValidatingAdmissionPolicyBinding metadata : name : c0001-binding spec : policyName : kubescape-c-0001-deny-forbidden-container-registries paramRef : name : basic-control-configuration matchResources : namespaceSelector : matchLabels : policy : enforced Summary Converting our controls to CEL was simple, in most cases. We cannot port the whole K...

Kubernetes SRE superpowers with AI

Developers and compliance teams get a new SBOM generation tool for cloud repositories.

You can build containers quickly using familiar tools and still end up shipping images with tiny attack surfaces.

Use any linux distribution inside your terminal. Enable both backward and forward compatibility with software and freedom to use whatever distribution you’re more comfortable with. Mirror available...

Developer News

Getting started in any open-source community can be daunting, especially if it’s a big one like Kubernetes. I wrote this post to share my experience and encourage others to join up. All it takes is some curiosity and a willingness to show up! Here’s how my journey unfolded at a high level: What am I interested in? Is there a SIG (Special Interest Group) or a WG (Working Group) that is dedicated to that topic, or something similar? Sign up for their mailing list and start hopping on meetings. When (never if!) there are opportunities to help out and it aligns with your skills and desired growth areas, raise your hand. Ask for lots of help and don’t be shy about not knowing everything (or anything!) Keep plugging along, even if progress isn’t as fast as you would like it to be. Starting up First things first. What are you interested in learning more about? There are so many wonderful SIGs and working groups in the Kubernetes community: there’s something for everyone. And continuing to show up and participate will be so much easier if you think what you are doing is interesting. Likewise, continued participation is what keeps the community thriving, so that interest will drive you to have more of an impact. Also: it’s ok to show up knowing nothing! I remember showing up knowing very little about Kubernetes or how the community itself worked. And while I know more about how the community functions today, I am still learning all the time about it and the project. Fortunately, the community is full of friendly people who want to help you learn. Learning as you go is expected and celebrated. When you raise your hand to do something, even if you know nothing, people will cheer and help you along the way. This method was my exact story. It was my first or second meeting with SIG Security , and Pushkar Joglekar mentioned that he needed a lead for a subproject he was creating after having done a security assessment of Cluster API . Everyone was so friendly in the meeting that I thought, “Hey, why not try it out?” And since then, I have received so much support and encouragement from my co-leads who are delighted to have me, especially because I am a beginner; new participation is what keeps the community healthy. Always learning My participation has also been a great learning experience on several fronts. First, I have been exposed to techniques for how to build community consensus. It’s simple stuff: show up at other SIG or working group meetings, share your ideas or where you are looking for help, find people who are interested and have the knowledge to help, build an action plan together, do it, and share as you execute. But the other thing that I’m learning is that building this consensus and executing it in a transparent, inviting way simply takes time. I also have to be patient with myself and remember that I am learning as I go. The Kubernetes git repo can be daunting to navigate. Knowing the next best step isn’t always obvious. But this is where my third learning curve, how to engage the community to get what I need, comes into play. It turns out that asking questions in the Kubernetes Slack workspace and bringing my topics to the SIG Security meetings when I need help is an amazing way to get what I need! Again, simple stuff, but until you do it, it’s not always obvious. Why you - a beginner - are important to the project In many ways, beginners are the most important part of the community. To put a finer point on it: asking for, receiving, and then giving help is a very relevant part of how the community grows and flourishes. When we take on and then pass on knowledge, we ensure that the community grows enough to keep supporting the needs of the people who rely on the project, whatever it is. You have superpowers as a beginner! I hope people who read this post have their curiosity peaked about getting involved in the community. It may seem scary. My experience has been such that, about halfway through your first step, you realize there are loads of people here who want to help you learn and are excited for you expressing interest and trying to participate, and the fear melts away. Sure, I’m still uncertain about a few things, but I know the community has my back and will support my growth. Come on in, that water’s fine!

How can you identify whether a potential employer has a burnout culture? Will the company support your well-being and productivity? Or will they leave you exhausted and looking for a new job again? The authors offer signals to look for, questions to ask, and ways to evaluate answers during your interview process.

I joined Javier Perez on a webinar reviewing the results of the 2023 State of Open Source survey, a collaborative effort between OpenLogic by Perforce and the Open Source Initiative (OSI). Open Source users from all eight global regions, working in 20+ industries in organizations of all sizes were anonymously surveyed. The resulting report is about

Alibaba said Tuesday it will split its company into six business groups, each with the ability to raise outside funding and go public.

Rewritten from scratch with native technology

GitHub, the popular developer platform, has laid off virtually its entire engineering team in India.

As an essential tool for managing servers, SSH (Secure Shell) provides a secure way to remotely access a server's command line. However, for best SSH

Bradley Curran. Senior Software Engineer.

A Guide to Selecting the Right Hardware for Self-Hosting Services

We are now less than 6 months away from the End Of Life (EOL) date for the OpenSSL 1.1.1 series. Users of OpenSSL 1.1.1 should consider their options …

null

With Raleigh-based First Citizens BancShares acquiring much of the remains of Silicon Valley Bank, the North Carolina region is positioned for a higher profile.