Shrink to Secure: Kubernetes and Secure Compact Containers
Reduce your Kubernetes container size to improve security.
These two concepts seem very much unrelated in other technical fields but when talking about Kubernetes containers you can reduce the first while improving the latter.
For once you don’t have to choose between security and performance. “You can have your cake and eat it too”.
More Michigan teens hit the brakes on learning to drive | Bridge Michigan
The state’s decision to stop funding driver’s education and high private driver’s education costs may be contributing to many Michigan teens delaying getting their driver’s licenses.
Bluesky is intervening in user’s mental health and other existential crisis messages on the platform. Imagine some junior dev contacting someone via email who has a gun to their head instead of letting the message out on to the platform…
I started working on Flyaway with the intention of becoming familiar with Wayland, its protocols and extensions, and the wlroots library. Instead, I ended up genuinely liking all three.
Hackers claim to take down Russian satellite communications provider
A group of previously unknown hackers has claimed responsibility for a cyberattack on the Russian satellite communications provider Dozor-Teleport, which is used by energy companies and the country's defense and security services.
From IBM to Databricks, It’s Been A Big Week For Tech M&A
There have been at least 23 acquisitions this week of companies that previously raised known venture or seed funding, Crunchbase data shows. Does that mean an M&A boom is finally here?
U.S. antitrust regulator plans to target Amazon's online marketplace, Bloomberg reports
The U.S. antitrust regulator is planning to file a suit targeting Amazon.com's core online marketplace in the coming weeks, Bloomberg News reported on Thursday, citing documents and people familiar with the matter.
We should figure out how to do a random mass private event the two weeks before the end of Reddit’s quarterly earnings | Reddit is telling protesting mods their communities ‘will not’ stay private
I remember when I took the reigns of the network at my last AFB. A physical and virtual security assessment was done and holy shit, y’all | Exposed Interfaces in US Federal Networks: A Breach Waiting to Happen
Follow us on Twitter @Hackread - Facebook @ /Hackread
Blog: Verifying Container Image Signatures Within CRI Runtimes
Author : Sascha Grunert
The Kubernetes community has been signing their container image-based artifacts
since release v1.24. While the graduation of the corresponding enhancement
from alpha to beta in v1.26 introduced signatures for the binary artifacts,
other projects followed the approach by providing image signatures for their
releases, too. This means that they either create the signatures within their
own CI/CD pipelines, for example by using GitHub actions, or rely on the
Kubernetes image promotion process to automatically sign the images by
proposing pull requests to the k/k8s.io repository. A requirement for
using this process is that the project is part of the kubernetes or
kubernetes-sigs GitHub organization, so that they can utilize the community
infrastructure for pushing images into staging buckets.
Assuming that a project now produces signed container image artifacts, how can
one actually verify the signatures? It is possible to do it manually like
outlined in the official Kubernetes documentation . The problem with this
approach is that it involves no automation at all and should be only done for
testing purposes. In production environments, tools like the sigstore
policy-controller can help with the automation. These tools
provide a higher level API by using Custom Resource Definitions (CRD) as
well as an integrated admission controller and webhook to verify
the signatures.
The general usage flow for an admission controller based verification is:
A key benefit of this architecture is simplicity: A single instance within the
cluster validates the signatures before any image pull can happen in the
container runtime on the nodes, which gets initiated by the kubelet. This
benefit also brings along the issue of separation: The node which should pull
the container image is not necessarily the same node that performs the admission. This
means that if the controller is compromised, then a cluster-wide policy
enforcement can no longer be possible.
One way to solve this issue is doing the policy evaluation directly within the
Container Runtime Interface (CRI) compatible container runtime. The
runtime is directly connected to the kubelet on a node and does all
the tasks like pulling images. CRI-O is one of those available runtimes
and will feature full support for container image signature verification in v1.28.
How does it work? CRI-O reads a file called policy.json , which
contains all the rules defined for container images. For example, you can define a
policy which only allows signed images quay.io/crio/signed for any tag or
digest like this:
{
"default" : [{ "type" : "reject" }],
"transports" : {
"docker" : {
"quay.io/crio/signed" : [
{
"type" : "sigstoreSigned" ,
"signedIdentity" : { "type" : "matchRepository" },
"fulcio" : {
"oidcIssuer" : "https://github.com/login/oauth" ,
"subjectEmail" : "sgrunert@redhat.com" ,
"caData" : "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUI5ekNDQVh5Z0F3SUJBZ0lVQUxaTkFQRmR4SFB3amVEbG9Ed3lZQ2hBTy80d0NnWUlLb1pJemowRUF3TXcKS2pFVk1CTUdBMVVFQ2hNTWMybG5jM1J2Y21VdVpHVjJNUkV3RHdZRFZRUURFd2h6YVdkemRHOXlaVEFlRncweQpNVEV3TURjeE16VTJOVGxhRncwek1URXdNRFV4TXpVMk5UaGFNQ294RlRBVEJnTlZCQW9UREhOcFozTjBiM0psCkxtUmxkakVSTUE4R0ExVUVBeE1JYzJsbmMzUnZjbVV3ZGpBUUJnY3Foa2pPUFFJQkJnVXJnUVFBSWdOaUFBVDcKWGVGVDRyYjNQUUd3UzRJYWp0TGszL09sbnBnYW5nYUJjbFlwc1lCcjVpKzR5bkIwN2NlYjNMUDBPSU9aZHhleApYNjljNWlWdXlKUlErSHowNXlpK1VGM3VCV0FsSHBpUzVzaDArSDJHSEU3U1hyazFFQzVtMVRyMTlMOWdnOTJqCll6QmhNQTRHQTFVZER3RUIvd1FFQXdJQkJqQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUlkKd0I1ZmtVV2xacWw2ekpDaGt5TFFLc1hGK2pBZkJnTlZIU01FR0RBV2dCUll3QjVma1VXbFpxbDZ6SkNoa3lMUQpLc1hGK2pBS0JnZ3Foa2pPUFFRREF3TnBBREJtQWpFQWoxbkhlWFpwKzEzTldCTmErRURzRFA4RzFXV2cxdENNCldQL1dIUHFwYVZvMGpoc3dlTkZaZ1NzMGVFN3dZSTRxQWpFQTJXQjlvdDk4c0lrb0YzdlpZZGQzL1Z0V0I1YjkKVE5NZWE3SXgvc3RKNVRmY0xMZUFCTEU0Qk5KT3NRNHZuQkhKCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0="
},
"rekorPublicKeyData" : "LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFMkcyWSsydGFiZFRWNUJjR2lCSXgwYTlmQUZ3cgprQmJtTFNHdGtzNEwzcVg2eVlZMHp1ZkJuaEM4VXIvaXk1NUdoV1AvOUEvYlkyTGhDMzBNOStSWXR3PT0KLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0tCg=="
}
]
}
}
}
CRI-O has to be started to use that policy as the global source of truth:
sudo crio --log-level debug --signature-policy ./policy.json
CRI-O is now able to pull the image while verifying its signatures. This can be
done by using crictl (cri-tools) , for example:
sudo crictl -D pull quay.io/crio/signed
DEBU[…] get image connection
DEBU[…] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:quay.io/crio/signed,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
DEBU[…] PullImageResponse: &PullImageResponse{ImageRef:quay.io/crio/signed@sha256:18b42e8ea347780f35d979a829affa178593a8e31d90644466396e1187a07f3a,}
Image is up to date for quay.io/crio/signed@sha256:18b42e8ea347780f35d979a829affa178593a8e31d90644466396e1187a07f3a
The CRI-O debug logs will also indicate that the signature got successfully
validated:
DEBU[…] IsRunningImageAllowed for image docker:quay.io/crio/signed:latest
DEBU[…] Using transport "docker" specific policy section quay.io/crio/signed
DEBU[…] Reading /var/lib/containers/sigstore/crio/signed@sha256=18b42e8ea347780f35d979a829affa178593a8e31d90644466396e1187a07f3a/signature-1
DEBU[…] Looking for sigstore attachments in quay.io/crio/signed:sha256-18b42e8ea347780f35d979a829affa178593a8e31d90644466396e1187a07f3a.sig
DEBU[…] GET https://quay.io/v2/crio/signed/manifests/sha256-18b42e8ea347780f35d979a829affa178593a8e31d90644466396e1187a07f3a.sig
DEBU[…] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[…] Found a sigstore attachment manifest with 1 layers
DEBU[…] Fetching sigstore attachment 1/1: sha256:8276724a208087e73ae5d9d6e8f872f67808c08b0acdfdc73019278807197c45
DEBU[…] Downloading /v2/crio/signed/blobs/sha256:8276724a208087e73ae5d9d6e8f872f67808c08b0acdfdc73019278807197c45
DEBU[…] GET https://quay.io/v2/crio/signed/blobs/sha256:8276724a208087e73ae5d9d6e8f872f67808c08b0acdfdc73019278807197c45
DEBU[…] Requirement 0: allowed
DEBU[…] Overall: allowed
All of the defined fields like oidcIssuer and subjectEmail in the policy
have to match, while fulcio.caData and rekorPublicKeyData are the public
keys from the upstream fulcio (OIDC PKI) and rekor
(transparency log) instances.
This means that if you now invalidate the subjectEmail of the policy, for example to
wrong@mail.com :
jq '.transports.docker."quay.io/crio/signed"[0].fulcio.subjectEmail = "wrong@mail.com"' policy.json new-policy.json
mv new-policy.json policy.json
Then remove the image, since it already exists locally:
sudo crictl rmi quay.io/crio/signed
Now when you pull the image, CRI-O complains that the required email is wrong:
sudo crictl pull quay.io/crio/signed
FATA[…] pulling image: rpc error: code = Unknown desc = Source image rejected: Required email wrong@mail.com not found (got []string{"sgrunert@redhat.com"})
It is also possible to test an unsigned image against the policy. For that you
have to modify the key quay.io/crio/signed to something like
quay.io/crio/unsigned :
sed -i 's;quay.io/crio/signed;quay.io/crio/unsigned;' policy.json
If you now pull the container image, CRI-O will complain that no signature exists
for it:
sudo crictl pull quay.io/crio/unsigned
FATA[…] pulling image: rpc error: code = Unknown desc = SignatureValidationFailed: Source image rejected: A signature was required, but no signature exists
It is important to mention that CRI-O will match the
.critical.identity.docker-reference field within the signature to match with
the image repository. For example, if you verify the image
registry.k8s.io/kube-apiserver-amd64:v1.28.0-alpha.3 , then the corresponding
docker-reference should be registry.k8s.io/kube-apiserver-amd64 :
cosign verify registry.k8s.io/kube-apiserver-amd64:v1.28.0-alpha.3 \
--certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com \
--certificate-oidc-issuer https://accounts.google.com \
| jq -r '.[0].critical.identity."docker-reference"'
…
registry.k8s.io/kubernetes/kube-apiserver-amd64
The Kubernetes community introduced registry.k8s.io as proxy mirror for
various registries. Before the release of kpromo v4.0.2 , images
had been signed with the actual mirror rather than registry.k8s.io :
cosign verify registry.k8s.io/kube-apiserver-amd64:v1.28.0-alpha.2 \
--certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com \
--certificate-oidc-issuer https://accounts.google.com \
| jq -r '.[0].critical.identity."docker-reference"'
…
asia-northeast2-docker.pkg.dev/k8s-artifacts-prod/images/kubernetes/kube-apiserver-amd64
The change of the docker-reference to registry.k8s.io makes it easier for
end users to validate the signatures, because they cannot know anything about the
underlying infrastructure being used. The feature to set the identity on image
signing has been added to cosign via the flag sign --sign-container-identity as well and will be part of its upcoming release.
The Kubernetes image pull error code SignatureValidationFailed got recently added to
Kubernetes and will be available from v1.28. This error code allows
end-users to understand image pull failures directly from the kubectl CLI. For
example, if you run CRI-O together with Kubernetes using the policy which requires
quay.io/crio/unsigned to be signed, then a pod definition like this:
apiVersion : v1
kind : Pod
metadata :
name : pod
spec :
containers :
- name : container
image : quay.io/crio/unsigned
Will cause the SignatureValidationFailed error when applying the pod manifest:
kubectl apply -f pod.yaml
pod/pod created
kubectl get pods
NAME READY STATUS RESTARTS AGE
pod 0/1 SignatureValidationFailed 0 4s
kubectl describe pod pod | tail -n8
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 58s d...
%2Fcdn.vox-cdn.com%2Fuploads%2Fchorus_asset%2Ffile%2F24761812%2F236714_Google_RSS_Reader_HHerrera.jpeg?width=92&height=&ar=&mode=&format=avif&dpr=2)
%2Fcloudfront-us-east-2.images.arcpublishing.com%2Freuters%2FJYHFENJOKNI43HKETZ2TX4ML6I.jpg?width=92&height=&ar=&mode=&format=avif&dpr=2)
%2Fcdn.vox-cdn.com%2Fuploads%2Fchorus_asset%2Ffile%2F23985506%2Facastro_STK024_02.jpg?width=92&height=&ar=&mode=&format=avif&dpr=2)
%2Fcloudfront-us-east-2.images.arcpublishing.com%2Freuters%2FWHR22BARAROE7FI6QUOGU4PW3A.jpg?width=92&height=&ar=&mode=&format=avif&dpr=2)
