HeyCLI/heyCLI_client: HeyCLI allows you to use your command line terminal in natural language
HeyCLI allows you to use your command line terminal in natural language - HeyCLI/heyCLI_client: HeyCLI allows you to use your command line terminal in natural language
Silicon Valley Bank collapses after failing to raise capital | CNN Business
Silicon Valley Bank collapsed Friday morning after a stunning 48 hours in which a bank run and a capital crisis led to the second-largest failure of a financial institution in US history.
Blog: k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know
Authors : Bob Killen (Google), Davanum Srinivas (AWS), Chris Short (AWS), Frederico Muñoz (SAS
Institute), Tim Bannister (The Scale Factory), Ricky Sadowski (AWS), Grace Nguyen (Expo), Mahamed
Ali (Rackspace Technology), Mars Toktonaliev (independent), Laura Santamaria (Dell), Kat Cosgrove
(Dell)
On Monday, March 20th, the k8s.gcr.io registry will be redirected to the community owned
registry ,
registry.k8s.io .
TL;DR: What you need to know about this change
On Monday, March 20th, traffic from the older k8s.gcr.io registry will be redirected to
registry.k8s.io with the eventual goal of sunsetting k8s.gcr.io.
If you run in a restricted environment, and apply strict domain name or IP address access policies
limited to k8s.gcr.io, the image pulls will not function after k8s.gcr.io starts redirecting
to the new registry.
A small subset of non-standard clients do not handle HTTP redirects by image registries, and will
need to be pointed directly at registry.k8s.io.
The redirect is a stopgap to assist users in making the switch. The deprecated k8s.gcr.io registry
will be phased out at some point. Please update your manifests as soon as possible to point to
registry.k8s.io .
If you host your own image registry, you can copy images you need there as well to reduce traffic
to community owned registries.
If you think you may be impacted, or would like to know more about this change, please keep reading.
Why did Kubernetes change to a different image registry?
k8s.gcr.io is hosted on a custom Google Container Registry
(GCR) domain that was set up solely for the Kubernetes
project. This has worked well since the inception of the project, and we thank Google for providing
these resources, but today, there are other cloud providers and vendors that would like to host
images to provide a better experience for the people on their platforms. In addition to Google’s
renewed commitment to donate $3
million to support the project's
infrastructure last year, Amazon Web Services announced a matching donation during their Kubecon NA
2022 keynote in Detroit . This will provide a better experience
for users (closer servers = faster downloads) and will reduce the egress bandwidth and costs from
GCR at the same time.
For more details on this change, check out registry.k8s.io: faster, cheaper and Generally Available
(GA) .
Why is a redirect being put in place?
The project switched to registry.k8s.io last year with the 1.25
release ; however, most of
the image pull traffic is still directed at the old endpoint k8s.gcr.io. This has not been
sustainable for us as a project, as it is not utilizing the resources that have been donated to the
project from other providers, and we are in the danger of running out of funds due to the cost of
serving this traffic.
A redirect will enable the project to take advantage of these new resources, significantly reducing
our egress bandwidth costs. We only expect this change to impact a small subset of users running in
restricted environments or using very old clients that do not respect redirects properly.
What images will be impacted?
ALL images on k8s.gcr.io will be impacted by this change. k8s.gcr.io hosts many images beyond
Kubernetes releases. A large number of Kubernetes subprojects host their images there as well. Some
examples include the dns/k8s-dns-node-cache , ingress-nginx/controller , and
node-problem-detector/node-problem-detector images.
What will happen to k8s.gcr.io?
Separate from the the redirect, k8s.gcr.io will be frozen and will not be updated with new images
after April 3rd, 2023 . k8s.gcr.io
will not get any new releases, patches, or security updates. It will continue to remain available to
help people migrate, but it WILL be phased out entirely in the future.
I run in a restricted environment. What should I do?
For impacted users that run in a restricted environment, the best option is to copy over the
required images to a private registry or configure a pull-through cache in their registry.
There are several tools to copy images between registries;
crane is one
of those tools, and images can be copied to a private registry by using crane copy SRC DST . There
are also vendor-specific tools, like e.g. Google’s
gcrane , that
perform a similar function but are streamlined for their platform.
How can I check registry.k8s.io is accessible from my cluster?
To test connectivity to registry.k8s.io and being able to pull images from there, here is a sample
command that can be executed in the namespace of your choosing:
kubectl run hello-world --tty --rm -i --image=registry.k8s.io/busybox:latest sh
When you run the command above, here’s what to expect when things work correctly:
$ kubectl run hello-world --tty --rm -i --image=registry.k8s.io/busybox:latest sh
If you don't see a command prompt, try pressing enter.
/ # exit
Session ended, resume using 'kubectl attach hello-world -c hello-world -i -t' command when the pod is running
pod "hello-world" deleted
What kind of errors will I see if I’m impacted?
Errors may depend on what kind of container runtime you are using, and what endpoint you are routed
to, but it should present such as ErrImagePull , ImagePullBackOff , or a container failing to be
created with the warning FailedCreatePodSandBox .
Below is an example error message showing a proxied deployment failing to pull due to an unknown
certificate:
FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = Error response from daemon: Head “https://us-west1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: x509: certificate signed by unknown authority
How can I find which images are using the legacy registry, and fix them?
Option 1 : See the one line kubectl command in our earlier blog
post :
kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c
Option 2 : A kubectl krew plugin has been developed called
community-images ,
that will scan and report any images using the k8s.gcr.io endpoint.
If you have krew installed, you can install it with:
kubectl krew install community-images
and generate a report with:
kubectl community-images
For alternate methods of install and example output, check out the repo:
kubernetes-sigs/community-images .
Option 3 : If you do not have access to a cluster directly, or manage many clusters - the best
way is to run a search over your manifests and charts for "k8s.gcr.io" .
Option 4 : If you wish to prevent k8s.gcr.io based images from running in your cluster, example
policies for Gatekeeper and
Kyverno are available in the AWS EKS Best Practices
repository
that will block them from being pulled. You can use these third-party policies with any Kubernetes
cluster.
Option 5 : As a LAST possible option, you can use a Mutating
Admission Webhook
to change the image address dynamically. This should only be
considered a stopgap till your manifests have been updated. You can
find a (third party) Mutating Webhook and Kyverno policy in
k8s-gcr-quickfix .
I still have questions, where should I go?
For more information on registry.k8s.io and why it was developed, see registry.k8s.io: faster,
cheaper and Generally Available .
If you would like to know more about the image freeze and the last images that will be available
there, see the blog post: k8s.gcr.io Image Registry Will Be Frozen From the 3rd of April
2023 .
Information on the architecture of registry.k8s.io and its request handling decision
tree
can be found in the kubernetes/registry.k8s.io
repo .
If you believe you have encountered a bug with the new registry or the redirect, please open an
issue in the kubernetes/registry.k8s.io
repo . Please check if there is an issue already
open similar to what you are seeing before you create a new issue .
Authors: Adrian Reber (Red Hat)
In my previous article, Forensic container checkpointing in
Kubernetes , I introduced checkpointing in Kubernetes
and how it has to be setup and how it can be used. The name of the
feature is Forensic container checkpointing, but I did not go into
any details how to do the actual analysis of the checkpoint created by
Kubernetes. In this article I want to provide details how the
checkpoint can be analyzed.
Checkpointing is still an alpha feature in Kubernetes and this article
wants to provide a preview how the feature might work in the future.
Preparation
Details about how to configure Kubernetes and the underlying CRI implementation
to enable checkpointing support can be found in my Forensic container
checkpointing in Kubernetes article.
As an example I prepared a container image (quay.io/adrianreber/counter:blog )
which I want to checkpoint and then analyze in this article. This container allows
me to create files in the container and also store information in memory which
I later want to find in the checkpoint.
To run that container I need a pod, and for this example I am using the following Pod manifest:
apiVersion : v1
kind : Pod
metadata :
name : counters
spec :
containers :
- name : counter
image : quay.io/adrianreber/counter:blog
This results in a container called counter running in a pod called counters .
Once the container is running I am performing following actions with that
container:
$ kubectl get pod counters --template ''
10.88.0.25
$ curl 10.88.0.25:8088/create?test-file
$ curl 10.88.0.25:8088/secret?RANDOM_1432_KEY
$ curl 10.88.0.25:8088
The first access creates a file called test-file with the content test-file
in the container and the second access stores my secret information
(RANDOM_1432_KEY ) somewhere in the container's memory. The last access just
adds an additional line to the internal log file.
The last step before I can analyze the checkpoint it to tell Kubernetes to create
the checkpoint. As described in the previous article this requires access to the
kubelet only checkpoint API endpoint.
For a container named counter in a pod named counters in a namespace named
default the kubelet API endpoint is reachable at:
# run this on the node where that Pod is executing
curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"
For completeness the following curl command-line options are necessary to
have curl accept the kubelet 's self signed certificate and authorize the
use of the kubelet checkpoint API:
--insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key
Once the checkpointing has finished the checkpoint should be available at
/var/lib/kubelet/checkpoints/checkpoint-pod-name_namespace-name-container-name-timestamp.tar
In the following steps of this article I will use the name checkpoint.tar
when analyzing the checkpoint archive.
Checkpoint archive analysis using checkpointctl
To get some initial information about the checkpointed container I am using the
tool checkpointctl like this:
$ checkpointctl show checkpoint.tar --print-stats
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
| CONTAINER | IMAGE | ID | RUNTIME | CREATED | ENGINE | IP | CHKPT SIZE | ROOT FS DIFF SIZE |
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
| counter | quay.io/adrianreber/counter:blog | 059a219a22e5 | runc | 2023-03-02T06:06:49 | CRI-O | 10.88.0.23 | 8.6 MiB | 3.0 KiB |
+-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
CRIU dump statistics
+---------------+-------------+--------------+---------------+---------------+---------------+
| FREEZING TIME | FROZEN TIME | MEMDUMP TIME | MEMWRITE TIME | PAGES SCANNED | PAGES WRITTEN |
+---------------+-------------+--------------+---------------+---------------+---------------+
| 100809 us | 119627 us | 11602 us | 7379 us | 7800 | 2198 |
+---------------+-------------+--------------+---------------+---------------+---------------+
This gives me already some information about the checkpoint in that checkpoint
archive. I can see the name of the container, information about the container
runtime and container engine. It also lists the size of the checkpoint (CHKPT SIZE ). This is mainly the size of the memory pages included in the checkpoint,
but there is also information about the size of all changed files in the
container (ROOT FS DIFF SIZE ).
The additional parameter --print-stats decodes information in the checkpoint
archive and displays them in the second table (CRIU dump statistics ). This
information is collected during checkpoint creation and gives an overview how much
time CRIU needed to checkpoint the processes in the container and how many
memory pages were analyzed and written during checkpoint creation.
Digging deeper
With the help of checkpointctl I am able to get some high level information
about the checkpoint archive. To be able to analyze the checkpoint archive
further I have to extract it. The checkpoint archive is a tar archive and can
be extracted with the help of tar xf checkpoint.tar .
Extracting the checkpoint archive will result in following files and directories:
bind.mounts - this file contains information about bind mounts and is needed
during restore to mount all external files and directories at the right location
checkpoint/ - this directory contains the actual checkpoint as created by
CRIU
config.dump and spec.dump - these files contain metadata about the container
which is needed during restore
dump.log - this file contains the debug output of CRIU created during
checkpointing
stats-dump - this file contains the data which is used by checkpointctl
to display dump statistics (--print-stats )
rootfs-diff.tar - this file contains all changed files on the container's
file-system
File-system changes - rootfs-diff.tar
The first step to analyze the container's checkpoint further is to look at
the files that have changed in my container. This can be done by looking at the
file rootfs-diff.tar :
$ tar xvf rootfs-diff.tar
home/counter/logfile
home/counter/test-file
Now the files that changed in the container can be studied:
$ cat home/counter/logfile
10.88.0.1 - - [02/Mar/2023 06:07:29] "GET /create?test-file HTTP/1.1" 200 -
10.88.0.1 - - [02/Mar/2023 06:07:40] "GET /secret?RANDOM_1432_KEY HTTP/1.1" 200 -
10.88.0.1 - - [02/Mar/2023 06:07:43] "GET / HTTP/1.1" 200 -
$ cat home/counter/test-file
test-file
Compared to the container image (quay.io/adrianreber/counter:blog ) this
container is based on, I can see that the file logfile contains information
about all access to the service the container provides and the file test-file
was created just as expected.
With the help of rootfs-diff.tar it is possible to inspect all files that
were created or changed compared to the base image of the container.
Analyzing the checkpointed processes - checkpoint/
The directory checkpoint/ contains data created by CRIU while checkpointing
the processes in the container. The content in the directory checkpoint/
consists of different image files which can be analyzed with the
help of the tool CRIT which is distributed as part of CRIU.
First lets get an overview of the processes inside of the container:
$ crit show checkpoint/pstree.img | jq .entries[] .pid
1
7
8
This output means that I have three processes inside of the container's PID
namespace with the PIDs: 1, 7, 8
This is only the view from the inside of the container's PID namespace. During
restore exactly these PIDs will be recreated. From the outside of the
container's PID namespace the PIDs will change after restore.
The next step is to get some additional information about these three processes:
$ crit show checkpoint/core-1.img | jq .entries[ 0] .tc.comm
"bash"
$ crit show checkpoint/core-7.img | jq .entries[ 0] .tc.comm
"counter.py"
$ crit show checkpoint/core-8.img | jq .entries[ 0] .tc.comm
"tee"
This means the three processes in my container are bash , counter.py (a Python
interpreter) and tee . For details about the parent child relations of these processes there
is more data to be analyzed in checkpoint/pstree.img .
Let's compare the so far collected information to the still running container:
$ crictl inspect --output go-template --template "" 059a219a22e56
722520
$ ps auxf | grep -A 2 722520
fedora 722520 \_ bash -c /home/counter/counter.py 2&1 | tee /home/counter/logfile
fedora 722541 \_ /usr/bin/python3 /home/counter/counter.py
fedora 722542 \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /home/counter/logfile
$ cat /proc/722520/comm
bash
$ cat /proc/722541/comm
counter.py
$ cat /proc/722542/comm
tee
In this output I am first retrieving the PID of the first process in the
container and then I am looking for that PID and child processes on the system
where the container is running. I am seeing three processes and the first one is
"bash" which is PID 1 inside of the containers PID namespace. Then I am looking
at /proc/PID/comm and I can find the exact same value
as in the checkpoint image.
Important to remember is that the checkpoint will contain the view from within the
container's PID namespace because that information is important to restore the
processes.
One last example of what crit can tell us about the container is the information
about the UTS namespace:
$ crit show checkpoint/utsns-12.img
{
"magic": "UTSNS",
"entries": [
{
"nodename": "counters",
"domainname": "(none)"
}
]
}
This tells me that the hostname inside of the UTS namespace is counters .
For every resource CRIU collected during checkpointing...
Not me. Interesting though. | Why I Will Never Use Alpine Linux Ever Again
p
Nowadays, Alpine Linux is one of the most popular options for container base images. Many people (maybe including you) use it for anything and everythi...
containerd completes fuzzing audit | Cloud Native Computing Foundation
Community post by Adam Korczynski and Phil Estes The containerd project is happy to announce the completion of a comprehensive fuzzing audit which added 28 fuzzers covering a wide range of container…
%2Fcdn.vox-cdn.com%2Fuploads%2Fchorus_asset%2Ffile%2F19713460%2Facastro_200207_3900_Tor_0001.0.jpg?width=92&height=&ar=&mode=&format=avif&dpr=2)
