Kubernetes v1.33: HorizontalPodAutoscaler Configurable Tolerance

https://kubernetes.io/blog/2025/04/28/kubernetes-v1-33-hpa-configurable-tolerance/

This post describes configurable tolerance for horizontal Pod autoscaling, a new alpha feature first available in Kubernetes 1.33.

What is it?

Horizontal Pod Autoscaling is a well-known Kubernetes feature that allows your workload to automatically resize by adding or removing replicas based on resource utilization.

Let's say you have a web application running in a Kubernetes cluster with 50 replicas. You configure the Horizontal Pod Autoscaler (HPA) to scale based on CPU utilization, with a target of 75% utilization. Now, imagine that the current CPU utilization across all replicas is 90%, which is higher than the desired 75%. The HPA will calculate the required number of replicas using the formula:

$$desiredReplicas = ceil\left\lceil currentReplicas \times \frac{currentMetricValue}{desiredMetricValue} \right\rceil$$

In this example:

$$50 \times (90/75) = 60$$

So, the HPA will increase the number of replicas from 50 to 60 to reduce the load on each pod. Similarly, if the CPU utilization were to drop below 75%, the HPA would scale down the number of replicas accordingly. The Kubernetes documentation provides a detailed description of the scaling algorithm.

In order to avoid replicas being created or deleted whenever a small metric fluctuation occurs, Kubernetes applies a form of hysteresis: it only changes the number of replicas when the current and desired metric values differ by more than 10%. In the example above, since the ratio between the current and desired metric values is (90/75), or 20% above target, exceeding the 10% tolerance, the scale-up action will proceed.

This default tolerance of 10% is cluster-wide; in older Kubernetes releases, it could not be fine-tuned. It's a suitable value for most usage, but too coarse for large deployments, where a 10% tolerance represents tens of pods. As a result, the community has long asked to be able to tune this value.

In Kubernetes v1.33, this is now possible.

How do I use it?

After enabling the HPAConfigurableTolerance feature gate in your Kubernetes v1.33 cluster, you can add your desired tolerance for your HorizontalPodAutoscaler object.

Tolerances appear under the spec.behavior.scaleDown and spec.behavior.scaleUp fields and can thus be different for scale up and scale down. A typical usage would be to specify a small tolerance on scale up (to react quickly to spikes), but higher on scale down (to avoid adding and removing replicas too quickly in response to small metric fluctuations).

For example, an HPA with a tolerance of 5% on scale-down, and no tolerance on scale-up, would look like the following:

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: my-app spec: ... behavior: scaleDown: tolerance: 0.05 scaleUp: tolerance: 0

I want all the details!

Get all the technical details by reading KEP-4951 and follow issue 4951 to be notified of the feature graduation.

via Kubernetes Blog https://kubernetes.io/

April 28, 2025 at 02:30PM

Kro vs Helm: Is It Time to Ditch Helm Charts?

Dive deep into the fascinating debate of Helm vs. kro in Kubernetes! Discover the major differences between these tools, how they compare, and their unique advantages. From Helm's templating engine to kro's Custom Resource Definitions, this video will guide you through deploying applications and managing resources effectively. Learn about the evolution of Kubernetes, the role of CRDs and controllers, and why custom solutions might be the future.

Helm #Kro #Kubernetes

Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join

▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/kubernetes/kro-vs-helm-is-it-time-to-ditch-helm-charts 🔗 kro: https://kro.run 🔗 UpCloud: https://upcloud.com 🎬 Is This the End of Crossplane? Compose Kubernetes Resources with kro: https://youtu.be/8zQtpcxmdhs

▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 Helm vs. Kro 03:14 Lower vs. Higher Level Controllers 15:59 Kubernetes Custom Resources 26:19 Create Your Own CRDs and Controllers 30:48 Wrap-Up

via YouTube https://www.youtube.com/watch?v=V4N1dHzHmXI

Kubernetes v1.33: User Namespaces enabled by default!

https://kubernetes.io/blog/2025/04/25/userns-enabled-by-default/

In Kubernetes v1.33 support for user namespaces is enabled by default. This means that, when the stack requirements are met, pods can opt-in to use user namespaces. To use the feature there is no need to enable any Kubernetes feature flag anymore!

In this blog post we answer some common questions about user namespaces. But, before we dive into that, let's recap what user namespaces are and why they are important.

What is a user namespace?

Note: Linux user namespaces are a different concept from Kubernetes namespaces. The former is a Linux kernel feature; the latter is a Kubernetes feature.

Linux provides different namespaces to isolate processes from each other. For example, a typical Kubernetes pod runs within a network namespace to isolate the network identity and a PID namespace to isolate the processes.

One Linux namespace that was left behind is the user namespace. It isolates the UIDs and GIDs of the containers from the ones on the host. The identifiers in a container can be mapped to identifiers on the host in a way where host and container(s) never end up in overlapping UID/GIDs. Furthermore, the identifiers can be mapped to unprivileged, non-overlapping UIDs and GIDs on the host. This brings three key benefits:

Prevention of lateral movement: As the UIDs and GIDs for different containers are mapped to different UIDs and GIDs on the host, containers have a harder time attacking each other, even if they escape the container boundaries. For example, suppose container A runs with different UIDs and GIDs on the host than container B. In that case, the operations it can do on container B's files and processes are limited: only read/write what a file allows to others, as it will never have permission owner or group permission (the UIDs/GIDs on the host are guaranteed to be different for different containers).

Increased host isolation: As the UIDs and GIDs are mapped to unprivileged users on the host, if a container escapes the container boundaries, even if it runs as root inside the container, it has no privileges on the host. This greatly protects what host files it can read/write, which process it can send signals to, etc. Furthermore, capabilities granted are only valid inside the user namespace and not on the host, limiting the impact a container escape can have.

Enablement of new use cases: User namespaces allow containers to gain certain capabilities inside their own user namespace without affecting the host. This unlocks new possibilities, such as running applications that require privileged operations without granting full root access on the host. This is particularly useful for running nested containers.

User namespace IDs allocation

If a pod running as the root user without a user namespace manages to breakout, it has root privileges on the node. If some capabilities were granted to the container, the capabilities are valid on the host too. None of this is true when using user namespaces (modulo bugs, of course 🙂).

Demos

Rodrigo created demos to understand how some CVEs are mitigated when user namespaces are used. We showed them here before (see here and here), but take a look if you haven't:

Mitigation of CVE 2024-21626 with user namespaces:

Mitigation of CVE 2022-0492 with user namespaces:

Everything you wanted to know about user namespaces in Kubernetes

Here we try to answer some of the questions we have been asked about user namespaces support in Kubernetes.

1. What are the requirements to use it?

The requirements are documented here. But we will elaborate a bit more, in the following questions.

Note this is a Linux-only feature.

2. How do I configure a pod to opt-in?

A complete step-by-step guide is available here. But the short version is you need to set the hostUsers: false field in the pod spec. For example like this:

apiVersion: v1 kind: Pod metadata: name: userns spec: hostUsers: false containers:

• name: shell command: ["sleep", "infinity"] image: debian

Yes, it is that simple. Applications will run just fine, without any other changes needed (unless your application needs the privileges).

User namespaces allows you to run as root inside the container, but not have privileges in the host. However, if your application needs the privileges on the host, for example an app that needs to load a kernel module, then you can't use user namespaces.

3. What are idmap mounts and why the file-systems used need to support it?

Idmap mounts are a Linux kernel feature that uses a mapping of UIDs/GIDs when accessing a mount. When combined with user namespaces, it greatly simplifies the support for volumes, as you can forget about the host UIDs/GIDs the user namespace is using.

In particular, thanks to idmap mounts we can:

Run each pod with different UIDs/GIDs on the host. This is key for the lateral movement prevention we mentioned earlier.

Share volumes with pods that don't use user namespaces.

Enable/disable user namespaces without needing to chown the pod's volumes.

Support for idmap mounts in the kernel is per file-system and different kernel releases added support for idmap mounts on different file-systems.

To find which kernel version added support for each file-system, you can check out the mount_setattr man page, or the online version of it here.

Most popular file-systems are supported, the notable absence that isn't supported yet is NFS.

4. Can you clarify exactly which file-systems need to support idmap mounts?

The file-systems that need to support idmap mounts are all the file-systems used by a pod in the pod.spec.volumes field.

This means: for PV/PVC volumes, the file-system used in the PV needs to support idmap mounts; for hostPath volumes, the file-system used in the hostPath needs to support idmap mounts.

What does this mean for secrets/configmaps/projected/downwardAPI volumes? For these volumes, the kubelet creates a tmpfs file-system. So, you will need a 6.3 kernel to use these volumes (note that if you use them as env variables it is fine).

And what about emptyDir volumes? Those volumes are created by the kubelet by default in /var/lib/kubelet/pods/. You can also use a custom directory for this. But what needs to support idmap mounts is the file-system used in that directory.

The kubelet creates some more files for the container, like /etc/hostname, /etc/resolv.conf, /dev/termination-log, /etc/hosts, etc. These files are also created in /var/lib/kubelet/pods/ by default, so it's important for the file-system used in that directory to support idmap mounts.

Also, some container runtimes may put some of these ephemeral volumes inside a tmpfs file-system, in which case you will need support for idmap mounts in tmpfs.

5. Can I use a kernel older than 6.3?

Yes, but you will need to make sure you are not using a tmpfs file-system. If you avoid that, you can easily use 5.19 (if all the other file-systems you use support idmap mounts in that kernel).

It can be tricky to avoid using tmpfs, though, as we just described above. Besides having to avoid those volume types, you will also have to avoid mounting the service account token. Every pod has it mounted by default, and it uses a projected volume that, as we mentioned, uses a tmpfs file-system.

You could even go lower than 5.19, all the way to 5.12. However, your container rootfs probably uses an overlayfs file-system, and support for overlayfs was added in 5.19. We wouldn't recommend to use a kernel older than 5.19, as not being able to use idmap mounts for the rootfs is a big limitation. If you absolutely need to, you can check this blog post Rodrigo wrote some years ago, about tricks to use user namespaces when you can't support idmap mounts on the rootfs.

6. If my stack supports user namespaces, do I need to configure anything else?

No, if your stack supports it and you are using Kubernetes v1.33, there is nothing you need to configure. You should be able to follow the task: Use a user namespace with a pod.

However, in case you have specific requirements, you may configure various options. You can find more information here. You can also enable a feature gate to relax the PSS rules.

7. The demos are nice, but are there more CVEs that this mitigates?

Yes, quite a lot, actually! Besides the ones in the demo, the KEP has more CVEs you can check. That list is not exhaustive, there are many more.

8. Can you sum up why user namespaces is important?

Think about running a process as root, maybe even an untrusted process. Do you think that is secure? What if we limit it by adding seccomp and apparmor, mask some files in /proc (so it can't crash the node, etc.) and some more tweaks?

Wouldn't it be better if we don't give it privileges in the first place, instead of trying to play whack-a-mole with all the possible ways root can escape?

This is what user namespaces does, plus some other goodies:

Run as an unprivileged user on the host without making changes to your application. Greg and Vinayak did a great talk on the pains you can face when trying to run unprivileged without user namespaces. The pains part starts in this minute.

All pods run with different UIDs/GIDs, we significantly improve the lateral movement. This is guaranteed with user namespaces (the kubelet chooses it for you). In the same talk, Greg and Vinayak show that to achieve the same without user namespaces, they went through a quite complex custom solution. This part starts in this minute.

The capabilities granted are only granted inside the user namespace. That means that if a pod breaks out of the container, they are not valid on the host. We can't provide that without user namespaces.

It enables new use-cases in a secure way. You can run docker in docker, unprivileged container builds, Kubernetes inside Kubernetes, etc all in a secure way. Most of the previous solutions to do this required privilege

Continuing the transition from Endpoints to EndpointSlices

https://kubernetes.io/blog/2025/04/24/endpoints-deprecation/

Since the addition of EndpointSlices (KEP-752) as alpha in v1.15 and later GA in v1.21, the Endpoints API in Kubernetes has been gathering dust. New Service features like dual-stack networking and traffic distribution are only supported via the EndpointSlice API, so all service proxies, Gateway API implementations, and similar controllers have had to be ported from using Endpoints to using EndpointSlices. At this point, the Endpoints API is really only there to avoid breaking end user workloads and scripts that still make use of it.

As of Kubernetes 1.33, the Endpoints API is now officially deprecated, and the API server will return warnings to users who read or write Endpoints resources rather than using EndpointSlices.

Eventually, the plan (as documented in KEP-4974) is to change the Kubernetes Conformance criteria to no longer require that clusters run the Endpoints controller (which generates Endpoints objects based on Services and Pods), to avoid doing work that is unneeded in most modern-day clusters.

Thus, while the Kubernetes deprecation policy means that the Endpoints type itself will probably never completely go away, users who still have workloads or scripts that use the Endpoints API should start migrating them to EndpointSlices.

Notes on migrating from Endpoints to EndpointSlices

Consuming EndpointSlices rather than Endpoints

For end users, the biggest change between the Endpoints API and the EndpointSlice API is that while every Service with a selector has exactly 1 Endpoints object (with the same name as the Service), a Service may have any number of EndpointSlices associated with it:

$ kubectl get endpoints myservice Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice NAME ENDPOINTS AGE myservice 10.180.3.17:443 1h

$ kubectl get endpointslice -l kubernetes.io/service-name=myservice NAME ADDRESSTYPE PORTS ENDPOINTS AGE myservice-7vzhx IPv4 443 10.180.3.17 21s myservice-jcv8s IPv6 443 2001:db8:0123::5 21s

In this case, because the service is dual stack, it has 2 EndpointSlices: 1 for IPv4 addresses and 1 for IPv6 addresses. (The Endpoints API does not support dual stack, so the Endpoints object shows only the addresses in the cluster's primary address family.) Although any Service with multiple endpoints can have multiple EndpointSlices, there are three main cases where you will see this:

An EndpointSlice can only represent endpoints of a single IP family, so dual-stack Services will have separate EndpointSlices for IPv4 and IPv6.

All of the endpoints in an EndpointSlice must target the same ports. So, for example, if you have a set of endpoint Pods listening on port 80, and roll out an update to make them listen on port 8080 instead, then while the rollout is in progress, the Service will need 2 EndpointSlices: 1 for the endpoints listening on port 80, and 1 for the endpoints listening on port 8080.

When a Service has more than 100 endpoints, the EndpointSlice controller will split the endpoints into multiple EndpointSlices rather than aggregating them into a single excessively-large object like the Endpoints controller does.

Because there is not a predictable 1-to-1 mapping between Services and EndpointSlices, there is no way to know what the actual name of the EndpointSlice resource(s) for a Service will be ahead of time; thus, instead of fetching the EndpointSlice(s) by name, you instead ask for all EndpointSlices with a "kubernetes.io/service-name" label pointing to the Service:

$ kubectl get endpointslice -l kubernetes.io/service-name=myservice

A similar change is needed in Go code. With Endpoints, you would do something like:

// Get the Endpoints named name in namespace. endpoint, err := client.CoreV1().Endpoints(namespace).Get(ctx, name, metav1.GetOptions{}) if err != nil { if apierrors.IsNotFound(err) { // No Endpoints exists for the Service (yet?) ... } // handle other errors ... }

// process endpoint ...

With EndpointSlices, this becomes:

// Get all EndpointSlices for Service name in namespace. slices, err := client.DiscoveryV1().EndpointSlices(namespace).List(ctx, metav1.ListOptions{LabelSelector: discoveryv1.LabelServiceName + "=" + name}) if err != nil { // handle errors ... } else if len(slices.Items) == 0 { // No EndpointSlices exist for the Service (yet?) ... }

// process slices.Items ...

Generating EndpointSlices rather than Endpoints

For people (or controllers) generating Endpoints, migrating to EndpointSlices is slightly easier, because in most cases you won't have to worry about multiple slices. You just need to update your YAML or Go code to use the new type (which organizes the information in a slightly different way than Endpoints did).

For example, this Endpoints object:

apiVersion: v1 kind: Endpoints metadata: name: myservice subsets:

• addresses:
• ip: 10.180.3.17 nodeName: node-4
• ip: 10.180.5.22 nodeName: node-9
• ip: 10.180.18.2 nodeName: node-7 notReadyAddresses:
• ip: 10.180.6.6 nodeName: node-8 ports:
• name: https protocol: TCP port: 443

would become something like:

apiVersion: discovery.k8s.io/v1 kind: EndpointSlice metadata: name: myservice labels: kubernetes.io/service-name: myservice addressType: IPv4 endpoints:

• addresses:
• 10.180.3.17 nodeName: node-4
• addresses:
• 10.180.5.22 nodeName: node-9
• addresses:
• 10.180.18.12 nodeName: node-7
• addresses:
• 10.180.6.6 nodeName: node-8 conditions: ready: false ports:
• name: https protocol: TCP port: 443

Some points to note:

This example uses an explicit name, but you could also use generateName and let the API server append a unique suffix. The name itself does not matter: what matters is the "kubernetes.io/service-name" label pointing back to the Service.

You have to explicitly indicate addressType: IPv4 (or IPv6).

An EndpointSlice is similar to a single element of the "subsets" array in Endpoints. An Endpoints object with multiple subsets will normally need to be expressed as multiple EndpointSlices, each with different "ports".

The endpoints and addresses fields are both arrays, but by convention, each addresses array only contains a single element. If your Service has multiple endpoints, then you need to have multiple elements in the endpoints array, each with a single element in its addresses array.

The Endpoints API lists "ready" and "not-ready" endpoints separately, while the EndpointSlice API allows each endpoint to have conditions (such as "ready: false") associated with it.

And of course, once you have ported to EndpointSlice, you can make use of EndpointSlice-specific features, such as topology hints and terminating endpoints. Consult the EndpointSlice API documentation for more information.

via Kubernetes Blog https://kubernetes.io/

April 24, 2025 at 02:30PM

Week Ending April 20, 2025

https://lwkd.info/2025/20250423

Developer News

Kubernetes 1.33 is released. Top features include supporting .kuberc, sidecars graduating, and lots of DRA features. There’s dozens of other graduations to stable, and seven more major new features in alpha. Don’t forget to check the deprecations and removals and the upgrade notes before upgrading.

Many dependencies were added, removed, or updated in 1.33 as well.

Interested in writing for the Kubernetes blog? SIG-Docs has published expanded blog guidelines to help you select appropriate topics and style.

Release Schedule

Next Deadline: Release day, 23 April

Kubernetes v1.33 is released today! We have a total of 63 tracked enhancements this cycle, higher than 1.32’s 45 enhancements. Thank you to everyone who contributed to Kubernetes v1.33 and made such a productive release cycle possible.

April patch releases are also out this week for 1.29.16, 1.30.12, 1.31.8, and 1.32.4. This is just a bugfix patch release.

Shoutouts

Wendy Ha, the Release Signal lead for v1.33 gave a shoutout to the v1.33 Release Signal Team: @Sean McGinnis, @elieser1101, @Rajalakshmi Girish, @ChengHao Yang (tico88612), @nitish for their incredible effort throughout this cycle. Folks stayed committed until the end while also contributing to other parts of the project. Huge thanks to everyone for their hard work!

Nina Polshakova: Shout out to the amazing v1.33 Docs team for a smooth Docs freeze this week and all your hard work this release! rayandas, Melony Q. (aka.cloudmelon ), Sreeram Venkitesh, Urvashi, Arvind Parekh, Michelle Nguyen, Shedrack Akintayo

Nina Polshakova: Huge shoutout to the amazing Kubernetes v1.33 Comms team: Ryota, aibarbetta, Aakanksha Bhende, Sneha Yadav, Udi Hofesh , for all your incredible work this cycle! You pulled off the biggest release announcement blog in recent memory — over 7,000 words, blowing past the previous record of ~5,000 from v1.31!

via Last Week in Kubernetes Development https://lwkd.info/

April 23, 2025 at 03:30PM

Kubernetes v1.33: Octarine

https://kubernetes.io/blog/2025/04/23/kubernetes-v1-33-release/

Editors: Agustina Barbetta, Aakanksha Bhende, Udi Hofesh, Ryota Sawada, Sneha Yadav

Similar to previous releases, the release of Kubernetes v1.33 introduces new stable, beta, and alpha features. The consistent delivery of high-quality releases underscores the strength of our development cycle and the vibrant support from our community.

This release consists of 64 enhancements. Of those enhancements, 18 have graduated to Stable, 20 are entering Beta, 24 have entered Alpha, and 2 are deprecated or withdrawn.

There are also several notable deprecations and removals in this release; make sure to read about those if you already run an older version of Kubernetes.

Release theme and logo

The theme for Kubernetes v1.33 is Octarine: The Color of Magic1, inspired by Terry Pratchett’s Discworld series. This release highlights the open-source magic2 that Kubernetes enables across the ecosystem.

If you’re familiar with the world of Discworld, you might recognize a small swamp dragon perched atop the tower of the Unseen University, gazing up at the Kubernetes moon above the city of Ankh-Morpork with 64 stars3 in the background.

As Kubernetes moves into its second decade, we celebrate both the wizardry of its maintainers, the curiosity of new contributors, and the collaborative spirit that fuels the project. The v1.33 release is a reminder that, as Pratchett wrote, “It’s still magic even if you know how it’s done.” Even if you know the ins and outs of the Kubernetes code base, stepping back at the end of the release cycle, you’ll realize that Kubernetes remains magical.

Kubernetes v1.33 is a testament to the enduring power of open-source innovation, where hundreds of contributors4 from around the world work together to create something truly extraordinary. Behind every new feature, the Kubernetes community works to maintain and improve the project, ensuring it remains secure, reliable, and released on time. Each release builds upon the other, creating something greater than we could achieve alone.

1. Octarine is the mythical eighth color, visible only to those attuned to the arcane—wizards, witches, and, of course, cats. And occasionally, someone who’s stared at IPtable rules for too long.

2. Any sufficiently advanced technology is indistinguishable from magic…?

3. It’s not a coincidence 64 KEPs (Kubernetes Enhancement Proposals) are also included in v1.33.

4. See the Project Velocity section for v1.33 🚀

Spotlight on key updates

Kubernetes v1.33 is packed with new features and improvements. Here are a few select updates the Release Team would like to highlight!

Stable: Sidecar containers

The sidecar pattern involves deploying separate auxiliary container(s) to handle extra capabilities in areas such as networking, logging, and metrics gathering. Sidecar containers graduate to stable in v1.33.

Kubernetes implements sidecars as a special class of init containers with restartPolicy: Always, ensuring that sidecars start before application containers, remain running throughout the pod's lifecycle, and terminate automatically after the main containers exit.

Additionally, sidecars can utilize probes (startup, readiness, liveness) to signal their operational state, and their Out-Of-Memory (OOM) score adjustments are aligned with primary containers to prevent premature termination under memory pressure.

To learn more, read Sidecar Containers.

This work was done as part of KEP-753: Sidecar Containers led by SIG Node.

Beta: In-place resource resize for vertical scaling of Pods

Workloads can be defined using APIs like Deployment, StatefulSet, etc. These describe the template for the Pods that should run, including memory and CPU resources, as well as the replica count of the number of Pods that should run. Workloads can be scaled horizontally by updating the Pod replica count, or vertically by updating the resources required in the Pods container(s). Before this enhancement, container resources defined in a Pod's spec were immutable, and updating any of these details within a Pod template would trigger Pod replacement.

But what if you could dynamically update the resource configuration for your existing Pods without restarting them?

The KEP-1287 is precisely to allow such in-place Pod updates. It was released as alpha in v1.27, and has graduated to beta in v1.33. This opens up various possibilities for vertical scale-up of stateful processes without any downtime, seamless scale-down when the traffic is low, and even allocating larger resources during startup, which can then be reduced once the initial setup is complete.

This work was done as part of KEP-1287: In-Place Update of Pod Resources led by SIG Node and SIG Autoscaling.

Alpha: New configuration option for kubectl with .kuberc for user preferences

In v1.33, kubectl introduces a new alpha feature with opt-in configuration file .kuberc for user preferences. This file can contain kubectl aliases and overrides (e.g. defaulting to use server-side apply), while leaving cluster credentials and host information in kubeconfig. This separation allows sharing the same user preferences for kubectl interaction, regardless of target cluster and kubeconfig used.

To enable this alpha feature, users can set the environment variable of KUBECTL_KUBERC=true and create a .kuberc configuration file. By default, kubectl looks for this file in ~/.kube/kuberc. You can also specify an alternative location using the --kuberc flag, for example: kubectl --kuberc /var/kube/rc.

This work was done as part of KEP-3104: Separate kubectl user preferences from cluster configs led by SIG CLI.

Features graduating to Stable

This is a selection of some of the improvements that are now stable following the v1.33 release.

Backoff limits per index for indexed Jobs

​This release graduates a feature that allows setting backoff limits on a per-index basis for Indexed Jobs. Traditionally, the backoffLimit parameter in Kubernetes Jobs specifies the number of retries before considering the entire Job as failed. This enhancement allows each index within an Indexed Job to have its own backoff limit, providing more granular control over retry behavior for individual tasks. This ensures that the failure of specific indices does not prematurely terminate the entire Job, allowing the other indices to continue processing independently.

This work was done as part of KEP-3850: Backoff Limit Per Index For Indexed Jobs led by SIG Apps.

Job success policy

Using .spec.successPolicy, users can specify which pod indexes must succeed (succeededIndexes), how many pods must succeed (succeededCount), or a combination of both. This feature benefits various workloads, including simulations where partial completion is sufficient, and leader-worker patterns where only the leader's success determines the Job's overall outcome.

This work was done as part of KEP-3998: Job success/completion policy led by SIG Apps.

Bound ServiceAccount token security improvements

This enhancement introduced features such as including a unique token identifier (i.e. JWT ID Claim, also known as JTI) and node information within the tokens, enabling more precise validation and auditing. Additionally, it supports node-specific restrictions, ensuring that tokens are only usable on designated nodes, thereby reducing the risk of token misuse and potential security breaches. These improvements, now generally available, aim to enhance the overall security posture of service account tokens within Kubernetes clusters.

This work was done as part of KEP-4193: Bound service account token improvements led by SIG Auth.

Subresource support in kubectl

The --subresource argument is now generally available for kubectl subcommands such as get, patch, edit, apply and replace, allowing users to fetch and update subresources for all resources that support them. To learn more about the subresources supported, visit the kubectl reference.

This work was done as part of KEP-2590: Add subresource support to kubectl led by SIG CLI.

Multiple Service CIDRs

This enhancement introduced a new implementation of allocation logic for Service IPs. Across the whole cluster, every Service of type: ClusterIP must have a unique IP address assigned to it. Trying to create a Service with a specific cluster IP that has already been allocated will return an error. The updated IP address allocator logic uses two newly stable API objects: ServiceCIDR and IPAddress. Now generally available, these APIs allow cluster administrators to dynamically increase the number of IP addresses available for type: ClusterIP Services (by creating new ServiceCIDR objects).

This work was done as part of KEP-1880: Multiple Service CIDRs led by SIG Network.

nftables backend for kube-proxy

The nftables backend for kube-proxy is now stable, adding a new implementation that significantly improves performance and scalability for Services implementation within Kubernetes clusters. For compatibility reasons, iptables remains the default on Linux nodes. Check the migration guide if you want to try it out.

This work was done as part of KEP-3866: nftables kube-proxy backend led by SIG Network.

Topology aware routing with trafficDistribution: PreferClose

This release graduates topology-aware routing and traffic distribution to GA, which would allow us to optimize service traffic in multi-zone clusters. The topology-aware hints in EndpointSlices would enable components like kube-proxy to prioritize routing traffic to endpoints within the same zone, thereby reducing latency and cross-zone data transfer costs. Building upon this, trafficDistribution field is added to the Service specification, with the PreferClose option directing traffic to the nearest available endpoints based on network topology. This configuration enhances performance and cost-efficiency by minimizing inter-zone communication.

This work w

Ep19 - Ask Me Anything About Anything with Scott Rosenberg and Ramiro Berrelleza

There are no restrictions in this AMA session. You can ask anything about DevOps, Cloud, Kubernetes, Platform Engineering, containers, or anything else. We'll have special guests Scott Rosenberg and Ramiro Berrelleza to help us out.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: Codefresh 🔗 Codefresh GitOps Cloud: https://codefresh.io ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

via YouTube https://www.youtube.com/watch?v=EZ6wodGif0Q

Replacing StatefulSets with a custom Kubernetes operator in our Postgres cloud platform, with Andrew Charlton

https://ku.bz/fhZ_pNXM3

Discover why standard Kubernetes StatefulSets might not be sufficient for your database workloads and how custom operators can provide better solutions for stateful applications.

Andrew Charlton, Staff Software Engineer at Timescale, explains how they replaced Kubernetes StatefulSets with a custom operator called Popper for their PostgreSQL Cloud Platform. He details the technical limitations they encountered with StatefulSets and how their custom approach provides more intelligent management of database clusters.

You will learn:

Why StatefulSets fall short for managing high-availability PostgreSQL clusters, particularly around pod ordering and volume management

How Timescale's instance matching approach solves complex reconciliation challenges when managing heterogeneous database workloads

The benefits of implementing discrete, idempotent actions rather than workflows in Kubernetes operators

Real-world examples of operations that became possible with their custom operator, including volume downsizing and availability zone consolidation

Sponsor

This episode is brought to you by mirrord — run local code like in your Kubernetes cluster without deploying first.

More info

Find all the links and info for this episode here: https://ku.bz/fhZ_pNXM3

Interested in sponsoring an episode? Learn more.

via KubeFM https://kube.fm

April 22, 2025 at 08:59AM

Kubernetes Multicontainer Pods: An Overview

https://kubernetes.io/blog/2025/04/22/multi-container-pods-overview/

As cloud-native architectures continue to evolve, Kubernetes has become the go-to platform for deploying complex, distributed systems. One of the most powerful yet nuanced design patterns in this ecosystem is the sidecar pattern—a technique that allows developers to extend application functionality without diving deep into source code.

The origins of the sidecar pattern

Think of a sidecar like a trusty companion motorcycle attachment. Historically, IT infrastructures have always used auxiliary services to handle critical tasks. Before containers, we relied on background processes and helper daemons to manage logging, monitoring, and networking. The microservices revolution transformed this approach, making sidecars a structured and intentional architectural choice. With the rise of microservices, the sidecar pattern became more clearly defined, allowing developers to offload specific responsibilities from the main service without altering its code. Service meshes like Istio and Linkerd have popularized sidecar proxies, demonstrating how these companion containers can elegantly handle observability, security, and traffic management in distributed systems.

Kubernetes implementation

In Kubernetes, sidecar containers operate within the same Pod as the main application, enabling communication and resource sharing. Does this sound just like defining multiple containers along each other inside the Pod? It actually does, and this is how sidecar containers had to be implemented before Kubernetes v1.29.0, which introduced native support for sidecars. Sidecar containers can now be defined within a Pod manifest using the spec.initContainers field. What makes it a sidecar container is that you specify it with restartPolicy: Always. You can see an example of this below, which is a partial snippet of the full Kubernetes manifest:

initContainers:

• name: logshipper image: alpine:latest restartPolicy: Always command: ['sh', '-c', 'tail -F /opt/logs.txt'] volumeMounts:
• name: data mountPath: /opt

That field name, spec.initContainers may sound confusing. How come when you want to define a sidecar container, you have to put an entry in the spec.initContainers array? spec.initContainers are run to completion just before main application starts, so they’re one-off, whereas sidecars often run in parallel to the main app container. It’s the spec.initContainers with restartPolicy:Always which differs classic init containers from Kubernetes-native sidecar containers and ensures they are always up.

When to embrace (or avoid) sidecars

While the sidecar pattern can be useful in many cases, it is generally not the preferred approach unless the use case justifies it. Adding a sidecar increases complexity, resource consumption, and potential network latency. Instead, simpler alternatives such as built-in libraries or shared infrastructure should be considered first.

Deploy a sidecar when:

You need to extend application functionality without touching the original code

Implementing cross-cutting concerns like logging, monitoring or security

Working with legacy applications requiring modern networking capabilities

Designing microservices that demand independent scaling and updates

Proceed with caution if:

Resource efficiency is your primary concern

Minimal network latency is critical

Simpler alternatives exist

You want to minimize troubleshooting complexity

Four essential multi-container patterns

Init container pattern

The Init container pattern is used to execute (often critical) setup tasks before the main application container starts. Unlike regular containers, init containers run to completion and then terminate, ensuring that preconditions for the main application are met.

Ideal for:

Preparing configurations

Loading secrets

Verifying dependency availability

Running database migrations

The init container ensures your application starts in a predictable, controlled environment without code modifications.

Ambassador pattern

An ambassador container provides Pod-local helper services that expose a simple way to access a network service. Commonly, ambassador containers send network requests on behalf of a an application container and take care of challenges such as service discovery, peer identity verification, or encryption in transit.

Perfect when you need to:

Offload client connectivity concerns

Implement language-agnostic networking features

Add security layers like TLS

Create robust circuit breakers and retry mechanisms

Configuration helper

A configuration helper sidecar provides configuration updates to an application dynamically, ensuring it always has access to the latest settings without disrupting the service. Often the helper needs to provide an initial configuration before the application would be able to start successfully.

Use cases:

Fetching environment variables and secrets

Polling configuration changes

Decoupling configuration management from application logic

Adapter pattern

An adapter (or sometimes façade) container enables interoperability between the main application container and external services. It does this by translating data formats, protocols, or APIs.

Strengths:

Transforming legacy data formats

Bridging communication protocols

Facilitating integration between mismatched services

Wrap-up

While sidecar patterns offer tremendous flexibility, they're not a silver bullet. Each added sidecar introduces complexity, consumes resources, and potentially increases operational overhead. Always evaluate simpler alternatives first. The key is strategic implementation: use sidecars as precision tools to solve specific architectural challenges, not as a default approach. When used correctly, they can improve security, networking, and configuration management in containerized environments. Choose wisely, implement carefully, and let your sidecars elevate your container ecosystem.

via Kubernetes Blog https://kubernetes.io/

April 21, 2025 at 08:00PM

Mirrord Magic: Write Code Locally, See It Remotely!

Learn how to develop applications locally while integrating with remote production-like environments using mirrord. We'll demonstrate how to mirror and steal requests, connect to remote databases, and set up filtering to ensure a seamless development process without impacting others. Follow along as we configure and run mirrord, leveraging its capabilities to create an efficient and isolated development environment. This video will help you optimize your development workflow. Watch now to see mirrord in action!

Development #Kubernetes #mirrord

Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join

▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/development/mirrord-magic-write-code-locally,-see-it-remotely 🔗 mirrord: https://mirrord.dev 🔗 UpCloud: https://upcloud.com

▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 Development Environments 02:17 Setting The Scene (Staging Environment) 06:24 Development Environments with mirrord 13:34 Stealing Requests with mirrord 15:15 Filtering Requests with mirrord 17:52 What Else? 19:32 mirrord Pros and Cons

via YouTube https://www.youtube.com/watch?v=NLa0K5mybzo