Not Every Problem Needs Kubernetes, with Danyl Novhorodov

https://ku.bz/BYhFw8RwW

Danyl Novhorodov, a veteran .NET engineer and architect at Eneco, presents his controversial thesis that 90% of teams don't actually need Kubernetes. He walks through practical decision-making frameworks, explores powerful alternatives like BEAM runtimes and Actor models, and explains why starting with modular monoliths often beats premature microservices adoption.

You will learn:

The COST decision framework - How to evaluate infrastructure choices based on Complexity, Ownership, Skills, and Time rather than industry hype

Platform engineering vs. managed services - How to honestly assess whether your team can compete with AWS, Azure, and Google's managed container platforms

Evolutionary architecture approach - Why modular monoliths with clear boundaries often provide better foundations than distributed systems from day one

Sponsor

This episode is brought to you by Testkube—where teams run millions of performance tests in real Kubernetes infrastructure. From air-gapped environments to massive scale deployments, orchestrate every testing tool in one platform. Check it out at testkube.io

More info

Find all the links and info for this episode here: https://ku.bz/BYhFw8RwW

Interested in sponsoring an episode? Learn more.

via KubeFM https://kube.fm

September 23, 2025 at 06:00AM

Kubernetes v1.34: Pod Level Resources Graduated to Beta

https://kubernetes.io/blog/2025/09/22/kubernetes-v1-34-pod-level-resources/

On behalf of the Kubernetes community, I am thrilled to announce that the Pod Level Resources feature has graduated to Beta in the Kubernetes v1.34 release and is enabled by default! This significant milestone introduces a new layer of flexibility for defining and managing resource allocation for your Pods. This flexibility stems from the ability to specify CPU and memory resources for the Pod as a whole. Pod level resources can be combined with the container-level specifications to express the exact resource requirements and limits your application needs.

Pod-level specification for resources

Until recently, resource specifications that applied to Pods were primarily defined at the individual container level. While effective, this approach sometimes required duplicating or meticulously calculating resource needs across multiple containers within a single Pod. As a beta feature, Kubernetes allows you to specify the CPU, memory and hugepages resources at the Pod-level. This means you can now define resource requests and limits for an entire Pod, enabling easier resource sharing without requiring granular, per-container management of these resources where it's not needed.

Why does Pod-level specification matter?

This feature enhances resource management in Kubernetes by offering flexible resource management at both the Pod and container levels.

It provides a consolidated approach to resource declaration, reducing the need for meticulous, per-container management, especially for Pods with multiple containers.

Pod-level resources enable containers within a pod to share unused resoures amongst themselves, promoting efficient utilization within the pod. For example, it prevents sidecar containers from becoming performance bottlenecks. Previously, a sidecar (e.g., a logging agent or service mesh proxy) hitting its individual CPU limit could be throttled and slow down the entire Pod, even if the main application container had plenty of spare CPU. With pod-level resources, the sidecar and the main container can share Pod's resource budget, ensuring smooth operation during traffic spikes - either the whole Pod is throttled or all containers work.

When both pod-level and container-level resources are specified, pod-level requests and limits take precedence. This gives you – and cluster administrators - a powerful way to enforce overall resource boundaries for your Pods.

For scheduling, if a pod-level request is explicitly defined, the scheduler uses that specific value to find a suitable node, insteaf of the aggregated requests of the individual containers. At runtime, the pod-level limit acts as a hard ceiling for the combined resource usage of all containers. Crucially, this pod-level limit is the absolute enforcer; even if the sum of the individual container limits is higher, the total resource consumption can never exceed the pod-level limit.

Pod-level resources are prioritized in influencing the Quality of Service (QoS) class of the Pod.

For Pods running on Linux nodes, the Out-Of-Memory (OOM) score adjustment calculation considers both pod-level and container-level resources requests.

Pod-level resources are designed to be compatible with existing Kubernetes functionalities, ensuring a smooth integration into your workflows.

How to specify resources for an entire Pod

Using PodLevelResources feature gate requires Kubernetes v1.34 or newer for all cluster components, including the control plane and every node. This feature gate is in beta and enabled by default in v1.34.

Example manifest

You can specify CPU, memory and hugepages resources directly in the Pod spec manifest at the resources field for the entire Pod.

Here’s an example demonstrating a Pod with both CPU and memory requests and limits defined at the Pod level:

apiVersion: v1 kind: Pod metadata: name: pod-resources-demo namespace: pod-resources-example spec:

The 'resources' field at the Pod specification level defines the overall

resource budget for all containers within this Pod combined.

resources: # Pod-level resources

'limits' specifies the maximum amount of resources the Pod is allowed to use.

The sum of the limits of all containers in the Pod cannot exceed these values.

limits: cpu: "1" # The entire Pod cannot use more than 1 CPU core. memory: "200Mi" # The entire Pod cannot use more than 200 MiB of memory.

'requests' specifies the minimum amount of resources guaranteed to the Pod.

This value is used by the Kubernetes scheduler to find a node with enough capacity.

requests: cpu: "1" # The Pod is guaranteed 1 CPU core when scheduled. memory: "100Mi" # The Pod is guaranteed 100 MiB of memory when scheduled. containers:

• name: main-app-container image: nginx ... # This container has no resource requests or limits specified.
• name: auxiliary-container image: fedora command: ["sleep", "inf"] ... # This container has no resource requests or limits specified.

In this example, the pod-resources-demo Pod as a whole requests 1 CPU and 100 MiB of memory, and is limited to 1 CPU and 200 MiB of memory. The containers within will operate under these overall Pod-level constraints, as explained in the next section.

Interaction with container-level resource requests or limits

When both pod-level and container-level resources are specified, pod-level requests and limits take precedence. This means the node allocates resources based on the pod-level specifications.

Consider a Pod with two containers where pod-level CPU and memory requests and limits are defined, and only one container has its own explicit resource definitions:

apiVersion: v1 kind: Pod metadata: name: pod-resources-demo namespace: pod-resources-example spec: resources: limits: cpu: "1" memory: "200Mi" requests: cpu: "1" memory: "100Mi" containers:

• name: main-app-container image: nginx resources: requests: cpu: "0.5" memory: "50Mi"
• name: auxiliary-container image: fedora command: [ "sleep", "inf"] # This container has no resource requests or limits specified.

Pod-Level Limits: The pod-level limits (cpu: "1", memory: "200Mi") establish an absolute boundary for the entire Pod. The sum of resources consumed by all its containers is enforced at this ceiling and cannot be surpassed.

Resource Sharing and Bursting: Containers can dynamically borrow any unused capacity, allowing them to burst as needed, so long as the Pod's aggregate usage stays within the overall limit.

Pod-Level Requests: The pod-level requests (cpu: "1", memory: "100Mi") serve as the foundational resource guarantee for the entire Pod. This value informs the scheduler's placement decision and represents the minimum resources the Pod can rely on during node-level contention.

Container-Level Requests: Container-level requests create a priority system within the Pod's guaranteed budget. Because main-app-container has an explicit request (cpu: "0.5", memory: "50Mi"), it is given precedence for its share of resources under resource pressure over the auxiliary-container, which has no such explicit claim.

Limitations

First of all, in-place resize of pod-level resources is not supported for Kubernetes v1.34 (or earlier). Attempting to modify the pod-level resource limits or requests on a running Pod results in an error: the resize is rejected. The v1.34 implementation of Pod level resources focuses on allowing initial declaration of an overall resource envelope, that applies to the entire Pod. That is distinct from in-place pod resize, which (despite what the name might suggest) allows you to make dynamic adjustments to container resource requests and limits, within a running Pod, and potentially without a container restart. In-place resizing is also not yet a stable feature; it graduated to Beta in the v1.33 release.

Only CPU, memory, and hugepages resources can be specified at pod-level.

Pod-level resources are not supported for Windows pods. If the Pod specification explicitly targets Windows (e.g., by setting spec.os.name: "windows"), the API server will reject the Pod during the validation step. If the Pod is not explicitly marked for Windows but is scheduled to a Windows node (e.g., via a nodeSelector), the Kubelet on that Windows node will reject the Pod during its admission process.

The Topology Manager, Memory Manager and CPU Manager do not align pods and containers based on pod-level resources as these resource managers don't currently support pod-level resources.

Getting started and providing feedback

Ready to explore Pod Level Resources feature? You'll need a Kubernetes cluster running version 1.34 or later. Remember to enable the PodLevelResources feature gate across your control plane and all nodes.

As this feature moves through Beta, your feedback is invaluable. Please report any issues or share your experiences via the standard Kubernetes communication channels:

Slack: #sig-node

Mailing list

Open Community Issues/PRs

via Kubernetes Blog https://kubernetes.io/

September 22, 2025 at 02:30PM

Blog: Spotlight on the Kubernetes Steering Committee

https://www.kubernetes.dev/blog/2025/09/22/k8s-steering-spotlight-2025/

This interview was conducted in August 2024, and due to the dynamic nature of the Steering Committee membership and election process it might not represent the actual composition accurately. The topics covered are, however, overwhelmingly relevant to understand its scope of work. As we approach the Steering Committee elections, it provides useful insights into the workings of the Committee.

The Kubernetes Steering Committee is the backbone of the Kubernetes project, ensuring that its vibrant community and governance structures operate smoothly and effectively. While the technical brilliance of Kubernetes is often spotlighted through its Special Interest Groups (SIGs) and Working Groups (WGs), the unsung heroes quietly steering the ship are the members of the Steering Committee. They tackle complex organizational challenges, empower contributors, and foster the thriving open source ecosystem that Kubernetes is celebrated for.

But what does it really take to lead one of the world’s largest open source communities? What are the hidden challenges, and what drives these individuals to dedicate their time and effort to such an impactful role? In this exclusive conversation, we sit down with current Steering Committee (SC) members — Ben, Nabarun, Paco, Patrick, and Maciej — to uncover the rewarding, and sometimes demanding, realities of steering Kubernetes. From their personal journeys and motivations to the committee’s vital responsibilities and future outlook, this Spotlight offers a rare behind-the-scenes glimpse into the people who keep Kubernetes on course.

Introductions

Sandipan: Can you tell us a little bit about yourself?

Ben: Hi, I’m Benjamin Elder, also known as BenTheElder. I started in Kubernetes as a Google Summer of Code student in 2015 and have been working at Google in the space since 2017. I have contributed a lot to many areas but especially build, CI, test tooling, etc. My favorite project so far was building KIND. I have been on the release team, a chair of SIG Testing, and currently a tech lead of SIG Testing and SIG K8s Infra.

Nabarun: Hi, I am Nabarun from India. I have been working on Kubernetes since 2019. I have been contributing across multiple areas in Kubernetes: SIG ContribEx (where I am also a chair), API Machinery, Architecture, and SIG Release, where I contributed to several releases including being the Release Team Lead of Kubernetes 1.21.

Paco: I am Paco from China. I worked as an open source team lead in DaoCloud, Shanghai. In the community, I participate mainly in kubeadm, SIG Node and SIG Testing. Besides, I helped in KCD China and was co-chair of the recent KubeCon+CloudNativeCon China 2024 in Hong Kong.

Patrick: Hello! I’m Patrick. I’ve contributed to Kubernetes since 2018. I started in SIG Storage and then got involved in more and more areas. Nowadays, I am a SIG Testing tech lead, logging infrastructure maintainer, organizer of the Structured Logging and Device Management working groups, contributor in SIG Scheduling, and of course member of the Steering Committee. My main focus area currently is Dynamic Resource Allocation (DRA), a new API for accelerators.

Maciej: Hey, my name is Maciej and I’ve been working on Kubernetes since late 2014 in various areas, including controllers, apiserver and kubectl. Aside from being part of the Steering Committee, I’m also helping guide SIG CLI, SIG Apps and WG Batch.

About the Steering Committee

Sandipan: What does Steering do?

Ben: The charter is the definitive answer, but I see Steering as helping resolve Kubernetes-organization-level “people problems” (as opposed to technical problems), such as clarifying project governance and liaising with the Cloud Native Computing Foundation (for example, to request additional resources and support) and other CNCF projects.

Maciej: Our charter nicely describes all the responsibilities. In short, we make sure the project runs smoothly by supporting our maintainers and contributors in their daily tasks.

Patrick: Ideally, we don’t do anything 😀 All of the day-to-day business has been delegated to SIGs and WGs. Steering gets involved when something pops up where it isn’t obvious who should handle it or when conflicts need to be resolved.

**Sandipan: And how is Steering different from SIGs?

Ben: From a governance perspective: Steering delegates all of the ownership of subprojects to the SIGs and/or committees (Security Response, Code Of Conduct, etc.). They’re very different. The SIGs own pieces of the project, and Steering handles some of the overarching people and policy issues. You’ll find all of the software development, releasing, communications and documentation work happening in the SIGs and committees.

Maciej: SIGs or WGs are primarily concerned with the technical direction of a particular area in Kubernetes. Steering, on the other hand, is primarily concerned with ensuring all the SIGs, WGs, and most importantly maintainers have everything they need to run the project smoothly. This includes anything from ensuring financing of our CI systems, through governance structures and policies all the way to supporting individual maintainers in various inquiries.

**Sandipan: You’ve mentioned projects, could you give us an example of a project Steering has worked on recently?

Ben: We’ve been discussing the logistics to sync a better definition of the project’s official maintainers to the CNCF, which are used, for example, to vote for the Technical Oversight Committee (TOC). Currently that list is the Steering Committee, with SIG Contributor Experience and Infra + Release leads having access to the CNCF service desk. This isn’t well standardized yet across CNCF projects but I think it’s important.

Maciej: For the past year I’ve been sitting on the SC, I believe the majority of tasks we’ve been involved in were around providing letters supporting visa applications. Also, like every year, we’ve been helping all the SIGs and WGs with their annual reports.

Patrick: Apparently it has been a quiet year since Maciej and I joined the Steering Committee at the end of 2023. That’s exactly how it should be.

Sandipan: Do you have any examples of projects that came to Steering, which you then redirected to SIGs?

Ben: We often get requests for test/build related resources that we redirect to SIG K8s Infra + SIG Testing, or more specifically about releasing for subprojects that we redirect to SIG K8s Infra / SIG Release.

The road to the Steering Committee

Sandipan: What motivated you to be part of the Steering Committee? What has your journey been like?

Ben: I had a few people reach out and prompt me to run, but I was motivated by my passion for this community and the project. I think we have something really special going here and I care deeply about the ongoing success. I’ve been involved in this space my whole career and while there’s always rough edges, this community has been really supportive and I hope we can keep it that way.

Paco: After my journey to the Kubernetes Contributor Summit EU 2023, I met and chatted with many maintainers and members there, and attended the steering AMA there for the first time as there hadn’t been a contributor summit in China since 2019, and I started to connect with contributors in China to make it later the year. Through conversations at KCS EU and with local contributors, I realized that it is quite important to make it easy to start a contributor journey for APAC contributors and want to attract more contributors to the community. Then, I was elected just after the KCS CN 2023.

Patrick: I had done a lot of technical work, of which some affects and (hopefully) benefits all contributors to Kubernetes (linting and testing) and users (better log output). I saw joining the Steering Committee as an opportunity to help also with the organizational aspects of running a big open source project.

Maciej: I’ve been going through the idea of running for SC for a while now. My biggest drive was conversations with various members of our community. Eventually last year, I decided to follow their advice, and got elected :-)

Sandipan: What is your favorite part of being part of Steering?

Ben: When we get to help contributors directly. For example, sometimes extensive contributors reach out for an official letter from Steering explaining their contribution and its value for visa support. When we get to just purely help out Kubernetes contributors, that’s my favorite part.

Patrick: It’s a good place to learn more about how the project is actually run, directly from the other great people who are doing it.

Maciej: The same thing as with the project — it’s always the people that surround us, that give us opportunities to collaborate and create something interesting and exciting.

Sandipan: What do you think is most challenging about being part of Steering?

Ben: I think we’ve all spent a lot of time grappling with the sustainability issues in the project and not having a single great answer to solve them. A lot of people are working on these problems but we have limited time and resources. We’ve officially delegated most of this (for example, to SIGs Contributor Experience and K8s Infra), but I think we all still consider it very important and deserving of more time and energy, yet we only have so much and the answers are not obvious. The balancing act is hard.

Paco: Sustainability of contributors and maintainers is one of the most challenging aspects to me. I am constantly advocating for OSS users and employers to join the community. Community is a place that developers can learn from each other, discuss issues they encounter, and share their experience or solutions. Ensuring everyone in the community to feel supported and valued is crucial for the long-term health of the project.

Patrick: There is documentation about how things are done,

Teaching AI Your Company Policies: Vector Search + Enforcement

Ever wondered why AI keeps failing at simple infrastructure tasks? The problem isn't AI itself - it's that AI doesn't know your company's policies. Most organizations have their rules scattered across wikis, Slack messages, and locked in people's heads, making it impossible for AI to make compliant decisions.

This video demonstrates a different approach: extracting tribal knowledge from your brain and turning it into both AI-searchable policies and automatic Kubernetes enforcement. Using a guided workflow, we'll create database regional compliance policies that simultaneously feed semantic search for AI guidance and generate Kyverno policies for cluster enforcement. Watch as AI learns to proactively recommend compliant configurations while Kubernetes blocks any attempts to violate your rules - creating a dual strategy that works whether someone follows the guidance or tries to bypass it entirely.

KubernetesPolicies #DevOpsAI #Kyverno

Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join

▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/ai/teaching-ai-your-company-policies-vector-search-+-enforcement 🔗 DevOps AI Toolkit: https://github.com/vfarcic/dot-ai 🎬 Stop Blaming AI: Vector DBs + RAG = Game Changer: https://youtu.be/zqpJr1qZhTg

▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 What are Policies? 05:36 AI Policy Extraction 13:36 Policy Enforcement Demo 17:51 Dual Policy Strategy

via YouTube https://www.youtube.com/watch?v=hLK9j2cn6c0

Kubernetes v1.34: Recovery From Volume Expansion Failure (GA)

https://kubernetes.io/blog/2025/09/19/kubernetes-v1-34-recover-expansion-failure/

Have you ever made a typo when expanding your persistent volumes in Kubernetes? Meant to specify 2TB but specified 20TiB? This seemingly innocuous problem was kinda hard to fix - and took the project almost 5 years to fix. Automated recovery from storage expansion has been around for a while in beta; however, with the v1.34 release, we have graduated this to general availability.

While it was always possible to recover from failing volume expansions manually, it usually required cluster-admin access and was tedious to do (See aformentioned link for more information).

What if you make a mistake and then realize immediately? With Kubernetes v1.34, you should be able to reduce the requested size of the PersistentVolumeClaim (PVC) and, as long as the expansion to previously requested size hadn't finished, you can amend the size requested. Kubernetes will automatically work to correct it. Any quota consumed by failed expansion will be returned to the user and the associated PersistentVolume should be resized to the latest size you specified.

I'll walk through an example of how all of this works.

Reducing PVC size to recover from failed expansion

Imagine that you are running out of disk space for one of your database servers, and you want to expand the PVC from previously specified 10TB to 100TB - but you make a typo and specify 1000TB.

kind: PersistentVolumeClaim apiVersion: v1 metadata: name: myclaim spec: accessModes:

• ReadWriteOnce resources: requests: storage: 1000TB # newly specified size - but incorrect!

Now, you may be out of disk space on your disk array or simply ran out of allocated quota on your cloud-provider. But, assume that expansion to 1000TB is never going to succeed.

In Kubernetes v1.34, you can simply correct your mistake and request a new PVC size, that is smaller than the mistake, provided it is still larger than the original size of the actual PersistentVolume.

kind: PersistentVolumeClaim apiVersion: v1 metadata: name: myclaim spec: accessModes:

• ReadWriteOnce resources: requests: storage: 100TB # Corrected size; has to be greater than 10TB. # You cannot shrink the volume below its actual size.

This requires no admin intervention. Even better, any surplus Kubernetes quota that you temporarily consumed will be automatically returned.

This fault recovery mechanism does have a caveat: whatever new size you specify for the PVC, it must be still higher than the original size in .status.capacity. Since Kubernetes doesn't support shrinking your PV objects, you can never go below the size that was originally allocated for your PVC request.

Improved error handling and observability of volume expansion

Implementing what might look like a relatively minor change also required us to almost fully redo how volume expansion works under the hood in Kubernetes. There are new API fields available in PVC objects which you can monitor to observe progress of volume expansion.

Improved observability of in-progress expansion

You can query .status.allocatedResourceStatus['storage'] of a PVC to monitor progress of a volume expansion operation. For a typical block volume, this should transition between ControllerResizeInProgress, NodeResizePending and NodeResizeInProgress and become nil/empty when volume expansion has finished.

If for some reason, volume expansion to requested size is not feasible it should accordingly be in states like - ControllerResizeInfeasible or NodeResizeInfeasible.

You can also observe size towards which Kubernetes is working by watching pvc.status.allocatedResources.

Improved error handling and reporting

Kubernetes should now retry your failed volume expansions at slower rate, it should make fewer requests to both storage system and Kubernetes apiserver.

Errors observerd during volume expansion are now reported as condition on PVC objects and should persist unlike events. Kubernetes will now populate pvc.status.conditions with error keys ControllerResizeError or NodeResizeError when volume expansion fails.

Fixes long standing bugs in resizing workflows

This feature also has allowed us to fix long standing bugs in resizing workflow such as Kubernetes issue #115294. If you observe anything broken, please report your bugs to https://github.com/kubernetes/kubernetes/issues, along with details about how to reproduce the problem.

Working on this feature through its lifecycle was challenging and it wouldn't have been possible to reach GA without feedback from @msau42, @jsafrane and @xing-yang.

All of the contributors who worked on this also appreciate the input provided by @thockin and @liggitt at various Kubernetes contributor summits.

via Kubernetes Blog https://kubernetes.io/

September 19, 2025 at 02:30PM

Week Ending September 14, 2025

https://lwkd.info/2025/20250918

Developer News

The Steering Committee Election is underway. Please make sure to vote before October 25th, and request an exception if you need one before October 20th.

The Kubernetes Steering Committee reaffirmed that SIG Release and the Release Team have full authority to enforce policies, deadlines, and requirements, including blocking releases if needed. Steering does not override release execution but will back policy updates and clearer communication to ensure safe, stable, and predictable releases.

A medium-severity vulnerability (CVE-2025-9708) affects the Kubernetes C# client ≤ v17.0.13, where improper certificate validation could enable man-in-the-middle attacks. Users are advised to upgrade to v17.0.14+ and review any custom CA usage in kubeconfig files. See the GitHub issue. for more details.

Release Schedule

Next Deadline: 1.35 Release Cycle Starts, September 15

Kubernetes 1.35 release cycle kicks off on Sept 15, targeting final release on Dec 17, 2025, with key milestones including Enhancements Freeze on Oct 16 and Code Freeze on Nov 6.

Patch releases v1.34.1, v1.33.5, v1.32.9, v1.31.13 were out last week, delivering the latest fix and updates.

KEP of the Week

KEP-3243: Respect PodTopologySpread after rolling upgrades

This KEP introduces a complementary field, MatchLabelKeys, in TopologySpreadConstraint to enhance pod topology spread. It allows users to specify only label keys, with kube-apiserver resolving their values from the incoming pod and merging them with the existing LabelSelector to identify the target pod group. This simplifies skew calculation, supports revision-level spreading during Deployment rollouts, and is also handled by kube-scheduler when used in cluster-level default constraints.

This KEP is tracked for beta in v1.34.

Other Merges

Remove container name from container event messages

Replace NewIndexerInformerWatcher with NewIndexerInformerWatcherWithLogger

Standardize not found error message of kubectl scale

validation-gen uses JSON names for error paths

Prevent ClusterIP load balancer loss with InternalTrafficPolicy: Local in kube-proxy

Avoid deadlock when gRPC connection to driver goes idle

validation-gen adds uuid format for string fields

client-go/cli-runtime fixes config override when ClientKey/ClientCertificate are set

Replace WaitForNamedCacheSync with WaitForNamedCacheSyncWithContext

Update PodObservedGenerationTracking description in OpenAPI

kubectl includes container fieldPath in event messages

StorageVersionMigrator adds discovery check to avoid stuck migrations

agnhost adds fake-registry-server for e2e image-pull tests

Add E2e test for cleaning of terminated containers

kube-apiserver protects against delete/finalizer race

Update pod resize test to accept new cpu.weight conversion

DRA accepts implicit device-class extended resource names even when extendedResourceName is set in the DeviceClass

Skip creating storage for non-stored and non-served versions

Allow OpenAPI model package names to be declared by APIs

kubelet fixes negative pod startup duration values

kube-scheduler statusz lists registered paths

applyconfiguration-gen preserves struct and field comments in generated code

Scheduler framework interfaces move to k8s.io/kube-scheduler

CRD validation ratchets the max selectableFields limit

apiserver storage only accesses keys under resourcePrefix

apiserver storage replace SetKeysFunc with EnableResourceSizeEstimation

Subprojects and Dependency Updates

grpc v1.75.0 introduces Spiffe verification, OTel C++ retry metrics, bug fixes, and Python and Ruby updates

nerdctl v2.1.4 adds manifest, export, import commands, improves networking, and drops containerd 1.6 support

vertical-pod-autoscaler v1.4.2 improves logging, fixes updater metrics, adjusts webhook CA, and falls back to eviction on failed updates

via Last Week in Kubernetes Development https://lwkd.info/

September 18, 2025 at 05:00PM

Kubernetes v1.34: DRA Consumable Capacity

https://kubernetes.io/blog/2025/09/18/kubernetes-v1-34-dra-consumable-capacity/

Dynamic Resource Allocation (DRA) is a Kubernetes API for managing scarce resources across Pods and containers. It enables flexible resource requests, going beyond simply allocating N number of devices to support more granular usage scenarios. With DRA, users can request specific types of devices based on their attributes, define custom configurations tailored to their workloads, and even share the same resource among multiple containers or Pods.

In this blog, we focus on the device sharing feature and dive into a new capability introduced in Kubernetes 1.34: DRA consumable capacity, which extends DRA to support finer-grained device sharing.

Background: device sharing via ResourceClaims

From the beginning, DRA introduced the ability for multiple Pods to share a device by referencing the same ResourceClaim. This design decouples resource allocation from specific hardware, allowing for more dynamic and reusable provisioning of devices.

In Kubernetes 1.33, the new support for partitionable devices allowed resource drivers to advertise slices of a device that are available, rather than exposing the entire device as an all-or-nothing resource. This enabled Kubernetes to model shareable hardware more accurately.

But there was still a missing piece: it didn't yet support scenarios where the device driver manages fine-grained, dynamic portions of a device resource — like network bandwidth — based on user demand, or to share those resources independently of ResourceClaims, which are restricted by their spec and namespace.

That’s where consumable capacity for DRA comes in.

Benefits of DRA consumable capacity support

Here's a taste of what you get in a cluster with the DRAConsumableCapacity feature gate enabled.

Device sharing across multiple ResourceClaims or DeviceRequests

Resource drivers can now support sharing the same device — or even a slice of a device — across multiple ResourceClaims or across multiple DeviceRequests.

This means that Pods from different namespaces can simultaneously share the same device, if permitted and supported by the specific DRA driver.

Device resource allocation

Kubernetes extends the allocation algorithm in the scheduler to support allocating a portion of a device's resources, as defined in the capacity field. The scheduler ensures that the total allocated capacity across all consumers never exceeds the device’s total capacity, even when shared across multiple ResourceClaims or DeviceRequests. This is very similar to the way the scheduler allows Pods and containers to share allocatable resources on Nodes; in this case, it allows them to share allocatable (consumable) resources on Devices.

This feature expands support for scenarios where the device driver is able to manage resources within a device and on a per-process basis — for example, allocating a specific amount of memory (e.g., 8 GiB) from a virtual GPU, or setting bandwidth limits on virtual network interfaces allocated to specific Pods. This aims to provide safe and efficient resource sharing.

DistinctAttribute constraint

This feature also introduces a new constraint: DistinctAttribute, which is the complement of the existing MatchAttribute constraint.

The primary goal of DistinctAttribute is to prevent the same underlying device from being allocated multiple times within a single ResourceClaim, which could happen since we are allocating shares (or subsets) of devices. This constraint ensures that each allocation refers to a distinct resource, even if they belong to the same device class.

It is useful for use cases such as allocating network devices connecting to different subnets to expand coverage or provide redundancy across failure domains.

How to use consumable capacity?

DRAConsumableCapacity is introduced as an alpha feature in Kubernetes 1.34. The feature gate DRAConsumableCapacity must be enabled in kubelet, kube-apiserver, kube-scheduler and kube-controller-manager.

--feature-gates=...,DRAConsumableCapacity=true

As a DRA driver developer

As a DRA driver developer writing in Golang, you can make a device within a ResourceSlice allocatable to multiple ResourceClaims (or devices.requests) by setting AllowMultipleAllocations to true.

Device { ... AllowMultipleAllocations: ptr.To(true), ... }

Additionally, you can define a policy to restrict how each device's Capacity should be consumed by each DeviceRequest by defining RequestPolicy field in the DeviceCapacity. The example below shows how to define a policy that requires a GPU with 40 GiB of memory to allocate at least 5 GiB per request, with each allocation in multiples of 5 GiB.

DeviceCapacity{ Value: resource.MustParse("40Gi"), RequestPolicy: &CapacityRequestPolicy{ Default: ptr.To(resource.MustParse("5Gi")), ValidRange: &CapacityRequestPolicyRange { Min: ptr.To(resource.MustParse("5Gi")), Step: ptr.To(resource.MustParse("5Gi")), } } }

This will be published to the ResourceSlice, as partially shown below:

apiVersion: resource.k8s.io/v1 kind: ResourceSlice ... spec: devices:

• name: gpu0 allowMultipleAllocations: true capacity: memory: value: 40Gi requestPolicy: default: 5Gi validRange: min: 5Gi step: 5Gi

An allocated device with a specified portion of consumed capacity will have a ShareID field set in the allocation status.

claim.Status.Allocation.Devices.Results[i].ShareID

This ShareID allows the driver to distinguish between different allocations that refer to the same device or same statically-partitioned slice but come from different ResourceClaim requests.

It acts as a unique identifier for each shared slice, enabling the driver to manage and enforce resource limits independently across multiple consumers.

As a consumer

As a consumer (or user), the device resource can be requested with a ResourceClaim like this:

apiVersion: resource.k8s.io/v1 kind: ResourceClaim ... spec: devices: requests: # for devices

• name: req0 exactly:
• deviceClassName: resource.example.com capacity: requests: # for resources which must be provided by those devices memory: 10Gi

This configuration ensures that the requested device can provide at least 10GiB of memory.

Notably that any resource.example.com device that has at least 10GiB of memory can be allocated. If a device that does not support multiple allocations is chosen, the allocation would consume the entire device. To filter only devices that support multiple allocations, you can define a selector like this:

selectors:

• cel: expression: |- device.allowMultipleAllocations == true

Integration with DRA device status

In device sharing, general device information is provided through the resource slice. However, some details are set dynamically after allocation. These can be conveyed using the .status.devices field of a ResourceClaim. That field is only published in clusters where the DRAResourceClaimDeviceStatus feature gate is enabled.

If you do have device status support available, a driver can expose additional device-specific information beyond the ShareID. One particularly useful use case is for virtual networks, where a driver can include the assigned IP address(es) in the status. This is valuable for both network service operations and troubleshooting.

You can find more information by watching our recording at: KubeCon Japan 2025 - Reimagining Cloud Native Networks: The Critical Role of DRA.

What can you do next?

Check out the CNI DRA Driver project for an example of DRA integration in Kubernetes networking. Try integrating with network resources like macvlan, ipvlan, or smart NICs.

Start enabling the DRAConsumableCapacity feature gate and experimenting with virtualized or partitionable devices. Specify your workloads with consumable capacity (for example: fractional bandwidth or memory).

Let us know your feedback:

✅ What worked well?

⚠️ What didn’t?

If you encountered issues to fix or opportunities to enhance, please file a new issue and reference KEP-5075 there, or reach out via Slack (#wg-device-management).

Conclusion

Consumable capacity support enhances the device sharing capability of DRA by allowing effective device sharing across namespaces, across claims, and tailored to each Pod’s actual needs. It also empowers drivers to enforce capacity limits, improves scheduling accuracy, and unlocks new use cases like bandwidth-aware networking and multi-tenant device sharing.

Try it out, experiment with consumable resources, and help shape the future of dynamic resource allocation in Kubernetes!

Further Reading

DRA in the Kubernetes documentation

KEP for DRA Partitionable Devices

KEP for DRA Device Status

KEP for DRA Consumable Capacity

Kubernetes 1.34 Release Notes

via Kubernetes Blog https://kubernetes.io/

September 18, 2025 at 02:30PM

Kubernetes v1.34: Pods Report DRA Resource Health

https://kubernetes.io/blog/2025/09/17/kubernetes-v1-34-pods-report-dra-resource-health/

The rise of AI/ML and other high-performance workloads has made specialized hardware like GPUs, TPUs, and FPGAs a critical component of many Kubernetes clusters. However, as discussed in a previous blog post about navigating failures in Pods with devices, when this hardware fails, it can be difficult to diagnose, leading to significant downtime. With the release of Kubernetes v1.34, we are excited to announce a new alpha feature that brings much-needed visibility into the health of these devices.

This work extends the functionality of KEP-4680, which first introduced a mechanism for reporting the health of devices managed by Device Plugins. Now, this capability is being extended to Dynamic Resource Allocation (DRA). Controlled by the ResourceHealthStatus feature gate, this enhancement allows DRA drivers to report device health directly into a Pod's .status field, providing crucial insights for operators and developers.

Why expose device health in Pod status?

For stateful applications or long-running jobs, a device failure can be disruptive and costly. By exposing device health in the .status field for a Pod, Kubernetes provides a standardized way for users and automation tools to quickly diagnose issues. If a Pod is failing, you can now check its status to see if an unhealthy device is the root cause, saving valuable time that might otherwise be spent debugging application code.

How it works

This feature introduces a new, optional communication channel between the Kubelet and DRA drivers, built on three core components.

A new gRPC health service

A new gRPC service, DRAResourceHealth, is defined in the dra-health/v1alpha1 API group. DRA drivers can implement this service to stream device health updates to the Kubelet. The service includes a NodeWatchResources server-streaming RPC that sends the health status (Healthy, Unhealthy, or Unknown) for the devices it manages.

Kubelet integration

The Kubelet’s DRAPluginManager discovers which drivers implement the health service. For each compatible driver, it starts a long-lived NodeWatchResources stream to receive health updates. The DRA Manager then consumes these updates and stores them in a persistent healthInfoCache that can survive Kubelet restarts.

Populating the Pod status

When a device's health changes, the DRA manager identifies all Pods affected by the change and triggers a Pod status update. A new field, allocatedResourcesStatus, is now part of the v1.ContainerStatus API object. The Kubelet populates this field with the current health of each device allocated to the container.

A practical example

If a Pod is in a CrashLoopBackOff state, you can use kubectl describe pod <pod-name> to inspect its status. If an allocated device has failed, the output will now include the allocatedResourcesStatus field, clearly indicating the problem:

status: containerStatuses:

• name: my-gpu-intensive-container # ... other container statuses allocatedResourcesStatus:
• name: "claim:my-gpu-claim" resources:
• resourceID: "example.com/gpu-a1b2-c3d4" health: "Unhealthy"

This explicit status makes it clear that the issue is with the underlying hardware, not the application.

Now you can improve the failure detection logic to react on the unhealthy devices associated with the Pod by de-scheduling a Pod.

How to use this feature

As this is an alpha feature in Kubernetes v1.34, you must take the following steps to use it:

Enable the ResourceHealthStatus feature gate on your kube-apiserver and kubelets.

Ensure you are using a DRA driver that implements the v1alpha1 DRAResourceHealth gRPC service.

DRA drivers

If you are developing a DRA driver, make sure to think about device failure detection strategy and ensure that your driver is integrated with this feature. This way, your driver will improve the user experience and simplify debuggability of hardware issues.

What's next?

This is the first step in a broader effort to improve how Kubernetes handles device failures. As we gather feedback on this alpha feature, the community is planning several key enhancements before graduating to Beta:

Detailed health messages: To improve the troubleshooting experience, we plan to add a human-readable message field to the gRPC API. This will allow DRA drivers to provide specific context for a health status, such as "GPU temperature exceeds threshold" or "NVLink connection lost".

Configurable health timeouts: The timeout for marking a device's health as "Unknown" is currently hardcoded. We plan to make this configurable, likely on a per-driver basis, to better accommodate the different health-reporting characteristics of various hardware.

Improved post-mortem troubleshooting: We will address a known limitation where health updates may not be applied to pods that have already terminated. This fix will ensure that the health status of a device at the time of failure is preserved, which is crucial for troubleshooting batch jobs and other "run-to-completion" workloads.

This feature was developed as part of KEP-4680, and community feedback is crucial as we work toward graduating it to Beta. We have more improvements of device failure handling in k8s and encourage you to try it out and share your experiences with the SIG Node community!

via Kubernetes Blog https://kubernetes.io/

September 17, 2025 at 02:30PM

Kubernetes v1.34: Moving Volume Group Snapshots to v1beta2

https://kubernetes.io/blog/2025/09/16/kubernetes-v1-34-volume-group-snapshot-beta-2/

Volume group snapshots were introduced as an Alpha feature with the Kubernetes 1.27 release and moved to Beta in the Kubernetes 1.32 release. The recent release of Kubernetes v1.34 moved that support to a second beta. The support for volume group snapshots relies on a set of extension APIs for group snapshots. These APIs allow users to take crash consistent snapshots for a set of volumes. Behind the scenes, Kubernetes uses a label selector to group multiple PersistentVolumeClaims for snapshotting. A key aim is to allow you restore that set of snapshots to new volumes and recover your workload based on a crash consistent recovery point.

This new feature is only supported for CSI volume drivers.

What's new in Beta 2?

While testing the beta version, we encountered an issue where the restoreSize field is not set for individual VolumeSnapshotContents and VolumeSnapshots if CSI driver does not implement the ListSnapshots RPC call. We evaluated various options here and decided to make this change releasing a new beta for the API.

Specifically, a VolumeSnapshotInfo struct is added in v1beta2, it contains information for an individual volume snapshot that is a member of a volume group snapshot. VolumeSnapshotInfoList, a list of VolumeSnapshotInfo, is added to VolumeGroupSnapshotContentStatus, replacing VolumeSnapshotHandlePairList. VolumeSnapshotInfoList is a list of snapshot information returned by the CSI driver to identify snapshots on the storage system. VolumeSnapshotInfoList is populated by the csi-snapshotter sidecar based on the CSI CreateVolumeGroupSnapshotResponse returned by the CSI driver's CreateVolumeGroupSnapshot call.

The existing v1beta1 API objects will be converted to the new v1beta2 API objects by a conversion webhook.

What’s next?

Depending on feedback and adoption, the Kubernetes project plans to push the volume group snapshot implementation to general availability (GA) in a future release.

How can I learn more?

The design spec for the volume group snapshot feature.

The code repository for volume group snapshot APIs and controller.

CSI documentation on the group snapshot feature.

How do I get involved?

This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. On behalf of SIG Storage, I would like to offer a huge thank you to the contributors who stepped up these last few quarters to help the project reach beta:

Ben Swartzlander (bswartz)

Hemant Kumar (gnufied)

Jan Šafránek (jsafrane)

Madhu Rajanna (Madhu-1)

Michelle Au (msau42)

Niels de Vos (nixpanic)

Leonardo Cecchi (leonardoce)

Saad Ali (saad-ali)

Xing Yang (xing-yang)

Yati Padia (yati1998)

For those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the Kubernetes Storage Special Interest Group (SIG). We always welcome new contributors.

We also hold regular Data Protection Working Group meetings. New attendees are welcome to join our discussions.

via Kubernetes Blog https://kubernetes.io/

September 16, 2025 at 02:30PM

VerticalPodAutoscaler Went Rogue: It Took Down Our Cluster, with Thibault Jamet

https://ku.bz/rf1pbWXdN

Running 30 Kubernetes clusters serving 300,000 requests per second sounds impressive until your Vertical Pod Autoscaler goes rogue and starts evicting critical system pods in an endless loop.

Thibault Jamet shares the technical details of debugging a complex VPA failure at Adevinta, where webhook timeouts triggered continuous pod evictions across their multi-tenant Kubernetes platform.

You will learn:

VPA architecture deep dive - How the recommender, updater, and mutating webhook components interact and what happens when the webhook fails

Hidden Kubernetes limits - How default QPS and burst rate limits in the Kubernetes Go client can cause widespread failures, and why these aren't well documented in Helm charts

Monitoring strategies for autoscaling - What metrics to track for webhook latency and pod eviction rates to catch similar issues before they become critical

Sponsor

This episode is brought to you by Testkube—where teams run millions of performance tests in real Kubernetes infrastructure. From air-gapped environments to massive scale deployments, orchestrate every testing tool in one platform. Check it out at testkube.io

More info

Find all the links and info for this episode here: https://ku.bz/rf1pbWXdN

Interested in sponsoring an episode? Learn more.

via KubeFM https://kube.fm

September 16, 2025 at 06:00AM

Kubernetes v1.34: Decoupled Taint Manager Is Now Stable

https://kubernetes.io/blog/2025/09/15/kubernetes-v1-34-decoupled-taint-manager-is-now-stable/

This enhancement separates the responsibility of managing node lifecycle and pod eviction into two distinct components. Previously, the node lifecycle controller handled both marking nodes as unhealthy with NoExecute taints and evicting pods from them. Now, a dedicated taint eviction controller manages the eviction process, while the node lifecycle controller focuses solely on applying taints. This separation not only improves code organization but also makes it easier to improve taint eviction controller or build custom implementations of the taint based eviction.

What's new?

The feature gate SeparateTaintEvictionController has been promoted to GA in this release. Users can optionally disable taint-based eviction by setting --controllers=-taint-eviction-controller in kube-controller-manager.

How can I learn more?

For more details, refer to the KEP and to the beta announcement article: Kubernetes 1.29: Decoupling taint manager from node lifecycle controller.

How to get involved?

We offer a huge thank you to all the contributors who helped with design, implementation, and review of this feature and helped move it from beta to stable:

Ed Bartosh (@bart0sh)

Yuan Chen (@yuanchen8911)

Aldo Culquicondor (@alculquicondor)

Baofa Fan (@carlory)

Sergey Kanzhelev (@SergeyKanzhelev)

Tim Bannister (@lmktfy)

Maciej Skoczeń (@macsko)

Maciej Szulik (@soltysh)

Wojciech Tyczynski (@wojtek-t)

via Kubernetes Blog https://kubernetes.io/

September 15, 2025 at 02:30PM

Terminal Agents: Codex vs. Crush vs. OpenCode vs. Cursor CLI vs. Claude Code

I love Claude Code, but I hate being locked into Anthropic models. What if I want to use GPT5 or whatever comes out next week? So I went on a quest to find a terminal-based coding agent that works with different models and doesn't suck compared to Claude Code.

I tested every terminal agent I could find: Codex CLI from OpenAI, Charm Crush, OpenCode, and Cursor CLI. My requirements were simple - intuitive interface, MCP servers support, saved prompts, and actual functionality for coding and operations. The results were... disappointing. From agents that couldn't even fetch their own documentation to beautiful UIs that prioritized looks over functionality, each had critical flaws that made them unusable for real work. Even GPT5, hyped as the best coding model ever, couldn't shine through these broken wrappers. By the end, you'll understand why having a great model isn't enough - you need the complete package, and right now, that's still painfully rare in the terminal agent space.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 👉 Grab your free seat to the 2-Day AI Mastermind: https://link.outskill.com/AIDOS2 🔐 100% Discount for the first 1000 people 💥 Dive deep into AI and Learn Automations, Build AI Agents, Make videos & images – all for free! 🎁 Bonuses worth $5100+ if you join and attend ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

TerminalAgents #CodingAI #GPT5

Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join

▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/ai/terminal-agents-codex-vs.-crush-vs.-opencode-vs.-cursor-cli-vs.-claude-code

▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 Terminal-Based Coding Agents 01:09 Outskill (sponsor) 02:31 Why Terminal AI Agents Matter 06:54 Codex CLI - OpenAI's Terminal Agent 12:03 Charm Crush - Beautiful Terminal UI Agent 17:18 OpenCode - SST's Terminal Agent 20:13 Cursor CLI - From Cursor IDE Makers 24:10 Terminal AI Agents - Final Verdict

via YouTube https://www.youtube.com/watch?v=MXOP4WELkCc

The making of Flux: The origin

https://ku.bz/5Sf5wpd8y

This episode unpacks the technical and governance milestones that secured Flux's place in the cloud-native ecosystem, from a 45-minute production outage that led to the birth of GitOps to the CNCF process that defines project maturity and the handover of stewardship after Weaveworks' closure.

You will learn:

How a single incident pushed Weaveworks to adopt Git as the source of truth, creating the foundation of GitOps.

How Flux sustained continuity after Weaveworks shut down through community governance.

Where Flux is heading next with security guidance, Flux v2, and an enterprise-ready roadmap.

Sponsor

Join the Flux maintainers and community at FluxCon, November 11th in Salt Lake City—register here

More info

Find all the links and info for this episode here: https://ku.bz/5Sf5wpd8y

Interested in sponsoring an episode? Learn more.

via KubeFM https://kube.fm

September 15, 2025 at 06:00AM

Kubernetes v1.34: Autoconfiguration for Node Cgroup Driver Goes GA

https://kubernetes.io/blog/2025/09/12/kubernetes-v1-34-cri-cgroup-driver-lookup-now-ga/

Historically, configuring the correct cgroup driver has been a pain point for users running new Kubernetes clusters. On Linux systems, there are two different cgroup drivers: cgroupfs and systemd. In the past, both the kubelet and CRI implementation (like CRI-O or containerd) needed to be configured to use the same cgroup driver, or else the kubelet would misbehave without any explicit error message. This was a source of headaches for many cluster admins. Now, we've (almost) arrived at the end of that headache.

Automated cgroup driver detection

In v1.28.0, the SIG Node community introduced the feature gate KubeletCgroupDriverFromCRI, which instructs the kubelet to ask the CRI implementation which cgroup driver to use. You can read more here. After many releases of waiting for each CRI implementation to have major versions released and packaged in major operating systems, this feature has gone GA as of Kubernetes 1.34.0.

In addition to setting the feature gate, a cluster admin needs to ensure their CRI implementation is new enough:

containerd: Support was added in v2.0.0

CRI-O: Support was added in v1.28.0

Announcement: Kubernetes is deprecating containerd v1.y support

While CRI-O releases versions that match Kubernetes versions, and thus CRI-O versions without this behavior are no longer supported, containerd maintains its own release cycle. containerd support for this feature is only in v2.0 and later, but Kubernetes 1.34 still supports containerd 1.7 and other LTS releases of containerd.

The Kubernetes SIG Node community has formally agreed upon a final support timeline for containerd v1.y. The last Kubernetes release to offer this support will be the last released version of v1.35, and support will be dropped in v1.36.0. To assist administrators in managing this future transition, a new detection mechanism is available. You are able to monitor the kubelet_cri_losing_support metric to determine if any nodes in your cluster are using a containerd version that will soon be outdated. The presence of this metric with a version label of 1.36.0 will indicate that the node's containerd runtime is not new enough for the upcoming requirements. Consequently, an administrator will need to upgrade containerd to v2.0 or a later version before, or at the same time as, upgrading the kubelet to v1.36.0.

via Kubernetes Blog https://kubernetes.io/

September 12, 2025 at 02:30PM

Kubernetes v1.34: Mutable CSI Node Allocatable Graduates to Beta

https://kubernetes.io/blog/2025/09/11/kubernetes-v1-34-mutable-csi-node-allocatable-count/

The functionality for CSI drivers to update information about attachable volume count on the nodes, first introduced as Alpha in Kubernetes v1.33, has graduated to Beta in the Kubernetes v1.34 release! This marks a significant milestone in enhancing the accuracy of stateful pod scheduling by reducing failures due to outdated attachable volume capacity information.

Background

Traditionally, Kubernetes CSI drivers report a static maximum volume attachment limit when initializing. However, actual attachment capacities can change during a node's lifecycle for various reasons, such as:

Manual or external operations attaching/detaching volumes outside of Kubernetes control.

Dynamically attached network interfaces or specialized hardware (GPUs, NICs, etc.) consuming available slots.

Multi-driver scenarios, where one CSI driver’s operations affect available capacity reported by another.

Static reporting can cause Kubernetes to schedule pods onto nodes that appear to have capacity but don't, leading to pods stuck in a ContainerCreating state.

Dynamically adapting CSI volume limits

With this new feature, Kubernetes enables CSI drivers to dynamically adjust and report node attachment capacities at runtime. This ensures that the scheduler, as well as other components relying on this information, have the most accurate, up-to-date view of node capacity.

How it works

Kubernetes supports two mechanisms for updating the reported node volume limits:

Periodic Updates: CSI drivers specify an interval to periodically refresh the node's allocatable capacity.

Reactive Updates: An immediate update triggered when a volume attachment fails due to exhausted resources (ResourceExhausted error).

Enabling the feature

To use this beta feature, the MutableCSINodeAllocatableCount feature gate must be enabled in these components:

kube-apiserver

kubelet

Example CSI driver configuration

Below is an example of configuring a CSI driver to enable periodic updates every 60 seconds:

apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: name: example.csi.k8s.io spec: nodeAllocatableUpdatePeriodSeconds: 60

This configuration directs kubelet to periodically call the CSI driver's NodeGetInfo method every 60 seconds, updating the node’s allocatable volume count. Kubernetes enforces a minimum update interval of 10 seconds to balance accuracy and resource usage.

Immediate updates on attachment failures

When a volume attachment operation fails due to a ResourceExhausted error (gRPC code 8), Kubernetes immediately updates the allocatable count instead of waiting for the next periodic update. The Kubelet then marks the affected pods as Failed, enabling their controllers to recreate them. This prevents pods from getting permanently stuck in the ContainerCreating state.

Getting started

To enable this feature in your Kubernetes v1.34 cluster:

Enable the feature gate MutableCSINodeAllocatableCount on the kube-apiserver and kubelet components.

Update your CSI driver configuration by setting nodeAllocatableUpdatePeriodSeconds.

Monitor and observe improvements in scheduling accuracy and pod placement reliability.

Next steps

This feature is currently in beta and the Kubernetes community welcomes your feedback. Test it, share your experiences, and help guide its evolution to GA stability.

Join discussions in the Kubernetes Storage Special Interest Group (SIG-Storage) to shape the future of Kubernetes storage capabilities.

via Kubernetes Blog https://kubernetes.io/

September 11, 2025 at 02:30PM

Week Ending September 7, 2025

https://lwkd.info/2025/20250910

Developer News

The Kubernetes v1.35 Release Team shadow application is open till Sept 14, 2025, with results by Sept 22 and the release cycle running Sept 15–Dec 17. Learn more in the Release Team Overview, Shadows Guide, Role Handbooks, and Selection Criteria. Updates will be shared in the #sig-release Slack channel and kubernetes/sig-release repo

A medium-severity flaw (CVE-2025-7445) in secrets-store-sync-controller < v0.0.2 may expose service account tokens in logs, risking cloud vault access. Upgrade to v0.0.2+ and check logs for leaked or misused tokens. See Kubernetes CVE detailsKubernetes CVE details here.

Steering Committee Election

The nomination period for the Kubernetes Steering Committee Election has ended.

Now it’s time for your vote! The Steering Committee Election[https://github.com/kubernetes/community/tree/master/elections/steering/2025#voting-process] begins on Friday, 12th September. You can check your eligibility to vote in the voting app, and file an exception request if you need to.

Release Schedule

Next Deadline: 1.35 Release Cycle Starts, September

The Kubernetes v1.35 Release Team shadow application opened on Sept 4 and will close on Sept 14, 2025 (midnight anywhere). Selected applicants will be notified by Sept 22, and the release cycle is expected to run from Sept 15 to Dec 17, 2025. This is a great opportunity to get involved with the release process!

The cherry pick deadlines closed on Sept 5 for Kubernetes 1.33.5, 1.32.9, and 1.31.13, all targeting release on Sept 9, 2025

Featured PRs

133097: Resolve confusing use of TooManyRequests error for eviction

This PR resolves an issue where pod eviction requests could return a TooManyRequests (429) error with an unrelated disruption budget message; The API server now reports a clearer error when eviction is blocked by the fail-safe mechanism in the DisruptionController, avoiding misleading responses.

133890: Fix missing kubeletvolume_stats* metrics

This PR fixes a regression in v1.34 where kubeletvolume_stats* metrics disappeared from the kubelet metrics endpoint; The bug was caused by multiple calls to Register(); The fix ensures the metrics are registered correctly and reported again.

KEP of the Week

KEP 740: Support external signing of service account tokens

This KEP enables Kubernetes to integrate with external key management solutions such as HSMs and cloud KMS for signing service account tokens. It supports out-of-process JWT signing and dynamic public key discovery, improving security and allowing key rotation without restarting kube-apiserver. Existing file-based key management remains supported as a fallback.

This KEP is tracked for beta in v1.34.

Other Merges

DRA kubelet : Avoid deadlock when gRPC connection to driver goes idle

Add k8s-long-name and k8s-short-name format validation tags

Prevent missing kubelet_volume_stats metrics

Show real error reason in pod STATUS when a pod has both Running and Error containers

Migrate plugin-manager logs to contextual logging — improves developer diagnostics, no user-facing impact

Add Close() API to remote runtime/image — enables graceful gRPC cleanup, prevents resource leaks

Add the correct error when eviction is blocked due to the failSafe mechanism of the DisruptionController

Configure JSON content type for generic webhook RESTClient

Disable estimating resource size for resources with watch cache disabled

Enforce that all resources set resourcePrefix

Prevent error logs by skipping stats collection for resources missing resourcePrefix

Add paths section to kubelet statusz endpoint

Lock down the AllowOverwriteTerminationGracePeriodSeconds feature gate.

Add +k8s:ifEnabled / +k8s:ifDisabled / +k8s:enumExclude tags for validation

Add stress test for pod cleanup on VolumeAttachmentLimitExceeded

Deprecated

Removed deprecated gogo protocol definitions from k8s.io/kubelet/pkg/apis/dra in favor of google.golang.org/protobuf.

Drop SizeMemoryBackedVolumes after the feature GA-ed in 1.32

Remove GA feature gate ComponentSLIs (now always on)

Version Updates

Update CNI plugins to v1.8.0

Bump gengo to v2.0.0-20250903151518-081d64401ab4

Subprojects and Dependency Updates

cloud-provider-aws v1.34.0 resolves nil pointer dereferences, updates topology labels and EC2 SDK, adds a TG reconciler for NLB hairpinning, and refreshes Go deps

coredns v1.12.4 fixes DoH context propagation, file plugin label offsets, gRPC/transfer leaks, and adds loadbalance prefer and metrics timeouts

cri-o v1.34.0 moves to Kubernetes v1.34 dev, switches to opencontainers/cgroups with runc 1.3, improves container monitoring, and fixes deadlocks and terminal resize issues.

minikube v1.37.0 adds krunkit driver for macOS GPU AI workloads, introduces kubetail addon, supports Kubernetes v1.34.0, deprecates HyperKit, and updates key addons and CNIs

via Last Week in Kubernetes Development https://lwkd.info/

September 10, 2025 at 06:00PM

Kubernetes v1.34: Use An Init Container To Define App Environment Variables

https://kubernetes.io/blog/2025/09/10/kubernetes-v1-34-env-files/

Kubernetes typically uses ConfigMaps and Secrets to set environment variables, which introduces additional API calls and complexity, For example, you need to separately manage the Pods of your workloads and their configurations, while ensuring orderly updates for both the configurations and the workload Pods.

Alternatively, you might be using a vendor-supplied container that requires environment variables (such as a license key or a one-time token), but you don’t want to hard-code them or mount volumes just to get the job done.

If that's the situation you are in, you now have a new (alpha) way to achieve that. Provided you have the EnvFiles feature gate enabled across your cluster, you can tell the kubelet to load a container's environment variables from a volume (the volume must be part of the Pod that the container belongs to). this feature gate allows you to load environment variables directly from a file in an emptyDir volume without actually mounting that file into the container. It’s a simple yet elegant solution to some surprisingly common problems.

What’s this all about?

At its core, this feature allows you to point your container to a file, one generated by an initContainer, and have Kubernetes parse that file to set your environment variables. The file lives in an emptyDir volume (a temporary storage space that lasts as long as the pod does), Your main container doesn’t need to mount the volume. The kubelet will read the file and inject these variables when the container starts.

How It Works

Here's a simple example:

apiVersion: v1 kind: Pod spec: initContainers:

• name: generate-config image: busybox command: ['sh', '-c', 'echo "CONFIG_VAR=HELLO" > /config/config.env'] volumeMounts:
• name: config-volume mountPath: /config containers:
• name: app-container image: gcr.io/distroless/static env:
• name: CONFIG_VAR valueFrom: fileKeyRef: path: config.env volumeName: config-volume key: CONFIG_VAR volumes:
• name: config-volume emptyDir: {}

Using this approach is a breeze. You define your environment variables in the pod spec using the fileKeyRef field, which tells Kubernetes where to find the file and which key to pull. The file itself resembles the standard for .env syntax (think KEY=VALUE), and (for this alpha stage at least) you must ensure that it is written into an emptyDir volume. Other volume types aren't supported for this feature. At least one init container must mount that emptyDir volume (to write the file), but the main container doesn’t need to—it just gets the variables handed to it at startup.

A word on security

While this feature supports handling sensitive data such as keys or tokens, note that its implementation relies on emptyDir volumes mounted into pod. Operators with node filesystem access could therefore easily retrieve this sensitive data through pod directory paths.

If storing sensitive data like keys or tokens using this feature, ensure your cluster security policies effectively protect nodes against unauthorized access to prevent exposure of confidential information.

Summary

This feature will eliminate a number of complex workarounds used today, simplifying apps authoring, and opening doors for more use cases. Kubernetes stays flexible and open for feedback. Tell us how you use this feature or what is missing.

via Kubernetes Blog https://kubernetes.io/

September 10, 2025 at 02:30PM

Ep34 - Ask Me Anything About Anything with Scott Rosenberg

There are no restrictions in this AMA session. You can ask anything about DevOps, AI, Cloud, Kubernetes, Platform Engineering, containers, or anything else. Scott Rosenberg, a regular guest, will be here to help us out.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: Codefresh 🔗 GitOps Argo CD Certifications: https://learning.codefresh.io (use "viktor" for a 50% discount) ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

via YouTube https://www.youtube.com/watch?v=IoBrA6gUESk

Kubernetes v1.34: Snapshottable API server cache

https://kubernetes.io/blog/2025/09/09/kubernetes-v1-34-snapshottable-api-server-cache/

For years, the Kubernetes community has been on a mission to improve the stability and performance predictability of the API server. A major focus of this effort has been taming list requests, which have historically been a primary source of high memory usage and heavy load on the etcd datastore. With each release, we've chipped away at the problem, and today, we're thrilled to announce the final major piece of this puzzle.

The snapshottable API server cache feature has graduated to Beta in Kubernetes v1.34, culminating a multi-release effort to allow virtually all read requests to be served directly from the API server's cache.

Evolving the cache for performance and stability

The path to the current state involved several key enhancements over recent releases that paved the way for today's announcement.

Consistent reads from cache (Beta in v1.31)

While the API server has long used a cache for performance, a key milestone was guaranteeing consistent reads of the latest data from it. This v1.31 enhancement allowed the watch cache to be used for strongly-consistent read requests for the first time, a huge win as it enabled filtered collections (e.g. "a list of pods bound to this node") to be safely served from the cache instead of etcd, dramatically reducing its load for common workloads.

Taming large responses with streaming (Beta in v1.33)

Another key improvement was tackling the problem of memory spikes when transmitting large responses. The streaming encoder, introduced in v1.33, allowed the API server to send list items one by one, rather than buffering the entire multi-gigabyte response in memory. This made the memory cost of sending a response predictable and minimal, regardless of its size.

The missing piece

Despite these huge improvements, a critical gap remained. Any request for a historical LIST—most commonly used for paginating through large result sets—still had to bypass the cache and query etcd directly. This meant that the cost of retrieving the data was still unpredictable and could put significant memory pressure on the API server.

Kubernetes 1.34: snapshots complete the picture

The snapshottable API server cache solves this final piece of the puzzle. This feature enhances the watch cache, enabling it to generate efficient, point-in-time snapshots of its state.

Here’s how it works: for each update, the cache creates a lightweight snapshot. These snapshots are "lazy copies," meaning they don't duplicate objects but simply store pointers, making them incredibly memory-efficient.

When a list request for a historical resourceVersion arrives, the API server now finds the corresponding snapshot and serves the response directly from its memory. This closes the final major gap, allowing paginated requests to be served entirely from the cache.

A new era of API Server performance 🚀

With this final piece in place, the synergy of these three features ushers in a new era of API server predictability and performance:

Get Data from Cache: Consistent reads and snapshottable cache work together to ensure nearly all read requests—whether for the latest data or a historical snapshot—are served from the API server's memory.

Send data via stream: Streaming list responses ensure that sending this data to the client has a minimal and constant memory footprint.

The result is a system where the resource cost of read operations is almost fully predictable and much more resiliant to spikes in request load. This means dramatically reduced memory pressure, a lighter load on etcd, and a more stable, scalable, and reliable control plane for all Kubernetes clusters.

How to get started

With its graduation to Beta, the SnapshottableCache feature gate is enabled by default in Kubernetes v1.34. There are no actions required to start benefiting from these performance and stability improvements.

Acknowledgements

Special thanks for designing, implementing, and reviewing these critical features go to:

Ahmad Zolfaghari (@ah8ad3)

Ben Luddy (@benluddy) – Red Hat

Chen Chen (@z1cheng) – Microsoft

Davanum Srinivas (@dims) – Nvidia

David Eads (@deads2k) – Red Hat

Han Kang (@logicalhan) – CoreWeave

haosdent (@haosdent) – Shopee

Joe Betz (@jpbetz) – Google

Jordan Liggitt (@liggitt) – Google

Łukasz Szaszkiewicz (@p0lyn0mial) – Red Hat

Maciej Borsz (@mborsz) – Google

Madhav Jivrajani (@MadhavJivrajani) – UIUC

Marek Siarkowicz (@serathius) – Google

NKeert (@NKeert)

Tim Bannister (@lmktfy)

Wei Fu (@fuweid) - Microsoft

Wojtek Tyczyński (@wojtek-t) – Google

...and many others in SIG API Machinery. This milestone is a testament to the community's dedication to building a more scalable and robust Kubernetes.

via Kubernetes Blog https://kubernetes.io/

September 09, 2025 at 02:30PM