Gateway API 1.4: New Features

https://kubernetes.io/blog/2025/11/06/gateway-api-v1-4/

Ready to rock your Kubernetes networking? The Kubernetes SIG Network community presented the General Availability (GA) release of Gateway API (v1.4.0)! Released on October 6, 2025, version 1.4.0 reinforces the path for modern, expressive, and extensible service networking in Kubernetes.

Gateway API v1.4.0 brings three new features to the Standard channel (Gateway API's GA release channel):

BackendTLSPolicy for TLS between gateways and backends

supportedFeatures in GatewayClass status

Named rules for Routes

and introduces three new experimental features:

Mesh resource for service mesh configuration

Default gateways to ease configuration burden

externalAuth filter for HTTPRoute

Graduations to Standard Channel

Backend TLS policy

Leads: Candace Holman, Norwin Schnyder, Katarzyna Łach

GEP-1897: BackendTLSPolicy

BackendTLSPolicy is a new Gateway API type for specifying the TLS configuration of the connection from the Gateway to backend pod(s). . Prior to the introduction of BackendTLSPolicy, there was no API specification that allowed encrypted traffic on the hop from Gateway to backend.

The BackendTLSPolicy validation configuration requires a hostname. This hostname serves two purposes. It is used as the SNI header when connecting to the backend and for authentication, the certificate presented by the backend must match this hostname, unless subjectAltNames is explicitly specified.

If subjectAltNames (SANs) are specified, the hostname is only used for SNI, and authentication is performed against the SANs instead. If you still need to authenticate against the hostname value in this case, you MUST add it to the subjectAltNames list.

BackendTLSPolicy validation configuration also requires either caCertificateRefs or wellKnownCACertificates. caCertificateRefs refer to one or more (up to 8) PEM-encoded TLS certificate bundles. If there are no specific certificates to use, then depending on your implementation, you may use wellKnownCACertificates, set to "System" to tell the Gateway to use an implementation-specific set of trusted CA Certificates.

In this example, the BackendTLSPolicy is configured to use certificates defined in the auth-cert ConfigMap to connect with a TLS-encrypted upstream connection where pods backing the auth service are expected to serve a valid certificate for auth.example.com. It uses subjectAltNames with a Hostname type, but you may also use a URI type.

apiVersion: gateway.networking.k8s.io/v1 kind: BackendTLSPolicy metadata: name: tls-upstream-auth spec: targetRefs:

• kind: Service name: auth group: "" sectionName: "https" validation: caCertificateRefs:
• group: "" # core API group kind: ConfigMap name: auth-cert subjectAltNames:
• type: "Hostname" hostname: "auth.example.com"

In this example, the BackendTLSPolicy is configured to use system certificates to connect with a TLS-encrypted backend connection where Pods backing the dev Service are expected to serve a valid certificate for dev.example.com.

apiVersion: gateway.networking.k8s.io/v1 kind: BackendTLSPolicy metadata: name: tls-upstream-dev spec: targetRefs:

• kind: Service name: dev group: "" sectionName: "btls" validation: wellKnownCACertificates: "System" hostname: dev.example.com

More information on the configuration of TLS in Gateway API can be found in Gateway API - TLS Configuration.

Status information about the features that an implementation supports

Leads: Lior Lieberman, Beka Modebadze

GEP-2162: Supported features in GatewayClass Status

GatewayClass status has a new field, supportedFeatures. This addition allows implementations to declare the set of features they support. This provides a clear way for users and tools to understand the capabilities of a given GatewayClass.

This feature's name for conformance tests (and GatewayClass status reporting) is SupportedFeatures. Implementations must populate the supportedFeatures field in the .status of the GatewayClass before the GatewayClass is accepted, or in the same operation.

Here’s an example of a supportedFeatures published under GatewayClass' .status:

apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass ... status: conditions:

• lastTransitionTime: "2022-11-16T10:33:06Z" message: Handled by Foo controller observedGeneration: 1 reason: Accepted status: "True" type: Accepted supportedFeatures:
• HTTPRoute
• HTTPRouteHostRewrite
• HTTPRoutePortRedirect
• HTTPRouteQueryParamMatching

Graduation of SupportedFeatures to Standard, helped improve the conformance testing process for Gateway API. The conformance test suite will now automatically run tests based on the features populated in the GatewayClass' status. This creates a strong, verifiable link between an implementation's declared capabilities and the test results, making it easier for implementers to run the correct conformance tests and for users to trust the conformance reports.

This means when the SupportedFeatures field is populated in the GatewayClass status there will be no need for additional conformance tests flags like –suported-features, or –exempt or –all-features. It's important to note that Mesh features are an exception to this and can be tested for conformance by using Conformance Profiles, or by manually providing any combination of features related flags until the dedicated resource graduates from the experimental channel.

Named rules for Routes

GEP-995: Adding a new name field to all xRouteRule types (HTTPRouteRule, GRPCRouteRule, etc.)

Leads: Guilherme Cassolato

This enhancement enables route rules to be explicitly identified and referenced across the Gateway API ecosystem. Some of the key use cases include:

Status: Allowing status conditions to reference specific rules directly by name.

Observability: Making it easier to identify individual rules in logs, traces, and metrics.

Policies: Enabling policies (GEP-713) to target specific route rules via the sectionName field in their targetRef[s].

Tooling: Simplifying filtering and referencing of route rules in tools such as gwctl, kubectl, and general-purpose utilities like jq and yq.

Internal configuration mapping: Facilitating the generation of internal configurations that reference route rules by name within gateway and mesh implementations.

This follows the same well-established pattern already adopted for Gateway listeners, Service ports, Pods (and containers), and many other Kubernetes resources.

While the new name field is optional (so existing resources remain valid), its use is strongly encouraged. Implementations are not expected to assign a default value, but they may enforce constraints such as immutability.

Finally, keep in mind that the name format is validated, and other fields (such as sectionName) may impose additional, indirect constraints.

Experimental channel changes

Enabling external Auth for HTTPRoute

Giving Gateway API the ability to enforce authentication and maybe authorization as well at the Gateway or HTTPRoute level has been a highly requested feature for a long time. (See the GEP-1494 issue for some background.)

This Gateway API release adds an Experimental filter in HTTPRoute that tells the Gateway API implementation to call out to an external service to authenticate (and, optionally, authorize) requests.

This filter is based on the Envoy ext_authz API, and allows talking to an Auth service that uses either gRPC or HTTP for its protocol.

Both methods allow the configuration of what headers to forward to the Auth service, with the HTTP protocol allowing some extra information like a prefix path.

A HTTP example might look like this (noting that this example requires the Experimental channel to be installed and an implementation that supports External Auth to actually understand the config):

apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: require-auth namespace: default spec: parentRefs:

• name: your-gateway-here rules:
• matches:
• path: type: Prefix value: /admin filters:
• type: ExternalAuth externalAuth: protocol: HTTP backendRef: name: auth-service http: # These headers are always sent for the HTTP protocol, # but are included here for illustrative purposes allowedHeaders:
• Host
• Method
• Path
• Content-Length
• Authorization backendRefs:
• name: admin-backend port: 8080

This allows the backend Auth service to use the supplied headers to make a determination about the authentication for the request.

When a request is allowed, the external Auth service will respond with a 200 HTTP response code, and optionally extra headers to be included in the request that is forwarded to the backend. When the request is denied, the Auth service will respond with a 403 HTTP response.

Since the Authorization header is used in many authentication methods, this method can be used to do Basic, Oauth, JWT, and other common authentication and authorization methods.

Mesh resource

Lead(s): Flynn

GEP-3949: Mesh-wide configuration and supported features

Gateway API v1.4.0 introduces a new experimental Mesh resource, which provides a way to configure mesh-wide settings and discover the features supported by a given mesh implementation. This resource is analogous to the Gateway resource and will initially be mainly used for conformance testing, with plans to extend its use to off-cluster Gateways in the future.

The Mesh resource is cluster-scoped and, as an experimental feature, is named XMesh and resides in the gateway.networking.x-k8s.io API group. A key field is controllerName, which specifies the mesh implementation responsible for the resource. The resource's status stanza indicates whether the mesh implementation has accepted it and lists the features the mesh supports.

One of the goals of this GEP is to avoid making it more difficult for users to adopt a mesh. To simplify adoption, mesh implementations are expect

Week Ending November 02, 2025

https://lwkd.info/2025/20251106

Developer News

The 2025 Steering Committee Election results are announced. Congratulations to Kat Cosgrove, Paco Xu, Rita Zhang and Maciej Szulik for being elected for their 2 year term in the steering committee. Maciej and Paco are returning steering committee members. Thank you to all the candidates and all the community members for voting. An election retro call is happening on 19th November, 8AM PT. If you have any feedback about the steering elections this year, please add it to the retro doc.

WG-LTS is winding down after the conclusion from all the discussions that a community supported LTS within the Kubernetes project is probably not the right answer. The Compatibility Versions feature is cited as an alternative for safer upgrades.

The Kustomize project is seeking proposals for a new logo. If you have any ideas for a new logo, do post it in the open issue!

Release Schedule

Next Deadline: Code Freeze, 7th November

The code freeze and test freeze deadline is on Friday 7th November 2025, 12:00 UTC. Please open an early exception for your KEP if you think you need more time!

Kubernetes v1.35.0-alpha.3 is live.

Featured PRs

[KEP-4330] add min-compatibility-version to control plane

This PR is part of a larger effort to introduce “compatibility versions” to control plane components and features, eventually permitting upgrades and rollbacks that span more than one Kubernetes version safely. This PR adds the field to apiserver, controller-manager, and scheduler.

KEP of the Week

KEP-4827: Component Statusz

As part of Kubernetes march towards structured data for everything, this KEP introduces a structured, standardized endpoint for health and status checking. It will enhance observability and enable building new monitoring and performance tools. Statusz is kicking off with v1alpha1 in 1.35

Other Merges

New k8s-resource-fully-qualified-name format for Declarative Validation

Enhance several different E2E tests (plus many more, kudos Lukasz Szaszkiewicz) to support EnableWatchListClient

CRD Conditions include an ObservedGeneration to deter race conditions

DRA APIs: migrate several [DRA validations] (https://github.com/kubernetes/kubernetes/pull/134963), use EachKey to map resources, make DeviceAttribute a Union type,

New tests to support Deployments terminating pods during Recreate and RollingUpdate

Benchmarking Shared Informers now

Allow some kubeadm functions to be exported

Support Declarative Validation for StorageClass

Use informer.RunWithContext in controller tests

Test stepwise volume expansion

Prevent AllocationMode: All failure

Allow DRA to process inactive workloads with Allocatable=0

ContextualLogging migrations: cpumanager

JWKS fetch metrics for structured authentication

Pod Generation E2E tests promoted to conformance

Promotions

KUBECTL_COMMAND_HEADERS to GA

InPlacePodVerticalScaling to GA

StorageVersionMigration to beta

SystemWatchdog to GA

MutableCSINodeAllocatableCount to beta

DeploymentReplicaSetTerminatingReplicas to beta

Deprecated

BlockOwnerDeletion is removed from resource claims

Stop providing taint keys in Pod statuses when scheduling fails

DynamicResourceAllocation feature gate locked on; will be removed in a few releases

Remove kubelet --pod-infra-container-image switch

Subprojects and Dependency Updates

containerd v2.2.0-rc.0 (pre-release) introduces a mount manager, adds conf.d include support in the default configuration, and supports back references in the garbage collector. It improves CRI with ListPodSandboxMetrics and image-volume subpaths, adds parallel image unpack and referrers fetcher, updates the EROFS snapshotter, enables OpenTelemetry traces and WASM plugin support in NRI, improves shim reload performance, and postpones v2.2 deprecations to v2.3.

nerdctl v2.2.0-rc.0 fixes a namestore directory regression, adds mount-manager support, and introduces new checkpoint commands (create, ls, rm). It adds a --estargz-gzip-helper flag for image conversion and updates bundled dependencies, including containerd v2.2.0-rc.0, runc v1.3.2, BuildKit v0.25.1, and Stargz Snapshotter v0.18.0.

cloud-provider-vsphere v1.33.1 updates CAPI to v1.10.1 and CAPV to v1.13.0, enables weekly security checks, updates API calls to use FQDN, and fixes Service deletion when VirtualMachineService is not found. It also bumps Kubernetes to v1.33.5 and refreshes documentation.

cloud-provider-vsphere v1.32.3 provides dependency updates across test suites, upgrades controller-runtime to v0.19.6 and govmomi to v0.46.3, introduces weekly security checks, and fixes VirtualMachineService deletion. It also adopts FQDN for Supervisor API calls and includes CVE patches.

vsphere-cpi-chart-1.33.1 and vsphere-cpi-chart-1.32.3 update Helm charts for vSphere CPI to align with recent vSphere provider releases.

ingress-nginx helm-chart-4.14.0, 4.13.4, and 4.12.8 deliver updated Helm charts for the NGINX ingress controller with alignment to current controller and Kubernetes versions.

cluster-autoscaler v1.30.7 backports the OCI CloudProvider feature to the v1.30 line and publishes multi-architecture images (v1.30.7).

prometheus v3.7.3 fixes a UI redirect regression involving -web.external-url and -web.route-prefix, resolves federation issues for some native histograms, corrects promtool check config failures when --lint=none is specified, and eliminates a remote-write queue resharding deadlock.

Shoutouts

Sreeram Venkitesh - Shoutout to everyone who helped run the 2025 steering committee elections smoothly - The EOs and alternate EOs: @cblecker @Nina Polshakova @Arujjwal @Rey Lejano, K8s infra liaison @mahamed, and @jberkus for all the support from the very beginning and also for helping us with Elekto. Also big thanks to all the previous (and continuing) steering committee members for their support in making the election a smooth and successful one. Thank you all!

via Last Week in Kubernetes Development https://lwkd.info/

November 06, 2025 at 02:37AM

Graphs in your head, or how to assess a Kubernetes workload, with Oleksii Kolodiazhnyi

https://ku.bz/zDThxGQsP

Understanding what's actually happening inside a complex Kubernetes system is one of the biggest challenges architects face.

Oleksii Kolodiazhnyi, Senior Architect at Mirantis, shares his structured approach to Kubernetes workload assessment. He breaks down how to move from high-level business understanding to detailed technical analysis, using visualization tools and systematic documentation.

You will learn:

A top-down assessment methodology that starts with business cases and use cases before diving into technical details

Practical visualization techniques using tools like KubeView, K9s, and Helm dashboard to quickly understand resource interactions

Systematic resource discovery approaches for different scenarios, from well-documented Helm-based deployments to legacy applications with hard-coded configurations buried in containers

Documentation strategies for creating consumable artifacts that serve different audiences, from business stakeholders to new team members joining the project

Sponsor

This episode is sponsored by StormForge by CloudBolt — automatically rightsize your Kubernetes workloads with ML-powered optimization

More info

Find all the links and info for this episode here: https://ku.bz/zDThxGQsP

Interested in sponsoring an episode? Learn more.

via KubeFM https://kube.fm

November 04, 2025 at 05:00AM

Best AI Models for DevOps & SRE: Real-World Agent Testing

A comprehensive, data-driven comparison of 10 leading large language models (LLMs) from Google, Anthropic, OpenAI, xAI, DeepSeek, and Mistral, specifically tested for DevOps, SRE, and platform engineering workflows. Instead of relying on traditional benchmarks or marketing claims, this evaluation runs real agent workflows through production scenarios: Kubernetes operations, cluster analysis, policy generation, manifest creation, and systematic troubleshooting—all with actual timeout constraints. The results reveal shocking gaps between benchmark promises and production reality: 70% of models couldn't complete tasks in reasonable timeframes, premium "reasoning" models failed on tasks cheaper alternatives handled easily, and the most expensive model ($120 per million output tokens) failed more tests than it passed.

The evaluation measures five key dimensions: overall performance quality, reliability and completion rates, consistency across different tasks, cost-performance value, and context window efficiency. Five distinct test scenarios push models through endurance tests (100+ consecutive interactions), rapid pattern recognition (5-minute workflows), comprehensive policy compliance analysis, extreme context pressure (100,000+ token loads), and systematic investigation loops requiring intelligent troubleshooting. The rankings reveal clear performance tiers, with Claude Haiku emerging as the overall winner for its exceptional efficiency and price-performance ratio, while Claude Sonnet takes the reliability crown with 98% completion rates. The video provides specific recommendations on which models to use, which to avoid, and why cost doesn't always correlate with capability in production environments.

LLMComparison #DevOps #AIforEngineers

Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join

▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/ai/best-ai-models-for-devops--sre-real-world-agent-testing 🔗 DevOps AI Toolkit: https://github.com/vfarcic/dot-ai 🎬 Analysis report: https://github.com/vfarcic/dot-ai/blob/main/eval/analysis/platform/synthesis-report.md

▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 Large Language Models (LLMs) Compared 01:54 How I Compare Large Language Models 05:01 LLM Evaluation Criteria and Test Scenarios 13:23 AI Model Benchmark Results 27:34 AI Model Rankings and Recommendations

via YouTube https://www.youtube.com/watch?v=r84kQ5IMIQM

Week Ending October 26, 2025

https://lwkd.info/2025/20251031

Developer News

The steering committee election voting period closed last week. The results will be announced in the public steering meeting next Wednesday.

Some reminders for folks attending KubeCon NA 2025 about the Kubernetes Contributor Hour and the SIG/WG Meet and Greet

Release Schedule

Next Deadline: Code Freeze, 7th November

With the feature blog freeze in place, KEP assignees are expected to open placeholder PRs for their blogs. Please reach out to the Release Comms team for more information. We’re one week away from the v1.35 code freeze. Get your PRs ready and don’t forget to file an early exception if you anticipate any delays!

October patch releases have been skipped altogether.

KEP of the Week

KEP-5007: DRA: Device Binding Conditions

This KEP introduces BindingConditions, enabling the scheduler to delay Pod binding until external resources such as fabric-attached GPUs or FPGAs are confirmed ready. This improves scheduling reliability by preventing premature bindings that could lead to Pod failures or require manual intervention. The mechanism also supports asynchronous or failure-prone scenarios, including remote accelerators and FPGA reprogramming.

This KEP is tracked for beta in v1.35.

Other Merges

DRA resources use eachKey declarative validation to mirror map-key checks and keep generated DV in sync with handwritten rules

CSI NodePublishVolumeRequest now carries pod service account tokens in the gRPC secrets field instead of volume_context

DRA DeviceAttribute now declares its non-discriminated union with +k8s:unionMember, so declarative validation can enforce “exactly one value set”

Add +k8s:maxLength (and +k8s:optional) to NetworkDeviceData so generated DV can cap interfaceName / hardwareAddress lengths and match handwritten validation

Wire storage.k8s.io (StorageClass) into declarative validation and mark provisioner as +k8s:required, so generated DV now matches the old handwritten strategy on create/update

StorageVersionMigration (SVM) graduates to v1beta1 and drops the old v1alpha1/unused fields, so clusters must clean up any storage.k8s.io/v1alpha1 SVM objects before upgrading

kubectl finally drops support for the long-deprecated certificates.k8s.io/v1beta1 CertificateSigningRequest.

Add mtlsclient and mtlsserver for the mtls validations

apiserver cacher’s lister_watcher now exposes WatchList semantics

Enable declarative validation for resource.k8s.io ResourceSlice (v1/v1beta1/v1beta2)

Introduce pod queuing in endpoint/slice controllers

Add k8s-resource-fully-qualified-name format

Implements synthetic create authz permission check for exec, attach, and portforward

Enable Declarative Validation(DV) support for ClusterRole and RoleBinding

Replace HandleCrash and HandleError calls to use context-aware alternative

Bump supported etcd version to v3.5.24 for release v1.32, v1.33, and v1.3

Promotions

Pod Generation to GA

ContainerRestartRules to beta

RelaxedServiceNameValidation to beta

PreferSameTrafficDistribution to GA

Version Updates

etcd sdk to v3.6.5

system-validators to v1.12.1

Subprojects and Dependency Updates

containerd v2.2.0-rc.0 (pre-release) adds a mount manager, supports conf.d includes in the default config, and adds back-references in the garbage collector. It improves CRI with ListPodSandboxMetrics and image-volume subpaths, adds parallel image unpack and a referrers fetcher, updates EROFS snapshotter, enables OTEL traces and WASM plugin support in NRI, speeds shim reloads, and postpones some deprecations to 2.3.

containerd API v1.10.0-rc.0 (pre-release) aligns with containerd 2.2, introducing the mount manager and parallel unpack support in the API.

prometheus v3.7.3 fixes a UI redirect regression with -web.external-url and -web.route-prefix, corrects federation for some native histograms, fixes a promtool check config failure when --lint=none is set, and resolves a remote-write queue resharding deadlock.

via Last Week in Kubernetes Development https://lwkd.info/

October 31, 2025 at 02:41PM

Ep38 - Ask Me Anything About Anything with Scott Rosenberg

There are no restrictions in this AMA session. You can ask anything about DevOps, AI, Cloud, Kubernetes, Platform Engineering, containers, or anything else. Scott Rosenberg, a regular guest, will be here to help us out.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: Octopus 🔗 Enterprise Support for Argo: https://octopus.com/support/enterprise-argo-support ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

via YouTube https://www.youtube.com/watch?v=-nYVMVQosHc