Graphical User Interface - Feat. Backstage and Port (You Choose!, Ch. 05, Ep. 04)
Graphical User Interface - Choose Your Own Adventure: The Dignified Pursuit of a Developer Platform
In this episode, we'll go through UIs typically used as a portal into the platform, etc. The tools we'll explore and compare are Backstage and Port.
Vote for your choice of a tool for signing artifacts at https://cloud-native.slack.com/archives/C05M2NFNVRN. If you have not already joined CNCF Slack, you can do so from https://slack.cncf.io.
This and all other episodes are available at https://www.youtube.com/playlist?list=PLyicRj904Z9-FzCPvGpVHgRQVYJpVmx3Z.
More information about the "Choose Your Own Adventure" project including the source code and links to all the videos can be found at https://github.com/vfarcic/cncf-demo.
٩( ᐛ )و Whitney's YouTube Channel → https://www.youtube.com/@wiggitywhitney
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ 🔗 CNCF Slack invite (if you’re not already there): https://communityinviter.com/apps/cloud-native/cncf 🔗 Link to #you-choose channel in CNCF Slack: https://bit.ly/3NV7nHW 🔗 Graphical User Interface (GUI): https://github.com/vfarcic/cncf-demo/blob/main/manuscript/gui/README.md
via YouTube https://www.youtube.com/watch?v=YTfKt3pkx10
The Cloud Controller Manager Chicken and Egg Problem
https://kubernetes.io/blog/2025/02/14/cloud-controller-manager-chicken-egg-problem/
Kubernetes 1.31 completed the largest migration in Kubernetes history, removing the in-tree cloud provider. While the component migration is now done, this leaves some additional complexity for users and installer projects (for example, kOps or Cluster API) . We will go over those additional steps and failure points and make recommendations for cluster owners. This migration was complex and some logic had to be extracted from the core components, building four new subsystems.
Cloud controller manager (KEP-2392)
API server network proxy (KEP-1281)
kubelet credential provider plugins (KEP-2133)
Storage migration to use CSI (KEP-625)
The cloud controller manager is part of the control plane. It is a critical component that replaces some functionality that existed previously in the kube-controller-manager and the kubelet.
One of the most critical functionalities of the cloud controller manager is the node controller, which is responsible for the initialization of the nodes.
As you can see in the following diagram, when the kubelet starts, it registers the Node object with the apiserver, Tainting the node so it can be processed first by the cloud-controller-manager. The initial Node is missing the cloud-provider specific information, like the Node Addresses and the Labels with the cloud provider specific information like the Node, Region and Instance type information.
sequenceDiagram autonumber rect rgb(191, 223, 255) Kubelet->>+Kube-apiserver: Create Node Note over Kubelet: Taint: node.cloudprovider.kubernetes.io Kube-apiserver->>-Kubelet: Node Created end Note over Kube-apiserver: Node is Not Ready Tainted, Missing Node Addresses*, ... Note over Kube-apiserver: Send Updates rect rgb(200, 150, 255) Kube-apiserver->>+Cloud-controller-manager: Watch: New Node Created Note over Cloud-controller-manager: Initialize Node: Cloud Provider Labels, Node Addresses, ... Cloud-controller-manager->>-Kube-apiserver: Update Node end Note over Kube-apiserver: Node is Ready
This new initialization process adds some latency to the node readiness. Previously, the kubelet was able to initialize the node at the same time it created the node. Since the logic has moved to the cloud-controller-manager, this can cause a chicken and egg problem during the cluster bootstrapping for those Kubernetes architectures that do not deploy the controller manager as the other components of the control plane, commonly as static pods, standalone binaries or daemonsets/deployments with tolerations to the taints and using hostNetwork (more on this below)
Examples of the dependency problem
As noted above, it is possible during bootstrapping for the cloud-controller-manager to be unschedulable and as such the cluster will not initialize properly. The following are a few concrete examples of how this problem can be expressed and the root causes for why they might occur.
These examples assume you are running your cloud-controller-manager using a Kubernetes resource (e.g. Deployment, DaemonSet, or similar) to control its lifecycle. Because these methods rely on Kubernetes to schedule the cloud-controller-manager, care must be taken to ensure it will schedule properly.
Example: Cloud controller manager not scheduling due to uninitialized taint
As noted in the Kubernetes documentation, when the kubelet is started with the command line flag --cloud-provider=external, its corresponding Node object will have a no schedule taint named node.cloudprovider.kubernetes.io/uninitialized added. Because the cloud-controller-manager is responsible for removing the no schedule taint, this can create a situation where a cloud-controller-manager that is being managed by a Kubernetes resource, such as a Deployment or DaemonSet, may not be able to schedule.
If the cloud-controller-manager is not able to be scheduled during the initialization of the control plane, then the resulting Node objects will all have the node.cloudprovider.kubernetes.io/uninitialized no schedule taint. It also means that this taint will not be removed as the cloud-controller-manager is responsible for its removal. If the no schedule taint is not removed, then critical workloads, such as the container network interface controllers, will not be able to schedule, and the cluster will be left in an unhealthy state.
Example: Cloud controller manager not scheduling due to not-ready taint
The next example would be possible in situations where the container network interface (CNI) is waiting for IP address information from the cloud-controller-manager (CCM), and the CCM has not tolerated the taint which would be removed by the CNI.
The Kubernetes documentation describes the node.kubernetes.io/not-ready taint as follows:
"The Node controller detects whether a Node is ready by monitoring its health and adds or removes this taint accordingly."
One of the conditions that can lead to a Node resource having this taint is when the container network has not yet been initialized on that node. As the cloud-controller-manager is responsible for adding the IP addresses to a Node resource, and the IP addresses are needed by the container network controllers to properly configure the container network, it is possible in some circumstances for a node to become stuck as not ready and uninitialized permanently.
This situation occurs for a similar reason as the first example, although in this case, the node.kubernetes.io/not-ready taint is used with the no execute effect and thus will cause the cloud-controller-manager not to run on the node with the taint. If the cloud-controller-manager is not able to execute, then it will not initialize the node. It will cascade into the container network controllers not being able to run properly, and the node will end up carrying both the node.cloudprovider.kubernetes.io/uninitialized and node.kubernetes.io/not-ready taints, leaving the cluster in an unhealthy state.
Our Recommendations
There is no one “correct way” to run a cloud-controller-manager. The details will depend on the specific needs of the cluster administrators and users. When planning your clusters and the lifecycle of the cloud-controller-managers please consider the following guidance:
For cloud-controller-managers running in the same cluster, they are managing.
Use host network mode, rather than the pod network: in most cases, a cloud controller manager will need to communicate with an API service endpoint associated with the infrastructure. Setting “hostNetwork” to true will ensure that the cloud controller is using the host networking instead of the container network and, as such, will have the same network access as the host operating system. It will also remove the dependency on the networking plugin. This will ensure that the cloud controller has access to the infrastructure endpoint (always check your networking configuration against your infrastructure provider’s instructions).
Use a scalable resource type. Deployments and DaemonSets are useful for controlling the lifecycle of a cloud controller. They allow easy access to running multiple copies for redundancy as well as using the Kubernetes scheduling to ensure proper placement in the cluster. When using these primitives to control the lifecycle of your cloud controllers and running multiple replicas, you must remember to enable leader election, or else your controllers will collide with each other which could lead to nodes not being initialized in the cluster.
Target the controller manager containers to the control plane. There might exist other controllers which need to run outside the control plane (for example, Azure’s node manager controller). Still, the controller managers themselves should be deployed to the control plane. Use a node selector or affinity stanza to direct the scheduling of cloud controllers to the control plane to ensure that they are running in a protected space. Cloud controllers are vital to adding and removing nodes to a cluster as they form a link between Kubernetes and the physical infrastructure. Running them on the control plane will help to ensure that they run with a similar priority as other core cluster controllers and that they have some separation from non-privileged user workloads.
It is worth noting that an anti-affinity stanza to prevent cloud controllers from running on the same host is also very useful to ensure that a single node failure will not degrade the cloud controller performance.
Ensure that the tolerations allow operation. Use tolerations on the manifest for the cloud controller container to ensure that it will schedule to the correct nodes and that it can run in situations where a node is initializing. This means that cloud controllers should tolerate the node.cloudprovider.kubernetes.io/uninitialized taint, and it should also tolerate any taints associated with the control plane (for example, node-role.kubernetes.io/control-plane or node-role.kubernetes.io/master). It can also be useful to tolerate the node.kubernetes.io/not-ready taint to ensure that the cloud controller can run even when the node is not yet available for health monitoring.
For cloud-controller-managers that will not be running on the cluster they manage (for example, in a hosted control plane on a separate cluster), then the rules are much more constrained by the dependencies of the environment of the cluster running the cloud-controller-manager. The advice for running on a self-managed cluster may not be appropriate as the types of conflicts and network constraints will be different. Please consult the architecture and requirements of your topology for these scenarios.
Example
This is an example of a Kubernetes Deployment highlighting the guidance shown above. It is important to note that this is for demonstration purposes only, for production uses please consult your clou
Week Ending February 09, 2025
https://lwkd.info/2025/20250212
Developer News
SIG-Etcd is recruiting for another Mentorship Cohort. If you’ve been interested in working on Etcd, or leveling up your skills to become an approver, please apply to be a mentee.
SIG Leads, quick reminder to check your old alpha/beta features to see if they can progress in 1.33, or should maybe be deprecated.
Arty folks, want to design the contributor tshirt for London? Submit your designs by February 18th.
Release Schedule
Next Deadline: Enhancements Freeze, February 13
Enhancements freeze is right around the corner, scheduled for 02:00 UTC Friday 14th February 2025 / 18:00 PST Thursday 13th February 2025. If your KEPs have been opted-in for the release make sure that you’ve fulfilled all the requirements in order to make the cut for enhancements freeze. If you’re expecting the need for an exception, you can talk with your SIG leads and apply for an early exception request.
1.33 alpha 1 is now available.
February patch releases are delayed 1 week to incorporate a golang update.
Featured PRs
Sidecar containers feature is finally promoted to GA in v1.33. The SidecarContainers feature gate has been locked to default value and will be removed in v1.36. The original KEP for sidecar containers was first introduced in v1.28. The KEP proposed built-in support for sidecar containers in Kubernetes with the sidecar containers running as restartable initContainers.
This PR implements resize for sidecar contianers, with sidecar containers now promoted to GA. Resize of sidecar containers now works the same as resize of regular containers. This PR adds the capability to resize sidecar containers, which are essentially restartable init containers, when in-place pod resizing is enabled. Resize of non-restartable init containers is still not allowed.
Other Merges
Fix for memory leak in kube-proxy occurring in clusters with high volume of UDP workflows
kube-apiserver featuregate ServiceAccountNodeAudienceRestriction to be enabled by default in v1.33
Fix for regression with ServiceAccountNodeAudienceRestriction for azureFile volumes errors
Scheduler queuesort plugins integration tests added
Fix for DRA allocation All mode when there are no devices available for allocation
statusz endpoint added for kube-controller-manager
statusz endpoint added for kube-proxy
Promotions
ControlPlaneKubeletLocalMode to beta
WaitForAllControlPlaneComponents to beta
Deprecated
WatchFromStorageWithoutResourceVersion deprecated
SeparateCacheWatchRPC deprecated
Version Updates
Update publishing-bot rules to use go1.22.10 for active release branches
Kubernetes is now built with Go 1.23.5
via Last Week in Kubernetes Development https://lwkd.info/
February 12, 2025 at 02:00PM
Ep11 - Ask Me Anything About DevOps, Cloud, Kubernetes, Platform Engineering,... w/Scott Rosenberg
There are no restrictions in this AMA session. You can ask anything about DevOps, Cloud, Kubernetes, Platform Engineering, containers, or anything else. We'll have a special guest Scott Rosenberg to help us out.
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox
via YouTube https://www.youtube.com/watch?v=3wekCMdzHmA
From Zero to Fully Operational Developer Platform in 5 Steps!
Learn how to build a fully operational Internal Developer Platform (IDP) in just 5 steps! Discover the principles, components, and tools needed to create your own developer platform. Follow along as we cover APIs, state management, one-shot actions (workflows), RBAC & policies, and optional user interfaces like custom CLIs and GitOps. By the end, you'll have a comprehensive understanding of how to assemble an IDP that fits your needs using open-source solutions. Watch now to master developer platforms and streamline your application deployment and observation processes!
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: KodeKloud 🔗 Free courses (Feb 10 to 17) - https://kode.wiki/4hNCku4 🔗 KodeKloud -https://kodekloud.com ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/internal-developer-platforms/from-zero-to-fully-operational-developer-platform-in-5-steps 🎬 Crossplane (tutorial): https://youtube.com/playlist?list=PLyicRj904Z99i8U5JaNW5X3AyBvfQz-16 🎬 Cloud-Native Apps With Open Application Model (OAM) And KubeVela: https://youtu.be/2CBu6sOTtwk 🎬 Kubernetes-Native Policy Management With Kyverno: https://youtu.be/DREjzfTzNpA 🎬 How to apply policies in Kubernetes using Open Policy Agent (OPA) and Gatekeeper: https://youtu.be/14lGc7xMAe4 🎬 Govern Kubernetes with Kubewarden and WASM Admission Controllers: https://youtu.be/KbKQu3AqhBY 🎬 Is This the End of Crossplane? Compose Kubernetes Resources with kro: https://youtu.be/8zQtpcxmdhs 🎬 What Is... GitOps Tools?: https://youtube.com/shorts/z9nUgMOwaDg 🎬 Argo CD - Applying GitOps Principles To Manage A Production Environment In Kubernetes: https://youtu.be/vpWQeoaiRM4 🎬 Flux CD v2 With GitOps Toolkit - Kubernetes Deployment And Sync Mechanism: https://youtu.be/R6OeIgb7lUI 🎬 Rancher Fleet: GitOps Across A Large Number Of Kubernetes Clusters: https://youtu.be/rIH_2CUXmwM 🎬 Mastering Developer Portals: Discover & Integrate API Schemas with Port: https://youtu.be/PV1sBiC85Yc 🎬 From UX to API: Mastering Platform Validations with Kubernetes Validating Admission Policies: https://youtu.be/8jtYN-qvSqA 🎬 Full Application Setup in Internal Developer Platform (IDP) with Crossplane: https://youtu.be/WpgiVlODt4I 🎬 How to Create Custom CLIs for Internal Developer Platforms with Nushell: https://youtu.be/TgQZz2kGysk 🎬 Getting Started with Backstage: From Zero to Operational Dev Portal: https://youtu.be/A-3Ai--Z-Gs
▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 Full Internal Developer Platform 03:16 KodeKloud (sponsor) 04:39 Components of a Platform 12:17 Step 1: APIs 16:17 Step 2: State management 21:54 Step 3: One-Shot Actions (Workflows) 25:56 Step 4: RBAC & Policies 28:57 Step 5: Custom User Interface for Operations (Optional) 30:04 Step 5-1: CLI UI 31:16 Step 5-2: GitOps UI 32:53 Step 5-3: Web UI 35:26 The Question?
via YouTube https://www.youtube.com/watch?v=fZ2DjdqT1e0
1_r/devopsish
2_News
3_Music
