Blog: Using Go workspaces in Kubernetes

https://www.kubernetes.dev/blog/2024/03/19/go-workspaces-in-kubernetes/

The Go programming language has played a huge role in the success of Kubernetes. As Kubernetes has grown, matured, and pushed the bounds of what “regular” projects do, the Go project team has also grown and evolved the language and tools. In recent releases, Go introduced a feature called “workspaces” which was aimed at making projects like Kubernetes easier to manage.

We’ve just completed a major effort to adopt workspaces in Kubernetes, and the results are great. Our codebase is simpler and less error-prone, and we’re no longer off on our own technology island.

GOPATH and Go modules

Kubernetes is one of the most visible open source projects written in Go. The earliest versions of Kubernetes, dating back to 2014, were built with Go 1.3. Today, 10 years later, Go is up to version 1.22 — and let’s just say that a whole lot has changed.

In 2014, Go development was entirely based on GOPATH. As a Go project, Kubernetes lived by the rules of GOPATH. In the buildup to Kubernetes 1.4 (mid 2016), we introduced a directory tree called staging. This allowed us to pretend to be multiple projects, but still exist within one git repository (which had advantages for development velocity). The magic of GOPATH allowed this to work.

Kubernetes depends on several code-generation tools which have to find, read, and write Go code packages. Unsurprisingly, those tools grew to rely on GOPATH. This all worked pretty well until Go introduced modules in Go 1.11 (mid 2018).

Modules were an answer to many issues around GOPATH. They gave more control to projects on how to track and manage dependencies, and were overall a great step forward. Kubernetes adopted them. However, modules had one major drawback — most Go tools could not work on multiple modules at once. This was a problem for our code-generation tools and scripts.

Thankfully, Go offered a way to temporarily disable modules (GO111MODULE to the rescue). We could get the dependency tracking benefits of modules, but the flexibility of GOPATH for our tools. We even wrote helper tools to create fake GOPATH trees and played tricks with symlinks in our vendor directory (which holds a snapshot of our external dependencies), and we made it all work.

And for the last 5 years it has worked pretty well. That is, it worked well unless you looked too closely at what was happening. Woe be upon you if you had the misfortune to work on one of the code-generation tools, or the build system, or the ever-expanding suite of bespoke shell scripts we use to glue everything together.

The problems

Like any large software project, we Kubernetes developers have all learned to deal with a certain amount of constant low-grade pain. Our custom staging mechanism let us bend the rules of Go; it was a little clunky, but when it worked (which was most of the time) it worked pretty well. When it failed, the errors were inscrutable and un-Googleable — nobody else was doing the silly things we were doing. Usually the fix was to re-run one or more of the update-* shell scripts in our aptly named hack directory.

As time went on we drifted farther and farher from “regular” Go projects. At the same time, Kubernetes got more and more popular. For many people, Kubernetes was their first experience with Go, and it wasn’t always a good experience.

Our eccentricities also impacted people who consumed some of our code, such as our client library and the code-generation tools (which turned out to be useful in the growing ecosystem of custom resources). The tools only worked if you stored your code in a particular GOPATH-compatible directory structure, even though GOPATH had been replaced by modules more than four years prior.

This state persisted because of the confluence of three factors:

Most of the time it only hurt a little (punctuated with short moments of more acute pain).

Kubernetes was still growing in popularity - we all had other, more urgent things to work on.

The fix was not obvious, and whatever we came up with was going to be both hard and tedious.

As a Kubernetes maintainer and long-timer, my fingerprints were all over the build system, the code-generation tools, and the hack scripts. While the pain of our mess may have been low on_average, I was one of the people who felt it regularly.

Enter workspaces

Along the way, the Go language team saw what we (and others) were doing and didn’t love it. They designed a new way of stitching multiple modules together into a new workspace concept. Once enrolled in a workspace, Go tools had enough information to work in any directory structure and across modules, without GOPATH or symlinks or other dirty tricks.

When I first saw this proposal I knew that this was the way out. This was how to break the logjam. If workspaces was the technical solution, then I would put in the work to make it happen.

The work

Adopting workspaces was deceptively easy. I very quickly had the codebase compiling and running tests with workspaces enabled. I set out to purge the repository of anything GOPATH related. That’s when I hit the first real bump - the code-generation tools.

We had about a dozen tools, totalling several thousand lines of code. All of them were built using an internal framework called gengo, which was built on Go’s own parsing libraries. There were two main problems:

Those parsing libraries didn’t understand modules or workspaces.

GOPATH allowed us to pretend that Go package paths and directories on disk were interchangeable in trivial ways. They are not.

Switching to a modules- and workspaces-aware parsing library was the first step. Then I had to make a long series of changes to each of the code-generation tools. Critically, I had to find a way to do it that was possible for some other person to review! I knew that I needed reviewers who could cover the breadth of changes and reviewers who could go into great depth on specific topics like gengo and Go’s module semantics. Looking at the history for the areas I was touching, I asked Joe Betz and Alex Zielenski (SIG API Machinery) to go deep on gengo and code-generation, Jordan Liggitt (SIG Architecture and all-around wizard) to cover Go modules and vendoring and the hack scripts, and Antonio Ojea (wearing his SIG Testing hat) to make sure the whole thing made sense. We agreed that a series of small commits would be easiest to review, even if the codebase might not actually work at each commit.

Sadly, these were not mechanical changes. I had to dig into each tool to figure out where they were processing disk paths versus where they were processing package names, and where those were being conflated. I made extensive use of the delve debugger, which I just can’t say enough good things about.

One unfortunate result of this work was that I had to break compatibility. The gengo library simply did not have enough information to process packages outside of GOPATH. After discussion with gengo and Kubernetes maintainers, we agreed to make gengo/v2. I also used this as an opportunity to clean up some of the gengo APIs and the tools’ CLIs to be more understandable and not conflate packages and directories. For example you can’t just string-join directory names and assume the result is a valid package name.

Once I had the code-generation tools converted, I shifted attention to the dozens of scripts in the hack directory. One by one I had to run them, debug, and fix failures. Some of them needed minor changes and some needed to be rewritten.

Along the way we hit some cases that Go did not support, like workspace vendoring. Kubernetes depends on vendoring to ensure that our dependencies are always available, even if their source code is removed from the internet (it has happened more than once!). After discussing with the Go team, and looking at possible workarounds, they decided the right path was to implement workspace vendoring.

The eventual Pull Request contained over 200 individual commits.

Results

Now that this work has been merged, what does this mean for Kubernetes users? Pretty much nothing. No features were added or changed. This work was not about fixing bugs (and hopefully none were introduced).

This work was mainly for the benefit of the Kubernetes project, to help and simplify the lives of the core maintainers. In fact, it would not be a lie to say that it was rather self-serving - my own life is a little bit better now.

This effort, while unusually large, is just a tiny fraction of the overall maintenance work that needs to be done. Like any large project, we have lots of “technical debt” — tools that made point-in-time assumptions and need revisiting, internal APIs whose organization doesn’t make sense, code which doesn’t follow conventions which didn’t exist at the time, and tests which aren’t as rigorous as they could be, just to throw out a few examples. This work is often called “grungy” or “dirty”, but in reality it’s just an indication that the project has grown and evolved. I love this stuff, but there’s far more than I can ever tackle on my own, which makes it an interesting way for people to get involved. As our unofficial motto goes: “chop wood and carry water”.

Kubernetes used to be a case-study of how not to do large-scale Go development, but now our codebase is simpler (and in some cases faster!) and more consistent. Things that previously seemed like they should work, but didn’t, now behave as expected.

Our project is now a little more “regular”. Not completely so, but we’re getting closer.

Thanks

This effort would not have been possible without tons of support.

First, thanks to the Go team for hearing our pain, taking feedback, and solving the problems for us.

Special mega-thanks goes to Michael Matloob, on the Go team at Google, who designed and implemented workspaces. He guided me every step of the way, and was very generous with his time, answering all m

Grand Finale (You Choose!, Ch. 3, Ep. 11)

We're at the end of the Security season of YouChoose! We'll go through all the tools we explored this season and do an ...

via YouTube https://www.youtube.com/watch?v=5UZnXarnicM

How to Propagate Secrets Everywhere with External Secrets Operator (ESO) and Crossplane

We dive into the powerful synergy between External Secrets Operator (ESO) and Crossplane to efficiently manage and propagate ...

via YouTube https://www.youtube.com/watch?v=CmkIMhO4AjY

“Multiple major fiber cables” reaching Africa are broken

Damage to multiple subsea internet cables off the African coast is disrupting internet services across the continent – with network providers invoking Force Majeure owing to several deepsea cable cuts and full repairs of the damaged cables estimated to take five weeks.

Tags:

via Pocket https://www.thestack.technology/fibre-cable-cut-africa-internet-outage/

March 18, 2024 at 06:13AM

Containers on a diet

Sentry – Launch week! New features and products all week long (so get comfy)! Tune in to Sentry’s YouTube and Discord daily at 9am PT to hear the latest scoop. Too busy? No problem - enter your email address to receive all the announcements (and win swag along the way).

Tags:

via Pocket https://changelog.com/shipit/95

March 18, 2024 at 03:11AM

Meet DBOS: A Database Alternative to Kubernetes

Turing Award laureate Dr. Mike Stonebrakerjust keeps on inventing databases. Forty years ago, it was the first relational system, Ingress, and thirty years ago, it was PostgreSQL, More recently, he co-created an in-memory transactional database system, VoltDB.

Tags:

via Pocket https://thenewstack.io/meet-dbos-a-database-alternative-to-kubernetes/

March 18, 2024 at 02:33AM

Measuring Developer Productivity via Humans

Measuring developer productivity is a difficult challenge. Conventional metrics focused on development cycle time and throughput are limited, and there aren't obvious answers for where else to turn.

Tags:

via Pocket https://martinfowler.com/articles/measuring-developer-productivity-humans.html

March 18, 2024 at 02:33AM

Week Ending March 10, 2024

https://lwkd.info/2024/20240315

Developer News

Kubernetes Contributor Summit EU is happening next Tuesday, March 19, 2024. Make sure to register by March 15. If you want to bring a family member to social send an email to summit-team@kubernetes.io. We’re eagerly looking forward to receiving your contributions to the unconference topics.

Also, don’t forget to help your SIG staff its table at the Kubernetes Meet and Greet on Kubecon Friday.

Take a peek at the upcoming Kubernetes v1.30 Release in this Blog.

Release Schedule

Next Deadline: Draft Doc PRs, Mar 12th

Kubernetes v1.30.0-beta.0 is live!

Your SIG should be working on any feature blogs, and discussing what “themes” to feature in the Release Notes.

Featured PR

123516: DRA: structured parameters

DRA, or Dynamic Resource Allocation, is a way to bridge new types of schedulable resources into Kubernetes. A common example of this is GPU accelerator cards but the system is built as generically as possible. Maybe you want to schedule based on cooling capacity, or cash register hardware, or nearby interns, it’s up to you. DRA launched as an alpha feature back in 1.26 but came with some hard limitations. Notably the bespoke logic for simulating scale ups and scale downs in cluster-autoscaler had no way to understand how those would interact with these opaque resources. This PR pulls back the veil a tiny bit, keeping things generic but allowing more forms of structured interaction so core tools like the scheduler and autoscalers can understand dynamic resources.

This happens from a few directions. First, on the node itself a DRA driver plugin provides information about what is available locally, which the kubelet publishes as a NodeResourceSlice object. In parallel, an operator component from the DRA implementation creates ResourceClaimParameters as needed to describe a particular resource claim. The claim parameters include CEL selector expressions for each piece of the claim, allowing anything which can evaluate CEL to check them independently of the DRA plugin. These two new objects combine with the existing ResourceClaim object to allow bidirectional communication between Kubernetes components and the DRA plugin without either side needing to wait for the other in most operations.

While this does increase the implementation complexity of a new DRA provider, it also dramatically expands their capabilities. New resources can be managed with effectively zero overhead and without the even greater complexity of custom schedulers or a plugin-driven autoscaler.

KEP of the Week

KEP-647: APIServer Tracing

This KEP proposes to update the kube-apiserver to allow tracing requests. This is proposed to be done with OpenTelemetry libraries and the data will be exported in the OpenTelemetry format. The kube-apiserver currently uses kubernetes/utils/trace for tracing, but we can make use of distributed tracing to improve ease of use and to make analysis of the data easier. The proposed implementation involves wrapping the API Server’s http server and http clients with otelhttp.

This KEP is tracked to graduate to stable in the upcoming v1.30 release.

Other Merges

podLogsDir key in kubelet configuration to configure default location of pod logs.

New custom flag to kubectl debug for adding custom debug profiles.

PodResources API now has initContainers with containerRestartPolicy of Always when SidecarContainers are enabled.

Fix to the disruption controller’s PDB status sync to maintain PDB conditions during an update.

Service NodePort can now be set to 0 if AllocateLoadBalancerNodePorts is false.

Field selector for services that allows filtering by clusterIP field.

The ‘.memorySwap.swapBehaviour’ field in kubelet configuration gets NoSwap as the default value.

kubectl get jobs now prints the status of the listed jobs.

Bugfix for initContainer with containerRestartPolicy Always where it couldn’t update its Pod state from terminated to non-terminated.

The StorageVersionMigration API, which was previously available as a CRD, is now a built-in API.

InitContainer’s image location will now be considered in scheduling when prioritizing nodes.

Almost all printable ASCII characters are now allowed in environment variables.

Added support for configuring multiple JWT authenticators in Structured Authentication Configuration.

New trafficDistribution field added to the Service spec which allows configuring how traffic is distributed to the endpoints of a Service.

JWT authenticator config set via the –authentication-config flag is now dynamically reloaded as the file changes on disk.

Promotions

StructuredAuthorizationConfiguration to beta.

HPAContainerMetrics to GA.

MinDomainsInPodTopologySpread to GA.

ValidatingAdmissionPolicy to GA.

StructuredAuthenticationConfiguration to beta.

KubeletConfigDropInDir to beta.

Version Updates

google.golang.org/protobuf updated to v1.33.0 to resolve CVE-2024-24786.

Subprojects and Dependency Updates

gRPC to v1.62.1 fix a bug that results in no matching virtual host found RPC errors

cloud-provider-openstack to v1.28.2 Implement imagePullSecret support for release 1.28

via Last Week in Kubernetes Development https://lwkd.info/

March 14, 2024 at 11:53PM