The following is an unroll of a tweet thread about Crossplane packages that I posted after reading Dave Anderson’s post “A better Kubernetes, from the ground up”. I found many of the points in the post compelling, but feel that the existing extensibility model of Kubernetes allows for implementing at least some of the desired features without tearing down the entire system, as evidenced by the implementation of Crossplane packages. It is my hope that some of the design decisions we have made with Crossplane can be generally abstracted and improved upon to continue to enhance the Kubernetes ecosystem.
A big thank you to folks like Dave and Vallery Lancey for challenging existing models and driving change, and an equally large thanks to many of my wonderful colleagues who work tirelessly to maintain Kubernetes and make it what it is today.
Interesting post from @dave_universetf. Lots of parallels to @crossplane_io packages in the "version controlled pods" and "explicit ownership" sections. More info in the docs, but I'll give an overview here.
— Daniel Mangum (@hasheddan) November 29, 2020
Post: https://t.co/gG918TFNoI
Docs: https://t.co/WqGuBUJABa
(1/n)
First, what are @crossplane_io packages? They are single layer OCI images that contain manifests for installing CRDs, XRDs, and @kubernetesio controllers to reconcile them. K8s controllers have become ubiquitous, and are typically installed using a tool like @HelmPack. (2/n)
— Daniel Mangum (@hasheddan) November 29, 2020
So why reinvent the wheel? Well, first of all, a single @crossplane_io cluster could have quite a few of these controllers, each bringing potentially hundreds of CRDs. The core Crossplane controller needs to have RBAC on all of these types so they can be composed. (3/n)
— Daniel Mangum (@hasheddan) November 29, 2020
Composition is the process of grouping together multiple "primitive" resources into a higher level abstraction. Think of an EKS cluster + Node Group + VPC + Subnets satisfying a "Cluster" abstraction. Kind of like OOP for infrastructure. (4/n)
— Daniel Mangum (@hasheddan) November 29, 2020
Composition machinery in core Crossplane needs to be able to create these primitives when it observes an instance of the abstraction they satisfy. In addition, the controller installed with the CRDs needs permissions to manage them. (5/n)
— Daniel Mangum (@hasheddan) November 29, 2020
The Crossplane RBAC manager runs as an isolated Deployment responsible for doling out these permissions, meaning it must be given RBAC privilege escalation powers. (6/n)
— Daniel Mangum (@hasheddan) November 29, 2020
However, isolating the RBAC manager means that we can constrain the logic that runs with escalation privileges, and a user can also opt not to deploy the RBAC manager and just manage permissions on their own. (7/n)
— Daniel Mangum (@hasheddan) November 29, 2020
So now that we can guarantee that an installed package will be given permissions to manage the types that it installs, the next issue is versioning. (8/n)
— Daniel Mangum (@hasheddan) November 29, 2020
Crossplane uses many of the ideas introduced in @kubernetesio Deployment revisions in our design of package revisions. Installing a Package will create a PackageRevision for the specified image. (9/n)
— Daniel Mangum (@hasheddan) November 29, 2020
Based on the package contents and the directives in the crossplane.yaml file, Crossplane will unpack and install the revision's contents. For instance, provider-aws will result in a slew of new CRDs and a Deployment that reconciles them. (10/n)
— Daniel Mangum (@hasheddan) November 29, 2020
When a new version of your package is available, you simply bump the tag, and a new revision is created. A revision can either be Active or Inactive, the former meaning it controls its installed objects, and the latter meaning it doesn't (more on this later). (11/n)
— Daniel Mangum (@hasheddan) November 29, 2020
When a new Active revision is introduced, the existing revision is transitioned to Inactive, meaning that its controllers are stopped, but the objects it installed remain. The new revision then attempts to install its own objects, or observe that they exist. (12/n)
— Daniel Mangum (@hasheddan) November 29, 2020
If all goes well, the new controllers are now reconciling the objects, and the old revision sits idly by. Users may also choose to have a manual activation policy, meaning Crossplane will create a new revision, but you must explicitly say to Activate it. (13/n)
— Daniel Mangum (@hasheddan) November 29, 2020
Over time, depending on your revision history limit, inactive revisions will be garbage collected, which means that if subsequent revisions did not install a given CRD, it will also be cleaned up with its parent revision. (14/n)
— Daniel Mangum (@hasheddan) November 29, 2020
So how does this ownership work? Primarily using built-in @kubernetesio controller and owner references. In order for a revision to reconcile a CRD, it must be the controller of it, and k8s mandates there can only be one. (15/n)
— Daniel Mangum (@hasheddan) November 29, 2020
When a revision becomes active, it first attempts to become the controller of every object kind it will reconcile (remember, for it to reconcile an object it needs permissions, which are assigned in a deterministic manner). (16/n)
— Daniel Mangum (@hasheddan) November 29, 2020
If all of the revision's object kinds do not exist, or do and it can become the controller, then it is free to begin reconciling those types. But if older revisions exist, how can the newer one be the sole controller? (17/n)
— Daniel Mangum (@hasheddan) November 29, 2020
When a revision is transitioned to Inactive, it changes all of the controller references it established to owner references, which a single object can have many of. When it finishes this process, the new revision is then free to become the controller. (18/n)
— Daniel Mangum (@hasheddan) November 29, 2020
This is also how we guard against two distinct packages attempting to install the same types and reconcile them. If the GVK is owned by a revision, no other revision, whether an iteration of the same package or an entirely different one can reconcile it. (19/n)
— Daniel Mangum (@hasheddan) November 29, 2020
The same process works when rolling backward to a revision that is Inactive. You can even have Crossplane periodically check for new versions of a package, and create new revisions when it observes a newer version. This can all be configured. (20/n)https://t.co/0ZkXREDQOq
— Daniel Mangum (@hasheddan) November 29, 2020
Lastly, I am excited to say that Crossplane v1.0, which is due to land in ~2 weeks. Will also support dependency resolution between packages! Here is a small demo of what this looks like. (21/21)https://t.co/slA3Nfv970
— Daniel Mangum (@hasheddan) November 29, 2020