From 542b7c7c4cd606dbf1029ca977c9ed6af6c09c01 Mon Sep 17 00:00:00 2001 From: natasha41575 Date: Thu, 4 Nov 2021 12:52:23 -0700 Subject: [PATCH] proposal for transformer annotations option --- proposals/21-11-transformer-annotations.md | 469 +++++++++++++++++++++ 1 file changed, 469 insertions(+) create mode 100644 proposals/21-11-transformer-annotations.md diff --git a/proposals/21-11-transformer-annotations.md b/proposals/21-11-transformer-annotations.md new file mode 100644 index 000000000..ddf14a4ad --- /dev/null +++ b/proposals/21-11-transformer-annotations.md @@ -0,0 +1,469 @@ + + +# Option for origin and transformer annotations + +**Authors**: +- natasha41575 + +**Reviewers**: +- monopole +- KnVerey +- yuwenma + +**Status**: implementable + +## Summary + +This proposal is to extend the `buildMetadata.originAnnotations` option in the kustomization to annotate +generated resources with information about the source of their generators, and to add a new option +`buildMetadata.transformerAnnotations` that will add annotations to each resource with information about +all the transformers that have processed it. These options will help users understand how their output was +created and debug it accordingly. + +## Motivation + +When a user is managing a large number of resources with kustomize, the output of `kustomize build` will be a +long stream of yaml resources, and it can be extremely tedious for the user to understand how each resource +was created and processed. However, we can provide the user with an option to annotate each resource with +origin and transformation data. This information will help kustomize users +debug their large kustomize configurations. Because these annotations can be used for debugging purposes, we +would like them to be verbose and communicate as much information as possible to the user. + +Kustomize previously had a [feature request](https://github.com/kubernetes-sigs/kustomize/issues/3979) to add an +option to kustomize build to retain the origin of a resource in an annotation, and a final design was settled +on after some discussion. It was implemented by [this pull request](https://github.com/kubernetes-sigs/kustomize/pull/4065) +and released in kustomize v4.3.0. This feature added a new field to the kustomization file, `buildMetadata`, an +array of boolean options such as `originAnnotations`. If `originAnnotations` is set, `kustomize build` will add annotations +to each resource describing the resource's origin. + +While the above feature gives us valuable information about where many resources originated from, there are two +pieces missing: + +- Origin annotations for generated resources. Resources can either be generated using a builtin generator such as +`ConfigMapGenerator` or `SecretGenerator`, or can be generated from a custom generator invoked from either the `generators` +or `transformers` field. + +- Everything in kustomize is a transformer, so we would like to have annotations about which transformers, builtin or +custom, touched each resource. + +For a user attempting to debug a large stream of output from `kustomize build`, knowing how each resource has been +transformed through various layers will make it much easier to understand how the output was rendered, easing debugging +and making changes to kustomization files. + +### Builtin Transformers + +Builtin transformers include those that are invoked through various kustomization +fields. Some of these transformers are: + - AnnotationsTransformer + - HashTransformer + - ImageTagTransformer + - LabelTransformer + - LegacyOrderTransformer + - NamespaceTransformer + - PatchJson6902Transformer + - PatchStrategicMergeTransformer + - PatchTransformer + - PrefixSuffixTransformer + - ReplacementTransformer + - ReplicaCountTransformer + - ValueAddTransformer + +Custom transformers are invoked via the `transformers` field in the kustomization file. Although KRM-style plugins can be +invoked via the `generators` and `validators` fields, it is impossible for plugins in these fields to transform existing resources, +as `generators` do not take input and `validators` are not permitted to change input. Therefore, all custom transformers will +appear in the `transformers` field. + +It is, however, possible to put a generator or validator in the `transformers` field, so we should try to differentiate +the plugins in the `transformers` field where we can, and only store data about actual transformers. + +**Goals:** +1. When `originAnnotations` is set, add annotations to generated resources that describe the source of the generator that +created them. +2. Add a new option, `transformerAnnotations`, to `buildMetadata` that, when set, will annotate each resource with +information about which transformers, builtin or custom, touched each resource. + +**Non-goals:** +1. Change the syntax of the existing `originAnnotations` option. +2. Add these annotations to the output of kustomize by default. + +## Proposal + +### Origin Annotation +To retain the information about the origin of a resource, the user can specify the `originAnnotations` +option in the `buildMetadata` field of the kustomization: + +```yaml +buildMetadata: [originAnnotations] +``` + +When this option is set, generated resources should receive an annotation with key `config.kubernetes.io/origin`, +containing data about the generator that produced it. If the resource is from the `resources` field, this annotation +contains data about the file it originated from. + +The possible fields of these annotations are: + +```yaml +config.kubernetes.io/origin: | + path: path.yaml # The path to the resource file itself + ref: v0.0.0 # If from a remote file or generator, the ref of the repo URL + repo: http://github.com/examplerepo # If from a remote file or generator, the repo source + configuredIn: path/to/generatorconfig # If a generated resource, the path to the generator config + configuredBy: # If a generated resource, the ObjectReference of the generator config + kind: Generator + apiVersion: builtin + name: foo + namespace: default +``` + +All local file paths are relative to the top-level kustomization, i.e. the kustomization file in the directory upon +which `kustomize build` was invoked. For example, if someone were to run `kustomize build foo`, all file paths +in the annotation output would be relative to`foo/kustomization.yaml`. + +All remote file paths are relative to the root of the remote repository. + +Any fields that are not applicable would be omitted from the final output. If a generator is invoked via +a field in the kustomization file, `configuredIn` would point to the kustomization file itself. + +#### Examples + +##### Resource declared from `resources` +A kustomization such as the following: + +```yaml +resources: +- deployment.yaml + +buildMetadata: [originAnnotations] +``` + +would produce a resource with an annotation like the following: + +```yaml +config.kubernetes.io/origin: | + path: deployment.yaml +``` + +##### Local custom generator +A kustomization such as the following: + +```yaml +generators: +- generator.yaml + +buildMetadata: [originAnnotations] +``` + +would produce a resource with an annotation like the following: + +```yaml +config.kubernetes.io/origin: | + configuredIn: generator.yaml + configuredBy: + kind: MyGenerator + apiVersion: v1 + name: generator +``` + +##### Remote builtin generator +We have a top-level kustomization such as the following: + +```yaml +resources: +- github.com/examplerepo/?ref=v1.0.6 + +buildMetadata: [originAnnotations] +``` + +which uses `github.com/examplerepo/?ref=v1.0.6` as a remote base. This remote base has the following kustomization +defined in `github.com/examplerepo/kustomization.yaml`: + +```yaml +configMapGenerator: +- name: my-java-server-env-vars + literals: + - JAVA_HOME=/opt/java/jdk + - JAVA_TOOL_OPTIONS=-agentlib:hprof +``` + +Running `kustomize build` on the top-level kustomization would produce the following output: + +```yaml +apiVersion: v1 +data: + JAVA_HOME: /opt/java/jdk + JAVA_TOOL_OPTIONS: -agentlib:hprof +kind: ConfigMap +metadata: + name: my-java-server-env-vars-44k658k8gk + annotations: + config.kubernetes.io/origin: | + ref: v1.0.6 + repo: github.com/examplerepo + configuredIn: kustomization.yaml + configuredBy: + kind: ConfigMapGenerator + apiVersion: builtin +``` + +### Transformer Annotations + +To retain the information about what transformers have acted on each resource, we can propose a new option +`transformerAnnotations` in the `buildMetadata` field of the kustomization: + +```yaml +buildMetadata: [transformerAnnotations] +``` + +It is possible for the user to set both `transformerAnnotations` and `originAnnotations`: + +```yaml +buildMetadata: [originAnnotations, transformerAnnotations] +``` + +When the `transformerAnnotations` option is set, kustomize will add annotations with information about what transformers +have acted on each resource. Transformers can be invoked either through various fields in the kustomization file +(e.g. the `replacements` field will invoke the ReplacementTransformer), or through the `transformers` field. + +The annotation key for transformer annotatinos will be `config.kubernetes.io/transformations`, which will contain a list of +transformer data: + +```yaml +config.kubernetes.io/transformations: | +- ref: v0.0.0 # If from a remote transformer, the ref of the repo URL + repo: http://github.com/examplerepo # If from a remote transformer, the repo source + configuredIn: path/to/transformerconfig # The path to the transformer config + configuredBy: # The ObjectReference of the transformer config + kind: Transformer + apiVersion: builtin + name: foo + namespace: default +``` + +The possible fields for each item in the list is identical to the possible fields in `config.kubernetes.io/origin`, except that +the transformer annotation does not have a `path` field for the path to the resource file itself. + +All local file paths are relative to the top-level kustomization, i.e. the kustomization file in the directory upon +which `kustomize build` was invoked. For example, if someone were to run `kustomize build foo`, all file paths +in the annotation output would be relative to`foo/kustomization.yaml`. + +All remote file paths are relative to the root of the remote repository. + +Any fields that are not applicable would be omitted from the final output. If a transformer is invoked via +a field in the kustomization file, `configuredIn` would point to the kustomization file itself. + + +#### Examples + +#### Local custom transformer +A kustomization such as the following: + +```yaml +transformers: +- transformer.yaml + +buildMetadata: [transformerAnnotations] + +``` + +would produce a resource with an annotation like the following: + +```yaml +config.kubernetes.io/transformations: | +- configuredIn: transformer.yaml + configuredBy: + kind: MyTransformer + apiVersion: v1 + name: transformer +``` + +##### Remote builtin transformer + local builtin transformer +We have a top-level kustomization such as the following: + +```yaml +resources: +- github.com/examplerepo/?ref=v1.0.6 + +buildMetadata: [transformerAnnotations] + +namespace: my-ns +``` + +which uses `github.com/examplerepo/?ref=v1.0.6` as a remote base. This remote base has the following kustomization +defined in `github.com/examplerepo/kustomization.yaml`: + +```yaml +resources: +- deployment.yaml + +namePrefix: pre- +``` + +`deployment.yaml` contains the following: + +```yaml +apiVersion: v1 +kind: Deployment +metadata: + name: deploy +``` + +Running `kustomize build` on the top-level kustomization would produce the following output: + +```yaml +apiVersion: v1 +kind: Deployment +metadata: + name: pre-deploy + annotations: + config.kubernetes.io/transformations: | + - ref: v1.0.6 + repo: github.com/examplerepo + configuredIn: kustomization.yaml + configuredBy: + kind: PrefixSuffixTransformer + apiVersion: builtin + - configuredIn: kustomization.yaml + configuredBy: + kind: NamespaceTransformer + apiVersion: builtin +``` + +### Kustomize edit + +We will want to provide convenient commands for users to edit the `buildMetadata` field in their kustomization. +We should support the following commands: + +`kustomize edit add buildMetadata originAnnotations` + +`kustomize edit add buildMetadata transformerAnnotations` + +`kustomize edit add buildMetadata *` + +`kustomize edit remove buildMetadata originAnnotaitons` + +`kustomize edit remove buildMetadata transformerAnnotations` + +`kustomize edit remove buildMetadata *` + +The wildcard match `*` should add/remove all possible options for `buildMetadata`. + +### Implementation + +If there is a large tree of kustomization files, the data stored for origin and transformer data could be potentially +very large, so we should be careful about implementation. We should: + +1. Ensure that we are only accumulating the data if the option is set. +2. Account for `generators` that may appear in the `transformers` field. When we are processing each plugin invoked +in the `transformers` field, we should check to see if the resources have an `origin` annotation. If not, we can assume that +it was generated by the transformer and we can add the corresponding `origin` annotation. If it already has an `origin` +annotation, we can add the corresponding `transformation` annotation. +3. A `ConfigMapGenerator` with `merge` set to true actually transforms instead of generates, so we should take care to +account for this special case. In this case, we should add a transformer annotation. + + +### User Stories + +#### Story 1 + +I am running kustomize build on some configuration that has already been created. This configuration has a large number +of resources and patches through multiple overlays in different directories. After running `kustomize build`, I get a +stream of 100+ resources, but I notice that a few of them have a typo, likely originating from the same base or patch +file. However, because my directory structure is large and complex, it is difficult to narrow down which base these +resources originated from and which patches have been applied to it. In the top-level kustomization file, I can add a +new `buildMetadata` field: + +```yaml +buildMetadata: [originAnnotations, transformerAnnotations] +``` + +The output of `kustomize build` will contain patched resources with annotations similar to the following: +``` +annotations: + config.kubernetes.io/origin: | + path: deployment.yaml + config.kubernetes.io/transformations: + - configuredIn: ../base/kustomization.yaml + configuredBy: + kind: PatchTransformer + apiVersion: builtin + - configuredIn: ../dev/kustomization.yaml + configuredBy: + kind: PatchStrategicMergeTransformer + apiVersion: builtin + - configuredIn: ../dev/kustomization.yaml + configuredBy: + kind: PatchesJson6902Transformer + apiVersion: builtin +``` + +From these annotations, I can narrow down the set of files that contributed to the resource's final output and +debug it accordingly. + +#### Story 2 + +I am using a tool that automatically runs `kustomize build` and deploys the output to the cluster, such as `skaffold deploy`, +`kubectl apply -k`, or Google's Config Sync. I am unfamiliar with kustomize CLI and do not wish to install it to my +local machine or learn how to use it beyond using tools that run `kustomize build`. I add the `originAnnotations` and +`transformationAnnotations` options to my top-level kustomization file. + +With one of these tools, I apply my kustomize configuration to my cluster, but later discover that something in my cluster +is not configured correctly. Because the annotations have persisted to the cluster, I can take a look at the resources in +my cluster and understand how each of them was created, helping me figure out what went wrong. + + +### Risks and Mitigations +N/A + +### Dependencies +N/A + +### Scalability +With a large input, the data collected here could be very large, so we should take care to only accumulate data when necessary. + +## Drawbacks +N/A + +## Alternatives +1. Putting this data in the comments of the rendered resources, but this would be a large project because +kustomize doesn't currently support comments. + +2. Making this a flag to `kustomize build` rather than additional field, but we would like to align +with the kustomize way of avoiding build-time side effects and have everything declared explicitly in the kustomization. + +3. Having a separate `kustomize` command that prints out only the origin and/or transformer data. While this could be very +useful as a debugging feature, there are use cases for tools that automatically run kustomize and deploy, where it would be more +useful to the user to have the annotations persist to the cluster. See User Story 2. + +## Rollout Plan +This is a fairly lightweight, minor and optional feature that can be implemented and +released without needing a rollout plan, much like the `originAnnotations` option that +we are extending. + +The `origin` annotation is simply being extended, so there is no need to mark this feature as alpha. The rollout plan +for the `transformation` annotation is outlined below: + +### Alpha +- Will the feature be gated by an "alpha" flag? Which one? + +The feature will not be gated by an alpha flag, but we will start the annotation with the prefix "alpha", so the +annotation will key be `alpha.config.kubernetes.io/transformations` + +- Will the feature be available in `kubectl kustomize` during alpha? Why or why not? + +Yes, the feature will be part of `kubectl kustomize` as an option in the `buildMetadata` field, and will likewise +start with the annotation key `alpha.config.kubernetes.io/transformations` + +### GA +We will wait for two `kubectl` releases before promoting the feature. We will iterate the feature based on feedback from +users of Config Sync, Skaffold, kustomize CLI, and kustomize-in-kubectl prior to promotion.