From 3ee1579688c2dfe3398686d61d111f2ac6ab1b7c Mon Sep 17 00:00:00 2001 From: Frank Farzan Date: Tue, 25 May 2021 16:09:32 -0700 Subject: [PATCH] Fully specify KRM Functions Spec and graduate it to v1. - Define the OpenAPI schema for ResourceList - Graduate `ResourceList` to apiVersion `v1`. Many functions have been implemented and we are effectively treating the spec as backwards compatible. - Add `results` field to the spec. This has been implemented in the orchestrator and couple of functions libraries for a while, but hasn't been yet standardized leading to minor discrepencies. Concurrent to this PR, we need to udpate the fn framework (kyaml) to be compatible with the standard and update the orchestrator to be backwards compatible. - Include per-resource annotations (previously a seperate doc) - Remove support for non-ResourceList wire formats. In practice, this is not widely used and complicates the standard. The orchestrators can easily covert a List to a ResourceList. --- cmd/config/docs/api-conventions/config-io.md | 65 --- .../docs/api-conventions/functions-spec.md | 455 +++++++++++++----- 2 files changed, 332 insertions(+), 188 deletions(-) delete mode 100644 cmd/config/docs/api-conventions/config-io.md diff --git a/cmd/config/docs/api-conventions/config-io.md b/cmd/config/docs/api-conventions/config-io.md deleted file mode 100644 index 35b013608..000000000 --- a/cmd/config/docs/api-conventions/config-io.md +++ /dev/null @@ -1,65 +0,0 @@ -# Configuration IO API Semantics - -Resource Configuration may be read / written from / to sources such as directories, -stdin|out or network. Tools may be composed using pipes such that the tools writing -Resource Configuration may be a different tool from the one that read the configuration. -In order for tools to be composed in this way, while preserving origin information -- -such as the original file, index, etc.: - -Tools **SHOULD** insert the following annotations when reading from sources, -and **SHOULD** delete the annotations when writing to sinks. - -### `config.kubernetes.io/path` - -Records the slash-delimited, OS-agnostic, relative file path to a Resource. - -This annotation **SHOULD** be set when reading Resources from files. -It **SHOULD** be unset when writing Resources to files. -When writing Resources to a directory, the Resource **SHOULD** be written to the corresponding -path relative to that directory. - -Example: - -```yaml -metadata: - annotations: - config.kubernetes.io/path: "relative/file/path.yaml" -``` - -### `config.kubernetes.io/index` - -Records the index of a Resource in file. In a multi-object YAML file, Resources are separated -by three dashes (`---`), and the index represents the position of the Resource starting from zero. - -This annotation **SHOULD** be set when reading Resources from files. -It **SHOULD** be unset when writing Resources to files. -When writing multiple Resources to the same file, the Resource **SHOULD** be written in the -relative order matching the index. - -When this annotation is not specified, it implies a value of `0`. - -Example: - -```yaml -metadata: - annotations: - config.kubernetes.io/path: "relative/file/path.yaml" - config.kubernetes.io/index: 2 -``` - -This represents the third Resource in the file. - -### `config.kubernetes.io/local-config` - -`config.kubernetes.io/local-config` declares that the configuration is to local tools -rather than a remote Resource. e.g. The `Kustomization` config in a `kustomization.yaml` -**SHOULD** contain this annotation so that tools know it is not intended to be sent to -the Kubernetes api server. - -Example: - -```yaml -metadata: - annotations: - config.kubernetes.io/local-config: "true" -``` diff --git a/cmd/config/docs/api-conventions/functions-spec.md b/cmd/config/docs/api-conventions/functions-spec.md index 7bc69df34..048d118ab 100644 --- a/cmd/config/docs/api-conventions/functions-spec.md +++ b/cmd/config/docs/api-conventions/functions-spec.md @@ -1,13 +1,18 @@ -# Configuration Functions Specification +# KRM Configuration Functions Specification + +_apiVersion: v1_ + +## Overview This document specifies a standard for client-side functions that operate on -Kubernetes declarative configurations. This standard enables creating -small, interoperable, and language-independent executable programs packaged as -containers that can be chained together as part of a configuration management pipeline. -The end result of such a pipeline are fully rendered configurations that can then be -applied to a control plane (e.g. Using ‘kubectl apply’ for Kubernetes control plane). -As such, although this document references Kubernetes Resource Model and API conventions, -it is completely decoupled from Kubernetes API machinery and does not depend on any +Kubernetes declarative configurations referred to as _KRM Functions_. This +standard enables creating small, interoperable, and language-independent +executable programs packaged as containers that can be chained together as part +of a configuration management pipeline. The end result of such a pipeline are +fully rendered configurations that can then be applied to a control plane (e.g. +Using ‘kubectl apply’ for Kubernetes control plane). As such, although this +document references Kubernetes Resource Model and API conventions, it is +completely decoupled from Kubernetes API machinery and does not depend on any in-cluster components. This document references terms described in [Kubernetes API Conventions][1]. @@ -18,168 +23,372 @@ interpreted as described in [RFC 2119][2]. ## Use Cases -_Configuration functions_ enable shift-left practices (client-side) through: +KRM functions enable shift-left practices (client-side) through: - Pre-commit / delivery validation and linting of configuration - - e.g. Fail if any containers don't have PodSecurityPolicy or CPU / Memory limits -- Implementation of abstractions as client actuated APIs (e.g. templating) - - e.g. Create a client-side _"CRD"_ for generating configuration checked into git -- Aspect Orient configuration / Injection of cross-cutting configuration - - e.g. T-Shirt size containers by annotating Resources with `small`, `medium`, `large` - and inject the cpu and memory resources into containers accordingly. - - e.g. Inject `init` and `side-car` containers into Resources based off of Resource - Type, annotations, etc. + - e.g. Fail if any containers don't have PodSecurityPolicy or CPU / Memory + limits +- Implementation of abstractions as client actuated APIs + - e.g. Create a client-side _"CRD"_ for generating configuration checked into + git +- Injection of cross-cutting configuration + - e.g. T-Shirt size containers by annotating resources with `small`, `medium`, + `large` and inject the cpu and memory resources into containers accordingly. + - e.g. Inject `init` and `side-car` containers into resources based off of + resource type, annotations, etc. Performing these on the client rather than the server enables: - Configuration to be reviewed prior to being sent to the API server - Configuration to be validated as part of the CI/CD pipeline -- Configuration for Resources to validated holistically rather than individually - per-Resource +- Configuration for resources to validated holistically rather than individually + per-resource - e.g. ensure the `Service.selector` and `Deployment.spec.template` labels match. - - e.g. MutatingWebHooks are scoped to a single Resource instance at a time. + - e.g. MutatingWebHooks are scoped to a single resource instance at a time. - Low-level tweaks to the output of high-level abstractions - - e.g. add an `init container` to a client _"CRD"_ Resource after it was generated. + - e.g. add an `init container` to a client _"CRD"_ resource after it was + generated. - Composition and layering of multiple functions together - Compose generation, injection, validation together -## Spec +## Definitions -### Input Type +- **function:** A containerized program conforming to the spec described in this + document. +- **orchestrator:** A program that invokes the function container, passing + arguments and processing its output. -A function MUST accept as input a single [Kubernetes List type][3]. -The `items` field in the input will contain a sequence of [Object types][3]. -A function MAY not support [Simple types][3] and List types. +## Interface -An example using `v1/ConfigMapList` as input: +The inter-process communication between the orchestrator and a function works as +follows: + +1. Orchestrator runs the function container and provides the input on `stdin`. + The input is a Kubernetes object of kind `ResourceList` as described below. +2. Function reads the input from `stdin`, performs computations, and provides + the output as a `ResourceList` to `stdout`. The function MAY also emit + non-structured error message on `stderr`. +3. Orchestrator uses the `stdout`, `stderr`, and the exit code of the function + as it sees fit following to the semantics described below. + +### Schema + +A function MUST accept input from `stdin` and MUST output to `stdout` a +Kubernetes object of kind `ResourceList` with the following OpenAPI schema: ```yaml -apiVersion: v1 -kind: ConfigMapList -items: - - apiVersion: v1 - kind: ConfigMap - metadata: - name: config1 - data: - p1: v1 - p2: v2 - - apiVersion: v1 - kind: ConfigMap - metadata: - name: config2 +swagger: "2.0" +info: + title: KRM Functions Specification (ResourceList) + version: v1 +definitions: + ResourceList: + type: object + description: ResourceList is the input/output wire format for KRM functions. + x-kubernetes-group-version-kind: + - group: config.kubernetes.io + kind: ResourceList + version: v1 + - group: config.kubernetes.io + kind: ResourceList + version: v1beta1 + required: + - items + properties: + apiVersion: + description: apiVersion of ResourceList + type: string + kind: + description: kind of ResourceList i.e. `ResourceList` + type: string + items: + type: array + description: | + [input/output] + Items is a list of Kubernetes objects: + https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#types-kinds). + + A function will read this field in the input ResourceList and populate + this field in the output ResourceList. + functionConfig: + type: object + description: | + [input] + FunctionConfig is an optional Kubernetes object for passing arguments to a + function invocation. + results: + type: array + description: | + [output] + Results is an optional list that can be used by function to emit results + for observability and debugging purposes. + items: + "$ref": "#/definitions/Result" + Result: + type: object + required: + - message + properties: + message: + type: string + description: Message is a human readable message. + severity: + type: string + enum: + - error + - warning + - info + default: error + description: | + Severity is the severity of a result: + + "error": indicates an error result. + "warning": indicates a warning result. + "info": indicates an informational result. + resourceRef: + type: object + description: | + ResourceRef is the metadata for referencing a Kubernetes object + associated with a result. + required: + - apiVersion + - kind + - name + properties: + apiVersion: + description: + APIVersion refers to the `apiVersion` field of the object + manifest. + type: string + kind: + description: Kind refers to the `kind` field of the object. + type: string + namespace: + description: + Namespace refers to the `metadata.namespace` field of the object + manifest. + type: string + name: + description: + Name refers to the `metadata.name` field of the object manifest. + type: string + field: + type: object + description: | + Field is the reference to a field in the object. + If defined, `ResourceRef` must also be provided. + required: + - path + properties: + path: + type: string + description: | + Path is the JSON path of the field + e.g. `spec.template.spec.containers[3].resources.limits.cpu` + currentValue: + description: | + CurrrentValue is the current value of the field. + Can be any value - string, number, boolean, array or object. + proposedValue: + description: | + PropposedValue is the proposed value of the field to fix an issue. + Can be any value - string, number, boolean, array or object. + file: + type: object + description: File references a file containing the resource. + required: + - path + properties: + path: + type: string + description: | + Path is the OS agnostic, slash-delimited, relative path. + e.g. `some-dir/some-file.yaml`. + index: + type: number + default: 0 + description: Index of the object in a multi-object YAML file. + tags: + type: object + additionalProperties: + type: string + description: | + Tags is an unstructured key value map stored with a result that may be set + by external tools to store and retrieve arbitrary metadata. ``` -An example using `v1/List` as input: +#### Examples + +The following is an example input, where the custom resource of kind +`FulfillmentCenter` is provided as `functionConfig`. The function will operate +on one resource of kind `Service`. ```yaml -apiVersion: v1 -kind: List -items: - - apiVersion: foo-corp.com/v1 - kind: FulfillmentCenter - metadata: - name: staging - address: "100 Main St." - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: namespace-reader - rules: - - resources: - - namespaces - apiGroups: - - "" - verbs: - - get - - watch - - list -``` - -In addition, a function MUST accept as input a List of kind `ResourceList` where the -`functionConfig` field, if present, will contain the invocation-specific configuration passed to the function -by the orchestrator. -Functions MAY consider this field optional so that they can be triggered in an ad-hoc fashion. - -An example using `config.kubernetes.io/v1beta1/ResourceList` as input: - -```yaml -apiVersion: config.kubernetes.io/v1beta1 +apiVersion: config.kubernetes.io/v1 kind: ResourceList functionConfig: apiVersion: foo-corp.com/v1 kind: FulfillmentCenter metadata: name: staging - metadata: - annotations: - config.kubernetes.io/function: | - container: - image: gcr.io/example/foo:v1.0.0 spec: address: "100 Main St." items: - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole + - apiVersion: v1 + kind: Service metadata: - name: namespace-reader - rules: - - resources: - - namespaces - apiGroups: - - "" - verbs: - - get - - watch - - list + name: wordpress + labels: + app: wordpress + annotations: + config.kubernetes.io/index: "0" + config.kubernetes.io/path: "service.yaml" + spec: # Example comment + type: LoadBalancer + selector: + app: wordpress + tier: frontend + ports: + - protocol: TCP + port: 80 ``` -Here `FulfillmentCenter` kind with name `staging` is passed as the invocation-specific configuration -to the function. +The following is an example output containing one result representing a +validation error: -### Output Type - -A function’s output MUST be the same as the input specification above --- i.e. `ResourceList` or `List`. -This is necessary to enable chaining two or more functions together in a pipeline. -The serialization format of the output SHOULD match that of its input on each invocation --- e.g. if the input was a `ResourceList`, the output should also be a `ResourceList`. +```yaml +apiVersion: config.kubernetes.io/v1 +kind: ResourceList +items: + - apiVersion: v1 + kind: Service + metadata: + name: wordpress + labels: + app: wordpress + annotations: + config.kubernetes.io/index: "0" + config.kubernetes.io/path: "service.yaml" + spec: # Example comment + type: LoadBalancer + selector: + app: wordpress + tier: frontend + ports: + - protocol: TCP + port: 80 +results: + - message: "Invalid type. Expected: integer, given: string" + severity: error + resourceRef: + apiVersion: v1 + kind: Service + name: wordpress + field: + path: spec.ports.0.port + file: + path: service.yaml +``` ### Serialization Format A function MUST support YAML as a serialization format for the input and output. -A function MUST use utf8 encoding (as YAML is a superset of JSON, JSON will also be supported -by any conforming function). - -### Operations - -A function MAY Create, Update, or Delete any number of items in the `items` field and output the -resultant list. - -A function MAY modify annotations with prefix `config.kubernetes.io`, but must be careful about -doing so since they’re used for orchestration purposes and will likely impact subsequent functions -in the pipeline. - -A function SHOULD preserve comments when input serialization format is YAML. -This allows for human authoring of configuration to coexist with changes made by functions. +A function MUST use utf8 encoding (as YAML is a superset of JSON, JSON will also +be supported by any conforming function). ### Containerization A function MUST be implemented as a container. -A function container MUST be capable of running as a non-root user if it does not require -access to host filesystem or makes network calls. +A function container MUST be capable of running as a non-root user `nobody` if +it does not require access to host filesystem or makes network calls. -### stdin/stdout/stderr and Exit Codes +### stderr -A function MUST accept input from stdin and emit output to stdout. +Any non-structured error messages MUST be emitted to `stderr`. `stdout` is +reserved for `ResourceList` as described above. -Any error messages MUST be emitted to stderr. +### Exit Code -An exit code of zero indicates function execution was successful. -A non-zero exit code indicates a failure. +An exit code of zero indicates function execution was successful. A non-zero +exit code indicates a failure. -[1]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md +### Operations + +A function MAY Create, Update, or Delete any number of items in the `items` +field and output the resultant list in the corresponding `items` field of the +output. + +A function MAY modify annotations with prefix `config.kubernetes.io`, but must +be careful about doing so since they’re used for orchestration purposes and will +likely impact subsequent functions in the pipeline. Several annotations and +their semantics are listed below. + +A function SHOULD preserve comments when input serialization format is YAML. +This allows for human authoring of configuration to coexist with changes made by +functions. + +### Annotations + +The orchestrator and function can communicate on a per-resource basis using the +following annotations: + +#### `config.kubernetes.io/path` + +Records the slash-delimited, OS-agnostic, relative file path to a resource. The +path is relative to a fix location on the filesystem. Different orchestrator +implementations can choose different fixed points. + +This annotation only exists in the wire format and is not persisted to the +filesystem. The orchestrator sets this annotation when reading files from the +local filesystem and removes the annotation when writing the output of functions +back to the filesystem. + +Example: + +```yaml +metadata: + annotations: + config.kubernetes.io/path: "relative/file/path.yaml" +``` + +#### `config.kubernetes.io/index` + +Records the index of a Resource in file. In a multi-object YAML file, resources +are separated by three dashes (`---`), and the index represents the position of +the Resource starting from zero. When this annotation is not specified, it +implies a value of `0`. + +Similar to `path` annotation, this annotation is not persisted when writing +configurations files on the file system. + +Example: + +```yaml +metadata: + annotations: + config.kubernetes.io/path: "relative/file/path.yaml" + config.kubernetes.io/index: 2 +``` + +This represents the third resource in the file. + +#### `config.kubernetes.io/local-config` + +This annotation declares that the resource is not meant to be applied to a +Kubernetes api server. It is only used for client-side tooling. + +Example: + +```yaml +metadata: + annotations: + config.kubernetes.io/local-config: "true" +``` + +[1]: + https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md [2]: https://tools.ietf.org/html/rfc2119 -[3]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#types-kinds +[3]: + https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#types-kinds