diff --git a/cmd/config/docs/api-conventions/config-io.md b/cmd/config/docs/api-conventions/config-io.md deleted file mode 100644 index 35b013608..000000000 --- a/cmd/config/docs/api-conventions/config-io.md +++ /dev/null @@ -1,65 +0,0 @@ -# Configuration IO API Semantics - -Resource Configuration may be read / written from / to sources such as directories, -stdin|out or network. Tools may be composed using pipes such that the tools writing -Resource Configuration may be a different tool from the one that read the configuration. -In order for tools to be composed in this way, while preserving origin information -- -such as the original file, index, etc.: - -Tools **SHOULD** insert the following annotations when reading from sources, -and **SHOULD** delete the annotations when writing to sinks. - -### `config.kubernetes.io/path` - -Records the slash-delimited, OS-agnostic, relative file path to a Resource. - -This annotation **SHOULD** be set when reading Resources from files. -It **SHOULD** be unset when writing Resources to files. -When writing Resources to a directory, the Resource **SHOULD** be written to the corresponding -path relative to that directory. - -Example: - -```yaml -metadata: - annotations: - config.kubernetes.io/path: "relative/file/path.yaml" -``` - -### `config.kubernetes.io/index` - -Records the index of a Resource in file. In a multi-object YAML file, Resources are separated -by three dashes (`---`), and the index represents the position of the Resource starting from zero. - -This annotation **SHOULD** be set when reading Resources from files. -It **SHOULD** be unset when writing Resources to files. -When writing multiple Resources to the same file, the Resource **SHOULD** be written in the -relative order matching the index. - -When this annotation is not specified, it implies a value of `0`. - -Example: - -```yaml -metadata: - annotations: - config.kubernetes.io/path: "relative/file/path.yaml" - config.kubernetes.io/index: 2 -``` - -This represents the third Resource in the file. - -### `config.kubernetes.io/local-config` - -`config.kubernetes.io/local-config` declares that the configuration is to local tools -rather than a remote Resource. e.g. The `Kustomization` config in a `kustomization.yaml` -**SHOULD** contain this annotation so that tools know it is not intended to be sent to -the Kubernetes api server. - -Example: - -```yaml -metadata: - annotations: - config.kubernetes.io/local-config: "true" -``` diff --git a/cmd/config/docs/api-conventions/functions-spec.md b/cmd/config/docs/api-conventions/functions-spec.md index 7bc69df34..048d118ab 100644 --- a/cmd/config/docs/api-conventions/functions-spec.md +++ b/cmd/config/docs/api-conventions/functions-spec.md @@ -1,13 +1,18 @@ -# Configuration Functions Specification +# KRM Configuration Functions Specification + +_apiVersion: v1_ + +## Overview This document specifies a standard for client-side functions that operate on -Kubernetes declarative configurations. This standard enables creating -small, interoperable, and language-independent executable programs packaged as -containers that can be chained together as part of a configuration management pipeline. -The end result of such a pipeline are fully rendered configurations that can then be -applied to a control plane (e.g. Using ‘kubectl apply’ for Kubernetes control plane). -As such, although this document references Kubernetes Resource Model and API conventions, -it is completely decoupled from Kubernetes API machinery and does not depend on any +Kubernetes declarative configurations referred to as _KRM Functions_. This +standard enables creating small, interoperable, and language-independent +executable programs packaged as containers that can be chained together as part +of a configuration management pipeline. The end result of such a pipeline are +fully rendered configurations that can then be applied to a control plane (e.g. +Using ‘kubectl apply’ for Kubernetes control plane). As such, although this +document references Kubernetes Resource Model and API conventions, it is +completely decoupled from Kubernetes API machinery and does not depend on any in-cluster components. This document references terms described in [Kubernetes API Conventions][1]. @@ -18,168 +23,372 @@ interpreted as described in [RFC 2119][2]. ## Use Cases -_Configuration functions_ enable shift-left practices (client-side) through: +KRM functions enable shift-left practices (client-side) through: - Pre-commit / delivery validation and linting of configuration - - e.g. Fail if any containers don't have PodSecurityPolicy or CPU / Memory limits -- Implementation of abstractions as client actuated APIs (e.g. templating) - - e.g. Create a client-side _"CRD"_ for generating configuration checked into git -- Aspect Orient configuration / Injection of cross-cutting configuration - - e.g. T-Shirt size containers by annotating Resources with `small`, `medium`, `large` - and inject the cpu and memory resources into containers accordingly. - - e.g. Inject `init` and `side-car` containers into Resources based off of Resource - Type, annotations, etc. + - e.g. Fail if any containers don't have PodSecurityPolicy or CPU / Memory + limits +- Implementation of abstractions as client actuated APIs + - e.g. Create a client-side _"CRD"_ for generating configuration checked into + git +- Injection of cross-cutting configuration + - e.g. T-Shirt size containers by annotating resources with `small`, `medium`, + `large` and inject the cpu and memory resources into containers accordingly. + - e.g. Inject `init` and `side-car` containers into resources based off of + resource type, annotations, etc. Performing these on the client rather than the server enables: - Configuration to be reviewed prior to being sent to the API server - Configuration to be validated as part of the CI/CD pipeline -- Configuration for Resources to validated holistically rather than individually - per-Resource +- Configuration for resources to validated holistically rather than individually + per-resource - e.g. ensure the `Service.selector` and `Deployment.spec.template` labels match. - - e.g. MutatingWebHooks are scoped to a single Resource instance at a time. + - e.g. MutatingWebHooks are scoped to a single resource instance at a time. - Low-level tweaks to the output of high-level abstractions - - e.g. add an `init container` to a client _"CRD"_ Resource after it was generated. + - e.g. add an `init container` to a client _"CRD"_ resource after it was + generated. - Composition and layering of multiple functions together - Compose generation, injection, validation together -## Spec +## Definitions -### Input Type +- **function:** A containerized program conforming to the spec described in this + document. +- **orchestrator:** A program that invokes the function container, passing + arguments and processing its output. -A function MUST accept as input a single [Kubernetes List type][3]. -The `items` field in the input will contain a sequence of [Object types][3]. -A function MAY not support [Simple types][3] and List types. +## Interface -An example using `v1/ConfigMapList` as input: +The inter-process communication between the orchestrator and a function works as +follows: + +1. Orchestrator runs the function container and provides the input on `stdin`. + The input is a Kubernetes object of kind `ResourceList` as described below. +2. Function reads the input from `stdin`, performs computations, and provides + the output as a `ResourceList` to `stdout`. The function MAY also emit + non-structured error message on `stderr`. +3. Orchestrator uses the `stdout`, `stderr`, and the exit code of the function + as it sees fit following to the semantics described below. + +### Schema + +A function MUST accept input from `stdin` and MUST output to `stdout` a +Kubernetes object of kind `ResourceList` with the following OpenAPI schema: ```yaml -apiVersion: v1 -kind: ConfigMapList -items: - - apiVersion: v1 - kind: ConfigMap - metadata: - name: config1 - data: - p1: v1 - p2: v2 - - apiVersion: v1 - kind: ConfigMap - metadata: - name: config2 +swagger: "2.0" +info: + title: KRM Functions Specification (ResourceList) + version: v1 +definitions: + ResourceList: + type: object + description: ResourceList is the input/output wire format for KRM functions. + x-kubernetes-group-version-kind: + - group: config.kubernetes.io + kind: ResourceList + version: v1 + - group: config.kubernetes.io + kind: ResourceList + version: v1beta1 + required: + - items + properties: + apiVersion: + description: apiVersion of ResourceList + type: string + kind: + description: kind of ResourceList i.e. `ResourceList` + type: string + items: + type: array + description: | + [input/output] + Items is a list of Kubernetes objects: + https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#types-kinds). + + A function will read this field in the input ResourceList and populate + this field in the output ResourceList. + functionConfig: + type: object + description: | + [input] + FunctionConfig is an optional Kubernetes object for passing arguments to a + function invocation. + results: + type: array + description: | + [output] + Results is an optional list that can be used by function to emit results + for observability and debugging purposes. + items: + "$ref": "#/definitions/Result" + Result: + type: object + required: + - message + properties: + message: + type: string + description: Message is a human readable message. + severity: + type: string + enum: + - error + - warning + - info + default: error + description: | + Severity is the severity of a result: + + "error": indicates an error result. + "warning": indicates a warning result. + "info": indicates an informational result. + resourceRef: + type: object + description: | + ResourceRef is the metadata for referencing a Kubernetes object + associated with a result. + required: + - apiVersion + - kind + - name + properties: + apiVersion: + description: + APIVersion refers to the `apiVersion` field of the object + manifest. + type: string + kind: + description: Kind refers to the `kind` field of the object. + type: string + namespace: + description: + Namespace refers to the `metadata.namespace` field of the object + manifest. + type: string + name: + description: + Name refers to the `metadata.name` field of the object manifest. + type: string + field: + type: object + description: | + Field is the reference to a field in the object. + If defined, `ResourceRef` must also be provided. + required: + - path + properties: + path: + type: string + description: | + Path is the JSON path of the field + e.g. `spec.template.spec.containers[3].resources.limits.cpu` + currentValue: + description: | + CurrrentValue is the current value of the field. + Can be any value - string, number, boolean, array or object. + proposedValue: + description: | + PropposedValue is the proposed value of the field to fix an issue. + Can be any value - string, number, boolean, array or object. + file: + type: object + description: File references a file containing the resource. + required: + - path + properties: + path: + type: string + description: | + Path is the OS agnostic, slash-delimited, relative path. + e.g. `some-dir/some-file.yaml`. + index: + type: number + default: 0 + description: Index of the object in a multi-object YAML file. + tags: + type: object + additionalProperties: + type: string + description: | + Tags is an unstructured key value map stored with a result that may be set + by external tools to store and retrieve arbitrary metadata. ``` -An example using `v1/List` as input: +#### Examples + +The following is an example input, where the custom resource of kind +`FulfillmentCenter` is provided as `functionConfig`. The function will operate +on one resource of kind `Service`. ```yaml -apiVersion: v1 -kind: List -items: - - apiVersion: foo-corp.com/v1 - kind: FulfillmentCenter - metadata: - name: staging - address: "100 Main St." - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: namespace-reader - rules: - - resources: - - namespaces - apiGroups: - - "" - verbs: - - get - - watch - - list -``` - -In addition, a function MUST accept as input a List of kind `ResourceList` where the -`functionConfig` field, if present, will contain the invocation-specific configuration passed to the function -by the orchestrator. -Functions MAY consider this field optional so that they can be triggered in an ad-hoc fashion. - -An example using `config.kubernetes.io/v1beta1/ResourceList` as input: - -```yaml -apiVersion: config.kubernetes.io/v1beta1 +apiVersion: config.kubernetes.io/v1 kind: ResourceList functionConfig: apiVersion: foo-corp.com/v1 kind: FulfillmentCenter metadata: name: staging - metadata: - annotations: - config.kubernetes.io/function: | - container: - image: gcr.io/example/foo:v1.0.0 spec: address: "100 Main St." items: - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole + - apiVersion: v1 + kind: Service metadata: - name: namespace-reader - rules: - - resources: - - namespaces - apiGroups: - - "" - verbs: - - get - - watch - - list + name: wordpress + labels: + app: wordpress + annotations: + config.kubernetes.io/index: "0" + config.kubernetes.io/path: "service.yaml" + spec: # Example comment + type: LoadBalancer + selector: + app: wordpress + tier: frontend + ports: + - protocol: TCP + port: 80 ``` -Here `FulfillmentCenter` kind with name `staging` is passed as the invocation-specific configuration -to the function. +The following is an example output containing one result representing a +validation error: -### Output Type - -A function’s output MUST be the same as the input specification above --- i.e. `ResourceList` or `List`. -This is necessary to enable chaining two or more functions together in a pipeline. -The serialization format of the output SHOULD match that of its input on each invocation --- e.g. if the input was a `ResourceList`, the output should also be a `ResourceList`. +```yaml +apiVersion: config.kubernetes.io/v1 +kind: ResourceList +items: + - apiVersion: v1 + kind: Service + metadata: + name: wordpress + labels: + app: wordpress + annotations: + config.kubernetes.io/index: "0" + config.kubernetes.io/path: "service.yaml" + spec: # Example comment + type: LoadBalancer + selector: + app: wordpress + tier: frontend + ports: + - protocol: TCP + port: 80 +results: + - message: "Invalid type. Expected: integer, given: string" + severity: error + resourceRef: + apiVersion: v1 + kind: Service + name: wordpress + field: + path: spec.ports.0.port + file: + path: service.yaml +``` ### Serialization Format A function MUST support YAML as a serialization format for the input and output. -A function MUST use utf8 encoding (as YAML is a superset of JSON, JSON will also be supported -by any conforming function). - -### Operations - -A function MAY Create, Update, or Delete any number of items in the `items` field and output the -resultant list. - -A function MAY modify annotations with prefix `config.kubernetes.io`, but must be careful about -doing so since they’re used for orchestration purposes and will likely impact subsequent functions -in the pipeline. - -A function SHOULD preserve comments when input serialization format is YAML. -This allows for human authoring of configuration to coexist with changes made by functions. +A function MUST use utf8 encoding (as YAML is a superset of JSON, JSON will also +be supported by any conforming function). ### Containerization A function MUST be implemented as a container. -A function container MUST be capable of running as a non-root user if it does not require -access to host filesystem or makes network calls. +A function container MUST be capable of running as a non-root user `nobody` if +it does not require access to host filesystem or makes network calls. -### stdin/stdout/stderr and Exit Codes +### stderr -A function MUST accept input from stdin and emit output to stdout. +Any non-structured error messages MUST be emitted to `stderr`. `stdout` is +reserved for `ResourceList` as described above. -Any error messages MUST be emitted to stderr. +### Exit Code -An exit code of zero indicates function execution was successful. -A non-zero exit code indicates a failure. +An exit code of zero indicates function execution was successful. A non-zero +exit code indicates a failure. -[1]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md +### Operations + +A function MAY Create, Update, or Delete any number of items in the `items` +field and output the resultant list in the corresponding `items` field of the +output. + +A function MAY modify annotations with prefix `config.kubernetes.io`, but must +be careful about doing so since they’re used for orchestration purposes and will +likely impact subsequent functions in the pipeline. Several annotations and +their semantics are listed below. + +A function SHOULD preserve comments when input serialization format is YAML. +This allows for human authoring of configuration to coexist with changes made by +functions. + +### Annotations + +The orchestrator and function can communicate on a per-resource basis using the +following annotations: + +#### `config.kubernetes.io/path` + +Records the slash-delimited, OS-agnostic, relative file path to a resource. The +path is relative to a fix location on the filesystem. Different orchestrator +implementations can choose different fixed points. + +This annotation only exists in the wire format and is not persisted to the +filesystem. The orchestrator sets this annotation when reading files from the +local filesystem and removes the annotation when writing the output of functions +back to the filesystem. + +Example: + +```yaml +metadata: + annotations: + config.kubernetes.io/path: "relative/file/path.yaml" +``` + +#### `config.kubernetes.io/index` + +Records the index of a Resource in file. In a multi-object YAML file, resources +are separated by three dashes (`---`), and the index represents the position of +the Resource starting from zero. When this annotation is not specified, it +implies a value of `0`. + +Similar to `path` annotation, this annotation is not persisted when writing +configurations files on the file system. + +Example: + +```yaml +metadata: + annotations: + config.kubernetes.io/path: "relative/file/path.yaml" + config.kubernetes.io/index: 2 +``` + +This represents the third resource in the file. + +#### `config.kubernetes.io/local-config` + +This annotation declares that the resource is not meant to be applied to a +Kubernetes api server. It is only used for client-side tooling. + +Example: + +```yaml +metadata: + annotations: + config.kubernetes.io/local-config: "true" +``` + +[1]: + https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md [2]: https://tools.ietf.org/html/rfc2119 -[3]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#types-kinds +[3]: + https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#types-kinds