# Replacements and Patch value in the structured data. **Authors**: - koba1t **Reviewers**: - natasha41575 - knverey **Status**: implementable ## Summary This proposal decides the interfaces to change values in the structured data (like json,yaml) inside a Kubernetes objects' field value and implements changing function target a few formats (mainly json). ## Motivation kustomize can apply structured edits to Kubernetes objects defined in yaml files.\ Sometimes structured multi-line or long single line string (ex. json,yaml, and other structured format data) is injected in Kubernetes objects' string literal field. From the kustomize perspective, these "structured" multiline strings are just arbitrary unstructured strings.\ So, kustomize can't manipulate one value on structured, formatted data in the Kubernetes object's string literal field. This function is expected behavior, but kustomize will be very helpful if it can change the value of structured data like json and yaml substrings in a string literal.\ This proposal allows the user to identify strings literals containing JSON/YAML data so that Kustomize can make structured edits to the data they contain. This allows the requested functionality to be added without violating Kustomize's core principle of supporting structured edits exclusively.\ For example, kustomize can't change the value `"REPLACE_TARGET_HOSTNAME"` in this yaml file straightforwardly. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: target-configmap data: config.json: |- {"config": { "id": "42", "hostname": "REPLACE_TARGET_HOSTNAME" }} ``` This function has been wanted for a long time. - https://github.com/kubernetes-sigs/kustomize/issues/680 - https://github.com/kubernetes-sigs/kustomize/issues/3787 - https://github.com/kubernetes-sigs/kustomize/issues/4517 This function can be an alternative deprecated [vars](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/vars/) function in many use cases. **Goals:** 1. Provide the way to update values in the structured data like kubernetes objects with JSON/YAML formats. **Non-goals:** 1. Do not provide [Unstructured edits, because kustomize eschews parameterization](https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/#unstructured-edits). 1. Do not provide a way for targeting/merging values within strings from the [patches](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/patches/) field. 1. Do not provide a way for customizing field merge strategies for the embedded data. ## Proposal ### Replacement the value in structured data I propose to extend the `fieldPath` and `fieldPaths` fields to replace the value in structured data with the replacements function.\ This idea is extending how to select any field in replacement [TargetSelector](https://github.com/kubernetes-sigs/kustomize/blob/8668691ade05bc17b3c6f44bcd4723735033196e/api/types/replacement.go#L52-L64). If the `source.fieldPath` and `targets.fieldPaths` had extra values after a specific string literal in Yaml, Kustomize tries to parse that string as structure data and tries to drill down using that additional values. #### Example. ```yaml ## replacement replacements: - source: kind: ConfigMap name: source-configmap fieldPath: data.HOSTNAME targets: - select: kind: ConfigMap name: target-configmap fieldPaths: - data.config\.json.config.hostname # A path after `config\.json` is pointing one place in the structured data. ``` Please check [Story 1](#Story-1). ### Disciplined merge the value in structured data with configMapGenerator and secretGenerator This Proposal is to add parameters for [configMapGenerator](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/configmapgenerator/) and [secretGenerator](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/secretgenerator/) to allow the merging of two string literals when the behavior option is set to merge and string literals value is structured.\ This idea is to add one parameter, `mergeValues` to [GeneratorArgs](https://github.com/kubernetes-sigs/kustomize/blob/672c751715be7dd0b43b4a2fce956c84452e0db9/api/types/generatorargs.go#L7-L27) used from configMapGenerator and secretGenerator. The `mergeValues` option is a list that contains two parameters, `key` and `format`. One element corresponds to one string literal, `key` is used to select a string literal to merge, and `format` is used to designate a format for string literal which is JSON or YAML.\ This merge operation will be implemented for a part of [Overriding Base ConfigMap Values](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/configmapgenerator/#overriding-base-configmap-values). It will execute to merge two string literal having same [key](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#define-the-key-to-use-when-creating-a-configmap-from-a-file) name when merging two configMap or secret.\ #### Example ```yaml configMapGenerator: - name: demo-settings behavior: merge # This function requires `behavior: merge`. mergeValues: - key: config.json # Key with a target to merge. format: json # Setting structured data format MUST be YAML/JSON. literals: - config.json: |- { "config": { "hostname": "REPLACE_TARGET_HOSTNAME", "value": { "foo": "bar" } } } ``` Please check [Story 2](#Story-2). #### Appendix - https://github.com/kubernetes-sigs/kustomize/issues/680#issuecomment-458834785 ## User Stories #### Story 1 Scenario summary: Replacement the value inside for structured data(json) in the configMap. kustomize patching overlay is very strong to manage common yaml when using many cluster.\ But, if you want to set cluster specific change value in the json with configMap data field, you have to replacement whole json file. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: target-configmap-dev data: config.json: |- {"config": { "id": "42", "hostname": "dev.example.com }} --- apiVersion: v1 kind: ConfigMap metadata: name: target-configmap-prod data: config.json: |- {"config": { "id": "42", "hostname": "prod.example.com" }} ``` So if we can replacement this value in the substring formatted with json, we can easy to overlay this difference. ```yaml ## source apiVersion: v1 kind: ConfigMap metadata: name: source-configmap data: HOSTNAME: www.example.com --- apiVersion: v1 kind: ConfigMap metadata: name: target-configmap data: config.json: |- {"config": { "id": "42", "hostname": "REPLACE_TARGET_HOSTNAME" }} ``` ```yaml ## replacement replacements: - source: kind: ConfigMap name: source-configmap fieldPath: data.HOSTNAME targets: - select: kind: ConfigMap name: target-configmap fieldPaths: - data.config\.json.config.hostname ``` ```yaml ## expected apiVersion: v1 kind: ConfigMap metadata: name: source-configmap data: HOSTNAME: www.example.com --- apiVersion: v1 kind: ConfigMap metadata: name: target-configmap data: config.json: '{"config":{"hostname":"www.example.com","id":"42"}}' ``` #### Story 2 Scenario summary: Merge the two values formatted by structured data(json) with configMapGenerator. Many application needs to be set with json format file. And, That file is handled with configMap when the application is running on kubernetes.\ So, If kustomize configMapGenerator can merge for json inside a configMap data value, A json format file will be simple and easy to handle. #### Source ```yaml # base/kustomization.yaml configMapGenerator: - name: demo literals: - config.json: |- { "config": { "loglevel": debug, "parameter": { "foo": "bar" } } } ``` ```yaml # overlay/kustomization.yaml resources: - ../base configMapGenerator: - name: demo behavior: merge mergeValues: - key: config.json # Key with a target to merge. format: json # Setting structured data format MUST be YAML/JSON. literals: - config.json: |- { "config": { "hostname": "www.example.com", "parameter": { "baz": "qux" } } } ``` #### Result ```yaml apiVersion: v1 data: config.json: |- { "config": { "loglevel": debug, "hostname": "www.example.com", "parameter": { "foo": "bar", "baz": "qux" } } } kind: ConfigMap metadata: name: demo-xxxxxxxxxx # name suffix hash ``` #### Story 3 Scenario summary: Replacement the value inside for yaml in the configMap. A few applications require to use yaml format config file, and some major cloudnative applications are using that.(ex, Prometheus,AlertManager)\ A value you want to overlay in yaml is usually into a nested yaml structure. So if you can overlay value inside yaml, you won't need to copy a whole yaml file. ```yaml ## source apiVersion: v1 kind: ConfigMap metadata: name: environment-config data: env: dev --- apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config data: prometheus.yml: |- global: external_labels: prometheus_env: TARGET_ENVIROMENT scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] ``` ```yaml ## replacement replacements: - source: kind: ConfigMap name: environment-config fieldPath: data.env targets: - select: kind: ConfigMap name: prometheus-config fieldPaths: - data.prometheus\.yml.global.external_labels?prometheus_env ``` ```yaml ## expected apiVersion: v1 kind: ConfigMap metadata: name: environment-config data: env: dev --- apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config data: prometheus.yml: |- global: external_labels: prometheus_env: dev scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] ``` #### Story 4 Scenario summary: Replacement the value inside for json in the annotations. A few times, an application on your cluster requires to set json format config for the *Annotations* on kubernetes yaml resources. We need to overlay in this position value. ##### Example ```yaml ## source apiVersion: cloud.google.com/v1 kind: BackendConfig metadata: name: debug-backend-config spec: securityPolicy: name: "debug-security-policy" --- apiVersion: v1 kind: Service metadata: name: appA-svc annotations: cloud-provider/backend-config: '{"ports": {"appA":"gke-default-backend-config"}}' spec: ports: - name: appA port: 1234 protocol: TCP targetPort: 8080 ``` ```yaml ## replacement replacements: - source: kind: BackendConfig name: debug-backend-config fieldPath: metadata.name targets: - select: kind: ConServicefigMap name: appA-svc fieldPaths: - metadata.annotations.cloud-provider/backend-config.ports.appA ``` ```yaml ## expected apiVersion: cloud.google.com/v1 kind: BackendConfig metadata: name: debug-backend-config spec: securityPolicy: name: "debug-security-policy" --- apiVersion: v1 kind: Service metadata: name: appA-svc annotations: cloud-provider/backend-config: '{"ports": {"appA":"debug-backend-config"}}' spec: ports: - name: appA port: 1234 protocol: TCP targetPort: 8080 ``` - https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#unique_backendconfig_per_service_port - https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/#listen-ports ### Risks and Mitigations ### Dependencies ### Scalability ## Drawbacks ## Alternatives ## Rollout Plan ### Alpha - Will the feature be gated by an "alpha" flag? Which one? - Will the feature be available in `kubectl kustomize` during alpha? Why or why not? ### Beta ### GA