kustomize/proposals/22-03-value-in-the-structured-data.md

Replacements and Patch value in the structured data.

Authors:

• koba1t

Reviewers:

• natasha41575
• knverey

Status: implementable

Summary

This proposal decides the interfaces to change values in the structured data (like json,yaml) inside a Kubernetes objects' field value and implements changing function target a few formats (mainly json).

Motivation

kustomize can apply structured edits to Kubernetes objects defined in yaml files.
Sometimes structured multi-line or long single line string (ex. json,yaml, and other structured format data) is injected in Kubernetes objects' string literal field. From the kustomize perspective, these "structured" multiline strings are just arbitrary unstructured strings.
So, kustomize can't manipulate one value on structured, formatted data in the Kubernetes object's string literal field. This function is expected behavior, but kustomize will be very helpful if it can change the value of structured data like json and yaml substrings in a string literal.
While kustomize won't support unstructured edits in general, theis proposal allows editing of an unstructured string literal as an exception if the string literal is a structured string of a well-known format like json.\

For example, kustomize can't change the value "REPLACE_TARGET_HOSTNAME" in this yaml file straightforwardly.

apiVersion: v1
kind: ConfigMap
metadata:
  name: target-configmap
data:
  config.json: |-
    {"config": {
      "id": "42",
      "hostname": "REPLACE_TARGET_HOSTNAME"
    }}

This function has been wanted for a long time.

• https://github.com/kubernetes-sigs/kustomize/issues/680
• https://github.com/kubernetes-sigs/kustomize/issues/3787
• https://github.com/kubernetes-sigs/kustomize/issues/4517

This function can be an alternative deprecated vars function in many use cases.

Goals:

1. Provide the way to update values in the structured data like kubernetes objects with many formats.
2. Be able to add replacement able format after.

Non-goals:

1. Do not provide Unstructured edits, because kustomize eschews parameterization.

Proposal

Replacement the value in structured data

I propose to add options for replacing the value in structured data to replacements function. My sample implementation is here.
This idea is add two parameter format and formatPath to options in replacement TargetSelector. The format option is used by select to structured data format like "json" or "yaml", and The formatPath option is "path" to target to change values in structured data with selected format from format option.
I think these two parameters can't select a specific default value. Therefore kustomize return error message for the user if only one parameter was set.\

Example.

## replacement
replacements:
- source:
    kind: ConfigMap
    name: source-configmap
    fieldPath: data.HOSTNAME
  targets:
  - select:
      kind: ConfigMap
      name: target-configmap
    fieldPaths:
    - data.config\.json
    options:
      format: 'json'                  # Setting structured data format.
      formatPath: '/config/hostname'  # Setting replacements path.

Please check Story 1.

Disciplined merge the value in structured data with configMapGenerator

This Proposal is add option for configMapGenerator to allow merge two string literals when the behavior option is setting to merge and string literals value is structured.
This idea is add one parameter for valueStructuredMergeFormat to option. The valueStructuredMergeFormat option is used by select to structured data format like "json" or "yaml". And this function needs to work requires setting behavior: merge.
This merge operation will be implemented for a part of Overriding Base ConfigMap Values. It will execute to merge two string literal having same key name when merging two configMap.

Example

configMapGenerator:
- name: demo-settings
  behavior: merge                     # This function requires `behavior: merge`.
  option:
    valueStructuredMergeFormat: json  # Setting structured data format.
  literals:
  - config.json: |-
      {
        "config": {
          "hostname": "REPLACE_TARGET_HOSTNAME",
          "value": {
            "foo": "bar"
          }
        }
      }

Please check Story 2.

Appendix

• https://github.com/kubernetes-sigs/kustomize/issues/680#issuecomment-458834785

User Stories

Story 1

Scenario summary: Replacement the value inside for structured data(json) in the configMap.

kustomize patching overlay is very strong to manage common yaml when using many cluster.
But, if you want to set cluster specific change value in the json with configMap data field, you have to replacement whole json file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: target-configmap-dev
data:
  config.json: |-
    {"config": {
      "id": "42",
      "hostname": "dev.example.com
    }}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: target-configmap-prod
data:
  config.json: |-
    {"config": {
      "id": "42",
      "hostname": "prod.example.com"
    }}

So if we can replacement this value in the substring formatted with json, we can easy to overlay this difference.

## source
apiVersion: v1
kind: ConfigMap
metadata:
  name: source-configmap
data:
  HOSTNAME: www.example.com
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: target-configmap
data:
  config.json: |-
    {"config": {
      "id": "42",
      "hostname": "REPLACE_TARGET_HOSTNAME"
    }}

## replacement
replacements:
- source:
    kind: ConfigMap
    name: source-configmap
    fieldPath: data.HOSTNAME
  targets:
  - select:
      kind: ConfigMap
      name: target-configmap
    fieldPaths:
    - data.config\.json
    options:
      format: 'json'
      formatPath: '/config/hostname'

## expected
apiVersion: v1
kind: ConfigMap
metadata:
  name: source-configmap
data:
  HOSTNAME: www.example.com
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: target-configmap
data:
  config.json: '{"config":{"hostname":"www.example.com","id":"42"}}'

Story 2

Scenario summary: Merge the two values formatted by structured data(json) with configMapGenerator.

Many application needs to be set with json format file. And, That file is handled with configMap when the application is running on kubernetes.
So, If kustomize configMapGenerator can overlay to one line inside a configMap data value, A json format file will be simple and easy to handle.

Source

# base/kustomization.yaml
configMapGenerator:
- name: demo
  behavior: merge
  literals:
  - config.json: |-
      {
        "config": {
          "loglevel": debug,
          "parameter": {
            "foo": "bar"
          }
        }
      }

# overlay/kustomization.yaml
resources:
- ../base
configMapGenerator:
- name: demo
  option:
    valueMergeFormat: json
  literals:
  - config.json: |-
      {
        "config": {
          "hostname": "www.example.com",
          "parameter": {
            "baz": "qux"
          }
        }
      }

Result

apiVersion: v1
data:
  config.json: |-
    {
      "config": {
        "loglevel": debug,
        "hostname": "www.example.com",
        "parameter": {
          "foo": "bar",
          "baz": "qux"
        }
      }
    }
kind: ConfigMap
metadata:
  name: demo-xxxxxxxxxx  # name suffix hash

Story 3

Scenario summary: Replacement the value inside for yaml in the configMap.

A few applications require to use yaml format config file, and some major cloudnative applications are using that.(ex, Prometheus,AlertManager)
A value you want to overlay in yaml is usually into a nested yaml structure. So if you can overlay value inside yaml, you won't need to copy a whole yaml file.

## source
apiVersion: v1
kind: ConfigMap
metadata:
  name: environment-config
data:
  env: dev
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |-
    global:
      external_labels:
        prometheus_env: TARGET_ENVIROMENT
    scrape_configs:
      - job_name: "prometheus"
        static_configs:
          - targets: ["localhost:9090"]

## replacement
replacements:
- source:
    kind: ConfigMap
    name: environment-config
    fieldPath: data.env
  targets:
  - select:
      kind: ConfigMap
      name: prometheus-config
    fieldPaths:
    - prometheus\.yml
    options:
      format: 'yaml'
      formatPath: '/global/external_labels/prometheus_env'

## expected
apiVersion: v1
kind: ConfigMap
metadata:
  name: environment-config
data:
  env: dev
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |-
    global:
      external_labels:
        prometheus_env: dev
    scrape_configs:
      - job_name: "prometheus"
        static_configs:
          - targets: ["localhost:9090"]

Story 4

Scenario summary: Replacement the value inside for json in the annotations.

A few times, an application on your cluster requires to set json format config for the Annotations on kubernetes yaml resources. We need to overlay in this position value.

Example

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud-provider/backend-config: '{"ports": {"appA":"referrence_from_appA"}}'
spec:
  ports:
  - name: appA
    port: 1234
    protocol: TCP
    targetPort: 8080

• https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#unique_backendconfig_per_service_port
• https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/#listen-ports

Risks and Mitigations

Dependencies

Scalability

Drawbacks

Alternatives

Rollout Plan

Alpha

• Will the feature be gated by an "alpha" flag? Which one?
• Will the feature be available in kubectl kustomize during alpha? Why or why not?

Beta

GA