From 05a91893bfcf2414773dfe27a3ed77493a199d2c Mon Sep 17 00:00:00 2001 From: Jingfang Liu Date: Mon, 23 Jul 2018 13:54:15 -0700 Subject: [PATCH] break helloWorld example into two examples: - one for declaring a ConfigMap as resources - one for declaring a ConfigMap from ConfigMapGenerator and rolling update --- examples/README.md | 5 +- examples/combineConfigs.md | 298 +++++++++++++++++ examples/configGeneration.md | 424 ++++++++++--------------- examples/helloWorld/README.md | 127 -------- examples/helloWorld/kustomization.yaml | 7 +- 5 files changed, 470 insertions(+), 391 deletions(-) create mode 100644 examples/combineConfigs.md diff --git a/examples/README.md b/examples/README.md index 6ac0dc8ed..0e0caad1d 100644 --- a/examples/README.md +++ b/examples/README.md @@ -23,9 +23,12 @@ go get github.com/kubernetes-sigs/kustomize * [springboot](springboot/README.md) - Create a Spring Boot application production configuration from scratch. - * [configGeneration](configGeneration.md) - + * [combineConfigs](combineConfigs.md) - Mixing configuration data from different owners (e.g. devops/SRE and developers). + + * [configGenerations](configGeneration.md) - + Rolling update when ConfigMapGenerator changes * [breakfast](breakfast.md) - Customize breakfast for Alice and Bob. diff --git a/examples/combineConfigs.md b/examples/combineConfigs.md new file mode 100644 index 000000000..114d727ae --- /dev/null +++ b/examples/combineConfigs.md @@ -0,0 +1,298 @@ +[overlay]: ../docs/glossary.md#overlay +[target]: ../docs/glossary.md#target + +# Demo: combining config data from devops and developers + +Scenario: you have a Java-based server storefront in +production that various internal development teams +(signups, checkout, search, etc.) contribute to. + +The server runs in different environments: +_development_, _testing_, _staging_ and _production_, +accepting configuration parameters from java property +files. + +Using one big properties file for each environment is +difficult to manage. The files change frequently, and +have to be changed by devops exclusively because + + 1. the files must at least partially agree on certain + values that devops cares about and that developers + ignore and + 1. because the production + properties contain sensitive data like production + database credentials. + +## Property sharding + +With some study, we notice that the properties are +separable into categories. + +### Common properties + +E.g. internationalization data, static data like +physical constants, location of external services, etc. + +_Things that are the same regardless of environment._ + +Only one set of values is needed. + +Place them in a file called + + * `common.properties` + +(relative location defined below). + +### Plumbing properties + +E.g. serving location of static content (HTML, CSS, +javascript), location of product and customer database +tables, ports expected by load balancers, log sinks, +etc. + +_The different values for these properties are +precisely what sets the environments apart._ + +Devops or SRE will want full control over the values +used in production. Testing will have fixed +databases supporting testing. Developers will want +to do whatever they want to try scenarios under +development. + +Places these values in + + * `development/plumbing.properties` + * `staging/plumbing.properties` + * `production/plumbing.properties` + + +### Secret properties + +E.g. location of actual user tables, database +credentials, decryption keys, etc. + +_Things that are a subset of devops controls, that +nobody else has (or should want) access to._ + +Places these values in + + * `development/secret.properties` + * `staging/secret.properties` + * `production/secret.properties` + +[kubernetes secret]: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/ + +and control access to them with (for example) unix file +owner and mode bits, or better yet, put them in +a server dedicated to storing password protected +secrets, and use a field called `secretGenerator` +in your _kustomization_ to create a kubernetes +secret holding them (not covering that here). + + + +## A mixin approach to management + +The way to create _n_ cluster environments that share +some common information is to create _n_ overlays of a +common base. + +For the rest of this example, we'll do _n==2_, just +_development_ and _production_, since adding more +environments follows the same pattern. + +A cluster environment is created by +running `kustomize build` on a [target] that happens to +be an [overlay]. + +[helloworld]: helloWorld/README.md + +The following example will do that, but will focus on +configMap construction, and not worry about how to +connect the configMaps to deployments (that is covered +in the [helloworld] example). + + +All files - including the shared property files +discussed above - will be created in a directory tree +that is consistent with the base vs overlay file layout +defined in the [helloworld] demo. + +It will all live in this work directory: + + +``` +DEMO_HOME=$(mktemp -d) +``` + +### Create the base + + + +Make a place to put the base configuration: + + +``` +mkdir -p $DEMO_HOME/base +``` + +Make the data for the base. This direction by +definition should hold resources common to all +environments. Here we're only defining a java +properties file, and a `kustomization` file that +references it. + + +``` +cat <$DEMO_HOME/base/common.properties +color=blue +height=10m +EOF + +cat <$DEMO_HOME/base/kustomization.yaml +configMapGenerator: +- name: my-configmap + files: + - common.properties +EOF +``` + + +### Create and use the overlay for _development_ + +Make an abbreviation for the parent of the overlay +directories: + + +``` +OVERLAYS=$DEMO_HOME/overlays +``` + +Create the files that define the _development_ overlay: + + +``` +mkdir -p $OVERLAYS/development + +cat <$OVERLAYS/development/plumbing.properties +port=30000 +EOF + +cat <$OVERLAYS/development/secret.properties +dbpassword=mothersMaidenName +EOF + +cat <$OVERLAYS/development/kustomization.yaml +bases: +- ../../base +namePrefix: dev- +configMapGenerator: +- name: my-configmap + behavior: merge + files: + - plumbing.properties + - secret.properties +EOF +``` + +One can now generate the configMaps for development: + + +``` +kustomize build $OVERLAYS/development +``` + +#### Check the ConfigMap name + +The name of the generated `ConfigMap` is visible in this +output. + +The name should be something like `dev-my-configmap-b5m75ck895`: + + * `"dev-"` comes from the `namePrefix` field, + * `"my-configmap"` comes from the `configMapGenerator/name` field, + * `"-b5m75ck895"` comes from a deterministic hash that `kustomize` + computes from the contents of the configMap. + +The hash suffix is critical. If the configMap content +changes, so does the configMap name, along with all +references to that name that appear in the YAML output +from `kustomize`. + +The name change means deployments will do a rolling +restart to get new data if this YAML is applied to the +cluster using a command like + +> ``` +> kustomize build $OVERLAYS/development | kubectl apply -f - +> ``` + +A deployment has no means to automatically know when or +if a configMap in use by the deployment changes. + +If one changes a configMap without changing its name +and all references to that name, one must imperatively +restart the cluster to pick up the change. + +The best practice is to treat configMaps as immutable. + +Instead of editing configMaps, modify your declarative +specification of the cluster's desired state to +point deployments to _new_ configMaps with _new_ names. +`kustomize` makes this easy with its +`configMapGenerator` directive and associated naming +controls. A GC process in the k8s master eventually +deletes unused configMaps. + + +### Create and use the overlay for _production_ + +Next, create the files for the _production_ overlay: + + + +``` +mkdir -p $OVERLAYS/production + +cat <$OVERLAYS/production/plumbing.properties +port=8080 +EOF + +cat <$OVERLAYS/production/secret.properties +dbpassword=thisShouldProbablyBeInASecretInstead +EOF + +cat <$OVERLAYS/production/kustomization.yaml +bases: +- ../../base +namePrefix: prod- +configMapGenerator: +- name: my-configmap + behavior: merge + files: + - plumbing.properties + - secret.properties +EOF +``` + +One can now generate the configMaps for production: + + +``` +kustomize build $OVERLAYS/production +``` + +A CICD process could apply this directly to +the cluser using: + +> ``` +> kustomize build $OVERLAYS/production | kubectl apply -f - +> ``` diff --git a/examples/configGeneration.md b/examples/configGeneration.md index 114d727ae..40efd0329 100644 --- a/examples/configGeneration.md +++ b/examples/configGeneration.md @@ -1,298 +1,208 @@ -[overlay]: ../docs/glossary.md#overlay -[target]: ../docs/glossary.md#target +[patch]: ../../docs/glossary.md#patch +[resource]: ../../docs/glossary.md#resource +[variant]: ../../docs/glossary.md#variant -# Demo: combining config data from devops and developers +## ConfigMap generation and rolling updates -Scenario: you have a Java-based server storefront in -production that various internal development teams -(signups, checkout, search, etc.) contribute to. +Kustomize provides two ways of adding ConfigMap in one `kustomization`, either by declaring ConfigMap as a [resource] or declaring ConfigMap from a ConfigMapGenerator. The formats inside `kustomization.yaml` are -The server runs in different environments: -_development_, _testing_, _staging_ and _production_, -accepting configuration parameters from java property -files. +> ``` +> # declare ConfigMap as a resource +> resources: +> - configmap.yaml +> +> # declare ConfigMap from a ConfigMapGenerator +> configMapGenerator: +> - name: a-configmap +> files: +> - configs/configfile +> - configs/another_configfile +> ``` -Using one big properties file for each environment is -difficult to manage. The files change frequently, and -have to be changed by devops exclusively because +The ConfigMaps declared as [resource] are treated the same way as other resources. Kustomize doesn't append any hash to the ConfigMap name. The ConfigMap declared from a ConfigMapGenerator is treated differently. A hash is appended to the name and any change in the ConfigMap will trigger a rolling update. - 1. the files must at least partially agree on certain - values that devops cares about and that developers - ignore and - 1. because the production - properties contain sensitive data like production - database credentials. +In this demo, the same [hello_world](helloWorld/README.md) is used while the ConfigMap declared as [resources] is replaced by a ConfigMap declared from a ConfigmapGenerator. The change in this ConfigMap will result in a hash change and a rolling update. -## Property sharding +### Establish base and staging -With some study, we notice that the properties are -separable into categories. - -### Common properties - -E.g. internationalization data, static data like -physical constants, location of external services, etc. - -_Things that are the same regardless of environment._ - -Only one set of values is needed. - -Place them in a file called - - * `common.properties` - -(relative location defined below). - -### Plumbing properties - -E.g. serving location of static content (HTML, CSS, -javascript), location of product and customer database -tables, ports expected by load balancers, log sinks, -etc. - -_The different values for these properties are -precisely what sets the environments apart._ - -Devops or SRE will want full control over the values -used in production. Testing will have fixed -databases supporting testing. Developers will want -to do whatever they want to try scenarios under -development. - -Places these values in - - * `development/plumbing.properties` - * `staging/plumbing.properties` - * `production/plumbing.properties` - - -### Secret properties - -E.g. location of actual user tables, database -credentials, decryption keys, etc. - -_Things that are a subset of devops controls, that -nobody else has (or should want) access to._ - -Places these values in - - * `development/secret.properties` - * `staging/secret.properties` - * `production/secret.properties` - -[kubernetes secret]: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/ - -and control access to them with (for example) unix file -owner and mode bits, or better yet, put them in -a server dedicated to storing password protected -secrets, and use a field called `secretGenerator` -in your _kustomization_ to create a kubernetes -secret holding them (not covering that here). - - - -## A mixin approach to management - -The way to create _n_ cluster environments that share -some common information is to create _n_ overlays of a -common base. - -For the rest of this example, we'll do _n==2_, just -_development_ and _production_, since adding more -environments follows the same pattern. - -A cluster environment is created by -running `kustomize build` on a [target] that happens to -be an [overlay]. - -[helloworld]: helloWorld/README.md - -The following example will do that, but will focus on -configMap construction, and not worry about how to -connect the configMaps to deployments (that is covered -in the [helloworld] example). - - -All files - including the shared property files -discussed above - will be created in a directory tree -that is consistent with the base vs overlay file layout -defined in the [helloworld] demo. - -It will all live in this work directory: - - +Establish the base with a configMapGenerator + ``` DEMO_HOME=$(mktemp -d) -``` -### Create the base +BASE=$DEMO_HOME/base +mkdir -p $BASE - +curl -s -o "$BASE/#1.yaml" "https://raw.githubusercontent.com\ +/kubernetes-sigs/kustomize\ +/master/examples/helloWorld\ +/{deployment,service}.yaml" -Make a place to put the base configuration: - - -``` -mkdir -p $DEMO_HOME/base -``` - -Make the data for the base. This direction by -definition should hold resources common to all -environments. Here we're only defining a java -properties file, and a `kustomization` file that -references it. - - -``` -cat <$DEMO_HOME/base/common.properties -color=blue -height=10m -EOF - -cat <$DEMO_HOME/base/kustomization.yaml -configMapGenerator: -- name: my-configmap - files: - - common.properties +cat <<'EOF' >$BASE/kustomization.yaml +commonLabels: + app: hello +resources: +- deployment.yaml +- service.yaml +configMapGenerator: +- name: the-map + literals: + - altGreeting=Good Morning! + - enableRisky="false" EOF ``` - -### Create and use the overlay for _development_ - -Make an abbreviation for the parent of the overlay -directories: - - +Establish the staging with a patch applied to the ConfigMap + ``` OVERLAYS=$DEMO_HOME/overlays -``` +mkdir -p $OVERLAYS/staging -Create the files that define the _development_ overlay: - - -``` -mkdir -p $OVERLAYS/development - -cat <$OVERLAYS/development/plumbing.properties -port=30000 -EOF - -cat <$OVERLAYS/development/secret.properties -dbpassword=mothersMaidenName -EOF - -cat <$OVERLAYS/development/kustomization.yaml +cat <<'EOF' >$OVERLAYS/staging/kustomization.yaml +namePrefix: staging- +commonLabels: + variant: staging + org: acmeCorporation +commonAnnotations: + note: Hello, I am staging! bases: - ../../base -namePrefix: dev- -configMapGenerator: -- name: my-configmap - behavior: merge - files: - - plumbing.properties - - secret.properties +patches: +- map.yaml +EOF + +cat <$OVERLAYS/staging/map.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: the-map +data: + altGreeting: "Have a pineapple!" + enableRisky: "true" EOF ``` -One can now generate the configMaps for development: +### Review - +The _hello-world_ deployment running in this cluster is +configured with data from a configMap. + +The deployment refers to this map by name: + + + ``` -kustomize build $OVERLAYS/development +grep -C 2 configMapKeyRef $BASE/deployment.yaml ``` -#### Check the ConfigMap name +Changing the data held by a live configMap in a cluster +is considered bad practice. Deployments have no means +to know that the configMaps they refer to have +changed, so such updates have no effect. -The name of the generated `ConfigMap` is visible in this -output. +The recommended way to change a deployment's +configuration is to -The name should be something like `dev-my-configmap-b5m75ck895`: + 1. create a new configMap with a new name, + 1. patch the _deployment_, modifying the name value of + the appropriate `configMapKeyRef` field. - * `"dev-"` comes from the `namePrefix` field, - * `"my-configmap"` comes from the `configMapGenerator/name` field, - * `"-b5m75ck895"` comes from a deterministic hash that `kustomize` - computes from the contents of the configMap. +This latter change initiates rolling update to the pods +in the deployment. The older configMap, when no longer +referenced by any other resource, is eventually garbage +collected. -The hash suffix is critical. If the configMap content -changes, so does the configMap name, along with all -references to that name that appear in the YAML output -from `kustomize`. +### How this works with kustomize -The name change means deployments will do a rolling -restart to get new data if this YAML is applied to the -cluster using a command like +The _staging_ [variant] here has a configMap [patch]: -> ``` -> kustomize build $OVERLAYS/development | kubectl apply -f - -> ``` - -A deployment has no means to automatically know when or -if a configMap in use by the deployment changes. - -If one changes a configMap without changing its name -and all references to that name, one must imperatively -restart the cluster to pick up the change. - -The best practice is to treat configMaps as immutable. - -Instead of editing configMaps, modify your declarative -specification of the cluster's desired state to -point deployments to _new_ configMaps with _new_ names. -`kustomize` makes this easy with its -`configMapGenerator` directive and associated naming -controls. A GC process in the k8s master eventually -deletes unused configMaps. - - -### Create and use the overlay for _production_ - -Next, create the files for the _production_ overlay: - - - + ``` -mkdir -p $OVERLAYS/production - -cat <$OVERLAYS/production/plumbing.properties -port=8080 -EOF - -cat <$OVERLAYS/production/secret.properties -dbpassword=thisShouldProbablyBeInASecretInstead -EOF - -cat <$OVERLAYS/production/kustomization.yaml -bases: -- ../../base -namePrefix: prod- -configMapGenerator: -- name: my-configmap - behavior: merge - files: - - plumbing.properties - - secret.properties -EOF +cat $OVERLAYS/staging/map.yaml ``` -One can now generate the configMaps for production: +This patch is by definition a named but not necessarily +complete resource spec intended to modify a complete +resource spec. - +The ConfigMap it modifies is declared from a configMapGenerator. + + ``` -kustomize build $OVERLAYS/production +grep -C 4 configMapGenerator $BASE/kustomization.yaml ``` -A CICD process could apply this directly to -the cluser using: +For a patch to work, the names in the `metadata/name` +fields must match. -> ``` -> kustomize build $OVERLAYS/production | kubectl apply -f - -> ``` +However, the name values specified in the file are +_not_ what gets used in the cluster. By design, +kustomize modifies names of ConfigMaps declared from ConfigMapGenerator. To see the names +ultimately used in the cluster, just run kustomize: + + +``` +kustomize build $OVERLAYS/staging |\ + grep -B 8 -A 1 staging-the-map +``` + +The configMap name is prefixed by _staging-_, per the +`namePrefix` field in +`$OVERLAYS/staging/kustomization.yaml`. + +The suffix to the configMap name is generated from a +hash of the maps content - in this case the name suffix +is _hhhhkfmgmk_: + + +``` +kustomize build $OVERLAYS/staging | grep hhhhkfmgmk +``` + +Now modify the map patch, to change the greeting +the server will use: + + +``` +sed -i 's/pineapple/kiwi/' $OVERLAYS/staging/map.yaml +``` + +See the new greeting: + +``` +kustomize build $OVERLAYS/staging |\ + grep -B 2 -A 3 kiwi +``` + +Run kustomize again to see the new configMap names: + + +``` +kustomize build $OVERLAYS/staging |\ + grep -B 8 -A 1 staging-the-map +``` + +Confirm that the change in configMap content resulted +in three new names ending in _khk45ktkd9_ - one in the +configMap name itself, and two in the deployment that +uses the map: + + +``` +test 3 == \ + $(kustomize build $OVERLAYS/staging | grep khk45ktkd9 | wc -l); \ + echo $? +``` + +Applying these resources to the cluster will result in +a rolling update of the deployments pods, retargetting +them from the _hhhhkfmgmk_ maps to the _khk45ktkd9_ +maps. The system will later garbage collect the +unused maps. + +## Rollback + +To rollback, one would undo whatever edits were made to +the configuation in source control, then rerun kustomize +on the reverted configuration and apply it to the +cluster. diff --git a/examples/helloWorld/README.md b/examples/helloWorld/README.md index aea6f2325..390f90a30 100644 --- a/examples/helloWorld/README.md +++ b/examples/helloWorld/README.md @@ -315,130 +315,3 @@ To deploy, pipe the above commands to kubectl apply: > kustomize build $OVERLAYS/production |\ > kubectl apply -f - > ``` - -## Rolling updates - -### Review - -The _hello-world_ deployment running in this cluster is -configured with data from a configMap. - -The deployment refers to this map by name: - - - -``` -grep -C 2 configMapKeyRef $DEMO_HOME/base/deployment.yaml -``` - -Changing the data held by a live configMap in a cluster -is considered bad practice. Deployments have no means -to know that the configMaps they refer to have -changed, so such updates have no effect. - -The recommended way to change a deployment's -configuration is to - - 1. create a new configMap with a new name, - 1. patch the _deployment_, modifying the name value of - the appropriate `configMapKeyRef` field. - -This latter change initiates rolling update to the pods -in the deployment. The older configMap, when no longer -referenced by any other resource, is eventually garbage -collected. - -### How this works with kustomize - -The _staging_ [variant] here has a configMap [patch]: - - -``` -cat $OVERLAYS/staging/map.yaml -``` - -This patch is by definition a named but not necessarily -complete resource spec intended to modify a complete -resource spec. - -The resource it modifies is here: - - -``` -cat $DEMO_HOME/base/configMap.yaml -``` - -For a patch to work, the names in the `metadata/name` -fields must match. - -However, the name values specified in the file are -_not_ what gets used in the cluster. By design, -kustomize modifies these names. To see the names -ultimately used in the cluster, just run kustomize: - - -``` -kustomize build $OVERLAYS/staging |\ - grep -B 8 -A 1 staging-the-map -``` - -The configMap name is prefixed by _staging-_, per the -`namePrefix` field in -`$OVERLAYS/staging/kustomization.yaml`. - -The suffix to the configMap name is generated from a -hash of the maps content - in this case the name suffix -is _hhhhkfmgmk_: - - -``` -kustomize build $OVERLAYS/staging | grep hhhhkfmgmk -``` - -Now modify the map patch, to change the greeting -the server will use: - - -``` -sed -i 's/pineapple/kiwi/' $OVERLAYS/staging/map.yaml -``` - -See the new greeting: - -``` -kustomize build $OVERLAYS/staging |\ - grep -B 2 -A 3 kiwi -``` - -Run kustomize again to see the new configMap names: - - -``` -kustomize build $OVERLAYS/staging |\ - grep -B 8 -A 1 staging-the-map -``` - -Confirm that the change in configMap content resulted -in three new names ending in _khk45ktkd9_ - one in the -configMap name itself, and two in the deployment that -uses the map: - - -``` -test 3 == \ - $(kustomize build $OVERLAYS/staging | grep khk45ktkd9 | wc -l); \ - echo $? -``` - -Applying these resources to the cluster will result in -a rolling update of the deployments pods, retargetting -them from the _hhhhkfmgmk_ maps to the _khk45ktkd9_ -maps. The system will later garbage collect the -unused maps. - -## Rollback - -To rollback, one would undo whatever edits were made to -the configuation in source control, then rerun kustomize -on the reverted configuration and apply it to the -cluster. diff --git a/examples/helloWorld/kustomization.yaml b/examples/helloWorld/kustomization.yaml index c0cfd1bfb..41965951e 100644 --- a/examples/helloWorld/kustomization.yaml +++ b/examples/helloWorld/kustomization.yaml @@ -6,9 +6,4 @@ commonLabels: resources: - deployment.yaml - service.yaml - -configMapGenerator: -- name: the-map - literals: - - altGreeting="Good Morning!" - - enableRisky="false" +- configMap.yaml