Merge pull request #925 from monopole/addGoPluginExample

Add goplugin KV generator example.

examples/README.md

@@ -28,14 +28,18 @@ go get sigs.k8s.io/kustomize
    (e.g. devops/SRE and developers).
    
  * [configGenerations](configGeneration.md) -
-   Rolling update when ConfigMapGenerator changes
+   Rolling update when ConfigMapGenerator changes.
 
- * [generatorOptions](generatorOptions.md) - Modifying behavior of all ConfigMap and Secret generators.  
+ * [secret generation](kvSourceGoPlugin.md) - Generating secrets.
+ 
+ * [generatorOptions](generatorOptions.md) -
+   Modifying behavior of all ConfigMap and Secret generators.
 
  * [breakfast](breakfast.md) - Customize breakfast for
    Alice and Bob.
    
- * [vars](wordpress/README.md) - Injecting k8s runtime data into container arguments (e.g. to point wordpress to a SQL service) by vars.
+ * [vars](wordpress/README.md) - Injecting k8s runtime data into
+    container arguments (e.g. to point wordpress to a SQL service) by vars.
  
  * [image names and tags](image.md) - Updating image names and tags without applying a patch.
 

examples/kvSourceGoPlugin.md

@@ -0,0 +1,279 @@
+[Secrets]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#secret-v1-core
+[ConfigMaps]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#configmap-v1-core
+[base64]: https://tools.ietf.org/html/rfc4648#section-4
+[Go plugin]: https://golang.org/pkg/plugin
+[v2.0.3]: https://github.com/kubernetes-sigs/kustomize/releases/tag/v2.0.3
+
+# Generating Secrets
+
+## What's a Secret?
+
+Kubernetes [ConfigMaps] and [Secrets] are both
+key:value (KV) maps, but the latter is intended to
+signal that its values have a sensitive nature -
+e.g. ssh keys or passwords.
+
+Kubernetes assumes that the values in a Secret are
+[base64] encoded, and decodes them before actual
+use (as, say, the argument to a container
+command).  The user that creates the Secret must
+base64 encode the data, or use a tool that does it
+for them.  This encoding doesn't protect the
+secret from anything other than an
+over-the-shoulder glance.
+
+Protecting the actual secrecy of a Secret value is
+up to the cluster operator. They must lock down
+the cluster (and its `etcd` data store) as tightly
+as desired, and likewise protect the bytes that
+feed into the cluster to ultimately become the
+content of a Secret value.
+
+## Make a place to work
+
+<!-- @establishBase @test -->
+```
+DEMO_HOME=$(mktemp -d)
+```
+
+## Secret values from local files
+
+kustomize has three different ways to generate a secret
+from local files:
+
+ * get them from so-called _env_ files (`NAME=VALUE`, one per line),
+ * consume the entire contents of a file to make one secret value,
+ * get literal values from the kustomization file itself.
+ 
+Here's an example combining all three methods:
+
+Make an env file with some short secrets:
+
+<!-- @makeEnvFile @test -->
+```
+cat <<'EOF' >$DEMO_HOME/foo.env
+ROUTER_PASSWORD=admin
+DB_PASSWORD=iloveyou
+EOF
+```
+
+Make a text file with a long secret:
+
+<!-- @makeLongSecretFile @test -->
+```
+cat <<'EOF' >$DEMO_HOME/longsecret.txt
+Lorem ipsum dolor sit amet,
+consectetur adipiscing elit,
+sed do eiusmod tempor incididunt
+ut labore et dolore magna aliqua.
+EOF
+```
+
+And make a kustomization file referring to the
+above and additionally defining some literal KV
+pairs:
+
+<!-- @makeKustomization1 @test -->
+```
+cat <<'EOF' >$DEMO_HOME/kustomization.yaml
+secretGenerator:
+- name: mysecrets
+  kvSources:
+  - name: envfiles
+    pluginType: builtin
+    args:
+    - foo.env
+  - name: files
+    pluginType: builtin 
+    args:
+    - longsecret.txt
+  - name: literals
+    pluginType: builtin
+    args:
+    - FRUIT=apple
+    - VEGETABLE=carrot
+EOF
+```
+
+> The above syntax is _alpha_ behavior at HEAD, for v2.1+.
+>
+> The default value of `pluginType` is `builtin`, so the
+> `pluginType` fields could be omitted.
+>
+> The equivalent [v2.0.3] syntax (still supported) is
+> ```
+> secretGenerator:
+> - name: mysecrets
+>   env: foo.env
+>   files:
+>   - longsecret.txt
+>   literals:
+>   - FRUIT=apple
+>   - VEGETABLE=carrot
+> ```
+
+Now generate the Secret:
+
+<!-- @build1 @test -->
+```
+result=$(kustomize build $DEMO_HOME)
+echo "$result"
+test 1 == $(echo "$result" | grep -c "FRUIT: YXBwbGU=")
+```
+
+This emits something like
+
+> ```
+> apiVersion: v1
+> kind: Secret
+> metadata:
+>   name: mysecrets-hfb5df789h
+> type: Opaque
+> data:
+>   FRUIT: YXBwbGU=
+>   VEGETABLE: Y2Fycm90
+>   ROUTER_PASSWORD: YWRtaW4=
+>   DB_PASSWORD: aWxvdmV5b3U=
+>   longsecret.txt: TG9yZW0gaXBzdW0gZG9sb3Igc2l0I... (elided)
+> ```
+
+The name of the resource is a prefix, `mysecrets`
+(as specfied in the kustomization file), followed
+by a hash of its contents.
+
+Use your favorite base64 decoder to confirm the raw
+versions of any of these values.
+
+The problem that these three approaches share is
+that the purported secrets must live on disk.
+
+This adds additional security questions - who can
+see the files, who installs them, who deletes
+them, etc.
+
+
+## Secret values from anywhere
+
+> New _alpha_ behavior at HEAD, for v2.1+
+
+A general alternative is to enshrine secret
+value generation in a Go plugin.
+
+The values can then come in via, say, an
+authenticated and authorized RPC to a password
+vault service.
+
+Here's a trivial plugin that provides
+hardcoded values:
+
+<!-- @makePlugin @test -->
+```
+cat <<'EOF' >$DEMO_HOME/kvMaker.go
+package main
+var database = map[string]string{
+  "TREE":      "oak",
+  "ROCKET":    "Saturn V",
+  "FRUIT":     "apple",
+  "VEGETABLE": "carrot",
+  "SIMPSON":   "homer",
+}
+
+type plugin struct{}
+var KVSource plugin
+func (p plugin) Get(root string, args []string) (map[string]string, error) {
+  r := make(map[string]string)
+  for _, k := range args {
+    v, ok := database[k]
+    if ok {
+      r[k] = v
+    }
+  }
+  return r, nil
+}
+EOF
+```
+
+
+The two crucial items needed to
+load and query the plugin are
+ 1) the public symbol `KVSource`,
+ 1) its public `Get` method and signature.
+
+Plugins that generate KV pairs for kustomize
+must be installed at
+
+> ```
+> $XDG_CONFIG_HOME/kustomize/plugins/kvSource
+> ```
+
+`XDG_CONFIG_HOME` is an environment variable
+used by many programs as the root of a
+configuration directory.  If unspecified, the
+default `$HOME/.config` is used.  The rest of
+the required directory path establishes that
+the files found there are kustomize plugins
+for generating KV pairs.
+
+Compile and install the plugin:
+
+<!-- @compilePlugin @test -->
+```
+kvSources=$DEMO_HOME/kustomize/plugins/kvSources
+mkdir -p $kvSources
+GOPATH=$DEMO_HOME:$GOPATH go build \
+  -buildmode plugin \
+  -o $kvSources/kvMaker.so \
+  $DEMO_HOME/kvMaker.go
+```
+
+Create a new kustomization file
+referencing this plugin:
+
+<!-- @makeKustomization2 @test -->
+```
+cat <<'EOF' >$DEMO_HOME/kustomization.yaml
+secretGenerator:
+- name: mysecrets
+  kvSources:
+  - name: kvMaker
+    pluginType: go
+    args:
+    - FRUIT
+    - VEGETABLE
+EOF
+```
+
+Finally, generate the secret, setting
+`XDG_CONFIG_HOME` appropriately:
+
+<!-- @build2 @test -->
+```
+result=$(XDG_CONFIG_HOME=$DEMO_HOME kustomize \
+  --enable_alpha_goplugins_accept_panic_risk \
+  build $DEMO_HOME)
+echo "$result"
+test 1 == $(echo "$result" | grep -c "FRUIT: YXBwbGU=")
+```
+
+Specify the `--enable_...` flag to enable Go
+plugins, which may fail if not compiled under
+the same conditions as the main program.  Try
+this command without the flag to see more
+explanation.
+
+This should emit something like:
+
+> ```
+> apiVersion: v1
+> kind: Secret
+> metadata:
+>   name: mysecrets-bdt27dbkd6
+> type: Opaque
+> data:
+>  FRUIT: YXBwbGU=
+>  VEGETABLE: Y2Fycm90
+> ```
+
+i.e. a subset of the same values as above.
+
+

k8sdeps/configmapandsecret/basefactory.go

@@ -91,7 +91,7 @@ func keyValuesFromLiteralSources(sources []string) ([]kv.Pair, error) {
 }
 
 func (bf baseFactory) keyValuesFromPlugins(sources []types.KVSource) ([]kv.Pair, error) {
-	var allKvs []kv.Pair
+	var result []kv.Pair
 	for _, s := range sources {
 		plug, err := bf.reg.Load(s.PluginType, s.Name)
 		if err != nil {
@@ -101,9 +101,11 @@ func (bf baseFactory) keyValuesFromPlugins(sources []types.KVSource) ([]kv.Pair,
 		if err != nil {
 			return nil, err
 		}
-		allKvs = append(allKvs, kvs...)
+		for k, v := range kvs {
+			result = append(result, kv.Pair{Key: k, Value: v})
+		}
 	}
-	return allKvs, nil
+	return result, nil
 }
 
 func (bf baseFactory) keyValuesFromFileSources(sources []string) ([]kv.Pair, error) {

k8sdeps/kv/plugin/builtin/envfiles.go

@@ -21,16 +21,16 @@ import (
 	"sigs.k8s.io/kustomize/pkg/ifc"
 )
 
-// Envfiles format should be a path to a file to read lines of key=val
+// EnvFiles format should be a path to a file to read lines of key=val
 // pairs to create a configmap.
 // i.e. a Docker .env file or a .ini file.
-type Envfiles struct {
+type EnvFiles struct {
 	Ldr ifc.Loader
 }
 
 // Get implements the interface for kv plugins.
-func (p Envfiles) Get(root string, args []string) ([]kv.Pair, error) {
+func (p EnvFiles) Get(root string, args []string) (map[string]string, error) {
-	var all []kv.Pair
+	all := make(map[string]string)
 	for _, path := range args {
 		if path == "" {
 			return nil, nil
@@ -43,7 +43,9 @@ func (p Envfiles) Get(root string, args []string) ([]kv.Pair, error) {
 		if err != nil {
 			return nil, err
 		}
-		all = append(all, kvs...)
+		for _, pair := range kvs {
+			all[pair.Key] = pair.Value
+		}
 	}
 	return all, nil
 }

k8sdeps/kv/plugin/builtin/files.go

@@ -32,8 +32,8 @@ type Files struct {
 }
 
 // Get implements the interface for kv plugins.
-func (p Files) Get(root string, args []string) ([]kv.Pair, error) {
+func (p Files) Get(root string, args []string) (map[string]string, error) {
-	var kvs []kv.Pair
+	kvs := make(map[string]string)
 	for _, s := range args {
 		k, fPath, err := kv.ParseFileSource(s)
 		if err != nil {
@@ -43,7 +43,7 @@ func (p Files) Get(root string, args []string) ([]kv.Pair, error) {
 		if err != nil {
 			return nil, err
 		}
-		kvs = append(kvs, kv.Pair{Key: k, Value: string(content)})
+		kvs[k] = string(content)
 	}
 	return kvs, nil
 }

k8sdeps/kv/plugin/builtin/literals.go

@@ -26,14 +26,14 @@ import (
 type Literals struct{}
 
 // Get implements the interface for kv plugins.
-func (p Literals) Get(root string, args []string) ([]kv.Pair, error) {
+func (p Literals) Get(root string, args []string) (map[string]string, error) {
-	var kvs []kv.Pair
+	kvs := make(map[string]string)
 	for _, s := range args {
 		k, v, err := kv.ParseLiteralSource(s)
 		if err != nil {
 			return nil, err
 		}
-		kvs = append(kvs, kv.Pair{Key: k, Value: v})
+		kvs[k] = v
 	}
 	return kvs, nil
 }

k8sdeps/kv/plugin/builtinplugin.go

@@ -34,7 +34,7 @@ func newBuiltinFactory(ldr ifc.Loader) *builtinFactory {
 		plugins: map[string]KVSource{
 			"literals": builtin.Literals{},
 			"files":    builtin.Files{Ldr: ldr},
-			"envfiles": builtin.Envfiles{Ldr: ldr},
+			"envfiles": builtin.EnvFiles{Ldr: ldr},
 		},
 	}
 }

k8sdeps/kv/plugin/goplugin.go

@@ -29,23 +29,20 @@ var _ Factory = &goFactory{}
 const (
 	kvSourcesDir            = "kvSources"
 	EnableGoPluginsFlagName = "enable_alpha_goplugins_accept_panic_risk"
-	EnableGoPluginsFlagHelp = `
+	EnableGoPluginsFlagHelp = `The main program may panic and exit on an attempt
-Warning: the main program may panic and exit on an
+to use a goplugin that was compiled under conditions
-attempt to use a goplugin that was compiled under
+differing from the those in effect when main was
-conditions differing from the those in effect when
+compiled. It's safest to use this flag in the
-main was compiled. It's safest to use this flag in
+context of a container image holding both the main
-the context of a container image holding both the
+and the goplugins it needs, all built on the same
-main and the goplugins it needs, all built on the
+machine, with the same transitive libs and the same
-same machine, with the same transitive libs and
+compiler version.`
-the same compiler version.
-`
 	errorFmt = `
 enable go plugins by specifying flag
   --%s
 Place .so files in
   %s
-%s
+%s`
-`
 )
 
 func newGoFactory(c *types.PluginConfig) *goFactory {
@@ -82,7 +79,7 @@ func (p *goFactory) load(name string) (KVSource, error) {
 		return nil, err
 	}
 
-	symbol, err := goPlugin.Lookup("Plugin")
+	symbol, err := goPlugin.Lookup("KVSource")
 	if err != nil {
 		return nil, err
 	}

k8sdeps/kv/plugin/plugin.go

@@ -17,13 +17,9 @@ limitations under the License.
 // Package plugin provides a plugin abstraction layer.
 package plugin
 
-import (
-	"sigs.k8s.io/kustomize/k8sdeps/kv"
-)
-
 // KVSource is the interface for kv source plugins.
 type KVSource interface {
-	Get(root string, args []string) ([]kv.Pair, error)
+	Get(root string, args []string) (map[string]string, error)
 }
 
 // Factory is the interface for new kv source plugin implementations.

pkg/commands/commands.go

@@ -55,12 +55,12 @@ See https://sigs.k8s.io/kustomize
 		PluginConfig: plugin.DefaultPluginConfig(),
 	}
 
-	c.Flags().BoolVar(
+	c.PersistentFlags().BoolVar(
 		&genMetaArgs.PluginConfig.GoEnabled,
 		plugin.EnableGoPluginsFlagName,
 		false, plugin.EnableGoPluginsFlagHelp)
 	// Not advertising this alpha feature.
-	c.Flags().MarkHidden(plugin.EnableGoPluginsFlagName)
+	c.PersistentFlags().MarkHidden(plugin.EnableGoPluginsFlagName)
 
 	uf := kunstruct.NewKunstructuredFactoryWithGeneratorArgs(&genMetaArgs)
 

pkg/pgmconfig/config.go

@@ -23,6 +23,7 @@ import (
 	"runtime"
 )
 
+//noinspection GoSnakeCaseUsage
 const (
 	XDG_CONFIG_HOME     = "XDG_CONFIG_HOME"
 	defaultConfigSubdir = ".config"