Commit Graph

1778 Commits

Author SHA1 Message Date
Richard Marshall
24c173a49b Detect ID conflicts in namespace transformer 2019-08-19 08:55:54 -07:00
Jeff Regan
be1d5478dc Merge pull request #1450 from damienr74/master
Add internal tooling for kustomize
2019-08-16 14:49:37 -07:00
Damien Robichaud
d3022ccd65 rename to tools directory 2019-08-16 11:25:20 -07:00
Damien Robichaud
fe45157b26 Update crawler to cache web request form github.
- Increase logging signal to noise ratio.
- Allow to specify the `http.Client` for github requests. (This allows
 the use of caching http.Clients).
- Clean up implementation.
2019-08-16 09:48:23 -07:00
Damien Robichaud
df779fd720 Modify document for elasticsearch migration. 2019-08-16 09:48:23 -07:00
Damien Robichaud
e0d388c6f7 Implements search query partitioning by filesize.
Binary searches through different ranges of file sizes to create search
queries with fewer than 1000 results. This is required since github will
only return the first 1000 result of any search query.

The implementation handles the case where some files may be deleted
while the search is running, and (possibly artificially) assures that the
number of files increases monotonically as the filesize range grows.

The implementation also caches queries and is expected to make fewer
than O((#files/1000) * lg(max file size)) API calls to retrieve the
range queries that can be used to index all of the files.

In practice running the search splitting takes a few minutes, while
retrieving all of the data takes a few hours.
2019-08-16 09:48:23 -07:00
Damien Robichaud
62edcae233 Implementation of configurable github crawler.
Currently I've left the search splitting by file size out of this commit
since it's ~200 lines of logic, and I think it's best to get it reviewed
separately.

In it's current state the crawler would only be able to get the last
1000 indexed files by Github, but it does show the general structure of
how the crawler is implemented.
2019-08-16 09:48:23 -07:00
Damien Robichaud
ac6918d70f Implementation of github query helper library.
To make this easier to read, use, and modify, I've abstracted the
important parts of the github query api into crawler/github/query.go
which allows to describe at a high level what is to be searched without
knowing the API syntax.
2019-08-16 09:48:23 -07:00
Damien Robichaud
ca41674df3 Implementation of basic crawler organisation.
`crawler.Crawler` interface is defined, where a crawler has to implement
a `Crawl` method that forwards document found by the crawler to a channel.

A helper function that launches a list of crawlers concurrently and
merges their channels into one main output channel, forwarding errors is
also implemented.

Finally, a test that verifies the correctness and concurrency of the
helper method is provided.
2019-08-16 09:48:23 -07:00
Damien Robichaud
c02b4f3a11 Initial (temporary) implementation of search doc.
Document describing how to convert a kustomization file into a
searchable document on appengine (will be changed to elasticsearch)
soon.
2019-08-16 09:48:23 -07:00
Kubernetes Prow Robot
c464fb0a81 Merge pull request #1436 from richardmarshall/kubectl_clarity
docs: Additional details for kubectl integration
2019-08-13 15:18:23 -07:00
Kubernetes Prow Robot
694cf23df8 Merge pull request #1432 from richardmarshall/lostreplicas
Retain replicas field in edit marshal path
2019-08-09 14:29:14 -07:00
Richard Marshall
e66656aa7f docs: Additional details for kubectl integration 2019-08-08 17:06:19 -07:00
Richard Marshall
eaae7af5fe Retain replicas field in edit marshal path 2019-08-08 15:45:56 -07:00
Kubernetes Prow Robot
f9fe138114 Merge pull request #1416 from anthonyho007/makefile
add Makefile for local development
2019-08-07 11:04:09 -07:00
Kubernetes Prow Robot
2a2a889c37 Merge pull request #1423 from sunny0826/master
Update zh-README.md & zh-example-README.md
2019-08-02 12:37:55 -07:00
郭旭东
34287e511f fix example-zh-README.md 2019-08-02 09:09:32 +08:00
Anthony Ho
e6fffc8ba4 add makefile 2019-08-01 11:23:38 -04:00
郭旭东
86f221611e Update zh-example-README.md 2019-08-01 15:30:11 +08:00
郭旭东
b4d6e89fa2 Update zh-README.md 2019-08-01 15:19:14 +08:00
Kubernetes Prow Robot
5937bd0259 Merge pull request #1394 from richardmarshall/namerefperformance
Simplify name reference candidate resmap building
2019-07-31 10:18:14 -07:00
Richard Marshall
ed3c29be12 Simplify name reference candidate resmap building
This patch removes a layer of looping in the name reference candiate
resmap building process by not checking if the resources already exist
in the new resmap.
2019-07-30 17:15:15 -07:00
Kubernetes Prow Robot
3d2e956b19 Merge pull request #1412 from richardmarshall/anchor_resmap_select
Automatically anchor resource selector patterns
2019-07-30 15:47:11 -07:00
Kubernetes Prow Robot
dd9d1f95e9 Merge pull request #1389 from Liujingfang1/repospec
make repospec memebers public
2019-07-30 15:27:51 -07:00
jingfangliu
a279c08f7d make repospec memebers public 2019-07-30 13:56:20 -07:00
Kubernetes Prow Robot
a798109161 Merge pull request #1413 from bai/fix-typo
Fix typo in patches definition
2019-07-30 10:14:51 -07:00
Vlad Gorodetsky
bafd6b5423 Fix typo in patches definition 2019-07-30 14:52:02 +03:00
Richard Marshall
963913f9ef Automatically anchor resource selector patterns 2019-07-29 17:57:33 -07:00
Jingfang Liu
46905588ac add document for inline patch (#1411) 2019-07-29 15:15:06 -07:00
Kubernetes Prow Robot
5426888df4 Merge pull request #1405 from Liujingfang1/inlinepatch
add inline patch support for Strategic Merge Patch and JSON patch
2019-07-29 14:28:49 -07:00
jingfangliu
35481ec6d9 add inline patch support for Strategic Merge Patch and JSON patch 2019-07-29 14:10:57 -07:00
Kubernetes Prow Robot
6c92c30e94 Merge pull request #1402 from damienr74/currentid-replicas
Allow replicas to find modified names.
2019-07-29 12:54:47 -07:00
Damien Robichaud
02f6b3ec98 Allow replicas to find modified names.
Also allows to test for modified resmaps instead of directly loading
them.
2019-07-26 18:00:59 -07:00
Kubernetes Prow Robot
a9848f2738 Merge pull request #1403 from Liujingfang1/inlinepatch
add testing for patch transformers
2019-07-26 15:05:57 -07:00
jingfangliu
b4038a6cd2 add testting for patch transformers 2019-07-26 14:02:52 -07:00
Kubernetes Prow Robot
95f3303493 Merge pull request #1400 from keleustes/defaultsa
Force the namespace value for the "default" service object.
v3.1.0
2019-07-26 10:51:21 -07:00
Jerome Brette
2faf4a491b Force the namespace value for the "default" service object.
The clusterrolebinding and rolebinding is pointing at a resource
which is not listed in the kustomize
2019-07-25 22:43:59 +00:00
Kubernetes Prow Robot
e646bba1ff Merge pull request #1396 from Liujingfang1/delete
support strategic merge patch with $patch: delete
2019-07-25 12:24:45 -07:00
Kubernetes Prow Robot
99a21b0a3c Merge pull request #1308 from keleustes/varequal
Demonstrate need for Var.DeepEqual method equivalent
2019-07-25 11:26:06 -07:00
Kubernetes Prow Robot
e7a22b6bc5 Merge pull request #1398 from keleustes/doc
Update v3.1.0 release notes.
2019-07-25 10:43:28 -07:00
Jerome Brette
d783bbc0bc DeepEqual method seems cleaner than adding Defaulting before every
reflect.DeepEqual call
2019-07-25 03:52:25 +00:00
Jerome Brette
b7405f3872 Test new types.Var.DeepEqual method. 2019-07-25 03:50:28 +00:00
Jerome Brette
abc419b5f9 Add Absorb method to VarSet and DeepEqual to Var 2019-07-25 03:42:40 +00:00
Jerome Brette
336378b114 Update release notes 2019-07-25 00:12:51 +00:00
jingfangliu
29959551da release note for v3.1.0 2019-07-24 14:16:09 -07:00
jingfangliu
fc78917191 support strategic merge patch with $patch: delete 2019-07-24 12:46:33 -07:00
Kubernetes Prow Robot
ffd95ef5a9 Merge pull request #1378 from keleustes/nameprefix
Name Prefix and Suffix Transformation for multi level level kustomize context
2019-07-24 11:33:21 -07:00
Jerome Brette
230090d790 Fix namereference and stacked kustomization contexts (3/3)
- Update unit and integration tests.
2019-07-24 18:02:29 +00:00
Jerome Brette
8fa3861ba3 Fix namereference and stacked kustomization contexts (2/3)
- Leverage nameprefix and namesuffix contextual data
2019-07-24 18:02:23 +00:00
Jerome Brette
69c90e3427 Fix namereference and stacked kustomization contexts (1/3)
- Update PrefixSuffixTransfomer to add empty prefix and suffix
2019-07-24 10:59:07 -05:00