1943 Commits

Author SHA1 Message Date
Damien Robichaud
df779fd720 Modify document for elasticsearch migration. 2019-08-16 09:48:23 -07:00
Damien Robichaud
e0d388c6f7 Implements search query partitioning by filesize.
Binary searches through different ranges of file sizes to create search
queries with fewer than 1000 results. This is required since github will
only return the first 1000 result of any search query.

The implementation handles the case where some files may be deleted
while the search is running, and (possibly artificially) assures that the
number of files increases monotonically as the filesize range grows.

The implementation also caches queries and is expected to make fewer
than O((#files/1000) * lg(max file size)) API calls to retrieve the
range queries that can be used to index all of the files.

In practice running the search splitting takes a few minutes, while
retrieving all of the data takes a few hours.
2019-08-16 09:48:23 -07:00
Damien Robichaud
62edcae233 Implementation of configurable github crawler.
Currently I've left the search splitting by file size out of this commit
since it's ~200 lines of logic, and I think it's best to get it reviewed
separately.

In it's current state the crawler would only be able to get the last
1000 indexed files by Github, but it does show the general structure of
how the crawler is implemented.
2019-08-16 09:48:23 -07:00
Damien Robichaud
ac6918d70f Implementation of github query helper library.
To make this easier to read, use, and modify, I've abstracted the
important parts of the github query api into crawler/github/query.go
which allows to describe at a high level what is to be searched without
knowing the API syntax.
2019-08-16 09:48:23 -07:00
Damien Robichaud
ca41674df3 Implementation of basic crawler organisation.
`crawler.Crawler` interface is defined, where a crawler has to implement
a `Crawl` method that forwards document found by the crawler to a channel.

A helper function that launches a list of crawlers concurrently and
merges their channels into one main output channel, forwarding errors is
also implemented.

Finally, a test that verifies the correctness and concurrency of the
helper method is provided.
2019-08-16 09:48:23 -07:00
Damien Robichaud
c02b4f3a11 Initial (temporary) implementation of search doc.
Document describing how to convert a kustomization file into a
searchable document on appengine (will be changed to elasticsearch)
soon.
2019-08-16 09:48:23 -07:00
Liviu Costea
64341a81fa Add short version flag 2019-08-16 09:46:37 +03:00
Richard Marshall
fe8ba8e44b Log loader errors during resource accumulation 2019-08-15 07:59:55 -07:00
Richard Marshall
54f1952195 Log output from git on errors 2019-08-15 07:34:38 -07:00
Richard Marshall
44b62a8ebc Fix indirect git resource cycle detection 2019-08-15 07:25:20 -07:00
Jordan Liggitt
8e9c08ea61 Fix patch path example 2019-08-14 15:55:55 -04:00
Kubernetes Prow Robot
c464fb0a81 Merge pull request #1436 from richardmarshall/kubectl_clarity
docs: Additional details for kubectl integration
2019-08-13 15:18:23 -07:00
Richard Marshall
9481e3fba6 docs: Update patchesStrategicMerge documentation 2019-08-13 13:38:42 -07:00
Richard Marshall
0e5206a251 test: Update target tests for SMP directives 2019-08-13 13:38:42 -07:00
Richard Marshall
96c5b4aa3e Handle ordering patches with SMP delete directives
This change enables the SMP patch merging process to support delete
directives in patches regardless of input order.
2019-08-13 13:38:41 -07:00
Jan Guth
6c44da52a2 add PriorityClass to the order list 2019-08-10 07:31:54 +02:00
Kubernetes Prow Robot
694cf23df8 Merge pull request #1432 from richardmarshall/lostreplicas
Retain replicas field in edit marshal path
2019-08-09 14:29:14 -07:00
Richard Marshall
e66656aa7f docs: Additional details for kubectl integration 2019-08-08 17:06:19 -07:00
Richard Marshall
eaae7af5fe Retain replicas field in edit marshal path 2019-08-08 15:45:56 -07:00
Luka Skugor
2de052ecd8 Download submodules when using base from git 2019-08-08 15:49:56 +02:00
guoxudong
6cf8b9e2b8 update examples-zh 2019-08-08 10:16:13 +08:00
Kubernetes Prow Robot
f9fe138114 Merge pull request #1416 from anthonyho007/makefile
add Makefile for local development
2019-08-07 11:04:09 -07:00
Vincent C
78c9729252 translate-zh: glossary.md 2019-08-03 20:40:58 +08:00
Kubernetes Prow Robot
2a2a889c37 Merge pull request #1423 from sunny0826/master
Update zh-README.md & zh-example-README.md
2019-08-02 12:37:55 -07:00
郭旭东
34287e511f fix example-zh-README.md 2019-08-02 09:09:32 +08:00
Anthony Ho
e6fffc8ba4 add makefile 2019-08-01 11:23:38 -04:00
郭旭东
86f221611e Update zh-example-README.md 2019-08-01 15:30:11 +08:00
郭旭东
b4d6e89fa2 Update zh-README.md 2019-08-01 15:19:14 +08:00
Richard Marshall
adbb6228a5 Handle git:: prefix in urls containing _git 2019-07-31 10:23:05 -07:00
Kubernetes Prow Robot
5937bd0259 Merge pull request #1394 from richardmarshall/namerefperformance
Simplify name reference candidate resmap building
2019-07-31 10:18:14 -07:00
Richard Marshall
eeafd43513 Remove import of k8sdeps from create command 2019-07-30 20:52:06 -07:00
Richard Marshall
a68f95b65f Rename commands utility function file 2019-07-30 20:50:52 -07:00
Richard Marshall
ed3c29be12 Simplify name reference candidate resmap building
This patch removes a layer of looping in the name reference candiate
resmap building process by not checking if the resources already exist
in the new resmap.
2019-07-30 17:15:15 -07:00
Kubernetes Prow Robot
3d2e956b19 Merge pull request #1412 from richardmarshall/anchor_resmap_select
Automatically anchor resource selector patterns
2019-07-30 15:47:11 -07:00
Kubernetes Prow Robot
dd9d1f95e9 Merge pull request #1389 from Liujingfang1/repospec
make repospec memebers public
2019-07-30 15:27:51 -07:00
jingfangliu
a279c08f7d make repospec memebers public 2019-07-30 13:56:20 -07:00
Kubernetes Prow Robot
a798109161 Merge pull request #1413 from bai/fix-typo
Fix typo in patches definition
2019-07-30 10:14:51 -07:00
Richard Marshall
5dfa929906 Add create subcommand 2019-07-30 09:17:29 -07:00
Richard Marshall
e904f612f3 Move commands/edit utils package up to commands 2019-07-30 08:45:06 -07:00
Vlad Gorodetsky
bafd6b5423 Fix typo in patches definition 2019-07-30 14:52:02 +03:00
Richard Marshall
963913f9ef Automatically anchor resource selector patterns 2019-07-29 17:57:33 -07:00
Jingfang Liu
46905588ac add document for inline patch (#1411) 2019-07-29 15:15:06 -07:00
Kubernetes Prow Robot
5426888df4 Merge pull request #1405 from Liujingfang1/inlinepatch
add inline patch support for Strategic Merge Patch and JSON patch
2019-07-29 14:28:49 -07:00
jingfangliu
35481ec6d9 add inline patch support for Strategic Merge Patch and JSON patch 2019-07-29 14:10:57 -07:00
Kubernetes Prow Robot
6c92c30e94 Merge pull request #1402 from damienr74/currentid-replicas
Allow replicas to find modified names.
2019-07-29 12:54:47 -07:00
Damien Robichaud
02f6b3ec98 Allow replicas to find modified names.
Also allows to test for modified resmaps instead of directly loading
them.
2019-07-26 18:00:59 -07:00
Kubernetes Prow Robot
a9848f2738 Merge pull request #1403 from Liujingfang1/inlinepatch
add testing for patch transformers
2019-07-26 15:05:57 -07:00
jingfangliu
b4038a6cd2 add testting for patch transformers 2019-07-26 14:02:52 -07:00
Kubernetes Prow Robot
95f3303493 Merge pull request #1400 from keleustes/defaultsa
Force the namespace value for the "default" service object.
v3.1.0
2019-07-26 10:51:21 -07:00
Jerome Brette
2faf4a491b Force the namespace value for the "default" service object.
The clusterrolebinding and rolebinding is pointing at a resource
which is not listed in the kustomize
2019-07-25 22:43:59 +00:00