Commit Graph

153 Commits

Author SHA1 Message Date
Phillip Wittrock
b1f514632a Switch to api version 0.3.1 2020-01-07 08:54:05 -08:00
Haiyan Meng
be2e03681d Remove unused param from IndexFunc 2019-12-18 15:56:44 -08:00
Haiyan Meng
127541f610 Support diffrent modes of running the crawler 2019-12-18 15:56:44 -08:00
Haiyan Meng
f5ff254203 Update deps 2019-12-18 15:56:44 -08:00
Haiyan Meng
a35f002139 Run goimports 2019-12-18 15:56:44 -08:00
Haiyan Meng
bef157d6b3 Fix insert/updating document logic 2019-12-18 15:56:44 -08:00
Haiyan Meng
2c2aa928cc Delete non-existing documents from the index 2019-12-18 15:56:44 -08:00
Haiyan Meng
1eb713157c Sort the string slice fields of a document to avoid updating the index
unnecessarily
2019-12-18 15:56:44 -08:00
Haiyan Meng
272b7a6fcd Use UpdateRequest to insert/update a document
Currently, `IndexRequest` is used to insert/update a document, which
increases the version of the document every time IndexRequest.Do is
called.
2019-12-18 15:56:44 -08:00
Haiyan Meng
5598d35e4b Add a summary for doCrawl 2019-12-18 15:56:44 -08:00
Haiyan Meng
8c89f0946c Avoid to index a document if FetchDcoument or SetCreated fails 2019-12-18 15:56:44 -08:00
Haiyan Meng
12fc8f41c7 Add support for github paths starting with "git@github.com:" 2019-12-18 15:56:44 -08:00
Haiyan Meng
e44d1298df Return errors if http Client.Do resp status code is not 2xx 2019-12-18 15:56:44 -08:00
Ilia Zlobin
cc8b100331 Handle variables in annotations 2019-12-18 17:04:43 +03:00
Phillip Wittrock
de824c2e4d Drop mdrip dependency from api/ because it has conflicting deps with kubectl 2019-12-17 13:48:29 -08:00
Jeffrey Regan
90597d56c9 Update go.sums 2019-12-16 11:37:35 -08:00
Jeffrey Regan
7e205b46b8 Update serialize-javascript 2019-12-13 14:45:43 -08:00
Jeff Regan
c6c099a9d1 Merge pull request #1948 from haiyanmeng/expose-es
Add supports for crawling a specific git user or repo
2019-12-13 13:24:03 -08:00
Haiyan Meng
a9244f759e Add supports for crawling a specific git user or repo 2019-12-13 11:18:33 -08:00
jregan
b8a13b6335 Pin to kustomize API v0.3.0 2019-12-12 18:41:05 -08:00
Jeffrey Regan
87cf1231c3 Drop api module's explicit dependence on pluginator. 2019-12-12 16:36:57 -08:00
Jeffrey Regan
bb03507d98 Go mod tidy b4 release. 2019-12-12 14:24:06 -08:00
Jeff Regan
7cdb157a55 Update execplugin_test.go 2019-12-12 13:57:00 -08:00
Haiyan Meng
50ce2a66a3 Separate the two types of crawling
1) crawling the documents in the index to update these documents;
2) crawling the whole github.
2019-12-12 13:42:07 -08:00
Jeff Regan
dae03a9618 Merge branch 'master' into fix-process-args-fields 2019-12-12 13:41:50 -08:00
Jeff Regan
6dd48142cf Update execplugin_test.go 2019-12-12 13:41:08 -08:00
Jeffrey Regan
61c5afdf83 Update npm deps. 2019-12-12 13:18:30 -08:00
Phillip Wittrock
7e847dc907 publish apply / diff as alpha commands 2019-12-12 10:28:10 -08:00
Phillip Wittrock
1a28e493cd Fix client-go version to match apimachinery version 2019-12-12 10:22:12 -08:00
Haiyan Meng
d9239104aa Escape spaces in the query paths of git commit requests 2019-12-12 10:03:15 -08:00
Jeffrey Regan
ee678d6ce9 Upgrade to mdrip 1.0.1 2019-12-12 09:26:11 -08:00
Kubernetes Prow Robot
e1205698e6 Merge pull request #1933 from haiyanmeng/expose-es
Expose ElasticSearch as a LoadBalancer-type service
2019-12-11 16:16:32 -08:00
Kubernetes Prow Robot
a6535c4be2 Merge pull request #1934 from pwittrock/unk8s
revert pseudo/k8s deps
2019-12-11 16:08:32 -08:00
Phillip Wittrock
c2dc7ae789 revert pseudo/k8s deps 2019-12-11 15:28:04 -08:00
Haiyan Meng
afd24c6faf Expose ElasticSearch as a LoadBalancer-type service 2019-12-11 15:05:10 -08:00
Haiyan Meng
0d79219e46 Avoid processing the nil pointer returned by kustomizationResultAdapter
Currently, the crawler job panics whenever a nil pointer is returned by
kustomizationResultAdapter.
2019-12-11 13:54:01 -08:00
Haiyan Meng
bffc0d7071 Mulitple improvements of the crawler
1) Set document IDs to avoid duplicating documents;
2) Set the `creationTime` field of each document in the index;
3) set the `values`, `kinds` and `identifiers` fields for all documents;
4) Add a `Copy` method into the `Document` struct: this fixes the issue
where all the documents existing in the index point to the same Document
object;
5) Avoid using keystore redis;
6) Set imagePullPolicy to `Always` for crawler jobs.
2019-12-11 11:10:48 -08:00
bzub
105a8e57c8 Detect path to plugins from krusty tests. 2019-12-07 13:16:15 -06:00
lalyos
95ad5d05b3 fix exec plugin args handling 2019-12-06 13:34:42 +01:00
Jeff Regan
969a1e8f37 Merge pull request #1892 from monopole/getRidOfFakeLoader
Get rid of the 'fake' loader.
2019-12-05 11:22:47 -08:00
Jeff Regan
184735fa89 Merge pull request #1887 from Liujingfang1/replacement-poc
add replacement type and an example transformer
2019-12-05 11:10:35 -08:00
Jeffrey Regan
caa71a73fe Get rid of the fake loader. 2019-12-05 10:50:42 -08:00
Haiyan Meng
d25b6ff3dc Remove duplicates in kinds 2019-12-04 11:24:54 -08:00
Haiyan Meng
68a196dbe5 Add a test case to demonstrate kinds have duplicates 2019-12-04 11:24:23 -08:00
Jingfang Liu
c7600bc079 add replacement type and an example transformer 2019-12-04 11:13:05 -08:00
Haiyan Meng
8aaa3f56f5 Set the ElasticSearch index creation configuration
Currently, the `kustomize` index in ElasticSearch is using dynamic
mapping, which sets the types of all the fields to `text`. However,
`text` field type is good for full-text value matching, and not good for
exact-value matching.  For exact-value matching, the `keyword` filed
type should be used.
2019-12-03 15:16:32 -08:00
Jeffrey Regan
e9ab3da164 Fix some nits in the crawler and elsewhere. 2019-12-03 10:44:44 -08:00
Jeff Regan
fd5db20a48 Update harness.go 2019-12-02 18:47:07 -08:00
Jeffrey Regan
861c86a70a Simplify plugin tests. 2019-12-02 17:12:59 -08:00
Jeffrey Regan
382c330f5b Consolidate test harness to one package. 2019-12-02 12:29:10 -08:00