Phillip Wittrock
b1f514632a
Switch to api version 0.3.1
2020-01-07 08:54:05 -08:00
Haiyan Meng
be2e03681d
Remove unused param from IndexFunc
2019-12-18 15:56:44 -08:00
Haiyan Meng
127541f610
Support diffrent modes of running the crawler
2019-12-18 15:56:44 -08:00
Haiyan Meng
f5ff254203
Update deps
2019-12-18 15:56:44 -08:00
Haiyan Meng
a35f002139
Run goimports
2019-12-18 15:56:44 -08:00
Haiyan Meng
bef157d6b3
Fix insert/updating document logic
2019-12-18 15:56:44 -08:00
Haiyan Meng
2c2aa928cc
Delete non-existing documents from the index
2019-12-18 15:56:44 -08:00
Haiyan Meng
1eb713157c
Sort the string slice fields of a document to avoid updating the index
...
unnecessarily
2019-12-18 15:56:44 -08:00
Haiyan Meng
272b7a6fcd
Use UpdateRequest to insert/update a document
...
Currently, `IndexRequest` is used to insert/update a document, which
increases the version of the document every time IndexRequest.Do is
called.
2019-12-18 15:56:44 -08:00
Haiyan Meng
5598d35e4b
Add a summary for doCrawl
2019-12-18 15:56:44 -08:00
Haiyan Meng
8c89f0946c
Avoid to index a document if FetchDcoument or SetCreated fails
2019-12-18 15:56:44 -08:00
Haiyan Meng
12fc8f41c7
Add support for github paths starting with "git@github.com:"
2019-12-18 15:56:44 -08:00
Haiyan Meng
e44d1298df
Return errors if http Client.Do resp status code is not 2xx
2019-12-18 15:56:44 -08:00
Ilia Zlobin
cc8b100331
Handle variables in annotations
2019-12-18 17:04:43 +03:00
Phillip Wittrock
de824c2e4d
Drop mdrip dependency from api/ because it has conflicting deps with kubectl
2019-12-17 13:48:29 -08:00
Jeffrey Regan
90597d56c9
Update go.sums
2019-12-16 11:37:35 -08:00
Jeffrey Regan
7e205b46b8
Update serialize-javascript
2019-12-13 14:45:43 -08:00
Jeff Regan
c6c099a9d1
Merge pull request #1948 from haiyanmeng/expose-es
...
Add supports for crawling a specific git user or repo
2019-12-13 13:24:03 -08:00
Haiyan Meng
a9244f759e
Add supports for crawling a specific git user or repo
2019-12-13 11:18:33 -08:00
jregan
b8a13b6335
Pin to kustomize API v0.3.0
2019-12-12 18:41:05 -08:00
Jeffrey Regan
87cf1231c3
Drop api module's explicit dependence on pluginator.
2019-12-12 16:36:57 -08:00
Jeffrey Regan
bb03507d98
Go mod tidy b4 release.
2019-12-12 14:24:06 -08:00
Jeff Regan
7cdb157a55
Update execplugin_test.go
2019-12-12 13:57:00 -08:00
Haiyan Meng
50ce2a66a3
Separate the two types of crawling
...
1) crawling the documents in the index to update these documents;
2) crawling the whole github.
2019-12-12 13:42:07 -08:00
Jeff Regan
dae03a9618
Merge branch 'master' into fix-process-args-fields
2019-12-12 13:41:50 -08:00
Jeff Regan
6dd48142cf
Update execplugin_test.go
2019-12-12 13:41:08 -08:00
Jeffrey Regan
61c5afdf83
Update npm deps.
2019-12-12 13:18:30 -08:00
Phillip Wittrock
7e847dc907
publish apply / diff as alpha commands
2019-12-12 10:28:10 -08:00
Phillip Wittrock
1a28e493cd
Fix client-go version to match apimachinery version
2019-12-12 10:22:12 -08:00
Haiyan Meng
d9239104aa
Escape spaces in the query paths of git commit requests
2019-12-12 10:03:15 -08:00
Jeffrey Regan
ee678d6ce9
Upgrade to mdrip 1.0.1
2019-12-12 09:26:11 -08:00
Kubernetes Prow Robot
e1205698e6
Merge pull request #1933 from haiyanmeng/expose-es
...
Expose ElasticSearch as a LoadBalancer-type service
2019-12-11 16:16:32 -08:00
Kubernetes Prow Robot
a6535c4be2
Merge pull request #1934 from pwittrock/unk8s
...
revert pseudo/k8s deps
2019-12-11 16:08:32 -08:00
Phillip Wittrock
c2dc7ae789
revert pseudo/k8s deps
2019-12-11 15:28:04 -08:00
Haiyan Meng
afd24c6faf
Expose ElasticSearch as a LoadBalancer-type service
2019-12-11 15:05:10 -08:00
Haiyan Meng
0d79219e46
Avoid processing the nil pointer returned by kustomizationResultAdapter
...
Currently, the crawler job panics whenever a nil pointer is returned by
kustomizationResultAdapter.
2019-12-11 13:54:01 -08:00
Haiyan Meng
bffc0d7071
Mulitple improvements of the crawler
...
1) Set document IDs to avoid duplicating documents;
2) Set the `creationTime` field of each document in the index;
3) set the `values`, `kinds` and `identifiers` fields for all documents;
4) Add a `Copy` method into the `Document` struct: this fixes the issue
where all the documents existing in the index point to the same Document
object;
5) Avoid using keystore redis;
6) Set imagePullPolicy to `Always` for crawler jobs.
2019-12-11 11:10:48 -08:00
bzub
105a8e57c8
Detect path to plugins from krusty tests.
2019-12-07 13:16:15 -06:00
lalyos
95ad5d05b3
fix exec plugin args handling
2019-12-06 13:34:42 +01:00
Jeff Regan
969a1e8f37
Merge pull request #1892 from monopole/getRidOfFakeLoader
...
Get rid of the 'fake' loader.
2019-12-05 11:22:47 -08:00
Jeff Regan
184735fa89
Merge pull request #1887 from Liujingfang1/replacement-poc
...
add replacement type and an example transformer
2019-12-05 11:10:35 -08:00
Jeffrey Regan
caa71a73fe
Get rid of the fake loader.
2019-12-05 10:50:42 -08:00
Haiyan Meng
d25b6ff3dc
Remove duplicates in kinds
2019-12-04 11:24:54 -08:00
Haiyan Meng
68a196dbe5
Add a test case to demonstrate kinds have duplicates
2019-12-04 11:24:23 -08:00
Jingfang Liu
c7600bc079
add replacement type and an example transformer
2019-12-04 11:13:05 -08:00
Haiyan Meng
8aaa3f56f5
Set the ElasticSearch index creation configuration
...
Currently, the `kustomize` index in ElasticSearch is using dynamic
mapping, which sets the types of all the fields to `text`. However,
`text` field type is good for full-text value matching, and not good for
exact-value matching. For exact-value matching, the `keyword` filed
type should be used.
2019-12-03 15:16:32 -08:00
Jeffrey Regan
e9ab3da164
Fix some nits in the crawler and elsewhere.
2019-12-03 10:44:44 -08:00
Jeff Regan
fd5db20a48
Update harness.go
2019-12-02 18:47:07 -08:00
Jeffrey Regan
861c86a70a
Simplify plugin tests.
2019-12-02 17:12:59 -08:00
Jeffrey Regan
382c330f5b
Consolidate test harness to one package.
2019-12-02 12:29:10 -08:00