Commit Graph

233 Commits

Author SHA1 Message Date
Kubernetes Prow Robot
0b1ad031a9 Merge pull request #2169 from yujunz/loader/go-getter
Replace git cloner with go getter to support various target
2020-03-06 15:45:34 -08:00
Kubernetes Prow Robot
300dd108d5 Merge pull request #2167 from yujunz/loader/http
Allow loading file from http
2020-03-06 15:31:34 -08:00
Kubernetes Prow Robot
65522b9e9f Merge pull request #2182 from chancez/per_resource_generator_options
Enable specifying GeneratorOptions per resource
2020-03-06 14:51:34 -08:00
Yujun Zhang
735cefa456 Use a lite fork of go-getter 2020-03-04 20:04:39 +08:00
Yujun Zhang
a24bf6aece Use a tailored go-getter to get rid of transitive depenencies 2020-03-04 20:04:39 +08:00
Yujun Zhang
fae66446a8 Replace git cloner with go getter to support various target 2020-03-04 20:04:39 +08:00
Rhomel Chinsio
f0ae77abd5 allow braces in image tag name (api) 2020-03-03 05:48:36 +00:00
Yujun Zhang
382425d974 Add unit test for http fileloader 2020-02-29 16:19:21 +08:00
Yujun Zhang
ff6250cdb4 Allow loading file from http 2020-02-29 16:19:21 +08:00
Chance Zibolski
62f21cbe69 Enable specifying GeneratorOptions per resource
In addition to specifying GeneratorOptions globally within a
kustomization, also allow users to set GeneratorOptions on a
per-resource level.
2020-02-26 09:54:43 -08:00
vanou
87b680e1c0 Correct explanation of ApendAll method in ResMap interface
This commit corrects expalation of ApendAll method in ResMap
interface. AppendAll method should fail on CurId collision,
not OrgId collision.
2020-02-19 23:53:10 +09:00
Haiyan Meng
b7b7a5a79f Fix typo 2020-02-10 15:44:51 -08:00
Haiyan Meng
807ca9c1e3 Add notes on backup and restore 2020-02-10 08:30:08 -08:00
Haiyan Meng
baccf58ccf Avoid tracking the change in github_api_secret.txt
This helps prevent commiting your Github personal access token into
Github by accident.
2020-02-05 12:06:21 -08:00
Haiyan Meng
c7bdb3fbe4 Add cmds to process the kustomize-stats log 2020-02-05 11:04:59 -08:00
Haiyan Meng
967fe44e3f Add curl commands for kustomize stats 2020-02-05 11:04:59 -08:00
Haiyan Meng
d0602c732b Remove the usage of github access token from the kustomize-stats job 2020-02-05 11:04:59 -08:00
Haiyan Meng
a4179fa87f Use the silence mode of curl 2020-02-05 11:04:59 -08:00
Haiyan Meng
c9bce3fc0a Add comments on backup and restore 2020-02-05 11:04:59 -08:00
Haiyan Meng
3ebeebabde Add comments for backup and restore 2020-02-03 12:37:18 -08:00
Haiyan Meng
a3b3449b1f Add curl commands for generator/transformer exploration 2020-02-03 09:59:52 -08:00
Haiyan Meng
1b8488da2c Add curl commands for snapshoting 2020-02-03 09:59:52 -08:00
Haiyan Meng
f5419e9f72 Check the incomplete_results field of github query responses
Currently, we don't check the `incomplete_results` field of a github
query response, which is problematic when incomplete query results are
used to split the query ranges: the splitted query ranges will
be very wild.
2020-02-03 09:59:52 -08:00
Haiyan Meng
7a87c84403 Reprocess the github filesize search ranges which have more than 1000 items 2020-02-03 09:59:52 -08:00
Haiyan Meng
0fcb3a014c Add config for index backup and restore 2020-02-03 09:59:52 -08:00
Haiyan Meng
0b38e6d284 Improve the analysis on generator and transformer 2020-02-03 09:59:52 -08:00
Haiyan Meng
d5c66cb3d4 Add KustomizationDocument.Copy method 2020-02-03 09:59:52 -08:00
Haiyan Meng
b35b5aa73d Check the checksums of documents in the index 2020-02-03 09:59:52 -08:00
Haiyan Meng
bb409a5ea8 Set up cronjob to run crawler every 7 days 2020-02-03 09:59:52 -08:00
Haiyan Meng
74e1b5d54b Add GCP service account into ESCluster config
This is necessary for index backup into GCS and index recovery from GCS
2020-02-03 09:59:52 -08:00
Jeff Regan
0ce076758d Merge pull request #2150 from haiyanmeng/stats
Add `fileType` and `User` into the index
2020-01-28 09:18:31 -08:00
Haiyan Meng
154208d331 Improve the efficiency of crawling github by skipping the documents
already in the index
2020-01-24 19:55:56 -08:00
Haiyan Meng
b7b88cae76 Add curl commands for querying different filetypes 2020-01-23 16:04:55 -08:00
HowJMay
00f68c12a8 fix typos
Fix typos
2020-01-23 23:35:38 +08:00
Haiyan Meng
0820865e1d Retry FindRangesForRepoSearch 2020-01-22 10:13:57 -08:00
Haiyan Meng
1120c6bc7a Add a User field into Document to make it easy to aggregate on github
user level.
2020-01-21 10:09:52 -08:00
Phani Teja Marupaka
0bd872e6d5 Do not remove empty lines in configmap/secret 2020-01-20 11:42:39 -08:00
Haiyan Meng
96ee9e9146 Add curl ElasticSearch cmd for using filter and range together 2020-01-17 15:49:14 -08:00
Haiyan Meng
377eb5b66d Fix the regexp for determining kustomization file 2020-01-17 15:48:38 -08:00
Haiyan Meng
f4636f8555 Add a fileType field into the index 2020-01-17 13:15:49 -08:00
Haiyan Meng
9f80da28ae Refactor the stats code for generators and transformers 2020-01-16 09:20:24 -08:00
Haiyan Meng
5477bde7e5 Use an env variable for index name and fix the call to NewKustomizeIndex in backend 2020-01-15 15:29:17 -08:00
Haiyan Meng
3ead42fe27 Add --index flag to kustomize_stats config file 2020-01-15 15:29:16 -08:00
Haiyan Meng
cf8d53a195 Move SeenMap to the utils dir 2020-01-15 15:29:16 -08:00
Haiyan Meng
aaaba99389 Use Document.Path instead of its fields 2020-01-15 12:10:08 -08:00
Haiyan Meng
29e50ab476 Collect stats on generators and transformers 2020-01-15 12:10:08 -08:00
Haiyan Meng
3519cc56a1 Add support to get files referred in the generators and tranformers
fields
2020-01-15 12:10:08 -08:00
Haiyan Meng
2e895c147e Use log.Print* instead of fmt.Print* 2020-01-14 15:50:35 -08:00
Haiyan Meng
af131c7471 Use flags to specify crawling mode and github user/repo info 2020-01-14 15:36:12 -08:00
Haiyan Meng
7ac573ae51 Add a flag to specify the index name 2020-01-14 14:25:29 -08:00