3 Commits

Author	SHA1	Message	Date
Haiyan Meng	bffc0d7071	Mulitple improvements of the crawler ... 1) Set document IDs to avoid duplicating documents; 2) Set the `creationTime` field of each document in the index; 3) set the `values`, `kinds` and `identifiers` fields for all documents; 4) Add a `Copy` method into the `Document` struct: this fixes the issue where all the documents existing in the index point to the same Document object; 5) Avoid using keystore redis; 6) Set imagePullPolicy to `Always` for crawler jobs.	2019-12-11 11:10:48 -08:00
Haiyan Meng	84b75afae4	Make the crawler work ... 1) add the crawler binary and fix the crawler library 2) remove the readiness probe in the search backend 3) add config for redis keystore 4) add github_api_secret.txt file with instructions	2019-11-26 09:50:51 -08:00
Haiyan Meng	f69d2d2e69	Move hack/crawl under api/internal	2019-11-14 13:17:28 -08:00