2 Commits

Author	SHA1	Message	Date
Haiyan Meng	bffc0d7071	Mulitple improvements of the crawler ... 1) Set document IDs to avoid duplicating documents; 2) Set the `creationTime` field of each document in the index; 3) set the `values`, `kinds` and `identifiers` fields for all documents; 4) Add a `Copy` method into the `Document` struct: this fixes the issue where all the documents existing in the index point to the same Document object; 5) Avoid using keystore redis; 6) Set imagePullPolicy to `Always` for crawler jobs.	2019-12-11 11:10:48 -08:00
Haiyan Meng	f69d2d2e69	Move hack/crawl under api/internal	2019-11-14 13:17:28 -08:00