5 Commits

Author	SHA1	Message	Date
Haiyan Meng	1120c6bc7a	Add a User field into Document to make it easy to aggregate on github ... user level.	2020-01-21 10:09:52 -08:00
Haiyan Meng	cf8d53a195	Move SeenMap to the utils dir	2020-01-15 15:29:16 -08:00
Haiyan Meng	12fc8f41c7	Add support for github paths starting with "git@github.com:"	2019-12-18 15:56:44 -08:00
Haiyan Meng	bffc0d7071	Mulitple improvements of the crawler ... 1) Set document IDs to avoid duplicating documents; 2) Set the `creationTime` field of each document in the index; 3) set the `values`, `kinds` and `identifiers` fields for all documents; 4) Add a `Copy` method into the `Document` struct: this fixes the issue where all the documents existing in the index point to the same Document object; 5) Avoid using keystore redis; 6) Set imagePullPolicy to `Always` for crawler jobs.	2019-12-11 11:10:48 -08:00
Haiyan Meng	f69d2d2e69	Move hack/crawl under api/internal	2019-11-14 13:17:28 -08:00