8 Commits

Author	SHA1	Message	Date
Haiyan Meng	1120c6bc7a	Add a User field into Document to make it easy to aggregate on github ... user level.	2020-01-21 10:09:52 -08:00
Haiyan Meng	f4636f8555	Add a fileType field into the index	2020-01-17 13:15:49 -08:00
Haiyan Meng	c801958d40	Log response status code to help debug ... Recently, the crawler job often fails after 10+ hours with the following error (10.0.47.27:9200 is the ElasticSearch master): dial tcp 10.0.47.27:9200: connect: connection refused	2020-01-10 11:37:22 -08:00
Haiyan Meng	745b58b3d0	Check whether a pointer is empty before accessing it to avoid SIGSEGV	2020-01-06 12:06:18 -08:00
Haiyan Meng	bef157d6b3	Fix insert/updating document logic	2019-12-18 15:56:44 -08:00
Haiyan Meng	272b7a6fcd	Use UpdateRequest to insert/update a document ... Currently, `IndexRequest` is used to insert/update a document, which increases the version of the document every time IndexRequest.Do is called.	2019-12-18 15:56:44 -08:00
Haiyan Meng	8aaa3f56f5	Set the ElasticSearch index creation configuration ... Currently, the `kustomize` index in ElasticSearch is using dynamic mapping, which sets the types of all the fields to `text`. However, `text` field type is good for full-text value matching, and not good for exact-value matching. For exact-value matching, the `keyword` filed type should be used.	2019-12-03 15:16:32 -08:00
Haiyan Meng	f69d2d2e69	Move hack/crawl under api/internal	2019-11-14 13:17:28 -08:00