Refactor code design

[mihneadb]

Need

The code should be nicely structured in order to make it easy to contribute.

Deliverables

• code has a new structure that makes more sense given the ETag vs polling mechanisms introduced in Incremental loading of Github events and issues. #16
• clean up debug logs
• lay out the overall process for fetching and indexing changes in an abstract class, and let subclasses decide on how to do each step. More or less like template method pattern
• do pagination by implementing an iterator
• use an executor instead of a thread

Solution

TODO

[mihneadb]

cc @LosD feel free to come up with a suggestion and I'll edit the issue with your ideas.

[LosD]

I have a few, though some of them may not be worth the effort.

• Lay out the overall process for fetching and indexing changes in an abstract class, and let subclasses decide on how to do each step. More or less like template method pattern.
• Do pagination by implementing an iterator. Then the complexities of paginating are well hidden, making it easier to work with a new type (i.e. Git data). I don't believe there is anything specific to each entry type, but in that case it should still be possible to make subclasses for specifics.
• Use an executor instead of a thread. That should make the River a little more resilient. Right now an unexpected exception will simply stop the thread, even with no exception in the logs. I know, because I screwed up an ES query :)

[mihneadb]

@LosD I'm thinking of rewriting the river [as soon as I find some time] in a better manner using Scala. Are you interested in hacking some Scala? :)

[LosD]

My Scala experience is rather limited (we use it for a few tests and analysis tools at my employer), but yeah :-)

On 24 January 2015 17:49:27 CET, Mihnea Dobrescu-Balaur [email protected] wrote:

@LosD I'm thinking of rewriting the river [as soon as I find some time]
in a better manner using Scala. Are you interested in hacking some
Scala? :)

---

Reply to this email directly or view it on GitHub:
#18 (comment)

[LosD]

Heads up: I've been at the ElasticSearch Core training today, and they warned me that rivers are deprecated. They will still be there in 2.0, but probably not much longer than that.

Logstash is now the expected way to get data into ES, so we might not want to put too much energy into updating the river. However, I know nothing about Logstash, so I don't know if it would even be viable for importing stuff from Github.

[LosD]

More info here https://www.elastic.co/blog/deprecating_rivers

[mihneadb]

Thanks for the info! :) Will keep that in mind.