Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Streaming Roadmap #192

Open
1 of 12 tasks
gaborhermann opened this issue Apr 21, 2016 · 0 comments
Open
1 of 12 tasks

Streaming Roadmap #192

gaborhermann opened this issue Apr 21, 2016 · 0 comments
Assignees
@gaborhermann
Labels
planning STREAMING

Comments

@gaborhermann
Copy link

gaborhermann commented Apr 21, 2016 •
edited
Loading

This is a meta-issue about what needs to be done for introducing StreamBags to Emma. A lot of more theoretical work needs to be done (especially at optimizations). This issue will reflect the state of implementations, and will change over time.

We will work on branch streaming for now.

User-facing API

Compilation

  • Resugaring comprehensions (reuse Emma batch code)
  • Comprehension normalization (reuse Emma batch code)
  • Define stream target language (i.e. dataflow with stateful flatMaps) (issue Stream target language #202)
  • Translate comprehensions to target language
  • Backends for target language (Kafka Streams, Flink, Storm)
  • Serialization of streams based on coalgebraic view (to allow nested StreamBags)

Optimizations

  • General comprehension optimizations (group by, equijoin etc.)
  • Time predicates (for windowing)
  • Make heterogeneous Bag-StreamBag comprehensions scalable
  • Windowing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gaborhermann gaborhermann

Labels
planning STREAMING
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gaborhermann