Skip to content

Latest commit

 

History

History
226 lines (194 loc) · 11.1 KB

README.md

File metadata and controls

226 lines (194 loc) · 11.1 KB

Hazelcast Jet Code Samples

A repository of code samples for Hazelcast Jet. The samples show you how to use the Pipeline API to solve a range of use cases, how to integrate Jet with other systems and how to connect to various data sources (both from a Hazelcast IMDG and 3rd-party systems). There is also a folder with samples using the Core API.

Stream Aggregation

  • apply a sliding window
  • perform basic aggregation (counting)
  • print the results on the console
  • like the above, plus:
  • add a second-level aggregation stage to find top/bottom N results
  • apply a session window
  • use a custom Core API processor as the event source
  • perform a composite aggregate operation (apply two aggregate functions in parallel).
  • print the results on the console
  • use the SourceBuilder to create a mock source of trade events from a stock market
  • apply a tumbling window, configure to emit early results
  • aggregate by summing a derived value
  • present the results in a live GUI chart
  • use SourceBuilder to create a mock source of trade events from a stock market
  • simple rolling aggregation (summing the price)
  • keep updating the target map with the current values of aggregation
  • present the results in a live GUI chart

Batch Aggregation

  • use an IMap as the data source
  • stateless transforms to clean up the input (flatMap + filter)
  • perform basic aggregation (counting)
  • print a table of the most frequent words on the console
  • serialize a small dataset to use as side input
  • fork a pipeline stage into two downstream stages
  • stateless transformations to clean up input
  • count distinct items
  • group by key, then group by secondary key
  • aggregate to a map of (secondary key -> result)
  • hash-join the forked stages
  • open an interactive GUI to try out the results

Joins

  • co-group three bounded data streams on a common key
  • for each distinct key, emit the co-grouped items in a 3-tuple of lists
  • store the results in an IMap and check they are as expected
  • use the Event Journal of an IMap as a streaming source
  • apply a sliding window
  • co-group three unbounded data streams on a common key
  • print the results on the console

Hash Join

Data Enrichment

  • the sample is in the enrichUsingIMap() method
  • use the Event Journal of an IMap as a streaming data source
  • apply the mapUsingIMap transform to fetch the enriching data from another IMap
  • enrich from two IMaps in two mapUsingIMap steps
  • print the results on the console
  • the sample is in the enrichUsingReplicatedMap() method
  • use the Event Journal of an IMap as a streaming data source
  • apply the mapUsingReplicatedMap transform to fetch the enriching data from another IMap
  • enrich from two ReplicatedMaps in two mapUsingReplicatedMap steps
  • print the results on the console
  • the sample is in the enrichUsingAsyncService() method
  • prepare a mock data server: a gRPC-based network service
  • use the Event Journal of an IMap as a streaming data source
  • enrich the unbounded data stream by making async gRPC calls to the mock service
  • print the results on the console
  • the sample is in the enrichUsingHashJoin() method
  • use the Event Journal of an IMap as a streaming data source
  • use a directory of files as a batch data source
  • hash-join an unbounded stream with two batch streams in one step
  • print the results on the console

Configuration

Job Management

Integration with Hazelcast IMDG

Integration with Other Systems

Custom Sources and Sinks

  • Custom Source:
    • start an Undertow HTTP server that collects basic JVM stats
    • construct a custom Jet source based on Java 11 HTTP client
    • apply a sliding window
    • compute linear trend of the JVM metric provided by the HTTP server
    • present the results in a live GUI chart
  • Custom Sink
    • construct a custom Hazelcast ITopic sink
  • Custom Side Input for Data Enrichment
    • look at the enrichUsingReplicatedMap() method

Integration with Frameworks

Spring Framework

Use a pre-trained TensorFlow model to enrich a stream of movie reviews. The model classifies natural-language text by the sentiment it expresses.

Integration with Deployment Environments

Code samples that show how to use the low-level Core DAG API directly. See the dedicated README for more details.

License

Hazelcast is available under the Apache 2 License. Please see the Licensing section for more information.

Copyright

Copyright (c) 2008-2019, Hazelcast, Inc. All Rights Reserved.

Visit www.hazelcast.com for more info.