Ylem is an open-source data streaming platform that is a one-stop-shop solution for orchestrating data streams on top of Apache Kafka, Amazon SQS, Google Pub/Sub, RabbitMQ, various APIs, and data storages.
This repository contains installation files that allow you to set up Ylem from the pre-build containers.
Ylem uses Docker to run its containers by default. If you don't yet have Docker 4 installed, install it from their official website for your OS.
This repository contains .env file where you can configure your connection to Apache Kafka and other integrations.
More information is in our documentation.
docker compose upOr
docker compose up -dYlem is available at http://localhost:7331/
Ylem uses Apache Kafka to exchange messages for processing pipelines and tasks. By default Ylem already comes with the pre-configured Apache Kafka container, however, you might already have an Apache Kafka cluster in your infrastructure and might want to reuse it.
In this case, you need to take the following steps:
To be able to work correctly, Ylem requires the following Apache Kafka topics to be created:
- task_runs
- task_runs_load_balanced
- task_run_results
- query_task_run_results
- notification_task_run_results
Open docker-compose.yml in the same folder with this README file and remove or comment out the following lines to exclude it from the network.
Also don't forget to remove these and these dependency checks.
Open .env and replace the following value with your Kafka host:
YLEM_KAFKA_BOOTSTRAP_SERVERS=ylem_kafka_broker:39092Now if you run docker compose up and everything is set up correctly Ylem should be able to work with your own Kafka cluster.
If you want to use an SSH connection to your MySQL or PostgreSQL database, you need to place your public RSA SSH key called id_rsa.pub in the folder keys next to this README file and it will be mounted to the container automatically.