Spring-Kafka-Etl-Api

This project occurs four application. The project read ndjson file that named product-views.json by line by and write the line to Kafka. And then, reads the line, that written to Kafka, from Kafka then write to PostgreSQL.

Plus, implements ETL process to products, orders and order_items tables that take place in PostgreSQL.

Also, with API return ten viewed products given userId and recommend products given userId.

Modules and Modules' description

Module	Use Technology	Description
view-producer	Java Spring	Reads product-views.json file by line by and transform the line to Event class and writes to Kafka.
stream-reader	Java Spring	Reads event from Kafka and transform the event to BrowsingHistory class and the class write to PostgreSQL Database.
etl-process	PySpark	Implements ETL to orders, products and order_items tables. These tables joins bestseller_product table.
api	Java Spring	Returns response for specific requests.

Click the module name to read its own README

Diagrams

View-Producer Diagram

Diagram

Stream-Reader Diagram

Diagram

ETL-process Diagram

Diagram

API Diagram

Diagram

Requirements Check

Module	Code	README	Unittest	Containerization
viewed-producer	✔️	✔️	❌	✔️
stream-reader	✔️	✔️	❌	✔️
etl_process	✔️	✔️	✔️	✔️
api	✔️	✔️	✔️	✔️

Install

If you don't want to install anything so run below command:

docker-compose -f project-docker-compose.yml up -d

If you want to install applications single by single except above installing. For this, run below command and click to module name you want to install.

docker-compose up -d