Skip to content

Latest commit

 

History

History
97 lines (82 loc) · 3.24 KB

README.md

File metadata and controls

97 lines (82 loc) · 3.24 KB
Raw

alarm-system-kafka

An alarm system that monitor hosts' CPU & memory indicators and send notifications according to user configurations.

Context

After took some kafka udemy courses, i did this mini project to apply some of the knowledge that i got.

Kafka topics covered

  • Consumer API
  • Producer API
  • Streams
  • Sink Connector.

Requisites

  • Multiple hosts send information about the memory and cpu usage.
  • The user has the possibility to configure mem/cpu limits and the time to trigger alarms.
  • If the limits are exceed inside a time window then a notification must be send.
  • All the messages must be audited in a text file.

Architeture

image

Configurations

audit-consumer

resources/fileconnector.properties

Config var Description Example
topics Topic to read user_alarms
file Log file to write /Users/andfaria/test.log

notifier-consumer

resources/application.properties

Config var Description Example
smtpServer Email server smtp.gmail.com
Port Server port 587
emailOrigin Sender [email protected]
username User [email protected]
password Password test
emailDestination Destionation email [email protected]
telegramToken Telegram Token blablabla231321
telegramNotification Set on/off telegram notification on
telegramChatId Conversation ID to send notification 123123

transformer-stream

resources/alarms.properties

Config var Description Example
memory Minimum free memory in MBs 500
cpu Max cpu usage 50
retention Time window to consider cpu and memory indicators Milliseconds 3600

Running the app

Test environment

  • MacOS Catalina 10.15.4
  • Kafka Server 2.3.0
  • Java 8

Note: In order to send emails successfully with gmail, i needed to disable the two-factor authenticator and enable less secure apps.

Compiling

Navigating to root dir and execute:


mvn clean install assembly:single

Start kafka and zookeeper:

docker-compose -f docker-compose.yaml up

Running

audit-consumer

cd audit-consumer/
export CLASSPATH="$(find target/ -type f -name '*.jar'| grep '\-package' | tr '\n' ':')"
connect-standalone.sh config/worker.properties config/fileconnector.properties

client-producer

cd client_producer/target
#arg1 -> bootstrapServer, arg2 -> topic to write, arg3 -> interval between writes
java -cp client-producer-1.0.0-jar-with-dependencies.jar org.aflabs.kafka.producer.client.Producer "127.0.0.1:9092" host_stats 10000

transformer-stream

cd transformer-stream/target
#arg1 -> bootstrapServer, arg2 -> applicationId. If need edit file in classes/alarms.properties
java -cp "classes:./*" org.aflabs.kafka.stream.transformer.AlarmsStream "127.0.0.1:9092" "alarms-system-stream-9"

notifier-consumer

cd notifier-consumer/target
#arg1 -> bootstrapServer, arg2 -> topic to read. If need edit file in classes/application.properties
java -cp classes:notifier-consumer-1.0.0-jar-with-dependencies.jar org.aflabs.kafka.consumer.notifier.Consumer  "127.0.0.1:9092" "alarm-consumer"