Real Time Kafka Spark Streaming

A Dockerized Kafka system for streaming server metrics and load balancer logs, with data processing using Spark and storage in a relational database and Hadoop.

This project involves designing a multi-node Kafka cluster for handling metrics and logs from a cluster of 10 servers and a load balancer. The goal is to process and store metrics and logs effectively using Kafka, a relational database, and Hadoop. The system architecture includes:

10 Servers: Each with an agent to send resource consumption metrics.

Load Balancer: Equipped with an agent to send log data.

Kafka Cluster: To handle the incoming data.

Relational Database(Postgres): For storing metrics.

Spark Application: To process logs and calculate moving window counts.