Big Data Pipeline

Using MariaDB to create a new database and a new table with some data then migrating with Sqoop to the HDFS

Then Real-Time Data Ingestion with Apache Flume and Apache Kafka:

by creating a python script that generate data into a log file then using Flume to send data into Kafka sink on real time basis