This project is a real-world data engineering ETL pipeline built using a Kaggle Netflix dataset. The goal is to demonstrate how raw, messy data can be validated, cleaned, verified, and prepared for downstream analytics or data warehouse loading.
The pipeline follows professional data engineering practices such as raw vs cleaned data separation, schema and quality validation, modular ETL design, and early failure on bad data.
🎯 What This Project Does
- Reads raw Netflix data from a CSV file
- Validates raw data for schema and basic sanity checks
- Cleans fixable data quality issues (nulls, invalid years, duplicates)
- Re-validates cleaned data with strict rules
- Saves a clean, analytics-ready dataset
- Automatically generates plots to visually verify data quality
- Simulates bad data to demonstrate pipeline failure behavior
Project Structure
🛠 Technologies Used
- Python
- Pandas
- Matplotlib
- JupyterLab
- Git & GitHub
▶️ How to Run the Project
- Create and activate virtual environment python -m venv .venv source .venv/bin/activate
- Install dependencies pip install -r requirements.txt
- Run ETL pipeline (normal run) python src/run_etl.py
- Run ETL with bad-data simulation (FAIL demo) python src/simulate_bad_data.py python src/run_etl.py data/raw/netflix_titles_BAD.csv
Outputs (Saved Plots)
Run
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
# Run with default raw dataset
python src/run_etl.py
# Run with a specific file (e.g., bad dataset test)
python src/run_etl.py data/raw/netflix_titles_BAD.csv
source .venv/bin/activate
pip install -r requirements.txt
# Run with default raw dataset
python src/run_etl.py
# Run with a specific file (e.g., bad dataset test)
python src/run_etl.py data/raw/netflix_titles_BAD.csv