Open Data Stack

Overview

Data engineering demos usually pick a side. Either a batch pipeline or a streaming one, never both, and almost always over a toy CSV. Open Data Stack runs both paths over the same live feed of stock prices from Yahoo Finance, so the difference between them is something you can watch instead of read about.

On the batch path, Airflow pulls prices on a schedule and lands them in DuckDB. On the streaming path, the same tickers go through Kafka and get processed by Spark. Both ends surface in Superset. The whole stack is open source and comes up with one docker-compose up.

Architecture

Key Features

• Two Paths, One Source - A scheduled Airflow job into DuckDB, and a Kafka stream into Spark, both reading the same API
• Real Market Data - Live prices for AAPL, GOOGL, MSFT, AMZN, and META. Nothing synthetic
• Dashboards Included - Superset comes pre-built, so the stack has something to show the minute it boots
• One Command - Docker Compose brings up every service
• 73 Tests - The pipeline code is covered, not just demoed

Tech Stack

• Data Source - yfinance (Yahoo Finance API)
• Message Queue - Apache Kafka
• Stream Processing - Apache Spark
• Orchestration - Apache Airflow
• Data Warehouse - DuckDB
• Visualization - Apache Superset
• Processing - Pandas, PySpark

Why Both Paths

The batch and streaming tradeoff is easier to feel than to argue about. Run them side by side on the same tickers and it stops being abstract. The Airflow path is cheap, simple to reason about, and always a little behind. The Kafka path is current, and you pay for that in moving parts: a broker, a consumer, and a Spark job that has to keep up. Same data, two costs.

Quick Start

# Clone and start
git clone https://github.com/AlharbiAbdullah/open_data_stack
cd open_data_stack
docker-compose up --build -d

# Access services
# Airflow:   http://localhost:8080
# Superset:  http://localhost:8088
# Kafka UI:  http://localhost:8082