← Back to Projects & Blogs

Microsoft Data Stack

View Code
Data EngineeringSQL ServerAirflowSupersetDocker

Overview

Plenty of shops still run on SQL Server, and almost none of the modern pipeline tutorials are written for them. Everything assumes Postgres or a cloud warehouse. This one runs end to end on SQL Server 2022, in containers, with Airflow driving it.

Data arrives from REST APIs and CSV files. Airflow walks it through four layers in the same database: Raw keeps exactly what showed up, Staging cleans it, the warehouse models it, and Analytics serves the marts. Superset reads the last layer.

Architecture

Key Features

  • Multi-Source Ingestion - REST endpoints and CSV drops land through the same orchestrated path
  • Four Layers - Raw, Staging, Data Warehouse, Analytics. Each one has a job, and nothing skips ahead
  • Quality Monitoring - Every load counts what it received, what it stored, and what it rejected
  • Pre-Built Dashboards - Sales analytics and data quality, both ready on first boot
  • Data Marts - Aggregates built once, so reports read instead of recompute

Tech Stack

  • Database - SQL Server 2022
  • Orchestration - Apache Airflow 2.8
  • Visualization - Apache Superset 3.1
  • Processing - Python (pandas, requests)
  • Infrastructure - Docker Compose

Counting What Gets Dropped

The part I care about most is the smallest. Each load writes three numbers: records received, records stored, records rejected. Those numbers drive a Superset dashboard, which means a load that quietly drops a third of its rows shows up as a gap on a chart instead of a question somebody asks three weeks later.

Quick Start

bash
# Clone and start
git clone https://github.com/AlharbiAbdullah/microsoft_data_stack
cd microsoft_data_stack
docker-compose up --build -d

# Access services
# Airflow:   http://localhost:8080
# Superset:  http://localhost:8088
# SQL Server: localhost:1433