Data quality
weather-data-pipeline-great-expectations
A weather data pipeline that collects hourly observations from 50+ global cities via the Open-Meteo API, validates every batch with Great Expectations, and transforms data through dbt into analytics-ready BigQuery tables.
Drawing
Highlights
- Quality gates check schema, null rates under 5%, value ranges, row counts and freshness under 3 hours.
- Failed checks halt the pipeline and send Slack alerts.
- 21 unit tests pass across 5 test suites.