AWS Data Engineering · Batch Pipeline

Ember Energy Pipeline — Architecture

Monthly European electricity-generation data flows from the Ember Energy API through a Python ingestion step into an S3-and-Glue batch pipeline, lands in a queryable Data Catalog, and surfaces as an interactive QuickSight dashboard.

Region eu-central-1 (Frankfurt) Countries DEU · FRA · CHE · ITA · ESP · POL · BEL · NLD Sources Solar · Wind · Hydro · Bioenergy · Nuclear · Coal · Gas Budget $50/mo cap Range 2020-01 → 2026-04
Client / Local Environment

Ember Energy API

External data source · monthly electricity generation, 88 geographies

ember-energy.org/data

Ingestion Script

Python + boto3 · fetches JSON, uploads to S3

src/ingest_ember_data.py
AWS Cloud — eu-central-1

S3 Raw Bucket

Landing zone · JSON, partitioned by extraction date

extraction_date=YYYY-MM-DD/

Glue ETL Job

Explode array → Flatten fields → Parquet/Snappy

EmberGlueETLJob-v1

S3 Processed Bucket

Analytics-ready columnar output

Parquet · Snappy compression

Glue Crawler

Infers schema from processed Parquet

EmberProcessedDataCrawler

Glue Data Catalog

Central metadata store

ember_energy_db

Amazon Athena

SQL validation & ad-hoc analytics

Results → dedicated S3 bucket

QuickSight Dashboard

Stacked area chart · KPI cards · geo map · Country / Source filters

SPICE import
data flow direct read access ● external / client-side

Service roles

ServiceRole
Amazon S3Raw & processed data lake (two buckets)
AWS Glue ETLJSON → Parquet, Explode + Flatten transforms
AWS Glue CrawlerAuto-discovers schema from processed Parquet
Glue Data CatalogCentral metadata store (ember_energy_db)
Amazon AthenaSQL validation & ad-hoc analytics
Amazon QuickSightInteractive dashboard for trends & energy mix

Known pipeline fixes

IssueResolution
Nested JSON in AthenaAdded Explode + Flatten transforms to the Glue job
Duplicate recordsClear processed bucket + catalog table before re-running ETL/crawler
Broken time axisCast date column to Date type before SPICE import