Amazon Prime Video
Automated quarterly business review metrics.
Overview
An automated pipeline and dashboard for the metrics behind the organisation's quarterly business review, replacing a collection process that had been done by hand every quarter.
Mostly a data engineering problem. Infrastructure is defined in CDK, the data stays raw until the reporting layer, and the aggregation is handled in SQL. The output is a QuickSight dashboard embedded in the internal wiki, where metrics can be read week by week and month by month rather than only at quarter end.
My manager's manager introduced it to the wider team as "this will make your lives 100x easier".
Stack
- AWS CDK
- CloudFormation
- Athena
- Lake Formation
- SQL
- QuickSight
Decisions
Choosing the simpler architecture
The design I first proposed ran a Glue ETL job aggregating into Redshift for the dashboard to read. Design review pushed on whether that complexity was earning anything, and it wasn't. It cost more to run, and changing a figure meant re-running jobs before you could see whether it was right. What I built keeps the data raw until the reporting layer, so each adjustment is a single query edit.
Checking the metrics before automating them
Once a number is on a dashboard people tend to trust it, so automating a metric before I properly understood it would just have made any mistake harder to spot. Tracing each one back to its source took longer than building the pipeline, but it was the part that made the output worth relying on.
Making it easy to pick up
The internship was going to end and someone else would own this, so it needed to be something another engineer could extend without me. The simpler pipeline helped, and I documented every data source and the steps for adding a new metric.
Results
- 50+
- Metrics
- 5
- Teams
- 80%
- Automated