Stock Price Data Pipeline
An automated market-data pipeline connecting Airflow orchestration, Spark processing, object storage, PostgreSQL, and Metabase.
A question. A process.
Work you can explore.
Follow the decisions behind the implementation, then inspect the original work.
Chapter 1 of 4 · The question
The question behind the build.
API availability, raw-data retention, transformation, loading, and delivery need explicit boundaries for a pipeline to be understandable and repeatable.
Follow the implementation.
- 01
Check API availability and ingest price data into MinIO raw storage.
- 02
Run Spark transformation in a Docker task.
- 03
Load the formatted output into a PostgreSQL warehouse.
- 04
Connect Metabase reporting and workflow completion notifications.
Try the project for yourself.
Read the DAG and architecture documentation to follow dependencies from ingestion to reporting. The project highlights the orchestration around an analytical dataset.
Choose where to look deeper.
Search the linked source inventory or narrow it to the kind of artifact you want to inspect. Files open at their original source.
7 linked source files
No linked files match this view.
The source
keeps moving.
Source updated
An end-to-end automated data pipeline built with Apache Airflow, Apache Spark, MinIO, PostgreSQL, and Metabase.
Current README excerpt
An end-to-end automated data pipeline built with Apache Airflow, Apache Spark, MinIO, PostgreSQL, and Metabase.
A source excerpt, not an independently verified outcome.