Laptop Data Platform
A complete modern data stack that runs on a laptop and costs nothing
By Renjith ·
Prerequisites
- Python 3.11+
- Basic SQL
- A terminal
The problem
Learning data engineering has an artificial entry cost. Tutorials assume a cloud warehouse, a paid orchestrator and a billing account, so people who want to learn the concepts end up spending their first weekend on IAM roles.
The concepts — incremental models, idempotency, testing, lineage, orchestration — do not require any of that. They are ideas about how to structure work, and they are learnable on a laptop.
What was built
A complete platform that clones and runs with two commands:
- Ingestion — Python extractors pulling from a public API and CSV files into raw tables.
- Storage — DuckDB, one file, no server.
- Transformation — dbt with staging, intermediate and mart layers.
- Testing — dbt tests on every model, plus freshness checks.
- Orchestration — Dagster assets with real dependencies and schedules.
- BI — a Streamlit app reading the marts.
Everything a cloud stack teaches, minus the credit card. The same dbt project runs against Snowflake or BigQuery by changing a profile — which is the point: the skills transfer, the bill does not.
How it works
Why this teaches the same things#
The gap between this and a cloud stack is scale and operations, not concepts. Every idea that makes data engineering hard is present here:
| Concept | Where it appears |
|---|---|
| Idempotency | Rerunning any asset produces the same result |
| Incremental processing | The events model, with a deliberately broken variant to study |
| Testing and contracts | dbt tests gating every layer |
| Lineage | Dagster's asset graph |
| Slowly changing dimensions | The customer dimension, type 2 |
| Backfills | A dated partition you can rerun for any window |
models/marts/fct_orders.sqlsql{{ config(materialized='incremental', unique_key='order_id') }}
-- The predicate is the whole lesson: without it this is a full refresh
-- wearing an incremental costume, and it will pass every test while
-- quietly reprocessing the entire history every run.
select * from {{ ref('stg_orders') }}
{% if is_incremental() %}
where updated_at > (select coalesce(max(updated_at), '1900-01-01') from {{ this }})
{% endif %}
Break it on purpose
The most useful exercise in the whole project is removing that where clause and watching the model still pass every test while silently reprocessing everything. Incremental models fail quietly, and the only reliable way to learn that is to cause it.
When you would outgrow it#
DuckDB handles single-node analytical work impressively — tens of millions of rows on a laptop is comfortable. You move off it when you need concurrent writers, data larger than local disk, or shared access across a team. That is a genuinely smaller set of situations than most architecture diagrams assume.
Build steps
- 1
Set up DuckDB and load raw data
One file, no server, no connection string. Load the sample data and query it immediately.
- 2
Build staging models
One staging model per source table: rename, cast, and nothing else. The discipline of a thin staging layer is most of what keeps a dbt project navigable.
- 3
Add tests before transforming further
Uniqueness, not-null and relationship tests on staging. Catching a broken assumption here costs minutes; catching it in a mart costs a day.
- 4
Build marts
Star-schema marts shaped for questions rather than for sources.
- 5
Make one model incremental
And deliberately break it, so the failure mode of a bad incremental key is something you have seen rather than read about.
- 6
Orchestrate with Dagster
Assets with real dependencies, a schedule and a UI showing lineage — the same mental model as any managed orchestrator.
Lessons learned
Local-first removes the excuse. Nobody abandons this project because their trial expired.
Thin staging models are the highest-leverage discipline in dbt.
Incremental models fail silently. Building a broken one deliberately is worth more than reading about it.
Most companies are running a cluster for a workload that fits on a laptop. That is a genuine finding, not a rhetorical flourish.
Limitations
Single-node and single-writer. Concurrent writes are not supported and the design assumes one process at a time.
No secrets management, network security or access control — all of which are real work in a production deployment and are deliberately out of scope.
The Streamlit BI layer is a demonstration, not a replacement for a semantic layer with governed metrics.
Future improvements
A profile-swap guide for pointing the same dbt project at Snowflake, BigQuery and Postgres, so the transfer of skills is explicit.
Data contracts between staging and marts using dbt's model contracts.
A worked example of a schema change flowing through every layer, since that is the recurring real-world event this kind of project never covers.
Related course
Modern Data Engineering on Your Laptop
DuckDB, dbt and Dagster — ingestion, transformation, testing, orchestration and lineage without a cloud bill.
Free resource
Retail Sample Dataset
A realistic multi-table retail dataset with deliberate data quality problems, for practice and portfolio work.
Related builds and resources
Modern Data Engineering on Your Laptop
DuckDB, dbt and Dagster — ingestion, transformation, testing, orchestration and lineage without a cloud bill.
DuckDB vs pandas vs Polars: When Each One Wins
A practical decision guide based on workload shape rather than benchmark scores — because the fastest tool depends on what you are actually doing.
Retail Sample Dataset
A realistic multi-table retail dataset with deliberate data quality problems, for practice and portfolio work.
The Entire Modern Data Stack, On a Laptop
DuckDB, dbt and Dagster — every concept a cloud stack teaches, without the bill.