Data Engineering

NumPy: Whole Columns at Once

Machine learning is maths on thousands of numbers at once. NumPy arrays let you do it in one line — no loops — and they're what every ML library runs on underneath.

pandas: Tables You Can Question

pandas is the spreadsheet of data science. Load 80 farms, select, filter, group and clean them — the everyday moves before any model is trained.

Files & Formats

Data moves between systems as files. Learn what each format keeps and loses — why CSV forgets your types, how JSON nests, and why data lakes run on Parquet — using the libraries data engineers use.

Data Modelling

A flat export repeats every customer and agent on every row — and quietly contradicts itself. Find the entities, normalise into linked tables with keys, and let a database enforce the rules.

Digitise a SACCO's Ledger

A savings cooperative in Nyeri keeps a year of member contributions in a spreadsheet. Profile its problems, design a proper database, load the clean rows (and report the rest), and export an analysis-ready Parquet file.

SQL for Analysts

Real data lives in databases, and SQL is how you ask for it. Build a database in your browser, then filter, group, join and clean with SQL — and hand the answers to pandas.

Window Functions & DuckDB

Running balances, rankings, month-on-month growth: the questions GROUP BY can't answer. Learn window functions in DuckDB — the fast analytics database that runs anywhere, even in your browser.

Indexes & Query Performance

Why does one query take milliseconds and another minutes? Read query plans, add the right index, measure the speed-up, and see why analytics databases store data by column.

Mobile-Money Ledger Analytics

A mobile-money provider's operations team needs three numbers every morning: daily volume and its trend, customers slipping away, and agents who handle unusual volumes. Build the queries in DuckDB and ship them as Parquet.

Extracting Data

Every pipeline starts by pulling data from somewhere else. Page through an API, survive the errors real networks throw, and download only what changed since last time.

Transform & Load

Turn raw API records into a clean table, and load them so that running the job twice, after a crash or for a backfill, never double-counts. This is the property that separates a script from a pipeline.

Orchestration with Airflow & dbt

Real pipelines are many tasks with dependencies, run on a schedule. Learn how Airflow runs them, build a small orchestrator yourself, then build a mini dbt that turns SQL files into a dependency graph.

A Crop Prices Pipeline

A maize and bean cooperative in Kitale wants to know each week where its crops fetch the best price. Build the pipeline end to end: resilient extract, clean transform, idempotent incremental load, a week of scheduled runs through an API outage, and the weekly report.

Validating Data

Stop bad data at the door. Write a contract for an API's records in JSON Schema, turn it into a pydantic model that parses and checks every record, and quarantine what fails instead of loading it or crashing.

Testing Pipelines

Write real pytest tests for pipeline code, check that they would actually catch bugs by running them against deliberately broken versions, and write dbt-style data tests that run against the tables themselves.

Monitoring & Lineage

The worst pipeline failures report success. Build the monitors that catch them (volume anomalies, late data, schema drift), then use lineage to work out what a failure broke downstream and who needs to know.

Trustworthy Immunisation Data

Thirty health facilities send monthly immunisation reports to a county health records office. Merge two exports whose schema changed mid-year, enforce a contract, keep the latest resubmission (with tests to prove it), and measure completeness and timeliness in a data-quality report.

Warehouses & Lakes

Where analytical data lives at scale. Build a small data lake of partitioned Parquet files, watch DuckDB skip the files a query doesn't need, and load a warehouse that can be rebuilt one partition at a time.

Dimensional Modelling

The shape analysts love: a fact table of events surrounded by dimensions that describe them. Model a year of mobile-money transactions as a star schema, query it, and keep history when agents change their names and towns.

Batch vs Streaming

Some questions can wait for tonight's batch; a suspicious payment can't. Learn how streaming systems like Kafka and Flink work: consume a log with offsets, survive a crash without double-counting, and count events in time windows even when they arrive late.

A Mobile-Money Analytics Pipeline

Everything in the track, end to end. Take a year of raw mobile-money transactions from export to lake, model a star schema with history, test it, and publish a daily report through an orchestrated pipeline that refuses to publish bad data.