Build streaming and batch pipelines that survive node failures, replays and replication lag — and know which numbers you can trust.
Your live dashboard says 4.1 million events today; the nightly job says 4.06 million, and nobody can say which one is right. A consumer restarts and replays 200,000 records into a sink that was never idempotent. A replica three time zones away is nine seconds behind, and a customer noticed before your alerts did. These are the failures that arrive once data outgrows a single database, and more servers do not fix them. You work through each one in runnable code: splitting write and read models with CQRS (Command Query Responsibility Segregation), then streaming every row change out of PostgreSQL with CDC (change data capture). From there come Kafka partitions, offsets and exactly-once semantics; sagas and the outbox pattern when one transaction crosses three services; hypertables for metrics arriving faster than an ordinary table accepts them; multi-region writes where CAP stops being theory and CRDTs (conflict-free replicated data types) merge the conflicts; Lambda versus Kappa pipelines with watermarks and windows; and PostgreSQL internals for the night EXPLAIN stops explaining. The capstone sustains 10,000 events per second straight through a simulated node failure.
Built by Lakshya Kumar
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
Tail the database’s own write log so every insert, update and delete reaches downstream systems in order, seconds after it lands.
An append-only partitioned log lets you reprocess yesterday for free and survive a consumer crash without dropping a message.
Two-phase commit blocks the moment a coordinator dies — compensation and idempotency keys keep money and inventory in step instead.
Keep billions of metric points queryable in milliseconds using partitioning, compression and pre-computed rollups.
Send uploads and downloads straight to S3 with presigned URLs, then cut the bill with versioning and lifecycle rules.
Segment, transcode and range-serve video so a viewer on a train drops to a lower bitrate instead of staring at a spinner.
Route people to their nearest region, settle conflicting writes, fail over without a human, and still obey data-residency law.
When the real-time dashboard and the nightly job disagree, watermarks and window semantics decide which one you ship.
Find out why a query with a perfect plan still crawls: dead row versions, autovacuum falling behind, pages that never reach cache.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Design and implement a production-grade data platform that ingests events from at least two sources via Kafka, stores them in a time-series store and a relational store using a CQRS pattern, exposes a read API backed by materialized views, and includes a Kappa-style analytics pipeline that computes rolling 1-minute and session-level aggregations. The platform must handle at least 10,000 events/second, survive a simulated node failure without data loss (demonstrated via WAL replay or Kafka offset replay), and produce an ops runbook covering vacuum, index maintenance, and upgrade decisions.
This is a Data-Intensive Systems course for experienced builders who already write production code. Every task involves real, runnable code — no pseudocode, no toy examples. When helping with streaming tasks, use Kafka client libraries (kafka-go, confluent-kafka, rdkafka, kafkajs) and real Kafka semantics (offsets, consumer groups, watermarks). When helping with database internals tasks, use actual PostgreSQL system views (pg_stat_user_tables, pg_statistic, pg_buffercache, pgstattuple) and real SQL — never simulated or mocked DB behavior. For distributed systems tasks, be precise about CAP trade-offs, consistency levels, and failure modes; avoid hand-waving. If a builder asks why their metric disagrees between streaming and batch, walk through the watermark and window semantics before suggesting a code fix.
Build a Kafka-driven pipeline that guarantees exactly-once semantics from producer to sink. Include idempotent producer config, transactional consumer-to-producer chain, and a chaos test that kills consumers mid-batch and verifies no duplicates and no losses in the sink store.
Build a CDC pipeline that replicates Postgres changes (using logical replication or Debezium) to a columnar warehouse (Snowflake, BigQuery, or DuckDB). Handle schema evolution, large transaction batches, and a backfill from existing data. Validate row-count parity over a 24-hour run.
Set up an Apache Iceberg-based lakehouse on S3 (or Minio): ingest 100M rows, run schema evolution (add/drop columns), demonstrate time travel via snapshot queries, and benchmark query performance vs parquet-only baseline. Document the metadata layer.
Build a feature store for an ML use case: batch features in offline store (S3 + Parquet), real-time features in online store (Redis), a pipeline that backfills offline -> online, and a serving layer that returns features at P95 < 5ms. Include feature versioning and rollback.
Goes deeper than Kleppmann on storage engine internals — essential companion for Module 10.