Build streaming and batch pipelines that survive node failures, replays and replication lag — and know which numbers you can trust.
Your live dashboard says 4.1 million events today; the nightly job says 4.06 million, and nobody can say which one is right. A consumer restarts and replays 200,000 records into a sink that was never idempotent. A replica three time zones away is nine seconds behind, and a customer noticed before your alerts did. These are the failures that arrive once data outgrows a single database, and more servers do not fix them. You work through each one in runnable code: splitting write and read models with CQRS (Command Query Responsibility Segregation), then streaming every row change out of PostgreSQL with CDC (change data capture) through a logical replication slot. From there come Kafka partitions, offsets and exactly-once semantics; sagas and the outbox pattern when one transaction crosses three services; hypertables for metrics arriving faster than an ordinary table accepts them; presigned uploads to object storage; HLS video segmentation; multi-region writes where CAP (consistency, availability, partition tolerance) stops being theory and CRDTs (conflict-free replicated data types) merge the conflicts; Lambda versus Kappa pipelines with watermarks and windows; and PostgreSQL internals for the night EXPLAIN stops explaining. The capstone sustains 10,000 events per second straight through a simulated node failure.
Built by Lakshya Kumar
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
Tail the database’s own write log so every insert, update and delete reaches downstream systems in order, seconds after it lands.
An append-only partitioned log lets you reprocess yesterday for free and survive a consumer crash without dropping a message.
Two-phase commit blocks the moment a coordinator dies — compensation and idempotency keys keep money and inventory in step instead.
Keep billions of metric points queryable in milliseconds using partitioning, compression and pre-computed rollups.
Send uploads and downloads straight to S3 with presigned URLs, then cut the bill with versioning and lifecycle rules.
Segment, transcode and range-serve video so a viewer on a train drops to a lower bitrate instead of staring at a spinner.
Route people to their nearest region, settle conflicting writes, fail over without a human, and still obey data-residency law.
When the real-time dashboard and the nightly job disagree, watermarks and window semantics decide which one you ship.
Find out why a query with a perfect plan still crawls: dead row versions, autovacuum falling behind, pages that never reach cache.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Design and implement a production-grade data platform that ingests events from at least two sources via Kafka, stores them in a time-series store and a relational store using a CQRS pattern, exposes a read API backed by materialized views, and includes a Kappa-style analytics pipeline that computes rolling 1-minute and session-level aggregations. The platform must handle at least 10,000 events/second, survive a simulated node failure without data loss (demonstrated via WAL replay or Kafka offset replay), and produce an ops runbook covering vacuum, index maintenance, and upgrade decisions.
This is a Data-Intensive Systems course for experienced builders who already write production code. Every task involves real, runnable code — no pseudocode, no toy examples. When helping with streaming tasks, use Kafka client libraries (kafka-go, confluent-kafka, rdkafka, kafkajs) and real Kafka semantics (offsets, consumer groups, watermarks). When helping with database internals tasks, use actual PostgreSQL system views (pg_stat_user_tables, pg_statistic, pg_buffercache, pgstattuple) and real SQL — never simulated or mocked DB behavior. For distributed systems tasks, be precise about CAP trade-offs, consistency levels, and failure modes; avoid hand-waving. If a builder asks why their metric disagrees between streaming and batch, walk through the watermark and window semantics before suggesting a code fix.
Build a Kafka-driven pipeline that guarantees exactly-once semantics from producer to sink. Include idempotent producer config, transactional consumer-to-producer chain, and a chaos test that kills consumers mid-batch and verifies no duplicates and no losses in the sink store.
Build a CDC pipeline that replicates Postgres changes (using logical replication or Debezium) to a columnar warehouse (Snowflake, BigQuery, or DuckDB). Handle schema evolution, large transaction batches, and a backfill from existing data. Validate row-count parity over a 24-hour run.
Set up an Apache Iceberg-based lakehouse on S3 (or Minio): ingest 100M rows, run schema evolution (add/drop columns), demonstrate time travel via snapshot queries, and benchmark query performance vs parquet-only baseline. Document the metadata layer.
Build a feature store for an ML use case: batch features in offline store (S3 + Parquet), real-time features in online store (Redis), a pipeline that backfills offline -> online, and a serving layer that returns features at P95 < 5ms. Include feature versioning and rollback.
Goes deeper than Kleppmann on storage engine internals — essential companion for Module 10.