Design an Ad Click Aggregator (Event Counting at Scale)

Prompt

Capture ad click events (billions/day), and serve aggregated metrics — clicks per ad per minute/hour/day — to advertiser dashboards with near-real-time freshness and accurate historical rollups. Money depends on the counts, so they must eventually be exact.

1. Requirements

Functional: record each click; query aggregates by ad and time bucket (minute → day); top-N ads. Non-functional: very high write throughput; low-latency reads of pre-aggregated data; idempotency (a click counted once despite retries/duplicates); fast path can be approximate, but historical totals must be exact (billing).

2. Estimate

~10B clicks/day ≈ 115k writes/s average, peaks several × that. Raw events at ~100 bytes ≈ ~1 TB/day — store raw for reconciliation, but never aggregate by scanning raw per query.

3. Architecture

clients ─▶ ingestion API ─▶ [ Kafka ]
                              partitioned by adId
                                   │
                       ┌───────────┴───────────┐
                       ▼ (speed layer)          ▼ (batch layer)
              stream processor            raw event store (S3)
              (Flink/Spark): windowed     │ periodic exact
              counts, ~seconds fresh      │ recompute (reconcile)
                       │                  │
                       ▼                  ▼
                 OLAP / time-series store  ◀── corrected totals
                       │
                       ▼
               advertiser dashboard (read pre-aggregated buckets)

The read path never touches raw events — it reads pre-aggregated minute/hour/day buckets from an OLAP/time-series store. All the work is on write.

4. The decisions that matter

• Idempotency / dedupe: each click carries a unique event_id; the processor drops duplicates (a dedupe window keyed on event_id) so retries and at-least-once delivery from the queue don't inflate counts.
• Windowing: aggregate into tumbling windows (fixed minute buckets), then roll minutes → hours → days. Emit on window close.
• Late events & watermarks: a click can arrive after its window closed (mobile retry, clock skew). Use watermarks to wait a bounded grace period, then either fold late events into a correction or let the batch layer fix them.
• Hot partitions (skew): a viral ad concentrates traffic on one Kafka partition / one aggregator key. Mitigate by salting the key (adId#shard) to spread load, then summing the shards at read time.
• Approximate fast path: for unique users use HyperLogLog; for top-N use count-min sketch. Cheap and fast; the batch layer provides the exact number later.

5. Lambda vs Kappa

• Lambda: a fast speed layer (approximate, seconds) + a slow batch layer (exact, scans raw, overwrites the speed numbers). Two code paths to maintain.
• Kappa: one streaming pipeline; "recompute" = replay the Kafka log. Simpler, now the common default.

Either way the principle is the same: serve fast/approximate immediately, then reconcile to exact from durable raw events — the answer to "what if the stream processor double-counts or drops?"

Design drills