Time-Series Databases

A time-series database (TSDB) is optimised for data that is a sequence of timestamped measurements: CPU usage every 10 seconds, a sensor reading every second, a stock price on every trade. These workloads are append-heavy, read in time ranges, and lose value with age, so TSDBs are built very differently from general-purpose databases.

What makes time-series data special

• Writes vastly outnumber reads, and almost all writes are appends of new points; old points are rarely updated.
• Queries are over time windows and aggregates: "average latency per minute for the last 6 hours, grouped by region".
• Recent data is hot, old data is cold and can be summarised.
• Huge volume: 10,000 servers × 200 metrics every 10 seconds is 200,000 points per second.

The data model

A series is identified by a metric name plus labels (tags), and holds (timestamp, value) points:

http_requests_total{service="checkout", region="ap-south-1", status="500"}
  1792123200  1532
  1792123210  1541
  1792123220  1555

Every unique combination of label values is a separate series. Cardinality, the number of series, is the main scaling constraint: a label like user_id with millions of values creates millions of series and can overwhelm a TSDB. Keep high-cardinality identifiers out of metric labels and put them in logs or traces instead.

How TSDBs are built

• Time-partitioned storage: data is written into chunks or blocks per time window (for example 2 hours), so a query only opens blocks in its range and old blocks are dropped cheaply.
• Compression: consecutive timestamps are evenly spaced and consecutive values similar, so delta-of-delta encoding for timestamps and XOR encoding for floats (from Facebook's Gorilla) shrink points to around 1–2 bytes each.
• Append-optimised writes: LSM-style or columnar layouts. See database indexing.
• An inverted index of labels to find all series matching service="checkout" quickly.

Keeping storage bounded: retention policies and rollups

Storing raw 10-second data forever is rarely worth it. A retention policy drops raw data after days or weeks, and downsampling keeps aggregates (min, max, average, count, percentiles via sketches) at coarser resolution for longer. Storing min and max, not just the average, keeps short spikes visible in long-range graphs.

Time-series patterns you can build anywhere

Pre-computed aggregates (see materialized views) are themselves time series, and they grow forever. Three patterns keep them fast and bounded, whether you use a TSDB or plain Redis.

Bucketing on fixed time boundaries

Store each aggregate under a key that includes its aligned time boundary:

metric:page_views:minute:2024-01-15T10:05:00
metric:page_views:minute:2024-01-15T10:06:00
metric:page_views:minute:2024-01-15T10:07:00

"Views in the last 10 minutes" is exactly 10 key reads: no scanning, no filtering, and you know which keys to fetch before reading anything. Arbitrary timestamps would force a scan-and-filter; aligned buckets turn range queries into point lookups.

Rollups: coarser as data ages

A year of 1-second data is about 31 million points per metric, yet nobody zooms into one second from six months ago. A rollup aggregates fine buckets into coarser ones as data ages:

1. A background job reads the 60 one-minute buckets for 10:00–10:59.
2. It computes sum, average, min, max and count into one 1-hour bucket.
3. It writes the hourly bucket.
4. The minute buckets are deleted or left to expire.

Run minute→hour rollups hourly and hour→day rollups daily, after late data for the period has arrived. Keep the aggregates your queries need: sums for totals; sum and count (not just the average) for latency, so averages stay correct across rolled-up periods; min and max for peaks and alerting.

Retention with TTLs

Many stores (Redis, most TSDBs, some NoSQL databases) expire keys automatically. Set TTLs per tier: minute buckets 24 hours, hour buckets 30 days, day buckets a year or forever, and cleanup needs no deletion jobs. Tune from query logs: if nobody reads minute-level data older than 2 hours, shorten that TTL and save storage with no user impact.

import redis, time
r = redis.Redis()

def record(metric: str, value: float, ts: float):
    minute = int(ts) // 60 * 60
    key = f"metric:{metric}:minute:{minute}"
    pipe = r.pipeline()
    pipe.hincrbyfloat(key, "sum", value); pipe.hincrby(key, "count", 1)
    pipe.expire(key, 24 * 3600)                     # tier 1: keep minutes for a day
    pipe.execute()

def rollup_hour(metric: str, hour_start: int):      # run hourly, after late data settles
    keys = [f"metric:{metric}:minute:{hour_start + 60 * i}" for i in range(60)]
    rows = [r.hgetall(k) for k in keys]
    total = sum(float(x.get(b"sum", 0)) for x in rows)
    count = sum(int(x.get(b"count", 0)) for x in rows)
    hkey = f"metric:{metric}:hour:{hour_start}"
    r.hset(hkey, mapping={"sum": total, "count": count})
    r.expire(hkey, 30 * 24 * 3600)                  # tier 2: keep hours for 30 days

record("latency_ms", 182.0, time.time())
# average latency for an hour = sum / count, still exact after the rollup

High-cardinality dimensions

"Page views by user" with millions of users means millions of keys per bucket, mostly holding 1 or 2. Use approximations instead (see probabilistic data structures):

• HyperLogLog for distinct counts: "unique users today" within a few percent using fixed memory (Redis uses about 12 KB for roughly 0.8% error), whether the answer is 1,847,293 or 1,852,106.
• Count-min sketch for approximate frequencies ("how often did each URL appear?") without storing every distinct value.
• Sampling: keep 1% of events in full detail and use aggregates for the rest; statistically valid for large-scale patterns.

For dashboards, "roughly 1.8 million users" is as useful as the exact number, at a tiny fraction of the storage.

Options