Real-Time Analytics Architecture: Stream to Cache, TSDB and Hot/Cold

Stream processors, materialized views and time-series storage are the parts. How do they fit into a working analytics system? An operational dashboard needs different infrastructure from a BI platform for ad-hoc questions. Three architectures cover most needs: stream to cache, time-series database, and hybrid hot + cold.

Pattern 1: stream to cache

1. Events are published to a broker such as Kafka.
2. A stream processor aggregates by window and dimension.
3. Results go to Redis with TTL-based expiry.
4. The dashboard reads Redis on each refresh.

Example: an ops dashboard of request counts and error rates per minute per service. The processor computes (service, minute) → count, Redis serves it, and queries return in single-digit milliseconds. Limits: Redis is memory-bound, so everything must fit in RAM; high-cardinality dimensions (per-user metrics) exhaust it; and you can only answer what you pre-computed ("p95 latency by region and device?" has no answer without reprocessing). Best for low-cardinality operational dashboards, simple aggregations and minimal infrastructure.

Pattern 2: time-series database

Time-series databases are built for exactly this: they compress timestamped data very well (often roughly 10–20× better than general-purpose databases for this workload, depending on the data), make "CPU for the last hour" a native query, and include rollup and retention policies. Because data lives on disk they hold far more dimensions than Redis. Limits: less flexible than general SQL (complex joins and ad-hoc exploration are harder) and one more system to operate, back up and capacity-plan. Best for infrastructure monitoring, application metrics and IoT sensor data, where queries follow time ranges and a fixed set of dimensions.

Pattern 3: hybrid (hot + cold)

The hot path optimizes for latency (pre-computed aggregates in memory, millisecond reads); the cold path for flexibility (raw data, full SQL, any question given enough query time). Trade-offs: two systems to run, and the paths may briefly disagree (data reaches Redis before the warehouse, or the reverse). That eventual consistency is usually fine for analytics, but show "as of" timestamps and reconcile periodically to catch drift. Best for organizations needing both operational monitoring and business analytics (conversion funnels, cohort analysis). See OLTP vs OLAP.

Choosing and evolving

Start simple. A common path: stream-to-cache for operational metrics; when analysts want history, add a cold path into a warehouse while the hot path keeps serving dashboards, and you've grown into the hybrid pattern naturally.

• Late data: don't block results waiting for stragglers; show "as of" times. "Roughly 500 errors per minute" is already actionable. See watermarks.
• Hot–cold consistency: the hot path may say 1,234 requests for 10:00–10:05 and the warehouse 1,241 once late data lands. Usually fine; exact consistency needs a single path, meaning slower hot results or a less flexible cold side.
• Cardinality: keep hot-path dimensions few. Per-user metrics for millions of users belong in the warehouse, where storage is cheap and query time acceptable.

These pieces compose. A production system might stream database changes with CDC, process them in Flink, serve live dashboards from Redis and keep history in a warehouse, choosing each part by latency, flexibility and operational cost.