Feature Stores: Online vs Offline Features and Training-Serving Skew

A feature store is the system that computes, stores and serves the input features for machine learning models, so the same feature definitions are used when building training data and when making live predictions. It usually has an offline store for large historical datasets and an online store for low-latency lookups at prediction time, plus a registry of feature definitions. Its main job is preventing training-serving skew: a model trained on one version of a feature and served another.

Why teams build one

• Training-serving skew: a data scientist computes "orders in the last 7 days" in a notebook with SQL; an engineer reimplements it in the serving path in Java with a subtly different window. The model performs well offline and poorly in production.
• Point-in-time leakage: a training row for a prediction made on 1 March accidentally includes data from 5 March. The model learns from the future and offline metrics lie.
• Duplicated work: five teams compute the same customer features five ways.
• Latency: live prediction needs features in milliseconds, which the warehouse can't serve.

Architecture

Point-in-time correct training data

To build a training set, you start from labelled events, each with an entity key and a timestamp (a transaction and whether it turned out to be fraud). For each event, you need each feature's value as of that timestamp, not today's value. That's a point-in-time ("as of") join: for every event, take the latest feature row whose timestamp is at or before the event time.

-- For each labelled transaction, the customer's 7-day order count as it was at that moment
select t.transaction_id, t.label, f.orders_7d
from labelled_transactions t
left join lateral (
  select orders_7d from customer_features f
  where f.customer_id = t.customer_id and f.feature_time <= t.event_time
  order by f.feature_time desc
  limit 1
) f on true;

Also respect when a feature *became available*, not only when it describes. A daily batch feature for 1 March that lands at 04:00 on 2 March wasn't available to a prediction at 23:00 on 1 March.

Batch, streaming and on-demand features

Streaming features need the same windows and watermarks care as any stream job, and the offline store must record the same values the online store served, or backfill them identically.

Serving considerations

• Latency budget: a model call with a 50 ms budget might allow 5-10 ms for feature lookups, so batch lookups by entity key in one round trip.
• Freshness and defaults: define what the model gets when a feature is missing or stale, and train with the same default.
• Log what was served: logging features at prediction time gives exact training data for the next model and makes skew measurable.
• Hot entities: a popular item or merchant can become a hot key in the online store; cache it.