Feature Engineering for ML Systems: Encoding, Scaling and Aggregates
Feature engineering for production ML: scaling, bucketing, one-hot and hashing, embeddings, feature crosses, windowed aggregates and training-serving skew.
Feature engineering turns raw data (events, profiles, text, images) into the numbers a model consumes. In production systems the hard part isn't inventing clever features; it's computing them the same way in training and at serving time, fresh enough to matter, and cheap enough to fetch within a latency budget. This guide covers the transformations interviewers expect you to name and the system choices behind them.
Feature groups to name in an interview
| Group | Examples (video recommender) | Changes |
|---|---|---|
| User | Age bucket, country, languages, account age, long-term topic interests | Slowly |
| Item | Category, duration, creator, upload age, global CTR, embedding | Slowly (stats daily) |
| Context | Time of day, day of week, device, network, page | Per request |
| User-item | Has the user watched this creator? Similarity of user and item embeddings | Per request |
| Real-time behaviour | Videos watched in the last 10 minutes, current session topics | Seconds |
Listing groups like this, then picking two or three high-value features per group, is clearer than a long flat list.
Transforming numeric features
- Scaling: min-max to [0, 1] or z-score standardisation, so features with large ranges don't dominate gradient-based models. Tree models don't need it.
- Log transform: for heavy-tailed counts (followers, views, price):
log(1 + x). - Bucketing (discretisation): turn age or distance into ranges; useful for linear models and for crossing with other features.
- Clipping: cap outliers at a percentile so one bot with a million clicks doesn't distort the feature.
- Missing values: impute with a default or median and add an
is_missingflag; missingness is often informative.
Encoding categorical features
- One-hot: one binary column per value. Simple, but the vector grows with cardinality.
- Hashing trick: hash the value into a fixed number of buckets (say 2^20). Handles unseen values and bounded memory; collisions add a little noise.
- Embeddings: a learned dense vector per value, trained with the model. The standard for ids with millions of values. See embeddings.
- Target encoding: replace a category by its historical label rate (for example a merchant's past fraud rate). Powerful but leaks the label unless computed only from earlier data.
Feature crosses and interactions
A feature cross combines features, such as country × hour_of_day or user_language × video_language, so a linear model can learn interactions. Deep models and gradient-boosted trees learn many interactions themselves, but explicit crosses are still common in ad and recommendation models (the "wide" part of wide-and-deep models).
Aggregates over time windows
Many of the strongest features are counts and rates over windows: a card's transactions in the last hour, a video's CTR over the last day, a user's watches in the last 7 days. They are computed by batch jobs for long windows and by stream processors for short ones.
Training-serving skew
Skew happens when a feature's value at serving time differs from its value in training for the same situation. Causes and fixes:
| Cause | Example | Fix |
|---|---|---|
| Two implementations | SQL in training, Java in the service; they round differently | Define each feature once; generate both paths from it (a feature store) |
| Different freshness | Training used end-of-day counts; serving reads counts 5 minutes old | Log served features and train on the logs |
| Future leakage | Training aggregates include events after the prediction | Point-in-time joins |
| Missing at serving | A feature service times out and returns 0 | Explicit defaults and missing flags; monitor null rates |
The strongest guarantee is log-and-train: record the exact feature vector used for every prediction, and build training data from those logs. What the model trains on is then exactly what it saw.
Feature selection and cost
- Each feature costs storage, a lookup at serving time, and an on-call burden when its pipeline breaks.
- Use feature importance and ablations to drop features that add little.
- Prefer features you can compute for new users and items, or cold start gets worse.
- Watch for features that encode protected attributes or their proxies; see fairness checks.
Where it’s used
Velocity features (transactions per card per minute, new device, distance from the last transaction) from a stream, joined with long-term merchant risk from batch.
Billions of sparse id features hashed or embedded, with crosses between user segments, ad categories and placements.
Real-time traffic and restaurant load from streams, plus historical route and kitchen times from batch.
In an interview
- Group features into user, item, context, interaction and real-time, with two examples each.
- Choose encodings by cardinality: one-hot, hashing or embeddings.
- Say which features come from batch, stream or the request, and their freshness.
- Explain how you prevent training-serving skew (one definition, point-in-time joins, logged features).
Frequently asked questions
How do you encode high-cardinality categorical features?
Use the hashing trick to map values into a fixed number of buckets, or learn embeddings for each value. One-hot encoding is only practical for features with a small number of values.
What is training-serving skew?
A difference between a feature's value in training and at serving time for the same situation, caused by separate implementations, different freshness or future leakage. Prevent it by defining features once, using point-in-time joins, and training on logged serving features.
Further reading
- Feature Hashing for Large Scale Multitask LearningWeinberger et al., ICML 2009
- Wide & Deep Learning for Recommender SystemsCheng et al., Google, 2016
- Rules of Machine LearningGoogle for Developers