Autoscaling

Autoscaling adjusts capacity automatically to match demand: adding instances when load rises and removing them when it falls. Done well, it keeps latency steady through daily peaks and sudden spikes while avoiding paying for idle machines at night. Done poorly, it reacts too late, oscillates, or scales the wrong tier.

Load balancers spread traffic over instances, but what if traffic is bursty or uneven over time? A social network spikes during a big sports match; a store spikes on Black Friday. You could run enough instances for the maximum possible traffic all the time, but that wastes money most of the day. Autoscaling instead changes the number of compute resources (usually VMs or containers) automatically, based on current or predicted demand, working hand in hand with the load balancer so the application stays responsive and cost-efficient.

How autoscaling works

It starts with continuous monitoring of the application's metrics: CPU utilization, memory, network traffic, request count, and any custom metrics you define. Rules or policies decide when to act: when demand rises the system scales out (adds resources), and when it falls it scales in (removes them).

Horizontal vs vertical

• Horizontal autoscaling adds or removes instances (pods, VMs, containers). It requires stateless services behind a load balancer and is the default for web tiers. See scalability.
• Vertical autoscaling changes CPU and memory of existing instances. Useful for workloads that can't scale out easily (some databases, single-process jobs), but usually means a restart.
• Cluster autoscaling adds or removes the underlying nodes when pods can't be scheduled or nodes sit empty.

How a scaling loop works

Most autoscalers use target tracking: keep a metric near a target value, with desired capacity ≈ current capacity × (current metric ÷ target).

Auto Scaling Groups

An Auto Scaling Group (ASG) is a logical group of identical instances that the platform keeps at the right size, even replacing instances that become unhealthy. You configure five things:

1. Desired capacity: how many instances should be running now.
2. Minimum and maximum: the bounds the group may scale between.
3. Health checks: the group monitors each instance and automatically replaces failed ones.
4. Scaling policies: the conditions for scaling, for example "add an instance when CPU > 70%".
5. Launch template: the configuration of new instances: machine type, image, network and startup script.

Reactive vs predictive scaling

• Reactive scaling responds to current demand using real-time metrics. Best for unpredictable workloads.
• Predictive scaling uses historical data (often machine learning) to forecast demand and scale before it arrives. Best for predictable patterns such as daily or weekly cycles.

A classic interview question: how do you handle peak traffic on Black Friday? Use predictive (or scheduled) scaling to scale out before the sale starts, using last year's traffic to size it, and scale back in after the sale ends. Reactive scaling alone would lag behind the opening rush.

Benefits

• Cost: you pay only for the capacity you need, with automatic reduction when demand is low.
• Availability: performance holds through spikes, and unhealthy instances are replaced automatically.
• Consistent user experience: latency stays low even at peak.
• Less operational work: no one has to add and remove servers by hand.

Working with the load balancer

Autoscaling and load balancers work as a pair. When a new instance launches, the autoscaler registers it with the load balancer, which starts sending it traffic once it passes health checks. When an instance is removed, the system first drains it, letting existing connections finish before deregistering and terminating it, so no requests are dropped.

Choosing the right metric

Scaling the wrong tier makes things worse. If latency is high because the database is saturated, adding app servers adds more database connections and load. Know where the bottleneck is before automating.

The lag problem

Capacity arrives late: the metric has to cross the threshold (often averaged over minutes), the autoscaler acts, instances boot (seconds for containers, minutes for VMs), applications warm caches and JIT compilers, and readiness checks pass. A spike that doubles traffic in 30 seconds will hurt before new capacity helps. Mitigations:

• Headroom: target 50–60% utilisation, not 90%, so existing capacity absorbs the first minutes of a spike.
• Scale out fast, scale in slowly: aggressive scale-up policies, conservative scale-down with cooldowns to avoid oscillation (flapping).
• Scheduled scaling for known peaks (business hours, a sale at 8 pm, a live event).
• Predictive scaling from historical patterns, scaling ahead of the daily curve.
• Warm pools of pre-initialised instances, and fast startup (small images, lazy initialisation).
• Load shedding to protect the system during the gap. See load shedding.

Limits and safety

• Set minimum instances (for availability and to avoid cold starts) and maximum instances (to cap cost and protect downstream dependencies from a scale-out stampede).
• Check account quotas and dependency limits (database connections, third-party rate limits) for the maximum scale.
• Make scale-in safe: drain connections and finish in-flight work before terminating. See health checks.

Best practices

• Scale on metrics that truly reflect the application's health and load, not on whatever is easiest to measure.
• Avoid scaling too often: add cooldown periods between scaling actions so the group doesn't oscillate up and down.
• Test your policies by simulating different load patterns and checking the scaling behaves as expected.
• Watch costs: set billing alerts and review scaling patterns regularly to keep utilization efficient.

More than servers

VMs and containers are the most common things to autoscale, but modern clouds autoscale much more of the stack:

• Databases: adding or removing read replicas to spread query load, growing storage as data grows, resizing connection pools, and adjusting compute (for example serverless database tiers).
• Network: load balancers scale their own capacity with traffic; bandwidth and connection limits flex with demand; API gateways can adjust rate limits to usage and service capacity.
• Serverless and event processing: function concurrency scales automatically with incoming requests; stream and queue processors scale on queue depth; worker pools grow and shrink with the workload. See serverless.

Cloud implementations

The core functionality is similar everywhere; the platforms differ in their integrations and extra features.