Object Storage and Blob Stores

Object storage stores files as objects (data plus metadata) in flat buckets, addressed by a key and accessed over HTTP. Amazon S3, Google Cloud Storage and Azure Blob Storage are the best-known services. It is the default home for anything big and unstructured: images, videos, backups, logs, ML datasets and static websites.

The problem with files and databases

A video service stores millions of videos, each megabytes to gigabytes; Netflix-scale catalogs hold on the order of 100 million video files, and YouTube hosts billions of thumbnails. Where should these bytes live?

• File systems don't scale out. Directories like /videos/2024/01/video1.mp4 must be traversed, and "list all January uploads" scans thousands of subdirectories; file systems weren't built for billions of files. One server holds limited storage, adding servers needs manual partitioning, replication is complex, and clients must know which server holds a file.
• Databases aren't built for big binaries. You can store BLOBs, but a 1 GB video in PostgreSQL strains memory, makes backups enormous and slows replication. Databases are optimized for structured queries.
• Global users need nearby copies. A user in Tokyo requesting a video stored in a US data center sees high latency, and file systems have no built-in global distribution.

How object storage works

Each file becomes an independent object in a flat namespace: no directories, no hierarchy. An object has three parts: the data (the bytes), metadata (content type, size, upload date, custom tags) and a unique key such as videos/paris.mp4.

{
  "ObjectKey": "videos/paris.mp4",
  "Metadata": { "ContentType": "video/mp4", "UploadDate": "2024-01-15", "Duration": "120s" },
  "Data": "<binary video data>"
}

• Buckets are top-level containers: create my-videos and store millions of objects in it, each with a key unique within the bucket.
• No real folders: images/2024/cat.jpg looks like a path but is just a string; no /images/ directory exists. Lookups by key use hash-table-like indexes, so performance stays flat from 1,000 objects to a billion.
• RESTful HTTP API: upload with PUT, download with GET, delete with DELETE, e.g. https://storage.example.com/my-videos/paris.mp4. The API hides the distributed storage behind it, and CDNs can cache objects at edge locations for low latency worldwide.
• Built-in durability: an upload is automatically copied to several servers (often 3+) across availability zones; if one fails, another copy serves the file. Services such as S3 are designed for 99.999999999% (11 nines) annual durability.
• Versioning: overwrite report.pdf or delete it and the old version is kept, protecting against application bugs and user mistakes.
• Lifecycle policies: move objects to cheaper tiers automatically (say after 90 days) or delete them (say after a year).
• Consistency: S3 now offers strong read-after-write consistency (upload, then read immediately); some systems are eventually consistent, which is cheaper but may briefly return stale data.

When to use object storage

• Multimedia: YouTube stores billions of videos and Netflix serves petabytes of streaming content; each file is an object with an HTTP URL browsers can fetch directly, no application server in the path.
• Backups and disaster recovery: terabyte-scale database backups stored cheaply, with geographic replication against data-center loss. See disaster recovery.
• Static website hosting: upload HTML, CSS, JavaScript and images and let the bucket serve them; cheaper than running web servers.
• Data lakes: IoT sensors produce millions of events a day; store raw JSON or CSV as objects and let Spark or a query engine read them in place, separating storage from compute.

Don't use it for frequently updated files, appends or low-latency small reads and writes; use a database or block storage for those.

import boto3

s3 = boto3.client("s3")
s3.upload_file("video.mp4", "my-bucket", "videos/video.mp4")              # upload
s3.download_file("my-bucket", "videos/video.mp4", "local-video.mp4")      # download

# Share it for 1 hour without making the bucket public
url = s3.generate_presigned_url("get_object", Params={"Bucket": "my-bucket", "Key": "videos/video.mp4"}, ExpiresIn=3600)

s3.delete_object(Bucket="my-bucket", Key="videos/video.mp4")              # delete

Object vs block vs file storage

Key properties

• Durability: objects are stored redundantly (replicas or erasure coding across devices and data centers). S3 is designed for 99.999999999% (eleven nines) durability.
• Flat namespace: photos/2026/10/abc.jpg is one key; the slashes only look like folders. Listing by prefix emulates directories.
• Metadata: content type, custom tags and user-defined metadata travel with each object.
• Strong read-after-write consistency: S3, GCS and Azure now return the new object immediately after a successful write.
• Storage classes: hot, infrequent-access and archive tiers (S3 Glacier) trade retrieval speed for much lower cost; lifecycle rules move or delete objects by age automatically.

Uploads and downloads without your servers

Streaming large files through your application servers wastes bandwidth and ties up workers. Instead, issue presigned URLs: short-lived URLs signed with your credentials that let a client upload or download one specific object directly.

• Multipart upload splits large files into parts uploaded in parallel and retried individually; essential for video and multi-GB files.
• Event notifications (object created) trigger processing: thumbnails, transcoding, virus scanning, indexing. See message queues.
• Serve downloads through a CDN with the bucket as origin, using signed URLs or cookies for private content.

Design patterns

• Store bytes in object storage, metadata in a database: the database row holds the object key, owner, size, status and permissions; never store large blobs in the database itself.
• Content-addressed keys: name objects by a hash of their content to deduplicate identical uploads (common in backup and file-sync systems).
• Immutability: write new keys instead of overwriting; enable versioning for accidental-delete protection.
• Data lakes: store analytics data as Parquet files in buckets and query them with Spark, Trino or Athena.

Object storage is not a file system. Renaming a "folder" means copying every object, appends require rewriting the object, and small random reads are slow. Don't put a database's data files on it.