Large File Uploads: Chunking, Resumable Uploads and Deduplication

Uploading large files reliably means not sending them as one giant request. A robust design splits files into chunks, uploads chunks directly to object storage with presigned URLs, tracks which chunks arrived so an interrupted upload resumes instead of restarting, verifies integrity with checksums, and often deduplicates chunks that the system already has. This is the core of cloud storage products like Dropbox and Google Drive, and of video upload pipelines.

Why a single request fails

• A 4 GB upload over a mobile network will be interrupted, and restarting from zero each time may never finish.
• Proxying gigabytes through application servers ties up their connections and bandwidth for no benefit.
• Request size and timeout limits on load balancers and gateways reject large bodies.
• Without integrity checks, a corrupted transfer is stored silently.

The chunked upload flow

1. Create a session: the client tells the API the file's name, size and hash; the API records an upload in the metadata database with status uploading and returns an upload ID.
2. Upload chunks (often 5-100 MB) directly to object storage using presigned URLs or a multipart upload API, several in parallel. See object storage.
3. Track progress: the service (or the storage system) knows which parts completed, so after a network drop the client asks which chunks are missing and uploads only those.
4. Complete: the client sends the list of parts; the service verifies them, commits the object, marks the file available, and triggers processing (thumbnails, virus scans, transcoding).
5. Clean up: abandoned sessions expire and their parts are deleted after a timeout.

Integrity

• Send a checksum with each chunk so storage rejects corrupted parts.
• Verify the whole file's hash on completion against what the client declared.
• Store the hash in metadata so later downloads and sync can be verified too.

Deduplication and sync

Cloud storage systems split files into chunks and identify each chunk by its content hash. Before uploading, the client asks which hashes the server already has, and uploads only new chunks. A file is stored as a list of chunk hashes in the metadata database.

• Fixed-size chunks are simple, but inserting one byte at the start of a file shifts every chunk boundary, so nothing deduplicates.
• Content-defined chunking uses a rolling hash to choose boundaries based on the content itself, so an insertion changes only nearby chunks. Editing a large document then uploads a few chunks, not the whole file.
• Sync: a file version is a new list of chunk hashes; devices compare lists and download only missing chunks. Merkle-style hashes over directories find changes quickly.
• Privacy: cross-user deduplication can leak whether someone else has a file; many systems deduplicate only within one account, or encrypt per user.

Metadata and access

• Keep file metadata (owner, name, folder, version, chunk list, size, hash, status) in a database, separate from the bytes in object storage.
• Serve downloads through short-lived presigned URLs or a CDN, after an authorisation check.
• Version files by keeping old chunk lists, which gives history and restore almost for free.