Design an AI Video Generator Like Seedance: Architecture and Scale

The question: design an AI video generator like ByteDance's Seedance. A user types a prompt, optionally adds reference images, video or audio, and receives a realistic clip of up to about 15 seconds with camera moves, several shots and sound. Video is the most demanding generative workload in production today: a single clip can hold a GPU for many seconds or minutes, so model design, inference optimisation, long-job scheduling and safety all decide whether the product can exist at all.

What's public and what's our design. ByteDance has published detailed technical reports on Seedance's models, data and inference speed-ups, and those facts are cited throughout. It hasn't published its production serving system. The fleet, queues and delivery sections here are a reasoned design for the same kind of product, and their numbers are illustrative.

1. Clarify requirements

2. Why video is so expensive

A frame is an image, and a clip has hundreds of them, and they must also agree with each other over time. A flicker that nobody would notice in a still image becomes obvious in motion. Here's the size of the task for a 5-second clip:

Pixels       : 1920 × 1080 × 120 frames ≈ 249M (vs ~1M for one image)
Compression  : Seedance 1.0's VAE shrinks time by 4 and each side by 16,
               storing 48 channels: overall 1/64 of the raw data size
Latent tokens: at 1080p: 31 latent frames × 120 × 68 ≈ 253K tokens
               at 480p : 31 latent frames × 53 × 30 ≈ 49K tokens
Attention    : naive full attention over 49K tokens compares
               49K × 49K ≈ 2.4B pairs, at every layer and every step

3. How a video model works

Most leading video models are latent diffusion transformers, an extension of image diffusion. A compression network (VAE) turns video into a small latent. A transformer learns to turn noise into a latent, guided by text. Then the VAE decoder turns the result back into pixels. Sora's public report described this as turning video into spacetime patches that act as transformer tokens. Wan and Meta's Movie Gen publish similar designs.

Decoupling space and time makes it practical. Comparing every token with every other token is impossible at this size. Seedance 1.0 uses separate layers: spatial layers attend within one frame (and also see the text), and temporal layers attend across frames, using windows. That captures motion at a small fraction of the cost.

Flow matching. Seedance 1.0 trains with flow matching and velocity prediction: the model learns the direction from noise to data at each point, which tends to allow straighter paths and fewer steps than older diffusion formulations. For an interview, knowing that step count times per-step cost equals your GPU bill is more important than the math.

4. Inside Seedance

The Seedance 1.0 report is one of the most detailed public accounts of a top video model, so it makes a good case study.

Multi-shot generation. Older models produce one continuous shot, so a story needs several unrelated clips, each with a new character. Seedance 1.0 supports multiple shots natively. Each shot has its own caption, ordered by time, and the positional encoding tells the model which visual tokens belong with which caption.

One model for many tasks. Noisy latents are combined channel by channel with conditioning frames and a binary mask. For text-to-video, the conditioning frames are zeros. For image-to-video, the first frame is real. Seedance 1.0 mixes these in training, and raised image-to-video to 40% in continued training.

5. Data and training

A video model is largely shaped by its data pipeline. The Seedance 1.0 report describes shot-aware segmentation with a 12-second maximum clip length, removal of overlays like logos and subtitles, quality and safety filtering, semantic deduplication, rebalancing and dense bilingual captions.

Rights in data have become a major issue for this whole field, and this is covered in the safety section below. In your design, record where each training clip came from, and keep the ability to remove sources, since a future legal or licensing change may require it.

6. Inference optimisation

Seedance 1.0 reports a 5-second 1080p clip in 41.4 seconds on an NVIDIA L20, which it credits to over 10× end-to-end speedup from multi-stage distillation and system work, with no drop in model performance. The biggest gains come from reducing steps. Every other optimisation multiplies on top of that.

• Distillation: train a student model that takes larger, fewer steps; several methods can be stacked to keep quality.
• Thin VAE decoder: a smaller decoder trained to match the original, which is a 2× speedup with no visible quality loss in their tests.
• Kernel fusion and mixed precision: run the same math more efficiently on the GPU, with fine-grained low precision where it's safe.
• Multi-GPU parallelism: split attention across GPUs while cutting communication, plus sending less data with FP8.
• Async offloading: move models from GPU memory when idle, so large cascades fit, at under a 2% cost.

The VAE decoder is a bottleneck you may not expect. Turning a full latent into 120 frames at 1080p needs a lot of memory at once. Seedance splits this across GPUs by space and time, decoding overlapping tiles and blending them so no seams are visible.

7. Estimates

Jobs        : 1M/day ≈ 12/s average, ~30/s peak
GPU time    : 41.4 GPU-seconds for 5 s at 1080p (Seedance 1.0's figure)
              assume 10 s clips on average → ~83 GPU-seconds per job
Fleet       : 30 jobs/s × 83 s ≈ 2,500 GPUs busy at peak
              × 1.3 headroom ≈ 3,250 GPUs
Cost        : 3,250 × ~$2/hour × 24 ≈ $156K/day ≈ $0.16 per job
Storage     : 10 s at ~10 Mbps = ~12.5 MB; plus smaller versions
              ≈ 20 MB per job → ~20 TB/day
Reads       : most views are of a small popular share, served from a CDN
(illustrative; assumes one GPU per clip and the 1080p time for all jobs)

Compute cost can decide whether the product survives. In March 2026, OpenAI announced that it was shutting down the Sora app and API, about six months after the Sora 2 launch. OpenAI didn't give a detailed reason, but several reports linked the decision to the high cost of serving video. Do this estimate early, and check that your pricing covers it.

8. Architecture

A worked example. Dev wants: *"A red fox runs through snow at dawn, slow motion, camera tracks alongside."* At 10 seconds, 1080p, on the fast tier:

video_jobs(job_id, user_id, tier, status, stage, prompt, seed, duration_s,
           resolution, model_version, gpu_seconds_est, gpu_seconds_used,
           checkpoint_path, created_at, finished_at)
video_refs(job_id, kind[image|video|audio], storage_path, safety_status)
videos(video_id, job_id, master_path, hls_path, poster_path,
       provenance_id, visibility)

9. Scheduling long GPU jobs

A video job holds expensive hardware for a long time, so scheduling decides your costs and your wait times. Treating all jobs as equal doesn't work, because a 15-second 1080p clip costs far more than a 5-second 480p draft.

• Estimate cost in GPU-seconds from duration, resolution, step count and model. Use that for quotas, pricing and queue ordering.
• Allocate gangs: multi-GPU stages need all their GPUs at once. Without care, large jobs wait forever while small ones keep filling the gaps, so reserve capacity for them.
• Run stages separately: a base stage needs different memory than a decode stage, so run them on the pools that best fit.
• Checkpoint between stages: save the latent after each stage. A GPU failure then costs one stage, not the whole job.
• Use cheap interruptible GPUs for relaxed tiers: a preempted job restarts from its last checkpoint.

Drafts are both a product feature and a major cost control. If most people reject a video in the first second of viewing, then spending 10 seconds on a cheap draft avoids spending 80 seconds on an unwanted full render. Track the draft-to-final conversion rate, and adjust your prices with it.

10. Reference inputs and audio

Seedance 2.0, whose technical report came out in April 2026, moves to native audio-video generation from a single unified model. It accepts text, image, audio and video inputs, generates clips of 4-15 seconds at native 480p and 720p, and its open platform takes up to 3 video clips, 9 images and 3 audio clips as references. A Fast version targets low-latency use.

Native audio changes the design. If a separate step adds sound afterwards, lip movement and footsteps rarely line up. Generating both together helps synchronisation, but it raises the stakes on safety, since a generated person can now say anything. Your output safety check must then cover speech and voice, not just pictures.

11. Encoding and delivery

A finished clip is ordinary video from here on, and standard streaming practice applies. Keep the high-quality master, encode a ladder of versions, and let a CDN serve them. See CDN and object storage.

12. Safety, likeness and copyright

Realistic video makes abuse much more convincing, so this section matters as much as the GPU fleet. Seedance 2.0 shows what's at stake. After its launch in February 2026, Disney sent ByteDance a cease-and-desist letter claiming use of its characters, the Motion Picture Association publicly objected, and SAG-AFTRA objected to the use of performers' voices and likenesses. ByteDance said it would strengthen its safeguards, and reports said it suspended uploads of images of real people.

Provenance works in layers. OpenAI's Sora 2 system card described C2PA metadata on all assets, a visible moving watermark on downloads, and internal detection tools to trace content. Each has limits: metadata can be stripped, watermarks can be cropped, and detection isn't perfect. Together, they raise the bar. (OpenAI announced in March 2026 that it was shutting down the Sora app, but this layered approach remains a useful reference.)

Scanning a video is itself a cost problem. Sampling frames can miss a brief harmful moment, and scanning every frame is expensive. A common compromise is to sample densely around scene changes, and to raise the sampling rate for riskier accounts and prompts. See harmful content detection.

13. Evaluating quality

There's no single score for good video. A clip must follow the prompt, have believable motion, and look appealing. Seedance's reward models target those same dimensions, and it uses expert and public preference tests. External leaderboards such as Artificial Analysis also compare models, though these are snapshots.

14. Compared with other video models

Published models share most of their design, with different choices in size and scope. Knowing a few of them helps you discuss trade-offs, including when an open model could replace an API.

Follow-up questions to expect

• *How would you add longer videos, up to 5 minutes?* Generate shot by shot with a plan, using the last frames or a character reference to stay consistent, then join them. Quality drift and cost are the main issues.
• *Would you build your own model or use an API?* Start with an API or open model to test demand. Build only when you have scale, a data advantage and research talent, as training costs are huge.
• *A GPU dies during the refine stage. What happens?* The worker's lease expires, and the job resumes from the base checkpoint on a new worker, with no extra credits charged.
• *How do you handle a viral surge?* Slow down the relaxed tier first, lower the default resolution for free users, require drafts, and show honest wait times.
• *Generated clips show flicker. Where do you start?* Compare the outputs of each stage, as below.