A pre-drawn architecture is already failing. Talk through what broke and how you would fix it with the AI interviewer.
Drills follow the same access as their parent question in the library — free rounds stay free; Plus rounds need Plus to start.
Or start any drill directly — grouped by parent question.
During a marquee sports event, click tracking latencies surge to over 10 seconds. Ingestion gateways report HTTP 503 backpressure errors, while total system thr…
Advertiser dashboards display sudden 2x-3x spikes in aggregate clicks during a 10-minute window, resulting in false budget exhaustion alerts and billing complai…
Finance reports a persistent 5% discrepancy between daily customer billing statements and live dashboard telemetry, leading to publisher payment disputes.…
Finance reports a 3× gap between predicted click volume and billed clicks for several large advertisers. Reliability charts show average pCTR at 0.9% while obse…
Ad request p99 jumped from 45 ms to 220 ms during a major campaign launch. One parameter-server shard pegged CPU at 100% while others sat idle; feature store ti…
Revenue per mille dropped 18% for 36 hours after a meme creative went viral. Offline AUC still looks healthy but live calibration error doubled and under-delive…
Launch day reports show new campaigns stuck below 200 impressions despite high budgets. Auction simulators show their ads ranked last because pCTR estimates sit…
Feed ad load p99 exceeded 120 ms after enabling richer cross features. Traces show dozens of serial RPCs to the feature store before GPU batch inference even st…
P99 answer latency jumped from 1.5s to 14s. LLM latency remains low, but web search and page fetch threads are starving the application pool.…
User reports show answers containing fabricated statistics attributed to real source links, causing factual drift in answers.…
All customer API requests are failing with HTTP 504 Gateway Timeout or timing out after 10 seconds. The control plane DB went offline during routine maintenance…
100% of global API traffic is dropping with HTTP 500 errors immediately after an operator saved a updated route pattern in the management portal.…
Global gateway P99 request latency spiked from 4ms to 650ms. CPU utilization on the centralized Redis cluster reached 100%, causing request timeouts for all ten…
p99 autocomplete query latency spiked from 4ms to 850ms on single-character queries during morning traffic peaks. Application CPU utilization hit 100%.…
Breaking news search queries are missing from autocomplete suggestions. Stream processing metrics show rising consumer lag and backpressure.…
Every night at 02:00 UTC when the daily batch index deploys, search servers show 500 error spikes and read timeout rates jump to 35% for 3 minutes.…
Primary PostgreSQL database disk space utilization reached 98% and alerted critical after the CDC connector went offline during a deployment.…
Data warehouse reports missing record updates for mutations that occurred while the multi-hour initial database snapshot process was executing.…
Data warehouse tables display stale entity values even though the source OLTP database committed newer updates.…
The downstream CDC target ingestion job is throwing unhandled deserialization exceptions, backing up the pipeline queue.…
Hard-deleted records in the production database remain permanently visible in downstream analytical reports.…
Users report sent messages show a single checkmark (sent) but are never received by online recipients until the recipient restarts their mobile app.…
Push notification queues are backing up with millions of delayed messages. Online users report getting duplicate push notifications for messages they already re…
GPU cluster memory utilization reaches 100% capacity and queue depth increases rapidly, even though active user count remains steady. Users report pressing 'Sto…
Time-To-First-Token (TTFT) increases by 500% across the platform and GPU compute costs double following a minor update to conversation rendering logic.…
During a traffic peak, the API Gateway returns 504 Gateway Timeouts to 40% of incoming users. Meanwhile, GPU worker utilization is 100%, but processing throughp…
Users on long flights or in remote areas report mobile app crashes on launch. Crashes spike specifically for users who haven't reconnected to the internet for o…
Data warehouse tables are missing 40% of clickstream events following the release of iOS App v4.2. Downstream analytics pipelines report schema mismatch excepti…
During a global push notification push, event ingestion latency increases from 40ms to over 15 seconds. HTTP 504 Gateway Timeouts peak at 65% for 10 minutes.…
IDE completions jumped from 800 ms to 6 s p95 after enabling whole-repo retrieval. GPU utilization is low while the context assembly service pegs CPU and queues…
Security scan found live AWS keys in exported prompt traces. Three enterprise tenants threatened to disable the product after a support engineer downloaded debu…
A single user session issued 400 shell tool calls in ten minutes, tripping rate limits for the whole org. Billing alerts show 50× normal token usage with no suc…
Product analytics show suggestion accept rate fell from 32% to 19% over two weeks, yet internal SLO dashboards report 99.9% availability. Leadership asks why me…
After autoscaling added GPUs, 12% of requests timed out at 30 s. New pods spent 25 s loading weights from object storage before accepting traffic.…
Support tickets report divergent paragraph text for the same doc link. OT engine logs show conflicting transform branches without a single canonical snapshot.…
Enterprise docs above 2 MB take 90 s to open. Storage costs for document history grew 8× after switching to full snapshot on every keystroke batch.…
After a brief CDN outage, documents contain repeated bullet lists. Metrics show 40k reconnects/min replaying the same buffered ops twice.…
Designers report undo removed a coworker's paragraph from an hour ago. Audit logs show shared global undo stack keyed only by document id.…
Document corruption alerts fired in EU region. Two sync nodes both advertised primary, accepting conflicting writes for 90 seconds during a switch failover.…
Design review boards lose ink when two people draw overlapping lines. Remote clients see gaps while the local author still sees their stroke.…
Importing a 200-page PDF to the whiteboard drops client FPS to 2 and WebGL context crashes on mid-tier laptops.…
Workshops show 30+ cursors though only eight people are connected. Presence badges confuse facilitators about who is active.…
Users returning from sleep see a blank canvas until hard refresh. Console shows websocket reconnected but snapshot fetch 404 on stale revision token.…
Teachers report entire lessons missing after Chrome killed a tab. Autosave metrics show last successful persist was 45 minutes before crash.…
All CDN edge agents restarted in a loop after a config publish. Global error rate hit 40% until rollback; schema allowed a negative TTL field that agents parse …
A critical TLS fix remains on 5% of agents for 72 hours. Promotion pipeline blocked because canary metrics compare to wrong baseline window.…
Operators rolled back a bad release but five minutes later the broken version returned. Two CI jobs raced writing the same config channel.…
When the control plane API blipped for three minutes, every API request blocked waiting for fresh route tables. p99 latency hit 12 s.…
Tenant B suddenly routed to Tenant A's payment service after an operator typo in a shared namespace key. Blast radius crossed isolation boundary.…
After tuning NSFW model threshold down, creator reach dropped sharply and appeals queue grew 10×. False positive rate jumped while recall barely improved.…
Average time-to-decision in the review queue hit 18 hours during a protest event. Reported content stayed visible while moderators worked FIFO without risk scor…
Weekly retrain job accuracy collapsed after moderators started using a new 'borderline' label not mapped in the training pipeline. Production still serves old m…
A coordinated spam campaign used thin noise borders to drop NSFW scores just below threshold. Automated removal rate fell 60% for two days.…
Leadership dashboards show flat violation rate while user reports tripled. Denominator uses impressions instead of viewed posts, hiding surge in smaller creator…
During a deploy, CPU metric flapped every 30 s causing 12,000 alert evaluations per minute. PagerDuty throttled and real outages went unpageable.…
Database failover generated 22 identical PagerDuty incidents because each shard evaluator opened a new alert key instead of deduping on correlation id.…
Night shift received 400 CPU alerts from individual Kafka brokers though cluster capacity was fine. Mean time to acknowledge rose above 30 minutes.…
A typo in label matcher stopped all payment error alerts for six days until finance noticed reconciliation gaps. CI had no alert rule tests.…
Every Friday evening on-call gets paged for 'anomaly' on signups though pattern is normal weekly seasonality. Trust in anomaly alerts eroded.…
Warehouse load failed at 2 a.m. because one Kafka partition had corrupt JSON for three hours. Downstream marts empty for morning dashboards.…
CEO dashboard showed blank revenue for a day after an engineer renamed a field in the raw layer. No alert fired because downstream job silently null-filled.…
Customer age field became 90% null after a mobile bug but quality checks only ran monthly. Segmentation models degraded silently.…
Payments pipeline stalled because a new status enum value appeared without consumer update. Spark job threw enum parse exceptions for 6 hours.…
Daily quality scan missed SLA because one huge tenant partition dominated scan time; other partitions finished in minutes.…
A memory leak canary reached 50% fleet before anyone noticed RSS climb; auto-promotion continued because HTTP 500 rate stayed flat while OOM kills rose.…
Engineering could not ship hotfixes because deployment pipeline waits on a synchronous additive column migration on a 40 TB table.…
Release analysis compared canary p99 latency to the wrong baseline window (deploy day vs prior week), green-lighting a 40 ms regression.…
Security audit found 12 pods running an image digest with no build record. Incident response cannot map artifact to git commit or SBOM.…
Updating 8,000 microservices caused reconcile queue depth of 2M; new deploys stuck pending for hours while controller CPU pegged.…
Expanding the memcached ring from 20 to 22 nodes dropped hit ratio from 92% to 40% for an hour. Origin DB CPU doubled during remapping.…
Feature flag updates take up to five minutes to appear. Incidents traced to reads served from async replica lagging 200k writes behind primary.…
Flash sale SKU drove 500k RPS against one memcached instance while siblings idle. p99 product page latency hit 2 s though overall traffic normal.…
Every hour on the hour, cache hit ratio drops from 95% to 60% and database IOPS spike. Metrics show mass key expiry at :00.…
Intermittent cache misses and timeouts after a rack power event. Gossip still lists a dead node for 90 s while clients keep routing keys there.…
Two clients both believed they held the global inventory lock during a zone partition. Oversell of 1,200 units resulted before merge.…
Batch pipeline stalled 30 minutes because a worker died holding the shard lock. No other worker could proceed until lease expired.…
Payment settlement job double-charged accounts after a JVM GC pause exceeded lock TTL. Two workers processed the same batch concurrently.…
Black Friday checkout saw 50k threads blocked on the same promo lock; when holder released, thundering herd crashed payment service.…
Rare duplicate cron runs at midnight when NTP drift caused lock service to think lease expired 400 ms early on two nodes.…
Enterprise customer ingestion lagged hours while other partitions empty. Consumer lag dashboard showed partition 7 at 50M messages.…
After a rack outage, brokers accepted produce acks with min.insync.replicas not met; consumers read truncated log losing 30 seconds of orders.…
Consumer processing stalled 40% of the day as group rebalanced every few minutes when partition leaders oscillated between two brokers.…
Operator reset consumer group offset to earliest during debugging; billing service reprocessed seven days of events duplicating invoices.…
Warehouse shipped duplicate packages after consumer retries on transient DB errors. Exactly-once claim but no idempotency keys on write path.…
Ingestion queue depth hit 200k after customers bulk-uploaded 400-page scanned PDFs. Parser pods OOMKilled looping retries on the same files.…
Legal team sees two 'latest' versions of the same contract after parallel uploads. Search index shows both as current without merge.…
GDPR erasure requests completed in primary store but documents still appear in search results weeks later. Tombstone stream consumer lagging 14 days.…
Night backfill job rewrote entire Elasticsearch index using old snapshot, dropping a day of new documents from search.…
Parser cluster outage after nested zip expanded to 800 GB logical size. Antivirus scan skipped for 'trusted' enterprise uploads.…
Dynamic pricing raised rides 25% in a dense city; completed trips fell 40% while competitors held flat. Elasticity model trained on pre-pandemic data.…
Finance noticed control city revenue drop matching treatment algorithm. Feature flag misconfigured to 100% rollout in neighboring geofence.…
Support tickets spiked about 'bait price' when quotes changed 15% between two refreshes seconds apart. Smoothing filter disabled during load test leftover flag.…
During partial dispatch outage, surge hit 8× normal while supply recovered; regulators contacted about price gouging. Hard cap feature flag off in prod.…
Surge pricing activated citywide though plenty of drivers were online. Location pipeline lagged 12 minutes dropping available supply counts.…
Daily revenue KPI jumped 12% without sales growth. Analysts found duplicate line items per order after replaying checkout events twice.…
Category performance YoY report shows NULL category for half of last year after SKU merge. SCD2 effective dates have a gap day without overlap handling.…
Weekly inventory snapshot understated returns by $2M because incremental job cut off at warehouse close while returns arrived from stores until 3 a.m.…
Finance close double-subtracted refunds appearing in both gross revenue and refund fact without netting logic. Net revenue off by $800k for the month.…
Analyst query costs 10× after mart denormalized 300 columns into one table. BigQuery bytes scanned per dashboard refresh exceeded budget.…
ETAs underestimated by 18 minutes across a corridor after bridge closure. Traffic feature pipeline still served pre-closure speeds for 45 minutes.…
Complaints rose when meals arrived outside the 5-minute quoted window though model RMSE was fine. Interval always ±3 minutes regardless of route variance.…
Nightly ETA retrain failed after city expansion; all-pairs shortest path matrix exceeded memory on single Spark driver.…
Live ETA bias spiked after deploy though offline eval looked good. Serving hash encodes road segments differently than training pipeline after library upgrade.…
Mean absolute error crept up 30% over two weeks after map vendor change. Monitoring dashboard only tracked p95 latency not error vs actuals.…
Job retry after worker kill inserted duplicate fact rows doubling inventory counts. Framework marks stage complete before commit flush.…
Same-day sales mart missed SLA while backfill consumed all Spark executors for 48 hours. Live jobs queued behind low-priority backfill flag mis-set.…
Payments fact job killed when adjacent team's experimental job spiked memory on shared Kubernetes node without limits.…
Marketing funnel KPI flatlined after upstream renamed event field. ETL framework lineage graph outdated; no impact analysis ran on schema change.…
Every warehouse load failed at midnight after DB password rotation. Framework still mounted old secret volume until manual pod restart.…
Launch review showed neutral results but product saw lift. Exposure logs dropped 35% of mobile sessions after SDK upgrade missed assignment callback.…
Experiment declared winner on 20% CTR lift but metric divided clicks by sessions instead of impressions, overstating lift for heavy users.…
After launch, chi-square test shows 54/46 split instead of 50/50 with p-value 0.001. Browser bug sent more Chrome users to treatment.…
Team stopped experiment at 95% significance day three; effect disappeared by week two. No sequential testing correction applied.…
Experiment exposure counts exceeded unique users by 22% because assignment used device id while outcomes joined on account id created after login.…
A new fraud model shows 0.98 offline AUC but only 0.71 in shadow traffic. Data science suspects label leakage in the historical training join.…
Checkout recommendation models miss latency SLO: P99 feature fetch latency jumps from 12ms to 800ms during a flash sale while online store CPU pegs at 100%.…
Click-through rate on the home feed drops 18% after a data engineer fixed a batch aggregation bug without updating the streaming pipeline.…
Fraud scores swing wildly for all traffic after a team redeployed feature logic by overwriting rows in the online store without versioning.…
Personalization quality degrades silently; dashboards show healthy Kafka lag but online engagement features are 36 hours old for 12% of users.…
Feed relevance scores flatline during prime time; feature fetch timeouts cause the ranker to fall back to popularity-only ordering.…
Policy team reports a 5x spike in low-quality clickbait in top feed slots despite unchanged model weights.…
Users report seeing the same posts after reopening the app; session dedupe cache never expires and blocks new candidates.…
An experiment shows +4% clicks for 3 days but 30-day retention drops 2%; experiment analysis only aggregates daily metrics.…
During evening peak, feed API P99 exceeds 2s; ranker queue depth grows unbounded and clients receive HTTP 503.…
Users report failed 10GB video uploads after brief Wi-Fi drops; mobile clients restart entire uploads from byte zero.…
API fleet memory saturates during a campus-wide file share event; upload throughput collapses despite auto-scaled pods.…
Cross-device sync delays spike; users see stale folder listings on secondary devices after edits on primary.…
Support tickets spike for duplicate files named doc (conflict 1) after users edited the same note offline on two devices.…
Storage costs jump 40% after a meme circulates; each user upload creates a unique object key with full byte copy.…
Checkout P99 fraud decision latency exceeds 400ms; payment DB CPU spikes correlate with card velocity rule evaluation.…
New fraud model shows great offline metrics but live false-positive rate triples after deployment.…
40% of checkout authorizations time out at 120ms; tracing shows five sequential remote calls on the fraud path.…
Compliance audit fails: analysts cannot explain why transaction ID 8842 was declined last Tuesday.…
Feature store regional outage causes 100% checkout hard failures instead of degraded fraud screening.…
Legal reports 12,000 overdue erasure tickets; lake scans rewrite entire Parquet partitions nightly at unsustainable cost.…
Privacy review shows deleted accounts visible via table time travel queries used by support tooling.…
Disaster recovery restore from last week's backup reintroduces 4,000 users who completed GDPR erasure.…
Erasure team cannot rewrite immutable glacier archives within regulatory window; legal asks for cryptographic deletion.…
Finance compliance requires 7-year invoices but erasure pipeline deletes entire customer records including billing artifacts.…
Users in EU and APAC simultaneously update profile settings; merged record oscillates between conflicting values.…
Audit log shows purchase B before purchase A despite A happening first globally; support refunds wrong orders.…
Users post content then immediately refresh and see empty feed; reads hit stale replica seconds after write ACK.…
Region outage loses 2 minutes of confirmed writes; customers report missing transactions after failover.…
Network blip triggers dual primaries in US and EU; duplicate charge IDs appear for 8 minutes.…
Instagram-style feed crashes on mid-tier Android after long scroll sessions; heap dumps show thousands of retained image views.…
Data usage complaints spike; feed loads 4K images into 400px thumbnails burning memory and bandwidth.…
Users see blank tiles when scrolling back up; decoded bitmap cache thrashes under memory pressure.…
Offline users see endless loading spinner with no cached content or explicit offline state.…
Users report missing posts when scrolling; offset-based pagination skips items when new content inserts at top.…
Temperature charts show impossible spikes after devices reconnect; hourly rollups double-count late batches.…
Dashboard year-view queries exceed 60s timeout; raw 1-second samples scanned for single device chart.…
After outage, 200k devices reconnect within 60s; ingest gateway returns 503 and drops telemetry batches.…
Fleet dashboard shows future-dated readings and negative durations; alerting misfires on clock-skewed devices.…
Critical freezer sensors stop reporting for 45 minutes before anyone notices; SLA requires 1-minute detection.…
Adding nodes during peak causes GET misses and 404 spikes; clients hit wrong owners after ring change.…
Disk failure on one node permanently loses writes that were acknowledged with W=1.…
Cart counters intermittently revert; two clients increment same key and one update vanishes.…
Weekly consistency audit finds 0.3% key mismatch across replicas; manual rebuild required.…
New nodes never receive traffic; routing table still lists decommissioned hosts as alive.…
Lake queries that used to take minutes now take hours; metastore CPU pegged after streaming ingest lands micro-files.…
Finance dashboards time out scanning 500TB events table for single-day filters.…
ML team cannot reproduce March model; time travel query returns empty after aggressive vacuum deleted history.…
Audit finds analysts selecting raw email column via ad-hoc SQL bypassing approved views.…
Executive dashboards fail during nightly feature backfill; shared cluster queues BI behind 10k Spark tasks.…
Downstream job scheduler sees constant leader changes; duplicate cron triggers fire intermittently.…
Two scheduler leaders run simultaneously assigning conflicting shard ownership during WAN blip.…
After failover, old leader resumes writing job checkpoints corrupting orchestration metadata.…
Brief 200ms network jitter triggers cluster-wide re-elections and 5-minute orchestration freeze.…
After leader change, 30% of client requests still hit old node returning HTTP 409 for two minutes.…
Release quality gate flips pass/fail on identical prompts week to week; grader model updated silently.…
New prompt template passes eval once then fails in production; eval runs each case only once.…
Staging eval run processes 200 live refund API calls totaling $18k during tool-use agent test.…
Known-bad model ships after eval job fails but CI treats eval step as advisory only.…
Finance alerts on $40k LLM eval charges after teams rerun full suites with expensive judge models.…
Provider A at 50% errors becomes 100% unavailable as gateway retries double traffic with no jitter.…
Long streaming completion finishes at 2x token budget; finance cannot attribute overage until after response ends.…
Gateway stops accepting traffic at 10k concurrent streams; long-lived SSE ties up worker threads.…
Security scan finds user emails and medical prompts in centralized log index used by all teams.…
Support bot gives outdated policy answer; cache key fuzzy-matches distinct user questions.…
Interactive chat P99 latency hits 4s while GPU utilization looks healthy at 85%.…
Nightly embedding job colocated on same GPU pool spikes interactive latency 10x at 2am UTC globally.…
GPU memory exhausted serving many fine-tunes; platform loads separate full weights per adapter.…
Product launch causes 503s for 12 minutes while new GPU nodes load 70GB checkpoints sequentially.…
New quantized model promoted to 100% traffic after 5-minute canary with no quality guardrails.…
During outage every service logs 50x errors; Kafka consumer lag hits 6 hours and search falls behind.…
Security audit logs dropped during spike while verbose debug logs from one team fill retention tier.…
Half of microservices invisible to structured queries; parser drops malformed multiline stack traces.…
Cluster health red after team logs random UUID field names; mapping count exceeds cluster limit.…
DLP audit finds PAN sequences searchable in production log index accessible to all engineers.…
Metrics cluster OOM after deploy adds user_id label; active series jump from 2M to 180M.…
Metrics storage bill up 6x; SRE asked to keep 15s resolution for a year on all series.…
Database outage pages mobile team; alert routing rules stale after service ownership migration.…
Mass pod restart emits metric burst; remote write drops 30% samples and SLO dashboards go blank.…
On-call flooded with resolve/reopen cycles on noisy CPU alert during batch cron overlap.…
Checkout micro-frontend loads before shell router; white screen on 20% of page loads due to race.…
Lighthouse shows 1.2MB duplicate React bundles from four micro-frontends on dashboard.…
Uncaught TypeError in recommendations widget takes down navigation and payments MFEs.…
Cart MFE listens to ad-hoc window events; refactor in checkout team breaks cart count silently.…
Team ships fix but users still hit broken bundle for 30 minutes; CDN and manifest cache not invalidated together.…
App launch regression after moving all flag evaluation to blocking server round-trip on startup.…
New installs on flights see experimental checkout enabled; no bundled default flag snapshot.…
Kill switch toggled off bad feature globally but Android 12 clients keep crashing until app update.…
Experiment shows null effect; analytics pipeline logs conversions but not flag exposure impressions.…
Flags config download grows to 400KB; startup latency rises as hundreds of stale flags ship to clients.…
Checkout conversion looks stable in business dashboards, but fraud block rate dropped 18% over two weeks. Ground-truth chargeback labels lag 30 days, so model a…
PagerDuty fired 400 alerts in one hour after a minor deployment. Most cohorts have fewer than 50 events per day, and static thresholds were copied from the glob…
Security review flagged that model monitoring logs store full billing addresses and emails. A data analyst exported 2M rows for drift analysis without field-lev…
Image classification p99 latency jumped from 120 ms to 2.1 s during a traffic spike. GPU utilization is healthy at 85%, but clients time out at 500 ms.…
A/B test shows new model v3 underperforming v2, but shadow metrics on v3 look excellent. Support reports specific enterprise tenants always get v2 scores labele…
A trending product launch drove QPS from 200 to 3,000 in three minutes. Error rate hit 40% with HTTP 503 while cluster autoscaler added GPU nodes over eight min…
After a cross-region network blip, two regions both accepted write traffic for 90 seconds. Order IDs collided and inventory counts diverged by 1,200 units.…
Failover to the secondary region succeeded in DNS, but users saw account balances from six hours ago for 20 minutes. Payment retries caused duplicate holds.…
Regional failover appeared successful, but checkout still failed because the DR stack synchronously called an identity service that remained pinned to the down …
Posting for a user with 80M followers timed out after 30 s. Fan-out worker queue depth hit 2M jobs and normal users’ feeds lagged 45 minutes behind.…
Moderation removed a harmful post, but millions of precomputed feeds still show it 12 hours later. Mobile pull-to-refresh intermittently surfaces the deleted co…
Redis memory usage climbed 3x after enabling rich media previews in feeds. Editing one viral post required scanning millions of feed keys to patch embedded JSON…
Push opt-out rate spiked 12% after a marketing campaign. Some users received 28 pushes in one day despite a product policy of max 3 ranked notifications per day…
Complaints flooded support after users in Europe received shopping alerts at 3 a.m. local time. The ranker schedules on UTC hour buckets using signup timezone o…
Users report duplicate alerts for the same order shipment on push, email, and SMS within two minutes. Engagement dropped because each channel ranker optimizes i…
During a security incident, password-reset pushes failed for 18% of users. APNs returned HTTP 429 and the service had no retry backoff or priority queue separat…
Users received the same promo push twice within 30 seconds. Logs show two delivery attempts with identical payload hash but different worker attempt IDs.…
Enterprise customers report missing in-app alerts while push volume looked normal. Internal metrics show 35% of enqueue requests rejected with HTTP 429 from the…
A user edited a shared note on phone and laptop while offline for a flight. After reconnect, one device’s paragraph vanished with no warning.…
Users deleted tasks while offline, but items popped back after syncing from an older tablet that still held the rows locally.…
After a week offline, the app froze on ‘Syncing…’ and consumed 400 MB in minutes. Users on cellular complained about data caps.…
Checkout p99 rose from 180 ms to 950 ms after enabling a new graph feature. The fraud service blocks the payment API synchronously on every request.…
A flash sale on one merchant drove fraud feature Redis CPU to 100%. Latency correlated with a single feature key read millions of times per minute.…
When the ML scoring cluster crashed for 12 minutes, checkout returned HTTP 503 for every card payment. No rule-based fallback path existed.…
A paste linked from a trending forum drove read QPS to 200k on a single short id. Origin DB CPU pegged at 100% despite Redis in front.…
Compliance audit found 1.2M pastes older than their TTL still downloadable. The nightly cron sweeper fell 18 hours behind after a DB migration.…
Security team flagged malware download links served from new pastes for up to 90 seconds. Upload API returns 201 immediately while async scanning queues behind …
Merchants report duplicate charges when shoppers double-tap pay or when mobile clients retry timed-out requests. Two capture records share the same cart id but …
Finance reconciliation shows $240k in processor settlements without matching internal ledger entries. App servers updated payment status then crashed before pub…
Shoppers saw payment failures and placed second orders, but the processor actually captured both attempts. Internal status stayed PENDING after HTTP timeout to …
Merchants received duplicate bank transfers for the same settlement batch. Ops re-uploaded a CSV after a timeout, and both runs inserted payout rows with differ…
Daily reconciliation report shows a persistent $1.8M mismatch that disappears tomorrow. Processor settles on UTC midnight while internal ledger cuts off at US/P…
Auditors flagged thousands of 1-cent variances between order totals, fee splits, and processor settlements after introducing floating-point math in a fee servic…
Users see friend suggestions for people they unfriended months ago. The daily graph feature batch finished 36 hours late and serving still reads yesterday’s pre…
A user who blocked their ex still received a ‘people you may know’ card with that person’s photo. Support escalations cite harassment concerns.…
New users see empty PYMK for 48 hours while veterans get instant suggestions. Realtime friend accept events queue behind a backlog, so online features miss fres…
Analytics shows average session length of 4 minutes for a 2-hour film. Heartbeat events every 30s create a new session whenever mobile OS pauses the app briefly…
Reported watch minutes exceed available content duration by 3x for binge watchers. Scrubbing backward generates overlapping progress events summed naively.…
Session funnels show impossible sequences: ‘complete’ before ‘start’. Devices offline for hours upload buffered events that land out of order in the stream proc…
An employee asked the assistant about a confidential HR policy and received excerpts from executive-only docs. Vector search ranked by similarity only; ACL filt…
Support bot told customers an old refund policy after legal published an update. Search index refresh runs nightly; doc deletes are not tombstoned in the vector…
Users received confident wrong answers about API rate limits. Retrieval returned weak matches but the LLM still generated specifics not present in any chunk.…
API Gateway endpoints return 500 Internal Server Errors across all routes. Upstream services report 0 incoming requests while overall API availability drops to …
Latency spikes to 1,500ms for high-tier API users. CPU usage on a single Redis cluster shard hits 100%, while adjacent shards remain under 5% utilization.…
Audit logs show a client with a strict tier limit of 1,000 requests/minute successfully executed over 12,000 requests in a single second without receiving HTTP …
Live dashboard showed 2x actual signups during a Kafka consumer rebalance. Unique user counts looked fine but event totals inflated.…
Hourly active user chart undercounted mobile traffic every evening. Offline clients burst-upload events that arrive 40 minutes late and were dropped by strict t…
During region failover drill, global QPS metric spiked to 2x actual. Both us-east and eu-west aggregators wrote the same event ids to the global Redis counter f…
Day-0 retention dropped after launch in a new country. Users with no click history received empty recommendation responses because the ranker returned 404 when …
Home feed p99 latency hit 800 ms after expanding candidate retrieval from 200 to 10,000 items. Ranker GPU queue backed up while ANN search stayed fast.…
Offline AUC improved 8% but online CTR dropped after deploy. Training logs used imputed defaults for missing features while serving returned null and a differen…
Riders reported two drivers arriving for the same trip during surge. Support logs show two ACCEPT transitions on one ride id within 200 ms in different regions.…
Riders waited 12 minutes while the app showed three drivers ‘2 min away’. Those drivers had gone offline or moved neighborhoods; location index TTL was 5 minute…
After a stadium concert, surge multiplier stayed at 3.5x for 90 minutes in a geohash that no longer had demand. Drivers avoided the area while riders still saw …
Customer dimension queries for ‘as of March 15’ return two active addresses. A CRM correction effective February 10 arrived on March 20 and was inserted without…
Customer dimension row count grew 40% month-over-month though business changes were flat. Nightly extracts re-insert identical rows because change detection com…
Privacy request verified removal from the active customer row, but analysts still queried historical SCD2 versions containing email and phone from prior years.…
After retraining the search ranker on six months of click logs, NDCG on human-rated sets drops 8%. Merchandising reports that highly relevant products in positi…
Zero-result rate jumps from 2% to 11% after a catalog taxonomy migration. Queries like 'running shoes' return empty lists even though inventory exists under the…
During a flash sale, search p99 latency breaches 200 ms for 25 minutes while p50 stays near 45 ms. Error budgets burn on scatter-gather timeouts returning parti…
A new ranker wins the A/B test on overall conversion (+1.2%) but support tickets rise for long-tail queries. Reformulation rate on tail segments increases 35% w…
Sellers complain new SKUs are invisible in search for 72+ hours despite matching titles. Only products with historical click features rank on page one.…
Security detects an unknown process listing every secret path under /prod/* from a compromised microservice pod. Cloud audit logs show 40,000 Secret Manager rea…
During scheduled DB password rotation, 800 services restart simultaneously and checkout API error rate hits 40% for 12 minutes. Connection pools hold stale pass…
After revoking a leaked partner API key, traffic with the old key continues for 45 minutes. Incident response cannot confirm full containment.…
A signing key for JWT issuance is compromised. Security triggers global revoke, but edge POPs continue minting valid tokens with the old key for 20 minutes.…
Compliance audit flags inability to prove who read production TLS private keys during a suspected insider incident. Vault metrics show read volume but not calle…
After a deploy, 18% of checkout requests return HTTP 503 for 4 minutes. Clients still call pod IPs removed from the registry but cached locally.…
Two AZs report different instance counts for payment-service. Some clients see 12 healthy nodes, others see 4, during a network partition between control-plane …
When 30% of catalog-service pods crash, latency on survivors jumps 5x. Load balancer still sends traffic to dead instances for 90 seconds.…
During a downstream DB slowdown, the mesh ejects every recommendation-service instance within 30 seconds. The service appears entirely unavailable despite pods …
External partners hitting api.example.com via DNS still connect to decommissioned load balancer IPs 20 minutes after cutover.…
On-call cannot reproduce payments-service prod config from the standard template. Missing metrics sidecar and wrong autoscaling limits caused a 2-hour outage.…
A Sev-1 fix required kubectl exec into production as root, disabling network policy and exporting customer data to a laptop. Post-incident review finds no appro…
A new service deploys without CPU limits and starves neighboring pods on shared nodes. Noisy neighbor causes cluster-wide latency spike.…
Fifty teams queue on a single Terraform state lock for the shared platform stack. A stuck apply from yesterday prevents any new service onboarding.…
Scanner flags CVE-2024-XXXX in OpenSSL across 400 services. Only 12% rebuilt images after the platform team published a patched base tag two weeks ago.…
Finance reports $120k in unauthorized refunds in 48 hours traced to the support chatbot. Logs show the model called RefundOrder with full amount after user clai…
After a Friday prompt tweak, CSAT rises slightly but regulated-topic hallucinations jump 3x. Compliance flags incorrect legal advice in 40 conversations.…
Customers repeat their issue three times after bot escalation. Average handle time for escalations rises 40%. CRM ticket shows only 'Transferred from bot'.…
Customers receive three shipping-label emails for one request. Tool logs show UpdateTicket invoked multiple times with identical arguments after LLM retries.…
P99 agent response time hits 25s on threads over 80 turns. Monthly LLM spend doubles while containment rate stays flat.…
Scheduler DB CPU pegs at 95% during peak. Job dispatch latency rises from 200ms to 8s. EXPLAIN shows millions of SELECT ... FOR UPDATE SKIP LOCKED scans per min…
Billing export job emails customers duplicate PDFs. Two workers claim the same job_id within 500ms during NTP drift on one host.…
During leader failover, 18% of minute-level cron tasks never enqueue. Metrics show a 90-second gap in schedule ticks.…
Finance reconciliation ran twice on fall-back night and skipped entirely on spring-forward in US/Eastern cron jobs.…
Platform-wide job latency SLO breaches while one enterprise tenant enqueues 2M backup jobs in an hour. Other tenants wait hours in FIFO queue.…
Paid tier p99 time-to-first-pixel exceeds 8 minutes during a viral prompt trend. Free tier jobs dominate the single FIFO GPU queue.…
Trust & safety flags 200 generated images in an hour containing disallowed content. Prompts use unicode homoglyphs and split banned tokens across languages.…
Queue depth hits 400k jobs during marketing launch. New GPU nodes take 12 minutes to provision while users see multi-hour waits.…
Background thumbnail jobs age over 24 hours while interactive requests keep arriving. Batch SLA missed for a week during sustained traffic.…
Simple 256px icon requests take 45s and cost 12x more than necessary. Product analytics show 70% of prompts are short and low resolution.…
Concert on-sale generates 200 support tickets for duplicate seat assignments. Two confirmation emails reference seat A-12 for different orders.…
Show appears sold out while UI shows 400 seats in 'held' state. Revenue lost because holds from abandoned carts never expire.…
Seat map API shows green available seats that fail at payment with 'already sold'. Replica lag averages 8 seconds during on-sale.…
Payment provider shows 1,200 successful captures without matching confirmed orders in booking DB after DB failover during on-sale.…
Cluster shows 80 free GPUs while 15 large training jobs stay pending for 6 hours. Each job needs 8 GPUs simultaneously but cluster fragments into 7-GPU islands.…
Job resumed from checkpoint after node failure diverges immediately with NaN loss. Object store contains truncated shard files from crash mid-upload.…
Input pipeline stalls; GPU utilization drops to 30% cluster-wide. NFS metadata ops/sec peg while thousands of workers list same dataset directory.…
Regulator asks which training snapshot produced model v3.2; team cannot reproduce metrics because dataset was overwritten in place with same path.…
128-GPU job hangs every 20 minutes at all-reduce; NCCL timeouts fire after one worker on a noisy network link slows collective.…
Primary key collisions spike on orders table after VM clock correction. Two ID generator nodes emit identical 64-bit IDs milliseconds apart.…
Collision rate jumps when Kubernetes HPA doubles ID generator pods. Two pods assigned worker_id=17 via modulo on pod name hash clash.…
ID API latency spikes to 2s during flash sale while QPS is 80k per node. Generator busy-waits for next millisecond after exhausting 12-bit sequence in one ms.…
Sharding router sends 30% of traffic to wrong database after server adds datacenter bits to ID format. Mobile apps embed timestamp extraction logic.…
Users double-charged when checkout retried ID fetch after 504 timeout though server actually allocated ID. Two orders share different IDs for one cart.…
A celebrity shared a short link and redirect p99 latency jumped from 20 ms to over 800 ms. The cache layer shows near 100% CPU and the database read IOPS triple…
Two API servers occasionally return the same short code for different long URLs. Support tickets mention wrong redirects after heavy create traffic.…
Home feed shows generic trending only for 20% of users during peak. p99 retrieval stage exceeds 120ms budget and ranker receives zero candidates.…
Average watch time drops 15% while click-through rises. Model amplifies misleading thumbnails because training labels are click-only.…
Uploads from new channels get under 50 impressions unless creator is manually featured. Ranker scores rely on channel historical watch hours unavailable for new…
Users binge a new topic but feed keeps pushing old genre for 6 hours. Real-time clicks not reflected in ranker feature store refresh lag.…
New ranker shows +5% offline AUC but online A/B is flat and long-session users churn. Offline eval uses random negatives not matching production candidate gener…
Major series premiere shows 'format unavailable' on TV apps for 6 hours. Transcode queue depth exceeds 50k jobs while only SD renditions exist.…
Live match viewers see quality ping-pong between 1080p and 360p every 10 seconds with frequent rebuffering on stable 50 Mbps links.…
Viral clip triggers 900 Gbps origin egress; origin error rate 40% while edges show low hit ratio on new URL variants with unique query params.…
Support hears playback complaints but dashboards show 99.9% success. Client QoE beacons dropped on ad blocker and batch flush failures during background mobile.…
Monthly CDN egress bill jumps 35% after UI change defaults mobile autoplay to highest rung. Most phones never switch down on cellular.…
Merchants upload inventory photos but visual search returns no matches until next day. Marketing campaign underperforms because index embedding pipeline batches…
Team ships v2 vision model with incompatible embedding space; mixed v1/v2 vectors in same index collapse search relevance until full 5-day reembed completes.…
Users tap visually similar items and land on 404 product pages for delisted SKUs. Vector index retains embeddings after catalog tombstone without delete propaga…
Mobile visual search p99 hits 2.8s after adding cross-encoder rerank on 200 ANN candidates sequentially on CPU.…
Offline recall@10 improves 10% on benchmark but user-shot mirror selfies perform poorly in beta. Eval set lacks noisy real-world query images.…
Users interrupt the assistant mid-sentence; session abandon rate rises 22%. Traces show 1.4s from end-of-speech to first audio byte due to serial STT, LLM, TTS …
Customers yell 'stop' repeatedly while TTS continues for 3 seconds. Turn-taking state machine ignores VAD during playback.…
Callers stay on hold indefinitely after assistant says 'connecting you to a specialist'. Logs show no TransferCall tool invocation.…
New callers hear busy tone during campaign while CPU is only 60%. Session manager limit of 2k concurrent WebRTC legs hit before autoscaler adds nodes.…
After STT v3 deploy, misrouted intents rise 18% on car Bluetooth calls. Offline WER improved on clean LibriSpeech but not on telephony noise set.…
Major news site returns HTTP 403 for all crawler IPs. Rate limit logs show 40 concurrent connections to one host during sitemap burst.…
Crawl rate drops from 40k pages/sec to 2k after dedupe moved to Postgres primary. CPU on DB cluster saturates on SELECT by URL hash.…
Object storage costs jump 3x in a quarter; 60% of bytes are near-identical syndicated articles at different URLs.…
Fetcher CPU idle at 30% while crawl QPS flatlines. Profiles show 70% of worker time blocked on getaddrinfo to corporate DNS.…
After spot instance reclaim, 2M URLs never finish crawling. Frontier marks them IN_PROGRESS with no lease expiry.…