Ops signals are observable indicators—metrics, logs, traces, and events—that reflect pipeline health across prompt parsing, model loading, inference, upscaling, frame interpolation, and asset delivery. In creative workloads, the most important signals connect system performance to viewer or creator experience (e.g., FPS health in animation previews, render latency for page layouts, and error-free completions for batch jobs).

• Experience
  • FPS health (animation preview/playback)
  • End-to-end render latency (prompt-to-image/frame)
  • Success ratio (completed vs. attempted renders)
• Flow
  • Throughput (images/sec, frames/sec, jobs/min)
  • Queue depth and wait time
• System
  • GPU utilization and VRAM pressure
  • Model load/unload time; cache hit rate (weights, VAE, LoRA)
  • I/O bandwidth (disk/network) for model and asset fetches
• Quality
  • Dropped/duplicated frames
  • Post-process timing (upscaler, denoise, interpolation)
  • Optional: perceptual metrics or rejection rate from automatic QC

FPS health captures visible smoothness for animation previews and timeline scrubbing. Track both produced FPS (generator output) and delivered FPS (viewer/client) to detect bottlenecks across inference, post-processing, and playback. Use FPS health to drive alerting, rollback, and autoscaling policies because it directly maps to perceived quality.

• Add structured events at stage boundaries: prompt_received, model_loaded, inference_started, inference_finished, postprocess_done, frame_emitted, deliver_complete.
• Emit timers for per-stage latency and counters for successes/errors/timeouts.
• Attach trace/span IDs across services (scheduler → inference → post-process → CDN/client).
• Tag with model_id, sampler, steps, resolution, batch size, and hardware class to isolate regressions.
• Sample high-volume events (frames) but keep full fidelity for errors and tail latency.

• Animation previews: delivered FPS ≥24 (99th percentile over 5 min), dropped frames <2%.
• Single-image render: p95 latency ≤3–5s at 768×768 on standard GPU class.
• Batch comic panels: success ratio ≥99.5%, queue wait p95 ≤30s.
• System safety: VRAM headroom ≥10–15%, GPU utilization target 70–90% under load.
• Error budget: timeouts + OOMs ≤0.5% of requests per day. Note: Calibrate thresholds per model (base vs. finetune), resolution, and post-process chain.

• Experience: delivered vs. produced FPS; frame drop/duplication; render latency distribution (p50/p95/p99).
• Flow: queue depth, throughput, scheduler admit/deny rate.
• System: GPU utilization, VRAM used/free, model load time, cache hit/miss, I/O wait.
• Errors: timeouts, OOMs, CUDA errors, retry loops; top failing models/settings.
• Correlation: overlay deploys/config changes with FPS health and latency shifts.

• Multi-signal alerts: combine FPS health drop + queue surge to reduce false positives.
• Use short + long windows (e.g., 2 min and 15 min) to detect spikes and drifts.
• Route by impact: UX-breaking (paging) vs. capacity (ticket) vs. anomaly (email).
• Add auto-silence during planned heavy loads (e.g., model cache warmup).
• Include runbook links with each alert: probable causes and commands to verify.

• Low delivered FPS, normal produced FPS → client/CDN issue or network bottleneck.
• Low produced FPS, high GPU utilization → model too heavy or VRAM thrash; reduce resolution/steps or scale out.
• Latency tail grows, queue depth rising → scheduler saturation; add workers or prioritize smaller jobs.
• Spiky OOMs after deploy → new model/LoRA size or batch config; roll back or split batches.
• High post-process time only → upscaler/interpolator regression; isolate and toggle feature flag.

• Define your golden signals: FPS health, render latency, success ratio, queue depth, GPU/VRAM.
• Instrument stage timers and error counters with trace IDs.
• Set SLOs per workload and wire alerts with dual windows.
• Build dashboards per workload; overlay deploys.
• Review weekly: error budget burn, tail latency, and FPS health regressions.

How is FPS health different from throughput? FPS health reflects viewer-perceived smoothness; throughput measures production rate. You need both to avoid smooth-looking but backlogged systems.

What if I can’t hit 24 FPS? Target stable delivery (no drops) and communicate constraints; consider caching, interpolation, or lowering resolution during preview.

Do I need perceptual quality metrics? Optional. Start with operational signals (errors, latency, FPS). Add perceptual checks when you automate QC or compare styles.