Video LLMs are multimodal large language models that accept video (or sequences of frames) and text, returning structured analyses or generation-ready instructions. Unlike pure video generators, video LLMs focus on understanding: temporal events, shot boundaries, camera moves, dialogue timing, and visual semantics. In anime/comic pipelines they help plan and supervise generation tools (image-to-video, diffusion, motion control) for consistency and speed.

• Convert scripts and reference clips into shot lists and animatics.
• Maintain style, character continuity, and camera language across scenes.
• Produce motion prompts for image-to-video tools (pan, tilt, dolly, arc, timing, easing).
• Align dialogue to phoneme-level timing for lip-sync and subtitles.
• Extract color keys, lighting notes, and prop continuity from reference footage.

• Shot/scene segmentation: detect cuts, duration, and pacing.
• Camera understanding: classifies shot size (WS/MS/CU), angle (high/low), movement (pan/tilt/dolly), and lens feel.
• Action and beat summaries: who does what, when, for how long.
• Visual attributes: style tags (cel, line weight), lighting (rim, bounce), palette, mood.
• Dialogue/timecode alignment: timestamps, per-line duration, phoneme hints.
• Safety and rights checks: flag watermarks or obvious copyrighted overlays to avoid misuse.

• General-purpose multimodal: GPT-4o family (video understanding), Gemini 1.5 (long-context video), Claude Vision (frame-sequence reasoning; check current video limits).
• Open research models: LLaVA/LLaVA-Video variants, Video-LLaMA family, InternVideo/InternVL, Qwen-VL with video support. Capabilities vary: frame count, fps handling, and temporal reasoning depth.
• Generators that pair well: diffusion image-to-video, multimodal video generators (for rendering), motion controllers (for camera paths). Use video LLMs to author the plan; use generators to render.

1. Script-to-animatic

• Input: script + character sheets + style bible.
• Output: shot list CSV/JSON, timing, camera moves, keyframes, temp VO timings.
• Render: stills-to-animatic, then upgrade to image-to-video.

2. Video-to-shotlist (reference breakdown)

• Input: reference anime clip.
• Output: per-shot attributes (size, angle, movement, action), palette notes, style tags.
• Use: match a director’s style or produce a learning set of prompts.

3. Image-to-video motion authoring

• Input: key art or panels.
• Video LLM returns motion prompts (camera path, easing, duration) and continuity notes.
• Feed to your image-to-video tool as structured parameters.

4. Lip-sync + subtitle timing

• Input: dialogue lines and rough takes.
• Output: timestamps and phoneme hints; generate viseme curves or subtitle SRT.

5. Continuity control

• Maintain persistent character descriptors (hair color, eye shape, outfit layers) and enforce across scenes using LLM checks before rendering.

Use constrained outputs so your generator can consume them directly. Example JSON for a shot list:

{ "scene_id": "S03", "shot_id": "S03-005", "start": 12.40, "end": 16.00, "shot_size": "MCU", "angle": "low", "camera_move": {"type": "push-in", "easing": "easeInOut", "duration": 3.6}, "subject": "Hero", "action": "turns, determined look", "lighting": "cool rim, warm key", "palette": ["#121A2C", "#FFB86C"], "style_tags": ["cel", "hard shadows", "thin lines"], "prompt": "anime cel shading, thin ink lines, dramatic low angle", "neg_prompt": "motion smear, off-model face", "fps": 24, "aspect": "16:9" }

Prompt tips:

• Ask for fixed keys and explicit units (seconds, fps).
• Enforce enumerations for shot_size, angle, and camera_move.
• Request short, atomic sentences to avoid ambiguity.

• Structural: cut accuracy, duration error per shot, coverage of script beats.
• Visual: temporal consistency (hair, eyes, outfit), palette drift, line weight stability.
• Audio: lip-sync offset (ms), subtitle timing error, ADR alignment.
• Motion: camera jerk/judder, unintended zoom/perspective warp.
• User tests: clarity of staging, readability of action, emotional intent.

• Timing drift: models may misread fps; always normalize to project fps.
• Hallucinated camera terms: constrain to a glossary.
• Copyright: do not ingest unlicensed footage; follow tool ToS.
• Over-specified prompts can fight the renderer; start minimal and iterate.
• Long videos: chunk by scene; pass a rolling context summary to maintain continuity.

• Define your shot taxonomy and camera glossary upfront.
• Collect 3–5 reference clips for style and pacing.
• Create JSON schemas for shot lists, motion prompts, and lip-sync.
• Build a validation script to catch missing keys and unit mismatches.
• Iterate: script → animatic → image-to-video → polish passes.

Are video LLMs the same as video generators?

• No. LLMs analyze/plan; generators render frames.

Can I use still panels as input?

• Yes. Provide ordered frames; include intended fps and durations.

How do I keep characters on-model?

• Use character sheets in context and add automated LLM checks before render.

What’s a safe fps workflow?

• Normalize inputs to project fps (e.g., 24). Convert timestamps after analysis.