Add ShotSpec / Storyboard planning layer for controllable video generation

[ninghan11111-ai]

Summary

Introduce a ShotSpec / Storyboard planning layer before script generation, timeline planning, and future generative-shot workflows. The goal is to make OpenStoryline plan what each shot should do before selecting, generating, editing, and rendering media.

This issue is intentionally design-first. Phase 1 should not require Seedance integration or a new UI. It should establish a stable intermediate representation that later nodes can consume.

Motivation

The current workflow is mostly node-driven:

media -> shot splitting -> visual understanding -> filtering/grouping -> script -> timeline -> render

This works for automatic editing, but the system does not yet have an explicit shot-level plan that explains:

• why each shot exists
• what visual subject it should contain
• whether it should come from user media, search media, existing clips, or future video generation
• how it maps to narration, style, duration, and timeline placement

Adding a storyboard layer would make the pipeline more controllable, easier to debug, and easier to extend with Seedance or other video generation providers.

Proposed Scope

Phase 1: Add the planning layer only

Add a new plan_storyboard node that produces a structured storyboard from existing upstream outputs.

Proposed workflow:

load_media
  -> split_shots
  -> understand_clips
  -> filter_clips
  -> group_clips
  -> plan_storyboard
  -> generate_script
  -> generate_voiceover
  -> select_bgm
  -> plan_timeline_pro
  -> render_video

The first implementation should be backward-compatible:

• if storyboard exists, downstream nodes may use it
• if storyboard does not exist, existing behavior should continue

Proposed Data Model

Add schema objects similar to:

class ShotSpec(BaseModel):
    shot_id: str
    purpose: str
    source_type: Literal["user_media", "pexels", "seedance", "existing_clip"] = "existing_clip"
    duration_ms: int
    visual_prompt: str
    narration_hint: str = ""
    required_subject: str = ""
    mood: str = ""
    group_id: Optional[str] = None
    clip_ids: List[str] = Field(default_factory=list)
    locked: bool = False


class Storyboard(BaseModel):
    title: str
    target_duration_ms: int
    style: str
    shots: List[ShotSpec]


class PlanStoryboardInput(BaseModel):
    mode: Literal["auto", "skip", "default"] = "auto"
    user_request: str = ""

The exact field names can be adjusted during implementation, but the important part is to create a stable shot-level planning contract.

Proposed Node

Add PlanStoryboardNode under src/open_storyline/nodes/core_nodes/plan_storyboard.py.

Proposed node metadata:

meta = NodeMeta(
    name="plan_storyboard",
    description="Plan a shot-level storyboard before script generation and timeline planning",
    node_id="plan_storyboard",
    node_kind="storyboard",
    require_prior_kind=["group_clips", "understand_clips"],
    default_require_prior_kind=["group_clips"],
    next_available_node=["generate_script", "plan_timeline_pro"],
)

Expected output shape:

{
  "storyboard": {
    "title": "Coffee shop vlog",
    "target_duration_ms": 30000,
    "style": "warm, light, lifestyle short video",
    "shots": [
      {
        "shot_id": "shot_001",
        "purpose": "opening",
        "source_type": "existing_clip",
        "duration_ms": 2500,
        "visual_prompt": "Exterior shot of the coffee shop to establish the scene",
        "narration_hint": "Today we found a very cozy coffee shop",
        "required_subject": "coffee shop exterior",
        "mood": "relaxed",
        "group_id": "group_0001",
        "clip_ids": ["clip_0003"],
        "locked": false
      }
    ]
  }
}

Prompt Templates

Add prompt templates for the new node:

prompts/tasks/plan_storyboard/zh/system.md
prompts/tasks/plan_storyboard/zh/user.md
prompts/tasks/plan_storyboard/en/system.md
prompts/tasks/plan_storyboard/en/user.md

The prompt should ask the model to output strict JSON matching the storyboard schema.

Downstream Compatibility

generate_script

Enhance generate_script to optionally read storyboard and generate narration/subtitles according to shots[].

Fallback behavior:

• if no storyboard is present, keep the current group_clips-based logic.

plan_timeline_pro

Enhance plan_timeline_pro to optionally read storyboard:

• preserve shot order from shots[]
• prefer shot.clip_ids when available
• use duration_ms as a planning hint
• fallback to current timeline logic if no storyboard is present

Artifact Traceability

For future debugging and UI support, artifacts should eventually record the mapping between planned shots and actual media:

{
  "shot_id": "shot_003",
  "planned_source_type": "seedance",
  "actual_source": "generated_clip_001",
  "actual_path": ".../shot_003.mp4",
  "status": "ok"
}

This is useful even before video generation is implemented, because it makes planned-vs-actual behavior inspectable.

Future Seedance Extension

This issue should reserve the path for Seedance without requiring it in Phase 1.

Future extension ideas:

• source_type = "seedance" for missing b-roll, transition shots, product detail shots, or emotional cutaways
• use ShotSpec.visual_prompt as the generation prompt
• optionally use previous/next shot frames as references
• save generated clip paths and provider metadata into artifacts
• allow generated shots to be reviewed, regenerated, locked, or replaced

Possible future generation metadata:

{
  "shot_id": "shot_003",
  "source_type": "seedance",
  "visual_prompt": "Close-up of a latte on a wooden table with warm sunlight",
  "duration_ms": 2500,
  "generation": {
    "provider": "seedance",
    "aspect_ratio": "9:16",
    "reference_clip_id": "clip_0002",
    "previous_shot_id": "shot_002",
    "next_shot_id": "shot_004",
    "status": "planned"
  }
}

Non-Goals for Phase 1

• Do not implement Seedance generation yet
• Do not build a complex storyboard UI yet
• Do not add automatic VLM review/retry yet
• Do not change the existing render path unless storyboard data is present
• Do not require existing workflows to use storyboard immediately

Acceptance Criteria

• A new plan_storyboard node can be registered as an MCP tool
• The node can produce a valid storyboard artifact from upstream clip/group data
• The schema is documented in code and validates model output
• Existing workflows still work when plan_storyboard is skipped or unavailable
• generate_script and/or plan_timeline_pro can optionally consume storyboard data without breaking fallback behavior
• The design leaves a clear extension path for Seedance-generated shots