diff --git a/.agents/skills/3d-asset-generation/SKILL.md b/.agents/skills/3d-asset-generation/SKILL.md new file mode 100644 index 00000000..6a7416a2 --- /dev/null +++ b/.agents/skills/3d-asset-generation/SKILL.md @@ -0,0 +1,101 @@ +--- +name: 3d-asset-generation +description: Generate, reconstruct, inspect, and route production 3D assets for OpenMontage worlds using Atlas Cloud, fal.ai, licensed catalogs, and Blender. +--- + +# 3D Asset Generation + +Use this skill when a production needs real meshes rather than primitive stand-ins. +It complements `threejs-world-generation`: that skill owns semantic world planning; +this skill owns how unique and repeated meshes enter the world with provenance. + +## Route by asset role + +| Need | Tool | Model/path | Why | +|---|---|---|---| +| Repeated vegetation, rocks, generic props | `threejs_asset_catalog` | CC0 Kenney catalog | Free, coherent, instancing-friendly | +| Unique object described in words | `atlas_3d` | `tripo-h3.1/text-to-3d` | Direct textured/PBR GLB with seeds and face limit | +| Object matching a concept image | `fal_3d` | Hunyuan 3D v3.1 Rapid image-to-3D | Better silhouette/style conditioning from one image | +| Several objects extracted from one regional composition | `fal_3d` | SAM 3D Objects | Individual and combined GLBs plus placement metadata | +| Terrain, composition, lighting, camera, final frames | `blender_world` | Blender 4.5 LTS / Eevee Next | Scene-level control; generated-asset APIs are not world renderers | + +Never ask a text-to-3D model to generate a whole cinematic world in one mesh. +Generate hero objects, use licensed catalogs for high-volume scatter, and assemble +everything in Blender from a semantic specification. + +## Paid-call discipline + +Before every first provider call, state the exact provider, model, operation, +estimated unit cost, and number of requested outputs. Generate one sample before +a batch. As of 2026-08-13: + +- Atlas Tripo H3.1: $0.22 untextured; $0.33 standard textures; $0.44 HD + textures; detailed geometry adds $0.22; quad mesh adds $0.055. +- fal Hunyuan 3D v3.1 Rapid: $0.225 per generation; PBR adds $0.15. +- fal SAM 3D Objects: $0.02 per reconstruction. + +Pricing changes. Confirm the provider page before quoting or running a batch. + +## Asset prompt contract + +Each request describes one isolated object, not a shot: + +1. Name the object and silhouette. +2. Specify construction materials and visible wear. +3. Specify the project's art style and color constraints. +4. State scale and orientation. +5. Exclude ground plane, backdrop, extra objects, labels, and lighting rigs. + +For image-to-3D, use a simple background and make the object occupy more than +half the frame. Request PBR only for assets close enough to benefit from it. + +## Mandatory mesh QA + +Do not approve from the provider thumbnail alone. Import the downloaded artifact +into Blender and inspect: + +- front, rear, and silhouette; +- geometry holes and floating pieces; +- ground contact, scale, and orientation; +- UV seams and missing texture slots; +- base-color, roughness, metallic, and normal response; +- triangle count and whether the intended camera distance justifies it. + +Record provider, model id, prompt, seeds, source page, cost, and output path in +the asset provenance manifest. Failed samples remain failed; do not silently +swap provider or spend on another model. + +### Assembly normalization is mandatory + +Provider and catalog GLBs rarely share units, origins, or up-axis assumptions. +Never compensate with arbitrary scene-level scale guesses. The Blender assembly +spec declares `target_height`; the renderer measures the imported bounding box, +normalizes to that target, and offsets the bounding-box floor to the sampled +terrain height. Review the resulting real-world scale and ground contact. + +Repeated scatter must declare semantic `exclusion_zones` around settlements, +roads, rivers, landmark apertures, and hero camera sightlines. Density that +occludes the subject is not production detail. Waterways and paths must use flat +terrain-following ribbons; beveled 3D curves read as pipes from aerial cameras. + +Landmarks whose reveal timing matters declare visibility windows in the scene +spec. Camera occlusion remains preferred for natural reveals, but deterministic +visibility keys are the hard guarantee for approved timing. + +## World fidelity budget + +A reference-grade region needs three simultaneous density layers: + +- macro: authored terrain silhouettes, waterways, paths, settlements; +- meso: hero buildings, bridges, cliffs, canopy clusters, props; +- micro: ground cover, rocks, flowers, debris, material breakup. + +The asset gate must show global, regional, and walk-height Blender stills. A +wide aerial alone can hide broken contacts; a walk shot alone can hide an empty +world. Primitive-only previews must be labeled `blockout` and cannot pass as a +production-fidelity review. + +For a final animation, render a small bounded frame range first and measure the +per-frame time. Choose the full-render resolution from that measurement rather +than intuition, render a numbered PNG sequence, and call `blender_world` with +`resume: true` after interruption so it starts at the first missing frame. diff --git a/.agents/skills/3d-asset-generation/agents/openai.yaml b/.agents/skills/3d-asset-generation/agents/openai.yaml new file mode 100644 index 00000000..9c385d52 --- /dev/null +++ b/.agents/skills/3d-asset-generation/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "3D Asset Generation" + short_description: "Generate and assemble production 3D assets" + default_prompt: "Use $3d-asset-generation to source, generate, validate, and assemble production-ready 3D assets for this world." diff --git a/.agents/skills/lyria/SKILL.md b/.agents/skills/lyria/SKILL.md new file mode 100644 index 00000000..d9dc780b --- /dev/null +++ b/.agents/skills/lyria/SKILL.md @@ -0,0 +1,128 @@ +--- +name: lyria +description: Generate and validate music with Google Lyria 3 through the Gemini Interactions API. Use before calling OpenMontage `google_music`, designing Lyria 3 Clip or Pro prompts, using image-to-music or custom lyrics, choosing between Lyria 3 and Lyria RealTime, diagnosing Google music-generation failures, or preparing exact-duration music for a video. +--- + +# Google Lyria 3 + +Use Lyria 3 as a single-turn music generator. Keep it distinct from Lyria RealTime, which is an experimental WebSocket model for continuously steered instrumental performance. + +Read [references/api-and-prompting.md](references/api-and-prompting.md) when choosing a model, designing vocals or custom lyrics, using image inputs, debugging the response, or checking current limits and pricing. + +## Required Workflow + +1. Confirm the intended role: underscore, loop, instrumental cue, or full song. +2. Confirm vocals, language, target duration, musical structure, and delivery format. +3. Announce the provider, exact model, estimated per-request cost, and whether the call is exploratory or final. +4. Write one structured prompt using the contract below. +5. Make one approved generation call. Treat a retry as another potentially billable stochastic generation. +6. Preserve the returned provider source unchanged. +7. Probe the actual file and listen before recording duration, format, or approval metadata. +8. Derive a separate production master when the edit requires an exact duration. + +## Choose The Model Deliberately + +| Need | Model | Contract | +|---|---|---| +| Prompt iteration, preview, loop, exact 30-second source | `lyria-3-clip-preview` | Always generates a 30-second MP3; currently $0.04/request | +| Full song, vocals, longer structure, image-conditioned score | `lyria-3-pro-preview` | Prompt-influenced duration up to roughly three minutes; currently $0.08/request | +| Live, continuously steered instrumental performance | `lyria-realtime-exp` | Separate WebSocket workflow; do not route through `google_music` | + +The current OpenMontage `google_music` adapter is locked to `lyria-3-pro-preview`. It does not expose Clip, WAV response selection, multiple images, or RealTime controls. Surface that limitation rather than implying those options are available through the adapter. + +Do not change models silently. For a 30-second video, either obtain approval for Pro plus exact-duration mastering or use Clip through an explicitly supported path. + +## Build The Prompt + +Specify, in this order: + +1. **Purpose and duration** — what the music supports and the requested length. +2. **Genre and era** — use musical vocabulary, not a living artist imitation. +3. **Tempo and harmony** — BPM or tempo range, meter, key or tonal center. +4. **Instrumentation and texture** — name lead, rhythm, bass, and ambient layers. +5. **Structure** — timestamped sections or `[Intro]`, `[Verse]`, `[Chorus]`, `[Bridge]`, `[Outro]`. +6. **Dynamics and synchronization** — entrances, rests, builds, hits, and holds tied to edit times. +7. **Vocal policy** — instrumental-only constraints, a vocal profile, or clearly separated custom lyrics. +8. **Mix and ending** — density, foreground/background role, headroom character, and final decay. +9. **Exclusions** — unwanted vocals, instruments, gestures, clichés, abrupt endings, or copyrighted material. + +For video underscore, use timestamp windows that cover the full requested duration. Ask for one primary change per window and identify the exact synchronization moment. + +For instrumental-only output, say all of the following when they matter: + +```text +Instrumental only. No lead or backing vocals, speech, choir, humming, +vocal chops, spoken samples, lyrical fragments, or recognizable quotations. +``` + +For custom lyrics, put performance direction before a separate `Lyrics:` block and use section labels. Prompt in the language the singer should use. + +## Direct Vocals Deliberately + +When vocals are requested, define these before writing the prompt: + +1. **Vocal role** — solo lead, duet, call-and-response, backing ensemble, or vocal texture. +2. **Language and script** — name the sung language and keep the custom lyrics in one intentional script; do not silently transliterate or code-switch. +3. **Singer profile** — voice type or range, timbre, intensity, diction, ornamentation, and emotional distance. Do not imitate a named artist. +4. **Section behavior** — state where the lead enters, where harmonies or echoes appear, and which sections remain instrumental. +5. **Lyric contract** — separate directions from a `Lyrics:` block, use `[Verse]`, `[Chorus]`, `[Bridge]`, and `[Outro]`, and reserve parentheses for intentional backing-vocal echoes. + +Treat the returned vocal as untrusted until auditioned. Check lyric adherence, language drift, pronunciation, intelligibility, unwanted backing vocals, vocal/instrument balance, and whether the performance follows the requested emotional arc. A technically valid file with poor diction or altered lyrics is not an approved vocal result. + +## Treat Duration As Untrusted Until Probed + +Lyria 3 Pro duration is controlled through prompt instructions and timestamps, not an exact API parameter. The OpenMontage adapter appends a target-duration instruction, but its returned `duration_seconds` field is the request, not a media probe. + +Always inspect the generated file: + +```bash +ffprobe -v error -show_entries \ + format=duration,format_name,bit_rate:stream=codec_name,sample_rate,channels \ + -of json output.mp3 +``` + +If exact duration is required: + +- keep the provider source untouched; +- record requested and measured durations separately; +- derive a new master by trimming at a musically sensible boundary and applying a short fade; +- do not stretch, loop, or regenerate without the approved production plan; +- record the derivation and probe the master again. + +## Authenticate And Diagnose Safely + +The Gemini API commonly uses `GEMINI_API_KEY`. OpenMontage also supports `GOOGLE_API_KEY` and Vertex service-account credentials. + +- Use one known credential path per run. +- Never print keys or edit credential files while debugging. +- Do not assume a rejected first key will fall through to a second configured key. +- In the current OpenMontage resolver, `GOOGLE_API_KEY` takes precedence over `GEMINI_API_KEY` when both are non-empty. +- Treat `403`, `API_KEY_SERVICE_BLOCKED`, and project/service restrictions as authentication or Google-project configuration failures, not prompt-quality failures. +- Do not spend retries on permission failures. Resolve the credential/project path first. +- Retry only transient rate-limit or timeout failures within the approved retry and budget policy. + +## Parse And Record The Result + +Prefer `interaction.output_audio`. For interleaved responses, traverse `model_output` steps and select the audio block; capture output text separately if lyrics or a structure description are relevant. + +Record: + +- provider and exact model; +- original prompt and any image provenance; +- requested duration and probed duration; +- actual codec, sample rate, channels, and file path; +- cost per call and total attempts; +- whether vocals were requested and whether any were detected by listening; +- the provider source and any separately derived production master. + +## Quality Checklist + +- The prompt states purpose, tempo, instruments, structure, dynamics, vocal policy, and ending. +- Timestamp windows cover the intended runtime without contradictions. +- No artist impersonation or copyrighted lyrics are requested. +- The output file exists, is non-empty, decodes, and contains an audio stream. +- Actual duration and technical properties come from a probe, not request metadata. +- Instrumental output is checked for accidental vocal material. +- The opening, synchronization moments, transitions, and ending are auditioned. +- The untouched source is preserved and any production master has explicit provenance. +- SynthID watermarking and preview-model instability are acknowledged where provenance matters. diff --git a/.agents/skills/lyria/agents/openai.yaml b/.agents/skills/lyria/agents/openai.yaml new file mode 100644 index 00000000..a9840762 --- /dev/null +++ b/.agents/skills/lyria/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Google Lyria" + short_description: "Generate and validate music with Lyria 3" + default_prompt: "Use $lyria to design a production-ready Lyria 3 music generation prompt and verification plan." diff --git a/.agents/skills/lyria/references/api-and-prompting.md b/.agents/skills/lyria/references/api-and-prompting.md new file mode 100644 index 00000000..79797443 --- /dev/null +++ b/.agents/skills/lyria/references/api-and-prompting.md @@ -0,0 +1,188 @@ +# Lyria 3 API and Prompting Reference + +Last verified: 2026-07-18. + +## Provider Families + +### Lyria 3 Clip + +- Model: `lyria-3-clip-preview` +- Best for: prompt iteration, previews, loops, short cues +- Input: text or images +- Output: MP3 audio plus text containing lyrics or song structure +- Duration: always 30 seconds +- Gemini Developer API price: $0.04 per request; no free tier + +### Lyria 3 Pro + +- Model: `lyria-3-pro-preview` +- Best for: full songs, vocals, verses, choruses, bridges, longer scores +- Input: text or up to 10 images through the underlying API +- Output: MP3 by default; the underlying Pro API can request audio/WAV response format +- Duration: prompt-influenced, up to roughly three minutes; not an exact media contract +- Gemini Developer API price: $0.08 per request; no free tier + +### Lyria RealTime + +- Model: `lyria-realtime-exp` +- Experimental, instrumental, persistent WebSocket session +- Supports continuous steering of BPM, scale, density, brightness, and guidance +- Not the Interactions API and not the model used by OpenMontage `google_music` + +## Interactions API Shape + +Python: + +```python +from google import genai + +client = genai.Client() +interaction = client.interactions.create( + model="lyria-3-pro-preview", + input="A structured instrumental score ...", +) + +audio = interaction.output_audio +text = interaction.output_text +``` + +The convenience properties are preferred for ordinary responses. For interleaved responses, iterate `interaction.steps`; inspect `model_output` content blocks and keep `audio` and `text` blocks separate. + +The OpenMontage adapter currently: + +- sends a list containing one text block and optionally one image block; +- appends `[Target Duration: N seconds]` to the text; +- calls `client.interactions.create()` with `lyria-3-pro-preview`; +- extracts audio through `output_audio`, then legacy outputs, then step traversal; +- writes MP3 and reports the requested duration without probing the result. + +## Prompt Template: Video Underscore + +```text +Create an original instrumental background score for [purpose], target [N] seconds. + +Style: [genre/blend/era], [mood], [foreground or background role]. +Tempo and harmony: [BPM], [meter], [key/scale or tonal behavior]. +Instrumentation: [lead], [rhythm], [bass], [texture]. + +[0:00-0:06] [one musical state and entrance rule]. +[0:06-0:12] [one development]. +[0:12-0:15] [build or reduction that prepares the sync point]. +[0:15-0:18] [exact synchronization event and required hold]. +[0:18-0:30] [release and ending behavior]. + +Mix: [density, space, transient character, dynamic range]. +Ending: [tail, final resonance, no abrupt cutoff]. +Instrumental only. No vocals, speech, choir, humming, vocal chops, +spoken samples, lyrical fragments, or recognizable quotations. +Avoid: [unwanted instruments, clichés, oversized hits, abrupt fade]. +``` + +Use timestamps as structural instructions, not as proof that Pro will return an exact-length file. + +## Prompt Template: Song With Custom Lyrics + +```text +Create a [duration] [genre] song in [key] at [BPM]. +Vocal profile: [range, tone, delivery, language]. +Instrumentation and production: [details]. +Structure: [Intro], [Verse], [Chorus], [Bridge], [Outro]. + +Lyrics: +[Verse 1] +... + +[Chorus] +... +``` + +Separate lyrics from production direction. Use round brackets only when backing-vocal echoes are desired. + +## Vocals And Custom Lyrics + +Lyria generates vocals and lyrics unless the prompt clearly requests an instrumental. Google recommends a detailed singer profile covering voice type or gender presentation, timbre, and vocal range. The model generates lyrics in the language of the prompt, and an explicit language instruction can override that default. + +For a vocal song, specify: + +- **Role:** solo lead, duet, call-and-response, backing ensemble, or wordless texture. +- **Profile:** range, timbre, delivery, diction, intensity, ornamentation, and register changes. +- **Language:** the sung language, intended register, and whether any code-switching is allowed. +- **Section map:** where vocals begin, where harmonies enter, which passage is instrumental, and how the vocal exits. +- **Backing policy:** use parentheses only for intentional echoes or backing singers. +- **Exclusions:** unwanted choir, spoken phrases, ad-libs, humming, vocal chops, or named-artist imitation. + +Keep production direction outside the lyric block: + +```text +Create a [duration] [genre] song in [key] at [BPM]. +Vocal role: [solo, duet, call-and-response, or ensemble]. +Singer profile: [range], [timbre], [delivery], and [diction]. +Sing in [language and intended register]. Use backing vocals only for words +shown in parentheses. No [unwanted vocal gestures or effects], and do not +imitate a named singer. + +Lyrics: +[Verse 1] +[custom lyrics in the selected language and script] + +[Chorus] +[lead line] ([intentional backing-vocal echo]) +``` + +### Multilingual Work + +- Name the sung language explicitly even when custom lyrics make it apparent. +- Choose the writing system or transliteration deliberately; do not convert scripts without user approval. +- Keep one script inside a lyric version so the model does not infer accidental language switches. +- Describe the intended register, dialect, formality, and any intentional code-switching. +- Preserve the approved lyric text separately from the generation prompt. Lyria may alter, omit, or repeat words, so compare the returned text and audible performance against that source. +- Treat pronunciation and language coverage as output-level QA. The API documentation says Lyria follows the prompt language, but it does not promise perfect diction for every language or regional register. + +### Vocal QA + +Audition and record: + +1. Whether the requested lead, duet, or ensemble roles are present. +2. Whether every approved lyric line is sung, omitted, altered, or repeated. +3. Whether the language, script-derived pronunciation, and intended register remain consistent. +4. Whether the requested range, timbre, dynamics, ornamentation, and section entrances are followed. +5. Whether backing vocals appear only where requested. +6. Whether the lead remains intelligible against the arrangement without clipping or masking. +7. Whether the ending contains a complete final phrase and musically useful decay. + +Do not classify a diction or lyric-adherence failure as a file-format problem. It is a stochastic prompt/adherence failure; any retry is a new paid generation and requires the approved retry policy. + +## Image-To-Music + +The underlying API accepts up to 10 base64-encoded images with MIME types. Prompt for the musical interpretation explicitly: mood, palette-to-timbre mapping, motion-to-rhythm mapping, and the desired structure. The current OpenMontage adapter accepts one `image_path` or one `image_url`. + +## Limitations And Safety + +- Generation is stochastic; identical prompts can differ. +- Lyria 3 is single-turn; generated tracks cannot currently be refined through a multi-turn edit chain. +- Safety filters may reject artist-voice imitation, copyrighted lyrics, or other restricted content. +- All generated audio contains an imperceptible SynthID watermark. +- Preview models and rate limits can change before stable release. +- Official pages currently disagree between 44.1 kHz and 48 kHz descriptions. Probe every returned file and record the observed value. + +## Failure Triage + +| Symptom | Class | Action | +|---|---|---| +| `401` or invalid key | Auth | Verify the selected credential without printing it | +| `403`, `API_KEY_SERVICE_BLOCKED` | Auth/project policy | Stop retries; fix the Google key, project, API enablement, or service restriction | +| `429` | Rate limit | Retry only within the approved retry/budget policy | +| Timeout | Transient/provider | Use the tool timeout policy; do not launch parallel duplicate paid calls | +| No `output_audio` | Response parsing | Traverse `model_output` step content for an `audio` block | +| Output longer/shorter than requested | Model behavior | Preserve source, probe, and create a separate exact-duration master | +| Accidental vocals | Prompt/adherence | Strengthen the full instrumental exclusion list; audition the next paid result | +| Safety rejection | Policy/prompt | Remove artist imitation or copyrighted material; do not disguise the request | + +## Official Sources + +- Gemini API music generation: https://ai.google.dev/gemini-api/docs/music-generation +- Gemini Interactions API: https://ai.google.dev/gemini-api/docs/interactions-overview +- Gemini Developer API pricing: https://ai.google.dev/gemini-api/docs/pricing +- Google DeepMind Lyria prompt guide: https://deepmind.google/models/lyria/prompt-guide/ +- Lyria RealTime: https://ai.google.dev/gemini-api/docs/realtime-music-generation +- Google announcement for Lyria 3 Pro: https://blog.google/innovation-and-ai/technology/ai/lyria-3-pro/ diff --git a/.agents/skills/manimgl-best-practices/templates/3d_scene.py b/.agents/skills/manimgl-best-practices/templates/3d_scene.py index 5c2da873..a9c13dca 100644 --- a/.agents/skills/manimgl-best-practices/templates/3d_scene.py +++ b/.agents/skills/manimgl-best-practices/templates/3d_scene.py @@ -254,5 +254,5 @@ class LightingTemplate(Scene): if __name__ == "__main__": - import os - os.system(f"manimgl {__file__} ThreeDSceneTemplate") + import subprocess + subprocess.run(["manimgl", __file__, "ThreeDSceneTemplate"]) diff --git a/.agents/skills/manimgl-best-practices/templates/basic_scene.py b/.agents/skills/manimgl-best-practices/templates/basic_scene.py index c46d65ad..1df13734 100644 --- a/.agents/skills/manimgl-best-practices/templates/basic_scene.py +++ b/.agents/skills/manimgl-best-practices/templates/basic_scene.py @@ -130,5 +130,5 @@ class AnimationShowcase(Scene): if __name__ == "__main__": # This allows you to run: python basic_scene.py # (though using manimgl is recommended) - import os - os.system(f"manimgl {__file__} BasicSceneTemplate") + import subprocess + subprocess.run(["manimgl", __file__, "BasicSceneTemplate"]) diff --git a/.agents/skills/manimgl-best-practices/templates/math_scene.py b/.agents/skills/manimgl-best-practices/templates/math_scene.py index 201d7d8c..bdab3585 100644 --- a/.agents/skills/manimgl-best-practices/templates/math_scene.py +++ b/.agents/skills/manimgl-best-practices/templates/math_scene.py @@ -329,5 +329,5 @@ class MatrixTemplate(Scene): if __name__ == "__main__": - import os - os.system(f"manimgl {__file__} MathSceneTemplate") + import subprocess + subprocess.run(["manimgl", __file__, "MathSceneTemplate"]) diff --git a/.agents/skills/threejs-world-generation/SKILL.md b/.agents/skills/threejs-world-generation/SKILL.md new file mode 100644 index 00000000..1ced8df4 --- /dev/null +++ b/.agents/skills/threejs-world-generation/SKILL.md @@ -0,0 +1,130 @@ +--- +name: threejs-world-generation +description: Build deterministic, editable, free-viewpoint Three.js worlds from text or structured briefs. Use for cinematic 3D terrain, semantic regions, procedural biomes, explicit landmarks, environmental scattering, camera fly-throughs, world diagnostics, or requests for a real 3D environment rather than generated 2D footage. Integrates OpenMontage's threejs_world tool with HyperFrames; do not use for a single isolated 3D object or a flat parallax scene. +--- + +# Three.js World Generation + +For production meshes and Blender assembly, also read `3d-asset-generation`. +Three.js remains the semantic interactive/blockout renderer; Blender is the +production renderer when the brief calls for dense reference-grade scenery. + +The production handoff must include target dimensions for imported assets, +semantic scatter exclusion zones, terrain-following water/path geometry, +landmark visibility policy, camera clearance, and global/regional/walk review +frames. These are world-spec contracts, not manual Blender cleanup notes. + +Create a persistent scene graph, not a sequence of unrelated 2D shots. Preserve the user's explicit constraints, infer missing construction details separately, establish the global terrain first, and refine selected regions without disturbing the world-wide spatial contract. + +## Choose the fidelity tier explicitly + +- `blockout`: procedural primitives, vertex colors, semantic/layout validation, fast iteration. Never call this production-quality, reference-grade, or visually equivalent to WorldClaw. +- `production`: licensed local GLTF/GLB catalogs, a minimum eight-model palette across four semantic categories, three PBR terrain layers, asset provenance, walk-level repetition review, and no primitive landmark fallback. + +For a hero video or any reference showing populated textured environments, use `production`. If its catalog/material/provider requirements cannot be met, stop at preflight or the asset gate. Do not render a blockout as the final deliverable. + +## Read first + +- Read [references/worldclaw-principles.md](references/worldclaw-principles.md) when planning or explaining the coarse-to-fine method. +- Read [references/world-spec.md](references/world-spec.md) before authoring a `world_spec` or calling `threejs_world`. +- Read `hyperframes-core`, `hyperframes-animation`, and `hyperframes-animation/adapters/three.md` before editing the emitted workspace. +- Read `threejs-loaders`, `threejs-materials`, `threejs-textures`, `threejs-lighting`, and `threejs-postprocessing` for production-tier work. + +## Route the request + +- Use the `animation` pipeline for design-led, explanatory, abstract, or music-led world films. +- Use the `cinematic` pipeline for trailer-like mood, dramatic reveals, or source-plus-world edits. +- Choose HyperFrames when the deliverable is the code-native Three.js world. Choose Blender for reference-grade hero rendering and FFmpeg only to package Blender's numbered frames and approved audio. Record that choice at proposal; do not silently switch after approval. +- Keep this as a capability inside existing pipelines. Do not create a new pipeline merely because a scene is 3D. + +## Workflow + +### 1. Separate intent from completion + +Record two lists before planning: + +- `explicit_constraints`: only facts the user supplied. +- `inferred_details`: scale, region coverage, terrain operators, densities, palette refinements, and camera details added to make the world executable. + +Never smuggle an inferred landmark, biome, or story beat into the explicit list. + +### 2. Plan globally + +Author one shared `world_spec` containing: + +- world scale, terrain resolution, elevation range, and seed; +- semantic regions with normalized centers, radii, landform operators, palette, and scatter recipes; +- atmosphere and lighting shared across all regions; +- explicit landmarks with stable IDs and world-space placement; +- a complete camera path with time, position, target, and field of view. + +Prefer 3-7 regions. Each region must contribute a distinct silhouette, surface read, or functional role. + +### 3. Build the terrain foundation + +For production, first call `threejs_asset_catalog` to install rights-safe catalogs under `projects//assets/3d/catalogs//`. Record source, license, archive hash, model inventory, and every selected model in the asset manifest. Then call `threejs_world` with `quality_tier: "production"` and the installed catalog paths. + +```python +from tools.graphics.threejs_world import ThreeJSWorld + +result = ThreeJSWorld().execute({ + "operation": "build", + "world_spec": world_spec, + "output_path": "projects//hyperframes", + "duration_seconds": 60, + "render_mode": "cinematic", + "quality_tier": "production", + "asset_catalog_paths": ["projects//assets/3d/catalogs/kenney-nature-kit"], +}) +``` + +Treat `world.json`, `world-spec.js`, `world-runtime.js`, and `world-report.json` as editable assets. Do not flatten them into a video until the assets gate is approved. + +### 4. Inspect regionally + +Build a second pass with `render_mode: "semantic"` or `"wireframe"` when spatial problems are hard to see in the cinematic material pass. Inspect snapshots from global, regional, and walk-level viewpoints. + +Maintain an issue queue with stable subjects: + +- terrain transition or silhouette; +- landmark scale, pose, or contact; +- scatter density, slope rejection, or repetition; +- material contrast and atmosphere; +- camera clearance, clipping, or weak framing. + +Fix only the affected region or object when possible. Preserve the seed, region IDs, camera times, and unrelated parameters. + +### 5. Refine with bounded loops + +Run at most three render-guided refinement rounds: + +1. build the workspace; +2. run the unified HyperFrames `check` gate and snapshot representative times; +3. inspect frames and update the issue queue; +4. change the narrowest relevant spec fields; +5. rebuild with the same seed and compare. + +Stop when no substantial issue remains or the iteration budget is reached. Report residual limitations rather than disguising them with overlays. + +### 6. Compose without overwriting + +For browser-native delivery, set `render_runtime: "hyperframes"` and `composition_mode: "atelier"`; `video_compose` must preserve the authored workspace. For reference-grade video, render a Blender PNG sequence with `resume: true`, then set `render_runtime: "ffmpeg"` for packaging. Preserve the world spec and `.blend` as the editable source of truth. + +## Quality gates + +- Terrain is continuous and region boundaries blend without obvious seams. +- Every landmark touches its support surface and remains inside world bounds. +- Scatter respects region affinity, slope limits, and deterministic seed behavior. +- Global, regional, and walk-level frames all read as the same continuous world. +- Camera paths remain above terrain, avoid clipping, and provide at least one scale-establishing reveal. +- World source remains editable after render: regions, landmarks, camera keys, and palette have stable IDs or fields. +- HyperFrames `check` and post-render review pass before delivery. Use the legacy `validate` or `inspect` operations only when supporting an older runtime. +- Production beauty frames contain textured assets at foreground, midground, and background depths; no dominant object may read as an untextured box, cone, octahedron, or dodecahedron. +- Production requires at least four semantic asset categories, eight distinct models, three PBR terrain layers, one regional composition review per camera-critical region, and explicit repetition/contact findings. + +## Boundaries + +- The production catalog path materially improves geometry and surface richness, but it still does not reproduce WorldClaw's GPT-Image-2, SAM3, SAM3D, Hunyuan3D, BlenderMCP, or four-H20 implementation. +- Do not claim articulated assets, game physics, navigation meshes, or interaction logic unless another tool explicitly adds them. +- Do not use unseeded randomness, wall-clock animation, remote models, or render-time asset fetches. +- Do not delete the lower-level `threejs-*` skills. They are the subsystem references used when extending this runtime. diff --git a/.agents/skills/threejs-world-generation/agents/openai.yaml b/.agents/skills/threejs-world-generation/agents/openai.yaml new file mode 100644 index 00000000..151a3232 --- /dev/null +++ b/.agents/skills/threejs-world-generation/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Three.js World Generation" + short_description: "Build semantic, editable cinematic 3D worlds" + default_prompt: "Use $threejs-world-generation to turn this world brief into a deterministic, editable Three.js world and camera sequence." diff --git a/.agents/skills/threejs-world-generation/references/world-spec.md b/.agents/skills/threejs-world-generation/references/world-spec.md new file mode 100644 index 00000000..aa4a79fd --- /dev/null +++ b/.agents/skills/threejs-world-generation/references/world-spec.md @@ -0,0 +1,136 @@ +# `threejs_world` specification + +Use normalized region coordinates in `[-1, 1]`; the tool converts them to world space. Positions for landmarks and cameras are world-space `[x, y, z]` values. + +## Minimal shape + +```json +{ + "version": "1.0", + "title": "The Luminous Divide", + "seed": 2048, + "explicit_constraints": ["a volcanic rift divides a living valley"], + "inferred_details": ["three semantic regions", "dawn atmosphere"], + "world": { + "size": 120, + "resolution": 160, + "elevation_scale": 18, + "water_level": -1.5 + }, + "atmosphere": { + "sky_color": "#07111f", + "fog_color": "#13263a", + "fog_density": 0.009, + "sun_color": "#ffd7a3", + "sun_intensity": 3.2, + "sun_position": [45, 70, 20] + }, + "terrain_materials": [ + { + "id": "mossy-ground", + "regions": ["ember-rift"], + "base_color": "assets/materials/mossy-ground/diffuse.jpg", + "normal": "assets/materials/mossy-ground/normal.jpg", + "roughness": "assets/materials/mossy-ground/roughness.jpg", + "meters_per_repeat": 7 + } + ], + "asset_palette": [ + { + "id": "rift-tree-a", + "catalog_id": "kenney-nature-kit", + "model_id": "tree-pine-a", + "category": "tree", + "region_id": "ember-rift", + "count": 80, + "scale_range": [0.8, 1.5] + } + ], + "regions": [ + { + "id": "ember-rift", + "label": "Ember Rift", + "center": [-0.35, 0.12], + "radius": 0.58, + "base_elevation": 0.3, + "amplitude": 1.0, + "frequency": 1.4, + "landform": "ridge", + "blend_width": 0.2, + "color": "#5b271f", + "accent_color": "#ff6b2c", + "scatter": {"rock": 90, "crystal": 22, "tree": 0} + } + ], + "landmarks": [ + { + "id": "rift-gate", + "type": "arch", + "region_id": "ember-rift", + "position": [-24, 0, 8], + "scale": 5.5, + "color": "#2a2020", + "accent_color": "#ff7a35" + } + ], + "camera_path": [ + {"time": 0, "position": [72, 42, 72], "target": [0, 3, 0], "fov": 46}, + {"time": 60, "position": [-54, 13, -32], "target": [-12, 4, 5], "fov": 40} + ] +} +``` + +## Regions + +Required: `id`, `center`, `radius`, `color`. + +Useful fields: + +- `label`: human-facing diagnostic label. +- `base_elevation`: normalized vertical offset before `elevation_scale`. +- `amplitude`: relief contribution. +- `frequency`: macro noise frequency. +- `landform`: `plain`, `peak`, `ridge`, `dune`, `terrace`, `basin`, or `canyon`. +- `blend_width`: softness of the semantic boundary. +- `accent_color`: used by semantic and environmental details. +- `scatter`: counts for `tree`, `rock`, and `crystal` prototypes. +- `slope_limit`: maximum accepted slope proxy for scattered instances. + +Region weights are normalized at every terrain sample. The fallback region must still cover the full domain, so avoid tiny isolated regions with no broad neighbor. + +## Landmarks + +Supported procedural types: `monolith`, `arch`, `tower`, `ruin`, `crystal`, `settlement`, and `ring`. + +Each landmark is placed at sampled terrain height. `position[1]` is an additional vertical offset, not an absolute Y coordinate. Keep IDs stable through refinement so review notes remain addressable. + +## Camera path + +- Provide at least two keys. +- First key time must be `0`; last key should match the requested duration. +- Keep keys ordered and inside the duration. +- The runtime interpolates position, target, and FOV with smoothstep easing. +- Add higher keys for regional and walk-level passes; do not attempt to encode cuts with teleporting adjacent keys. +- Keep the camera at least `2` world units above sampled terrain unless a deliberate ground skim is reviewed. + +## Render modes + +- `cinematic`: PBR vertex colors, fog, water, shadows, overlays. +- `semantic`: saturated region colors and labels for layout diagnosis. +- `wireframe`: terrain topology and explicit scene nodes for geometry diagnosis. + +## Tool outputs + +`operation: "build"` writes: + +- `index.html`: HyperFrames composition root. +- `world.json`: normalized editable specification. +- `world-spec.js`: browser-loadable specification. +- `world-runtime.js`: deterministic Three.js scene construction. +- `world.css`: full-frame canvas and production overlays. +- `world-report.json`: validation, performance estimate, and warnings. +- `hyperframes.json`: local registry configuration. + +Production builds additionally write `asset-catalog-index.json` and `asset-catalog.js`, and copy the selected catalogs into `assets/models/` so rendering never depends on a remote model fetch. + +`operation: "validate"` performs specification checks without writing a workspace. diff --git a/.agents/skills/threejs-world-generation/references/worldclaw-principles.md b/.agents/skills/threejs-world-generation/references/worldclaw-principles.md new file mode 100644 index 00000000..e3b5f383 --- /dev/null +++ b/.agents/skills/threejs-world-generation/references/worldclaw-principles.md @@ -0,0 +1,39 @@ +# WorldClaw principles adapted for OpenMontage + +Source: [WorldClaw: Agentic 3D Open-World Generation at Scale](https://arxiv.org/html/2608.05248v1), Guo et al., arXiv:2608.05248v1 (2026). + +WorldClaw's public repository currently contains the paper and assets, not the executable generation stack. OpenMontage therefore adopts the architectural ideas, not private code or model weights. + +## Transferable architecture + +1. **Intent extraction precedes completion.** Keep user-stated facts separate from inferred construction parameters. +2. **Shared structured intermediates coordinate agents.** A world spec carries regions, terrain, objects, appearance, and spatial relations across stages. +3. **Global constraints precede local detail.** Establish semantic layout, scale, terrain, atmosphere, and major relationships once. +4. **Terrain is the spatial contract.** Use the same region weights for height, palette, scattering, and later placement. +5. **Reusable environmental prototypes differ from functional landmarks.** Scatter rocks, vegetation, and crystals globally; place named structures explicitly. +6. **Local development is selective.** Spend detail and iteration on regions that matter to the camera path or delivery promise. +7. **Objects stay independent.** Stable IDs and transforms preserve editability and replacement. +8. **Placement is contact-aware.** Sample terrain height and slope, reject implausible candidates, align instances, and diagnose floating or penetration. +9. **Refinement is render-guided and bounded.** Use global, regional, walk, semantic, and wireframe views; change the narrowest responsible parameters. +10. **Executable representations improve reuse.** Code-native terrain, materials, placement, and camera paths remain parameterized and animatable. + +## Local mapping + +| WorldClaw concept | OpenMontage implementation | +|---|---| +| Structured scene specification | `world_spec` JSON and tool schema | +| Semantic layout map | Continuous normalized region-weight field | +| Region-aware height field | Weighted procedural landform operators | +| Terrain materials | Blockout: vertex colors. Production: catalogued PBR texture layers | +| Reusable terrain assets | Blockout: primitives. Production: licensed textured GLTF/GLB palettes | +| Regional objects | Stable explicit scene nodes; production forbids primitive landmark fallback | +| Blender refinement agents | HyperFrames snapshots plus agent issue queue | +| Free-viewpoint render | Deterministic Three.js camera path responding to `hf-seek` | +| Editable textured meshes | Editable code-native geometry, materials, regions, and transforms | + +## Deliberate scope differences + +- No single-view object reconstruction, segmentation, or generated PBR texture maps. +- No Blender or Unreal dependency in the current runnable path; this limits reconstruction and offline-render fidelity. +- No claim of photoreal asset diversity comparable to large generative 3D models. +- Stronger portability and determinism for browser-rendered OpenMontage video work. diff --git a/.claude/skills/manimgl-best-practices/templates/3d_scene.py b/.claude/skills/manimgl-best-practices/templates/3d_scene.py index 5c2da873..a9c13dca 100644 --- a/.claude/skills/manimgl-best-practices/templates/3d_scene.py +++ b/.claude/skills/manimgl-best-practices/templates/3d_scene.py @@ -254,5 +254,5 @@ class LightingTemplate(Scene): if __name__ == "__main__": - import os - os.system(f"manimgl {__file__} ThreeDSceneTemplate") + import subprocess + subprocess.run(["manimgl", __file__, "ThreeDSceneTemplate"]) diff --git a/.claude/skills/manimgl-best-practices/templates/basic_scene.py b/.claude/skills/manimgl-best-practices/templates/basic_scene.py index c46d65ad..1df13734 100644 --- a/.claude/skills/manimgl-best-practices/templates/basic_scene.py +++ b/.claude/skills/manimgl-best-practices/templates/basic_scene.py @@ -130,5 +130,5 @@ class AnimationShowcase(Scene): if __name__ == "__main__": # This allows you to run: python basic_scene.py # (though using manimgl is recommended) - import os - os.system(f"manimgl {__file__} BasicSceneTemplate") + import subprocess + subprocess.run(["manimgl", __file__, "BasicSceneTemplate"]) diff --git a/.claude/skills/manimgl-best-practices/templates/math_scene.py b/.claude/skills/manimgl-best-practices/templates/math_scene.py index 201d7d8c..bdab3585 100644 --- a/.claude/skills/manimgl-best-practices/templates/math_scene.py +++ b/.claude/skills/manimgl-best-practices/templates/math_scene.py @@ -329,5 +329,5 @@ class MatrixTemplate(Scene): if __name__ == "__main__": - import os - os.system(f"manimgl {__file__} MathSceneTemplate") + import subprocess + subprocess.run(["manimgl", __file__, "MathSceneTemplate"]) diff --git a/.env.example b/.env.example index 40b442ad..95226750 100644 --- a/.env.example +++ b/.env.example @@ -2,70 +2,101 @@ # Copy this to .env and fill in your keys # --- Image + video gateway --- -FAL_KEY= # FLUX images, Google Veo video, Kling video, MiniMax video, Recraft images - # Get one at https://fal.ai/dashboard/keys -FAL_AI_API_KEY= # Alias for FAL_KEY (some SDKs/docs use this name); either one is read. +# FLUX images, Google Veo video, Kling video, MiniMax video, Recraft images. +# Get one at https://fal.ai/dashboard/keys +FAL_KEY= +# Alias for FAL_KEY (some SDKs/docs use this name); either one is read. +FAL_AI_API_KEY= # --- Replicate --- -REPLICATE_API_TOKEN= # Replicate-hosted video gen (seedance_replicate). Needed to make the - # Replicate-backed Seedance path selectable alongside the fal.ai one. - # Get one at https://replicate.com/account/api-tokens +# Replicate-hosted video gen (seedance_replicate). Needed to make the +# Replicate-backed Seedance path selectable alongside the fal.ai one. +# Get one at https://replicate.com/account/api-tokens +REPLICATE_API_TOKEN= # --- Higgsfield --- -HIGGSFIELD_API_KEY= # Higgsfield Cloud key (higgsfield_video). Pair with the secret below, -HIGGSFIELD_API_SECRET= # or use the combined HIGGSFIELD_KEY=":" form instead. +# Higgsfield Cloud key (higgsfield_video). Pair with the secret below, +# or use the combined HIGGSFIELD_KEY=":" form instead. +HIGGSFIELD_API_KEY= +HIGGSFIELD_API_SECRET= # HIGGSFIELD_KEY= # Combined key:secret — set this INSTEAD of the _KEY/_SECRET pair if you prefer. # --- Kling official direct API --- -KLING_API_KEY= # Official Kling API key; enables video, image, TTS, avatar, lip sync -KLING_API_BASE_URL= # Optional endpoint override; leave blank for default https://api-singapore.klingai.com - # Mainland China accounts can use https://api-beijing.klingai.com +# Official Kling API key; enables video, image, TTS, avatar, lip sync. +KLING_API_KEY= +# Optional endpoint override; leave blank for default https://api-singapore.klingai.com +# Mainland China accounts can use https://api-beijing.klingai.com +KLING_API_BASE_URL= # --- Google (one key unlocks image gen + TTS + video) --- -GOOGLE_API_KEY= # Google Imagen images, Google Cloud TTS (700+ voices, 50+ languages), - # Gemini Omni video (generation + conversational editing, paid tier) - # Get one at https://aistudio.google.com/apikey +# Google Imagen images, Google Cloud TTS (700+ voices, 50+ languages), +# Gemini Omni video (generation + conversational editing, paid tier). +# Get one at https://aistudio.google.com/apikey +GOOGLE_API_KEY= # GEMINI_API_KEY= # Alias for GOOGLE_API_KEY (takes precedence when both are set) # Alternative to the API key: service-account JSON auth. # TTS uses Cloud Text-to-Speech; Imagen routes to Vertex AI. -GOOGLE_APPLICATION_CREDENTIALS= # path to a service-account JSON key file -GOOGLE_CLOUD_PROJECT= # GCP project id (required for Imagen via Vertex AI) -GOOGLE_CLOUD_LOCATION= # Vertex AI region, default us-central1 +# Path to a service-account JSON key file. +GOOGLE_APPLICATION_CREDENTIALS= +# GCP project id (required for Imagen via Vertex AI). +GOOGLE_CLOUD_PROJECT= +# Vertex AI region, default us-central1. +GOOGLE_CLOUD_LOCATION= # --- Voice --- -ELEVENLABS_API_KEY= # TTS narration, music generation, sound effects -OPENAI_API_KEY= # OpenAI TTS fallback and GPT Image 2 image generation -XAI_API_KEY= # Grok image generation/editing and Grok video generation -DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (new console API Key) -DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type, e.g. zh_female_vv_uranus_bigtts +# TTS narration, music generation, sound effects. +ELEVENLABS_API_KEY= +# OpenAI TTS fallback and GPT Image 2 image generation. +OPENAI_API_KEY= +# Grok image generation/editing and Grok video generation. +XAI_API_KEY= +# Volcengine Doubao Speech TTS (new console API Key). +DOUBAO_SPEECH_API_KEY= +# Default Doubao speaker/voice type, e.g. zh_female_vv_uranus_bigtts. +DOUBAO_SPEECH_VOICE_TYPE= # Piper local voices do not require env vars; install `piper-tts` via pip # --- DashScope (Alibaba Cloud Bailian) --- -DASHSCOPE_API_KEY= # Qwen image gen (qwen-image-2.0-pro), TTS (qwen3-tts-flash), ASR with word timestamps (qwen3-asr-flash-filetrans) - # Get one at https://dashscope.aliyun.com/ +# Qwen image gen (qwen-image-2.0-pro), TTS (qwen3-tts-flash), ASR with word timestamps (qwen3-asr-flash-filetrans). +# Get one at https://dashscope.aliyun.com/ +DASHSCOPE_API_KEY= # --- Music --- -SUNO_API_KEY= # Suno AI music generation (full songs, instrumentals, any genre) +# Suno AI music generation (full songs, instrumentals, any genre). +SUNO_API_KEY= # --- Video Generation --- -HEYGEN_API_KEY= # HeyGen API (VEO, Sora, Runway, Kling, Seedance via single key) -RUNWAY_API_KEY= # Runway Gen-4 (direct API, alternative to fal.ai routing) -VIDEO_GEN_LOCAL_ENABLED= # Set to "true" for local video gen (needs GPU + diffusers) -VIDEO_GEN_LOCAL_MODEL= # Local model: wan2.1-1.3b, wan2.1-14b, hunyuan-1.5, ltx2-local, cogvideo-5b -MODAL_LTX2_ENDPOINT_URL= # Modal self-hosted LTX-2 endpoint (optional) +# HeyGen API (VEO, Sora, Runway, Kling, Seedance via single key). +HEYGEN_API_KEY= +# Runway Gen-4 (direct API, alternative to fal.ai routing). +RUNWAY_API_KEY= +# Volcengine Jimeng (即梦 AI) video generation via official API (HMAC-SHA256 V4 signing). +VOLC_ACCESSKEY= +# Secret Access Key paired with VOLC_ACCESSKEY. Get both at https://console.volcengine.com/iam/keymanage +VOLC_SECRETKEY= +# Set to "true" for local video gen (needs GPU + diffusers). +VIDEO_GEN_LOCAL_ENABLED= +# Local model: wan2.1-1.3b, wan2.1-14b, hunyuan-1.5, ltx2-local, cogvideo-5b. +VIDEO_GEN_LOCAL_MODEL= +# Modal self-hosted LTX-2 endpoint (optional). +MODAL_LTX2_ENDPOINT_URL= # --- Stock Media --- -PEXELS_API_KEY= # Pexels stock footage/images (free) -PIXABAY_API_KEY= # Pixabay stock footage/images (free) -UNSPLASH_ACCESS_KEY= # Unsplash stock images (free developer key) +# Pexels stock footage/images (free). +PEXELS_API_KEY= +# Pixabay stock footage/images (free). +PIXABAY_API_KEY= +# Unsplash stock images (free developer key). +UNSPLASH_ACCESS_KEY= # --- Analysis --- -HF_TOKEN= # HuggingFace token — enables speaker diarization in transcriber +# HuggingFace token — enables speaker diarization in transcriber. +HF_TOKEN= # Speech: optional Azure AI Speech. One key/region unlocks both directions — # azure_stt (Fast Transcription cloud STT) and azure_tts (neural cloud TTS). # The local faster-whisper transcriber / piper_tts remain the default offline paths. -AZURE_SPEECH_KEY= # Azure AI Speech resource key ('Keys and Endpoint' page) -AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus +AZURE_SPEECH_KEY= +AZURE_SPEECH_REGION= # AZURE_SPEECH_ENDPOINT= # Optional: full custom STT endpoint URL (overrides region) # AZURE_TTS_ENDPOINT= # Optional: full custom TTS host (e.g. https://.tts.speech.microsoft.com) diff --git a/.gitignore b/.gitignore index 52051e5d..85473523 100644 --- a/.gitignore +++ b/.gitignore @@ -97,3 +97,7 @@ venv/ # Backlot local cache (thumbnails) .backlot/ + +# Workspace-local third-party runtimes (for example the portable Blender LTS +# used by blender_world). These are checksum-verified but never committed. +.runtime/ diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md index dc5abee1..0b0afd9f 100644 --- a/AGENT_GUIDE.md +++ b/AGENT_GUIDE.md @@ -474,6 +474,10 @@ Key capability families to look for in the output: - **analysis** — Transcription, scene detection, frame sampling. - **avatar** — Talking head and lip sync generation. - **character_animation** — Local character specs, SVG rigs, pose libraries, action timelines, previews, and QA. +- **3d_world_generation** — Local semantic terrain, procedural biome scattering, explicit landmarks, diagnostic passes, and deterministic HyperFrames/Three.js camera fly-throughs. Route through `threejs_world` and read `skills/creative/3d-world-generation.md` plus `.agents/skills/threejs-world-generation/SKILL.md`. +- **3d_asset_acquisition** — Rights-safe GLTF/GLB catalogs for production-tier Three.js worlds. Route through `threejs_asset_catalog`; never substitute blockout primitives for a requested detailed or reference-grade environment. +- **3d_asset_generation** — Unique textured/PBR meshes from text or concept images. Route text-described hero assets through `atlas_3d`, image-conditioned assets and regional object extraction through `fal_3d`, and read `.agents/skills/3d-asset-generation/SKILL.md`. Announce provider/model/unit cost before every first paid call and sample before batching. +- **3d_world_rendering** — Production assembly, terrain, lighting, materials, camera, and image-sequence rendering in Blender. Route through `blender_world`; use Three.js for interactive/blockout review, not as a substitute for Blender when reference-grade scene density is requested. - **enhancement** — Upscale, background removal, face enhance, color grading. Each tool in the registry declares `best_for`, `install_instructions`, `runtime` (LOCAL, API, LOCAL_GPU, HYBRID), and `status`. Read these fields — do not assume tool strengths from memory. @@ -528,9 +532,10 @@ Music is a critical part of any video. **Surface the music situation to the user Check music availability in this order and present the options: -1. **User music library (`music_library/`):** Check if this folder exists and contains tracks. If so, list available tracks with durations and let the user pick one. -2. **Music generation APIs:** Check which music tools are available via the registry (`registry.get_by_capability("music_generation")`). Report their status honestly — include quota status if known. -3. **Royalty-free sources:** Note if the user can provide their own track (e.g., from YouTube Audio Library, Jamendo, or other free sources). Offer the `music_library/` drop path. +1. **User music library:** Check `registry.get_by_capability("music_library")` and inspect `music_library/`. If tracks exist, list durations and let the user pick one. +2. **Royalty-free search:** Check `registry.get_by_capability("music_search")` for configured search/download tools. Report licensing constraints and whether a key is required. +3. **Music generation APIs:** Check `registry.get_by_capability("music_generation")`. Report status, quota, cost, and quality tradeoffs honestly. +4. **Bring your own:** Note that the user can provide a track (for example from YouTube Audio Library or Jamendo) through the `music_library/` drop path. **Always present the user with explicit choices:** - Use a track from their library (which one?) @@ -672,7 +677,7 @@ The `.agents/skills/` directory is large. When you're not coming in through a to | Category | Skills | |---|---| -| **Composition runtime** | `remotion`, `remotion-best-practices`, `synthetic-screen-recording` (fake terminal/UI demos via Remotion TerminalScene) | +| **Composition runtime** | `remotion`, `remotion-best-practices`, `synthetic-screen-recording` (fake terminal/UI demos via Remotion TerminalScene), `threejs-world-generation` (semantic terrain and free-viewpoint HyperFrames worlds) | | **Animation knowledge (generic)** | `gsap-core`, `gsap-timeline`, `gsap-plugins` (SplitText / MorphSVG / DrawSVG / MotionPath / Flip / CustomEase), `gsap-utils`, `gsap-react`, `gsap-performance`, `gsap-scrolltrigger`, `gsap-frameworks`, `framer-motion` (Disney 12 principles), `lottie-bodymovin` (Lottie export) | | **Character animation** | `character-rigging`, `svg-character-animation`, `pose-library-design`, `canvas-procedural-animation`, `character-animation-qa` | | **Image generation** | `bfl-api`, `flux-best-practices` | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..b1fd5bf1 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,7 @@ +# Contributing to OpenMontage + +Thank you for improving OpenMontage. We do not require a Contributor License Agreement (CLA), Developer Certificate of Origin sign-off, or any separate contribution contract. + +By submitting a pull request, you confirm that you have the right to contribute its contents. Contributions must be compatible with the repository's [AGPLv3 license](LICENSE), and accepted contributions remain available under that license. + +Please keep pull requests focused and include tests when behavior changes. diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md index 136856bd..5f0db301 100644 --- a/PROJECT_CONTEXT.md +++ b/PROJECT_CONTEXT.md @@ -71,11 +71,17 @@ Each tool's `agent_skills[]` field bridges Layer 1 → Layer 3. See `skills/INDE | `tools/cost_tracker.py` | Budget governance | | `tools/video/video_stitch.py` | Multi-clip assembly (stitch, spatial, validate, preview) | | `tools/video/video_compose.py` | Runtime-aware composition orchestrator — routes to Remotion / HyperFrames / FFmpeg based on `edit_decisions.render_runtime` | -| `tools/video/hyperframes_compose.py` | HyperFrames runtime — workspace materialization, `hyperframes lint`/`validate`/`render`, FFmpeg floor check | +| `tools/video/hyperframes_compose.py` | HyperFrames runtime — templated workspace materialization plus authored-workspace unified `check`/`render`, FFmpeg floor check | +| `tools/graphics/threejs_world.py` | Local semantic 3D-world authoring with explicit blockout/production fidelity tiers, region-aware terrain, diagnostics, and HyperFrames atelier workspaces | +| `tools/graphics/threejs_asset_catalog.py` | CC0 GLTF/GLB catalog acquisition, inventory, and provenance for production-fidelity world builds | +| `tools/graphics/atlas_3d.py` | Atlas Cloud Tripo H3.1 text-to-3D for unique textured/PBR GLB assets | +| `tools/graphics/fal_3d.py` | fal.ai Hunyuan 3D and SAM 3D routes for image-conditioned and multi-object GLB generation | +| `tools/graphics/blender_world.py` | Blender 4.5 LTS production world assembly, terrain, lighting, camera, and Eevee Next rendering | | `tools/character/character_animation.py` | Local character-animation tools — character specs, SVG rig plans, pose libraries, action timelines, HyperFrames packages, and QA reports | | `lib/hyperframes_style_bridge.py` | Playbook → CSS custom properties + `DESIGN.md` bridge for HyperFrames workspaces | | `remotion-composer/src/components/` | 8 Remotion components (TextCard, StatCard, ProgressBar, CalloutBox, ComparisonCard + charts/) | | `.agents/skills/hyperframes*/` | Vendored HyperFrames Layer 3 skills (authoring contract, CLI, registry, website-to-video) | +| `.agents/skills/threejs-world-generation/` | Layer 3 coarse-to-fine semantic world construction and render-guided refinement workflow | | `skills/core/hyperframes.md` | Layer 2 — when OpenMontage should pick HyperFrames vs Remotion, artifact → workspace mapping | | `schemas/styles/playbook.schema.json` | Playbook schema v2 with design tokens (chart_palette, scale_system, weight_matrix, color_rules) | | `tests/qa/` | Quality validation test scripts for tool-by-tool output inspection | diff --git a/README.md b/README.md index d47cd649..6dd89ecd 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,20 @@

- OpenMontage + + + Monty the Clapper — the official mascot of OpenMontage +

+

Monty the Clapper — the official mascot of OpenMontage

+

OpenMontage

The first open-source, agentic video production system.

+

+ openmontage.video +

+

Paste A Video  ·  Quick Start  ·  @@ -48,8 +57,8 @@ - - + + @@ -223,6 +232,7 @@ This repo is built for agentic operation. If you're an OpenClaw-style agent, her # Image + video gateway: FAL_KEY=your-key # FLUX images + Google Veo, Kling, MiniMax video + Recraft images +ATLASCLOUD_API_KEY=your-key # Atlas Cloud — Seedream/Nano Banana/GPT Image + Kling/Seedance/Hailuo video # Kling official direct API: KLING_API_KEY=your-key # Official Kling video, image, TTS, avatar, lip sync @@ -375,8 +385,8 @@ Most "free AI video" stacks quietly mean "animate still images." OpenMontage can Edit your own talking-head footage. Generate a fully animated explainer from scratch. Cut a 2-hour podcast into a dozen social clips. Translate and dub your content into 10 languages. Build a cinematic brand teaser from stock footage and AI-generated scenes. **If a production team can make it, OpenMontage can orchestrate it.** - **12 production pipelines** — explainers, talking heads, screen demos, cinematic trailers, animations, podcasts, localization, documentary montages, and more -- **52 production tools** — spanning video generation, image creation, text-to-speech, music, audio mixing, subtitles, enhancement, and analysis -- **400+ agent skills** — production skills, pipeline directors, creative techniques, quality checklists, and deep technology knowledge packs that teach the agent how to use every tool like an expert +- **100+ production tools** — spanning video generation, image creation, text-to-speech, music, audio mixing, subtitles, enhancement, and analysis +- **700+ agent skill and production-knowledge files** — pipeline directors, creative techniques, quality checklists, and deep technology knowledge packs that teach the agent how to use every tool like an expert - **Reference-driven creation** — paste a video you like and the agent turns it into a grounded, differentiated production plan instead of forcing you to invent the perfect prompt from scratch - **Real-footage documentary creation without paid video models** — build actual edited videos from free/open motion footage and archival sources, not just Ken Burns over images - **Live web research built in** — before writing a single word of script, the agent runs 15-25+ web searches across YouTube, Reddit, news sites, and academic sources to ground your video in real, current data @@ -433,7 +443,7 @@ Final video output -- only if self-review passes ``` OpenMontage/ -├── tools/ # 48 Python tools (the agent's hands) +├── tools/ # 100+ Python tools (the agent's hands) │ ├── video/ # 13 video gen tools + compose, stitch, trim │ ├── audio/ # 4 TTS providers + Suno/ElevenLabs music, mixing, enhancement │ ├── graphics/ # 9 image/graphics generation tools + diagrams, code snippets, math diff --git a/README_zh-CN.md b/README_zh-CN.md index ca663047..b8e3b7f2 100644 --- a/README_zh-CN.md +++ b/README_zh-CN.md @@ -1,7 +1,12 @@

- OpenMontage + + + Monty the Clapper — OpenMontage 官方吉祥物 +

+

Monty the Clapper — OpenMontage 官方吉祥物

+

OpenMontage

首个开源的,代理化(agentic)的视频制作系统

diff --git a/assets/monty-dark.svg b/assets/monty-dark.svg new file mode 100644 index 00000000..dc13e5d4 --- /dev/null +++ b/assets/monty-dark.svg @@ -0,0 +1,47 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/monty-light.svg b/assets/monty-light.svg new file mode 100644 index 00000000..498da5fa --- /dev/null +++ b/assets/monty-light.svg @@ -0,0 +1,47 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/social_preview.png b/assets/social_preview.png index e6038409..86b4aeb7 100644 Binary files a/assets/social_preview.png and b/assets/social_preview.png differ diff --git a/assets/social_preview.source.html b/assets/social_preview.source.html new file mode 100644 index 00000000..02c90945 --- /dev/null +++ b/assets/social_preview.source.html @@ -0,0 +1,94 @@ + + + + + + + +
+
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+

OpenMontage

+
+

The first open-source, agentic
video production system.

+
12 pipelines·100+ tools·700+ agent skills
+
+
+ +
openmontage.video
+ + diff --git a/backlot/server.py b/backlot/server.py index 7e40e8fc..1016a256 100644 --- a/backlot/server.py +++ b/backlot/server.py @@ -10,6 +10,7 @@ from __future__ import annotations import asyncio import json import time +from contextlib import asynccontextmanager, suppress from pathlib import Path from typing import Optional @@ -147,18 +148,22 @@ async def _watch_projects() -> None: hub.publish(pid) +@asynccontextmanager +async def _lifespan(app: FastAPI): + """Own and cleanly stop the project watcher with FastAPI's lifespan API.""" + + task = asyncio.create_task(_watch_projects()) + app.state.watch_task = task + try: + yield + finally: + task.cancel() + with suppress(asyncio.CancelledError): + await task + + def create_app() -> FastAPI: - app = FastAPI(title="Backlot", docs_url=None, redoc_url=None) - - @app.on_event("startup") - async def _startup() -> None: - app.state.watch_task = asyncio.create_task(_watch_projects()) - - @app.on_event("shutdown") - async def _shutdown() -> None: - task = getattr(app.state, "watch_task", None) - if task: - task.cancel() + app = FastAPI(title="Backlot", docs_url=None, redoc_url=None, lifespan=_lifespan) # ---- API ---------------------------------------------------------- diff --git a/backlot/state.py b/backlot/state.py index 9ce50f7a..678cb829 100644 --- a/backlot/state.py +++ b/backlot/state.py @@ -9,6 +9,7 @@ from __future__ import annotations import json import re +from functools import lru_cache from pathlib import Path from typing import Any, Optional @@ -59,6 +60,7 @@ def _rel(project_dir: Path, path: Path) -> str: # Pipeline / stages # --------------------------------------------------------------------------- +@lru_cache(maxsize=32) def _load_pipeline_meta(pipeline_type: Optional[str]) -> dict[str, Any]: """Stage order + gate flags from the manifest; graceful fallback.""" if pipeline_type and pipeline_type != "unknown": @@ -69,6 +71,10 @@ def _load_pipeline_meta(pipeline_type: Optional[str]) -> dict[str, Any]: { "name": s["name"], "gated": bool(s.get("human_approval_default", False)), + "produces": [ + str(name) for name in (s.get("produces") or []) + if isinstance(name, str) and name + ], } for s in manifest.get("stages", []) if isinstance(s, dict) and s.get("name") @@ -83,7 +89,7 @@ def _load_pipeline_meta(pipeline_type: Optional[str]) -> dict[str, Any]: pass return { "pipeline_type": pipeline_type or "unknown", - "stages": [{"name": s, "gated": False} for s in FALLBACK_STAGES], + "stages": [{"name": s, "gated": False, "produces": []} for s in FALLBACK_STAGES], "known": False, } @@ -152,6 +158,7 @@ def _build_stage_rail( entry: dict[str, Any] = { "name": name, "gated": stage_def["gated"], + "produces": list(stage_def.get("produces") or []), "status": status or "pending", "timestamp": cp.get("timestamp") if cp else None, "review": cp.get("review") if cp else None, @@ -188,6 +195,11 @@ def _build_stage_rail( entry = { "name": name, "gated": False, + "produces": [ + str(artifact_name) + for artifact_name in (cp.get("artifacts") or {}) + if isinstance(artifact_name, str) and artifact_name + ], "status": cp.get("status") or "unknown", "timestamp": cp.get("timestamp"), "review": cp.get("review"), diff --git a/backlot/ui/board.css b/backlot/ui/board.css index 1b017f8e..736f8439 100644 --- a/backlot/ui/board.css +++ b/backlot/ui/board.css @@ -496,6 +496,7 @@ aside .panel:nth-of-type(1) { animation-delay: .32s } aside .panel:nth-of-type(2 /* ---------- layout ---------- */ .board { display: grid; grid-template-columns: 1fr 320px; gap: 22px; align-items: start; } +.board.solo { grid-template-columns: minmax(0, 1fr); } .main-col { min-width: 0; display: flex; flex-direction: column; gap: 22px; } .panel { background: var(--surface); border: 1px solid var(--border-soft); border-radius: 12px; } @@ -565,12 +566,92 @@ aside .panel:nth-of-type(1) { animation-delay: .32s } aside .panel:nth-of-type(2 font-family: var(--mono); font-size: calc(10px * var(--fs-scale)); color: var(--cream-ink-2); letter-spacing: .06em; z-index: 2; } -.script-approved { +.script-status { position: absolute; top: 20px; right: 26px; font-family: var(--mono); font-size: calc(10px * var(--fs-scale)); font-weight: 600; letter-spacing: .14em; - color: #2c7a4b; border: 1.5px solid #2c7a4b; border-radius: 3px; padding: 3px 8px; transform: rotate(6deg); opacity: .8; } +.script-approved { color: #2c7a4b; border: 1.5px solid #2c7a4b; border-radius: 3px; } +.script-pending { color: #9a5b00; border: 1.5px solid #9a5b00; border-radius: 3px; } +.script-draft { color: #59657a; border: 1.5px solid #59657a; border-radius: 3px; } + +/* ---------- approval review surface ---------- */ +.approval-review { + width: 100%; max-width: 1000px; margin: 0 auto; + border: 1px solid color-mix(in srgb, var(--amber) 52%, var(--border-soft)); + border-radius: 14px; + background: + radial-gradient(circle at 100% 0, color-mix(in srgb, var(--amber) 13%, transparent), transparent 42%), + var(--surface); + box-shadow: 0 28px 76px -54px rgba(0,0,0,.9); + overflow: hidden; + animation: rise .55s cubic-bezier(.2,.7,.3,1) .18s backwards; +} +.approval-review-head { + display: flex; align-items: flex-start; justify-content: space-between; gap: 24px; + padding: 24px 26px 20px; border-bottom: 1px solid var(--border-soft); +} +.approval-review-head h2 { + margin: 4px 0 6px; color: var(--text); font-size: calc(21px * var(--fs-scale)); + line-height: 1.2; text-transform: capitalize; +} +.approval-review-head p { max-width: 650px; color: var(--text-2); font-size: calc(12.5px * var(--fs-scale)); line-height: 1.5; } +.approval-eyebrow, .approval-status, .approval-artifact-kicker, +.approval-selected, .approval-rationale b, .approval-review-note b, +.approval-review-foot, .approval-fact span { + font-family: var(--mono); letter-spacing: .11em; text-transform: uppercase; +} +.approval-eyebrow { color: var(--amber); font-size: calc(9.5px * var(--fs-scale)); } +.approval-status { + flex: none; color: var(--amber); border: 1px solid var(--amber); border-radius: 99px; + padding: 5px 9px; font-size: calc(9px * var(--fs-scale)); white-space: nowrap; +} +.approval-review-note { + margin: 18px 26px 0; padding: 10px 12px; border-left: 2px solid var(--amber); + background: color-mix(in srgb, var(--amber) 7%, var(--surface-2)); + color: var(--text-2); font-size: calc(11.5px * var(--fs-scale)); line-height: 1.5; +} +.approval-review-note b { color: var(--amber); font-size: calc(9px * var(--fs-scale)); } +.approval-artifacts { display: grid; gap: 14px; padding: 20px 26px 22px; } +.approval-artifact { + padding: 20px; border: 1px solid var(--border-soft); border-radius: 10px; + background: color-mix(in srgb, var(--surface-2) 72%, transparent); +} +.approval-artifact-kicker { color: var(--amber); font-size: calc(9px * var(--fs-scale)); } +.approval-artifact > h2 { margin: 5px 0 14px; font-size: calc(18px * var(--fs-scale)); line-height: 1.25; } +.approval-lead { color: var(--text-2); font-size: calc(13px * var(--fs-scale)); line-height: 1.55; margin: 0 0 14px; } +.approval-facts { display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 14px; } +.approval-fact { + display: grid; gap: 2px; min-width: 112px; padding: 8px 10px; + border: 1px solid var(--border-soft); border-radius: 7px; background: var(--surface); +} +.approval-fact span { color: var(--text-3); font-size: calc(8px * var(--fs-scale)); } +.approval-fact b { color: var(--text-2); font-size: calc(11px * var(--fs-scale)); font-weight: 600; overflow-wrap: anywhere; } +.approval-items { list-style: none; margin: 4px 0 0; padding: 0; display: grid; gap: 8px; } +.approval-items li { padding: 10px 12px; border-left: 2px solid var(--border); background: var(--surface); } +.approval-items li.selected { border-left-color: var(--amber); } +.approval-item-title { display: flex; align-items: baseline; justify-content: space-between; gap: 10px; color: var(--text); font-size: calc(12px * var(--fs-scale)); font-weight: 600; } +.approval-items p { color: var(--text-3); font-size: calc(11px * var(--fs-scale)); line-height: 1.45; margin-top: 3px; } +.approval-selected { color: var(--amber); font-size: calc(8px * var(--fs-scale)); font-weight: 500; } +.approval-rationale, .approval-guidance { color: var(--text-3); font-size: calc(10.5px * var(--fs-scale)); line-height: 1.5; margin-top: 12px; } +.approval-rationale b { color: var(--text-2); font-size: calc(8.5px * var(--fs-scale)); } +.approval-guidance { color: var(--amber); } +.approval-missing { + padding: 16px; border: 1px solid color-mix(in srgb, var(--red) 45%, var(--border)); + border-radius: 9px; background: var(--red-dim); color: var(--text-2); font-size: calc(12px * var(--fs-scale)); +} +.approval-missing b { color: var(--red); } +.approval-review-foot { + display: flex; align-items: center; justify-content: space-between; gap: 16px; + padding: 14px 26px; border-top: 1px solid var(--border-soft); + color: var(--text-3); font-size: calc(9px * var(--fs-scale)); +} +.approval-review-foot button { + border: 1px solid var(--border); border-radius: 99px; padding: 6px 11px; + background: var(--surface-2); color: var(--text-2); font: inherit; letter-spacing: inherit; + cursor: pointer; +} +.approval-review-foot button:hover, .approval-review-foot button:focus-visible { color: var(--amber); border-color: var(--amber); outline: none; } /* ---------- right rail: decisions & activity ---------- */ .decision { padding: 11px 0; border-bottom: 1px solid var(--border-soft); } @@ -866,7 +947,8 @@ a { color: inherit; } body:not(.first) .slate, body:not(.first) .rail .stage, body:not(.first) .script-card, body:not(.first) .notice, body:not(.first) aside .panel, body:not(.first) .scene-card, -body:not(.first) .lib-card, body:not(.first) .drawer { animation: none; } +body:not(.first) .lib-card, body:not(.first) .drawer, +body:not(.first) .approval-review { animation: none; } /* stages that ran but aren't declared by the pipeline manifest */ .stage.undeclared .node { border-style: dashed; opacity: .85; } @@ -919,6 +1001,7 @@ body:not(.first) .lib-card, body:not(.first) .drawer { animation: none; } } .section-title { flex-wrap: wrap; } .section-title .meta { margin-left: 0; } + .approval-review-head { flex-wrap: wrap; gap: 12px; } } @media (max-width: 520px) { @@ -940,7 +1023,7 @@ body:not(.first) .lib-card, body:not(.first) .drawer { animation: none; } padding: 24px 20px 28px; border-radius: 5px; } - .script-approved { top: 14px; right: 16px; font-size: calc(9px * var(--fs-scale)); padding: 2px 6px; } + .script-status { top: 14px; right: 16px; font-size: calc(9px * var(--fs-scale)); padding: 2px 6px; } .script-card .sp-title { font-size: calc(14px * var(--fs-scale)); padding-right: 58px; } .script-card .sp-meta { margin-bottom: 18px; } .script-card .sp-slug .tc { float: none; display: block; margin-top: 2px; } @@ -951,4 +1034,9 @@ body:not(.first) .lib-card, body:not(.first) .drawer { animation: none; } .drawer .drawer-head { flex-wrap: wrap; } .drawer pre { font-size: calc(10.5px * var(--fs-scale)); } .scene-card { max-width: calc(100vw - 42px); } + .approval-review-head, .approval-artifacts, .approval-review-foot { padding-left: 16px; padding-right: 16px; } + .approval-review-note { margin-left: 16px; margin-right: 16px; } + .approval-review-foot { align-items: flex-start; flex-direction: column; } + .approval-review-foot button { width: 100%; } + .approval-status { white-space: normal; } } diff --git a/backlot/ui/board.js b/backlot/ui/board.js index edf82dd4..b87ecea6 100644 --- a/backlot/ui/board.js +++ b/backlot/ui/board.js @@ -169,6 +169,36 @@ const STAGE_ARTIFACTS = { publish: ["publish_log"], }; +function artifactNamesForStage(st) { + const declared = Array.isArray(st.produces) ? st.produces : []; + const fallback = STAGE_ARTIFACTS[st.name] || []; + return [...new Set([...declared, ...fallback].filter(Boolean))]; +} + +function reviewMetrics(review) { + const nested = review && review.summary && typeof review.summary === "object" + ? review.summary : {}; + return { + critical: Number((review && review.critical) ?? nested.critical ?? 0), + suggestions: Number((review && review.suggestions) ?? nested.suggestions ?? 0), + nitpicks: Number((review && review.nitpicks) ?? nested.nitpicks ?? 0), + }; +} + +function reviewSummaryText(review) { + if (!review) return ""; + if (typeof review.summary === "string") return review.summary; + const nested = review.summary && typeof review.summary === "object" ? review.summary : {}; + const counts = reviewMetrics(review); + return [ + review.decision, + `${counts.critical} critical`, + `${counts.suggestions} suggestion${counts.suggestions === 1 ? "" : "s"}`, + nested.review_focus_met ? `review focus ${nested.review_focus_met}` : null, + nested.schema_validation, + ].filter(Boolean).join(" · "); +} + function renderDrawer(s) { if (!selectedStage) return null; const st = s.stages.find((x) => x.name === selectedStage); @@ -177,15 +207,17 @@ function renderDrawer(s) { const body = el("div", { class: "drawer-body" }); if (st.review) { + const metrics = reviewMetrics(st.review); + const summary = reviewSummaryText(st.review); body.append(el("div", { class: "findings", style: "margin-bottom:12px" }, - el("span", { class: `f ${st.review.critical ? "crit" : ""}` }, `${st.review.critical ?? 0} critical`), - el("span", { class: `f ${st.review.suggestions ? "sugg" : ""}` }, `${st.review.suggestions ?? 0} suggestions`), - el("span", { class: "f" }, `${st.review.nitpicks ?? 0} nitpicks`), - typeof st.review.summary === "string" ? el("span", { style: "font-size:calc(11.5px * var(--fs-scale));color:var(--text-2);margin-left:8px" }, st.review.summary) : null, + el("span", { class: `f ${metrics.critical ? "crit" : ""}` }, `${metrics.critical} critical`), + el("span", { class: `f ${metrics.suggestions ? "sugg" : ""}` }, `${metrics.suggestions} suggestions`), + el("span", { class: "f" }, `${metrics.nitpicks} nitpicks`), + summary ? el("span", { style: "font-size:calc(11.5px * var(--fs-scale));color:var(--text-2);margin-left:8px" }, summary) : null, )); } - const names = STAGE_ARTIFACTS[st.name] || []; + const names = artifactNamesForStage(st); let shown = false; for (const name of names) { const artifact = s.artifacts[name]; @@ -243,10 +275,17 @@ function renderScriptCard(s) { const script = s.artifacts.script; if (!script) return null; const scriptStage = s.stages.find((x) => x.name === "script"); - const approved = scriptStage && scriptStage.status === "completed"; + const status = scriptStage ? scriptStage.status : "unknown"; + const stamp = status === "completed" + ? el("span", { class: "script-status script-approved" }, "APPROVED") + : status === "awaiting_human" + ? el("span", { class: "script-status script-pending" }, "PENDING APPROVAL") + : status === "in_progress" + ? el("span", { class: "script-status script-draft" }, "DRAFTING") + : null; const card = el("div", { class: "script-card script-preview", title: "Click to expand full script", onclick: openScriptModal }, - approved ? el("span", { class: "script-approved" }, "APPROVED") : null, + stamp, el("div", { class: "sp-title" }, script.title || s.title), el("div", { class: "sp-meta" }, `script · ${fmtDuration(script.total_duration_seconds)} · ${(script.sections || []).length} sections`), @@ -256,6 +295,238 @@ function renderScriptCard(s) { return card; } +function humanize(value) { + return String(value || "artifact").replaceAll("_", " "); +} + +function shortText(value, limit = 180) { + const text = String(value || "").trim(); + return text.length > limit ? `${text.slice(0, limit - 1)}…` : text; +} + +function reviewFact(label, value) { + if (value == null || value === "") return null; + return el("div", { class: "approval-fact" }, + el("span", {}, label), + el("b", {}, value), + ); +} + +function reviewFacts(items) { + const facts = items.filter(Boolean); + return facts.length ? el("div", { class: "approval-facts" }, facts) : null; +} + +function titledItems(items, selectedId = null) { + const rows = (items || []).slice(0, 4).map((item, index) => { + if (item == null) return null; + if (typeof item !== "object") { + return el("li", {}, shortText(item)); + } + const id = item.id || item.concept_id || item.option_id; + const title = item.title || item.name || item.display_name || item.label || id || item.path || item.platform || item.description || `Item ${index + 1}`; + const detail = item.hook || item.why_this_works || item.summary || item.description || item.silhouette_notes; + return el("li", { class: id && id === selectedId ? "selected" : "" }, + el("div", { class: "approval-item-title" }, shortText(title, 100), + id && id === selectedId ? el("span", { class: "approval-selected" }, "SELECTED") : null), + detail && detail !== title ? el("p", {}, shortText(detail)) : null, + ); + }).filter(Boolean); + return rows.length ? el("ul", { class: "approval-items" }, rows) : null; +} + +function genericArtifactSummary(artifact) { + const facts = []; + const items = []; + for (const [key, value] of Object.entries(artifact || {})) { + if (["version", "decision_log_ref"].includes(key)) continue; + if (["string", "number", "boolean"].includes(typeof value)) { + facts.push(reviewFact(humanize(key), shortText(value, 90))); + } else if (Array.isArray(value)) { + facts.push(reviewFact(humanize(key), `${value.length} item${value.length === 1 ? "" : "s"}`)); + if (!items.length && value.length) items.push(titledItems(value)); + } + if (facts.length >= 6) break; + } + return [reviewFacts(facts), ...items].filter(Boolean); +} + +function artifactReviewContent(name, artifact) { + if (name === "brief") { + return [ + artifact.hook ? el("p", { class: "approval-lead" }, artifact.hook) : null, + reviewFacts([ + reviewFact("platform", artifact.target_platform), + reviewFact("duration", artifact.target_duration_seconds != null ? fmtDuration(artifact.target_duration_seconds) : null), + reviewFact("tone", artifact.tone), + reviewFact("style", artifact.style), + ]), + titledItems(artifact.key_points), + ].filter(Boolean); + } + if (name === "proposal_packet") { + const selected = (artifact.selected_concept || {}).concept_id; + const plan = artifact.production_plan || {}; + const cost = artifact.cost_estimate || {}; + return [ + reviewFacts([ + reviewFact("runtime", plan.render_runtime), + reviewFact("pipeline", plan.pipeline), + reviewFact("estimated cost", cost.total_estimated_usd != null ? fmtMoney(cost.total_estimated_usd) : null), + reviewFact("concepts", Array.isArray(artifact.concept_options) ? artifact.concept_options.length : null), + ]), + titledItems(artifact.concept_options, selected), + (artifact.selected_concept || {}).rationale + ? el("p", { class: "approval-rationale" }, + el("b", {}, "WHY THIS CONCEPT "), shortText(artifact.selected_concept.rationale)) + : null, + ].filter(Boolean); + } + if (name === "research_brief") { + return [ + artifact.topic ? el("p", { class: "approval-lead" }, artifact.topic) : null, + reviewFacts([ + reviewFact("sources", Array.isArray(artifact.sources) ? artifact.sources.length : null), + reviewFact("data points", Array.isArray(artifact.data_points) ? artifact.data_points.length : null), + reviewFact("angles", Array.isArray(artifact.angles_discovered) ? artifact.angles_discovered.length : null), + ]), + titledItems(artifact.angles_discovered), + ].filter(Boolean); + } + if (name === "script") { + const first = (artifact.sections || [])[0]; + return [ + reviewFacts([ + reviewFact("duration", fmtDuration(artifact.total_duration_seconds)), + reviewFact("sections", (artifact.sections || []).length), + ]), + first && first.text ? el("p", { class: "approval-lead" }, shortText(first.text, 220)) : null, + el("p", { class: "approval-guidance" }, "The complete script preview is shown directly below."), + ].filter(Boolean); + } + if (name === "scene_plan") { + const scenes = artifact.scenes || []; + const end = scenes.reduce((max, scene) => Math.max(max, Number(scene.end_seconds) || 0), 0); + return [ + reviewFacts([ + reviewFact("scenes", scenes.length), + reviewFact("duration", end ? fmtDuration(end) : null), + ]), + titledItems(scenes), + el("p", { class: "approval-guidance" }, "Review timing and shot coverage in the storyboard below."), + ].filter(Boolean); + } + if (name === "asset_manifest") { + const assets = artifact.assets || []; + const types = [...new Set(assets.map((asset) => asset.type).filter(Boolean))]; + return [ + reviewFacts([ + reviewFact("assets", assets.length), + reviewFact("types", types.join(", ")), + reviewFact("generation cost", artifact.total_cost_usd != null ? fmtMoney(artifact.total_cost_usd) : null), + ]), + titledItems(assets), + el("p", { class: "approval-guidance" }, "Inspect every generated take in the filmstrip below before approving compose."), + ].filter(Boolean); + } + if (name === "edit_decisions") { + return [ + reviewFacts([ + reviewFact("cuts", Array.isArray(artifact.cuts) ? artifact.cuts.length : null), + reviewFact("runtime", artifact.render_runtime || (artifact.metadata || {}).render_runtime), + ]), + titledItems(artifact.cuts), + ].filter(Boolean); + } + if (name === "render_report") { + return [ + reviewFacts([ + reviewFact("outputs", Array.isArray(artifact.outputs) ? artifact.outputs.length : null), + reviewFact("duration", artifact.duration_seconds != null ? fmtDuration(artifact.duration_seconds) : null), + ]), + titledItems(artifact.outputs), + ].filter(Boolean); + } + if (name === "publish_log") { + return [ + reviewFacts([reviewFact("destinations", Array.isArray(artifact.entries) ? artifact.entries.length : null)]), + titledItems((artifact.entries || []).map((entry) => ({ + title: entry.platform || entry.destination || "Publish destination", + description: [entry.status, entry.url].filter(Boolean).join(" · "), + }))), + ].filter(Boolean); + } + return genericArtifactSummary(artifact); +} + +function artifactReviewTitle(name, artifact, s) { + if (name === "proposal_packet") { + const selected = (artifact.selected_concept || {}).concept_id; + const concept = (artifact.concept_options || []).find((item) => item.id === selected); + return (concept && concept.title) || "Production proposal"; + } + if (name === "research_brief") return artifact.topic || "Research brief"; + if (name === "scene_plan") return "Scene plan"; + if (name === "asset_manifest") return "Generated assets"; + if (name === "edit_decisions") return "Edit decisions"; + if (name === "render_report") return "Render report"; + if (name === "publish_log") return "Publish plan"; + return artifact.title || artifact.name || s.title; +} + +function renderApprovalReview(s) { + const awaiting = s.stages.find((item) => item.status === "awaiting_human"); + if (!awaiting) return null; + + const names = artifactNamesForStage(awaiting); + const entries = names + .filter((name) => name !== "decision_log") + .map((name) => [name, s.artifacts[name]]) + .filter(([, artifact]) => artifact && typeof artifact === "object"); + const stageIndex = s.stages.findIndex((item) => item.name === awaiting.name); + const nextStage = stageIndex >= 0 ? s.stages[stageIndex + 1] : null; + const review = awaiting.review || {}; + const reviewSummary = reviewSummaryText(review); + + const artifacts = entries.map(([name, artifact]) => el("article", { + class: "approval-artifact", + "data-artifact": name, + }, + el("div", { class: "approval-artifact-kicker" }, humanize(name)), + el("h2", {}, artifactReviewTitle(name, artifact, s)), + ...artifactReviewContent(name, artifact), + )); + + if (!artifacts.length) { + artifacts.push(el("div", { class: "approval-missing", role: "alert" }, + el("b", {}, "Nothing reviewable was found. "), + names.length + ? `The ${awaiting.name} checkpoint declares ${names.map(humanize).join(", ")}, but Backlot could not load it.` + : `The ${awaiting.name} checkpoint does not declare an artifact.`, + )); + } + + return el("section", { class: "approval-review", "data-stage": awaiting.name }, + el("div", { class: "approval-review-head" }, + el("div", {}, + el("div", { class: "approval-eyebrow" }, "REVIEW GATE"), + el("h2", {}, `${humanize(awaiting.name)} is ready for your review`), + el("p", {}, "Review the artifact here, then reply in chat to approve it or request changes."), + ), + el("span", { class: "approval-status" }, "PENDING APPROVAL"), + ), + reviewSummary ? el("div", { class: "approval-review-note" }, + el("b", {}, "SELF-REVIEW "), shortText(reviewSummary, 260)) : null, + el("div", { class: "approval-artifacts" }, artifacts), + el("div", { class: "approval-review-foot" }, + el("span", {}, nextStage + ? `Approval unlocks ${humanize(nextStage.name)}.` + : "This is the final approval gate."), + el("button", { type: "button", onclick: () => toggleDrawer(awaiting.name) }, "OPEN FULL ARTIFACT"), + ), + ); +} + function openScriptModal() { const script = state && state.artifacts.script; if (!script) return; @@ -799,6 +1070,8 @@ function render() { if (noState) app.append(noState); const main = el("div", { class: "main-col" }); + const approvalReview = renderApprovalReview(s); + if (approvalReview) main.append(approvalReview); const script = renderScriptCard(s); if (script) main.append(script); const aside = el("aside", {}); @@ -813,11 +1086,12 @@ function render() { const found = renderFoundMedia(s); const renders = renderRenders(s); - if (script || decisions || activity) { + if (approvalReview || script || decisions || activity) { for (const section of [storyboard, found, renders]) { if (section) main.append(section); } - app.append(el("div", { class: "board" }, main, aside)); + const hasAside = Boolean(decisions || activity); + app.append(el("div", { class: `board${hasAside ? "" : " solo"}` }, main, hasAside ? aside : null)); } else { for (const section of [storyboard, found, renders]) { if (section) app.append(section); @@ -830,6 +1104,9 @@ function render() { function normalize(s) { s.pipeline = s.pipeline || { pipeline_type: "unknown", stages: [], known: false }; s.stages = Array.isArray(s.stages) ? s.stages : []; + for (const stage of s.stages) { + stage.produces = Array.isArray(stage.produces) ? stage.produces : []; + } s.artifacts = s.artifacts || {}; s.media = s.media || {}; s.media.renders = Array.isArray(s.media.renders) ? s.media.renders : []; diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md index 1f55c0ea..cd3d3a17 100644 --- a/docs/PROVIDERS.md +++ b/docs/PROVIDERS.md @@ -104,6 +104,53 @@ OpenMontage now uses those published rates in the Grok tool estimators. --- +### Volcengine Jimeng — 即梦 AI Video Generation + +> **Direct ByteDance API via V4 signing.** Calls the Volcengine visual API (visual.volcengineapi.com) with HMAC-SHA256 request signing using IAM AK/SK credentials. Supports text-to-video and image-to-video via Jimeng 3.0 Pro. + +**Tools unlocked:** `jimeng_video` +**Env vars:** `VOLC_ACCESSKEY` (Access Key ID) + `VOLC_SECRETKEY` (Secret Access Key) + +#### Setup + +1. Go to [console.volcengine.com/iam/keymanage](https://console.volcengine.com/iam/keymanage) +2. Create a Volcengine account if you don't have one +3. Create an Access Key pair (AK + SK) +4. Ensure your account has access to Jimeng AI (即梦) video generation service +5. Add to `.env`: `VOLC_ACCESSKEY=...` and `VOLC_SECRETKEY=...` + +#### What it's best for + +- Direct ByteDance/Volcengine API quota usage +- Jimeng 3.0 Pro text-to-video and image-to-video +- Chinese-language prompt understanding +- Configurable frame count (121=5s, 241=10s) and aspect ratio + +#### API notes + +Authentication uses Volcengine IAM V4 signing (HMAC-SHA256), not a Bearer token. The signing process builds a canonical request, derives a signing key from SK → date → region → service, and signs the request. + +API flow: `POST ?Action=CVSync2AsyncSubmitTask` → poll `POST ?Action=CVSync2AsyncGetResult` → download `video_url`. + +The implementation uses the compatible generic `CVSync2Async*` route (API version `2022-08-31`) rather than the model-specific `2024-06-06` actions presented in the public API explorer. This is intentional — the generic route supports the same Jimeng 3.0 Pro model via `req_key` while remaining stable across model updates. + +The `req_key` for video is `jimeng_ti2v_v30_pro`. Success code is `10000`. Task statuses: `in_queue`, `generating`, `done`, `not_found`, `expired`. + +**Authoritative API reference:** [Jimeng TI2V V30 Pro SubmitTask](https://api.volcengine.com/api-docs/view?action=JimengTI2VV30PROSubmitTask&serviceCode=cv&version=2024-06-06) + +**Schema constraints** (enforced by `input_schema` to prevent paid-call failures): +- `prompt`: max 800 characters +- `frames`: must be exactly `121` (5s) or `241` (10s) at 24fps +- `seed`: `-1` for random, or any non-negative integer + +#### Pricing + +| Model | Price | +|------|-------| +| Jimeng 3.0 Pro (video) | ~$0.05/sec (check Volcengine console for actual rate) | + +--- + ### Alibaba DashScope — Qwen Image + TTS + ASR > **Best for Chinese-language production.** One key unlocks Qwen-Image generation, Qwen-TTS Mandarin narration, and Qwen-ASR with word-level timestamps — the only DashScope path that provides word-level granularity for subtitle alignment. diff --git a/lib/checkpoint.py b/lib/checkpoint.py index e35e6164..f0cc4b99 100644 --- a/lib/checkpoint.py +++ b/lib/checkpoint.py @@ -95,9 +95,29 @@ class CheckpointValidationError(ValueError): """Raised when a checkpoint or its canonical artifacts are invalid.""" +def _validate_style_playbook(style_playbook: str | None) -> None: + """Fail closed when a checkpoint names a visual identity that cannot load.""" + + if style_playbook is None: + return + try: + from styles.playbook_loader import list_playbooks, load_playbook + + load_playbook(style_playbook) + except Exception as exc: + try: + available = list_playbooks() + except Exception: + available = [] + raise CheckpointValidationError( + f"Unknown or invalid style_playbook {style_playbook!r}. " + f"Available playbooks: {available}. Underlying error: {exc}" + ) from exc + + @lru_cache(maxsize=1) def _load_checkpoint_schema() -> dict[str, Any]: - with open(CHECKPOINT_SCHEMA_PATH) as f: + with open(CHECKPOINT_SCHEMA_PATH, encoding="utf-8") as f: return json.load(f) @@ -192,6 +212,7 @@ def init_project( Idempotent: re-running preserves the original created_at and merges fields. Returns the project directory. """ + _validate_style_playbook(style_playbook) base = pipeline_dir or PROJECTS_DIR project_dir = base / project_id for sub in ( @@ -208,7 +229,7 @@ def init_project( marker: dict[str, Any] = {} if marker_path.exists(): try: - with open(marker_path) as f: + with open(marker_path, encoding="utf-8") as f: marker = json.load(f) except (json.JSONDecodeError, OSError): marker = {} @@ -221,7 +242,7 @@ def init_project( if style_playbook is not None: marker["style_playbook"] = style_playbook - with open(marker_path, "w") as f: + with open(marker_path, "w", encoding="utf-8") as f: json.dump(marker, f, indent=2) return project_dir @@ -260,6 +281,71 @@ def _stage_requires_approval(pipeline_type: Optional[str], stage: str) -> Option return get_stage_human_approval_default(manifest, stage) +def _enforce_stage_prerequisites( + pipeline_dir: Path, + project_id: str, + pipeline_type: str | None, + stage: str, + status: str, +) -> None: + """Require completed, approved predecessors before advancing a stage. + + ``in_progress`` and failure heartbeats remain writable so an operator can + inspect or resume a broken run. Only lifecycle advancement + (``awaiting_human``/``completed``) is gated. + """ + + if status not in {"awaiting_human", "completed"}: + return + if not pipeline_type or pipeline_type == "unknown": + return + + stages = get_pipeline_stages(pipeline_type) + if stage not in stages: + return + + incomplete: list[str] = [] + unapproved: list[str] = [] + for predecessor in stages[: stages.index(stage)]: + path = _checkpoint_path(pipeline_dir, project_id, predecessor) + if not path.exists(): + incomplete.append(predecessor) + continue + try: + with open(path, encoding="utf-8") as handle: + checkpoint = json.load(handle) + validate_checkpoint(checkpoint) + except (OSError, json.JSONDecodeError, CheckpointValidationError): + incomplete.append(predecessor) + continue + if ( + checkpoint.get("project_id") != project_id + or checkpoint.get("pipeline_type") != pipeline_type + or checkpoint.get("stage") != predecessor + ): + incomplete.append(predecessor) + continue + if checkpoint.get("status") != "completed": + incomplete.append(predecessor) + continue + if _stage_requires_approval(pipeline_type, predecessor) and not checkpoint.get( + "human_approved" + ): + unapproved.append(predecessor) + + if incomplete or unapproved: + details = [] + if incomplete: + details.append(f"incomplete or missing: {incomplete}") + if unapproved: + details.append(f"completed without required approval: {unapproved}") + raise CheckpointValidationError( + f"PREREQUISITE VIOLATION: stage {stage!r} cannot advance; " + + "; ".join(details) + + f". Pipeline order: {stages}." + ) + + def _archive_superseded_checkpoint(path: Path, stage: str) -> None: """Copy an existing checkpoint into history/ before it is overwritten. @@ -275,7 +361,7 @@ def _archive_superseded_checkpoint(path: Path, stage: str) -> None: if not path.exists(): return try: - with open(path) as f: + with open(path, encoding="utf-8") as f: existing = json.load(f) except (json.JSONDecodeError, OSError): existing = {} @@ -314,7 +400,7 @@ def _merge_decision_log( """ path = _decision_log_path(pipeline_dir, project_id) if path.exists(): - with open(path) as f: + with open(path, encoding="utf-8") as f: existing = json.load(f) else: existing = { @@ -329,7 +415,7 @@ def _merge_decision_log( existing["decisions"].append(decision) path.parent.mkdir(parents=True, exist_ok=True) - with open(path, "w") as f: + with open(path, "w", encoding="utf-8") as f: json.dump(existing, f, indent=2) @@ -351,19 +437,22 @@ def write_checkpoint( metadata: Optional[dict] = None, ) -> Path: """Write a checkpoint file for a pipeline stage.""" - # Backfill a missing pipeline_type from the project marker so that - # omitting the kwarg doesn't quietly bypass gate enforcement. - if not pipeline_type: - marker = None - marker_path = pipeline_dir / project_id / PROJECT_MARKER_FILENAME - if marker_path.exists(): - try: - with open(marker_path) as f: - marker = json.load(f) - except (json.JSONDecodeError, OSError): - marker = None - if isinstance(marker, dict) and marker.get("pipeline_type"): + # Backfill identity fields from the project marker so omitted kwargs + # cannot bypass either gate enforcement or style validation. + marker = None + marker_path = pipeline_dir / project_id / PROJECT_MARKER_FILENAME + if marker_path.exists() and (not pipeline_type or not style_playbook): + try: + with open(marker_path, encoding="utf-8") as f: + marker = json.load(f) + except (json.JSONDecodeError, OSError): + marker = None + if isinstance(marker, dict): + if not pipeline_type and marker.get("pipeline_type"): pipeline_type = marker["pipeline_type"] + if not style_playbook and marker.get("style_playbook"): + style_playbook = marker["style_playbook"] + _validate_style_playbook(style_playbook) valid_stages = ( set(get_pipeline_stages(pipeline_type)) if pipeline_type @@ -404,6 +493,14 @@ def write_checkpoint( f"re-write with status='completed', human_approved=True." ) + _enforce_stage_prerequisites( + pipeline_dir, + project_id, + pipeline_type, + stage, + status, + ) + checkpoint = { "version": "1.0", "project_id": project_id, @@ -456,7 +553,7 @@ def write_checkpoint( # current checkpoint; then archive the superseded file and swap in the # new one atomically. tmp_path = path.with_suffix(".json.tmp") - with open(tmp_path, "w") as f: + with open(tmp_path, "w", encoding="utf-8") as f: json.dump(checkpoint, f, indent=2) # Preserve run history: a superseded completed/awaiting_human checkpoint # is copied to history/ (stage versioning, gate audit trail, replay). @@ -474,7 +571,7 @@ def read_checkpoint( path = _checkpoint_path(pipeline_dir, project_id, stage) if not path.exists(): return None - with open(path) as f: + with open(path, encoding="utf-8") as f: checkpoint = json.load(f) validate_checkpoint(checkpoint) return checkpoint @@ -496,7 +593,7 @@ def get_latest_checkpoint( if not checkpoints: return None - with open(checkpoints[0]) as f: + with open(checkpoints[0], encoding="utf-8") as f: checkpoint = json.load(f) validate_checkpoint(checkpoint) return checkpoint diff --git a/lib/clip_embedder.py b/lib/clip_embedder.py index b9ee82eb..82f0e1cd 100644 --- a/lib/clip_embedder.py +++ b/lib/clip_embedder.py @@ -61,6 +61,20 @@ def model_info() -> dict: } +def _as_feature_tensor(features): + """Normalize CLIP feature return values across transformers versions. + + Transformers 4 returned the projected tensor directly. Transformers 5 may + wrap that tensor in a model-output object whose ``pooler_output`` contains + the same shared-space embedding. Do not project it again: the vision + projection expects the pre-projection width, while ``pooler_output`` is + already the final CLIP width. + """ + + pooled = getattr(features, "pooler_output", None) + return features if pooled is None else pooled + + def embed_images(image_paths: Sequence[Union[str, Path]]) -> np.ndarray: """Embed a list of image files into a (N, 512) float32 matrix. @@ -82,7 +96,7 @@ def embed_images(image_paths: Sequence[Union[str, Path]]) -> np.ndarray: inputs = _PROCESSOR(images=images, return_tensors="pt").to(_DEVICE) with torch.no_grad(): - features = _MODEL.get_image_features(**inputs) + features = _as_feature_tensor(_MODEL.get_image_features(**inputs)) features = features / features.norm(dim=-1, keepdim=True).clamp_min(1e-8) arr = features.cpu().numpy().astype(np.float32, copy=False) # Close PIL handles to avoid leaking file handles on Windows @@ -116,7 +130,7 @@ def embed_texts(texts: Sequence[str]) -> np.ndarray: max_length=77, ).to(_DEVICE) with torch.no_grad(): - features = _MODEL.get_text_features(**inputs) + features = _as_feature_tensor(_MODEL.get_text_features(**inputs)) features = features / features.norm(dim=-1, keepdim=True).clamp_min(1e-8) return features.cpu().numpy().astype(np.float32, copy=False) diff --git a/lib/corpus.py b/lib/corpus.py index 4aefd52b..068d617a 100644 --- a/lib/corpus.py +++ b/lib/corpus.py @@ -407,14 +407,21 @@ class Corpus: best_i = -1 best_score = -1e9 picked_mat = self.clip_embeddings[np.array(picked)] - for i in remaining: + # Normalize the position term to [0, 1] so it lives on the + # same scale as cosine similarity. An absolute index grows + # with the pool, drowning the similarity term: at the 0.5 + # default a candidate one slot later needed a similarity gap + # > 1.0 to be preferred — impossible for non-negative + # cosines — so diversify() degenerated to input order. + denom = max(1, len(remaining) - 1) + for pos, i in enumerate(remaining): sim_picked = float(np.max(self.clip_embeddings[i] @ picked_mat.T)) # We want LOW similarity, so we negate. score = -sim_picked # Diversity weights how hard we penalize similarity. At # diversity=1 we always pick the most different; at # diversity=0 we just take them in input order. - score = diversity * score + (1.0 - diversity) * (-remaining.index(i)) + score = diversity * score + (1.0 - diversity) * (-pos / denom) if score > best_score: best_score = score best_i = i diff --git a/lib/pipeline_loader.py b/lib/pipeline_loader.py index 6ced59c7..e2f2003d 100644 --- a/lib/pipeline_loader.py +++ b/lib/pipeline_loader.py @@ -26,7 +26,7 @@ from functools import lru_cache @lru_cache(maxsize=1) def _load_manifest_schema() -> dict: - with open(SCHEMA_PATH) as f: + with open(SCHEMA_PATH, encoding="utf-8") as f: return json.load(f) @@ -61,7 +61,7 @@ def load_pipeline(name: str, defs_dir: Optional[Path] = None) -> dict[str, Any]: if not path.exists(): raise FileNotFoundError(f"Pipeline manifest not found: {path}") - with open(path) as f: + with open(path, encoding="utf-8") as f: manifest = yaml.safe_load(f) schema = _load_manifest_schema() diff --git a/pipeline_defs/animation.yaml b/pipeline_defs/animation.yaml index 1002eabf..20f964a9 100644 --- a/pipeline_defs/animation.yaml +++ b/pipeline_defs/animation.yaml @@ -2,7 +2,7 @@ name: animation version: "2.0" description: > Animation-first pipeline for motion graphics, diagram-led explainers, kinetic typography, - math visuals, and stylized illustrative sequences. Features a research-first pre-production + math visuals, explicit Three.js worlds, and stylized illustrative sequences. Features a research-first pre-production phase: the agent researches the topic and animation techniques, proposes concepts with animation mode selection and cost estimates, and gets explicit user approval before any assets are generated. @@ -175,14 +175,18 @@ stages: - proposal_packet produces: - asset_manifest - required_tools: - - tts_selector optional_tools: + - tts_selector - image_selector - video_selector - math_animate - diagram_gen - code_snippet + - threejs_world + - threejs_asset_catalog + - atlas_3d + - fal_3d + - blender_world - music_gen tools_available: - tts_selector @@ -191,6 +195,11 @@ stages: - math_animate - diagram_gen - code_snippet + - threejs_world + - threejs_asset_catalog + - atlas_3d + - fal_3d + - blender_world - music_gen checkpoint_required: true human_approval_default: true @@ -208,6 +217,7 @@ stages: - Schema-valid asset_manifest artifact - All referenced asset files exist on disk - layer3_skills_read list is present and includes all tools used for generation + - Real 3D-world briefs include semantic or wireframe diagnostic review before compose - name: edit skill: pipelines/animation/edit-director diff --git a/pipeline_defs/cinematic.yaml b/pipeline_defs/cinematic.yaml index 0f806d6b..23f44201 100644 --- a/pipeline_defs/cinematic.yaml +++ b/pipeline_defs/cinematic.yaml @@ -1,7 +1,8 @@ name: cinematic version: "2.0" description: > - Mood-led cinematic pipeline for trailers, brand films, montages, and short-form dramatic edits. + Mood-led cinematic pipeline for trailers, brand films, montages, explicit 3D-world fly-throughs, + and short-form dramatic edits. Works best with supplied footage, stills, or source media, and can optionally use generated support visuals for gap filling or concept-led inserts. EP orchestration adds quality gates for emotional pacing, color consistency, and audio dynamics. @@ -171,6 +172,11 @@ stages: - audio_enhance - image_selector - video_selector + - threejs_world + - threejs_asset_catalog + - atlas_3d + - fal_3d + - blender_world - pixabay_music - freesound_music - music_gen @@ -179,6 +185,11 @@ stages: - audio_enhance - image_selector - video_selector + - threejs_world + - threejs_asset_catalog + - atlas_3d + - fal_3d + - blender_world - pixabay_music - freesound_music - music_gen @@ -189,6 +200,8 @@ stages: - Motion-required beats use actual video clips rather than still-image substitutes - Music and ambience plan matches the beat map - Optional generated inserts stay limited and justified + - Explicit 3D worlds preserve one coherent scene graph across global, regional, and walk views + - Hero 3D worlds use the production fidelity tier with licensed textured models and PBR terrain layers; primitive-only blockouts cannot pass success_criteria: - Schema-valid asset_manifest artifact - All referenced asset files exist on disk diff --git a/remotion-composer/package-lock.json b/remotion-composer/package-lock.json index 4b527227..da9eabb2 100644 --- a/remotion-composer/package-lock.json +++ b/remotion-composer/package-lock.json @@ -15,6 +15,7 @@ "@remotion/player": "^4.0.484", "@remotion/transitions": "^4.0.484", "d3-geo": "^3.1.1", + "fast-uri": "^3.1.5", "react": "^18.2.0", "react-dom": "^18.2.0", "remotion": "^4.0.484", @@ -1951,9 +1952,9 @@ "license": "MIT" }, "node_modules/fast-uri": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.2.tgz", - "integrity": "sha512-rVjf7ArG3LTk+FS6Yw81V1DLuZl1bRbNrev6Tmd/9RaroeeRRJhAt7jg/6YFxbvAQXUCavSoZhPPj6oOx+5KjQ==", + "version": "3.1.5", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.5.tgz", + "integrity": "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw==", "funding": [ { "type": "github", @@ -2253,9 +2254,9 @@ "license": "MIT" }, "node_modules/nanoid": { - "version": "3.3.15", - "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.15.tgz", - "integrity": "sha512-y7Wygv/7mEOvxTuEQDB8StXdMRBWf1kR/tlhAzBRUFkB2jfcLOAxO/SHmOO2zgz1pVgK29/kyupn059/bCHdjA==", + "version": "3.3.16", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.16.tgz", + "integrity": "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q==", "funding": [ { "type": "github", @@ -2345,9 +2346,9 @@ "license": "ISC" }, "node_modules/postcss": { - "version": "8.5.15", - "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.15.tgz", - "integrity": "sha512-FfR8sjd4em2T6fb3I2MwAJU7HWVMr9zba+enmQeeWFfCbm+UOC/0X4DS8XtpUTMwWMGbjKYP7xjfNekzyGmB3A==", + "version": "8.5.25", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.25.tgz", + "integrity": "sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw==", "funding": [ { "type": "opencollective", @@ -2364,7 +2365,7 @@ ], "license": "MIT", "dependencies": { - "nanoid": "^3.3.12", + "nanoid": "^3.3.16", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" }, diff --git a/remotion-composer/package.json b/remotion-composer/package.json index 2e5bfad9..e913bb2c 100644 --- a/remotion-composer/package.json +++ b/remotion-composer/package.json @@ -15,6 +15,7 @@ "@remotion/player": "^4.0.484", "@remotion/transitions": "^4.0.484", "d3-geo": "^3.1.1", + "fast-uri": "^3.1.5", "react": "^18.2.0", "react-dom": "^18.2.0", "remotion": "^4.0.484", diff --git a/remotion-composer/src/CinematicRenderer.tsx b/remotion-composer/src/CinematicRenderer.tsx index f3c27412..926c59c7 100644 --- a/remotion-composer/src/CinematicRenderer.tsx +++ b/remotion-composer/src/CinematicRenderer.tsx @@ -8,31 +8,13 @@ import { Sequence, interpolate, spring, - staticFile, useCurrentFrame, useVideoConfig, } from "remotion"; -function resolveAsset(src: string): string { - if (src.startsWith("http://") || src.startsWith("https://") || src.startsWith("data:")) { - return src; - } - const clean = src.replace(/^file:\/\/\/?/, ""); - if (clean.startsWith("/") || /^[A-Za-z]:[/\\]/.test(clean)) { - const posix = clean.replace(/\\/g, "/"); - // POSIX absolute paths already have a leading "/" — file:// + posix - // gives exactly three slashes. Windows drive paths (C:/...) need the - // extra slash added explicitly. Do not merge these branches — adding - // "file:///" unconditionally double-slashes POSIX paths (file:////...). - if (posix.startsWith("/")) { - return `file://${posix}`; - } - return `file:///${posix}`; - } - return staticFile(clean); -} import { CinematicRendererProps, CinematicTone, CinematicVideoScene } from "./cinematic/types"; import { CaptionOverlay } from "./components/CaptionOverlay"; +import { resolveAsset } from "./lib/resolveAsset"; const FPS = 30; @@ -57,10 +39,11 @@ const toneGradient = (tone: CinematicTone) => { const SceneVideo: React.FC<{ scene: CinematicVideoScene }> = ({ scene }) => { const frame = useCurrentFrame(); - const { durationInFrames, fps } = useVideoConfig(); + const { fps } = useVideoConfig(); + const durationInFrames = Math.max(1, Math.round(scene.durationSeconds * fps)); const fadeInFrames = scene.fadeInFrames ?? 10; const fadeOutFrames = scene.fadeOutFrames ?? 10; - const fadeOutStart = Math.max(fadeInFrames, durationInFrames - fadeOutFrames); + const fadeOutStart = Math.max(0, durationInFrames - fadeOutFrames); const fadeInOpacity = fadeInFrames === 0 ? 1 @@ -98,6 +81,7 @@ const SceneVideo: React.FC<{ scene: CinematicVideoScene }> = ({ scene }) => { src={resolveAsset(scene.src)} trimBefore={trimBefore} trimAfter={trimAfter} + playbackRate={scene.playbackRate} style={{ width: "100%", height: "100%", diff --git a/remotion-composer/src/CollageBurst.tsx b/remotion-composer/src/CollageBurst.tsx index b0400095..77cf0c74 100644 --- a/remotion-composer/src/CollageBurst.tsx +++ b/remotion-composer/src/CollageBurst.tsx @@ -6,12 +6,12 @@ import { interpolate, random, spring, - staticFile, useCurrentFrame, useVideoConfig, } from "remotion"; import React from "react"; import { loadFont as loadPlayfair } from "@remotion/google-fonts/PlayfairDisplay"; +import { resolveAsset } from "./lib/resolveAsset"; const { fontFamily: playfairFamily } = loadPlayfair("normal", { weights: ["400", "700"], @@ -22,23 +22,6 @@ const { fontFamily: playfairItalic } = loadPlayfair("italic", { subsets: ["latin"], }); -function resolveAsset(src: string): string { - if (src.startsWith("http://") || src.startsWith("https://") || src.startsWith("data:")) return src; - const clean = src.replace(/^file:\/\/\/?/, ""); - if (clean.startsWith("/") || /^[A-Za-z]:[\\/]/.test(clean)) { - const posix = clean.replace(/\\/g, "/"); - // POSIX absolute paths already have a leading "/" — file:// + posix - // gives exactly three slashes. Windows drive paths (C:/...) need the - // extra slash added explicitly. Do not merge these branches — adding - // "file:///" unconditionally double-slashes POSIX paths (file:////...). - if (posix.startsWith("/")) { - return `file://${posix}`; - } - return `file:///${posix}`; - } - return staticFile(clean); -} - export type CollageTransition = | "pop" | "slide-zoom" @@ -63,13 +46,13 @@ export interface CollageClip { seed?: number; } -export interface CollageBurstProps { +export type CollageBurstProps = { backgroundSrc: string; backgroundInSeconds?: number; curtainStartSeconds: number; curtainEndSeconds: number; clips: CollageClip[]; -} +}; // ---------------------------------------------------------------------------- // Opening text — elegant serif card that lives in the pre-reveal black, then diff --git a/remotion-composer/src/Explainer.tsx b/remotion-composer/src/Explainer.tsx index cf7ae03f..91b0da67 100644 --- a/remotion-composer/src/Explainer.tsx +++ b/remotion-composer/src/Explainer.tsx @@ -6,34 +6,10 @@ import { Sequence, interpolate, spring, - staticFile, useCurrentFrame, useVideoConfig, } from "remotion"; import { loadFont } from "@remotion/google-fonts/SpaceGrotesk"; - -// Resolve asset path — handle URLs, absolute paths (Windows/Unix), and public/ relative paths -function resolveAsset(src: string): string { - if (src.startsWith("http://") || src.startsWith("https://") || src.startsWith("data:")) { - return src; - } - // Strip any file:// prefix - const clean = src.replace(/^file:\/\/\/?/, ""); - // Absolute paths (Unix: /foo, Windows: C:\foo or C:/foo) — convert to file:// URI - // staticFile() only accepts relative paths within public/, so absolute paths must bypass it - if (clean.startsWith("/") || /^[A-Za-z]:[\\/]/.test(clean)) { - const posix = clean.replace(/\\/g, "/"); - // POSIX absolute paths already have a leading "/" — file:// + posix - // gives exactly three slashes. Windows drive paths (C:/...) need the - // extra slash added explicitly. Do not merge these branches — adding - // "file:///" unconditionally double-slashes POSIX paths (file:////...). - if (posix.startsWith("/")) { - return `file://${posix}`; - } - return `file:///${posix}`; - } - return staticFile(clean); -} import { TextCard } from "./components/TextCard"; import { StatCard } from "./components/StatCard"; import { CalloutBox } from "./components/CalloutBox"; @@ -54,6 +30,7 @@ import type { TerminalStep } from "./components/TerminalScene"; import { ScreenshotScene } from "./components/ScreenshotScene"; import type { ScreenshotStep } from "./components/ScreenshotScene"; import { ProviderChip } from "./components/ProviderChip"; +import { resolveAsset } from "./lib/resolveAsset"; import type { ParticleType } from "./components/ParticleOverlay"; import { resolveTheme, type ThemeConfig, DEFAULT_THEME } from "./Root"; @@ -254,6 +231,7 @@ interface Cut { animation?: string; transition_in?: string; transition_out?: string; + transition_duration?: number; transform?: { animation?: string; scale?: number; @@ -425,22 +403,49 @@ const ImageScene: React.FC<{ src: string; animation?: string }> = ({ // Enhanced Video Scene // --------------------------------------------------------------------------- -const VideoScene: React.FC<{ src: string; startFrom?: number }> = ({ +const VideoScene: React.FC<{ + src: string; + startFrom?: number; + transitionIn?: string; + transitionOut?: string; + transitionDuration?: number; + sceneDurationSeconds: number; + backgroundColor?: string; +}> = ({ src, startFrom = 0, + transitionIn, + transitionOut, + transitionDuration, + sceneDurationSeconds, + backgroundColor = "#0F172A", }) => { const frame = useCurrentFrame(); - const { fps, durationInFrames } = useVideoConfig(); + const { fps } = useVideoConfig(); + const durationInFrames = Math.max(1, Math.round(sceneDurationSeconds * fps)); - const fadeIn = spring({ frame, fps, config: { damping: 20 } }); - const fadeOutStart = durationInFrames - 8; - const fadeOut = interpolate(frame, [fadeOutStart, durationInFrames], [1, 0.3], { - extrapolateLeft: "clamp", - extrapolateRight: "clamp", - }); + const hardIn = ["cut", "none"].includes((transitionIn || "").toLowerCase()); + const hardOut = ["cut", "none"].includes((transitionOut || "").toLowerCase()); + const transitionFrames = Math.max( + 1, + Math.round((transitionDuration ?? 8 / fps) * fps), + ); + const fadeIn = hardIn + ? 1 + : interpolate(frame, [0, transitionFrames], [0, 1], { + extrapolateLeft: "clamp", + extrapolateRight: "clamp", + }); + const fadeOutStart = Math.max(0, durationInFrames - transitionFrames); + const fadeOut = hardOut + ? 1 + : interpolate(frame, [fadeOutStart, durationInFrames], [1, 0], { + extrapolateLeft: "clamp", + extrapolateRight: "clamp", + }); return ( - + = ({ cut, theme } if (cut.source && isVideo(cut.source)) { - return maybeWrapWithBg(); + return maybeWrapWithBg( + , + ); } // Final fallback — try as image if source exists, otherwise show text_card @@ -741,7 +756,7 @@ const OverlayRenderer: React.FC<{ overlay: Overlay }> = ({ overlay }) => { if (overlay.type === "section_title") { return ( = ({ overlay }) => { if (overlay.type === "stat_reveal") { return ( = ({ overlay }) => { ); } if (overlay.type === "hero_title") { - return ; + return ; } if (overlay.type === "provider_chip" && overlay.providers) { return ( diff --git a/remotion-composer/src/LyricOverlay.tsx b/remotion-composer/src/LyricOverlay.tsx index 4896cf1e..287110f7 100644 --- a/remotion-composer/src/LyricOverlay.tsx +++ b/remotion-composer/src/LyricOverlay.tsx @@ -3,46 +3,29 @@ import { Audio, OffthreadVideo, interpolate, - staticFile, useCurrentFrame, useVideoConfig, } from "remotion"; import React from "react"; import { loadFont as loadPlayfair } from "@remotion/google-fonts/PlayfairDisplay"; +import { resolveAsset } from "./lib/resolveAsset"; const { fontFamily: playfairItalic } = loadPlayfair("italic", { weights: ["400", "700"], subsets: ["latin"], }); -function resolveAsset(src: string): string { - if (src.startsWith("http://") || src.startsWith("https://") || src.startsWith("data:")) return src; - const clean = src.replace(/^file:\/\/\/?/, ""); - if (clean.startsWith("/") || /^[A-Za-z]:[\\/]/.test(clean)) { - const posix = clean.replace(/\\/g, "/"); - // POSIX absolute paths already have a leading "/" — file:// + posix - // gives exactly three slashes. Windows drive paths (C:/...) need the - // extra slash added explicitly. Do not merge these branches — adding - // "file:///" unconditionally double-slashes POSIX paths (file:////...). - if (posix.startsWith("/")) { - return `file://${posix}`; - } - return `file:///${posix}`; - } - return staticFile(clean); -} - export interface Lyric { text: string; inSeconds: number; outSeconds: number; } -export interface LyricOverlayProps { +export type LyricOverlayProps = { videoSrc: string; lyrics: Lyric[]; bottomY?: number; // 0..1, vertical center of subtitle band -} +}; const LyricLine: React.FC<{ lyric: Lyric; bottomY: number }> = ({ lyric, bottomY }) => { const frame = useCurrentFrame(); diff --git a/remotion-composer/src/TitledVideo.tsx b/remotion-composer/src/TitledVideo.tsx index 0129271c..0f710cae 100644 --- a/remotion-composer/src/TitledVideo.tsx +++ b/remotion-composer/src/TitledVideo.tsx @@ -5,12 +5,12 @@ import { Sequence, interpolate, spring, - staticFile, useCurrentFrame, useVideoConfig, } from "remotion"; import { getVideoMetadata } from "@remotion/media-utils"; import { loadFont } from "@remotion/google-fonts/PlayfairDisplay"; +import { resolveAsset } from "./lib/resolveAsset"; // Editorial serif for the tagline — Playfair Display at its boldest weight. // Loaded once at module scope so every render reuses the same font face. @@ -19,7 +19,7 @@ const { fontFamily } = loadFont("normal", { subsets: ["latin"], }); -export interface TitledVideoProps { +export type TitledVideoProps = { videoSrc: string; tagline: string; // When the tagline starts animating in, in seconds from the start of the video. @@ -32,33 +32,10 @@ export interface TitledVideoProps { fontSize?: number; // Accent color used for the underline and the glow halo. accentColor?: string; -} +}; // Resolve asset path — handle URLs, absolute paths, and public/ relative paths. // Mirrors the helper in Explainer.tsx so absolute Windows/Unix paths work. -function resolveAsset(src: string): string { - if ( - src.startsWith("http://") || - src.startsWith("https://") || - src.startsWith("data:") - ) { - return src; - } - const clean = src.replace(/^file:\/\/\/?/, ""); - if (clean.startsWith("/") || /^[A-Za-z]:[\\/]/.test(clean)) { - const posix = clean.replace(/\\/g, "/"); - // POSIX absolute paths already have a leading "/" — file:// + posix - // gives exactly three slashes. Windows drive paths (C:/...) need the - // extra slash added explicitly. Do not merge these branches — adding - // "file:///" unconditionally double-slashes POSIX paths (file:////...). - if (posix.startsWith("/")) { - return `file://${posix}`; - } - return `file:///${posix}`; - } - return staticFile(clean); -} - // --------------------------------------------------------------------------- // EditorialTagline — big bold serif, upper-third, drawn underline, warm glow. // Letter-by-letter spring entrance. Designed to feel like a printed headline, diff --git a/remotion-composer/src/cinematic/types.ts b/remotion-composer/src/cinematic/types.ts index 67d8689b..7669968d 100644 --- a/remotion-composer/src/cinematic/types.ts +++ b/remotion-composer/src/cinematic/types.ts @@ -12,6 +12,7 @@ export interface CinematicVideoScene extends CinematicBaseScene { tone?: CinematicTone; trimBeforeSeconds?: number; trimAfterSeconds?: number; + playbackRate?: number; filter?: string; fadeInFrames?: number; fadeOutFrames?: number; diff --git a/remotion-composer/src/components/AnimeScene.tsx b/remotion-composer/src/components/AnimeScene.tsx index acfcd694..c2ab1741 100644 --- a/remotion-composer/src/components/AnimeScene.tsx +++ b/remotion-composer/src/components/AnimeScene.tsx @@ -3,27 +3,11 @@ import { Img, interpolate, spring, - staticFile, useCurrentFrame, useVideoConfig, } from "remotion"; import { ParticleOverlay, type ParticleType } from "./ParticleOverlay"; - -/** - * Resolve asset path — use staticFile() for local paths, passthrough URLs. - * Duplicated from Explainer.tsx to keep the component self-contained. - */ -function resolveAsset(src: string): string { - if ( - src.startsWith("http://") || - src.startsWith("https://") || - src.startsWith("data:") - ) { - return src; - } - const clean = src.replace(/^file:\/\/\/?/, ""); - return staticFile(clean); -} +import { resolveAsset } from "../lib/resolveAsset"; // --------------------------------------------------------------------------- // Types diff --git a/remotion-composer/src/components/CaptionOverlay.tsx b/remotion-composer/src/components/CaptionOverlay.tsx index e31123c6..1f9cf207 100644 --- a/remotion-composer/src/components/CaptionOverlay.tsx +++ b/remotion-composer/src/components/CaptionOverlay.tsx @@ -14,7 +14,7 @@ export interface WordCaption { endMs: number; } -interface CaptionOverlayProps { +type CaptionOverlayProps = { words: WordCaption[]; // How many words to show at once in a "page" wordsPerPage?: number; @@ -23,7 +23,7 @@ interface CaptionOverlayProps { highlightColor?: string; backgroundColor?: string; fontFamily?: string; -} +}; interface CaptionPage { words: WordCaption[]; diff --git a/remotion-composer/src/components/EndTag.tsx b/remotion-composer/src/components/EndTag.tsx index 6ffbfefb..ab785c51 100644 --- a/remotion-composer/src/components/EndTag.tsx +++ b/remotion-composer/src/components/EndTag.tsx @@ -6,7 +6,7 @@ import { useVideoConfig, } from "remotion"; -export interface EndTagProps { +export type EndTagProps = { text: string; palette?: "cool_offwhite_on_black" | "warm_ivory_on_black"; // Optional extra fade hold controls (all in seconds) @@ -19,7 +19,7 @@ export interface EndTagProps { // AbsoluteFill drops its background fill — caller is responsible for // rendering with an alpha-capable codec (VP9/WebM or ProRes 4444). overlay?: boolean; -} +}; const PALETTES = { cool_offwhite_on_black: { diff --git a/remotion-composer/src/components/HeroTitle.tsx b/remotion-composer/src/components/HeroTitle.tsx index 381dd173..d4b6321d 100644 --- a/remotion-composer/src/components/HeroTitle.tsx +++ b/remotion-composer/src/components/HeroTitle.tsx @@ -6,10 +6,10 @@ import { useVideoConfig, } from "remotion"; -interface HeroTitleProps { +type HeroTitleProps = { title: string; subtitle?: string; -} +}; export const HeroTitle: React.FC = ({ title, subtitle }) => { const frame = useCurrentFrame(); diff --git a/remotion-composer/src/components/ProductReveal.tsx b/remotion-composer/src/components/ProductReveal.tsx index c8967ab9..4ad81dba 100644 --- a/remotion-composer/src/components/ProductReveal.tsx +++ b/remotion-composer/src/components/ProductReveal.tsx @@ -9,14 +9,14 @@ import { Easing, } from "remotion"; -export interface ProductRevealProps { +export type ProductRevealProps = { productImage: string; productName: string; price: string; tagline: string; closer: string; accentColor?: string; -} +}; export const ProductReveal: React.FC = ({ productImage, diff --git a/remotion-composer/src/components/ProviderChip.tsx b/remotion-composer/src/components/ProviderChip.tsx index fdc69372..0f11230a 100644 --- a/remotion-composer/src/components/ProviderChip.tsx +++ b/remotion-composer/src/components/ProviderChip.tsx @@ -53,7 +53,7 @@ export const ProviderChip: React.FC = ({ const translateY = interpolate(springIn, [0, 1], [12, 0]); return ( - +
+ src.startsWith("http://") || + src.startsWith("https://") || + src.startsWith("data:"); + +const isWindowsAbsolutePath = (src: string): boolean => + /^[A-Za-z]:[\\/]/.test(src); + +/** Resolve public assets and absolute filesystem paths consistently. */ +export function resolveAsset(src: string): string { + if (isRemoteAsset(src)) { + return src; + } + + const withoutScheme = src.replace(/^file:\/\//i, ""); + const clean = /^\/[A-Za-z]:[\\/]/.test(withoutScheme) + ? withoutScheme.slice(1) + : withoutScheme; + + if (clean.startsWith("/") || isWindowsAbsolutePath(clean)) { + const posix = clean.replace(/\\/g, "/"); + return posix.startsWith("/") ? `file://${posix}` : `file:///${posix}`; + } + + return staticFile(clean); +} diff --git a/schemas/artifacts/__init__.py b/schemas/artifacts/__init__.py index acf15edf..147b2093 100644 --- a/schemas/artifacts/__init__.py +++ b/schemas/artifacts/__init__.py @@ -39,7 +39,7 @@ def load_schema(name: str) -> dict: path = SCHEMA_DIR / f"{name}.schema.json" if not path.exists(): raise FileNotFoundError(f"Schema not found: {path}") - with open(path) as f: + with open(path, encoding="utf-8") as f: return json.load(f) diff --git a/schemas/artifacts/asset_manifest.schema.json b/schemas/artifacts/asset_manifest.schema.json index 291b0b05..57324e36 100644 --- a/schemas/artifacts/asset_manifest.schema.json +++ b/schemas/artifacts/asset_manifest.schema.json @@ -16,7 +16,7 @@ "id": { "type": "string" }, "type": { "type": "string", - "enum": ["image", "video", "audio", "narration", "music", "sfx", "diagram", "animation", "code_snippet", "subtitle", "font", "lut"] + "enum": ["image", "video", "audio", "narration", "music", "sfx", "diagram", "animation", "3d_asset", "3d_world", "code_snippet", "subtitle", "font", "lut"] }, "path": { "type": "string", "description": "Relative path within the pipeline project directory" }, "source_tool": { "type": "string" }, diff --git a/schemas/artifacts/edit_decisions.schema.json b/schemas/artifacts/edit_decisions.schema.json index 63eeca4f..b4dc0089 100644 --- a/schemas/artifacts/edit_decisions.schema.json +++ b/schemas/artifacts/edit_decisions.schema.json @@ -57,6 +57,10 @@ "minimum": 0, "description": "Duration of transition in seconds" }, + "backgroundColor": { + "type": "string", + "description": "Backing color used behind this cut by composition runtimes" + }, "reason": { "type": "string" } }, "additionalProperties": false diff --git a/schemas/pipelines/pipeline_manifest.schema.json b/schemas/pipelines/pipeline_manifest.schema.json index ebf73b40..a7737e43 100644 --- a/schemas/pipelines/pipeline_manifest.schema.json +++ b/schemas/pipelines/pipeline_manifest.schema.json @@ -11,7 +11,7 @@ "description": { "type": "string" }, "category": { "type": "string", - "enum": ["talking_head", "generated", "hybrid", "screen_recording", "animation", "cinematic", "custom"] + "enum": ["talking_head", "generated", "hybrid", "screen_recording", "animation", "cinematic", "documentary", "custom"] }, "stability": { "type": "string", diff --git a/schemas/tools/atlas_3d.schema.json b/schemas/tools/atlas_3d.schema.json new file mode 100644 index 00000000..dc3405ca --- /dev/null +++ b/schemas/tools/atlas_3d.schema.json @@ -0,0 +1,24 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "openmontage/tools/atlas_3d", + "title": "Atlas Cloud Text-to-3D Input", + "type": "object", + "required": ["prompt", "output_path"], + "properties": { + "prompt": {"type": "string", "minLength": 3, "maxLength": 1024}, + "negative_prompt": {"type": "string"}, + "output_path": {"type": "string"}, + "texture": {"type": "boolean"}, + "pbr": {"type": "boolean"}, + "texture_quality": {"type": "string", "enum": ["standard", "detailed"]}, + "geometry_quality": {"type": "string", "enum": ["standard", "detailed"]}, + "face_limit": {"type": "integer", "minimum": 1000, "maximum": 2000000}, + "model_seed": {"type": "integer"}, + "image_seed": {"type": "integer"}, + "texture_seed": {"type": "integer"}, + "auto_size": {"type": "boolean"}, + "quad": {"type": "boolean"}, + "poll_timeout_seconds": {"type": "integer", "minimum": 30, "maximum": 1800} + }, + "additionalProperties": false +} diff --git a/schemas/tools/blender_world.schema.json b/schemas/tools/blender_world.schema.json new file mode 100644 index 00000000..45015e63 --- /dev/null +++ b/schemas/tools/blender_world.schema.json @@ -0,0 +1,23 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "openmontage/tools/blender_world", + "title": "Blender World Input", + "type": "object", + "required": ["operation"], + "properties": { + "operation": {"type": "string", "enum": ["doctor", "build", "render_still", "render_animation"]}, + "world_spec": {"type": "object"}, + "output_path": {"type": "string"}, + "blend_path": {"type": "string"}, + "width": {"type": "integer", "minimum": 320, "maximum": 7680}, + "height": {"type": "integer", "minimum": 240, "maximum": 4320}, + "samples": {"type": "integer", "minimum": 1, "maximum": 256}, + "fps": {"type": "integer", "minimum": 1, "maximum": 120}, + "duration_seconds": {"type": "number", "minimum": 1, "maximum": 600}, + "start_frame": {"type": "integer", "minimum": 1}, + "end_frame": {"type": "integer", "minimum": 1}, + "frame": {"type": "integer", "minimum": 1}, + "resume": {"type": "boolean", "default": false} + }, + "additionalProperties": false +} diff --git a/schemas/tools/fal_3d.schema.json b/schemas/tools/fal_3d.schema.json new file mode 100644 index 00000000..e0ea6716 --- /dev/null +++ b/schemas/tools/fal_3d.schema.json @@ -0,0 +1,20 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "openmontage/tools/fal_3d", + "title": "fal.ai 3D Generation Input", + "type": "object", + "required": ["operation", "output_path"], + "properties": { + "operation": {"type": "string", "enum": ["text_to_3d", "image_to_3d", "reconstruct_objects"]}, + "prompt": {"type": "string"}, + "image_url": {"type": "string"}, + "image_path": {"type": "string"}, + "output_path": {"type": "string"}, + "enable_pbr": {"type": "boolean"}, + "seed": {"type": "integer"}, + "export_textured_glb": {"type": "boolean"}, + "detection_threshold": {"type": "number", "minimum": 0.1, "maximum": 1.0}, + "poll_timeout_seconds": {"type": "integer", "minimum": 30, "maximum": 1800} + }, + "additionalProperties": false +} diff --git a/schemas/tools/threejs_asset_catalog.schema.json b/schemas/tools/threejs_asset_catalog.schema.json new file mode 100644 index 00000000..c69eabfb --- /dev/null +++ b/schemas/tools/threejs_asset_catalog.schema.json @@ -0,0 +1,16 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "openmontage/tools/threejs_asset_catalog", + "title": "Three.js Asset Catalog Input", + "type": "object", + "required": ["operation"], + "properties": { + "operation": {"type": "string", "enum": ["list", "install", "inspect"]}, + "catalog_id": { + "type": "string", + "enum": ["kenney-nature-kit", "kenney-fantasy-town-kit", "kenney-survival-kit"] + }, + "output_path": {"type": "string"} + }, + "additionalProperties": false +} diff --git a/schemas/tools/threejs_world.schema.json b/schemas/tools/threejs_world.schema.json new file mode 100644 index 00000000..8bd2f36c --- /dev/null +++ b/schemas/tools/threejs_world.schema.json @@ -0,0 +1,37 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "openmontage/tools/threejs_world", + "title": "Three.js World Tool Input", + "type": "object", + "required": ["operation", "world_spec"], + "properties": { + "operation": {"type": "string", "enum": ["build", "validate"]}, + "output_path": {"type": "string"}, + "duration_seconds": {"type": "number", "minimum": 1, "maximum": 600, "default": 60}, + "width": {"type": "integer", "minimum": 320, "maximum": 7680, "default": 1920}, + "height": {"type": "integer", "minimum": 240, "maximum": 4320, "default": 1080}, + "render_mode": {"type": "string", "enum": ["cinematic", "semantic", "wireframe"], "default": "cinematic"}, + "quality_tier": {"type": "string", "enum": ["blockout", "production"], "default": "blockout"}, + "asset_catalog_paths": {"type": "array", "items": {"type": "string"}, "default": []}, + "world_spec": { + "type": "object", + "required": ["regions", "camera_path"], + "properties": { + "version": {"type": "string"}, + "title": {"type": "string"}, + "seed": {"type": "integer"}, + "explicit_constraints": {"type": "array", "items": {"type": "string"}}, + "inferred_details": {"type": "array", "items": {"type": "string"}}, + "world": {"type": "object"}, + "atmosphere": {"type": "object"}, + "terrain_materials": {"type": "array", "items": {"type": "object"}}, + "asset_palette": {"type": "array", "items": {"type": "object"}}, + "regions": {"type": "array", "minItems": 1, "maxItems": 12, "items": {"type": "object"}}, + "landmarks": {"type": "array", "items": {"type": "object"}}, + "camera_path": {"type": "array", "minItems": 2, "items": {"type": "object"}} + }, + "additionalProperties": true + } + }, + "additionalProperties": false +} diff --git a/scripts/backlot_screenshot_stage.py b/scripts/backlot_screenshot_stage.py index 0230a6f6..e968d5a9 100644 --- a/scripts/backlot_screenshot_stage.py +++ b/scripts/backlot_screenshot_stage.py @@ -184,6 +184,15 @@ def stage_project(pid: str, title: str, palette: str, scenes: list, *, brief["topic"] = title cp("research", "completed", {"research_brief": brief}) + proposal = sample_artifact("proposal_packet") + cp("proposal", "awaiting_human", {"proposal_packet": proposal}) + cp( + "proposal", + "completed", + {"proposal_packet": proposal}, + human_approved=True, + ) + script = script_artifact(title, scenes) plan = scene_plan_artifact(scenes, hero) (art_dir / "decision_log.json").write_text(json.dumps(decision_log(pid), indent=2)) diff --git a/skills/INDEX.md b/skills/INDEX.md index 1b4aa3bf..1cdcab20 100644 --- a/skills/INDEX.md +++ b/skills/INDEX.md @@ -59,8 +59,14 @@ Key capability families to look for in the output: | `enhancement` | — | Mixed providers | | `analysis` | — | Mixed providers | | `character_animation` | — | Local character specs, SVG rigs, pose libraries, action timelines, previews, and QA | +| `3d_world_generation` | — | Local semantic terrain, procedural scattering, explicit landmarks, diagnostics, and HyperFrames/Three.js fly-through workspaces | +| `3d_asset_acquisition` | — | Rights-safe local GLTF/GLB catalogs and provenance | +| `3d_asset_generation` | — | Atlas/fal textured mesh generation and reconstruction for unique scene assets | +| `3d_world_rendering` | — | Blender assembly and production rendering of detailed worlds | | `graphics` | — | Local rendering tools | -| `music_generation` | — | Single-provider | +| `music_library` | — | Discovers user-provided local tracks | +| `music_search` | — | Discovers royalty-free search/download providers | +| `music_generation` | — | Discovers paid/local generation providers | | `subtitle` | — | Pure Python | | `avatar` | — | Local GPU models | | `video_post` | — | FFmpeg-based local tools | @@ -108,6 +114,7 @@ Key capability families to look for in the output: | ManimCE Usage | `creative/manim-usage.md` | Scene composition, animation timing, color usage | `manimce-best-practices` | | Image Gen Usage | `creative/image-gen-usage.md` | Prompt consistency, hero reference, batch strategy | `flux-best-practices`, `bfl-api` | | Image Provider Usage | `creative/image-provider-usage.md` | Provider selection (FLUX/Grok/OpenAI/Recraft/stock), cost-quality tradeoffs | `flux-best-practices`, `bfl-api`, `grok-media` | +| 3D World Generation | `creative/3d-world-generation.md` | Semantic world planning, asset sourcing/generation, Blender assembly, and fidelity review | `3d-asset-generation`, `threejs-world-generation` | | B-Roll Planning | `creative/broll-planning.md` | Stock vs. generated decision, query construction, footage evaluation | — | | Stock Sourcing Usage | `creative/stock-sourcing-usage.md` | Pexels/Pixabay usage, parameters, licensing, integration | — | | Scene Detect Usage | `creative/scene-detect-usage.md` | Threshold tuning, algorithm selection, content presets | â€" | @@ -130,6 +137,7 @@ Pipeline type skills provide production guidance for specific video formats, ind | Long-Form | `creative/long-form.md` | YouTube 10+ min â€" chapters, retention, end screens | | Screen Recording | `creative/screen-recording.md` | Code walkthroughs, tutorials, software demos | | Animation Pipeline | `creative/animation-pipeline.md` | Motion graphics, easing, transitions, composition | +| 3D World Generation | `creative/3d-world-generation.md` | Continuous Three.js terrain worlds with semantic regions, explicit blockout/production tiers, licensed GLTF/PBR assets, diagnostics, and deterministic camera paths | | Character Animation Pipeline | `pipelines/character-animation/` | Rigged local cartoon characters, pose libraries, action timelines, SVG/Canvas/Remotion/HyperFrames rendering | | Cinematic | `creative/cinematic.md` | Letterbox, film pacing, layered audio, color grading | @@ -311,7 +319,7 @@ Claude Code accesses them via symlinks in `.claude/skills/`. | **TTS & Audio** | `text-to-speech`, `speech-to-text` (whisper, default STT), `azure-speech-to-text` (optional cloud STT), `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` | | **Image Generation** | `flux-best-practices`, `bfl-api`, `grok-media` | `black-forest-labs/skills`, local OpenMontage skill | | **Math Animation** | `manimce-best-practices`, `manimgl-best-practices`, `manim-composer` | `adithya-s-k/manim_skill` | -| **3D Graphics** | `threejs-animation`, `threejs-fundamentals`, `threejs-geometry`, `threejs-interaction`, `threejs-lighting`, `threejs-loaders`, `threejs-materials`, `threejs-postprocessing`, `threejs-shaders`, `threejs-textures` | `cloudai-x/threejs-skills` | +| **3D Graphics** | `threejs-world-generation` (OpenMontage semantic-world workflow), `threejs-animation`, `threejs-fundamentals`, `threejs-geometry`, `threejs-interaction`, `threejs-lighting`, `threejs-loaders`, `threejs-materials`, `threejs-postprocessing`, `threejs-shaders`, `threejs-textures` | Local OpenMontage skill + `cloudai-x/threejs-skills` | | **Diagrams** | `beautiful-mermaid`, `d3-viz` | `intellectronica/agent-skills`, `davila7/claude-code-templates` | | **Animation** | `framer-motion`, `lottie-bodymovin` | `pproenca/dot-skills`, `dylantarre/animation-principles` | | **Design** | `tailwind-design-system`, `web-design-guidelines`, `vercel-react-best-practices`, `vercel-composition-patterns` | `wshobson/agents`, `vercel-labs/agent-skills` | diff --git a/skills/core/hyperframes.md b/skills/core/hyperframes.md index 120401ef..315fe346 100644 --- a/skills/core/hyperframes.md +++ b/skills/core/hyperframes.md @@ -125,9 +125,11 @@ projects// └── final.mp4 ``` -The workspace is generated at compose time by `hyperframes_compose` from -`edit_decisions` + `asset_manifest` + the active playbook. It's regenerable -and gitignored along with the rest of `projects/`. +Templated workspaces are generated at compose time by `hyperframes_compose` +from `edit_decisions` + `asset_manifest` + the active playbook. Atelier +workspaces are authored during assets and passed through unchanged via +`hyperframes_compose.render_existing`. Both live under `projects/` and are +gitignored with the rest of the production workspace. ### Why a dedicated workspace per project diff --git a/skills/creative/3d-world-generation.md b/skills/creative/3d-world-generation.md new file mode 100644 index 00000000..513a815e --- /dev/null +++ b/skills/creative/3d-world-generation.md @@ -0,0 +1,48 @@ +# 3D World Generation + +Use this Layer 2 skill when an animation or cinematic project needs a real, continuous Three.js environment rather than a stack of stills or generated video clips. + +## Capability route + +1. Query the registry for `3d_world_generation`, `3d_asset_acquisition`, `3d_asset_generation`, and `3d_world_rendering`. +2. Read the selected tools' `agent_skills`; production work requires both `.agents/skills/threejs-world-generation` and `.agents/skills/3d-asset-generation`. +3. Choose the delivery path at proposal. Use `render_runtime="hyperframes"` and `composition_mode="atelier"` for an editable browser-native Three.js deliverable. Use `render_runtime="ffmpeg"` for a Blender-rendered PNG sequence plus governed audio/video mux. The latter is still real 3D motion; FFmpeg only packages Blender's frames. +4. Use either the `animation` or `cinematic` pipeline. A 3D world is a reusable production capability, not a separate pipeline. +5. Lock a fidelity tier. `blockout` is only for semantic layout and camera iteration. Hero/reference-led output requires `production`, licensed catalogs for repeated assets, Atlas/fal for unique assets when useful, and `blender_world` for scene assembly/rendering. + +## Artifact mapping + +| Stage | Contract | +|---|---| +| proposal | Record world promise, explicit-vs-inferred policy, fidelity tier, and either the browser-native Three.js/HyperFrames route or production Blender/FFmpeg route. | +| script | Use beats or sparse titles; narration is optional. | +| scene_plan | Define global, regional, and walk-level camera beats in one continuous coordinate system. | +| assets | Install/inventory repeated assets with `threejs_asset_catalog`; sample unique hero assets with `atlas_3d` or `fal_3d`; assemble and render global/regional/walk stills with `blender_world`; register meshes as `type="3d_asset"` and the editable world spec/`.blend` as `type="3d_world"`. | +| assets review | Produce semantic/wireframe diagnostics and representative snapshots; log bounded refinement issues. | +| edit | Carry camera times without changing region IDs or seed. | +| compose | Browser-native: call `video_compose` on the authored HyperFrames workspace. Production Blender: render a numbered PNG sequence with resume enabled, then let `video_compose`/FFmpeg package frames and audio without pretending FFmpeg generated the motion. | + +## Required asset metadata + +Record: + +- `source_tool: "threejs_world"`; +- `provider: "threejs"`; +- `quality_tier`, `seed`, and the returned model identifier; +- catalog IDs, source URLs, licenses, archive hashes, selected model IDs, and PBR material maps; +- workspace path and `world.json` path; +- region, landmark, instance, and terrain-triangle counts; +- diagnostic warnings and refinement rounds; +- `layer3_skills_read: ["threejs-world-generation"]`. + +## Review focus + +- The terrain is continuous and establishes the large-scale silhouette. +- Region color, relief, scatter, and landmarks agree with one semantic layout. +- Environmental instances respect slope and contact constraints. +- Global, regional, and walk-level frames remain spatially coherent. +- The camera never tunnels through terrain or clips the far plane. +- The final render preserves editability: world, regions, landmarks, and camera keys remain structured files. +- Production frames use textured authored models at foreground, midground, and background depth; dominant primitives or untextured flat ground are asset-gate failures. + +Do not use this path for isolated product spins, a CSS parallax landscape, or an AI-generated fly-through with no explicit scene graph. diff --git a/skills/meta/animation-runtime-selector.md b/skills/meta/animation-runtime-selector.md index 8c656672..43592e6c 100644 --- a/skills/meta/animation-runtime-selector.md +++ b/skills/meta/animation-runtime-selector.md @@ -62,6 +62,8 @@ when both were available is a CRITICAL reviewer finding. | Kinetic typography, HTML/GSAP-native motion, product promo, launch reel | **hyperframes** | `skills/core/hyperframes.md` + `.agents/skills/hyperframes/SKILL.md` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type), `hyperframes-animation` (motion) | | Website → video, UI-driven composition | **hyperframes** | `.agents/skills/website-to-video/SKILL.md` (renamed from website-to-hyperframes in 0.7) | | Registry block needed (data-chart, grain-overlay, shader transitions, etc.) | **hyperframes** | `.agents/skills/hyperframes-registry/SKILL.md` | +| Editable browser-native 3D terrain/world and free-viewpoint fly-through | **hyperframes** | `skills/creative/3d-world-generation.md` + `.agents/skills/threejs-world-generation/SKILL.md` | +| Reference-grade 3D world film rendered in Blender | **ffmpeg packaging of Blender frames** | `skills/creative/3d-world-generation.md` + `.agents/skills/3d-asset-generation/SKILL.md` | | Beat-synced music video (audio drives scene timing) | **hyperframes** | `.agents/skills/music-to-video/SKILL.md` — uses `hyperframes beats` to detect drops, lays out frames on the beat grid | | Porting an existing Remotion composition to HyperFrames | **hyperframes** | `.agents/skills/remotion-to-hyperframes/SKILL.md` — migration guidance, ONLY for explicit port requests | | BGM / SFX / image / icon resolution (any pipeline, any runtime) | n/a | `.agents/skills/media-use/SKILL.md` — `resolve` verb against project cache + global cache + HeyGen catalog | @@ -94,6 +96,8 @@ decision matrix and the list of features that stay Remotion-only in Phase 1. | Data chart (bar/line/pie/KPI) | Remotion built-in chart components | `remotion-composer/SCENE_TYPES.md` | | HyperFrames composition — animation knowledge (rules, blueprints, transitions, runtime adapters) | HyperFrames + GSAP default | `.agents/skills/hyperframes-animation` (consolidated motion skill) + `.agents/skills/gsap-core`, `.agents/skills/gsap-timeline` | | HyperFrames composition structure (data-* timing, tracks, sub-compositions) | HyperFrames | `.agents/skills/hyperframes-core` | +| Explicit Three.js world (terrain, regions, landmarks, camera path) | HyperFrames + `threejs_world` | `.agents/skills/threejs-world-generation` | +| Detailed Blender world film (generated/catalog meshes, PBR, camera path) | Blender + FFmpeg packaging | `.agents/skills/3d-asset-generation` | | HyperFrames creative direction (palette, type, narration, beat planning) | HyperFrames | `.agents/skills/hyperframes-creative` | | HyperFrames audio/media (TTS, BGM, SFX, transcription, captions, bg-removal) | HyperFrames | `.agents/skills/hyperframes-media` | | HyperFrames composition CLI work (lint/validate/inspect/snapshot/benchmark/render/lambda) | HyperFrames CLI 0.7+ | `.agents/skills/hyperframes-cli` | diff --git a/skills/meta/bespoke-composition.md b/skills/meta/bespoke-composition.md index 84c8a9cf..0797ef9a 100644 --- a/skills/meta/bespoke-composition.md +++ b/skills/meta/bespoke-composition.md @@ -209,10 +209,11 @@ registry (`src/components`, `src/Explainer`, etc.), and warns if `art_direction` - Verify before render: `npx hyperframes lint . && npx hyperframes validate . && npx hyperframes snapshot . --at `. Snapshot is HF's native visual-spotcheck (contact-sheet of PNG frames at chosen timestamps) — use it the same way an atelier `final_review.visual_spotcheck` would. -- **Render**: `npx hyperframes render . --output renders/.mp4`. - > Known gap (F13): `hyperframes_compose.render` currently requires `edit_decisions.cuts[]` - > from the templated path. For hand-authored HF compositions it errors; call `npx` directly - > until the tool grows a bespoke branch. +- **Render**: call `video_compose` with `render_runtime: "hyperframes"`, + `composition_mode: "atelier"`, and the authored `workspace_path`. It routes to + `hyperframes_compose.render_existing`, which preserves `index.html` and runs + the unified check gate, strict render, and post-render review. Call `npx + hyperframes render` directly only while debugging the runtime outside a pipeline. ## Guardrails so this doesn't backfire diff --git a/skills/pipelines/animation/asset-director.md b/skills/pipelines/animation/asset-director.md index 59ab5c18..964a47d5 100644 --- a/skills/pipelines/animation/asset-director.md +++ b/skills/pipelines/animation/asset-director.md @@ -31,7 +31,7 @@ Quick routing for common animation-pipeline needs: |-------|----------|---------| | Schema | `schemas/artifacts/asset_manifest.schema.json` | Artifact validation | | Prior artifacts | `state.artifacts["scene_plan"]["scene_plan"]`, `state.artifacts["script"]["script"]`, `state.artifacts["proposal"]["proposal_packet"]` | Tool path and beat map | -| Tools | `tts_selector`, `image_selector`, `video_selector`, `math_animate`, `diagram_gen`, `code_snippet`, `music_gen` — selectors auto-discover all available providers from the registry | Asset production options | +| Tools | `tts_selector`, `image_selector`, `video_selector`, `math_animate`, `diagram_gen`, `code_snippet`, `threejs_world`, `music_gen` — selectors auto-discover all available providers from the registry | Asset production options | | Playbook | Active style playbook | Visual consistency | ## Process @@ -45,6 +45,13 @@ Prefer the lowest-variance useful path: - `math_animate` for real math motion, - provided artwork before new generation. +For a real 3D environment, read `skills/creative/3d-world-generation.md` and +the tool's `threejs-world-generation` Layer 3 skill, then use +`threejs_world` before any image or video generator. Build the cinematic +workspace plus a semantic or wireframe diagnostic pass. Register the editable +workspace as `type: "3d_world"`; snapshots belong in the assets review, while +the final MP4 belongs to compose. + ### 1b. Sample Preview (Prevents Wasted Spend) Before batch-generating assets, produce one sample of each expensive type and show the user: diff --git a/skills/pipelines/animation/compose-director.md b/skills/pipelines/animation/compose-director.md index ee953852..09fdbf06 100644 --- a/skills/pipelines/animation/compose-director.md +++ b/skills/pipelines/animation/compose-director.md @@ -11,8 +11,8 @@ Before any other work, read `edit_decisions.render_runtime`. It was locked at pr - **`render_runtime="hyperframes"`** — HTML/CSS/GSAP render. Do NOT follow the Remotion-specific sections below (public/ staging, Remotion composition JSON). Instead: 1. Read `skills/core/hyperframes.md` for the full routing model. 2. Read `.agents/skills/hyperframes/SKILL.md` and `.agents/skills/hyperframes-cli/SKILL.md` for authoring contract and CLI usage. - 3. Call `video_compose` with `edit_decisions.render_runtime="hyperframes"` — it delegates to `hyperframes_compose`, which owns workspace materialization under `projects//hyperframes/`, runs `hyperframes lint → validate → render`, and returns the MP4 path. - 4. `hyperframes lint` and `hyperframes validate` MUST both pass before render. Never skip validate; contrast can be deferred with `skip_contrast=true` during iteration but not for final delivery. + 3. Call `video_compose` with `edit_decisions.render_runtime="hyperframes"` — it delegates to `hyperframes_compose`, which owns workspace materialization under `projects//hyperframes/`, runs `hyperframes check → render`, and returns the MP4 path. + 4. `hyperframes check` MUST pass before render. It unifies lint, runtime, layout, motion, and WCAG contrast checks; contrast can be deferred with `skip_contrast=true` during iteration but not for final delivery. - **`render_runtime="ffmpeg"`** — simple concat/trim with no composition. Call `video_compose` directly; it will not auto-upgrade to Remotion. - **Runtime unavailable** — do NOT silently swap to a different engine. Surface the blocker to the user per AGENT_GUIDE.md > "Escalate Blockers Explicitly" and wait for approval (recorded as a `render_runtime_selection` decision in decision_log) before switching. diff --git a/skills/pipelines/animation/proposal-director.md b/skills/pipelines/animation/proposal-director.md index caffcc8c..e2df1455 100644 --- a/skills/pipelines/animation/proposal-director.md +++ b/skills/pipelines/animation/proposal-director.md @@ -31,6 +31,7 @@ Fit cheat-sheet for the recommendation (NOT an auto-decision): | Kinetic typography, product promo, launch reel, HTML/GSAP-native motion | HyperFrames | | Website-to-video or UI-driven composition | HyperFrames | | Registry blocks needed (data-chart, grain-overlay, shader transitions) | HyperFrames | +| Real 3D terrain, semantic regions, editable landmarks, free-viewpoint camera | HyperFrames + `threejs_world` | | Word-level/karaoke caption burn required | Remotion (HyperFrames caption parity deferred) | | Simple source-footage concat, no composition | ffmpeg | diff --git a/skills/pipelines/cinematic/asset-director.md b/skills/pipelines/cinematic/asset-director.md index 05a0105d..b1f0ddf5 100644 --- a/skills/pipelines/cinematic/asset-director.md +++ b/skills/pipelines/cinematic/asset-director.md @@ -32,6 +32,20 @@ Before authoring title cards, name plates, or SVG overlays, read **`skills/meta/ ## Process +### Explicit 3D-world path + +When the approved delivery promise is a continuous, free-viewpoint 3D world, +`threejs_world` satisfies semantic planning and browser-native motion: it creates +a real scene graph and time-driven camera, not a still-image fallback. Read +`skills/creative/3d-world-generation.md` and `.agents/skills/threejs-world-generation/SKILL.md`, +build into `projects//hyperframes/`, and review global, regional, walk, +semantic, and wireframe views before the assets gate. Keep +`render_runtime="hyperframes"` and `composition_mode="atelier"` locked for a +browser-native deliverable. For reference-grade video, lock Blender as the 3D +renderer and `render_runtime="ffmpeg"` solely as the image-sequence/audio packager. + +For hero/reference-driven work, `quality_tier="production"` is mandatory. Install licensed catalogs with `threejs_asset_catalog`, generate unique meshes with Atlas/fal when useful, assemble and render in Blender, and reject the asset gate if dominant primitives, flat untextured ground, low regional object density, or obvious repetition remain. `blockout` exists only for layout/camera approval. + ### 1. Prioritize Source Selects Start with: diff --git a/skills/pipelines/cinematic/compose-director.md b/skills/pipelines/cinematic/compose-director.md index 10049b24..9bf349a8 100644 --- a/skills/pipelines/cinematic/compose-director.md +++ b/skills/pipelines/cinematic/compose-director.md @@ -9,9 +9,13 @@ Render the cinematic piece with careful attention to grade, audio dynamics, and Read `edit_decisions.render_runtime`. Cinematic work routes to: - **`render_runtime="remotion"`** — default for video-led trailers using `CinematicRenderer`. Keeps video clips, transitions, and ambient overlays in one React-based pass. -- **`render_runtime="hyperframes"`** — for kinetic title cards, HTML/GSAP-driven trailers, or launch-reel-style compositions where the visual grammar is HTML/CSS. See `skills/core/hyperframes.md`. `hyperframes lint` and `hyperframes validate` must both pass before render. +- **`render_runtime="hyperframes"`** — for kinetic title cards, HTML/GSAP-driven trailers, launch-reel-style compositions, or explicit Three.js world fly-throughs. See `skills/core/hyperframes.md`. `hyperframes check` must pass before render. - **`render_runtime="ffmpeg"`** — simple source-footage concat with no composition. +For a Blender world film, FFmpeg is the approved packager for the numbered +Blender image sequence and audio. It must not synthesize camera motion or replace +missing Blender frames with pan/zoom effects. + `delivery_promise.motion_required=true` means the locked runtime is a commitment. Silent swap to another runtime (including FFmpeg Ken Burns) is a CRITICAL governance violation. If the locked runtime fails, escalate per AGENT_GUIDE.md > "Escalate Blockers Explicitly." **Pass `proposal_packet` to `video_compose.execute()`** so the tool's `runtime_swap_detected` check compares directly against `proposal_packet.production_plan.render_runtime`. Without it the swap check is skipped in-tool and only the reviewer skill catches the drift. diff --git a/skills/pipelines/cinematic/idea-director.md b/skills/pipelines/cinematic/idea-director.md index 0f332c81..d6cbc7ce 100644 --- a/skills/pipelines/cinematic/idea-director.md +++ b/skills/pipelines/cinematic/idea-director.md @@ -88,23 +88,26 @@ Cinematic videos live and die by their audio. **Surface the music situation befo Check availability in this order: -1. **User music library (`music_library/`):** Check if this folder exists and contains tracks. List available tracks with durations and moods. Let the user choose. -2. **Music generation APIs:** Check `registry.get_by_capability("music_generation")`. Report status, quota, and cost per track. -3. **Royalty-free sources:** Note that the user can provide a track from YouTube Audio Library, Jamendo, or other free sources by dropping it in `music_library/`. +1. **User music library:** Check `registry.get_by_capability("music_library")` and inspect `music_library/`. List available tracks with durations and moods. +2. **Royalty-free search:** Check `registry.get_by_capability("music_search")`. Report available providers, licensing constraints, and any key requirement. +3. **Music generation APIs:** Check `registry.get_by_capability("music_generation")`. Report status, quota, cost, and quality tradeoffs. +4. **Bring your own:** The user can drop a track from another licensed source into `music_library/`. Present explicit options: ``` MUSIC PLAN ├── Your music library: [N tracks / empty] +├── Royalty-free search: [providers / unavailable] ├── AI generation: [provider] — [AVAILABLE/UNAVAILABLE] [cost] └── Bring your own: Drop a track in music_library/ before asset stage Options: (a) Use a library track (which one?) - (b) Provide your own track - (c) Generate via API (if available) - (d) Proceed without music (not recommended for cinematic) + (b) Search a royalty-free provider + (c) Provide your own track + (d) Generate via API (if available) + (e) Proceed without music (not recommended for cinematic) ``` Record the decision in `brief.metadata.music_strategy` with the chosen source and path/prompt. diff --git a/skills/pipelines/cinematic/proposal-director.md b/skills/pipelines/cinematic/proposal-director.md index 1bbdcbd4..096e7cc8 100644 --- a/skills/pipelines/cinematic/proposal-director.md +++ b/skills/pipelines/cinematic/proposal-director.md @@ -29,6 +29,12 @@ Fit cheat-sheet for the recommendation (NOT an auto-decision): **Motion-required deliverables**: if `delivery_promise.motion_required=true`, the chosen runtime is a commitment. Silent downgrade to FFmpeg Ken Burns or still-led animatic is forbidden. If the chosen runtime becomes unavailable at render time, compose must escalate, not substitute. +For an explicit 3D-world promise, query `3d_world_generation`. When +`threejs_world` and HyperFrames are available, this is a real motion path even +if cloud video generation is unavailable: it authors a continuous editable +scene graph with a deterministic camera. Record the tool, local $0 generation +cost, HyperFrames runtime, and atelier mode in the proposal. + A `render_runtime_selection` decision with only one option considered when both were available is a CRITICAL reviewer finding. ## Prerequisites @@ -222,14 +228,16 @@ Let the user select, combine, modify, or redirect entirely. Cinematic videos live and die by their audio. Surface the music situation before the user approves. Check availability in this order: -1. **User music library (`music_library/`)** — list available tracks -2. **Music generation APIs** — report status, cost, and quality honestly -3. **Bring-your-own path** — user can drop a track in `music_library/` +1. **User music library** — query `registry.get_by_capability("music_library")` and list available tracks +2. **Royalty-free search** — query `registry.get_by_capability("music_search")` and report providers/licensing +3. **Music generation APIs** — query `registry.get_by_capability("music_generation")` and report status, cost, and quality honestly +4. **Bring-your-own path** — user can drop a track in `music_library/` Present explicit options: ``` MUSIC PLAN ├── Your music library: [N tracks / empty] +├── Royalty-free search: [providers / unavailable] ├── AI generation: [provider] — [AVAILABLE/UNAVAILABLE] [cost] └── Bring your own: Drop a track in music_library/ before asset stage diff --git a/skills/pipelines/documentary-montage/idea-director.md b/skills/pipelines/documentary-montage/idea-director.md index 204187d2..4e4ab81e 100644 --- a/skills/pipelines/documentary-montage/idea-director.md +++ b/skills/pipelines/documentary-montage/idea-director.md @@ -89,11 +89,14 @@ like abandoned footage at compose time. Do not assume silence will earn itself. If the user has not mentioned music, ASSUME THEY WANT IT and pick: - user-provided track (put path in `music_plan.source_path`), -- music library pick (list what's in `music_library/`), +- music library pick (query `registry.get_by_capability("music_library")` and list tracks), +- royalty-free search (query `registry.get_by_capability("music_search")`, report provider and license), - generated (name the tool and prompt seed with register), - explicit opt-out (`source: "none"` + `opt_out_reason`). -**Warn the user if no music source is available.** Do not silently +Before declaring no source available, also query +`registry.get_by_capability("music_generation")`. **Warn the user if no music +source is available.** Do not silently defer this — it becomes an expensive surprise at the asset stage. ### 5. Note End-Tag Intent (MANDATORY) diff --git a/skills/pipelines/explainer/idea-director.md b/skills/pipelines/explainer/idea-director.md index f500804b..5dda09b9 100644 --- a/skills/pipelines/explainer/idea-director.md +++ b/skills/pipelines/explainer/idea-director.md @@ -129,7 +129,9 @@ If any dimension scores below 3, iterate before submitting. The reviewer will ch ### Step 7: Submit -Call `handle_explainer_idea(state, {"brief": brief_json})` to validate and persist. +Validate `brief_json` against the canonical brief schema, persist it through +the checkpoint protocol, and attach the stage review. There is no separate +explainer submit function. ## Playbook Selection Guide diff --git a/skills/pipelines/explainer/proposal-director.md b/skills/pipelines/explainer/proposal-director.md index db718b72..ffb482eb 100644 --- a/skills/pipelines/explainer/proposal-director.md +++ b/skills/pipelines/explainer/proposal-director.md @@ -369,9 +369,9 @@ Music is a critical part of the video's feel. **Surface the music situation to t **Check music availability in this order:** -1. **User music library (`music_library/`):** Check if this folder exists and contains tracks. If so, list available tracks with durations and let the user pick one. -2. **Music generation APIs:** Check which music tools are available via the registry (`registry.get_by_capability("music_generation")`). Report their status honestly. -3. **Stock music sources:** Note if stock music is available via any provider. +1. **User music library:** Check `registry.get_by_capability("music_library")` and inspect `music_library/`. List tracks with durations. +2. **Royalty-free search:** Check `registry.get_by_capability("music_search")`. Report providers, licensing constraints, and key requirements. +3. **Music generation APIs:** Check `registry.get_by_capability("music_generation")`. Report status, quota, cost, and quality tradeoffs honestly. **Present to the user:** @@ -381,15 +381,17 @@ MUSIC PLAN │ ├── cosmic_interstellar_space.mp3 (3:13) — ambient, cosmic │ ├── cinematic_epic.mp3 (2:45) — dramatic, building │ └── lofi_beat.mp3 (4:00) — chill, electronic +├── Royalty-free search: [providers / unavailable] ├── AI generation: music_gen (ElevenLabs) — UNAVAILABLE (plan limit) └── Recommendation: Use "cosmic_interstellar_space.mp3" from your library OR provide a different track before asset generation Would you like to: (a) Use a track from your library (which one?) - (b) Provide a different track (drop it in music_library/) - (c) Generate one via API (if available) - (d) Proceed without music + (b) Search a royalty-free provider + (c) Provide a different track (drop it in music_library/) + (d) Generate one via API (if available) + (e) Proceed without music ``` **If no music source is available:** Tell the user explicitly. Do NOT let this surface as a surprise at the asset stage. Offer the `music_library/` path so they can add a track before production starts. diff --git a/skills/pipelines/explainer/scene-director.md b/skills/pipelines/explainer/scene-director.md index 9642d666..131eb455 100644 --- a/skills/pipelines/explainer/scene-director.md +++ b/skills/pipelines/explainer/scene-director.md @@ -63,9 +63,17 @@ Transform each script section into 1-3 visual scenes. Each scene is a distinct v } ``` -#### Scene Types and When to Use Them +#### Render Templates and Scene Types -| Type | Best For | Available Tools | Duration Guidance | +The canonical `scene_plan.scenes[].type` vocabulary is `talking_head`, `broll`, +`animation`, `character_scene`, `diagram`, `text_card`, `transition`, +`generated`, and `screen_recording`. Names such as `hero_title`, `stat_card`, +and chart/card variants below are downstream Remotion `cut.type` templates, +not valid scene-plan types. During scene planning, use `text_card` or +`animation` as appropriate and state the intended render template in the scene +description; the Edit stage converts that intent into `cut.type`. + +| Render template or scene type | Best For | Available Tools | Duration Guidance | |------|----------|-----------------|-------------------| | `hero_title` | Opening titles, dramatic reveals | Remotion HeroTitle (theme-driven title treatment) | 3-5s | | `stat_card` | Big dramatic numbers, impactful metrics | Remotion StatCard (large stat + subtitle) | 4-6s | @@ -84,7 +92,11 @@ Transform each script section into 1-3 visual scenes. Each scene is a distinct v | `broll` | Context, real-world examples | Stock or generated footage | 3-6s | | `screen_recording` | Code demos, UI walkthroughs | Recorded or simulated | 5-15s | -**Zero-key scene selection:** When no image/video generation is available, prefer `hero_title`, `stat_card`, `bar_chart`, `line_chart`, `pie_chart`, `kpi_grid`, `comparison`, `callout`, `progress_bar`, and `text_card`. These render entirely from Remotion components with zero external dependencies and can still feel distinct if you derive color, typography, and pacing from the subject instead of defaulting to a generic dashboard aesthetic. +**Zero-key scene selection:** When no image/video generation is available, +plan `text_card`, `animation`, or `diagram` scenes and name an appropriate +downstream template (`hero_title`, `stat_card`, charts, `comparison`, +`callout`, or `progress_bar`) in each description. These render entirely from +Remotion components with zero external dependencies. ### Step 4: Apply the Visual Technique Library @@ -226,7 +238,9 @@ If any dimension scores below 3, revise. ### Step 8: Submit -Call `handle_explainer_scene_plan(state, {"scene_plan": scene_plan_json})` to validate and persist. +Validate `scene_plan_json` against the canonical scene-plan schema, persist it +through the checkpoint protocol, and attach the stage review. There is no +separate explainer submit function. ## Common Pitfalls diff --git a/skills/pipelines/explainer/script-director.md b/skills/pipelines/explainer/script-director.md index f00404c8..f33a9e93 100644 --- a/skills/pipelines/explainer/script-director.md +++ b/skills/pipelines/explainer/script-director.md @@ -206,7 +206,9 @@ If any dimension scores below 3, revise before submitting. ### Step 7: Submit -Call `handle_explainer_script(state, {"script": script_json})` to validate and persist. +Validate `script_json` against the canonical script schema, persist it through +the checkpoint protocol, and attach the stage review. There is no separate +explainer submit function. ### Mid-Production Fact Verification diff --git a/styles/playbook_loader.py b/styles/playbook_loader.py index 0ece61f6..70155eb4 100644 --- a/styles/playbook_loader.py +++ b/styles/playbook_loader.py @@ -26,7 +26,7 @@ SCHEMA_PATH = ( def _load_playbook_schema() -> dict: - with open(SCHEMA_PATH) as f: + with open(SCHEMA_PATH, encoding="utf-8") as f: return json.load(f) @@ -45,7 +45,7 @@ def load_playbook(name: str, styles_dir: Optional[Path] = None) -> dict[str, Any if not path.exists(): raise FileNotFoundError(f"Playbook not found: {path}") - with open(path) as f: + with open(path, encoding="utf-8") as f: playbook = yaml.safe_load(f) validate_playbook(playbook) diff --git a/tests/backlot/test_gate_scenarios.py b/tests/backlot/test_gate_scenarios.py index d00920c3..6d96c4de 100644 --- a/tests/backlot/test_gate_scenarios.py +++ b/tests/backlot/test_gate_scenarios.py @@ -7,7 +7,11 @@ import pytest from backlot import state as state_mod from backlot.state import load_board_state -from lib.checkpoint import CheckpointValidationError, write_checkpoint +from lib.checkpoint import ( + CANONICAL_STAGE_ARTIFACTS, + CheckpointValidationError, + write_checkpoint, +) def _script_artifact() -> dict: @@ -23,6 +27,22 @@ def _manifest_artifact() -> dict: return {"version": "1.0", "assets": [], "total_cost_usd": 0.0} +def _approve_predecessors(tmp_path, project_id, pipeline_type, *stages) -> None: + from tests.contracts.test_phase0_contracts import sample_artifact + + for stage in stages: + artifact_name = CANONICAL_STAGE_ARTIFACTS[stage] + write_checkpoint( + tmp_path, + project_id, + stage, + "completed", + {artifact_name: sample_artifact(artifact_name)}, + pipeline_type=pipeline_type, + human_approved=True, + ) + + def _write(path: Path, data: dict) -> None: path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(data), encoding="utf-8") @@ -73,6 +93,15 @@ def test_handwritten_completed_checkpoint_surfaces_gate_skip(tmp_path, monkeypat def test_awaiting_then_approved_archives_history_without_gate_skip(tmp_path): + _approve_predecessors( + tmp_path, + "film", + "cinematic", + "research", + "proposal", + "script", + "scene_plan", + ) write_checkpoint( tmp_path, "film", diff --git a/tests/backlot/test_state.py b/tests/backlot/test_state.py index 86fbce9b..000123c2 100644 --- a/tests/backlot/test_state.py +++ b/tests/backlot/test_state.py @@ -91,6 +91,9 @@ class TestBoardState: assert sc2["visual"]["exists"] is False # missing file flagged script_stage = next(x for x in s["stages"] if x["name"] == "script") assert script_stage["status"] == "completed" + assert script_stage["produces"] == ["script"] + proposal_stage = next(x for x in s["stages"] if x["name"] == "proposal") + assert proposal_stage["produces"] == ["proposal_packet", "decision_log"] def test_gate_skip_detection(self, projects_root): p = _make_project(projects_root, "sneaky") diff --git a/tests/backlot/test_ui_bug_bash.py b/tests/backlot/test_ui_bug_bash.py index 16ead774..7c1e3458 100644 --- a/tests/backlot/test_ui_bug_bash.py +++ b/tests/backlot/test_ui_bug_bash.py @@ -10,16 +10,132 @@ import urllib.request import pytest +from lib.checkpoint import CANONICAL_STAGE_ARTIFACTS, init_project, write_checkpoint +from lib.pipeline_loader import get_stage_order, load_pipeline from scripts import backlot_screenshot_stage +from tests.contracts.test_phase0_contracts import sample_artifact pytest.importorskip("playwright.sync_api") from playwright.sync_api import sync_playwright # noqa: E402 +APPROVAL_CASES = [ + ("gate-research", "framework-smoke", "research", "research_brief", "Test Topic"), + ("gate-idea", "hybrid", "idea", "brief", "Did you know?"), + ("gate-proposal", "cinematic", "proposal", "proposal_packet", "The Surprising Truth About X"), + ("gate-script", "cinematic", "script", "script", "Hello world"), + ("gate-scene-plan", "cinematic", "scene_plan", "scene_plan", "Host on camera"), + ("gate-assets", "cinematic", "assets", "asset_manifest", "asset-1"), + ("gate-edit", "documentary-montage", "edit", "edit_decisions", "cut-1"), + ("gate-compose", "cinematic", "compose", "render_report", "renders/output.mp4"), + ("gate-publish", "cinematic", "publish", "publish_log", "youtube"), +] + + +def _complete_predecessors(root, project_id: str, pipeline_type: str, stage: str) -> None: + order = get_stage_order(load_pipeline(pipeline_type)) + for predecessor in order[: order.index(stage)]: + artifact_name = CANONICAL_STAGE_ARTIFACTS.get(predecessor) + if artifact_name: + artifact = sample_artifact(artifact_name) + if artifact_name == "edit_decisions": + artifact["render_runtime"] = "ffmpeg" + artifacts = {artifact_name: artifact} + else: + artifacts = {} + write_checkpoint( + root, + project_id, + predecessor, + "completed", + artifacts, + pipeline_type=pipeline_type, + human_approved=True, + ) + + +def _build_approval_projects() -> None: + root = backlot_screenshot_stage.STAGE_DIR + for project_id, pipeline_type, stage, artifact_name, _visible_text in APPROVAL_CASES: + artifact = sample_artifact(artifact_name) + if artifact_name == "edit_decisions": + artifact["render_runtime"] = "ffmpeg" + review_summary = ( + { + "critical": 0, + "suggestions": 1, + "nitpicks": 0, + "review_focus_met": "9/9", + "schema_validation": "proposal_packet PASS", + } + if stage == "proposal" + else "Artifact is ready for human review." + ) + init_project( + project_id, + title=f"Approval fixture: {stage}", + pipeline_type=pipeline_type, + pipeline_dir=root, + ) + _complete_predecessors(root, project_id, pipeline_type, stage) + write_checkpoint( + root, + project_id, + stage, + "awaiting_human", + {artifact_name: artifact}, + pipeline_type=pipeline_type, + review={ + "round": 1, + "decision": "pass", + "critical": 0, + "suggestions": 1, + "nitpicks": 0, + "summary": review_summary, + }, + ) + + # A manifest-declared custom stage/artifact proves the fallback is driven + # by the stage contract rather than a hardcoded canonical-stage list. + init_project( + "gate-character-design", + title="Approval fixture: character design", + pipeline_type="character-animation", + pipeline_dir=root, + ) + _complete_predecessors( + root, + "gate-character-design", + "character-animation", + "character_design", + ) + write_checkpoint( + root, + "gate-character-design", + "character_design", + "awaiting_human", + {"character_design": { + "version": "1.0", + "characters": [{ + "id": "ada", + "display_name": "Ada", + "role": "explorer", + "body_type": "round", + "style": "flat graphic", + "silhouette_notes": "Round explorer with a bright orange field jacket", + "required_emotions": ["curious"], + "required_actions": ["wave"], + }], + }}, + pipeline_type="character-animation", + ) + + @pytest.fixture(scope="module") def staged_backlot_server(): backlot_screenshot_stage.build_stage() + _build_approval_projects() port = 4897 env = dict(os.environ) env["OPENMONTAGE_PROJECTS_DIR"] = str(backlot_screenshot_stage.STAGE_DIR) @@ -57,6 +173,8 @@ def test_project_pages_fit_mobile_and_tablet_widths(staged_backlot_server): "/p/the-slow-orchard?static=1", "/p/the-last-lighthouse?static=1", "/p/paper-boats?static=1", + "/p/gate-proposal?static=1", + "/p/gate-character-design?static=1", ] viewports = [ {"width": 390, "height": 844}, @@ -108,3 +226,69 @@ def test_static_navigation_invalid_route_and_active_takes(staged_backlot_server) assert page.locator(".takes .tk.active").count() >= 1 finally: browser.close() + + +@pytest.mark.parametrize( + ("project_id", "_pipeline_type", "stage", "artifact_name", "visible_text"), + APPROVAL_CASES, +) +def test_every_canonical_gate_promotes_its_artifact_before_approval( + staged_backlot_server, + project_id, + _pipeline_type, + stage, + artifact_name, + visible_text, +): + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True) + page = browser.new_page(viewport={"width": 1280, "height": 900}) + try: + page.goto( + staged_backlot_server + f"/p/{project_id}?static=1", + wait_until="networkidle", + ) + review = page.locator(f'.approval-review[data-stage="{stage}"]') + assert review.is_visible() + assert review.get_by_text("PENDING APPROVAL", exact=True).is_visible() + assert "[object Object]" not in review.inner_text() + artifact = review.locator(f'[data-artifact="{artifact_name}"]') + assert artifact.is_visible() + assert visible_text in artifact.inner_text() + + review.get_by_role("button", name="OPEN FULL ARTIFACT").click() + assert page.locator(".drawer").is_visible() + assert visible_text in page.locator(".drawer").inner_text() + finally: + browser.close() + + +def test_script_gate_keeps_script_visible_and_marks_pending_approval(staged_backlot_server): + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True) + page = browser.new_page(viewport={"width": 1280, "height": 900}) + try: + page.goto(staged_backlot_server + "/p/gate-script?static=1", wait_until="networkidle") + assert page.locator(".script-card").is_visible() + assert page.locator(".script-pending").inner_text() == "PENDING APPROVAL" + finally: + browser.close() + + +def test_manifest_declared_custom_gate_uses_generic_review_fallback(staged_backlot_server): + with sync_playwright() as pw: + browser = pw.chromium.launch(headless=True) + page = browser.new_page(viewport={"width": 1280, "height": 900}) + try: + page.goto( + staged_backlot_server + "/p/gate-character-design?static=1", + wait_until="networkidle", + ) + review = page.locator('.approval-review[data-stage="character_design"]') + assert review.is_visible() + artifact = review.locator('[data-artifact="character_design"]') + assert artifact.is_visible() + assert "Ada" in artifact.inner_text() + assert "Round explorer" in artifact.inner_text() + finally: + browser.close() diff --git a/tests/contracts/test_agent_instruction_integrity.py b/tests/contracts/test_agent_instruction_integrity.py new file mode 100644 index 00000000..82443ab5 --- /dev/null +++ b/tests/contracts/test_agent_instruction_integrity.py @@ -0,0 +1,31 @@ +from pathlib import Path + + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _read(relative_path: str) -> str: + return (REPO_ROOT / relative_path).read_text(encoding="utf-8") + + +def test_music_plans_discover_all_music_capabilities() -> None: + instruction_files = [ + "AGENT_GUIDE.md", + "skills/pipelines/cinematic/idea-director.md", + "skills/pipelines/cinematic/proposal-director.md", + "skills/pipelines/documentary-montage/idea-director.md", + "skills/pipelines/explainer/proposal-director.md", + ] + + for relative_path in instruction_files: + text = _read(relative_path) + for capability in ("music_library", "music_search", "music_generation"): + assert f'get_by_capability("{capability}")' in text, ( + f"{relative_path} omits the {capability!r} music source" + ) + + +def test_explainer_directors_do_not_reference_fictitious_submit_functions() -> None: + for stage in ("idea", "script", "scene"): + text = _read(f"skills/pipelines/explainer/{stage}-director.md") + assert "handle_explainer_" not in text diff --git a/tests/contracts/test_backlot_contract.py b/tests/contracts/test_backlot_contract.py index 50245baa..82a257fc 100644 --- a/tests/contracts/test_backlot_contract.py +++ b/tests/contracts/test_backlot_contract.py @@ -6,6 +6,7 @@ import json import pytest from lib.checkpoint import ( + CANONICAL_STAGE_ARTIFACTS, CheckpointValidationError, HISTORY_DIRNAME, PROJECT_MARKER_FILENAME, @@ -27,6 +28,22 @@ def _minimal_script() -> dict: } +def _approve_predecessors(tmp_path, project_id, pipeline_type, *stages) -> None: + from tests.contracts.test_phase0_contracts import sample_artifact + + for stage in stages: + artifact_name = CANONICAL_STAGE_ARTIFACTS[stage] + write_checkpoint( + tmp_path, + project_id, + stage, + "completed", + artifacts={artifact_name: sample_artifact(artifact_name)}, + pipeline_type=pipeline_type, + human_approved=True, + ) + + class TestGateEnforcement: """GI-4: gated stages cannot be completed without approval evidence.""" @@ -39,6 +56,9 @@ class TestGateEnforcement: ) def test_awaiting_human_is_the_correct_gate_state(self, tmp_path): + _approve_predecessors( + tmp_path, "proj", "animated-explainer", "research", "proposal" + ) path = write_checkpoint( tmp_path, "proj", "script", "awaiting_human", artifacts={"script": _minimal_script()}, @@ -51,6 +71,9 @@ class TestGateEnforcement: assert cp["human_approval_required"] is True def test_completed_with_approval_passes(self, tmp_path): + _approve_predecessors( + tmp_path, "proj", "animated-explainer", "research", "proposal" + ) path = write_checkpoint( tmp_path, "proj", "script", "completed", artifacts={"script": _minimal_script()}, @@ -84,6 +107,9 @@ class TestCheckpointHistory: """Superseded checkpoints are archived, not destroyed.""" def test_overwrite_archives_previous(self, tmp_path): + _approve_predecessors( + tmp_path, "proj", "animated-explainer", "research", "proposal" + ) write_checkpoint( tmp_path, "proj", "script", "awaiting_human", artifacts={"script": _minimal_script()}, diff --git a/tests/contracts/test_env_example.py b/tests/contracts/test_env_example.py new file mode 100644 index 00000000..a2dd245d --- /dev/null +++ b/tests/contracts/test_env_example.py @@ -0,0 +1,19 @@ +from pathlib import Path + +from dotenv import dotenv_values + + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def test_env_example_does_not_turn_comments_into_credentials() -> None: + """A fresh copy of .env.example must leave every documented key unset.""" + + values = dotenv_values(REPO_ROOT / ".env.example") + false_credentials = { + key: value + for key, value in values.items() + if isinstance(value, str) and value.lstrip().startswith("#") + } + + assert false_credentials == {} diff --git a/tests/contracts/test_jimeng_video.py b/tests/contracts/test_jimeng_video.py new file mode 100644 index 00000000..56236c18 --- /dev/null +++ b/tests/contracts/test_jimeng_video.py @@ -0,0 +1,403 @@ +"""Contract tests for the Volcengine Jimeng video provider tool. + +These tests verify that the tool satisfies the BaseTool contract without +requiring real Volcengine AK/SK credentials or making any API calls. + +Run: pytest tests/contracts/test_jimeng_video.py -v +""" + +import pytest + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ToolRuntime, + ToolStability, + ToolStatus, + ToolTier, +) +from tools.video.jimeng_video import JimengVideo + + +# ------------------------------------------------------------------ +# Contract compliance +# ------------------------------------------------------------------ + +class TestContract: + + def test_inherits_base_tool(self): + assert issubclass(JimengVideo, BaseTool) + + def test_has_required_identity(self): + tool = JimengVideo() + assert tool.name == "jimeng_video" + assert tool.version + assert tool.provider == "volcengine" + assert tool.capability == "video_generation" + assert tool.tier == ToolTier.GENERATE + assert tool.stability == ToolStability.EXPERIMENTAL + assert tool.runtime == ToolRuntime.API + + def test_execution_mode_is_async(self): + assert JimengVideo().execution_mode == ExecutionMode.ASYNC + + def test_has_input_schema(self): + schema = JimengVideo().input_schema + assert schema.get("type") == "object" + props = schema.get("properties", {}) + required = schema.get("required", []) + assert required == ["prompt"] + for field in required: + assert field in props + + def test_has_capabilities(self): + tool = JimengVideo() + assert "text_to_video" in tool.capabilities + assert "image_to_video" in tool.capabilities + + def test_has_agent_skills(self): + assert "ai-video-gen" in JimengVideo().agent_skills + + def test_has_fallbacks(self): + tool = JimengVideo() + assert "minimax_video" in tool.fallback_tools + assert "kling_video" in tool.fallback_tools + + def test_has_install_instructions(self): + tool = JimengVideo() + assert "VOLC_ACCESSKEY" in tool.install_instructions + assert "VOLC_SECRETKEY" in tool.install_instructions + + def test_get_info_returns_dict(self): + info = JimengVideo().get_info() + assert isinstance(info, dict) + assert info["name"] == "jimeng_video" + assert info["provider"] == "volcengine" + assert info["runtime"] == "api" + + def test_status_unavailable_without_keys(self, monkeypatch): + monkeypatch.delenv("VOLC_ACCESSKEY", raising=False) + monkeypatch.delenv("VOLC_SECRETKEY", raising=False) + assert JimengVideo().get_status() == ToolStatus.UNAVAILABLE + + def test_status_available_with_keys(self, monkeypatch): + monkeypatch.setenv("VOLC_ACCESSKEY", "fake-ak") + monkeypatch.setenv("VOLC_SECRETKEY", "fake-sk") + assert JimengVideo().get_status() == ToolStatus.AVAILABLE + + def test_status_unavailable_with_only_ak(self, monkeypatch): + monkeypatch.setenv("VOLC_ACCESSKEY", "fake-ak") + monkeypatch.delenv("VOLC_SECRETKEY", raising=False) + assert JimengVideo().get_status() == ToolStatus.UNAVAILABLE + + def test_has_resource_profile(self): + rp = JimengVideo().resource_profile + assert rp.network_required is True + assert rp.vram_mb == 0 + + def test_has_retry_policy(self): + assert JimengVideo().retry_policy.max_retries >= 0 + + def test_has_side_effects(self): + side = JimengVideo().side_effects + assert len(side) > 0 + assert any("API" in s for s in side) + + def test_has_user_visible_verification(self): + assert len(JimengVideo().user_visible_verification) > 0 + + def test_lazy_imports_requests(self, monkeypatch): + import importlib + import sys + mod_name = "tools.video.jimeng_video" + if "requests" in sys.modules: + monkeypatch.delitem(sys.modules, "requests") + importlib.reload(sys.modules[mod_name]) + + def test_estimate_cost_returns_float(self): + cost = JimengVideo().estimate_cost({"prompt": "x", "frames": 121}) + assert isinstance(cost, float) + assert cost > 0.0 + + def test_dry_run_returns_dict(self): + result = JimengVideo().dry_run({"prompt": "test"}) + assert isinstance(result, dict) + assert result["tool"] == "jimeng_video" + + +# ------------------------------------------------------------------ +# Idempotency keys +# ------------------------------------------------------------------ + +class TestIdempotencyKeys: + + def test_includes_all_output_affecting_fields(self): + fields = JimengVideo().idempotency_key_fields + for field in ("prompt", "operation", "image_url", "frames", "aspect_ratio", "seed"): + assert field in fields, f"missing idempotency field: {field}" + + def test_excludes_execution_only_fields(self): + fields = JimengVideo().idempotency_key_fields + for field in ("output_path", "poll_interval_seconds", "timeout_seconds"): + assert field not in fields + + def test_differs_on_frames(self): + tool = JimengVideo() + base = {"prompt": "x"} + assert tool.idempotency_key(base) != tool.idempotency_key({**base, "frames": 241}) + + def test_differs_on_aspect_ratio(self): + tool = JimengVideo() + base = {"prompt": "x"} + assert tool.idempotency_key({**base, "aspect_ratio": "16:9"}) != tool.idempotency_key( + {**base, "aspect_ratio": "9:16"} + ) + + def test_differs_on_seed(self): + tool = JimengVideo() + base = {"prompt": "x"} + assert tool.idempotency_key({**base, "seed": -1}) != tool.idempotency_key( + {**base, "seed": 42} + ) + + def test_differs_on_image_url(self): + tool = JimengVideo() + base = {"prompt": "x", "operation": "image_to_video"} + assert tool.idempotency_key(base) != tool.idempotency_key( + {**base, "image_url": "https://example.com/img.png"} + ) + + +# ------------------------------------------------------------------ +# Tool-specific behavior +# ------------------------------------------------------------------ + +class TestToolSpecific: + + def test_default_frames_is_121(self): + tool = JimengVideo() + assert tool.input_schema["properties"]["frames"]["default"] == 121 + + def test_default_aspect_ratio_is_16_9(self): + tool = JimengVideo() + assert tool.input_schema["properties"]["aspect_ratio"]["default"] == "16:9" + + def test_default_seed_is_negative_one(self): + tool = JimengVideo() + assert tool.input_schema["properties"]["seed"]["default"] == -1 + + def test_cost_scales_with_frames(self): + tool = JimengVideo() + cost_5s = tool.estimate_cost({"prompt": "x", "frames": 121}) + cost_10s = tool.estimate_cost({"prompt": "x", "frames": 241}) + assert cost_10s > cost_5s + + def test_build_payload_t2v(self): + tool = JimengVideo() + payload = tool._build_payload({"prompt": "a cat"}) + assert payload["req_key"] == "jimeng_ti2v_v30_pro" + assert payload["prompt"] == "a cat" + assert payload["frames"] == 121 + assert payload["aspect_ratio"] == "16:9" + assert payload["seed"] == -1 + assert "image_urls" not in payload + + def test_build_payload_i2v_includes_image(self): + tool = JimengVideo() + payload = tool._build_payload({ + "prompt": "motion", + "operation": "image_to_video", + "image_url": "https://example.com/img.png", + }) + assert payload["image_urls"] == ["https://example.com/img.png"] + + def test_build_payload_t2v_omits_image(self): + tool = JimengVideo() + payload = tool._build_payload({"prompt": "a cat", "operation": "text_to_video"}) + assert "image_urls" not in payload + + def test_i2v_without_image_fails(self, monkeypatch): + monkeypatch.setenv("VOLC_ACCESSKEY", "fake-ak") + monkeypatch.setenv("VOLC_SECRETKEY", "fake-sk") + result = JimengVideo().execute({"prompt": "test", "operation": "image_to_video"}) + assert result.success is False + assert "image_url" in result.error + + def test_no_keys_returns_error(self, monkeypatch): + monkeypatch.delenv("VOLC_ACCESSKEY", raising=False) + monkeypatch.delenv("VOLC_SECRETKEY", raising=False) + result = JimengVideo().execute({"prompt": "test"}) + assert result.success is False + assert "VOLC_ACCESSKEY" in result.error + assert "VOLC_SECRETKEY" in result.error + + def test_safe_error_redacts_keys(self, monkeypatch): + monkeypatch.setenv("VOLC_ACCESSKEY", "my-ak-secret") + monkeypatch.setenv("VOLC_SECRETKEY", "my-sk-secret") + redacted = JimengVideo._safe_error( + Exception("failed with ak=my-ak-secret sk=my-sk-secret") + ) + assert "my-ak-secret" not in redacted + assert "my-sk-secret" not in redacted + assert "[redacted]" in redacted + + def test_safe_error_no_empty_string_bug(self, monkeypatch): + """Regression: when no keys are set, _safe_error must not mangle.""" + monkeypatch.delenv("VOLC_ACCESSKEY", raising=False) + monkeypatch.delenv("VOLC_SECRETKEY", raising=False) + msg = JimengVideo._safe_error(Exception("abc")) + assert msg == "abc" + + def test_sign_returns_authorization_header(self): + headers = JimengVideo._sign( + "POST", "/", + {"Action": "CVSync2AsyncSubmitTask", "Version": "2022-08-31"}, + {}, b'{"prompt":"test"}', + "fake-ak", "fake-sk", + ) + assert "Authorization" in headers + assert "HMAC-SHA256" in headers["Authorization"] + assert "fake-ak" in headers["Authorization"] + assert "Host" in headers + assert "X-Date" in headers + assert "X-Content-Sha256" in headers + + def test_sign_includes_content_type(self): + headers = JimengVideo._sign("POST", "/", {}, {}, b"{}", "ak", "sk") + assert headers["Content-Type"] == "application/json" + + def test_json_or_raise_returns_dict(self): + class FakeResp: + status_code = 200 + def json(self): + return {"code": 10000, "data": {"task_id": "123"}} + assert JimengVideo._json_or_raise(FakeResp()) == {"code": 10000, "data": {"task_id": "123"}} + + def test_json_or_raise_raises_on_non_json(self): + class FakeResp: + status_code = 500 + def json(self): + raise ValueError("not JSON") + with pytest.raises(RuntimeError, match="Non-JSON"): + JimengVideo._json_or_raise(FakeResp()) + + def test_check_code_passes_on_success(self): + JimengVideo._check_code(200, {"code": 10000, "message": "Success"}) + + def test_check_code_raises_on_api_error(self): + with pytest.raises(RuntimeError, match="code=10008"): + JimengVideo._check_code(200, {"code": 10008, "message": "Insufficient balance"}) + + def test_check_code_raises_on_http_error(self): + with pytest.raises(RuntimeError, match="HTTP 401"): + JimengVideo._check_code(401, {"code": 10004, "message": "Auth failed"}) + + def test_check_code_defaults_to_success_when_code_missing(self): + """If code field is absent on HTTP 2xx, default to 10000 (success).""" + JimengVideo._check_code(200, {"data": {"task_id": "123"}}) + + +# ------------------------------------------------------------------ +# Registry discovery +# ------------------------------------------------------------------ + +class TestRegistryDiscovery: + + def test_discoverable(self): + from tools.tool_registry import ToolRegistry + registry = ToolRegistry() + registry.discover() + names = {t.name for t in registry._tools.values()} + assert "jimeng_video" in names + + def test_distinct_from_other_minimax_tools(self): + from tools.tool_registry import ToolRegistry + registry = ToolRegistry() + registry.discover() + jimeng = [t for t in registry._tools.values() if t.name == "jimeng_video"] + assert len(jimeng) == 1 + assert jimeng[0].provider == "volcengine" + + +# ------------------------------------------------------------------ +# Schema validation — reject invalid inputs before paid API call +# ------------------------------------------------------------------ + +class TestSchemaValidation: + + def test_frames_accepts_121(self): + schema = JimengVideo().input_schema + valid = schema["properties"]["frames"] + assert valid["enum"] == [121, 241] + + def test_frames_rejects_non_enum(self): + import jsonschema + schema = JimengVideo().input_schema + for invalid in [1, 100, 200, 500, 0, -1]: + instance = {"prompt": "test", "frames": invalid} + with pytest.raises(jsonschema.ValidationError): + jsonschema.validate(instance, schema) + + def test_prompt_max_length_800(self): + schema = JimengVideo().input_schema + assert schema["properties"]["prompt"]["maxLength"] == 800 + + def test_prompt_rejects_over_800_chars(self): + import jsonschema + schema = JimengVideo().input_schema + instance = {"prompt": "x" * 801} + with pytest.raises(jsonschema.ValidationError): + jsonschema.validate(instance, schema) + + def test_prompt_accepts_800_chars(self): + import jsonschema + schema = JimengVideo().input_schema + instance = {"prompt": "x" * 800} + jsonschema.validate(instance, schema) + + def test_seed_minimum_is_negative_one(self): + schema = JimengVideo().input_schema + assert schema["properties"]["seed"]["minimum"] == -1 + + def test_seed_rejects_below_negative_one(self): + import jsonschema + schema = JimengVideo().input_schema + for invalid in [-2, -10, -100]: + instance = {"prompt": "test", "seed": invalid} + with pytest.raises(jsonschema.ValidationError): + jsonschema.validate(instance, schema) + + def test_seed_accepts_negative_one(self): + import jsonschema + schema = JimengVideo().input_schema + jsonschema.validate({"prompt": "test", "seed": -1}, schema) + + def test_seed_accepts_zero_and_positive(self): + import jsonschema + schema = JimengVideo().input_schema + for valid in [0, 1, 42, 999999]: + jsonschema.validate({"prompt": "test", "seed": valid}, schema) + + +# ------------------------------------------------------------------ +# Selector duration → frames mapping +# ------------------------------------------------------------------ + +class TestSelectorDurationMapping: + + def test_duration_5_maps_to_121_frames(self): + payload = JimengVideo._build_payload({"prompt": "x", "duration": 5}) + assert payload["frames"] == 121 + + def test_duration_10_maps_to_241_frames(self): + payload = JimengVideo._build_payload({"prompt": "x", "duration": 10}) + assert payload["frames"] == 241 + + def test_duration_defaults_to_5_when_absent(self): + payload = JimengVideo._build_payload({"prompt": "x"}) + assert payload["frames"] == 121 + + def test_frames_takes_priority_over_duration(self): + payload = JimengVideo._build_payload({"prompt": "x", "frames": 241, "duration": 5}) + assert payload["frames"] == 241 diff --git a/tests/contracts/test_phase2_contracts.py b/tests/contracts/test_phase2_contracts.py index 6c76931f..9c0746cf 100644 --- a/tests/contracts/test_phase2_contracts.py +++ b/tests/contracts/test_phase2_contracts.py @@ -174,10 +174,14 @@ class TestPhase2ErrorHandling: # Either succeeds (provider available) or fails gracefully assert isinstance(r, ToolResult) - def test_diagram_gen_empty_boxes(self): + def test_diagram_gen_empty_boxes(self, tmp_path): tool = DiagramGen() if tool.get_status() == ToolStatus.AVAILABLE: - r = tool.execute({"diagram_type": "boxes", "boxes": []}) + r = tool.execute({ + "diagram_type": "boxes", + "boxes": [], + "output_path": str(tmp_path / "empty-boxes.png"), + }) assert isinstance(r, ToolResult) diff --git a/tests/contracts/test_phase3_contracts.py b/tests/contracts/test_phase3_contracts.py index bed5b747..b35f5fcf 100644 --- a/tests/contracts/test_phase3_contracts.py +++ b/tests/contracts/test_phase3_contracts.py @@ -9,6 +9,7 @@ import builtins import base64 import os import shutil +from types import SimpleNamespace from pathlib import Path from unittest.mock import MagicMock, patch @@ -186,6 +187,7 @@ class TestGoogleMusic: assert info["tier"] == "generate" assert info["capability"] == "music_generation" assert info["provider"] == "google" + assert info["agent_skills"] == ["lyria"] def test_duration_validation(self): tool = GoogleMusic() @@ -570,16 +572,26 @@ class TestVeoVideo: called_kwargs = mock_client.models.generate_videos.call_args[1] assert called_kwargs["image"] is not None - def test_vertex_ai_mode_rejection(self): + def test_vertex_ai_mode_requires_inline_video_bytes(self): tool = VeoVideo() mock_client = MagicMock() mock_client.vertexai = True - if hasattr(mock_client, "_api_client"): - delattr(mock_client, "_api_client") + mock_client.models.generate_videos.return_value = SimpleNamespace( + done=True, + error=None, + response=SimpleNamespace( + generated_videos=[ + SimpleNamespace(video=SimpleNamespace(video_bytes=None)) + ] + ), + ) with ( patch.dict(os.environ, {"GEMINI_API_KEY": "test_key"}), - patch("google.genai.Client", return_value=mock_client), + patch( + "tools.google_credentials.get_genai_client", + return_value=mock_client, + ), ): inputs = { "prompt": "cinematic shot", @@ -588,7 +600,7 @@ class TestVeoVideo: res = tool.execute(inputs) assert res.success is False assert res.error is not None - assert "only supported using the Gemini Developer API" in res.error + assert "without inline bytes" in res.error def test_missing_local_image_paths(self): tool = VeoVideo() diff --git a/tests/contracts/test_pipeline_manifest_categories.py b/tests/contracts/test_pipeline_manifest_categories.py new file mode 100644 index 00000000..54983bb1 --- /dev/null +++ b/tests/contracts/test_pipeline_manifest_categories.py @@ -0,0 +1,9 @@ +"""Contracts for the category vocabulary used by shipped pipelines.""" + +from lib.pipeline_loader import load_pipeline + + +def test_documentary_pipeline_uses_a_schema_supported_category() -> None: + manifest = load_pipeline("documentary-montage") + + assert manifest["category"] == "documentary" diff --git a/tests/contracts/test_remotion_video_transition_contract.py b/tests/contracts/test_remotion_video_transition_contract.py new file mode 100644 index 00000000..3d30934b --- /dev/null +++ b/tests/contracts/test_remotion_video_transition_contract.py @@ -0,0 +1,28 @@ +from pathlib import Path + + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def test_video_scene_honors_hard_cut_tokens_and_backing_color() -> None: + source = (REPO_ROOT / "remotion-composer/src/Explainer.tsx").read_text( + encoding="utf-8" + ) + + assert '["cut", "none"].includes((transitionIn || "").toLowerCase())' in source + assert '["cut", "none"].includes((transitionOut || "").toLowerCase())' in source + assert "transitionIn={cut.transition_in}" in source + assert "transitionOut={cut.transition_out}" in source + assert "sceneDurationSeconds={cut.out_seconds - cut.in_seconds}" in source + assert "Math.round(sceneDurationSeconds * fps)" in source + assert "durationInFrames - transitionFrames" in source + assert "backgroundColor={cut.backgroundColor}" in source + + +def test_cinematic_fades_are_bounded_by_each_scene_duration() -> None: + source = (REPO_ROOT / "remotion-composer/src/CinematicRenderer.tsx").read_text( + encoding="utf-8" + ) + + assert "Math.round(scene.durationSeconds * fps)" in source + assert "durationInFrames - fadeOutFrames" in source diff --git a/tests/contracts/test_utf8_file_io.py b/tests/contracts/test_utf8_file_io.py new file mode 100644 index 00000000..d3a7ba30 --- /dev/null +++ b/tests/contracts/test_utf8_file_io.py @@ -0,0 +1,30 @@ +import ast +from pathlib import Path + +import pytest + + +REPO_ROOT = Path(__file__).resolve().parents[2] +RUNTIME_FILES = [ + "lib/checkpoint.py", + "lib/pipeline_loader.py", + "schemas/artifacts/__init__.py", + "styles/playbook_loader.py", +] + + +@pytest.mark.parametrize("relative_path", RUNTIME_FILES) +def test_pipeline_contract_files_use_explicit_utf8(relative_path: str) -> None: + path = REPO_ROOT / relative_path + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + + bare_opens = [] + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + if not isinstance(node.func, ast.Name) or node.func.id != "open": + continue + if not any(keyword.arg == "encoding" for keyword in node.keywords): + bare_opens.append(node.lineno) + + assert bare_opens == [], f"bare open() calls at lines {bare_opens}" diff --git a/tests/lib/test_checkpoint_prerequisites.py b/tests/lib/test_checkpoint_prerequisites.py new file mode 100644 index 00000000..ee9b5f97 --- /dev/null +++ b/tests/lib/test_checkpoint_prerequisites.py @@ -0,0 +1,152 @@ +import json + +import pytest +from tests.contracts.test_phase0_contracts import sample_artifact + +from lib.checkpoint import ( + CheckpointValidationError, + init_project, + write_checkpoint, +) + + +def _script_artifact() -> dict: + return { + "version": "1.0", + "title": "Smoke", + "total_duration_seconds": 1, + "sections": [ + { + "id": "s1", + "text": "One second.", + "start_seconds": 0, + "end_seconds": 1, + } + ], + } + + +def test_later_stage_cannot_skip_a_missing_predecessor(tmp_path) -> None: + init_project( + "run", + title="Run", + pipeline_type="framework-smoke", + pipeline_dir=tmp_path, + ) + + with pytest.raises(CheckpointValidationError, match="PREREQUISITE VIOLATION"): + write_checkpoint( + tmp_path, + "run", + "script", + "completed", + {"script": _script_artifact()}, + pipeline_type="framework-smoke", + human_approved=True, + ) + + +def test_later_stage_rejects_unapproved_gated_predecessor(tmp_path) -> None: + project_dir = init_project( + "run", + title="Run", + pipeline_type="framework-smoke", + pipeline_dir=tmp_path, + ) + predecessor_path = write_checkpoint( + tmp_path, + "run", + "research", + "awaiting_human", + {"research_brief": sample_artifact("research_brief")}, + pipeline_type="framework-smoke", + ) + predecessor = json.loads(predecessor_path.read_text(encoding="utf-8")) + predecessor["status"] = "completed" + predecessor["human_approved"] = False + predecessor_path.write_text(json.dumps(predecessor), encoding="utf-8") + + with pytest.raises(CheckpointValidationError, match="completed without required approval"): + write_checkpoint( + tmp_path, + "run", + "script", + "completed", + {"script": _script_artifact()}, + pipeline_type="framework-smoke", + human_approved=True, + ) + + +def test_malformed_predecessor_cannot_forge_completion(tmp_path) -> None: + project_dir = init_project( + "run", + title="Run", + pipeline_type="framework-smoke", + pipeline_dir=tmp_path, + ) + (project_dir / "checkpoint_research.json").write_text( + json.dumps({"status": "completed", "human_approved": True}), + encoding="utf-8", + ) + + with pytest.raises(CheckpointValidationError, match="incomplete or missing"): + write_checkpoint( + tmp_path, + "run", + "script", + "completed", + {"script": _script_artifact()}, + pipeline_type="framework-smoke", + human_approved=True, + ) + + +def test_in_progress_heartbeat_is_not_blocked_by_prerequisites(tmp_path) -> None: + init_project( + "run", + title="Run", + pipeline_type="framework-smoke", + pipeline_dir=tmp_path, + ) + + path = write_checkpoint( + tmp_path, + "run", + "script", + "in_progress", + {}, + pipeline_type="framework-smoke", + ) + + assert path.exists() + + +def test_unknown_style_playbook_fails_before_project_creation(tmp_path) -> None: + with pytest.raises(CheckpointValidationError, match="style_playbook"): + init_project( + "run", + title="Run", + pipeline_type="framework-smoke", + pipeline_dir=tmp_path, + style_playbook="does-not-exist", + ) + + assert not (tmp_path / "run").exists() + + +def test_marker_derived_unknown_playbook_blocks_later_writes(tmp_path) -> None: + project_dir = tmp_path / "run" + project_dir.mkdir() + (project_dir / "project.json").write_text( + json.dumps({ + "version": "1.0", + "project_id": "run", + "pipeline_type": "framework-smoke", + "style_playbook": "does-not-exist", + }), + encoding="utf-8", + ) + + with pytest.raises(CheckpointValidationError, match="style_playbook"): + write_checkpoint(tmp_path, "run", "research", "in_progress", {}) diff --git a/tests/lib/test_clip_embedder_compat.py b/tests/lib/test_clip_embedder_compat.py new file mode 100644 index 00000000..fc9e4741 --- /dev/null +++ b/tests/lib/test_clip_embedder_compat.py @@ -0,0 +1,26 @@ +from lib.clip_embedder import _as_feature_tensor + + +class _Tensor: + pass + + +class _ModelOutput: + def __init__(self, pooler_output): + self.pooler_output = pooler_output + self.last_hidden_state = object() + + +def test_transformers_4_tensor_passes_through() -> None: + tensor = _Tensor() + assert _as_feature_tensor(tensor) is tensor + + +def test_transformers_5_output_unwraps_projected_pooler_output() -> None: + tensor = _Tensor() + assert _as_feature_tensor(_ModelOutput(tensor)) is tensor + + +def test_missing_pooler_output_does_not_replace_features_with_none() -> None: + output = _ModelOutput(None) + assert _as_feature_tensor(output) is output diff --git a/tests/lib/test_corpus_diversify_scale.py b/tests/lib/test_corpus_diversify_scale.py new file mode 100644 index 00000000..169b3fe3 --- /dev/null +++ b/tests/lib/test_corpus_diversify_scale.py @@ -0,0 +1,126 @@ +"""Regression tests: Corpus.diversify() must actually diversify at its default. + +The greedy score mixed two incommensurate scales: + + score = diversity * (-sim_picked) + (1 - diversity) * (-remaining.index(i)) + +`sim_picked` is a cosine in [-1, 1]; `remaining.index(i)` is an absolute +list position, unbounded. At the default diversity=0.5 a candidate one +slot later needed a similarity gap > 1.0 to be preferred — impossible for +non-negative cosines — so diversify() returned the input order verbatim, +even when the list held exact duplicates. Its one documented job ("make +sure no two adjacent slots are visually redundant") never happened, and +the effective meaning of `diversity` changed with pool size. + +The fix normalizes the position term to [0, 1] so both terms share a +scale. diversity=0 still returns input order; diversity=1 still picks the +most mutually dissimilar. +""" + +import sys +from pathlib import Path + +import numpy as np +import pytest + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(PROJECT_ROOT)) + +from lib.corpus import EMBED_DIM, ClipRecord, Corpus + + +def _corpus(vectors: list[list[float]]) -> Corpus: + """Build an in-memory corpus from small L2-normalised vectors.""" + corp = Corpus(Path("/nonexistent")) + emb = np.zeros((len(vectors), EMBED_DIM), dtype=np.float32) + for row, vec in enumerate(vectors): + emb[row, : len(vec)] = vec + emb[row] /= np.linalg.norm(emb[row]) + corp.clip_embeddings = emb + corp.records = [ + ClipRecord( + clip_id=f"c{row}", + source="test", + source_id=str(row), + source_url="", + local_path="", + ) + for row in range(len(vectors)) + ] + corp._id_to_row = {r.clip_id: i for i, r in enumerate(corp.records)} + return corp + + +def test_default_diversity_separates_exact_duplicates(): + # c1 is an exact duplicate of c0; c2/c3 are orthogonal to both. + corp = _corpus([[1, 0, 0], [1, 0, 0], [0, 1, 0], [0, 0, 1]]) + + kept = corp.diversify(["c0", "c1", "c2", "c3"], n=3) + + # The duplicate must not sit adjacent to its twin at the default. + assert kept[0] == "c0" + assert kept[1] != "c1" + + +def test_default_diversity_is_not_a_noop(): + # Three exact duplicates (c0, c2, c5) interleaved with distinct clips. + # The old scoring returned the input order verbatim here. + corp = _corpus( + [ + [1, 0, 0], + [0.8, 0.6, 0], + [1, 0, 0], + [0.6, 0.8, 0], + [0, 1, 0], + [1, 0, 0], + ] + ) + ids = [f"c{i}" for i in range(6)] + + kept = corp.diversify(ids, n=6) + + assert kept != ids # reorders when the input holds duplicates + for a, b in zip(kept, kept[1:]): # and no two adjacent slots are twins + va = corp.clip_embeddings[corp._id_to_row[a]] + vb = corp.clip_embeddings[corp._id_to_row[b]] + assert float(va @ vb) < 0.999 + + +def test_diversity_semantics_do_not_collapse_with_pool_size(): + # Duplicate of c0 right at position 1, moderately-related fillers, + # one orthogonal clip at the end. With the unnormalised position + # term the duplicate always won slot 2 at diversity=0.5 because the + # positional penalty scaled with the pool. + half_sqrt3 = np.sqrt(3) / 2 + vectors = [[1, 0, 0], [1, 0, 0]] + [[0.5, half_sqrt3, 0]] * 8 + [[0, 0, 1]] + corp = _corpus(vectors) + ids = [f"c{i}" for i in range(len(vectors))] + + kept = corp.diversify(ids, n=3, diversity=0.5) + + assert kept[0] == "c0" + assert "c1" not in kept[:2] # the exact duplicate no longer wins slot 2 + + +def test_diversity_zero_keeps_input_order(): + corp = _corpus([[1, 0, 0], [1, 0, 0], [0, 1, 0], [0, 0, 1]]) + ids = ["c0", "c1", "c2", "c3"] + + assert corp.diversify(ids, n=4, diversity=0.0) == ids + + +def test_diversity_one_picks_most_dissimilar(): + # c1 duplicates c0; c2 and c3 are mutually orthogonal. + corp = _corpus([[1, 0, 0], [1, 0, 0], [0, 1, 0], [0, 0, 1]]) + + kept = corp.diversify(["c0", "c1", "c2", "c3"], n=3, diversity=1.0) + + assert kept == ["c0", "c2", "c3"] + + +def test_empty_and_unknown_ids(): + corp = _corpus([[1, 0, 0]]) + + assert corp.diversify([], n=3) == [] + assert corp.diversify(["nope"], n=3) == [] + assert corp.diversify(["c0", "nope"], n=3) == ["c0"] diff --git a/tests/tools/test_3d_asset_generation.py b/tests/tools/test_3d_asset_generation.py new file mode 100644 index 00000000..9e86c33b --- /dev/null +++ b/tests/tools/test_3d_asset_generation.py @@ -0,0 +1,158 @@ +"""Contracts for cloud mesh generation and Blender world rendering.""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +import jsonschema + +from tools.base_tool import ToolStatus +from tools.graphics import atlas_3d, fal_3d +from tools.graphics.atlas_3d import Atlas3D +from tools.graphics import blender_world +from tools.graphics.blender_world import BlenderWorld, first_missing_frame +from tools.graphics.fal_3d import Fal3D +from tools.tool_registry import ToolRegistry + + +class _Response: + def __init__(self, payload=None, content=b""): + self._payload = payload + self.content = content + + def json(self): + return self._payload + + def raise_for_status(self): + return None + + +def test_registry_discovers_separate_3d_capabilities(): + registry = ToolRegistry() + registry.discover("tools") + assert {tool.name for tool in registry.get_by_capability("3d_asset_generation")} >= { + "atlas_3d", "fal_3d" + } + assert {tool.name for tool in registry.get_by_capability("3d_world_rendering")} >= { + "blender_world" + } + + +def test_atlas_cost_matrix_and_missing_key(monkeypatch, tmp_path): + for key in ("ATLASCLOUD_API_KEY", "ATLAS_CLOUD_API_KEY", "ATLAS_API_KEY"): + monkeypatch.delenv(key, raising=False) + tool = Atlas3D() + assert tool.get_status() == ToolStatus.UNAVAILABLE + assert tool.estimate_cost({"texture": False}) == 0.22 + assert tool.estimate_cost({"texture": True, "texture_quality": "standard"}) == 0.33 + assert tool.estimate_cost({"texture": True, "texture_quality": "detailed", "geometry_quality": "detailed"}) == 0.66 + result = tool.execute({"prompt": "a cottage", "output_path": str(tmp_path / "cottage.glb")}) + assert not result.success + assert "key" in (result.error or "").lower() + + +def test_fal_cost_matrix_and_input_validation(monkeypatch, tmp_path): + monkeypatch.delenv("FAL_KEY", raising=False) + monkeypatch.delenv("FAL_AI_API_KEY", raising=False) + tool = Fal3D() + assert tool.estimate_cost({"operation": "reconstruct_objects"}) == 0.02 + assert tool.estimate_cost({"operation": "image_to_3d", "enable_pbr": False}) == 0.225 + assert tool.estimate_cost({"operation": "image_to_3d", "enable_pbr": True}) == 0.375 + result = tool.execute({"operation": "text_to_3d", "output_path": str(tmp_path / "asset.glb")}) + assert not result.success + + +def test_blender_doctor_reports_detected_runtime(monkeypatch, tmp_path): + executable = tmp_path / "blender" + executable.write_bytes(b"") + monkeypatch.setattr(blender_world, "find_blender", lambda: executable) + monkeypatch.setattr(blender_world.subprocess, "run", lambda *args, **kwargs: subprocess.CompletedProcess( + args=args[0], returncode=0, stdout="OPENMONTAGE_BLENDER=4.5.10 LTS\n", stderr="", + )) + result = BlenderWorld().execute({"operation": "doctor"}) + assert result.success, result.error + assert result.data["version_line"].startswith("OPENMONTAGE_BLENDER=4.5.10") + + +def test_blender_doctor_explains_missing_optional_runtime(monkeypatch): + monkeypatch.setattr(blender_world, "find_blender", lambda: None) + result = BlenderWorld().execute({"operation": "doctor"}) + assert not result.success + assert "Blender not found" in (result.error or "") + + +def test_blender_resume_finds_first_missing_contiguous_frame(tmp_path): + prefix = tmp_path / "frame-" + for frame in (1, 2, 4): + (tmp_path / f"frame-{frame:04d}.png").write_bytes(b"png") + assert first_missing_frame(prefix, 1, 5) == 3 + (tmp_path / "frame-0003.png").write_bytes(b"png") + assert first_missing_frame(prefix, 1, 4) is None + + +def test_asset_manifest_accepts_generated_mesh_type(): + schema = json.loads(Path("schemas/artifacts/asset_manifest.schema.json").read_text(encoding="utf-8")) + jsonschema.validate({ + "version": "1.0", + "assets": [{ + "id": "hero-cottage", + "type": "3d_asset", + "path": "assets/3d/hero-cottage.glb", + "source_tool": "atlas_3d", + "scene_id": "village", + }], + }, schema) + + +def test_atlas_success_downloads_glb_and_provenance(monkeypatch, tmp_path): + monkeypatch.setenv("ATLASCLOUD_API_KEY", "test-key") + monkeypatch.setattr(atlas_3d.time, "sleep", lambda _seconds: None) + monkeypatch.setattr(atlas_3d.requests, "post", lambda *args, **kwargs: _Response({"data": {"id": "pred-1"}})) + + def fake_get(url, **_kwargs): + if "prediction/pred-1" in url: + return _Response({"data": {"status": "completed", "files": [{ + "type": "glb", "url": "https://cdn.example/asset.glb", + }]}}) + return _Response(content=b"glb-bytes") + + monkeypatch.setattr(atlas_3d.requests, "get", fake_get) + output = tmp_path / "asset.glb" + result = Atlas3D().execute({"prompt": "a weathered cottage", "output_path": str(output)}) + assert result.success, result.error + assert output.read_bytes() == b"glb-bytes" + provenance = json.loads(output.with_suffix(".provenance.json").read_text(encoding="utf-8")) + assert provenance["prediction_id"] == "pred-1" + assert provenance["model"] == "tripo-h3.1/text-to-3d" + + +def test_fal_success_downloads_glb_and_provenance(monkeypatch, tmp_path): + monkeypatch.setenv("FAL_KEY", "test-key") + monkeypatch.setattr(fal_3d.time, "sleep", lambda _seconds: None) + monkeypatch.setattr(fal_3d.requests, "post", lambda *args, **kwargs: _Response({ + "request_id": "req-1", + "status_url": "https://queue.example/status", + "response_url": "https://queue.example/result", + })) + + def fake_get(url, **_kwargs): + if url.endswith("/status"): + return _Response({"status": "COMPLETED"}) + if url.endswith("/result"): + return _Response({"model_urls": {"glb": { + "url": "https://cdn.example/asset.glb", "content_type": "model/gltf-binary", + }}}) + return _Response(content=b"fal-glb") + + monkeypatch.setattr(fal_3d.requests, "get", fake_get) + output = tmp_path / "fal-asset.glb" + result = Fal3D().execute({ + "operation": "text_to_3d", "prompt": "a stone bridge", "output_path": str(output), + }) + assert result.success, result.error + assert output.read_bytes() == b"fal-glb" + provenance = json.loads(output.with_suffix(".provenance.json").read_text(encoding="utf-8")) + assert provenance["request_id"] == "req-1" + assert provenance["provider"] == "fal" diff --git a/tests/tools/test_audio_mixer_target_duration.py b/tests/tools/test_audio_mixer_target_duration.py new file mode 100644 index 00000000..58068450 --- /dev/null +++ b/tests/tools/test_audio_mixer_target_duration.py @@ -0,0 +1,94 @@ +import shutil +import subprocess +import math +import struct +import wave +from pathlib import Path + +import pytest + +from tools.audio.audio_mixer import AudioMixer + + +pytestmark = pytest.mark.skipif( + shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None, + reason="ffmpeg and ffprobe are required", +) + + +def _tone(path: Path, frequency: int, duration: float) -> None: + subprocess.run( + [ + "ffmpeg", "-y", "-f", "lavfi", "-i", + f"sine=frequency={frequency}:duration={duration}", str(path), + ], + capture_output=True, + check=True, + timeout=30, + ) + + +def _duration(path: Path) -> float: + result = subprocess.run( + [ + "ffprobe", "-v", "error", "-show_entries", "format=duration", + "-of", "csv=p=0", str(path), + ], + capture_output=True, + check=True, + text=True, + timeout=30, + ) + return float(result.stdout.strip()) + + +def _tail_rms(path: Path, start_seconds: float, end_seconds: float) -> float: + with wave.open(str(path), "rb") as handle: + sample_width = handle.getsampwidth() + assert sample_width == 2 + frame_rate = handle.getframerate() + handle.setpos(int(start_seconds * frame_rate)) + raw = handle.readframes(int((end_seconds - start_seconds) * frame_rate)) + samples = struct.unpack(f"<{len(raw) // 2}h", raw) + return math.sqrt(sum(sample * sample for sample in samples) / len(samples)) + + +def test_full_mix_can_pin_the_composition_length(tmp_path) -> None: + speech = tmp_path / "speech.wav" + music = tmp_path / "music.wav" + output = tmp_path / "mix.wav" + _tone(speech, 440, 1) + _tone(music, 220, 3) + + result = AudioMixer().execute({ + "operation": "full_mix", + "tracks": [ + {"path": str(speech), "role": "speech"}, + {"path": str(music), "role": "music"}, + ], + "ducking": {"enabled": True}, + "normalize": False, + "target_duration": 3, + "output_path": str(output), + }) + + assert result.success, result.error + assert result.data["target_duration"] == 3 + assert _duration(output) == pytest.approx(3.0, abs=0.15) + assert _tail_rms(output, 2.0, 2.5) > 100 + + +@pytest.mark.parametrize("target", [0, -1, "not-a-number"]) +def test_full_mix_rejects_invalid_target_duration(tmp_path, target) -> None: + tone = tmp_path / "tone.wav" + _tone(tone, 440, 0.25) + + result = AudioMixer().execute({ + "operation": "full_mix", + "tracks": [{"path": str(tone), "role": "speech"}], + "target_duration": target, + "output_path": str(tmp_path / "mix.wav"), + }) + + assert result.success is False + assert "target_duration" in result.error diff --git a/tests/tools/test_audio_mixer_track_fades.py b/tests/tools/test_audio_mixer_track_fades.py new file mode 100644 index 00000000..caac6f14 --- /dev/null +++ b/tests/tools/test_audio_mixer_track_fades.py @@ -0,0 +1,95 @@ +"""Regression tests for delayed per-track fades in ``audio_mixer``.""" + +import shutil +import subprocess +import sys +from pathlib import Path +from types import SimpleNamespace + +import pytest + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(PROJECT_ROOT)) + +from tools.audio.audio_mixer import AudioMixer # noqa: E402 + + +@pytest.mark.parametrize("operation", ["mix", "full_mix"]) +def test_delayed_track_fades_use_the_source_timeline(tmp_path, monkeypatch, operation): + """Fade source audio before delay, and start fade-out at its real end.""" + tone = tmp_path / "tone.wav" + tone.write_bytes(b"stub") + commands = [] + + def fake_run(self, cmd, **kwargs): + commands.append(list(cmd)) + if cmd[0] == "ffprobe": + return SimpleNamespace(stdout="2.0\n", stderr="") + return SimpleNamespace(stdout="", stderr="") + + monkeypatch.setattr(AudioMixer, "run_command", fake_run) + result = AudioMixer().execute( + { + "operation": operation, + "tracks": [{ + "path": str(tone), + "role": "speech", + "start_seconds": 1, + "fade_in_seconds": 0.25, + "fade_out_seconds": 0.5, + }], + "ducking": {"enabled": False}, + "normalize": False, + "output_path": str(tmp_path / "out.wav"), + } + ) + + assert result.success, result.error + ffmpeg_cmd = next(cmd for cmd in commands if cmd[0] == "ffmpeg") + filter_graph = ffmpeg_cmd[ffmpeg_cmd.index("-filter_complex") + 1] + assert ( + "afade=t=in:d=0.25,afade=t=out:st=1.5:d=0.5,adelay=1000|1000" + in filter_graph + ) + + +@pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg required") +def test_delayed_track_with_fade_out_remains_audible_after_its_start(tmp_path): + """End-to-end regression for #347: delayed audio must not be silenced.""" + tone = tmp_path / "tone.wav" + output = tmp_path / "mixed.wav" + subprocess.run( + ["ffmpeg", "-y", "-f", "lavfi", "-i", "sine=frequency=440:duration=2", str(tone)], + capture_output=True, + check=True, + timeout=30, + ) + + result = AudioMixer().execute( + { + "operation": "mix", + "tracks": [{ + "path": str(tone), + "role": "music", + "start_seconds": 1, + "fade_out_seconds": 0.5, + }], + "normalize": False, + "output_path": str(output), + } + ) + assert result.success, result.error + + measured = subprocess.run( + [ + "ffmpeg", "-ss", "1.25", "-t", "0.25", "-i", str(output), + "-vn", "-af", "volumedetect", "-f", "null", "-", + ], + capture_output=True, + text=True, + check=True, + timeout=30, + ) + mean_line = next(line for line in measured.stderr.splitlines() if "mean_volume" in line) + mean_db = float(mean_line.split("mean_volume:")[1].strip().split(" ")[0]) + assert mean_db > -60, f"delayed tone was unexpectedly silent: {mean_db} dB" diff --git a/tests/tools/test_bg_remove_api.py b/tests/tools/test_bg_remove_api.py new file mode 100644 index 00000000..a6d47a6b --- /dev/null +++ b/tests/tools/test_bg_remove_api.py @@ -0,0 +1,28 @@ +import sys +from unittest.mock import MagicMock + +from PIL import Image + + +def test_bg_remove_selects_model_through_rembg_session(monkeypatch, tmp_path) -> None: + fake_rembg = MagicMock() + fake_rembg.new_session.return_value = "selected-session" + fake_rembg.remove.side_effect = lambda image, **kwargs: image.convert("RGBA") + monkeypatch.setitem(sys.modules, "rembg", fake_rembg) + + input_path = tmp_path / "input.png" + Image.new("RGB", (8, 8), (10, 20, 30)).save(input_path) + + from tools.enhancement.bg_remove import BgRemove + + result = BgRemove().execute({ + "input_path": str(input_path), + "output_path": str(tmp_path / "output.png"), + "model": "isnet-general-use", + }) + + assert result.success, result.error + fake_rembg.new_session.assert_called_once_with("isnet-general-use") + kwargs = fake_rembg.remove.call_args.kwargs + assert kwargs["session"] == "selected-session" + assert "model_name" not in kwargs diff --git a/tests/tools/test_cinematic_remotion_adapter.py b/tests/tools/test_cinematic_remotion_adapter.py new file mode 100644 index 00000000..b3b02fa8 --- /dev/null +++ b/tests/tools/test_cinematic_remotion_adapter.py @@ -0,0 +1,177 @@ +import json +from pathlib import Path + +import pytest + +from tools.video.video_compose import VideoCompose + + +def test_cinematic_cut_adapter_builds_a_sequential_timeline() -> None: + scenes = VideoCompose._cuts_to_cinematic_scenes([ + { + "id": "v1", + "source": "clip.mp4", + "in_seconds": 2, + "out_seconds": 6, + "transition_in": "cut", + "transition_out": "none", + }, + { + "id": "title", + "source": "", + "type": "hero_title", + "text": "The signal arrives", + "in_seconds": 0, + "out_seconds": 3, + }, + ]) + + assert scenes[0] == { + "id": "v1", + "startSeconds": 0.0, + "durationSeconds": 4.0, + "kind": "video", + "src": "clip.mp4", + "trimBeforeSeconds": 2.0, + "trimAfterSeconds": 6.0, + "playbackRate": 1.0, + "fadeInFrames": 0, + "fadeOutFrames": 0, + } + assert scenes[1]["kind"] == "title" + assert scenes[1]["startSeconds"] == 4.0 + assert scenes[1]["text"] == "The signal arrives" + + +def test_cinematic_cut_adapter_preserves_playback_speed() -> None: + scenes = VideoCompose._cuts_to_cinematic_scenes([ + { + "id": "fast", + "source": "clip.mp4", + "in_seconds": 2, + "out_seconds": 6, + "speed": 2, + } + ]) + + assert scenes[0]["durationSeconds"] == 2 + assert scenes[0]["playbackRate"] == 2 + + +@pytest.mark.parametrize("uri_style", ["standard", "legacy_windows"]) +def test_remotion_media_staging_decodes_file_uris(tmp_path, uri_style) -> None: + source = tmp_path / "clip with space.mp4" + source.write_bytes(b"video") + public_dir = tmp_path / "public" + uri = source.as_uri() + if uri_style == "legacy_windows" and len(source.drive) == 2: + uri = f"file://{source.drive}{source.as_posix()[2:]}".replace(" ", "%20") + props = {"scenes": [{"src": uri}]} + + staged_count = VideoCompose._stage_remotion_media(props, public_dir) + + assert staged_count == 1 + assert props["scenes"][0]["src"] != uri + assert (public_dir / props["scenes"][0]["src"]).read_bytes() == b"video" + + +def test_remotion_render_adapts_cuts_and_stages_local_video(monkeypatch, tmp_path) -> None: + source = tmp_path / "source.mp4" + source.write_bytes(b"not-a-real-video") + output = tmp_path / "render.mp4" + captured = {} + + def fake_run_command(self, command, **kwargs): + captured["command"] = command + captured["timeout"] = kwargs["timeout"] + props_arg = next(arg for arg in command if arg.startswith("--props=")) + captured["props"] = json.loads(Path(props_arg.split("=", 1)[1]).read_text()) + public_arg = next(arg for arg in command if arg.startswith("--public-dir=")) + public_dir = Path(public_arg.split("=", 1)[1]) + captured["staged_exists_during_render"] = ( + public_dir / captured["props"]["scenes"][0]["src"] + ).exists() + output.write_bytes(b"rendered") + + monkeypatch.setattr(VideoCompose, "run_command", fake_run_command) + + result = VideoCompose()._remotion_render({ + "edit_decisions": { + "renderer_family": "cinematic-trailer", + "cuts": [ + { + "id": "v1", + "source": str(source), + "in_seconds": 0, + "out_seconds": 2, + } + ], + }, + "output_path": str(output), + }) + + assert result.success, result.error + assert "cuts" not in captured["props"] + assert captured["props"]["scenes"][0]["kind"] == "video" + assert captured["staged_exists_during_render"] is True + assert result.data["staged_media_count"] == 1 + + +def test_remotion_timeout_scales_with_scene_count(monkeypatch, tmp_path) -> None: + output = tmp_path / "render.mp4" + captured = {} + + def fake_run_command(self, command, **kwargs): + captured["timeout"] = kwargs["timeout"] + output.write_bytes(b"rendered") + + monkeypatch.setattr(VideoCompose, "run_command", fake_run_command) + cuts = [ + { + "id": f"title-{index}", + "source": "", + "type": "hero_title", + "text": str(index), + "in_seconds": 0, + "out_seconds": 1, + } + for index in range(50) + ] + + result = VideoCompose()._remotion_render({ + "edit_decisions": {"renderer_family": "cinematic-trailer", "cuts": cuts}, + "output_path": str(output), + }) + + assert result.success, result.error + assert captured["timeout"] == 750 + + +def test_remotion_render_preserves_direct_cinematic_scenes(monkeypatch, tmp_path) -> None: + output = tmp_path / "render.mp4" + captured = {} + + def fake_run_command(self, command, **kwargs): + props_arg = next(arg for arg in command if arg.startswith("--props=")) + captured["props"] = json.loads(Path(props_arg.split("=", 1)[1]).read_text()) + output.write_bytes(b"rendered") + + monkeypatch.setattr(VideoCompose, "run_command", fake_run_command) + scene = { + "id": "authored", + "kind": "title", + "text": "Keep me", + "startSeconds": 0, + "durationSeconds": 1, + } + + result = VideoCompose()._remotion_render({ + "composition_data": { + "renderer_family": "cinematic-trailer", + "scenes": [scene], + }, + "output_path": str(output), + }) + + assert result.success, result.error + assert captured["props"]["scenes"] == [scene] diff --git a/tests/tools/test_corpus_builder_total_failure.py b/tests/tools/test_corpus_builder_total_failure.py new file mode 100644 index 00000000..c0180380 --- /dev/null +++ b/tests/tools/test_corpus_builder_total_failure.py @@ -0,0 +1,59 @@ +from dataclasses import dataclass + +import pytest + +import tools.video.stock_sources as stock_sources +from tools.video.corpus_builder import CorpusBuilder + + +@dataclass +class _Candidate: + clip_id: str + + +class _Source: + name = "fake" + + def __init__(self, count: int) -> None: + self.count = count + + def is_available(self) -> bool: + return True + + def search(self, query, filters): + return [_Candidate(f"clip-{index}") for index in range(self.count)] + + +@pytest.fixture +def run_builder(monkeypatch, tmp_path): + def run(count: int, processor): + monkeypatch.setattr(stock_sources, "available_sources", lambda: [_Source(count)]) + monkeypatch.setattr(stock_sources, "source_summary", lambda: {}) + monkeypatch.setattr(CorpusBuilder, "_process_candidate", processor) + return CorpusBuilder().execute({ + "corpus_dir": str(tmp_path / f"corpus-{count}"), + "queries": [{"query": "city at night"}], + "max_new_clips": 50, + }) + + return run + + +def test_all_candidate_failures_fail_closed_with_diagnostics(run_builder) -> None: + def broken_clip_stack(*args, **kwargs): + raise AttributeError("BaseModelOutput has no attribute norm") + + result = run_builder(4, broken_clip_stack) + + assert result.success is False + assert result.data["candidates_seen"] == 4 + assert result.data["clips_failed"] == 4 + assert "corpus index is empty" in result.error + assert "BaseModelOutput" in result.error + + +def test_no_candidates_is_a_valid_empty_search(run_builder) -> None: + result = run_builder(0, lambda *args, **kwargs: None) + + assert result.success is True + assert result.data["candidates_seen"] == 0 diff --git a/tests/tools/test_google_imagen_multi_output.py b/tests/tools/test_google_imagen_multi_output.py new file mode 100644 index 00000000..51967cf6 --- /dev/null +++ b/tests/tools/test_google_imagen_multi_output.py @@ -0,0 +1,104 @@ +"""Regression tests: google_imagen must return every image it requests and bills for. + +`execute()` sends `sampleCount = number_of_images` to the Imagen API and +`estimate_cost` scales with `number_of_images`, but result handling was +hardcoded to `predictions[0]` — images 1..n-1 were decoded never, written +never, and absent from `artifacts`. Worse, `images_generated` reported +`len(predictions)`, so the result claimed n images while only one reached +disk. The user paid for n images and received one. + +Mirrors tests/tools/test_openai_image_multi_output.py, which covers the same +defect class in the OpenAI provider. +""" + +import base64 +import sys +import types +from pathlib import Path + +import pytest + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(PROJECT_ROOT)) + + +class _FakeResponse: + def __init__(self, count: int): + self._count = count + + def raise_for_status(self): + return None + + def json(self): + return { + "predictions": [ + {"bytesBase64Encoded": base64.b64encode(f"IMAGE_{i}".encode()).decode()} + for i in range(self._count) + ] + } + + +@pytest.fixture +def imagen_tool(monkeypatch): + # Stub `requests` so execute() runs fully offline; echo back sampleCount + # images so the fake provider honors what the tool asked and billed for. + fake = types.ModuleType("requests") + + def _post(url, headers=None, json=None, timeout=None): + return _FakeResponse(json["parameters"]["sampleCount"]) + + fake.post = _post + monkeypatch.setitem(sys.modules, "requests", fake) + monkeypatch.setenv("GOOGLE_API_KEY", "test-key") + from tools.graphics.google_imagen import GoogleImagen + + return GoogleImagen() + + +def test_all_requested_images_are_written(imagen_tool, tmp_path): + out = tmp_path / "gen.png" + result = imagen_tool.execute( + {"prompt": "p", "number_of_images": 4, "output_path": str(out)} + ) + + assert result.success + assert result.data["images_generated"] == 4 + assert len(result.artifacts) == 4 + + files = sorted(tmp_path.glob("*.png")) + assert len(files) == 4 # every image reached disk, none overwritten + contents = {f.read_bytes() for f in files} + assert contents == {b"IMAGE_0", b"IMAGE_1", b"IMAGE_2", b"IMAGE_3"} + + +def test_artifacts_match_billed_image_count(imagen_tool, tmp_path): + # What the user pays for must equal what they receive. + inputs = { + "prompt": "p", + "number_of_images": 3, + "output_path": str(tmp_path / "img.png"), + } + result = imagen_tool.execute(inputs) + billed = imagen_tool.estimate_cost(inputs) + + assert len(result.artifacts) == 3 + assert billed == pytest.approx(0.04 * 3) + + +def test_single_image_keeps_exact_output_path(imagen_tool, tmp_path): + out = tmp_path / "single.png" + result = imagen_tool.execute( + {"prompt": "p", "number_of_images": 1, "output_path": str(out)} + ) + + assert result.success + assert result.artifacts == [str(out)] + assert out.read_bytes() == b"IMAGE_0" + + +def test_multi_output_paths_are_suffixed_and_unique(): + from tools.graphics.google_imagen import GoogleImagen + + paths = GoogleImagen._output_paths("/tmp/art/pic.png", 3) + assert [p.name for p in paths] == ["pic_1.png", "pic_2.png", "pic_3.png"] + assert len(set(paths)) == 3 diff --git a/tests/tools/test_google_vertex_backends.py b/tests/tools/test_google_vertex_backends.py new file mode 100644 index 00000000..375ba14f --- /dev/null +++ b/tests/tools/test_google_vertex_backends.py @@ -0,0 +1,97 @@ +from pathlib import Path +from types import SimpleNamespace + +import pytest + + +def test_blank_google_location_uses_documented_default(monkeypatch) -> None: + monkeypatch.setenv("GOOGLE_CLOUD_LOCATION", "") + from tools.google_credentials import resolve_google_location + + assert resolve_google_location() == "us-central1" + + +def test_google_music_requests_the_global_vertex_location(monkeypatch, tmp_path) -> None: + monkeypatch.setenv("GOOGLE_API_KEY", "test-key") + import tools.google_credentials as credentials + + captured = {} + + def stop_before_network(http_options=None, location=None): + captured["location"] = location + raise RuntimeError("stop before network") + + monkeypatch.setattr(credentials, "get_genai_client", stop_before_network) + + from tools.audio.google_music import GoogleMusic + + result = GoogleMusic().execute({ + "prompt": "solo piano", + "output_path": str(tmp_path / "music.mp3"), + }) + + assert result.success is False + assert captured["location"] == "global" + + +class _VideoAsset: + uri = None + + def __init__(self, video_bytes): + self.video_bytes = video_bytes + + def save(self, path): + Path(path).write_bytes(self.video_bytes or b"") + + +class _Models: + def __init__(self, video_bytes): + self.video_bytes = video_bytes + + def generate_videos(self, **kwargs): + asset = _VideoAsset(self.video_bytes) + generated = SimpleNamespace(video=asset) + response = SimpleNamespace(generated_videos=[generated]) + return SimpleNamespace(done=True, error=None, response=response) + + +class _VertexClient: + vertexai = True + + def __init__(self, video_bytes): + self.models = _Models(video_bytes) + self.files = SimpleNamespace( + download=lambda **kwargs: pytest.fail("Vertex must not call files.download") + ) + + +@pytest.mark.parametrize( + ("video_bytes", "expected_success"), + [(b"VIDEO_BYTES", True), (None, False)], +) +def test_veo_accepts_vertex_and_requires_inline_bytes( + monkeypatch, tmp_path, video_bytes, expected_success +) -> None: + monkeypatch.setenv("GOOGLE_API_KEY", "test-key") + import tools.google_credentials as credentials + + monkeypatch.setattr( + credentials, + "get_genai_client", + lambda http_options=None, location=None: _VertexClient(video_bytes), + ) + + from tools.video.veo_video import VeoVideo + + output = tmp_path / "video.mp4" + result = VeoVideo().execute({ + "backend": "google", + "prompt": "wind moving across grassland", + "output_path": str(output), + }) + + assert result.success is expected_success + if expected_success: + assert output.read_bytes() == b"VIDEO_BYTES" + else: + assert "without inline bytes" in result.error diff --git a/tests/tools/test_green_screen_chromakey.py b/tests/tools/test_green_screen_chromakey.py new file mode 100644 index 00000000..ab4bcf3a --- /dev/null +++ b/tests/tools/test_green_screen_chromakey.py @@ -0,0 +1,116 @@ +"""Regression tests: chromakey compositing must not collapse frames to 1x1. + +`_process_chromakey` built the background from a 1x1 lavfi color source and +tried to size it with `[0:v]scale=iw:ih` — a no-op, since iw/ih are the 1x1 +source's own dimensions. FFmpeg's `overlay` takes the size of its first input +(the 1x1 background), so every processed frame was clipped to a single pixel. +`_reconstruct_video` then upscaled those 1x1 frames, yielding a solid-color +video with the keyed subject entirely gone. The fix uses `scale2ref` to resize +the background to the actual frame dimensions. +""" + +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(PROJECT_ROOT)) + +from tools.video.green_screen_processor import GreenScreenProcessor # noqa: E402 + + +def test_chromakey_filter_scales_background_to_frame(): + """Offline: the built filtergraph must not leave a 1x1 background.""" + captured = [] + + def fake_run(self, cmd, **kwargs): + captured.append(list(cmd)) + + class _R: + returncode = 0 + stdout = "" + stderr = "" + + return _R() + + tool = GreenScreenProcessor() + orig = GreenScreenProcessor.run_command + GreenScreenProcessor.run_command = fake_run + try: + frames_dir = Path(PROJECT_ROOT) / "tests" / "tools" # any dir; glob may be empty + # Drive the filter build directly with a temp frame present. + import tempfile + + with tempfile.TemporaryDirectory() as td: + fd = Path(td) / "frames" + pd = Path(td) / "processed" + fd.mkdir() + pd.mkdir() + (fd / "frame_0000.png").write_bytes(b"stub") + tool._process_chromakey(fd, pd, "#0E172A", 1, 320, 240) + finally: + GreenScreenProcessor.run_command = orig + + ffmpeg_cmds = [c for c in captured if c and c[0] == "ffmpeg"] + assert ffmpeg_cmds, "no ffmpeg command built" + cmd = ffmpeg_cmds[0] + fc = cmd[cmd.index("-filter_complex") + 1] + # Background must be sized to the frame, not the old 1x1 no-op. + assert "size=320x240" in " ".join(cmd), "background not sized to frame" + assert "size=1x1" not in " ".join(cmd), "still using the 1x1 background" + assert "[0:v]scale=iw:ih[bg]" not in fc, "still using the 1x1 no-op scale" + # Alpha must be forced so keyed transparency survives on every FFmpeg build. + assert "format=yuva420p" in fc, f"keyed alpha not forced: {fc}" + + +@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None, + reason="ffmpeg/ffprobe required") +def test_chromakey_preserves_frame_size_and_keys(tmp_path): + """End-to-end: output keeps the source size and keys green -> background.""" + frames_dir = tmp_path / "frames" + processed_dir = tmp_path / "processed" + frames_dir.mkdir() + processed_dir.mkdir() + + frame = frames_dir / "frame_0000.png" + subprocess.run( + ["ffmpeg", "-y", "-f", "lavfi", "-i", + "color=c=0x00FF00:size=320x240,drawbox=x=100:y=80:w=120:h=80:color=red:t=fill", + "-frames:v", "1", str(frame)], + capture_output=True, check=True, timeout=60, + ) + + ok = GreenScreenProcessor()._process_chromakey( + frames_dir, processed_dir, "#0E172A", 1, 320, 240 + ) + assert ok + + out = processed_dir / "frame_0000.png" + assert out.exists() + + def _size(p): + r = subprocess.run( + ["ffprobe", "-v", "quiet", "-show_entries", "stream=width,height", + "-of", "csv=p=0", str(p)], + capture_output=True, text=True, timeout=30, + ) + w, h = r.stdout.strip().split(",") + return int(w), int(h) + + assert _size(out) == (320, 240), "frame collapsed instead of keeping source size" + + def _pixel(p, x, y): + r = subprocess.run( + ["ffmpeg", "-v", "quiet", "-i", str(p), "-vf", f"crop=1:1:{x}:{y}", + "-f", "rawvideo", "-pix_fmt", "rgb24", "-"], + capture_output=True, timeout=30, + ) + return r.stdout[:3] + + center = _pixel(out, 160, 120) # red box -> stays red + corner = _pixel(out, 10, 10) # was green -> keyed to dark background + assert center[0] > 150 and center[1] < 80, f"subject lost, center={center!r}" + assert corner[0] < 60 and corner[1] < 60, f"green not keyed, corner={corner!r}" diff --git a/tests/tools/test_hyperframes_compose.py b/tests/tools/test_hyperframes_compose.py index 41997990..d3780fcf 100644 --- a/tests/tools/test_hyperframes_compose.py +++ b/tests/tools/test_hyperframes_compose.py @@ -250,6 +250,11 @@ def test_runtime_check_succeeds_when_npm_resolves(monkeypatch): "_resolve_npm_package", classmethod(lambda cls: {"version": "0.4.5"}), ) + monkeypatch.setattr( + HyperFramesCompose, + "_probe_cli", + classmethod(lambda cls: {"status": "ok"}), + ) rc = HyperFramesCompose()._runtime_check() # Local binaries must still pass for this to go green. if rc["node_major"] is None or not rc["ffmpeg_available"] or not rc["npx_available"]: @@ -259,6 +264,31 @@ def test_runtime_check_succeeds_when_npm_resolves(monkeypatch): assert rc["reasons"] == [] +def test_runtime_check_fails_when_published_cli_crashes(monkeypatch): + monkeypatch.setattr( + HyperFramesCompose, + "_resolve_npm_package", + classmethod(lambda cls: {"version": "0.7.89"}), + ) + monkeypatch.setattr( + HyperFramesCompose, + "_probe_cli", + classmethod( + lambda cls: { + "error": 'doctor failed: The "file" argument must be of type string' + } + ), + ) + + rc = HyperFramesCompose()._runtime_check() + + if rc["node_major"] is None or not rc["ffmpeg_available"] or not rc["npx_available"]: + pytest.skip("Local runtime floor not met on this machine") + assert rc["runtime_available"] is False + assert rc["cli_probe_error"] is not None + assert any("not executable" in reason for reason in rc["reasons"]) + + def test_video_compose_render_engines_follow_hyperframes_runtime_check(monkeypatch): """Regression: `video_compose.get_info()['render_engines']['hyperframes']` must track the true availability, not just the local-binary floor. @@ -353,6 +383,40 @@ def test_hyperframes_render_requires_workspace(): assert ("workspace" in err) or ("runtime" in err) or ("hyperframes" in err) +def test_hyperframes_render_resolves_relative_output_path_once(tmp_path, monkeypatch): + """A successful CLI render must not be reported missing for a relative path.""" + import subprocess + + from tools.base_tool import ToolResult + + workspace = tmp_path / "workspace" + workspace.mkdir() + tool = HyperFramesCompose() + monkeypatch.chdir(tmp_path) + monkeypatch.setattr(tool, "_runtime_check", lambda: {"runtime_available": True}) + monkeypatch.setattr(tool, "_scaffold", lambda inputs: ToolResult(success=True, data={})) + monkeypatch.setattr(tool, "_lint", lambda inputs: ToolResult(success=True, data={})) + monkeypatch.setattr(tool, "_validate", lambda inputs: ToolResult(success=True, data={})) + + def run_render(args, *, cwd, timeout, check): + output = Path(args[args.index("--output") + 1]) + rendered_output = output if output.is_absolute() else cwd / output + rendered_output.parent.mkdir(parents=True, exist_ok=True) + rendered_output.write_bytes(b"rendered") + return subprocess.CompletedProcess(args, 0, "", "") + + monkeypatch.setattr(tool, "_run_hf", run_render) + + result = tool._render( + {"workspace_path": str(workspace), "output_path": "renders/final.mp4"} + ) + + expected = tmp_path / "renders" / "final.mp4" + assert result.success, result.error + assert result.data["output"] == str(expected) + assert result.artifacts == [str(expected)] + + # ------------------------------------------------------------------ # video_compose runtime routing # ------------------------------------------------------------------ diff --git a/tests/tools/test_mps_device.py b/tests/tools/test_mps_device.py index 844d633e..b3f5fdbd 100644 --- a/tests/tools/test_mps_device.py +++ b/tests/tools/test_mps_device.py @@ -213,8 +213,9 @@ def test_upscale_build_upsampler_uses_signature_guard(monkeypatch): # Build a fake RealESRGANer whose __init__ DOES accept device= class FakeRealESRGANer: - def __init__(self, *, scale, model_path, model, dni_weight, half, device=None): + def __init__(self, *, scale, model_path, model, dni_weight, half, tile=0, tile_pad=10, device=None): self.called_with_device = device + self.called_with_tile = tile fake_realesrganer_cls = FakeRealESRGANer monkeypatch.setitem(sys.modules, "torch", fake_torch) @@ -239,6 +240,7 @@ def test_upscale_build_upsampler_uses_signature_guard(monkeypatch): tool = upscale.Upscale() result = tool._build_upsampler(scale=4, model_name="RealESRGAN_x4plus", denoise_strength=0.5, face_enhance=False) assert result.called_with_device == "device(mps)" + assert result.called_with_tile == 256 def test_upscale_build_upsampler_skips_device_when_unsupported(monkeypatch): @@ -252,8 +254,9 @@ def test_upscale_build_upsampler_skips_device_when_unsupported(monkeypatch): # Build a fake RealESRGANer whose __init__ does NOT accept device= class FakeRealESRGANerNoDevice: - def __init__(self, *, scale, model_path, model, dni_weight, half): + def __init__(self, *, scale, model_path, model, dni_weight, half, tile=0, tile_pad=10): self.called_with_device = None # no device param + self.called_with_tile = tile fake_realesrganer_cls = FakeRealESRGANerNoDevice monkeypatch.setitem(sys.modules, "torch", fake_torch) @@ -279,6 +282,7 @@ def test_upscale_build_upsampler_skips_device_when_unsupported(monkeypatch): # Should NOT raise TypeError about unexpected keyword argument 'device' result = tool._build_upsampler(scale=4, model_name="RealESRGAN_x4plus", denoise_strength=0.5, face_enhance=False) assert result.called_with_device is None + assert result.called_with_tile == 256 # ------------------------------------------------------------------ diff --git a/tests/tools/test_remotion_audio_mux.py b/tests/tools/test_remotion_audio_mux.py new file mode 100644 index 00000000..210129b8 --- /dev/null +++ b/tests/tools/test_remotion_audio_mux.py @@ -0,0 +1,59 @@ +import shutil +import subprocess +from pathlib import Path + +import pytest + +from tools.video.video_compose import VideoCompose + + +pytestmark = pytest.mark.skipif( + shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None, + reason="ffmpeg and ffprobe are required", +) + + +def _run(command: list[str]) -> None: + subprocess.run(command, capture_output=True, check=True, timeout=30) + + +def test_external_audio_mux_adds_audible_stream_without_changing_video_length(tmp_path) -> None: + video = tmp_path / "video.mp4" + audio = tmp_path / "audio.wav" + _run([ + "ffmpeg", "-y", "-f", "lavfi", "-i", + "color=c=red:s=320x180:d=2:r=30", + "-c:v", "libx264", "-pix_fmt", "yuv420p", str(video), + ]) + _run([ + "ffmpeg", "-y", "-f", "lavfi", "-i", + "sine=frequency=440:duration=1", str(audio), + ]) + + result = VideoCompose()._mux_external_audio(video, audio) + + assert result.success, result.error + streams = subprocess.run( + [ + "ffprobe", "-v", "error", "-show_entries", + "stream=codec_type", "-of", "csv=p=0", str(video), + ], + capture_output=True, + check=True, + text=True, + timeout=30, + ).stdout.splitlines() + duration = float(subprocess.run( + [ + "ffprobe", "-v", "error", "-show_entries", "format=duration", + "-of", "csv=p=0", str(video), + ], + capture_output=True, + check=True, + text=True, + timeout=30, + ).stdout.strip()) + + assert "video" in streams + assert "audio" in streams + assert duration == pytest.approx(2.0, abs=0.15) diff --git a/tests/tools/test_threejs_asset_catalog.py b/tests/tools/test_threejs_asset_catalog.py new file mode 100644 index 00000000..f68c1d22 --- /dev/null +++ b/tests/tools/test_threejs_asset_catalog.py @@ -0,0 +1,45 @@ +import json +import zipfile +from pathlib import Path + +from tools.graphics.threejs_asset_catalog import CATALOGS, ThreeJSAssetCatalog + + +def test_catalog_list_is_rights_explicit(): + result = ThreeJSAssetCatalog().execute({"operation": "list"}) + assert result.success + assert result.data["catalogs"] + assert all(item["license"] == "CC0-1.0" for item in result.data["catalogs"].values()) + + +def test_catalog_install_inventories_gltf(tmp_path, monkeypatch): + source_zip = tmp_path / "fixture.zip" + with zipfile.ZipFile(source_zip, "w") as package: + package.writestr("Models/GLTF format/Tree.gltf", json.dumps({"asset": {"version": "2.0"}})) + package.writestr("Models/GLTF format/Tree.bin", b"mesh") + package.writestr("Textures/tree.png", b"texture") + + fixture_id = "fixture-catalog" + monkeypatch.setitem(CATALOGS, fixture_id, { + "title": "Fixture", + "source_url": "https://example.test/source", + "download_url": "https://example.test/catalog.zip", + "license": "CC0-1.0", + "license_url": "https://creativecommons.org/publicdomain/zero/1.0/", + "tags": ["fixture"], + }) + + def fake_download(_url: str, destination: Path) -> None: + destination.write_bytes(source_zip.read_bytes()) + + monkeypatch.setattr("tools.graphics.threejs_asset_catalog._download", fake_download) + output = tmp_path / "installed" + result = ThreeJSAssetCatalog().execute({ + "operation": "install", + "catalog_id": fixture_id, + "output_path": str(output), + }) + assert result.success, result.error + assert result.data["model_count"] == 1 + assert result.data["texture_count"] == 1 + assert (output / "catalog-manifest.json").exists() diff --git a/tests/tools/test_threejs_world.py b/tests/tools/test_threejs_world.py new file mode 100644 index 00000000..47815bf6 --- /dev/null +++ b/tests/tools/test_threejs_world.py @@ -0,0 +1,243 @@ +"""Contracts for semantic Three.js world generation and atelier rendering.""" + +from __future__ import annotations + +import hashlib +import json +import subprocess +from pathlib import Path + +from tools.base_tool import ToolResult +from tools.graphics.threejs_world import ThreeJSWorld +from tools.tool_registry import ToolRegistry +from tools.video.hyperframes_compose import HyperFramesCompose +from tools.video.video_compose import VideoCompose + + +def _world_spec(duration: float = 12.0) -> dict: + return { + "version": "1.0", + "title": "The Luminous Divide", + "seed": 260805248, + "explicit_constraints": ["one continuous explorable world"], + "inferred_details": ["cyan emissive accents provide visual continuity"], + "world": { + "size": 96, + "resolution": 72, + "elevation_scale": 12, + "water_level": -1.5, + }, + "regions": [ + { + "id": "wetlands", + "center": [-0.45, 0.15], + "radius": 0.9, + "landform": "basin", + "color": "#204a43", + "accent_color": "#71f7c4", + "scatter": {"tree": 18, "rock": 8, "crystal": 6}, + }, + { + "id": "rift", + "center": [0.5, -0.1], + "radius": 0.9, + "landform": "canyon", + "color": "#503040", + "accent_color": "#ff765f", + "scatter": {"tree": 0, "rock": 18, "crystal": 9}, + }, + ], + "landmarks": [ + { + "id": "threshold-ring", + "type": "ring", + "region_id": "wetlands", + "position": [-22, 0, 8], + "scale": 3.5, + } + ], + "camera_path": [ + {"time": 0, "position": [-42, 23, 38], "target": [-18, 0, 4]}, + {"time": duration / 2, "position": [0, 16, 24], "target": [10, 0, -4]}, + {"time": duration, "position": [42, 25, -34], "target": [20, 0, -5]}, + ], + } + + +def test_threejs_world_contract_and_registry_discovery(): + tool = ThreeJSWorld() + assert tool.capability == "3d_world_generation" + assert tool.provider == "threejs" + assert "threejs-world-generation" in tool.agent_skills + assert {"cinematic", "semantic", "wireframe"} == set( + tool.input_schema["properties"]["render_mode"]["enum"] + ) + + registry = ToolRegistry() + registry.discover("tools") + assert "threejs_world" in { + discovered.name + for discovered in registry.get_by_capability("3d_world_generation") + } + + +def test_threejs_world_validate_emits_worldclaw_diagnostics(): + result = ThreeJSWorld().execute( + {"operation": "validate", "world_spec": _world_spec(), "duration_seconds": 12} + ) + assert result.success, result.error + report = result.data["report"] + assert report["valid"] is True + assert report["stats"]["region_count"] == 2 + assert report["stats"]["terrain_triangles"] > 0 + assert set(report["diagnostic_passes"]) == {"cinematic", "semantic", "wireframe"} + assert report["review_views"] == [ + "global", + "regional", + "walk", + "semantic", + "wireframe", + ] + + +def test_threejs_world_build_is_deterministic_and_editable(tmp_path): + workspaces = [tmp_path / "first", tmp_path / "second"] + hashes = [] + for workspace in workspaces: + result = ThreeJSWorld().execute( + { + "operation": "build", + "world_spec": _world_spec(), + "output_path": str(workspace), + "duration_seconds": 12, + "width": 1280, + "height": 720, + "render_mode": "semantic", + } + ) + assert result.success, result.error + for filename in ( + "index.html", + "world.css", + "world-runtime.js", + "world.json", + "world-spec.js", + "world-report.json", + "hyperframes.json", + ): + assert (workspace / filename).is_file() + index = (workspace / "index.html").read_text(encoding="utf-8") + assert "--world-width: 1280px" in index + assert 'data-render-mode="semantic"' in index + hashes.append( + hashlib.sha256((workspace / "world-spec.js").read_bytes()).hexdigest() + ) + assert hashes[0] == hashes[1] + assert json.loads((workspaces[0] / "world.json").read_text(encoding="utf-8"))[ + "seed" + ] == 260805248 + + +def test_threejs_world_rejects_incomplete_camera_path(): + spec = _world_spec() + spec["camera_path"][-1]["time"] = 11 + result = ThreeJSWorld().execute( + {"operation": "validate", "world_spec": spec, "duration_seconds": 12} + ) + assert not result.success + assert "Last camera key" in (result.error or "") + + +def test_production_tier_rejects_primitive_only_spec(): + result = ThreeJSWorld().execute({ + "operation": "validate", + "world_spec": _world_spec(), + "duration_seconds": 12, + "quality_tier": "production", + "asset_catalog_paths": [], + }) + assert not result.success + assert "asset catalog" in (result.error or "").lower() + assert "asset-palette" in (result.error or "").lower() + assert "terrain material" in (result.error or "").lower() + + +def test_blockout_tier_is_labeled_as_nonproduction(): + result = ThreeJSWorld().execute({ + "operation": "validate", + "world_spec": _world_spec(), + "duration_seconds": 12, + "quality_tier": "blockout", + }) + assert result.success + assert result.data["report"]["quality_tier"] == "blockout" + assert any("do not present" in warning.lower() for warning in result.data["report"]["warnings"]) + + +def test_hyperframes_render_existing_preserves_authored_entry(tmp_path, monkeypatch): + workspace = tmp_path / "world" + workspace.mkdir() + entry = workspace / "index.html" + entry.write_text("
", encoding="utf-8") + tool = HyperFramesCompose() + monkeypatch.setattr(tool, "_runtime_check", lambda: {"runtime_available": True}) + monkeypatch.setattr(tool, "_check", lambda inputs: ToolResult(success=True, data={"ok": True})) + + def fake_run(args, *, cwd, timeout, check): + output = Path(args[args.index("--output") + 1]) + output.parent.mkdir(parents=True, exist_ok=True) + output.write_bytes(b"rendered") + return subprocess.CompletedProcess(args, 0, "", "") + + monkeypatch.setattr(tool, "_run_hf", fake_run) + output = tmp_path / "renders" / "final.mp4" + result = tool.execute( + { + "operation": "render_existing", + "workspace_path": str(workspace), + "output_path": str(output), + "quality": "draft", + } + ) + assert result.success, result.error + assert result.data["authored_entry_preserved"] is True + assert entry.read_text(encoding="utf-8") == "
" + assert output.is_file() + + +def test_video_compose_routes_empty_cut_atelier_to_existing_workspace(tmp_path, monkeypatch): + captured = {} + output = tmp_path / "final.mp4" + + def fake_hyperframes_execute(self, inputs): + captured.update(inputs) + Path(inputs["output_path"]).write_bytes(b"fake mp4") + return ToolResult(success=True, data={"output": inputs["output_path"]}) + + monkeypatch.setattr(VideoCompose, "_hyperframes_available", lambda self: True) + monkeypatch.setattr(HyperFramesCompose, "execute", fake_hyperframes_execute) + monkeypatch.setattr( + VideoCompose, + "_run_final_review", + lambda self, *args, **kwargs: {"status": "pass", "issues_found": []}, + ) + + result = VideoCompose().execute( + { + "operation": "render", + "workspace_path": str(tmp_path / "world"), + "output_path": str(output), + "edit_decisions": { + "version": "1.0", + "cuts": [], + "render_runtime": "hyperframes", + "renderer_family": "bespoke", + "composition_mode": "atelier", + "bespoke": {"entry": "index.html"}, + }, + } + ) + assert result.success, result.error + assert captured["operation"] == "render_existing" + assert captured["asset_manifest"] == {"version": "1.0", "assets": []} + assert captured["edit_decisions"]["cuts"] == [] diff --git a/tests/tools/test_transcriber_device_selection.py b/tests/tools/test_transcriber_device_selection.py new file mode 100644 index 00000000..11d50012 --- /dev/null +++ b/tests/tools/test_transcriber_device_selection.py @@ -0,0 +1,83 @@ +import sys +from types import SimpleNamespace + +from tools.analysis.transcriber import Transcriber + + +class _Info: + language = "en" + duration = 1.0 + + +def test_transcriber_uses_ctranslate2_cuda_without_torch(monkeypatch, tmp_path) -> None: + devices = [] + + class FakeWhisperModel: + def __init__(self, model_size, *, device, compute_type): + devices.append((device, compute_type)) + + def transcribe(self, *args, **kwargs): + return iter(()), _Info() + + monkeypatch.setitem( + sys.modules, + "faster_whisper", + SimpleNamespace(WhisperModel=FakeWhisperModel), + ) + monkeypatch.setitem( + sys.modules, + "ctranslate2", + SimpleNamespace( + get_cuda_device_count=lambda: 1, + get_supported_compute_types=lambda device: {"float16", "float32"}, + ), + ) + input_path = tmp_path / "audio.wav" + input_path.write_bytes(b"fake") + + result = Transcriber().execute({"input_path": str(input_path), "output_dir": str(tmp_path)}) + + assert result.success, result.error + assert devices == [("cuda", "float16")] + assert result.data["device"] == "cuda" + + +def test_transcriber_falls_back_when_cuda_fails_during_iteration(monkeypatch, tmp_path) -> None: + devices = [] + + class FakeWhisperModel: + def __init__(self, model_size, *, device, compute_type): + self.device = device + devices.append((device, compute_type)) + + def transcribe(self, *args, **kwargs): + if self.device == "cuda": + def broken_iterator(): + raise RuntimeError("cublas64_12.dll not found") + yield + + return broken_iterator(), _Info() + return iter(()), _Info() + + monkeypatch.setitem( + sys.modules, + "faster_whisper", + SimpleNamespace(WhisperModel=FakeWhisperModel), + ) + monkeypatch.setitem( + sys.modules, + "ctranslate2", + SimpleNamespace( + get_cuda_device_count=lambda: 1, + get_supported_compute_types=lambda device: {"float16"}, + ), + ) + input_path = tmp_path / "audio.wav" + input_path.write_bytes(b"fake") + + result = Transcriber().execute({"input_path": str(input_path), "output_dir": str(tmp_path)}) + + assert result.success, result.error + assert devices == [("cuda", "float16"), ("cpu", "int8")] + assert result.data["device"] == "cpu" + assert "cublas64_12.dll" in result.data["gpu_fallback_reason"] diff --git a/tools/analysis/transcriber.py b/tools/analysis/transcriber.py index bb309fc5..8e384152 100644 --- a/tools/analysis/transcriber.py +++ b/tools/analysis/transcriber.py @@ -135,50 +135,76 @@ class Transcriber(BaseTool): start = time.time() - # Load model (CPU by default, CUDA if available) + # faster-whisper executes through CTranslate2, so that runtime—not + # PyTorch—is authoritative for CUDA availability and compute types. + device = "cpu" + compute_type = "int8" try: - import torch - device = "cuda" if torch.cuda.is_available() else "cpu" - compute_type = "float16" if device == "cuda" else "int8" - except ImportError: + import ctranslate2 + + if ctranslate2.get_cuda_device_count() > 0: + supported = ctranslate2.get_supported_compute_types("cuda") + for candidate in ("float16", "int8_float16", "float32"): + if candidate in supported: + device = "cuda" + compute_type = candidate + break + except Exception: + # Probing is advisory. CPU remains a safe deterministic baseline. + pass + + def _transcribe_on(selected_device: str, selected_compute_type: str): + model = WhisperModel( + model_size, + device=selected_device, + compute_type=selected_compute_type, + ) + segments_iter, transcription_info = model.transcribe( + str(input_path), + language=language, + word_timestamps=True, + vad_filter=True, + ) + + parsed_segments = [] + parsed_words = [] + # faster-whisper evaluates lazily. Draining the iterator here keeps + # missing CUDA runtime libraries inside the fallback boundary. + for seg in segments_iter: + seg_data = { + "id": seg.id, + "start": round(seg.start, 3), + "end": round(seg.end, 3), + "text": seg.text.strip(), + } + + if seg.words: + words = [] + for word in seg.words: + word_entry = { + "word": word.word, + "start": round(word.start, 3), + "end": round(word.end, 3), + "probability": round(word.probability, 3), + } + words.append(word_entry) + parsed_words.append(word_entry) + seg_data["words"] = words + + parsed_segments.append(seg_data) + + return parsed_segments, parsed_words, transcription_info + + gpu_fallback_reason = None + try: + segments, word_timestamps, info = _transcribe_on(device, compute_type) + except Exception as exc: + if device == "cpu": + raise + gpu_fallback_reason = f"{type(exc).__name__}: {exc}" device = "cpu" compute_type = "int8" - - model = WhisperModel(model_size, device=device, compute_type=compute_type) - - # Transcribe - segments_iter, info = model.transcribe( - str(input_path), - language=language, - word_timestamps=True, - vad_filter=True, - ) - - segments = [] - word_timestamps = [] - - for seg in segments_iter: - seg_data = { - "id": seg.id, - "start": round(seg.start, 3), - "end": round(seg.end, 3), - "text": seg.text.strip(), - } - - if seg.words: - words = [] - for w in seg.words: - word_entry = { - "word": w.word, - "start": round(w.start, 3), - "end": round(w.end, 3), - "probability": round(w.probability, 3), - } - words.append(word_entry) - word_timestamps.append(word_entry) - seg_data["words"] = words - - segments.append(seg_data) + segments, word_timestamps, info = _transcribe_on(device, compute_type) detected_language = language or info.language duration = info.duration @@ -198,6 +224,8 @@ class Transcriber(BaseTool): "duration_seconds": round(duration, 3), "model_size": model_size, "device": device, + "compute_type": compute_type, + "gpu_fallback_reason": gpu_fallback_reason, } # Write transcript JSON diff --git a/tools/audio/audio_mixer.py b/tools/audio/audio_mixer.py index f3520236..48dfcb6a 100644 --- a/tools/audio/audio_mixer.py +++ b/tools/audio/audio_mixer.py @@ -184,6 +184,14 @@ class AudioMixer(BaseTool): "default": 0.5, "description": "Duration of fade in/out at segment boundaries (seconds).", }, + "target_duration": { + "type": "number", + "exclusiveMinimum": 0, + "description": ( + "full_mix only. Exact output length in seconds. Pads a short " + "mix and trims a long mix so audio matches the composition." + ), + }, }, } @@ -213,6 +221,39 @@ class AudioMixer(BaseTool): target = max(-40.0, min(0.0, target)) return f"[{in_label}]loudnorm=I={target}:LRA=11:TP=-1.5[{out_label}]" + def _track_filters(self, track: dict[str, Any]) -> list[str]: + """Build per-track filters on the source timeline before scheduling it. + + ``afade=t=out`` defaults to ``st=0``. Applying it after ``adelay`` + therefore fades the delay silence instead of the source audio, leaving + a delayed track silent by the time it starts. Fade source samples first + and add the timeline delay last so both fades follow the track itself. + """ + filters = [] + volume = track.get("volume", 1.0) + delay_ms = int(track.get("start_seconds", 0) * 1000) + fade_in = track.get("fade_in_seconds", 0) + fade_out = track.get("fade_out_seconds", 0) + + if volume != 1.0: + filters.append(f"volume={volume}") + if fade_in > 0: + filters.append(f"afade=t=in:d={fade_in}") + if fade_out > 0: + duration_cmd = [ + "ffprobe", "-v", "error", + "-show_entries", "format=duration", + "-of", "csv=p=0", + track["path"], + ] + duration = float(self.run_command(duration_cmd).stdout.strip().split("\n")[0]) + fade_start = max(0.0, duration - float(fade_out)) + filters.append(f"afade=t=out:st={fade_start}:d={fade_out}") + if delay_ms > 0: + filters.append(f"adelay={delay_ms}|{delay_ms}") + + return filters + def execute(self, inputs: dict[str, Any]) -> ToolResult: operation = inputs["operation"] start = time.time() @@ -256,20 +297,7 @@ class AudioMixer(BaseTool): for i, track in enumerate(tracks): input_args.extend(["-i", track["path"]]) - volume = track.get("volume", 1.0) - delay_ms = int(track.get("start_seconds", 0) * 1000) - fade_in = track.get("fade_in_seconds", 0) - fade_out = track.get("fade_out_seconds", 0) - - filters = [] - if volume != 1.0: - filters.append(f"volume={volume}") - if delay_ms > 0: - filters.append(f"adelay={delay_ms}|{delay_ms}") - if fade_in > 0: - filters.append(f"afade=t=in:d={fade_in}") - if fade_out > 0: - filters.append(f"afade=t=out:d={fade_out}") + filters = self._track_filters(track) if filters: filter_chain = ",".join(filters) @@ -480,6 +508,15 @@ class AudioMixer(BaseTool): output_path.parent.mkdir(parents=True, exist_ok=True) normalize = inputs.get("normalize", True) ducking = inputs.get("ducking", {"enabled": True}) + target_duration = inputs.get("target_duration") + target: float | None = None + if target_duration is not None: + try: + target = float(target_duration) + except (TypeError, ValueError): + return ToolResult(success=False, error="target_duration must be a positive number") + if target <= 0: + return ToolResult(success=False, error="target_duration must be greater than zero") speech_tracks = [t for t in tracks if t.get("role") in ("speech", "primary")] music_tracks = [t for t in tracks if t.get("role") in ("music", "secondary")] @@ -500,20 +537,7 @@ class AudioMixer(BaseTool): for i, track in enumerate(all_tracks): input_args.extend(["-i", track["path"]]) - volume = track.get("volume", 1.0) - delay_ms = int(track.get("start_seconds", 0) * 1000) - fade_in = track.get("fade_in_seconds", 0) - fade_out = track.get("fade_out_seconds", 0) - - filters = [] - if volume != 1.0: - filters.append(f"volume={volume}") - if delay_ms > 0: - filters.append(f"adelay={delay_ms}|{delay_ms}") - if fade_in > 0: - filters.append(f"afade=t=in:d={fade_in}") - if fade_out > 0: - filters.append(f"afade=t=out:d={fade_out}") + filters = self._track_filters(track) if filters: filter_chain = ",".join(filters) @@ -540,7 +564,14 @@ class AudioMixer(BaseTool): ) else: filter_parts.append(f"[a{speech_indices[0]}]acopy[speech_all]") - filter_parts.append("[speech_all]asplit=2[speech_key][speech_out]") + if target is not None: + filter_parts.append("[speech_all]asplit=2[speech_key_raw][speech_out]") + filter_parts.append( + f"[speech_key_raw]apad=whole_dur={target}," + f"atrim=duration={target},asetpts=PTS-STARTPTS[speech_key]" + ) + else: + filter_parts.append("[speech_all]asplit=2[speech_key][speech_out]") # Mix music tracks together music_start = len(speech_tracks) @@ -589,19 +620,34 @@ class AudioMixer(BaseTool): f"{all_labels}amix=inputs={len(all_tracks)}:duration=longest:dropout_transition=2[premix]" ) + # A ducked music stream is gated by the speech sidechain, so its tail + # can disappear when narration ends. If the caller knows the video + # duration, make that the authoritative mix length before loudness + # normalization: apad extends short audio and atrim caps long audio. + premix_label = "premix" + if target is not None: + filter_parts.append( + f"[premix]apad=whole_dur={target},atrim=duration={target}," + "asetpts=PTS-STARTPTS[premix_duration]" + ) + premix_label = "premix_duration" + # Normalize if normalize: - filter_parts.append(self._loudnorm_filter(inputs, "premix", "out")) + filter_parts.append(self._loudnorm_filter(inputs, premix_label, "out")) out_label = "[out]" else: - out_label = "[premix]" + out_label = f"[{premix_label}]" filter_complex = ";".join(p for p in filter_parts if p) cmd = ["ffmpeg", "-y"] cmd.extend(input_args) cmd.extend(["-filter_complex", filter_complex]) - cmd.extend(["-map", out_label, str(output_path)]) + cmd.extend(["-map", out_label]) + if target is not None: + cmd.extend(["-t", str(target)]) + cmd.append(str(output_path)) self.run_command(cmd) @@ -614,6 +660,7 @@ class AudioMixer(BaseTool): "sfx_tracks": len(sfx_tracks), "ducking_enabled": duck_enabled, "normalized": normalize, + "target_duration": target_duration, "output": str(output_path), }, artifacts=[str(output_path)], diff --git a/tools/audio/google_music.py b/tools/audio/google_music.py index defb56fd..67f25d87 100644 --- a/tools/audio/google_music.py +++ b/tools/audio/google_music.py @@ -44,7 +44,7 @@ class GoogleMusic(BaseTool): " - Or set GOOGLE_APPLICATION_CREDENTIALS for Vertex AI service account." ) fallback_tools = ["music_gen"] - agent_skills = ["music"] + agent_skills = ["lyria"] capabilities = [ "generate_background_music", @@ -146,7 +146,8 @@ class GoogleMusic(BaseTool): from tools.google_credentials import get_genai_client, GOOGLE_API_TIMEOUT_MS http_options = types.HttpOptions(timeout=GOOGLE_API_TIMEOUT_MS) - client = get_genai_client(http_options=http_options) + # Lyria 3 is served only from Vertex's global location. + client = get_genai_client(http_options=http_options, location="global") except ImportError as e: return ToolResult( success=False, diff --git a/tools/enhancement/bg_remove.py b/tools/enhancement/bg_remove.py index 7802f45a..7d1c40e8 100644 --- a/tools/enhancement/bg_remove.py +++ b/tools/enhancement/bg_remove.py @@ -130,7 +130,7 @@ class BgRemove(BaseTool): result_image = rembg.remove( input_image, - model_name=model_name, + session=rembg.new_session(model_name), alpha_matting=alpha_matting, ) diff --git a/tools/enhancement/upscale.py b/tools/enhancement/upscale.py index 03648405..1fa18ccd 100644 --- a/tools/enhancement/upscale.py +++ b/tools/enhancement/upscale.py @@ -297,6 +297,11 @@ class Upscale(BaseTool): "model": model, "dni_weight": denoise_strength, "half": half, + # Full-frame x4 inference can terminate the process on low-memory + # CPU/MPS hosts before Python can raise an exception. Bound the + # working set there; keep CUDA on the faster single-pass path. + "tile": 0 if _device == "cuda" else 256, + "tile_pad": 10, } # Guard: only pass device= if the installed version accepts it if "device" in inspect.signature(RealESRGANer.__init__).parameters: diff --git a/tools/google_credentials.py b/tools/google_credentials.py index 71158c73..77565c2e 100644 --- a/tools/google_credentials.py +++ b/tools/google_credentials.py @@ -18,6 +18,12 @@ from typing import Any # Broad scope that covers Cloud Text-to-Speech and Vertex AI prediction. CLOUD_PLATFORM_SCOPE = "https://www.googleapis.com/auth/cloud-platform" + +def resolve_google_location(location: str | None = None) -> str: + """Return a Vertex location, treating blank env values as unset.""" + + return location or os.environ.get("GOOGLE_CLOUD_LOCATION") or "us-central1" + # Shared constants for long-running Google/Vertex AI generation calls (e.g. music, video) GOOGLE_API_TIMEOUT_SECONDS = 600 GOOGLE_API_TIMEOUT_MS = GOOGLE_API_TIMEOUT_SECONDS * 1000 @@ -38,8 +44,15 @@ def has_google_credentials() -> bool: ) -def get_genai_client(http_options: Any | None = None) -> Any: - """Lazily import and initialize the Google GenAI Client based on configured credentials.""" +def get_genai_client( + http_options: Any | None = None, + location: str | None = None, +) -> Any: + """Initialize Google GenAI using the configured credential mode. + + ``location`` overrides the Vertex region for globally hosted models. It is + deliberately ignored by the API-key backend, which has no region setting. + """ from google import genai api_key = os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY") @@ -51,7 +64,7 @@ def get_genai_client(http_options: Any | None = None) -> Any: if use_vertex or (not api_key and service_account_configured()): kwargs = { "vertexai": True, - "location": os.environ.get("GOOGLE_CLOUD_LOCATION", "us-central1"), + "location": resolve_google_location(location), "http_options": http_options, } project_id = resolve_project_id() diff --git a/tools/graphics/atlas_3d.py b/tools/graphics/atlas_3d.py new file mode 100644 index 00000000..8178058c --- /dev/null +++ b/tools/graphics/atlas_3d.py @@ -0,0 +1,227 @@ +"""Text-to-3D asset generation through Atlas Cloud. + +The tool deliberately exposes mesh generation as its own capability. Atlas's +HTTP endpoint happens to be named ``generateImage`` for historical reasons; +that implementation detail must not make 3D assets look like image outputs to +the OpenMontage registry or pipeline. +""" + +from __future__ import annotations + +import json +import os +import time +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + +import requests + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ResourceProfile, + RetryPolicy, + ToolResult, + ToolRuntime, + ToolStability, + ToolStatus, + ToolTier, +) + + +_MODEL = "tripo-h3.1/text-to-3d" +_ENV_KEYS = ("ATLASCLOUD_API_KEY", "ATLAS_CLOUD_API_KEY", "ATLAS_API_KEY") + + +def _api_key() -> str | None: + return next((os.environ.get(name) for name in _ENV_KEYS if os.environ.get(name)), None) + + +def _extension(url: str, content_type: str | None, fallback: str = ".glb") -> str: + suffix = Path(urlparse(url).path).suffix.lower() + if suffix in {".glb", ".gltf", ".fbx", ".obj", ".zip"}: + return suffix + if content_type == "model/gltf-binary": + return ".glb" + return fallback + + +class Atlas3D(BaseTool): + name = "atlas_3d" + version = "0.1.0" + tier = ToolTier.GENERATE + capability = "3d_asset_generation" + provider = "atlas_cloud" + stability = ToolStability.BETA + execution_mode = ExecutionMode.ASYNC + determinism = Determinism.SEEDED + runtime = ToolRuntime.API + dependencies = ["env:ATLASCLOUD_API_KEY"] + install_instructions = ( + "Set ATLASCLOUD_API_KEY (ATLAS_CLOUD_API_KEY and ATLAS_API_KEY are also accepted). " + "Create a key at https://www.atlascloud.ai/." + ) + agent_skills = ["3d-asset-generation", "threejs-loaders", "threejs-materials"] + capabilities = ["text_to_3d", "textured_glb", "pbr_mesh", "seeded_mesh_generation"] + supports = { + "text_to_3d": True, + "texture": True, + "pbr": True, + "detailed_geometry": True, + "face_limit": True, + "seed": True, + "glb": True, + } + best_for = [ + "Unique hero props and environment pieces described in text", + "Textured PBR GLB assets for Blender or Three.js", + ] + not_good_for = [ + "Whole coherent worlds in one request", + "Repeated foliage or rocks that should come from a licensed local catalog", + ] + input_schema = { + "type": "object", + "required": ["prompt", "output_path"], + "properties": { + "prompt": {"type": "string", "minLength": 3, "maxLength": 1024}, + "negative_prompt": {"type": "string"}, + "output_path": {"type": "string"}, + "texture": {"type": "boolean", "default": True}, + "pbr": {"type": "boolean", "default": True}, + "texture_quality": {"type": "string", "enum": ["standard", "detailed"], "default": "standard"}, + "geometry_quality": {"type": "string", "enum": ["standard", "detailed"], "default": "standard"}, + "face_limit": {"type": "integer", "minimum": 1000, "maximum": 2000000}, + "model_seed": {"type": "integer"}, + "image_seed": {"type": "integer"}, + "texture_seed": {"type": "integer"}, + "auto_size": {"type": "boolean", "default": True}, + "quad": {"type": "boolean", "default": False}, + "poll_timeout_seconds": {"type": "integer", "minimum": 30, "maximum": 1800, "default": 900}, + }, + } + output_schema = {"type": "object"} + artifact_schema = {"artifact": "3d_asset"} + resource_profile = ResourceProfile(cpu_cores=1, ram_mb=512, disk_mb=1000, network_required=True) + retry_policy = RetryPolicy(max_retries=1, retryable_errors=["rate_limit", "timeout"]) + idempotency_key_fields = [ + "prompt", "negative_prompt", "texture", "pbr", "texture_quality", + "geometry_quality", "face_limit", "model_seed", "image_seed", "texture_seed", + ] + side_effects = ["calls the Atlas Cloud API", "writes a generated mesh and provenance manifest"] + user_visible_verification = [ + "Inspect the downloaded mesh from front, back, silhouette, UV, and PBR material views before scene assembly" + ] + quality_score = 0.86 + + def get_status(self) -> ToolStatus: + return ToolStatus.AVAILABLE if _api_key() else ToolStatus.UNAVAILABLE + + def estimate_cost(self, inputs: dict[str, Any]) -> float: + texture = bool(inputs.get("texture", True)) + texture_quality = inputs.get("texture_quality", "standard") + cost = 0.22 if not texture else (0.44 if texture_quality == "detailed" else 0.33) + if inputs.get("geometry_quality", "standard") == "detailed": + cost += 0.22 + if inputs.get("quad", False): + cost += 0.055 + return round(cost, 3) + + def execute(self, inputs: dict[str, Any]) -> ToolResult: + key = _api_key() + if not key: + return ToolResult(success=False, error="Atlas Cloud API key not set. " + self.install_instructions) + + output = Path(str(inputs["output_path"])).expanduser().resolve() + output.parent.mkdir(parents=True, exist_ok=True) + payload: dict[str, Any] = { + "model": _MODEL, + "prompt": inputs["prompt"], + "texture": bool(inputs.get("texture", True)), + "pbr": bool(inputs.get("pbr", True)), + "texture_quality": inputs.get("texture_quality", "standard"), + "geometry_quality": inputs.get("geometry_quality", "standard"), + "auto_size": bool(inputs.get("auto_size", True)), + "quad": bool(inputs.get("quad", False)), + } + for key_name in ("negative_prompt", "face_limit", "model_seed", "image_seed", "texture_seed"): + if inputs.get(key_name) is not None: + payload[key_name] = inputs[key_name] + + headers = {"Authorization": f"Bearer {key}", "Content-Type": "application/json"} + started = time.time() + try: + submit = requests.post( + "https://api.atlascloud.ai/api/v1/model/generateImage", + headers=headers, + json=payload, + timeout=45, + ) + submit.raise_for_status() + prediction = submit.json()["data"] + prediction_id = prediction["id"] + deadline = time.monotonic() + int(inputs.get("poll_timeout_seconds", 900)) + while time.monotonic() < deadline: + poll = requests.get( + f"https://api.atlascloud.ai/api/v1/model/prediction/{prediction_id}", + headers=headers, + timeout=30, + ) + poll.raise_for_status() + prediction = poll.json().get("data", poll.json()) + status = str(prediction.get("status", "")).lower() + if status in {"completed", "succeeded"}: + break + if status in {"failed", "cancelled"}: + raise RuntimeError(str(prediction.get("error") or f"prediction {status}")) + time.sleep(3) + else: + raise TimeoutError(f"Prediction {prediction_id} exceeded the poll timeout") + + files = list(prediction.get("files") or []) + mesh_file = next( + (item for item in files if str(item.get("type", "")).lower() == "glb"), + None, + ) or next( + (item for item in files if _extension(str(item.get("url", "")), item.get("content_type")) == ".glb"), + None, + ) + if mesh_file is None: + outputs = [url for url in prediction.get("outputs") or [] if isinstance(url, str)] + mesh_url = next((url for url in outputs if Path(urlparse(url).path).suffix.lower() == ".glb"), None) + if mesh_url is None: + raise RuntimeError("Atlas prediction completed without a GLB output") + mesh_file = {"url": mesh_url, "content_type": "model/gltf-binary"} + + mesh_url = str(mesh_file["url"]) + if output.suffix.lower() != ".glb": + output = output.with_suffix(_extension(mesh_url, mesh_file.get("content_type"))) + download = requests.get(mesh_url, timeout=180) + download.raise_for_status() + output.write_bytes(download.content) + + manifest = output.with_suffix(".provenance.json") + manifest.write_text(json.dumps({ + "version": "1.0", + "provider": "atlas_cloud", + "model": _MODEL, + "prediction_id": prediction_id, + "prompt": inputs["prompt"], + "parameters": {key: value for key, value in payload.items() if key != "prompt"}, + "source_url": "https://www.atlascloud.ai/models/tripo-h3.1/text-to-3d", + "output": str(output), + }, indent=2), encoding="utf-8") + except Exception as exc: + return ToolResult(success=False, error=f"Atlas Cloud 3D generation failed: {exc}") + + return ToolResult( + success=True, + data={"provider": "atlas_cloud", "model": _MODEL, "output": str(output), "prediction_id": prediction_id}, + artifacts=[str(output), str(manifest)], + cost_usd=self.estimate_cost(inputs), + duration_seconds=round(time.time() - started, 2), + seed=inputs.get("model_seed"), + model=_MODEL, + ) diff --git a/tools/graphics/blender_world.py b/tools/graphics/blender_world.py new file mode 100644 index 00000000..a3b930a6 --- /dev/null +++ b/tools/graphics/blender_world.py @@ -0,0 +1,241 @@ +"""Deterministic Blender assembly and rendering for production 3D worlds.""" + +from __future__ import annotations + +import json +import os +import shutil +import subprocess +import time +from pathlib import Path +from typing import Any + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ResourceProfile, + ToolResult, + ToolRuntime, + ToolStability, + ToolStatus, + ToolTier, +) + + +_REPO_ROOT = Path(__file__).resolve().parents[2] +_PORTABLE_BLENDER = ( + _REPO_ROOT / ".runtime" / "blender" / "blender-4.5.10-windows-x64" / "blender.exe" +) +_RUNTIME_SCRIPT = Path(__file__).resolve().parent / "templates" / "blender-world-runtime.py" + + +def find_blender() -> Path | None: + configured = os.environ.get("BLENDER_PATH") + if configured and Path(configured).is_file(): + return Path(configured).resolve() + if _PORTABLE_BLENDER.is_file(): + return _PORTABLE_BLENDER.resolve() + discovered = shutil.which("blender") + return Path(discovered).resolve() if discovered else None + + +def first_missing_frame(output_prefix: str | Path, start_frame: int, end_frame: int) -> int | None: + """Return the first missing PNG in a contiguous Blender image sequence.""" + prefix = Path(output_prefix).expanduser().resolve() + for frame in range(start_frame, end_frame + 1): + candidate = prefix.parent / f"{prefix.name}{frame:04d}.png" + if not candidate.is_file(): + return frame + return None + + +class BlenderWorld(BaseTool): + name = "blender_world" + version = "0.3.0" + tier = ToolTier.GENERATE + capability = "3d_world_rendering" + provider = "blender" + stability = ToolStability.BETA + execution_mode = ExecutionMode.SYNC + determinism = Determinism.SEEDED + runtime = ToolRuntime.LOCAL_GPU + dependencies: list[str] = [] + install_instructions = ( + "Install Blender 4.5 LTS, set BLENDER_PATH, or place the portable runtime at " + ".runtime/blender/blender-4.5.10-windows-x64/blender.exe." + ) + agent_skills = [ + "3d-asset-generation", "threejs-world-generation", "threejs-loaders", "threejs-materials", + "threejs-textures", "threejs-lighting", "threejs-postprocessing", + ] + capabilities = [ + "gltf_glb_scene_assembly", "procedural_terrain", "linked_asset_scatter", + "pbr_materials", "eevee_next_render", "camera_flythrough", "blend_project_export", + "asset_unit_normalization", "bounding_box_ground_contact", "semantic_scatter_exclusions", + "terrain_following_ribbons", "visibility_windows", "title_safe_final_hold", + ] + supports = { + "glb": True, + "gltf": True, + "pbr": True, + "linked_instances": True, + "still": True, + "animation": True, + "transparent_background": True, + } + best_for = [ + "Production-quality world assembly from many generated and licensed assets", + "Dense terrain, lighting, material, camera, and contact-shadow work", + "Rendering a final image sequence for governed video composition", + ] + not_good_for = ["Interactive browser delivery", "Text-to-mesh generation"] + input_schema = { + "type": "object", + "required": ["operation"], + "properties": { + "operation": {"type": "string", "enum": ["doctor", "build", "render_still", "render_animation"]}, + "world_spec": {"type": "object"}, + "output_path": {"type": "string"}, + "blend_path": {"type": "string"}, + "width": {"type": "integer", "minimum": 320, "maximum": 7680, "default": 1920}, + "height": {"type": "integer", "minimum": 240, "maximum": 4320, "default": 1080}, + "samples": {"type": "integer", "minimum": 1, "maximum": 256, "default": 32}, + "fps": {"type": "integer", "minimum": 1, "maximum": 120, "default": 30}, + "duration_seconds": {"type": "number", "minimum": 1, "maximum": 600, "default": 60}, + "start_frame": {"type": "integer", "minimum": 1}, + "end_frame": {"type": "integer", "minimum": 1}, + "frame": {"type": "integer", "minimum": 1}, + "resume": {"type": "boolean", "default": False}, + }, + } + output_schema = {"type": "object"} + artifact_schema = {"artifact": "3d_world"} + resource_profile = ResourceProfile(cpu_cores=8, ram_mb=8192, vram_mb=6000, disk_mb=20000) + idempotency_key_fields = ["operation", "world_spec", "width", "height", "samples", "fps", "duration_seconds"] + side_effects = ["writes a .blend project", "may render an image or PNG sequence"] + user_visible_verification = [ + "Review global, regional, and walk-height stills before an animation render", + "Check imported mesh scale, ground contact, texture color space, shadowing, and camera clearance", + ] + quality_score = 0.94 + + def get_status(self) -> ToolStatus: + return ToolStatus.AVAILABLE if find_blender() and _RUNTIME_SCRIPT.is_file() else ToolStatus.UNAVAILABLE + + def estimate_runtime(self, inputs: dict[str, Any]) -> float: + if inputs.get("operation") == "render_animation": + return float(inputs.get("duration_seconds", 60)) * float(inputs.get("fps", 30)) * 2.0 + return 30.0 + + def execute(self, inputs: dict[str, Any]) -> ToolResult: + blender = find_blender() + if not blender: + return ToolResult(success=False, error="Blender not found. " + self.install_instructions) + operation = str(inputs.get("operation") or "") + if operation == "doctor": + process = subprocess.run( + [str(blender), "--background", "--python-expr", "import bpy; print('OPENMONTAGE_BLENDER=' + bpy.app.version_string)"], + capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=60, + ) + ok = process.returncode == 0 and "OPENMONTAGE_BLENDER=" in process.stdout + return ToolResult( + success=ok, + data={"blender_path": str(blender), "version_line": next((line for line in process.stdout.splitlines() if line.startswith("OPENMONTAGE_BLENDER=")), "")}, + error=None if ok else (process.stderr or process.stdout)[-1000:], + model="blender-4.5-lts", + ) + + if operation not in {"build", "render_still", "render_animation"}: + return ToolResult(success=False, error=f"Unknown operation: {operation}") + if not isinstance(inputs.get("world_spec"), dict): + return ToolResult(success=False, error="world_spec is required") + output_raw = inputs.get("output_path") + if operation != "build" and not output_raw: + return ToolResult(success=False, error="output_path is required for rendering") + blend_raw = inputs.get("blend_path") or ( + str(Path(str(output_raw)).with_suffix(".blend")) if output_raw else "blender-world.blend" + ) + blend_path = Path(str(blend_raw)).expanduser().resolve() + blend_path.parent.mkdir(parents=True, exist_ok=True) + spec_path = blend_path.with_suffix(".world.json") + spec_path.write_text(json.dumps(inputs["world_spec"], indent=2), encoding="utf-8") + + command = [ + str(blender), "--background", "--python", str(_RUNTIME_SCRIPT), "--", + "--operation", operation, + "--spec", str(spec_path), + "--blend", str(blend_path), + "--width", str(int(inputs.get("width", 1920))), + "--height", str(int(inputs.get("height", 1080))), + "--samples", str(int(inputs.get("samples", 32))), + "--fps", str(int(inputs.get("fps", 30))), + "--duration", str(float(inputs.get("duration_seconds", 60))), + ] + requested_start = int(inputs.get("start_frame", 1)) + requested_end = int(inputs.get("end_frame") or round( + float(inputs.get("duration_seconds", 60)) * int(inputs.get("fps", 30)) + )) + effective_start = requested_start + if operation == "render_animation" and inputs.get("resume"): + if not output_raw: + return ToolResult(success=False, error="output_path is required to resume a render") + missing = first_missing_frame(output_raw, requested_start, requested_end) + if missing is None: + return ToolResult( + success=True, + data={ + "blender_path": str(blender), + "blend_path": str(blend_path), + "output": str(output_raw), + "already_complete": True, + "start_frame": requested_start, + "end_frame": requested_end, + }, + model="blender-4.5-lts-eevee-next", + ) + effective_start = missing + if inputs.get("start_frame") is not None or operation == "render_animation": + command.extend(["--start-frame", str(effective_start)]) + if inputs.get("end_frame") is not None or operation == "render_animation": + command.extend(["--end-frame", str(requested_end)]) + if inputs.get("frame") is not None: + command.extend(["--frame", str(int(inputs["frame"]))]) + if output_raw: + command.extend(["--output", str(Path(str(output_raw)).expanduser().resolve())]) + + started = time.time() + try: + process = subprocess.run( + command, capture_output=True, text=True, encoding="utf-8", errors="replace", + timeout=max(120, int(self.estimate_runtime(inputs) * 2.5)), + ) + except Exception as exc: + return ToolResult(success=False, error=f"Blender invocation failed: {exc}") + if process.returncode != 0: + return ToolResult(success=False, error="Blender world build failed: " + (process.stderr or process.stdout)[-3000:]) + + artifacts = [str(spec_path), str(blend_path)] + if output_raw: + output = Path(str(output_raw)).expanduser().resolve() + if output.exists(): + artifacts.append(str(output)) + report_path = blend_path.with_suffix(".report.json") + if report_path.exists(): + artifacts.append(str(report_path)) + return ToolResult( + success=True, + data={ + "blender_path": str(blender), + "blend_path": str(blend_path), + "output": str(output_raw or ""), + "report": str(report_path), + "start_frame": effective_start if operation == "render_animation" else None, + "end_frame": requested_end if operation == "render_animation" else None, + "resumed": bool(operation == "render_animation" and inputs.get("resume") and effective_start > requested_start), + }, + artifacts=artifacts, + duration_seconds=round(time.time() - started, 2), + seed=inputs["world_spec"].get("seed"), + model="blender-4.5-lts-eevee-next", + ) diff --git a/tools/graphics/fal_3d.py b/tools/graphics/fal_3d.py new file mode 100644 index 00000000..854aa328 --- /dev/null +++ b/tools/graphics/fal_3d.py @@ -0,0 +1,213 @@ +"""Text/image-to-3D and object reconstruction through fal.ai.""" + +from __future__ import annotations + +import json +import os +import time +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + +import requests + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ResourceProfile, + RetryPolicy, + ToolResult, + ToolRuntime, + ToolStability, + ToolStatus, + ToolTier, +) + + +_MODELS = { + "text_to_3d": "fal-ai/hunyuan-3d/v3.1/rapid/text-to-3d", + "image_to_3d": "fal-ai/hunyuan-3d/v3.1/rapid/image-to-3d", + "reconstruct_objects": "fal-ai/sam-3/3d-objects", +} + + +def _api_key() -> str | None: + return os.environ.get("FAL_KEY") or os.environ.get("FAL_AI_API_KEY") + + +def _download_file(file_info: dict[str, Any], destination: Path) -> Path: + url = str(file_info["url"]) + suffix = Path(urlparse(url).path).suffix.lower() + if suffix not in {".glb", ".gltf", ".obj", ".fbx", ".ply", ".zip"}: + suffix = ".glb" if file_info.get("content_type") == "model/gltf-binary" else destination.suffix + target = destination.with_suffix(suffix or ".glb") + response = requests.get(url, timeout=180) + response.raise_for_status() + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(response.content) + return target + + +class Fal3D(BaseTool): + name = "fal_3d" + version = "0.1.0" + tier = ToolTier.GENERATE + capability = "3d_asset_generation" + provider = "fal" + stability = ToolStability.BETA + execution_mode = ExecutionMode.ASYNC + determinism = Determinism.SEEDED + runtime = ToolRuntime.API + dependencies = ["env:FAL_KEY"] + install_instructions = "Set FAL_KEY (or FAL_AI_API_KEY). Create a key at https://fal.ai/dashboard/keys." + agent_skills = ["3d-asset-generation", "threejs-loaders", "threejs-materials"] + capabilities = ["text_to_3d", "image_to_3d", "multi_object_reconstruction", "textured_glb", "pbr_mesh"] + supports = { + "text_to_3d": True, + "image_to_3d": True, + "multi_object": True, + "pbr": True, + "glb": True, + "seed": True, + } + best_for = [ + "Image-conditioned hero props whose silhouette must match concept art", + "Extracting multiple textured GLBs and placements from a regional concept image", + "Rapid textured environment assets", + ] + not_good_for = ["Rendering a complete cinematic world", "Large repeated scatter libraries"] + input_schema = { + "type": "object", + "required": ["operation", "output_path"], + "properties": { + "operation": {"type": "string", "enum": list(_MODELS)}, + "prompt": {"type": "string"}, + "image_url": {"type": "string"}, + "image_path": {"type": "string"}, + "output_path": {"type": "string"}, + "enable_pbr": {"type": "boolean", "default": True}, + "seed": {"type": "integer"}, + "export_textured_glb": {"type": "boolean", "default": True}, + "detection_threshold": {"type": "number", "minimum": 0.1, "maximum": 1.0}, + "poll_timeout_seconds": {"type": "integer", "minimum": 30, "maximum": 1800, "default": 900}, + }, + } + output_schema = {"type": "object"} + artifact_schema = {"artifact": "3d_asset"} + resource_profile = ResourceProfile(cpu_cores=1, ram_mb=512, disk_mb=2000, network_required=True) + retry_policy = RetryPolicy(max_retries=1, retryable_errors=["rate_limit", "timeout"]) + idempotency_key_fields = ["operation", "prompt", "image_url", "image_path", "enable_pbr", "seed"] + side_effects = ["calls fal.ai", "may upload a local input image", "writes generated 3D assets and provenance"] + user_visible_verification = ["Inspect silhouette, back-side completion, topology, texture seams, and material response"] + quality_score = 0.88 + + def get_status(self) -> ToolStatus: + return ToolStatus.AVAILABLE if _api_key() else ToolStatus.UNAVAILABLE + + def estimate_cost(self, inputs: dict[str, Any]) -> float: + if inputs.get("operation") == "reconstruct_objects": + return 0.02 + return 0.225 + (0.15 if inputs.get("enable_pbr", True) else 0.0) + + def execute(self, inputs: dict[str, Any]) -> ToolResult: + key = _api_key() + if not key: + return ToolResult(success=False, error="fal.ai API key not set. " + self.install_instructions) + operation = str(inputs.get("operation") or "") + if operation not in _MODELS: + return ToolResult(success=False, error=f"Unknown operation {operation!r}") + if operation == "text_to_3d" and not inputs.get("prompt"): + return ToolResult(success=False, error="prompt is required for text_to_3d") + if operation != "text_to_3d" and not (inputs.get("image_url") or inputs.get("image_path")): + return ToolResult(success=False, error=f"image_url or image_path is required for {operation}") + + payload: dict[str, Any] = {} + if operation == "text_to_3d": + payload["prompt"] = inputs["prompt"] + payload["enable_pbr"] = bool(inputs.get("enable_pbr", True)) + else: + image_url = inputs.get("image_url") + if not image_url: + from tools.video._shared import upload_image_fal + image_url = upload_image_fal(str(inputs["image_path"])) + payload["image_url" if operation == "reconstruct_objects" else "input_image_url"] = image_url + if operation == "image_to_3d": + payload["enable_pbr"] = bool(inputs.get("enable_pbr", True)) + else: + payload["export_textured_glb"] = bool(inputs.get("export_textured_glb", True)) + if inputs.get("prompt"): + payload["prompt"] = inputs["prompt"] + if inputs.get("detection_threshold") is not None: + payload["detection_threshold"] = inputs["detection_threshold"] + if inputs.get("seed") is not None: + payload["seed"] = inputs["seed"] + + headers = {"Authorization": f"Key {key}", "Content-Type": "application/json"} + model = _MODELS[operation] + started = time.time() + try: + submit = requests.post(f"https://queue.fal.run/{model}", headers=headers, json=payload, timeout=45) + submit.raise_for_status() + queued = submit.json() + status_url = queued["status_url"] + response_url = queued["response_url"] + deadline = time.monotonic() + int(inputs.get("poll_timeout_seconds", 900)) + while time.monotonic() < deadline: + status_response = requests.get(status_url, headers=headers, timeout=30) + status_response.raise_for_status() + status = str(status_response.json().get("status", "")).upper() + if status == "COMPLETED": + break + if status in {"FAILED", "CANCELLED"}: + raise RuntimeError(f"request {status.lower()}") + time.sleep(3) + else: + raise TimeoutError("fal.ai request exceeded the poll timeout") + result_response = requests.get(response_url, headers=headers, timeout=45) + result_response.raise_for_status() + data = result_response.json() + + destination = Path(str(inputs["output_path"])).expanduser().resolve() + file_infos: list[dict[str, Any]] = [] + if operation == "reconstruct_objects": + if data.get("model_glb"): + file_infos.append(data["model_glb"]) + file_infos.extend(data.get("individual_glbs") or []) + else: + urls = data.get("model_urls") or {} + candidate = urls.get("glb") or data.get("model_glb") or urls.get("obj") + if candidate: + file_infos.append(candidate) + if not file_infos: + raise RuntimeError("fal.ai completed without a downloadable mesh") + + artifacts: list[str] = [] + for index, file_info in enumerate(file_infos): + target = destination if index == 0 else destination.with_name(f"{destination.stem}-{index:02d}{destination.suffix}") + artifacts.append(str(_download_file(file_info, target))) + provenance = destination.with_suffix(".provenance.json") + provenance.write_text(json.dumps({ + "version": "1.0", + "provider": "fal", + "model": model, + "request_id": queued.get("request_id"), + "operation": operation, + "prompt": inputs.get("prompt"), + "metadata": data.get("metadata"), + "source_url": f"https://fal.ai/models/{model}", + "outputs": artifacts, + }, indent=2), encoding="utf-8") + artifacts.append(str(provenance)) + except Exception as exc: + return ToolResult(success=False, error=f"fal.ai 3D generation failed: {exc}") + + return ToolResult( + success=True, + data={"provider": "fal", "model": model, "operation": operation, "outputs": artifacts[:-1]}, + artifacts=artifacts, + cost_usd=self.estimate_cost(inputs), + duration_seconds=round(time.time() - started, 2), + seed=inputs.get("seed"), + model=model, + ) diff --git a/tools/graphics/google_imagen.py b/tools/graphics/google_imagen.py index a222374c..2850a8e3 100644 --- a/tools/graphics/google_imagen.py +++ b/tools/graphics/google_imagen.py @@ -22,6 +22,7 @@ from tools.base_tool import ( ) from tools.google_credentials import ( get_access_token, + resolve_google_location, resolve_project_id, service_account_configured, has_google_credentials, @@ -145,6 +146,25 @@ class GoogleImagen(BaseTool): ] user_visible_verification = ["Inspect generated image for relevance and quality"] + @staticmethod + def _output_paths(output_path: str | None, count: int) -> list[Path]: + """Derive one output path per generated image. + + With a single image, honor the requested path as-is. With several, + suffix each with `_1`, `_2`, … so no image overwrites another. + """ + ext = ".png" + if not output_path: + return [Path(f"generated_image_{idx + 1}{ext}") for idx in range(count)] + + path = Path(output_path) + suffix = path.suffix or ext + if count == 1: + return [path if path.suffix else path.with_suffix(suffix)] + + base = path.with_suffix("") if path.suffix else path + return [base.parent / f"{base.name}_{idx + 1}{suffix}" for idx in range(count)] + def _get_api_key(self) -> str | None: return os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY") @@ -222,7 +242,7 @@ class GoogleImagen(BaseTool): } if bearer_token: - location = os.environ.get("GOOGLE_CLOUD_LOCATION", "us-central1") + location = resolve_google_location() url = ( f"https://{location}-aiplatform.googleapis.com/v1/projects/" f"{project_id}/locations/{location}/publishers/google/models/" @@ -261,11 +281,16 @@ class GoogleImagen(BaseTool): success=False, error="No images returned from Imagen API" ) - image_bytes = base64.b64decode(predictions[0]["bytesBase64Encoded"]) - - output_path = Path(inputs.get("output_path", "generated_image.png")) - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_bytes(image_bytes) + output_paths = self._output_paths( + inputs.get("output_path"), len(predictions) + ) + outputs: list[str] = [] + for prediction, out_path in zip(predictions, output_paths): + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_bytes( + base64.b64decode(prediction["bytesBase64Encoded"]) + ) + outputs.append(str(out_path)) except Exception as e: return ToolResult(success=False, error=f"Imagen generation failed: {e}") @@ -277,10 +302,11 @@ class GoogleImagen(BaseTool): "model": model, "prompt": prompt, "aspect_ratio": aspect_ratio, - "output": str(output_path), - "images_generated": len(predictions), + "output": outputs[0], + "outputs": outputs, + "images_generated": len(outputs), }, - artifacts=[str(output_path)], + artifacts=outputs, cost_usd=self.estimate_cost(inputs), duration_seconds=round(time.time() - start, 2), model=model, diff --git a/tools/graphics/templates/blender-world-runtime.py b/tools/graphics/templates/blender-world-runtime.py new file mode 100644 index 00000000..429908fa --- /dev/null +++ b/tools/graphics/templates/blender-world-runtime.py @@ -0,0 +1,434 @@ +"""Blender-side deterministic world builder used by ``blender_world``. + +No creative decisions live here: palette, density, asset choices, regions, +paths, water, camera, and lighting arrive in the JSON world specification. +""" + +from __future__ import annotations + +import argparse +import json +import math +import random +import sys +from pathlib import Path + +import bpy +from mathutils import Vector +from mathutils.noise import fractal, hetero_terrain, noise_vector, seed_set + + +def args_after_separator() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--operation", required=True) + parser.add_argument("--spec", required=True) + parser.add_argument("--blend", required=True) + parser.add_argument("--output", default="") + parser.add_argument("--width", type=int, default=1920) + parser.add_argument("--height", type=int, default=1080) + parser.add_argument("--samples", type=int, default=32) + parser.add_argument("--fps", type=int, default=30) + parser.add_argument("--duration", type=float, default=60.0) + parser.add_argument("--start-frame", type=int) + parser.add_argument("--end-frame", type=int) + parser.add_argument("--frame", type=int) + return parser.parse_args(sys.argv[sys.argv.index("--") + 1 :]) + + +def material(name: str, color: list[float], roughness: float = 0.7, metallic: float = 0.0, emission: float = 0.0): + mat = bpy.data.materials.new(name) + mat.diffuse_color = (*color[:3], color[3] if len(color) > 3 else 1.0) + mat.use_nodes = True + bsdf = mat.node_tree.nodes.get("Principled BSDF") + bsdf.inputs["Base Color"].default_value = mat.diffuse_color + bsdf.inputs["Roughness"].default_value = roughness + bsdf.inputs["Metallic"].default_value = metallic + if emission: + bsdf.inputs["Emission Color"].default_value = mat.diffuse_color + bsdf.inputs["Emission Strength"].default_value = emission + return mat + + +def terrain_height(x: float, y: float, spec: dict) -> float: + terrain = spec.get("terrain", {}) + scale = float(terrain.get("height_scale", 16.0)) + frequency = float(terrain.get("frequency", 0.018)) + base = hetero_terrain(Vector((x * frequency, y * frequency, 0)), 1.0, 2.0, 5.0, 0.7) + detail = fractal(Vector((x * frequency * 4.2, y * frequency * 4.2, 4.3)), 1.1, 2.0, 3.0) + height = (base - 0.65) * scale + detail * scale * 0.12 + for region in spec.get("regions", []): + cx, cy = region.get("center", [0, 0])[:2] + radius = max(1.0, float(region.get("radius", 40))) + distance = math.hypot(x - cx, y - cy) + influence = max(0.0, 1.0 - distance / radius) + influence = influence * influence * (3.0 - 2.0 * influence) + height += float(region.get("height_offset", 0)) * influence + if region.get("flatten") is not None: + target = float(region["flatten"]) + strength = float(region.get("flatten_strength", 0.75)) * influence + height = height * (1.0 - strength) + target * strength + return height + + +def clear_scene(): + bpy.ops.object.select_all(action="SELECT") + bpy.ops.object.delete(use_global=False) + for block in (bpy.data.meshes, bpy.data.curves, bpy.data.materials, bpy.data.cameras, bpy.data.lights): + for item in list(block): + if item.users == 0: + block.remove(item) + + +def build_terrain(spec: dict): + terrain = spec.get("terrain", {}) + size = float(terrain.get("size", 240)) + resolution = max(32, min(320, int(terrain.get("resolution", 180)))) + vertices = [] + faces = [] + for iy in range(resolution): + y = -size / 2 + size * iy / (resolution - 1) + for ix in range(resolution): + x = -size / 2 + size * ix / (resolution - 1) + vertices.append((x, y, terrain_height(x, y, spec))) + for iy in range(resolution - 1): + for ix in range(resolution - 1): + a = iy * resolution + ix + faces.append((a, a + 1, a + resolution + 1, a + resolution)) + mesh = bpy.data.meshes.new("WorldTerrainMesh") + mesh.from_pydata(vertices, [], faces) + mesh.update() + obj = bpy.data.objects.new("WorldTerrain", mesh) + bpy.context.collection.objects.link(obj) + palette = terrain.get("palette", [[0.11, 0.28, 0.08, 1], [0.28, 0.45, 0.10, 1], [0.35, 0.28, 0.16, 1]]) + mats = [material(f"Terrain-{index}", list(color), 0.92) for index, color in enumerate(palette)] + region_material_start = len(mats) + for region in spec.get("regions", []): + color = region.get("color") + if color: + mats.append(material(f"Region-{region.get('id', len(mats))}", list(color), float(region.get("roughness", 0.88)))) + for mat in mats: + obj.data.materials.append(mat) + for polygon in mesh.polygons: + center = sum((mesh.vertices[index].co for index in polygon.vertices), Vector()) / len(polygon.vertices) + z = center.z + normal_z = polygon.normal.z + region_choice = None + region_strength = 0.0 + for region_index, region in enumerate(spec.get("regions", [])): + if not region.get("color"): + continue + cx, cy = region.get("center", [0, 0])[:2] + radius = max(1.0, float(region.get("radius", 40))) + strength = max(0.0, 1.0 - math.hypot(center.x - cx, center.y - cy) / radius) + if strength > region_strength: + region_choice = region_index + region_strength = strength + if normal_z < 0.67: + polygon.material_index = min(2, len(palette) - 1) + elif region_choice is not None and region_strength > 0.18: + polygon.material_index = region_material_start + region_choice + else: + polygon.material_index = 1 if z > 4.0 and len(palette) > 1 else 0 + bevel = obj.modifiers.new("Terrain micro bevel", "BEVEL") + bevel.width = 0.18 + bevel.segments = 2 + return obj + + +def make_ribbon(name: str, points: list[list[float]], width: float, mat, z_offset: float = 0.25): + """Build a flat terrain-following ribbon, avoiding tube-like curve bevels.""" + vertices = [] + faces = [] + half_width = width / 2.0 + for index, source in enumerate(points): + x, y = source[:2] + previous = points[max(0, index - 1)] + following = points[min(len(points) - 1, index + 1)] + dx = float(following[0]) - float(previous[0]) + dy = float(following[1]) - float(previous[1]) + length = max(0.001, math.hypot(dx, dy)) + nx, ny = -dy / length, dx / length + for side in (-1.0, 1.0): + vx, vy = x + nx * half_width * side, y + ny * half_width * side + vz = source[2] if len(source) > 2 else terrain_height(vx, vy, WORLD_SPEC) + z_offset + vertices.append((vx, vy, vz)) + if index: + base = index * 2 + faces.append((base - 2, base, base + 1, base - 1)) + mesh = bpy.data.meshes.new(f"{name}Mesh") + mesh.from_pydata(vertices, [], faces) + mesh.update() + obj = bpy.data.objects.new(name, mesh) + bpy.context.collection.objects.link(obj) + obj.data.materials.append(mat) + bevel = obj.modifiers.new(f"{name} edge softness", "BEVEL") + bevel.width = min(0.22, width * 0.03) + bevel.segments = 2 + return obj + + +def import_asset_collection(path: Path, asset_id: str): + before = set(bpy.context.scene.objects) + if path.suffix.lower() in {".glb", ".gltf"}: + bpy.ops.import_scene.gltf(filepath=str(path)) + elif path.suffix.lower() == ".fbx": + bpy.ops.import_scene.fbx(filepath=str(path)) + elif path.suffix.lower() == ".obj": + bpy.ops.wm.obj_import(filepath=str(path)) + else: + raise ValueError(f"Unsupported asset format: {path}") + imported = [obj for obj in bpy.context.scene.objects if obj not in before] + collection = bpy.data.collections.new(f"ASSET::{asset_id}") + bpy.context.scene.collection.children.link(collection) + for obj in imported: + for owner in list(obj.users_collection): + owner.objects.unlink(obj) + collection.objects.link(obj) + if obj.type == "MESH": + for polygon in obj.data.polygons: + polygon.use_smooth = True + # Keep the source collection available for collection instances without + # rendering its authoring copy at the origin. + bpy.context.scene.collection.children.unlink(collection) + z_values = [] + for obj in imported: + if obj.type == "MESH": + z_values.extend((obj.matrix_world @ Vector(corner)).z for corner in obj.bound_box) + source_height = max(z_values) - min(z_values) if z_values else 1.0 + source_floor = min(z_values) if z_values else 0.0 + return collection, max(0.001, source_height), source_floor + + +def scatter_assets(spec: dict): + rng = random.Random(int(spec.get("seed", 1))) + report = {"asset_sources": 0, "instances": 0, "missing_assets": []} + for asset in spec.get("assets", []): + path = Path(asset["path"]).expanduser().resolve() + if not path.is_file(): + report["missing_assets"].append(str(path)) + continue + asset_id = str(asset.get("id") or path.stem) + collection, source_height, source_floor = import_asset_collection(path, asset_id) + target_height = float(asset.get("target_height", source_height)) + normalization = target_height / source_height + report["asset_sources"] += 1 + placements = list(asset.get("placements") or []) + if not placements: + center = asset.get("center", [0, 0]) + radius = float(asset.get("radius", 30)) + count = int(asset.get("count", 1)) + exclusions = list(asset.get("exclusion_zones") or []) + attempts = 0 + while len(placements) < count and attempts < count * 24: + attempts += 1 + angle = rng.random() * math.tau + distance = radius * math.sqrt(rng.random()) + x = center[0] + math.cos(angle) * distance + y = center[1] + math.sin(angle) * distance + if any( + math.hypot(x - zone.get("center", [0, 0])[0], y - zone.get("center", [0, 0])[1]) + < float(zone.get("radius", 0)) + for zone in exclusions + ): + continue + placements.append({ + "position": [x, y], + "rotation": rng.random() * math.tau, + "scale": rng.uniform(float(asset.get("scale_min", 1)), float(asset.get("scale_max", asset.get("scale_min", 1)))), + }) + for index, placement in enumerate(placements): + x, y = placement.get("position", [0, 0])[:2] + z = terrain_height(float(x), float(y), spec) + float(placement.get("z_offset", 0)) + instance = bpy.data.objects.new(f"{asset_id}-{index:03d}", None) + instance.instance_type = "COLLECTION" + instance.instance_collection = collection + instance.location = (x, y, z) + if placement.get("rotation_euler_degrees") is not None: + instance.rotation_euler = [math.radians(float(value)) for value in placement["rotation_euler_degrees"]] + else: + instance.rotation_euler[2] = float(placement.get("rotation", 0)) + scale = placement.get("scale", 1) + if isinstance(scale, list): + instance.scale = [component * normalization for component in scale] + instance.location.z = z - source_floor * instance.scale.z + else: + effective_scale = scale * normalization + instance.scale = (effective_scale, effective_scale, effective_scale) + instance.location.z = z - source_floor * effective_scale + visible_from = placement.get("visible_from_seconds", asset.get("visible_from_seconds")) + visible_until = placement.get("visible_until_seconds", asset.get("visible_until_seconds")) + if visible_from is not None: + reveal_frame = max(1, round(float(visible_from) * int(spec.get("fps", 30)))) + instance.hide_render = True + instance.hide_viewport = True + instance.keyframe_insert("hide_render", frame=max(1, reveal_frame - 1)) + instance.keyframe_insert("hide_viewport", frame=max(1, reveal_frame - 1)) + instance.hide_render = False + instance.hide_viewport = False + instance.keyframe_insert("hide_render", frame=reveal_frame) + instance.keyframe_insert("hide_viewport", frame=reveal_frame) + if visible_until is not None: + hide_frame = max(1, round(float(visible_until) * int(spec.get("fps", 30)))) + instance.hide_render = False + instance.hide_viewport = False + instance.keyframe_insert("hide_render", frame=hide_frame) + instance.keyframe_insert("hide_viewport", frame=hide_frame) + instance.hide_render = True + instance.hide_viewport = True + instance.keyframe_insert("hide_render", frame=hide_frame + 1) + instance.keyframe_insert("hide_viewport", frame=hide_frame + 1) + bpy.context.collection.objects.link(instance) + report["instances"] += 1 + return report + + +def look_at(obj, target): + obj.rotation_euler = (Vector(target) - obj.location).to_track_quat("-Z", "Y").to_euler() + + +def setup_camera_and_lights(spec: dict, args: argparse.Namespace): + camera_spec = spec.get("camera", {}) + camera_data = bpy.data.cameras.new("HeroCamera") + camera = bpy.data.objects.new("HeroCamera", camera_data) + bpy.context.collection.objects.link(camera) + camera.location = camera_spec.get("position", [105, -125, 95]) + camera_data.lens = float(camera_spec.get("lens", 44)) + camera_data.sensor_width = 36 + target = bpy.data.objects.new("CameraTarget", None) + target.empty_display_type = "SPHERE" + target.empty_display_size = 1.0 + target.location = camera_spec.get("target", [0, 0, 3]) + bpy.context.collection.objects.link(target) + tracking = camera.constraints.new(type="TRACK_TO") + tracking.target = target + tracking.track_axis = "TRACK_NEGATIVE_Z" + tracking.up_axis = "UP_Y" + bpy.context.scene.camera = camera + + camera.data.lens = float(camera_spec.get("lens", 44)) + + for key_index, key in enumerate(camera_spec.get("path", [])): + frame = 1 + round(float(key["time"]) * args.fps) + camera.location = key["position"] + target.location = key["target"] + if key.get("lens") is not None: + camera.data.lens = float(key["lens"]) + camera.data.keyframe_insert("lens", frame=frame) + camera.keyframe_insert("location", frame=frame) + target.keyframe_insert("location", frame=frame) + for animated in (camera, target): + for curve in animated.animation_data.action.fcurves if animated.animation_data and animated.animation_data.action else []: + for point in curve.keyframe_points: + point.interpolation = "BEZIER" + + lighting = spec.get("lighting", {}) + sun_data = bpy.data.lights.new("Sun", "SUN") + sun_data.energy = float(lighting.get("sun_energy", 3.0)) + sun_data.angle = math.radians(float(lighting.get("sun_angle_degrees", 18))) + sun = bpy.data.objects.new("Sun", sun_data) + sun.rotation_euler = [math.radians(value) for value in lighting.get("sun_rotation_degrees", [35, -28, -32])] + bpy.context.collection.objects.link(sun) + + area_data = bpy.data.lights.new("SkyFill", "AREA") + area_data.energy = float(lighting.get("fill_energy", 850)) + area_data.shape = "DISK" + area_data.size = 70 + area = bpy.data.objects.new("SkyFill", area_data) + area.location = (-35, -20, 70) + look_at(area, [0, 0, 0]) + bpy.context.collection.objects.link(area) + + world = bpy.context.scene.world or bpy.data.worlds.new("World") + bpy.context.scene.world = world + world.use_nodes = True + background = world.node_tree.nodes.get("Background") + background.inputs["Color"].default_value = lighting.get("world_color", [0.16, 0.24, 0.34, 1]) + background.inputs["Strength"].default_value = float(lighting.get("world_strength", 0.5)) + + +def setup_title(spec: dict, args: argparse.Namespace): + title = spec.get("title_card") + if not title: + return + curve = bpy.data.curves.new("FinalTitleText", "FONT") + curve.body = str(title.get("text", "")) + curve.align_x = "CENTER" + curve.align_y = "CENTER" + curve.size = float(title.get("size", 0.62)) + curve.extrude = 0.012 + curve.bevel_depth = 0.004 + text = bpy.data.objects.new("FinalTitle", curve) + bpy.context.collection.objects.link(text) + text.parent = bpy.context.scene.camera + text.location = title.get("camera_local_position", [0, -0.92, -5.2]) + text.rotation_euler = (0, 0, 0) + text.data.materials.append(material("FinalTitleGold", title.get("color", [0.95, 0.68, 0.24, 1]), 0.38, 0.05, 0.12)) + start_frame = round(float(title.get("start_seconds", 58.0)) * args.fps) + text.hide_render = True + text.hide_viewport = True + text.keyframe_insert("hide_render", frame=max(1, start_frame - 1)) + text.keyframe_insert("hide_viewport", frame=max(1, start_frame - 1)) + text.hide_render = False + text.hide_viewport = False + text.keyframe_insert("hide_render", frame=start_frame) + text.keyframe_insert("hide_viewport", frame=start_frame) + + +def setup_render(args: argparse.Namespace): + scene = bpy.context.scene + scene.render.engine = "BLENDER_EEVEE_NEXT" + scene.eevee.taa_render_samples = args.samples + scene.render.resolution_x = args.width + scene.render.resolution_y = args.height + scene.render.resolution_percentage = 100 + scene.render.image_settings.file_format = "PNG" + scene.render.film_transparent = False + scene.render.fps = args.fps + scene.frame_start = args.start_frame or 1 + scene.frame_end = args.end_frame or max(1, round(args.duration * args.fps)) + scene.render.image_settings.color_mode = "RGBA" + scene.view_settings.look = "AgX - Medium High Contrast" + scene.render.filepath = args.output + + +def build(spec: dict, args: argparse.Namespace): + global WORLD_SPEC + WORLD_SPEC = spec + clear_scene() + seed_set(int(spec.get("seed", 1))) + build_terrain(spec) + water = spec.get("water") + if water: + water_mat = material("Water", water.get("color", [0.03, 0.30, 0.48, 0.82]), 0.13, 0.05) + make_ribbon("River", water.get("points", [[-90, -20], [0, 0], [90, 25]]), float(water.get("width", 5)), water_mat, float(water.get("z_offset", 0.6))) + path_spec = spec.get("path") + if path_spec: + path_mat = material("Path", path_spec.get("color", [0.55, 0.36, 0.14, 1]), 0.95) + make_ribbon("Path", path_spec.get("points", []), float(path_spec.get("width", 2.2)), path_mat, float(path_spec.get("z_offset", 0.32))) + report = scatter_assets(spec) + setup_camera_and_lights(spec, args) + setup_title(spec, args) + setup_render(args) + bpy.ops.wm.save_as_mainfile(filepath=args.blend) + Path(args.blend).with_suffix(".report.json").write_text(json.dumps({ + "version": "1.0", "engine": "BLENDER_EEVEE_NEXT", "seed": spec.get("seed"), **report, + }, indent=2), encoding="utf-8") + return report + + +def main(): + args = args_after_separator() + spec = json.loads(Path(args.spec).read_text(encoding="utf-8")) + report = build(spec, args) + if args.operation == "render_still": + bpy.context.scene.frame_set(args.frame or 1) + bpy.context.scene.render.filepath = args.output + bpy.ops.render.render(write_still=True) + elif args.operation == "render_animation": + bpy.context.scene.render.filepath = args.output + bpy.ops.render.render(animation=True) + print("OPENMONTAGE_WORLD_REPORT=" + json.dumps(report, sort_keys=True)) + + +WORLD_SPEC = {} +main() diff --git a/tools/graphics/templates/threejs_world/index.html b/tools/graphics/templates/threejs_world/index.html new file mode 100644 index 00000000..838332fb --- /dev/null +++ b/tools/graphics/templates/threejs_world/index.html @@ -0,0 +1,66 @@ + + + + + + __TITLE__ + + + + +
+
+ + + + +
+
OPENMONTAGE · EXPLICIT WORLD 01
+

__TITLE__

+

ONE CONTINUOUS WORLD · FREE VIEWPOINT · SEEDED & EDITABLE

+
+ + + +
BUILDING WORLD GRAPH…
+
+
+ + + + + + diff --git a/tools/graphics/templates/threejs_world/world-runtime.js b/tools/graphics/templates/threejs_world/world-runtime.js new file mode 100644 index 00000000..1b53e828 --- /dev/null +++ b/tools/graphics/templates/threejs_world/world-runtime.js @@ -0,0 +1,478 @@ +import * as THREE from "https://cdn.jsdelivr.net/npm/three@0.181.2/+esm"; +import { GLTFLoader } from "https://cdn.jsdelivr.net/npm/three@0.181.2/examples/jsm/loaders/GLTFLoader.js"; +import { WORLD_SPEC } from "./world-spec.js"; +import { ASSET_CATALOG } from "./asset-catalog.js"; + +const root = document.getElementById("world-root"); +const canvas = document.getElementById("world-canvas"); +const status = document.getElementById("world-status"); +const regionName = document.getElementById("world-region-name"); +const timecode = document.getElementById("world-timecode"); +const altitude = document.getElementById("world-altitude"); +const renderMode = root.dataset.renderMode || "cinematic"; +const width = Number(root.dataset.width || canvas.width || 1920); +const height = Number(root.dataset.height || canvas.height || 1080); +const qualityTier = window.__WORLD_QUALITY_TIER__ || "blockout"; + +function mulberry32(seed) { + let value = seed >>> 0; + return () => { + value += 0x6d2b79f5; + let t = value; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +function hashString(text) { + let hash = 2166136261; + for (let i = 0; i < text.length; i += 1) { + hash ^= text.charCodeAt(i); + hash = Math.imul(hash, 16777619); + } + return hash >>> 0; +} + +function clamp(value, low, high) { return Math.max(low, Math.min(high, value)); } +function smoothstep(value) { const t = clamp(value, 0, 1); return t * t * (3 - 2 * t); } + +function regionWeights(nx, nz) { + const raw = WORLD_SPEC.regions.map((region) => { + const dx = nx - region.center[0]; + const dz = nz - region.center[1]; + const distance = Math.hypot(dx, dz) / Math.max(0.05, region.radius); + const softness = Math.max(0.02, region.blend_width); + return Math.max(0.00001, Math.exp(-Math.pow(Math.max(0, distance - 0.05), 2) / (softness * 2.8))); + }); + const total = raw.reduce((sum, value) => sum + value, 0) || 1; + return raw.map((value) => value / total); +} + +function landform(kind, dx, dz, distance) { + if (kind === "peak") return Math.pow(Math.max(0, 1 - distance), 2.2); + if (kind === "ridge") return Math.max(0, 1 - Math.abs(dx * 1.8 + Math.sin(dz * 5) * 0.16)); + if (kind === "dune") return (Math.sin((dx + dz * 0.25) * 18) + 1) * 0.24; + if (kind === "terrace") return Math.floor(Math.max(0, 1 - distance) * 5) / 5; + if (kind === "basin") return -Math.pow(Math.max(0, 1 - distance), 1.7); + if (kind === "canyon") return -Math.pow(Math.max(0, 1 - Math.abs(dx + Math.sin(dz * 7) * 0.1)), 2); + return 0; +} + +function heightAt(x, z) { + const half = WORLD_SPEC.world.size / 2; + const nx = x / half; + const nz = z / half; + const weights = regionWeights(nx, nz); + const seed = WORLD_SPEC.seed * 0.01337; + let elevation = 0; + WORLD_SPEC.regions.forEach((region, index) => { + const frequency = region.frequency; + const noise = ( + Math.sin((nx * 3.1 + seed + index) * frequency * Math.PI) + + Math.cos((nz * 2.7 - seed * 0.7 + index) * frequency * Math.PI) + + 0.5 * Math.sin((nx + nz) * frequency * 7.3 + seed * 3 + index) + ) / 2.5; + const dx = nx - region.center[0]; + const dz = nz - region.center[1]; + const distance = Math.hypot(dx, dz) / Math.max(0.05, region.radius); + elevation += weights[index] * ( + region.base_elevation + region.amplitude * (noise * 0.48 + landform(region.landform, dx, dz, distance) * 0.8) + ); + }); + return elevation * WORLD_SPEC.world.elevation_scale; +} + +function dominantRegion(x, z) { + const half = WORLD_SPEC.world.size / 2; + const weights = regionWeights(x / half, z / half); + let index = 0; + for (let i = 1; i < weights.length; i += 1) if (weights[i] > weights[index]) index = i; + return { region: WORLD_SPEC.regions[index], weight: weights[index] }; +} + +const renderer = new THREE.WebGLRenderer({ canvas, antialias: true, alpha: false, powerPreference: "high-performance" }); +renderer.setSize(width, height, false); +renderer.setPixelRatio(1); +renderer.outputColorSpace = THREE.SRGBColorSpace; +renderer.toneMapping = THREE.ACESFilmicToneMapping; +renderer.toneMappingExposure = renderMode === "cinematic" ? 1.05 : 1; +renderer.shadowMap.enabled = renderMode === "cinematic"; +renderer.shadowMap.type = THREE.PCFSoftShadowMap; + +const scene = new THREE.Scene(); +scene.background = new THREE.Color(WORLD_SPEC.atmosphere.sky_color); +if (renderMode === "cinematic" && WORLD_SPEC.atmosphere.fog_density > 0) { + scene.fog = new THREE.FogExp2(WORLD_SPEC.atmosphere.fog_color, WORLD_SPEC.atmosphere.fog_density); +} + +const camera = new THREE.PerspectiveCamera(45, width / height, 0.2, WORLD_SPEC.world.size * 4); +const terrainGroup = new THREE.Group(); +terrainGroup.name = "terrain-foundation"; +const environmentGroup = new THREE.Group(); +environmentGroup.name = "environment-prototypes"; +const landmarkGroup = new THREE.Group(); +landmarkGroup.name = "regional-landmarks"; +scene.add(terrainGroup, environmentGroup, landmarkGroup); + +const hemi = new THREE.HemisphereLight( + WORLD_SPEC.atmosphere.sky_color, + WORLD_SPEC.atmosphere.ground_color, + renderMode === "cinematic" ? 1.45 : 2.2, +); +scene.add(hemi); + +const sun = new THREE.DirectionalLight(WORLD_SPEC.atmosphere.sun_color, WORLD_SPEC.atmosphere.sun_intensity); +sun.position.fromArray(WORLD_SPEC.atmosphere.sun_position); +sun.castShadow = renderMode === "cinematic"; +sun.shadow.mapSize.set(1024, 1024); +const shadowSpan = WORLD_SPEC.world.size * 0.62; +sun.shadow.camera.left = -shadowSpan; +sun.shadow.camera.right = shadowSpan; +sun.shadow.camera.top = shadowSpan; +sun.shadow.camera.bottom = -shadowSpan; +sun.shadow.camera.near = 1; +sun.shadow.camera.far = WORLD_SPEC.world.size * 3; +sun.shadow.bias = -0.0003; +sun.shadow.normalBias = 0.035; +scene.add(sun); + +const terrainGeometry = new THREE.PlaneGeometry( + WORLD_SPEC.world.size, + WORLD_SPEC.world.size, + WORLD_SPEC.world.resolution, + WORLD_SPEC.world.resolution, +); +terrainGeometry.rotateX(-Math.PI / 2); +const position = terrainGeometry.attributes.position; +const colors = new Float32Array(position.count * 3); +const color = new THREE.Color(); +const mixed = new THREE.Color(); +for (let index = 0; index < position.count; index += 1) { + const x = position.getX(index); + const z = position.getZ(index); + position.setY(index, heightAt(x, z)); + const weights = regionWeights(x / (WORLD_SPEC.world.size / 2), z / (WORLD_SPEC.world.size / 2)); + mixed.setRGB(0, 0, 0); + WORLD_SPEC.regions.forEach((region, regionIndex) => { + color.set(renderMode === "semantic" ? region.accent_color : region.color); + mixed.r += color.r * weights[regionIndex]; + mixed.g += color.g * weights[regionIndex]; + mixed.b += color.b * weights[regionIndex]; + }); + colors[index * 3] = mixed.r; + colors[index * 3 + 1] = mixed.g; + colors[index * 3 + 2] = mixed.b; +} +position.needsUpdate = true; +terrainGeometry.setAttribute("color", new THREE.BufferAttribute(colors, 3)); +terrainGeometry.computeVertexNormals(); +terrainGeometry.computeBoundingSphere(); + +const terrainMaterial = renderMode === "wireframe" + ? new THREE.MeshBasicMaterial({ color: 0x8de8ff, wireframe: true, transparent: true, opacity: 0.82 }) + : new THREE.MeshStandardMaterial({ vertexColors: true, roughness: 0.92, metalness: 0.02, flatShading: false }); +const terrain = new THREE.Mesh(terrainGeometry, terrainMaterial); +terrain.name = "semantic-terrain"; +terrain.receiveShadow = renderMode === "cinematic"; +terrainGroup.add(terrain); + +let water = null; +if (renderMode !== "wireframe") { + const waterGeometry = new THREE.PlaneGeometry(WORLD_SPEC.world.size * 1.08, WORLD_SPEC.world.size * 1.08, 1, 1); + waterGeometry.rotateX(-Math.PI / 2); + const waterMaterial = new THREE.MeshPhysicalMaterial({ + color: renderMode === "semantic" ? 0x1765a3 : 0x143d55, + roughness: 0.22, + metalness: 0.08, + transmission: renderMode === "cinematic" ? 0.22 : 0, + transparent: true, + opacity: renderMode === "cinematic" ? 0.76 : 0.9, + depthWrite: false, + }); + water = new THREE.Mesh(waterGeometry, waterMaterial); + water.name = "global-water-plane"; + water.position.y = WORLD_SPEC.world.water_level; + terrainGroup.add(water); +} + +const instanceStats = { tree: 0, rock: 0, crystal: 0 }; +function slopeAt(x, z) { + const step = 0.65; + return Math.abs(heightAt(x + step, z) - heightAt(x - step, z)) + + Math.abs(heightAt(x, z + step) - heightAt(x, z - step)); +} + +function scatterPoints(region, count, salt) { + const random = mulberry32((WORLD_SPEC.seed ^ hashString(region.id + salt)) >>> 0); + const points = []; + const half = WORLD_SPEC.world.size / 2; + for (let index = 0; index < count; index += 1) { + let accepted = null; + for (let attempt = 0; attempt < 14; attempt += 1) { + const angle = random() * Math.PI * 2; + const radius = Math.sqrt(random()) * region.radius * half; + const x = region.center[0] * half + Math.cos(angle) * radius; + const z = region.center[1] * half + Math.sin(angle) * radius; + const dominant = dominantRegion(x, z); + if (dominant.region.id !== region.id || dominant.weight < 0.34) continue; + if (slopeAt(x, z) > region.slope_limit) continue; + accepted = { x, z, y: heightAt(x, z), rotation: random() * Math.PI * 2, scale: 0.72 + random() * 0.72 }; + break; + } + if (accepted) points.push(accepted); + } + return points; +} + +function makeInstanced(geometry, material, points, transform) { + if (!points.length) return null; + const mesh = new THREE.InstancedMesh(geometry, material, points.length); + const dummy = new THREE.Object3D(); + points.forEach((point, index) => { + transform(dummy, point, index); + dummy.updateMatrix(); + mesh.setMatrixAt(index, dummy.matrix); + }); + mesh.instanceMatrix.needsUpdate = true; + mesh.castShadow = renderMode === "cinematic"; + mesh.receiveShadow = renderMode === "cinematic"; + environmentGroup.add(mesh); + return mesh; +} + +const catalogModels = new Map(); +for (const catalog of ASSET_CATALOG.catalogs || []) { + for (const model of catalog.models || []) { + catalogModels.set(`${catalog.catalog_id}:${model.id}`, model.runtime_path); + } +} + +async function loadProductionPalette() { + if (qualityTier !== "production") return; + const loader = new GLTFLoader(); + const prototypes = new Map(); + const palette = WORLD_SPEC.asset_palette || []; + await Promise.all(palette.map(async (entry) => { + const key = `${entry.catalog_id}:${entry.model_id}`; + const path = catalogModels.get(key); + if (!path || prototypes.has(key)) return; + const gltf = await loader.loadAsync(path); + gltf.scene.traverse((node) => { + if (!node.isMesh) return; + node.castShadow = renderMode === "cinematic"; + node.receiveShadow = renderMode === "cinematic"; + if (node.material) node.material.envMapIntensity = 0.8; + }); + prototypes.set(key, gltf.scene); + })); + + palette.forEach((entry, entryIndex) => { + const prototype = prototypes.get(`${entry.catalog_id}:${entry.model_id}`); + if (!prototype) return; + const region = WORLD_SPEC.regions.find((item) => item.id === entry.region_id) || WORLD_SPEC.regions[entryIndex % WORLD_SPEC.regions.length]; + const points = scatterPoints(region, Math.min(180, Math.max(1, Number(entry.count || 12))), `catalog-${entry.id || entryIndex}`); + points.forEach((point, pointIndex) => { + const clone = prototype.clone(true); + const random = mulberry32((WORLD_SPEC.seed ^ hashString(`${entry.id || entryIndex}:${pointIndex}`)) >>> 0); + const scaleRange = Array.isArray(entry.scale_range) ? entry.scale_range : [0.8, 1.4]; + const scale = THREE.MathUtils.lerp(Number(scaleRange[0]), Number(scaleRange[1]), random()) * Number(entry.base_scale || 1); + clone.position.set(point.x, point.y + Number(entry.y_offset || 0), point.z); + clone.rotation.y = random() * Math.PI * 2; + clone.scale.setScalar(scale); + clone.name = `catalog-${entry.id || entryIndex}-${pointIndex}`; + environmentGroup.add(clone); + }); + }); +} + +WORLD_SPEC.regions.forEach((region) => { + const regionColor = new THREE.Color(renderMode === "semantic" ? region.accent_color : region.color); + const accentColor = new THREE.Color(region.accent_color); + + const rocks = scatterPoints(region, region.scatter.rock, "rock"); + instanceStats.rock += rocks.length; + makeInstanced( + new THREE.DodecahedronGeometry(0.72, 0), + new THREE.MeshStandardMaterial({ color: regionColor.clone().multiplyScalar(0.72), roughness: 0.95, wireframe: renderMode === "wireframe" }), + rocks, + (dummy, point) => { + dummy.position.set(point.x, point.y + 0.42 * point.scale, point.z); + dummy.rotation.set(point.rotation * 0.17, point.rotation, point.rotation * 0.11); + dummy.scale.set(point.scale * 1.1, point.scale * 0.72, point.scale); + }, + ); + + const crystals = scatterPoints(region, region.scatter.crystal, "crystal"); + instanceStats.crystal += crystals.length; + makeInstanced( + new THREE.OctahedronGeometry(0.72, 0), + new THREE.MeshStandardMaterial({ color: accentColor, emissive: accentColor, emissiveIntensity: renderMode === "cinematic" ? 1.7 : 0.25, roughness: 0.28, metalness: 0.28, wireframe: renderMode === "wireframe" }), + crystals, + (dummy, point) => { + dummy.position.set(point.x, point.y + 0.82 * point.scale, point.z); + dummy.rotation.set(0.08, point.rotation, 0.05); + dummy.scale.set(point.scale * 0.46, point.scale * 1.75, point.scale * 0.46); + }, + ); + + const trees = scatterPoints(region, region.scatter.tree, "tree"); + instanceStats.tree += trees.length; + const trunkMaterial = new THREE.MeshStandardMaterial({ color: renderMode === "semantic" ? region.accent_color : 0x3e2b22, roughness: 1, wireframe: renderMode === "wireframe" }); + const canopyMaterial = new THREE.MeshStandardMaterial({ color: regionColor.clone().offsetHSL(0, 0.08, 0.09), roughness: 0.94, wireframe: renderMode === "wireframe" }); + makeInstanced(new THREE.CylinderGeometry(0.16, 0.23, 1.75, 6), trunkMaterial, trees, (dummy, point) => { + dummy.position.set(point.x, point.y + 0.88 * point.scale, point.z); + dummy.rotation.set(0, point.rotation, 0); + dummy.scale.setScalar(point.scale); + }); + makeInstanced(new THREE.ConeGeometry(0.92, 2.4, 7), canopyMaterial, trees, (dummy, point) => { + dummy.position.set(point.x, point.y + 2.35 * point.scale, point.z); + dummy.rotation.set(0, point.rotation, 0); + dummy.scale.setScalar(point.scale); + }); +}); + +function materialPair(landmark) { + const base = new THREE.Color(renderMode === "semantic" ? landmark.accent_color : landmark.color); + const accent = new THREE.Color(landmark.accent_color); + return { + base: new THREE.MeshStandardMaterial({ color: base, roughness: 0.72, metalness: 0.18, wireframe: renderMode === "wireframe" }), + accent: new THREE.MeshStandardMaterial({ color: accent, emissive: accent, emissiveIntensity: renderMode === "cinematic" ? 1.25 : 0.2, roughness: 0.28, metalness: 0.38, wireframe: renderMode === "wireframe" }), + }; +} + +function addMesh(group, geometry, material, positionValue, scaleValue = [1, 1, 1], rotationValue = [0, 0, 0]) { + const mesh = new THREE.Mesh(geometry, material); + mesh.position.set(...positionValue); + mesh.scale.set(...scaleValue); + mesh.rotation.set(...rotationValue); + mesh.castShadow = renderMode === "cinematic"; + mesh.receiveShadow = renderMode === "cinematic"; + group.add(mesh); + return mesh; +} + +function buildLandmark(landmark) { + const group = new THREE.Group(); + group.name = landmark.id; + const materials = materialPair(landmark); + const s = landmark.scale; + const random = mulberry32((WORLD_SPEC.seed ^ hashString(landmark.id)) >>> 0); + + if (landmark.type === "arch") { + addMesh(group, new THREE.BoxGeometry(1, 1, 1), materials.base, [-0.72 * s, 0.7 * s, 0], [0.34 * s, 1.4 * s, 0.42 * s]); + addMesh(group, new THREE.BoxGeometry(1, 1, 1), materials.base, [0.72 * s, 0.7 * s, 0], [0.34 * s, 1.4 * s, 0.42 * s]); + addMesh(group, new THREE.BoxGeometry(1, 1, 1), materials.accent, [0, 1.48 * s, 0], [1.06 * s, 0.24 * s, 0.42 * s]); + } else if (landmark.type === "tower") { + addMesh(group, new THREE.CylinderGeometry(0.52, 0.68, 2.4, 8), materials.base, [0, 1.2 * s, 0], [s, s, s]); + addMesh(group, new THREE.TorusGeometry(0.68, 0.09, 8, 24), materials.accent, [0, 2.08 * s, 0], [s, s, s], [Math.PI / 2, 0, 0]); + addMesh(group, new THREE.ConeGeometry(0.72, 1.2, 8), materials.accent, [0, 2.72 * s, 0], [s, s, s]); + } else if (landmark.type === "ruin") { + for (let i = 0; i < 7; i += 1) { + const angle = (i / 7) * Math.PI * 2 + random() * 0.2; + const radius = s * (0.55 + random() * 0.45); + const h = s * (0.45 + random() * 1.1); + addMesh(group, new THREE.BoxGeometry(1, 1, 1), i === 3 ? materials.accent : materials.base, [Math.cos(angle) * radius, h / 2, Math.sin(angle) * radius], [s * 0.25, h, s * 0.25], [0, random() * Math.PI, (random() - 0.5) * 0.14]); + } + } else if (landmark.type === "crystal") { + for (let i = 0; i < 5; i += 1) { + const angle = (i / 5) * Math.PI * 2; + const localScale = s * (i === 0 ? 1.45 : 0.62 + random() * 0.35); + addMesh(group, new THREE.OctahedronGeometry(0.55, 0), materials.accent, [Math.cos(angle) * s * 0.42, localScale * 0.62, Math.sin(angle) * s * 0.42], [localScale * 0.44, localScale * 1.25, localScale * 0.44], [0.06, angle, 0.04]); + } + } else if (landmark.type === "settlement") { + for (let i = 0; i < 9; i += 1) { + const angle = (i / 9) * Math.PI * 2 + random() * 0.3; + const radius = s * (0.35 + random() * 1.05); + const h = s * (0.24 + random() * 0.52); + addMesh(group, new THREE.CylinderGeometry(0.42, 0.56, 1, 6), materials.base, [Math.cos(angle) * radius, h / 2, Math.sin(angle) * radius], [s * 0.42, h, s * 0.42], [0, angle, 0]); + addMesh(group, new THREE.ConeGeometry(0.64, 0.7, 6), materials.accent, [Math.cos(angle) * radius, h + s * 0.18, Math.sin(angle) * radius], [s * 0.42, s * 0.42, s * 0.42], [0, angle, 0]); + } + } else if (landmark.type === "ring") { + addMesh(group, new THREE.TorusGeometry(1, 0.12, 12, 64), materials.accent, [0, 1.25 * s, 0], [s, s, s], [0, 0, 0]); + addMesh(group, new THREE.CylinderGeometry(0.28, 0.48, 1.4, 8), materials.base, [0, 0.7 * s, 0], [s, s, s]); + } else { + addMesh(group, new THREE.BoxGeometry(1, 1, 1), materials.base, [0, 0.95 * s, 0], [0.62 * s, 1.9 * s, 0.62 * s], [0.04, 0.35, -0.03]); + addMesh(group, new THREE.OctahedronGeometry(0.32, 0), materials.accent, [0, 2.08 * s, 0], [s, s, s]); + } + + const terrainY = heightAt(landmark.position[0], landmark.position[2]); + group.position.set(landmark.position[0], terrainY + landmark.position[1], landmark.position[2]); + group.rotation.set(...landmark.rotation); + landmarkGroup.add(group); +} +if (qualityTier === "blockout") WORLD_SPEC.landmarks.forEach(buildLandmark); + +function interpolateVector(left, right, amount) { + return new THREE.Vector3( + THREE.MathUtils.lerp(left[0], right[0], amount), + THREE.MathUtils.lerp(left[1], right[1], amount), + THREE.MathUtils.lerp(left[2], right[2], amount), + ); +} + +function cameraAt(time) { + const keys = WORLD_SPEC.camera_path; + if (time <= keys[0].time) return { ...keys[0], positionV: new THREE.Vector3(...keys[0].position), targetV: new THREE.Vector3(...keys[0].target) }; + if (time >= keys[keys.length - 1].time) { + const key = keys[keys.length - 1]; + return { ...key, positionV: new THREE.Vector3(...key.position), targetV: new THREE.Vector3(...key.target) }; + } + for (let index = 0; index < keys.length - 1; index += 1) { + const left = keys[index]; + const right = keys[index + 1]; + if (time >= left.time && time <= right.time) { + const amount = smoothstep((time - left.time) / Math.max(0.0001, right.time - left.time)); + return { + label: amount < 0.5 ? left.label : right.label, + positionV: interpolateVector(left.position, right.position, amount), + targetV: interpolateVector(left.target, right.target, amount), + fov: THREE.MathUtils.lerp(left.fov, right.fov, amount), + }; + } + } + const fallback = keys[keys.length - 1]; + return { ...fallback, positionV: new THREE.Vector3(...fallback.position), targetV: new THREE.Vector3(...fallback.target) }; +} + +function formatTime(value) { + const minutes = Math.floor(value / 60).toString().padStart(2, "0"); + const seconds = (value % 60).toFixed(1).padStart(4, "0"); + return `${minutes}:${seconds}`; +} + +function renderAt(timeValue) { + const time = Math.max(0, Number(timeValue) || 0); + const state = cameraAt(time); + camera.position.copy(state.positionV); + camera.fov = state.fov; + camera.updateProjectionMatrix(); + camera.lookAt(state.targetV); + + if (water) water.material.opacity = (renderMode === "cinematic" ? 0.73 : 0.88) + Math.sin(time * 0.42) * 0.035; + sun.intensity = WORLD_SPEC.atmosphere.sun_intensity * (0.96 + Math.sin(time * 0.09) * 0.04); + + const regionState = dominantRegion(state.targetV.x, state.targetV.z); + regionName.textContent = state.label || regionState.region.label; + timecode.textContent = formatTime(time); + altitude.textContent = camera.position.y.toFixed(1).padStart(5, "0"); + renderer.render(scene, camera); +} + +async function finalizeWorld() { + await loadProductionPalette(); + window.addEventListener("hf-seek", (event) => renderAt(event.detail.time)); + window.__worldRenderAt = renderAt; + window.__worldGraph = { scene, camera, terrainGroup, environmentGroup, landmarkGroup, instanceStats }; + window.__worldReady = true; + status.textContent = `WORLD READY · ${WORLD_SPEC.regions.length} REGIONS · ${WORLD_SPEC.landmarks.length} LANDMARKS · ${qualityTier.toUpperCase()}`; + status.style.opacity = "0"; + renderAt(window.__hfThreeTime || 0); +} + +finalizeWorld().catch((error) => { + window.__worldReady = false; + window.__worldError = String(error?.stack || error); + status.textContent = "WORLD ASSET LOAD FAILED"; + console.error(error); +}); diff --git a/tools/graphics/templates/threejs_world/world.css b/tools/graphics/templates/threejs_world/world.css new file mode 100644 index 00000000..899d5b16 --- /dev/null +++ b/tools/graphics/templates/threejs_world/world.css @@ -0,0 +1,79 @@ +:root { + color-scheme: dark; + font-family: Inter, sans-serif; + background: #05080d; +} + +* { box-sizing: border-box; } +html, body { width: 100%; height: 100%; margin: 0; overflow: hidden; background: #05080d; } + +#world-root { + position: relative; + width: var(--world-width, 1920px); + height: var(--world-height, 1080px); + overflow: hidden; + background: transparent; + color: #f5f8ff; +} + +.world-stage { position: absolute; inset: 0; width: 100%; height: 100%; overflow: hidden; background: #05080d; } +#world-canvas { position: absolute; inset: 0; width: 100%; height: 100%; display: block; } + +#world-vignette { + position: absolute; + inset: 0; + pointer-events: none; + background: + radial-gradient(circle at 50% 43%, transparent 42%, rgba(3, 6, 11, 0.28) 73%, rgba(1, 3, 7, 0.86) 100%), + linear-gradient(180deg, rgba(1, 5, 10, 0.05), rgba(1, 5, 10, 0.24)); +} + +#world-grain { + position: absolute; + inset: 0; + pointer-events: none; + opacity: 0.09; + mix-blend-mode: soft-light; + background-image: url("data:image/svg+xml,%3Csvg viewBox='0 0 180 180' xmlns='http://www.w3.org/2000/svg'%3E%3Cfilter id='n'%3E%3CfeTurbulence type='fractalNoise' baseFrequency='.9' numOctaves='3' stitchTiles='stitch'/%3E%3C/filter%3E%3Crect width='100%25' height='100%25' filter='url(%23n)' opacity='.65'/%3E%3C/svg%3E"); +} + +.world-title-card { + position: absolute; + left: clamp(38px, 5vw, 96px); + bottom: clamp(48px, 9.6vh, 104px); + width: min(920px, calc(100% - clamp(76px, 10vw, 192px))); + opacity: 0; + text-shadow: 0 4px 36px rgba(0, 0, 0, 0.84); +} + +.eyebrow { margin-bottom: 18px; font: 600 18px/1.2 "JetBrains Mono", Consolas, monospace; letter-spacing: 0.22em; color: #9fdff2; } +.world-title-card h1 { margin: 0; max-width: 900px; font: 720 clamp(42px, 4.3vw, 82px)/0.94 Inter, sans-serif; letter-spacing: -0.055em; text-transform: uppercase; } +.world-title-card p { margin: 22px 0 0; font: 600 16px/1.4 "JetBrains Mono", Consolas, monospace; letter-spacing: 0.16em; color: rgba(236, 245, 255, 0.72); } + +.world-hud { + position: absolute; + top: clamp(32px, 6.5vh, 70px); + right: clamp(32px, 3.9vw, 74px); + width: min(330px, calc(100% - 64px)); + padding: 22px 24px 20px; + border: 1px solid rgba(170, 225, 244, 0.24); + border-radius: 2px; + background: linear-gradient(135deg, rgba(4, 12, 20, 0.72), rgba(6, 13, 20, 0.24)); + box-shadow: 0 18px 60px rgba(0, 0, 0, 0.28), inset 0 0 24px rgba(111, 212, 243, 0.035); + backdrop-filter: blur(8px); + opacity: 0; +} + +.hud-rule { width: 54px; height: 3px; margin-bottom: 18px; background: #9fdff2; box-shadow: 0 0 16px rgba(159, 223, 242, 0.55); } +.hud-label { font: 600 12px/1.2 "JetBrains Mono", Consolas, monospace; letter-spacing: 0.2em; color: rgba(213, 238, 248, 0.52); } +.hud-value { margin-top: 7px; min-height: 54px; font: 680 28px/1.02 Inter, sans-serif; letter-spacing: -0.03em; text-transform: uppercase; } +.hud-grid { display: grid; grid-template-columns: 74px 1fr; gap: 9px 16px; padding-top: 17px; border-top: 1px solid rgba(172, 224, 241, 0.16); font: 500 12px/1.1 "JetBrains Mono", Consolas, monospace; letter-spacing: 0.1em; } +.hud-grid span { color: rgba(209, 236, 246, 0.68); } +.hud-grid strong { text-align: right; color: rgba(235, 249, 255, 0.88); text-transform: uppercase; } + +#world-status { position: absolute; left: 50%; top: 50%; transform: translate(-50%, -50%); padding: 13px 18px; border: 1px solid rgba(174, 232, 249, 0.32); background: rgba(3, 9, 15, 0.72); font: 600 13px/1 "JetBrains Mono", Consolas, monospace; letter-spacing: 0.15em; color: #cceefa; } + +[data-render-mode="semantic"] #world-vignette, +[data-render-mode="wireframe"] #world-vignette, +[data-render-mode="semantic"] #world-grain, +[data-render-mode="wireframe"] #world-grain { display: none; } diff --git a/tools/graphics/threejs_asset_catalog.py b/tools/graphics/threejs_asset_catalog.py new file mode 100644 index 00000000..9026f2ba --- /dev/null +++ b/tools/graphics/threejs_asset_catalog.py @@ -0,0 +1,171 @@ +"""Licensed local GLTF/PBR catalog ingestion for Three.js worlds. + +This module intentionally handles acquisition and provenance only. Creative +selection and placement remain agent decisions expressed through world_spec. +""" + +from __future__ import annotations + +import hashlib +import json +import shutil +import urllib.request +import zipfile +from pathlib import Path +from typing import Any + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ResourceProfile, + ToolResult, + ToolRuntime, + ToolStability, + ToolTier, +) + + +CATALOGS: dict[str, dict[str, Any]] = { + "kenney-nature-kit": { + "title": "Kenney Nature Kit", + "source_url": "https://kenney.nl/assets/nature-kit", + "download_url": "https://kenney.nl/media/pages/assets/nature-kit/37ac38a37b-1677698939/kenney_nature-kit.zip", + "license": "CC0-1.0", + "license_url": "https://creativecommons.org/publicdomain/zero/1.0/", + "tags": ["nature", "tree", "rock", "foliage"], + }, + "kenney-fantasy-town-kit": { + "title": "Kenney Fantasy Town Kit 2.0", + "source_url": "https://kenney.nl/assets/fantasy-town-kit", + "download_url": "https://kenney.nl/media/pages/assets/fantasy-town-kit/efe948d309-1754222374/kenney_fantasy-town-kit_2.0.zip", + "license": "CC0-1.0", + "license_url": "https://creativecommons.org/publicdomain/zero/1.0/", + "tags": ["medieval", "village", "building", "wall", "prop"], + }, + "kenney-survival-kit": { + "title": "Kenney Survival Kit 2.0", + "source_url": "https://kenney.nl/assets/survival-kit", + "download_url": "https://kenney.nl/media/pages/assets/survival-kit/4065a8185b-1712149243/kenney_survival-kit.zip", + "license": "CC0-1.0", + "license_url": "https://creativecommons.org/publicdomain/zero/1.0/", + "tags": ["survival", "camp", "nature", "prop"], + }, +} + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _download(url: str, destination: Path) -> None: + request = urllib.request.Request(url, headers={"User-Agent": "OpenMontage/threejs-asset-catalog"}) + with urllib.request.urlopen(request, timeout=120) as response, destination.open("wb") as output: + shutil.copyfileobj(response, output) + + +class ThreeJSAssetCatalog(BaseTool): + """Install inspectable, rights-safe world asset catalogs.""" + + name = "threejs_asset_catalog" + version = "0.1.0" + tier = ToolTier.SOURCE + capability = "3d_asset_acquisition" + provider = "multi" + stability = ToolStability.BETA + execution_mode = ExecutionMode.SYNC + determinism = Determinism.DETERMINISTIC + runtime = ToolRuntime.HYBRID + dependencies: list[str] = [] + install_instructions = "Network access for install; no API key. Bundled catalogs are CC0." + agent_skills = ["threejs-world-generation", "threejs-loaders", "threejs-materials", "threejs-textures"] + best_for = [ + "Installing rights-safe GLTF/GLB libraries for detailed Three.js worlds", + "Recording model-level provenance before asset-gate review", + ] + not_good_for = [ + "Generating a unique mesh from text or an image", + "Downloading assets whose license is absent or incompatible", + ] + capabilities = ["cc0_catalog_install", "gltf_inventory", "asset_provenance"] + input_schema = { + "type": "object", + "required": ["operation"], + "properties": { + "operation": {"type": "string", "enum": ["list", "install", "inspect"]}, + "catalog_id": {"type": "string"}, + "output_path": {"type": "string"}, + }, + } + output_schema = {"type": "object"} + artifact_schema = {"artifact": "3d_world"} + resource_profile = ResourceProfile(cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=1000, network_required=True) + idempotency_key_fields = ["operation", "catalog_id", "output_path"] + side_effects = ["downloads and extracts a licensed asset archive for install operations"] + fallback_tools: list[str] = [] + user_visible_verification = ["Review catalog-manifest.json and the model inventory before production use"] + + def execute(self, params: dict[str, Any]) -> ToolResult: + operation = params.get("operation") + if operation == "list": + return ToolResult(success=True, data={"catalogs": CATALOGS}) + + catalog_id = str(params.get("catalog_id") or "") + if catalog_id not in CATALOGS: + return ToolResult(success=False, error=f"Unknown catalog_id {catalog_id!r}; choose one of {sorted(CATALOGS)}") + + output_path = params.get("output_path") + if not output_path: + return ToolResult(success=False, error="output_path is required for install and inspect") + root = Path(output_path).expanduser().resolve() + manifest_path = root / "catalog-manifest.json" + + if operation == "inspect": + if not manifest_path.exists(): + return ToolResult(success=False, error=f"No installed catalog manifest at {manifest_path}") + return ToolResult(success=True, data=json.loads(manifest_path.read_text(encoding="utf-8"))) + + if operation != "install": + return ToolResult(success=False, error=f"Unsupported operation {operation!r}") + + source = CATALOGS[catalog_id] + root.mkdir(parents=True, exist_ok=True) + archive = root / f"{catalog_id}.zip" + if not archive.exists(): + _download(source["download_url"], archive) + extract_root = root / "source" + if not extract_root.exists(): + extract_root.mkdir(parents=True) + with zipfile.ZipFile(archive) as package: + package.extractall(extract_root) + + models = sorted( + path for path in extract_root.rglob("*") + if path.is_file() and path.suffix.lower() in {".gltf", ".glb"} + ) + textures = sorted( + path for path in extract_root.rglob("*") + if path.is_file() and path.suffix.lower() in {".png", ".jpg", ".jpeg", ".webp"} + ) + manifest = { + "version": "1.0", + "catalog_id": catalog_id, + **source, + "archive_sha256": _sha256(archive), + "model_count": len(models), + "texture_count": len(textures), + "models": [ + { + "id": path.stem.lower().replace(" ", "-"), + "path": path.relative_to(root).as_posix(), + "format": path.suffix.lower().lstrip("."), + } + for path in models + ], + } + manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8") + return ToolResult(success=True, data=manifest, artifacts=[str(manifest_path)]) diff --git a/tools/graphics/threejs_world.py b/tools/graphics/threejs_world.py new file mode 100644 index 00000000..e939f9b7 --- /dev/null +++ b/tools/graphics/threejs_world.py @@ -0,0 +1,732 @@ +"""Deterministic semantic Three.js world authoring for HyperFrames. + +The agent owns creative planning. This tool validates and normalizes a structured +world specification, materializes an editable Three.js workspace, and emits a +diagnostic report. Rendering remains the responsibility of video_compose / +hyperframes_compose so pipeline governance and review stay intact. +""" + +from __future__ import annotations + +import copy +import html +import json +import math +import re +import shutil +import time +from pathlib import Path +from typing import Any + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ResourceProfile, + ToolResult, + ToolRuntime, + ToolStability, + ToolTier, +) + + +_HEX = re.compile(r"^#[0-9a-fA-F]{6}$") +_LANDFORMS = {"plain", "peak", "ridge", "dune", "terrace", "basin", "canyon"} +_LANDMARKS = {"monolith", "arch", "tower", "ruin", "crystal", "settlement", "ring"} +_RENDER_MODES = {"cinematic", "semantic", "wireframe"} +_QUALITY_TIERS = {"blockout", "production"} + + +def _clamp(value: float, low: float, high: float) -> float: + return max(low, min(high, value)) + + +def _number(value: Any, default: float) -> float: + try: + number = float(value) + return number if math.isfinite(number) else default + except (TypeError, ValueError): + return default + + +def _integer(value: Any, default: int) -> int: + try: + return int(value) + except (TypeError, ValueError): + return default + + +def _slug(value: Any, fallback: str) -> str: + text = re.sub(r"[^a-z0-9]+", "-", str(value or "").lower()).strip("-") + return text or fallback + + +def _color(value: Any, default: str) -> str: + text = str(value or "") + return text if _HEX.fullmatch(text) else default + + +def _vec(value: Any, length: int, default: list[float]) -> list[float]: + if not isinstance(value, (list, tuple)) or len(value) != length: + return list(default) + return [_number(component, default[index]) for index, component in enumerate(value)] + + +class ThreeJSWorld(BaseTool): + """Build and validate an editable semantic world workspace.""" + + name = "threejs_world" + version = "0.2.0" + tier = ToolTier.GENERATE + capability = "3d_world_generation" + provider = "threejs" + stability = ToolStability.BETA + execution_mode = ExecutionMode.SYNC + determinism = Determinism.SEEDED + runtime = ToolRuntime.LOCAL + dependencies: list[str] = [] + install_instructions = ( + "World authoring is dependency-free. Final rendering requires the configured " + "HyperFrames runtime (Node.js >= 22, npx, and FFmpeg)." + ) + agent_skills = ["threejs-world-generation"] + capabilities = [ + "semantic_region_planning", + "procedural_height_field", + "region_aware_asset_scattering", + "explicit_landmark_placement", + "deterministic_camera_flythrough", + "semantic_and_wireframe_diagnostics", + "hyperframes_atelier_workspace", + "licensed_gltf_asset_palette", + "production_fidelity_gate", + "pbr_terrain_material_contract", + ] + best_for = [ + "Editable cinematic 3D worlds and terrain fly-throughs", + "Free-viewpoint environments built without paid generation APIs", + "Region-aware terrain, biomes, landmarks, and diagnostic passes", + "Production worlds assembled from local licensed GLTF/PBR catalogs", + ] + not_good_for = [ + "Single-view mesh reconstruction without a separately configured provider", + "Articulated characters, physics, navmeshes, or interactive game logic", + "Single isolated product models where a normal Three.js scene is simpler", + ] + input_schema = { + "type": "object", + "required": ["operation", "world_spec"], + "properties": { + "operation": {"type": "string", "enum": ["build", "validate"]}, + "world_spec": {"type": "object"}, + "output_path": {"type": "string"}, + "duration_seconds": {"type": "number", "minimum": 1, "maximum": 600}, + "width": {"type": "integer", "minimum": 320, "maximum": 7680}, + "height": {"type": "integer", "minimum": 240, "maximum": 4320}, + "render_mode": { + "type": "string", + "enum": ["cinematic", "semantic", "wireframe"], + }, + "quality_tier": {"type": "string", "enum": ["blockout", "production"]}, + "asset_catalog_paths": {"type": "array", "items": {"type": "string"}}, + }, + } + output_schema = { + "type": "object", + "properties": { + "workspace": {"type": "string"}, + "entry": {"type": "string"}, + "world_spec": {"type": "object"}, + "report": {"type": "object"}, + }, + } + artifact_schema = {"artifact": "3d_world"} + resource_profile = ResourceProfile( + cpu_cores=2, ram_mb=1024, vram_mb=1024, disk_mb=2000, network_required=True + ) + idempotency_key_fields = ["operation", "world_spec", "duration_seconds", "render_mode", "quality_tier", "asset_catalog_paths"] + side_effects = [ + "writes an editable HyperFrames/Three.js workspace to output_path", + "writes normalized world and diagnostic JSON files", + ] + fallback_tools: list[str] = [] + user_visible_verification = [ + "Inspect semantic, regional, and walk-level snapshots before final render", + "Verify landmark contact, camera clearance, and stable region identities", + "Open index.html with HyperFrames preview to explore the authored camera path", + ] + + def execute(self, inputs: dict[str, Any]) -> ToolResult: + started = time.time() + operation = str(inputs.get("operation", "")) + duration = _clamp(_number(inputs.get("duration_seconds"), 60.0), 1.0, 600.0) + width = int(_clamp(_integer(inputs.get("width"), 1920), 320, 7680)) + height = int(_clamp(_integer(inputs.get("height"), 1080), 240, 4320)) + render_mode = str(inputs.get("render_mode") or "cinematic").lower() + if render_mode not in _RENDER_MODES: + return ToolResult(success=False, error=f"Unknown render_mode: {render_mode}") + quality_tier = str(inputs.get("quality_tier") or "blockout").lower() + if quality_tier not in _QUALITY_TIERS: + return ToolResult(success=False, error=f"Unknown quality_tier: {quality_tier}") + catalog_paths = [Path(str(path)).expanduser().resolve() for path in inputs.get("asset_catalog_paths") or []] + + spec, normalize_warnings = self._normalize_spec( + inputs.get("world_spec") or {}, duration=duration + ) + report = self._report(spec, duration=duration, warnings=normalize_warnings) + report["quality_tier"] = quality_tier + report["asset_catalog_paths"] = [str(path) for path in catalog_paths] + fidelity_errors, fidelity_warnings = self._fidelity_gate(spec, quality_tier, catalog_paths) + report["errors"].extend(fidelity_errors) + report["warnings"].extend(fidelity_warnings) + + if operation == "validate": + return ToolResult( + success=not report["errors"], + data={"world_spec": spec, "report": report}, + error="; ".join(report["errors"]) if report["errors"] else None, + duration_seconds=round(time.time() - started, 2), + seed=spec["seed"], + model=f"threejs-world-{quality_tier}-v2", + ) + + if operation != "build": + return ToolResult(success=False, error=f"Unknown operation: {operation}") + if report["errors"]: + return ToolResult( + success=False, + data={"world_spec": spec, "report": report}, + error="World specification failed validation: " + "; ".join(report["errors"]), + ) + + output_raw = inputs.get("output_path") + if not output_raw: + return ToolResult(success=False, error="output_path is required for operation='build'") + workspace = Path(str(output_raw)).expanduser().resolve() + + try: + artifacts = self._write_workspace( + workspace=workspace, + spec=spec, + report=report, + duration=duration, + width=width, + height=height, + render_mode=render_mode, + quality_tier=quality_tier, + catalog_paths=catalog_paths, + ) + except Exception as exc: + return ToolResult(success=False, error=f"3D world build failed: {exc}") + + return ToolResult( + success=True, + data={ + "workspace": str(workspace), + "entry": str(workspace / "index.html"), + "world_spec": spec, + "report": report, + "render_mode": render_mode, + "duration_seconds": duration, + "width": width, + "height": height, + }, + artifacts=artifacts, + duration_seconds=round(time.time() - started, 2), + seed=spec["seed"], + model=f"threejs-world-{quality_tier}-v2", + ) + + @staticmethod + def _fidelity_gate( + spec: dict[str, Any], quality_tier: str, catalog_paths: list[Path] + ) -> tuple[list[str], list[str]]: + if quality_tier == "blockout": + return [], [ + "Blockout tier may use procedural primitives and flat materials; " + "do not present it as reference-grade or production-fidelity output." + ] + + errors: list[str] = [] + warnings: list[str] = [] + manifests: list[dict[str, Any]] = [] + for catalog_path in catalog_paths: + manifest_path = catalog_path / "catalog-manifest.json" + if not manifest_path.is_file(): + errors.append(f"Production catalog manifest missing: {manifest_path}") + continue + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + errors.append(f"Production catalog manifest unreadable: {manifest_path}: {exc}") + continue + if manifest.get("license") not in {"CC0", "CC0-1.0"}: + errors.append(f"Catalog {manifest_path} lacks an approved CC0 license declaration.") + if int(manifest.get("model_count") or 0) <= 0: + errors.append(f"Catalog {manifest_path} contains no GLTF/GLB models.") + manifests.append(manifest) + + asset_palette = spec.get("asset_palette") or [] + terrain_materials = spec.get("terrain_materials") or [] + if not catalog_paths: + errors.append("Production tier requires at least one installed asset catalog path.") + if len(asset_palette) < 8: + errors.append("Production tier requires at least 8 distinct asset-palette entries.") + if len(terrain_materials) < 3: + errors.append("Production tier requires at least 3 terrain material layers.") + if any(not item.get("catalog_id") or not item.get("model_id") for item in asset_palette): + errors.append("Every production asset-palette entry requires catalog_id and model_id.") + if any(not item.get("base_color") or not item.get("normal") or not item.get("roughness") for item in terrain_materials): + errors.append("Every production terrain material requires base_color, normal, and roughness maps.") + + unique_categories = {str(item.get("category") or "") for item in asset_palette} + if len(unique_categories - {""}) < 4: + errors.append("Production asset palette requires at least 4 semantic categories.") + if len(manifests) == 1: + warnings.append("Only one asset catalog is installed; repetition must be checked at walk level.") + return errors, warnings + + @classmethod + def _normalize_spec( + cls, raw: dict[str, Any], *, duration: float + ) -> tuple[dict[str, Any], list[str]]: + source = copy.deepcopy(raw) if isinstance(raw, dict) else {} + warnings: list[str] = [] + world_raw = source.get("world") if isinstance(source.get("world"), dict) else {} + atmosphere_raw = ( + source.get("atmosphere") if isinstance(source.get("atmosphere"), dict) else {} + ) + terrain_materials_raw = source.get("terrain_materials") if isinstance(source.get("terrain_materials"), list) else [] + asset_palette_raw = source.get("asset_palette") if isinstance(source.get("asset_palette"), list) else [] + + world = { + "size": _clamp(_number(world_raw.get("size"), 120.0), 24.0, 500.0), + "resolution": int( + _clamp(_integer(world_raw.get("resolution"), 144), 24, 256) + ), + "elevation_scale": _clamp( + _number(world_raw.get("elevation_scale"), 16.0), 1.0, 80.0 + ), + "water_level": _clamp( + _number(world_raw.get("water_level"), -2.0), -60.0, 60.0 + ), + } + atmosphere = { + "sky_color": _color(atmosphere_raw.get("sky_color"), "#07111f"), + "fog_color": _color(atmosphere_raw.get("fog_color"), "#13263a"), + "fog_density": _clamp( + _number(atmosphere_raw.get("fog_density"), 0.008), 0.0, 0.08 + ), + "sun_color": _color(atmosphere_raw.get("sun_color"), "#ffd7a3"), + "sun_intensity": _clamp( + _number(atmosphere_raw.get("sun_intensity"), 3.0), 0.0, 12.0 + ), + "sun_position": _vec( + atmosphere_raw.get("sun_position"), 3, [45.0, 70.0, 20.0] + ), + "ground_color": _color(atmosphere_raw.get("ground_color"), "#151c23"), + } + + palette = ["#315b48", "#73523d", "#2d5968", "#6f4b78", "#8b753f"] + accent_palette = ["#8ee6b1", "#ff9a62", "#64d8ff", "#d4a8ff", "#ffe27a"] + regions: list[dict[str, Any]] = [] + raw_regions = source.get("regions") if isinstance(source.get("regions"), list) else [] + for index, item in enumerate(raw_regions[:12]): + item = item if isinstance(item, dict) else {} + region_id = _slug(item.get("id") or item.get("label"), f"region-{index + 1}") + landform = str(item.get("landform") or "plain").lower() + if landform not in _LANDFORMS: + warnings.append( + f"Region {region_id}: unknown landform {landform!r}; using 'plain'." + ) + landform = "plain" + scatter_raw = item.get("scatter") if isinstance(item.get("scatter"), dict) else {} + center = _vec(item.get("center"), 2, [0.0, 0.0]) + center = [_clamp(center[0], -1.0, 1.0), _clamp(center[1], -1.0, 1.0)] + regions.append( + { + "id": region_id, + "label": str(item.get("label") or region_id.replace("-", " ").title()), + "center": center, + "radius": _clamp(_number(item.get("radius"), 0.75), 0.12, 2.5), + "base_elevation": _clamp( + _number(item.get("base_elevation"), 0.0), -2.0, 2.0 + ), + "amplitude": _clamp(_number(item.get("amplitude"), 0.65), 0.0, 2.5), + "frequency": _clamp(_number(item.get("frequency"), 1.0), 0.15, 8.0), + "landform": landform, + "blend_width": _clamp( + _number(item.get("blend_width"), 0.22), 0.02, 1.0 + ), + "color": _color(item.get("color"), palette[index % len(palette)]), + "accent_color": _color( + item.get("accent_color"), accent_palette[index % len(accent_palette)] + ), + "scatter": { + "tree": int( + _clamp(_integer(scatter_raw.get("tree"), 0), 0, 1200) + ), + "rock": int( + _clamp(_integer(scatter_raw.get("rock"), 35), 0, 1200) + ), + "crystal": int( + _clamp(_integer(scatter_raw.get("crystal"), 0), 0, 1200) + ), + }, + "slope_limit": _clamp( + _number(item.get("slope_limit"), 1.8), 0.1, 12.0 + ), + } + ) + + landmarks: list[dict[str, Any]] = [] + raw_landmarks = ( + source.get("landmarks") if isinstance(source.get("landmarks"), list) else [] + ) + fallback_region = regions[0]["id"] if regions else "" + for index, item in enumerate(raw_landmarks[:80]): + item = item if isinstance(item, dict) else {} + landmark_id = _slug(item.get("id"), f"landmark-{index + 1}") + kind = str(item.get("type") or "monolith").lower() + if kind not in _LANDMARKS: + warnings.append( + f"Landmark {landmark_id}: unknown type {kind!r}; using 'monolith'." + ) + kind = "monolith" + landmarks.append( + { + "id": landmark_id, + "type": kind, + "region_id": _slug(item.get("region_id"), fallback_region), + "position": _vec(item.get("position"), 3, [0.0, 0.0, 0.0]), + "rotation": _vec(item.get("rotation"), 3, [0.0, 0.0, 0.0]), + "scale": _clamp(_number(item.get("scale"), 4.0), 0.2, 30.0), + "color": _color(item.get("color"), "#30343b"), + "accent_color": _color(item.get("accent_color"), "#74e5ff"), + } + ) + + camera_path: list[dict[str, Any]] = [] + raw_camera = ( + source.get("camera_path") if isinstance(source.get("camera_path"), list) else [] + ) + for index, item in enumerate(raw_camera[:40]): + item = item if isinstance(item, dict) else {} + default_time = (duration * index / max(1, len(raw_camera) - 1)) if raw_camera else 0 + camera_path.append( + { + "time": _clamp(_number(item.get("time"), default_time), 0.0, duration), + "position": _vec(item.get("position"), 3, [60.0, 35.0, 60.0]), + "target": _vec(item.get("target"), 3, [0.0, 0.0, 0.0]), + "fov": _clamp(_number(item.get("fov"), 45.0), 18.0, 90.0), + "label": str(item.get("label") or ""), + } + ) + camera_path.sort(key=lambda key: key["time"]) + + spec = { + "version": str(source.get("version") or "1.0"), + "title": str(source.get("title") or "Untitled Three.js World"), + "seed": _integer(source.get("seed"), 1337), + "explicit_constraints": [ + str(value) + for value in source.get("explicit_constraints", []) + if str(value).strip() + ] + if isinstance(source.get("explicit_constraints"), list) + else [], + "inferred_details": [ + str(value) + for value in source.get("inferred_details", []) + if str(value).strip() + ] + if isinstance(source.get("inferred_details"), list) + else [], + "world": world, + "atmosphere": atmosphere, + "terrain_materials": [copy.deepcopy(item) for item in terrain_materials_raw if isinstance(item, dict)], + "asset_palette": [copy.deepcopy(item) for item in asset_palette_raw if isinstance(item, dict)], + "regions": regions, + "landmarks": landmarks, + "camera_path": camera_path, + } + return spec, warnings + + @classmethod + def _report( + cls, spec: dict[str, Any], *, duration: float, warnings: list[str] + ) -> dict[str, Any]: + errors: list[str] = [] + warnings = list(warnings) + regions = spec["regions"] + landmarks = spec["landmarks"] + camera_path = spec["camera_path"] + + if not regions: + errors.append("At least one semantic region is required.") + region_ids = [region["id"] for region in regions] + if len(region_ids) != len(set(region_ids)): + errors.append("Region IDs must be unique.") + landmark_ids = [landmark["id"] for landmark in landmarks] + if len(landmark_ids) != len(set(landmark_ids)): + errors.append("Landmark IDs must be unique.") + for landmark in landmarks: + if landmark["region_id"] not in set(region_ids): + errors.append( + f"Landmark {landmark['id']} references unknown region " + f"{landmark['region_id']!r}." + ) + + if len(camera_path) < 2: + errors.append("Camera path requires at least two time keys.") + else: + if abs(camera_path[0]["time"]) > 1e-6: + errors.append("First camera key must start at time 0.") + if abs(camera_path[-1]["time"] - duration) > 1e-3: + errors.append( + f"Last camera key must end at duration {duration:g} seconds." + ) + times = [key["time"] for key in camera_path] + if any(right <= left for left, right in zip(times, times[1:])): + errors.append("Camera key times must be strictly increasing.") + + size = spec["world"]["size"] + half = size / 2.0 + for landmark in landmarks: + x, _, z = landmark["position"] + if abs(x) > half or abs(z) > half: + warnings.append(f"Landmark {landmark['id']} is outside world bounds.") + + coverage: dict[str, int] = {region_id: 0 for region_id in region_ids} + if regions: + for iz in range(15): + for ix in range(15): + x = (ix / 14.0) * 2.0 - 1.0 + z = (iz / 14.0) * 2.0 - 1.0 + weights = cls._region_weights(spec, x, z) + winner = max(range(len(weights)), key=weights.__getitem__) + coverage[regions[winner]["id"]] += 1 + for region_id, samples in coverage.items(): + if samples == 0: + warnings.append( + f"Region {region_id} never dominates the sampled semantic layout." + ) + + min_clearance: float | None = None + if len(camera_path) >= 2 and regions: + for sample_index in range(121): + sample_time = duration * sample_index / 120.0 + position = cls._interpolate_camera(camera_path, sample_time) + terrain_y = cls._height_at(spec, position[0], position[2]) + clearance = position[1] - terrain_y + min_clearance = clearance if min_clearance is None else min(min_clearance, clearance) + if min_clearance is not None and min_clearance < 2.0: + warnings.append( + f"Camera path minimum terrain clearance is {min_clearance:.2f}; " + "review for clipping." + ) + + resolution = spec["world"]["resolution"] + instance_count = sum(sum(region["scatter"].values()) for region in regions) + return { + "valid": not errors, + "errors": errors, + "warnings": warnings, + "stats": { + "region_count": len(regions), + "landmark_count": len(landmarks), + "camera_key_count": len(camera_path), + "terrain_triangles": resolution * resolution * 2, + "environment_instances": instance_count, + "semantic_coverage_samples": coverage, + "minimum_camera_clearance": ( + round(min_clearance, 3) if min_clearance is not None else None + ), + }, + "review_views": ["global", "regional", "walk", "semantic", "wireframe"], + "diagnostic_passes": { + "cinematic": "lit beauty render for final review", + "semantic": "stable region-color pass for layout review", + "wireframe": "explicit terrain and asset geometry pass", + }, + } + + @classmethod + def _region_weights(cls, spec: dict[str, Any], nx: float, nz: float) -> list[float]: + raw: list[float] = [] + for region in spec["regions"]: + dx = nx - region["center"][0] + dz = nz - region["center"][1] + radius = max(0.05, region["radius"]) + distance = math.sqrt(dx * dx + dz * dz) / radius + softness = max(0.02, region["blend_width"]) + value = math.exp(-max(0.0, distance - 0.05) ** 2 / (softness * 2.8)) + raw.append(max(1e-5, value)) + total = sum(raw) or 1.0 + return [value / total for value in raw] + + @classmethod + def _height_at(cls, spec: dict[str, Any], x: float, z: float) -> float: + size = spec["world"]["size"] + nx = x / (size / 2.0) + nz = z / (size / 2.0) + weights = cls._region_weights(spec, nx, nz) + seed = spec["seed"] * 0.01337 + elevation = 0.0 + for index, (region, weight) in enumerate(zip(spec["regions"], weights)): + frequency = region["frequency"] + noise = ( + math.sin((nx * 3.1 + seed + index) * frequency * math.pi) + + math.cos((nz * 2.7 - seed * 0.7 + index) * frequency * math.pi) + + 0.5 + * math.sin((nx + nz) * frequency * 7.3 + seed * 3.0 + index) + ) / 2.5 + dx = nx - region["center"][0] + dz = nz - region["center"][1] + distance = math.sqrt(dx * dx + dz * dz) / max(0.05, region["radius"]) + landform = cls._landform(region["landform"], dx, dz, distance) + elevation += weight * ( + region["base_elevation"] + + region["amplitude"] * (noise * 0.48 + landform * 0.8) + ) + return elevation * spec["world"]["elevation_scale"] + + @staticmethod + def _landform(kind: str, dx: float, dz: float, distance: float) -> float: + if kind == "peak": + return max(0.0, 1.0 - distance) ** 2.2 + if kind == "ridge": + return max(0.0, 1.0 - abs(dx * 1.8 + math.sin(dz * 5.0) * 0.16)) + if kind == "dune": + return (math.sin((dx + dz * 0.25) * 18.0) + 1.0) * 0.24 + if kind == "terrace": + return math.floor(max(0.0, 1.0 - distance) * 5.0) / 5.0 + if kind == "basin": + return -max(0.0, 1.0 - distance) ** 1.7 + if kind == "canyon": + return -max(0.0, 1.0 - abs(dx + math.sin(dz * 7.0) * 0.1)) ** 2.0 + return 0.0 + + @staticmethod + def _interpolate_camera(camera_path: list[dict[str, Any]], time_value: float) -> list[float]: + if time_value <= camera_path[0]["time"]: + return list(camera_path[0]["position"]) + if time_value >= camera_path[-1]["time"]: + return list(camera_path[-1]["position"]) + for left, right in zip(camera_path, camera_path[1:]): + if left["time"] <= time_value <= right["time"]: + span = max(1e-6, right["time"] - left["time"]) + t = _clamp((time_value - left["time"]) / span, 0.0, 1.0) + smooth = t * t * (3.0 - 2.0 * t) + return [ + left["position"][axis] + + (right["position"][axis] - left["position"][axis]) * smooth + for axis in range(3) + ] + return list(camera_path[-1]["position"]) + + @staticmethod + def _write_workspace( + *, + workspace: Path, + spec: dict[str, Any], + report: dict[str, Any], + duration: float, + width: int, + height: int, + render_mode: str, + quality_tier: str, + catalog_paths: list[Path], + ) -> list[str]: + template_dir = Path(__file__).resolve().parent / "templates" / "threejs_world" + required = ["index.html", "world.css", "world-runtime.js"] + missing = [name for name in required if not (template_dir / name).is_file()] + if missing: + raise FileNotFoundError(f"Missing Three.js world templates: {', '.join(missing)}") + + workspace.mkdir(parents=True, exist_ok=True) + (workspace / "assets").mkdir(exist_ok=True) + (workspace / "renders").mkdir(exist_ok=True) + + catalog_index: dict[str, Any] = {"version": "1.0", "catalogs": []} + model_root = workspace / "assets" / "models" + model_root.mkdir(parents=True, exist_ok=True) + for catalog_path in catalog_paths: + manifest_path = catalog_path / "catalog-manifest.json" + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + catalog_id = str(manifest["catalog_id"]) + target = model_root / catalog_id + source = catalog_path / "source" + if target.exists(): + shutil.rmtree(target) + shutil.copytree(source, target) + copied = copy.deepcopy(manifest) + for model in copied.get("models", []): + original = Path(model["path"]) + relative_inside_source = Path(*original.parts[1:]) if original.parts and original.parts[0] == "source" else original + model["runtime_path"] = (Path("assets") / "models" / catalog_id / relative_inside_source).as_posix() + copied.pop("download_url", None) + catalog_index["catalogs"].append(copied) + + index_template = (template_dir / "index.html").read_text(encoding="utf-8") + index_html = ( + index_template.replace("__TITLE__", html.escape(spec["title"], quote=True)) + .replace("__DURATION__", f"{duration:g}") + .replace("__WIDTH__", str(width)) + .replace("__HEIGHT__", str(height)) + .replace("__RENDER_MODE__", render_mode) + .replace("__QUALITY_TIER__", quality_tier) + ) + + index_path = workspace / "index.html" + css_path = workspace / "world.css" + runtime_path = workspace / "world-runtime.js" + world_json_path = workspace / "world.json" + world_js_path = workspace / "world-spec.js" + report_path = workspace / "world-report.json" + catalog_index_path = workspace / "asset-catalog-index.json" + catalog_js_path = workspace / "asset-catalog.js" + config_path = workspace / "hyperframes.json" + + index_path.write_text(index_html, encoding="utf-8") + shutil.copyfile(template_dir / "world.css", css_path) + shutil.copyfile(template_dir / "world-runtime.js", runtime_path) + world_json_path.write_text(json.dumps(spec, indent=2), encoding="utf-8") + world_js_path.write_text( + "export const WORLD_SPEC = " + json.dumps(spec, indent=2) + ";\n", + encoding="utf-8", + ) + report_path.write_text(json.dumps(report, indent=2), encoding="utf-8") + catalog_index_path.write_text(json.dumps(catalog_index, indent=2), encoding="utf-8") + catalog_js_path.write_text( + "export const ASSET_CATALOG = " + json.dumps(catalog_index, indent=2) + ";\n", + encoding="utf-8", + ) + config_path.write_text( + json.dumps( + { + "registry": ( + "https://raw.githubusercontent.com/heygen-com/hyperframes/main/registry" + ), + "paths": { + "blocks": "compositions", + "components": "compositions/components", + "assets": "assets", + }, + }, + indent=2, + ), + encoding="utf-8", + ) + + return [ + str(index_path), + str(css_path), + str(runtime_path), + str(world_json_path), + str(world_js_path), + str(report_path), + str(catalog_index_path), + str(catalog_js_path), + str(config_path), + ] diff --git a/tools/video/corpus_builder.py b/tools/video/corpus_builder.py index f426b40c..cb52ff6a 100644 --- a/tools/video/corpus_builder.py +++ b/tools/video/corpus_builder.py @@ -392,6 +392,38 @@ class CorpusBuilder(BaseTool): except Exception as e: cache_snapshot = {"error": f"{type(e).__name__}: {e}"} + # Per-candidate tolerance is useful only while at least one item + # survives. If every discovered candidate fails, reporting success + # persists an empty index and hides a systemic codec/CLIP failure + # until retrieval. A no-result or skip-only run remains valid. + total_failure = bool(candidates_seen) and not added_ids and not skipped + if total_failure: + first_errors = "; ".join( + item["error"] + for item in errors + if item.get("phase") == "process" + )[:400] + return ToolResult( + success=False, + error=( + f"All {failed} of {candidates_seen} candidates failed to " + "process; corpus index is empty. Check the media decoder " + "and the CLIP `transformers`/`torch` compatibility. " + f"First errors: {first_errors or '(none recorded)'}" + ), + data={ + "corpus_dir": str(corpus_dir), + "queries_run": len(queries), + "candidates_seen": candidates_seen, + "clips_added": 0, + "clips_skipped_existing": skipped, + "clips_failed": failed, + "total_corpus_size": len(corp), + "errors": errors[:25], + }, + duration_seconds=round(elapsed, 2), + ) + return ToolResult( success=True, data={ diff --git a/tools/video/green_screen_processor.py b/tools/video/green_screen_processor.py index de074589..fdc9b7bf 100644 --- a/tools/video/green_screen_processor.py +++ b/tools/video/green_screen_processor.py @@ -161,7 +161,7 @@ class GreenScreenProcessor(BaseTool): if method == "chromakey": ok = self._process_chromakey( - frames_dir, processed_dir, bg_color, frame_count + frames_dir, processed_dir, bg_color, frame_count, width, height ) else: ok = self._process_rembg( @@ -445,6 +445,8 @@ class GreenScreenProcessor(BaseTool): processed_dir: Path, bg_color: str, frame_count: int, + width: int, + height: int, ) -> bool: """Process frames using FFmpeg chromakey filter. @@ -461,13 +463,23 @@ class GreenScreenProcessor(BaseTool): out_path = processed_dir / frame.name cmd = [ "ffmpeg", "-y", - "-f", "lavfi", "-i", f"color=c={ffmpeg_bg}:size=1x1", + # Background sized to the frame up front. The old code used a 1x1 + # color source and `[0:v]scale=iw:ih` — a no-op (iw/ih were the + # 1x1 source's own size) — and overlay takes the size of its + # FIRST input, so every frame was clipped to a single pixel and + # the whole video came out a solid color with the subject gone. + "-f", "lavfi", "-i", f"color=c={ffmpeg_bg}:size={width}x{height}", "-i", str(frame), "-filter_complex", ( - f"[0:v]scale=iw:ih[bg];" - f"[1:v]chromakey=color=0x00FF00:similarity=0.3:blend=0.08[fg];" - f"[bg][fg]overlay=0:0" + # Force an explicit alpha format after chromakey so the keyed + # transparency survives filter negotiation on every FFmpeg + # build — without it, some Linux builds carry the keyed frame + # forward without an alpha plane and overlay draws opaque + # green over the background instead of compositing. + f"[1:v]chromakey=color=0x00FF00:similarity=0.3:blend=0.08," + f"format=yuva420p[fg];" + f"[0:v][fg]overlay=0:0:format=auto,format=yuv420p" ), "-frames:v", "1", str(out_path), diff --git a/tools/video/hyperframes_compose.py b/tools/video/hyperframes_compose.py index 330d9c78..351529d1 100644 --- a/tools/video/hyperframes_compose.py +++ b/tools/video/hyperframes_compose.py @@ -2,10 +2,11 @@ Sibling to `video_compose` (FFmpeg + Remotion). This tool owns the HyperFrames runtime end-to-end: workspace materialization, `hyperframes lint`, -`hyperframes validate`, and `hyperframes render`. It is invoked by +`hyperframes check`, and `hyperframes render`. It is invoked by `video_compose` when `edit_decisions.render_runtime == "hyperframes"`, and can also be called directly by pipelines that want HyperFrames-specific -operations (lint-only, validate-only, scaffold-only). +operations (check/lint/validate/inspect, scaffold-only, or an +existing-workspace atelier render that preserves authored HTML). This tool deliberately does NOT attempt parity with every Remotion scene component. See `skills/core/hyperframes.md` for what is in scope in Phase 1 @@ -49,7 +50,7 @@ _AUDIO_EXTENSIONS = {".mp3", ".wav", ".m4a", ".aac", ".ogg", ".flac"} class HyperFramesCompose(BaseTool): name = "hyperframes_compose" - version = "0.1.0" + version = "0.2.0" tier = ToolTier.CORE capability = "video_post" provider = "hyperframes" @@ -72,7 +73,7 @@ class HyperFramesCompose(BaseTool): "hyperframes", "hyperframes-cli", "hyperframes-registry", - "website-to-hyperframes", + "website-to-video", "gsap-core", "gsap-timeline", ] @@ -81,8 +82,11 @@ class HyperFramesCompose(BaseTool): "hyperframes_render", "hyperframes_lint", "hyperframes_validate", + "hyperframes_inspect", + "hyperframes_check", "hyperframes_doctor", "scaffold_workspace", + "render_existing_workspace", "add_block", ] @@ -91,6 +95,7 @@ class HyperFramesCompose(BaseTool): "Motion-graphics-heavy briefs where the scene library in remotion-composer/ doesn't fit", "Website-to-video / UI-driven compositions", "Registry-block-driven scenes (hyperframes add data-chart, grain-overlay, etc.)", + "Hand-authored atelier workspaces, including deterministic Three.js worlds", ] not_good_for = [ "Word-level caption burn (stays on Remotion in Phase 1)", @@ -107,16 +112,22 @@ class HyperFramesCompose(BaseTool): "type": "string", "enum": [ "render", + "render_existing", "lint", "validate", + "inspect", + "check", "doctor", "scaffold_workspace", "add_block", ], "description": ( "render: materialize workspace + lint + validate + render to MP4. " + "render_existing: preserve an authored index.html, then check + render it. " "lint: run `hyperframes lint` on an existing workspace. " "validate: run `hyperframes validate` (browser-based). " + "inspect: seek an existing workspace and audit layout/runtime issues. " + "check: run the current unified lint/runtime/layout/motion/contrast gate. " "doctor: run `hyperframes doctor` to check environment. " "scaffold_workspace: materialize HTML/CSS/assets but do not render. " "add_block: run `hyperframes add ` to install a registry " @@ -141,7 +152,7 @@ class HyperFramesCompose(BaseTool): }, "output_path": { "type": "string", - "description": "Output MP4 path. Used by operation='render'.", + "description": "Output MP4 path. Used by render and render_existing.", }, "edit_decisions": { "type": "object", @@ -191,15 +202,25 @@ class HyperFramesCompose(BaseTool): "type": "boolean", "default": False, "description": ( - "Skip the WCAG contrast audit during validate. Acceptable " + "Skip the WCAG contrast audit during check. Acceptable " "while iterating; forbidden for final delivery." ), }, + "strict_check": { + "type": "boolean", + "default": False, + "description": "Treat HyperFrames check warnings as errors.", + }, + "snapshots": { + "type": "boolean", + "default": False, + "description": "Save representative quality-check snapshots.", + }, }, } resource_profile = ResourceProfile( - cpu_cores=4, ram_mb=3072, vram_mb=0, disk_mb=2000, network_required=False + cpu_cores=4, ram_mb=3072, vram_mb=0, disk_mb=2000, network_required=True ) retry_policy = RetryPolicy(max_retries=0) resume_support = ResumeSupport.FROM_START @@ -226,6 +247,7 @@ class HyperFramesCompose(BaseTool): # We cache per-process so the first call pays ~2-5s and subsequent calls # (get_info spam from the registry) are free. _npm_resolve_cache: Optional[dict[str, str]] = None + _cli_probe_cache: Optional[dict[str, str]] = None @classmethod def _node_major_version(cls) -> Optional[int]: @@ -301,6 +323,45 @@ class HyperFramesCompose(BaseTool): cls._npm_resolve_cache = {"version": version} return cls._npm_resolve_cache + @classmethod + def _probe_cli(cls) -> dict[str, str]: + """Run the published CLI's doctor command once per process. + + Package resolution alone does not prove that the executable can start: + an upstream packaging regression can publish successfully while every + CLI command crashes during bootstrap. Provider preflight must not call + that state available. + """ + if cls._cli_probe_cache is not None: + return cls._cli_probe_cache + + npx = shutil.which("npx") + if not npx: + cls._cli_probe_cache = {"error": "npx not on PATH"} + return cls._cli_probe_cache + + try: + proc = subprocess.run( + [npx, "--yes", cls._NPM_PACKAGE, "doctor", "--json"], + capture_output=True, + text=True, + timeout=20, + ) + except subprocess.TimeoutExpired: + cls._cli_probe_cache = {"error": "doctor timed out after 20s"} + return cls._cli_probe_cache + except (OSError, subprocess.SubprocessError) as exc: + cls._cli_probe_cache = {"error": f"doctor failed: {type(exc).__name__}"} + return cls._cli_probe_cache + + if proc.returncode != 0: + output = "\n".join(filter(None, [proc.stderr, proc.stdout])).strip() + tail = output.splitlines()[-1][:200] if output else f"exit {proc.returncode}" + cls._cli_probe_cache = {"error": f"doctor failed: {tail}"} + else: + cls._cli_probe_cache = {"status": "ok"} + return cls._cli_probe_cache + def _runtime_check(self) -> dict[str, Any]: """Return availability state for the HyperFrames runtime. @@ -336,6 +397,12 @@ class HyperFramesCompose(BaseTool): f"{npm_resolve['error']}" ) + cli_probe: dict[str, str] = {} + if not reasons: + cli_probe = self._probe_cli() + if "error" in cli_probe: + reasons.append(f"published CLI is not executable: {cli_probe['error']}") + return { "runtime_available": not reasons, "node_major": node_major, @@ -344,6 +411,8 @@ class HyperFramesCompose(BaseTool): "npm_package": self._NPM_PACKAGE, "npm_package_version": npm_resolve.get("version"), "npm_resolve_error": npm_resolve.get("error"), + "cli_probe_status": cli_probe.get("status"), + "cli_probe_error": cli_probe.get("error"), "reasons": reasons, } @@ -399,8 +468,14 @@ class HyperFramesCompose(BaseTool): result = self._lint(inputs) elif operation == "validate": result = self._validate(inputs) + elif operation == "inspect": + result = self._inspect(inputs) + elif operation == "check": + result = self._check(inputs) elif operation == "render": result = self._render(inputs) + elif operation == "render_existing": + result = self._render_existing(inputs) elif operation == "add_block": result = self._add_block(inputs) else: @@ -592,6 +667,56 @@ class HyperFramesCompose(BaseTool): error=None if ok else f"hyperframes validate exit {proc.returncode}", ) + def _inspect(self, inputs: dict[str, Any]) -> ToolResult: + """Seek through an authored workspace and audit runtime/layout issues.""" + workspace = self._require_workspace(inputs) + if not (workspace / "index.html").exists(): + return ToolResult( + success=False, + error=f"No index.html in {workspace}.", + ) + proc = self._run_hf(["inspect", "--json"], cwd=workspace, timeout=300, check=False) + data: dict[str, Any] = {"exit_code": proc.returncode} + payload = self._parse_json_output(proc.stdout) + if payload is not None: + data["report"] = payload + else: + data["stdout_tail"] = (proc.stdout or "")[-4000:] + data["stderr_tail"] = (proc.stderr or "")[-2000:] + ok = proc.returncode == 0 + return ToolResult( + success=ok, + data=data, + error=None if ok else f"hyperframes inspect exit {proc.returncode}", + ) + + def _check(self, inputs: dict[str, Any]) -> ToolResult: + """Run the unified HyperFrames quality gate for authored workspaces.""" + workspace = self._require_workspace(inputs) + if not (workspace / "index.html").exists(): + return ToolResult(success=False, error=f"No index.html in {workspace}.") + args = ["check", "--json"] + if inputs.get("skip_contrast", False): + args.append("--no-contrast") + if inputs.get("strict_check", False): + args.append("--strict") + if inputs.get("snapshots", False): + args.append("--snapshots") + proc = self._run_hf(args, cwd=workspace, timeout=300, check=False) + data: dict[str, Any] = {"exit_code": proc.returncode} + payload = self._parse_json_output(proc.stdout) + if payload is not None: + data["report"] = payload + else: + data["stdout_tail"] = (proc.stdout or "")[-4000:] + data["stderr_tail"] = (proc.stderr or "")[-2000:] + ok = proc.returncode == 0 + return ToolResult( + success=ok, + data=data, + error=None if ok else f"hyperframes check exit {proc.returncode}", + ) + def _add_block(self, inputs: dict[str, Any]) -> ToolResult: """Install a registry block or component via `hyperframes add`. @@ -654,7 +779,9 @@ class HyperFramesCompose(BaseTool): ) workspace = self._require_workspace(inputs) - output_path = Path(inputs.get("output_path") or (workspace / "renders" / "final.mp4")) + output_path = Path( + inputs.get("output_path") or (workspace / "renders" / "final.mp4") + ).expanduser().resolve() output_path.parent.mkdir(parents=True, exist_ok=True) steps: dict[str, Any] = {} @@ -748,6 +875,111 @@ class HyperFramesCompose(BaseTool): artifacts=[str(output_path)], ) + def _render_existing(self, inputs: dict[str, Any]) -> ToolResult: + """Validate and render a hand-authored workspace without scaffolding it. + + Atelier compositions own their HTML, CSS, JavaScript, and local assets. + Re-running `_scaffold` would destroy that authored work, so this path + performs the mandatory gates against the files already on disk. + """ + runtime_ok = self._runtime_check() + if not runtime_ok["runtime_available"]: + return ToolResult( + success=False, + error=( + "HyperFrames runtime not available: " + + "; ".join(runtime_ok["reasons"]) + + ". Per governance, do not swap runtimes silently." + ), + data={"runtime_check": runtime_ok}, + ) + + workspace = self._require_workspace(inputs) + entry = workspace / "index.html" + if not entry.is_file(): + return ToolResult( + success=False, + error=f"No authored index.html in {workspace}.", + ) + original_digest = self._file_digest(entry) + output_path = Path( + inputs.get("output_path") or (workspace / "renders" / "final.mp4") + ).expanduser().resolve() + output_path.parent.mkdir(parents=True, exist_ok=True) + steps: dict[str, Any] = {} + + quality_check = self._check( + { + "workspace_path": str(workspace), + "skip_contrast": inputs.get("skip_contrast", False), + "strict_check": inputs.get("strict_check", False), + "snapshots": inputs.get("snapshots", False), + } + ) + steps["check"] = quality_check.data + if not quality_check.success: + return ToolResult( + success=False, + error=f"Quality check failed for authored workspace: {quality_check.error}", + data={"steps": steps}, + ) + + _, _, fps = self._resolve_dimensions( + inputs.get("profile"), inputs.get("fps", 30) + ) + quality = inputs.get("quality", "standard") + args = [ + "render", + "--output", str(output_path), + "--fps", str(fps), + "--quality", quality, + "--strict", + ] + proc = self._run_hf(args, cwd=workspace, timeout=1800, check=False) + steps["render"] = { + "exit_code": proc.returncode, + "stdout_tail": (proc.stdout or "")[-4000:], + "stderr_tail": (proc.stderr or "")[-4000:], + } + if proc.returncode != 0: + return ToolResult( + success=False, + error=f"hyperframes render exit {proc.returncode}", + data={"steps": steps}, + ) + if not output_path.is_file(): + return ToolResult( + success=False, + error=f"HyperFrames exited 0 but output is missing: {output_path}", + data={"steps": steps}, + ) + if self._file_digest(entry) != original_digest: + return ToolResult( + success=False, + error="Authored index.html changed during render_existing.", + data={"steps": steps}, + ) + + return ToolResult( + success=True, + data={ + "operation": "render_existing", + "output": str(output_path), + "workspace": str(workspace), + "fps": fps, + "quality": quality, + "authored_entry_preserved": True, + "steps": steps, + }, + artifacts=[str(output_path)], + ) + + @staticmethod + def _file_digest(path: Path) -> str: + import hashlib + + return hashlib.sha256(path.read_bytes()).hexdigest() + # ------------------------------------------------------------------ # Workspace generation helpers # ------------------------------------------------------------------ diff --git a/tools/video/jimeng_video.py b/tools/video/jimeng_video.py new file mode 100644 index 00000000..9000054e --- /dev/null +++ b/tools/video/jimeng_video.py @@ -0,0 +1,405 @@ +"""Volcengine Jimeng (即梦 AI) video generation via the official API. + +Calls the Volcengine visual API directly (visual.volcengineapi.com) using +HMAC-SHA256 V4 request signing with AK/SK credentials. Supports text-to-video +and image-to-video via the Hailuo/Jimeng 3.0 Pro model. + +API flow: POST CVSync2AsyncSubmitTask -> poll CVSync2AsyncGetResult -> +download video_url. + +Authentication uses Volcengine IAM V4 signing (not Bearer token), which +requires an Access Key ID (AK) and Secret Access Key (SK) pair from +console.volcengine.com/iam/keymanage. +""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import os +import time +import urllib.parse +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ResourceProfile, + RetryPolicy, + ToolResult, + ToolRuntime, + ToolStability, + ToolStatus, + ToolTier, +) + + +_HOST = "visual.volcengineapi.com" +_REGION = "cn-north-1" +_SERVICE = "cv" +_ALGORITHM = "HMAC-SHA256" +_API_VERSION = "2022-08-31" +_REQ_KEY_VIDEO = "jimeng_ti2v_v30_pro" + + +class JimengVideo(BaseTool): + name = "jimeng_video" + version = "0.1.0" + tier = ToolTier.GENERATE + capability = "video_generation" + provider = "volcengine" + stability = ToolStability.EXPERIMENTAL + execution_mode = ExecutionMode.ASYNC + determinism = Determinism.STOCHASTIC + runtime = ToolRuntime.API + + dependencies = ["env:VOLC_ACCESSKEY", "env:VOLC_SECRETKEY"] + install_instructions = ( + "Set VOLC_ACCESSKEY and VOLC_SECRETKEY to your Volcengine IAM credentials.\n" + " Get them at https://console.volcengine.com/iam/keymanage\n" + " Ensure your account has access to Jimeng AI (即梦) video generation." + ) + agent_skills = ["ai-video-gen"] + + capabilities = ["text_to_video", "image_to_video"] + supports = { + "text_to_video": True, + "image_to_video": True, + "native_audio": False, + "seed": True, + } + best_for = [ + "Jimeng 3.0 Pro text-to-video and image-to-video via Volcengine", + "direct ByteDance API quota usage (not through a gateway)", + "Chinese-language prompt understanding", + ] + not_good_for = ["offline generation", "users without Volcengine AK/SK"] + fallback_tools = ["minimax_video", "kling_video", "veo_video"] + + input_schema = { + "type": "object", + "required": ["prompt"], + "properties": { + "prompt": { + "type": "string", + "maxLength": 800, + "description": "Video description. Max 800 chars. Supports Chinese.", + }, + "operation": { + "type": "string", + "enum": ["text_to_video", "image_to_video"], + "default": "text_to_video", + }, + "image_url": { + "type": "string", + "description": ( + "First frame image URL for image-to-video. " + "Must be publicly accessible." + ), + }, + "frames": { + "type": "integer", + "enum": [121, 241], + "default": 121, + "description": "Total frames. 121=5s, 241=10s at 24fps.", + }, + "aspect_ratio": { + "type": "string", + "enum": ["16:9", "9:16", "1:1", "4:3", "3:4", "21:9"], + "default": "16:9", + }, + "seed": { + "type": "integer", + "minimum": -1, + "default": -1, + "description": "Random seed. -1 for random.", + }, + "output_path": {"type": "string"}, + "poll_interval_seconds": { + "type": "number", + "minimum": 2, + "default": 5.0, + }, + "timeout_seconds": { + "type": "integer", + "minimum": 60, + "default": 600, + }, + }, + } + + resource_profile = ResourceProfile( + cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True + ) + retry_policy = RetryPolicy( + max_retries=2, + backoff_seconds=2.0, + retryable_errors=["rate_limit", "timeout"], + ) + idempotency_key_fields = [ + "prompt", + "operation", + "image_url", + "frames", + "aspect_ratio", + "seed", + ] + side_effects = [ + "writes video file to output_path", + "calls Volcengine Jimeng API (V4-signed submit + poll + download)", + ] + user_visible_verification = [ + "Watch generated clip for motion coherence and prompt adherence", + ] + + def _ak(self) -> str | None: + val = os.environ.get("VOLC_ACCESSKEY", "") + if val and not val.strip().startswith("#"): + return val.strip() + return None + + def _sk(self) -> str | None: + val = os.environ.get("VOLC_SECRETKEY", "") + if val and not val.strip().startswith("#"): + return val.strip() + return None + + def get_status(self) -> ToolStatus: + if self._ak() and self._sk(): + return ToolStatus.AVAILABLE + return ToolStatus.UNAVAILABLE + + def estimate_cost(self, inputs: dict[str, Any]) -> float: + frames = int(inputs.get("frames", 121)) + seconds = frames / 24.0 + return round(0.05 * seconds, 2) + + def estimate_runtime(self, inputs: dict[str, Any]) -> float: + return 120.0 + int(inputs.get("frames", 121)) * 1.0 + + def execute(self, inputs: dict[str, Any]) -> ToolResult: + ak = self._ak() + sk = self._sk() + if not ak or not sk: + return ToolResult( + success=False, + error="VOLC_ACCESSKEY or VOLC_SECRETKEY not set. " + self.install_instructions, + ) + + operation = inputs.get("operation", "text_to_video") + if operation == "image_to_video" and not inputs.get("image_url"): + return ToolResult( + success=False, + error="image_to_video requires image_url (public URL).", + ) + + start = time.time() + try: + result = self._generate(inputs, ak=ak, sk=sk) + except Exception as exc: + return ToolResult( + success=False, + error=f"Jimeng video generation failed: {self._safe_error(exc)}", + ) + + result.duration_seconds = round(time.time() - start, 2) + return result + + def _generate(self, inputs: dict[str, Any], *, ak: str, sk: str) -> ToolResult: + import requests + + from tools.video._shared import probe_output + + payload = self._build_payload(inputs) + task_id = self._submit_task(payload, ak=ak, sk=sk) + video_url = self._poll_task( + task_id, ak=ak, sk=sk, + poll_interval=float(inputs.get("poll_interval_seconds", 5.0)), + timeout_seconds=int(inputs.get("timeout_seconds", 600)), + ) + + download = requests.get(video_url, timeout=120) + download.raise_for_status() + + output_path = Path(inputs.get("output_path", "jimeng_video.mp4")) + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_bytes(download.content) + + probed = probe_output(output_path) + return ToolResult( + success=True, + data={ + "provider": "volcengine", + "route": "jimeng_direct", + "model": _REQ_KEY_VIDEO, + "prompt": inputs["prompt"], + "operation": inputs.get("operation", "text_to_video"), + "frames": payload.get("frames", 121), + "aspect_ratio": payload.get("aspect_ratio", "16:9"), + "seed": payload.get("seed", -1), + "task_id": task_id, + "video_url": video_url, + "output": str(output_path), + "format": "mp4", + **probed, + }, + artifacts=[str(output_path)], + cost_usd=self.estimate_cost(inputs), + model=_REQ_KEY_VIDEO, + ) + + @staticmethod + def _duration_to_frames(duration: int) -> int: + if duration >= 10: + return 241 + return 121 + + @staticmethod + def _build_payload(inputs: dict[str, Any]) -> dict[str, Any]: + operation = inputs.get("operation", "text_to_video") + frames = inputs.get("frames") + if frames is None: + frames = JimengVideo._duration_to_frames(int(inputs.get("duration", 5))) + payload: dict[str, Any] = { + "req_key": _REQ_KEY_VIDEO, + "prompt": inputs["prompt"], + "frames": int(frames), + "aspect_ratio": inputs.get("aspect_ratio", "16:9"), + "seed": int(inputs.get("seed", -1)), + } + if operation == "image_to_video" and inputs.get("image_url"): + payload["image_urls"] = [inputs["image_url"]] + return payload + + def _submit_task(self, payload: dict[str, Any], *, ak: str, sk: str) -> str: + import requests + + query = {"Action": "CVSync2AsyncSubmitTask", "Version": _API_VERSION} + body = json.dumps(payload, ensure_ascii=False).encode("utf-8") + headers = self._sign("POST", "/", query, {}, body, ak, sk) + url = f"https://{_HOST}/?{urllib.parse.urlencode(sorted(query.items()))}" + resp = requests.post(url, data=body, headers=headers, timeout=30) + data = self._json_or_raise(resp) + self._check_code(resp.status_code, data) + task_id = data.get("data", {}).get("task_id") + if not task_id: + raise RuntimeError(f"Jimeng submit returned no task_id: {data}") + return task_id + + def _poll_task( + self, task_id: str, *, ak: str, sk: str, + poll_interval: float, timeout_seconds: int, + ) -> str: + import requests + + query = {"Action": "CVSync2AsyncGetResult", "Version": _API_VERSION} + body = json.dumps({ + "req_key": _REQ_KEY_VIDEO, + "task_id": task_id, + "req_json": json.dumps({"return_url": True}), + }, ensure_ascii=False).encode("utf-8") + + deadline = time.time() + timeout_seconds + while time.time() < deadline: + time.sleep(poll_interval) + headers = self._sign("POST", "/", query, {}, body, ak, sk) + url = f"https://{_HOST}/?{urllib.parse.urlencode(sorted(query.items()))}" + resp = requests.post(url, data=body, headers=headers, timeout=30) + data = self._json_or_raise(resp) + self._check_code(resp.status_code, data) + status = (data.get("data") or {}).get("status", "") + if status == "done": + video_url = (data.get("data") or {}).get("video_url") + if not video_url: + raise RuntimeError(f"Jimeng task done but no video_url: {data}") + return video_url + if status in ("not_found", "expired"): + raise RuntimeError(f"Jimeng task invalid: status={status}") + raise TimeoutError(f"Jimeng task {task_id} did not finish within {timeout_seconds}s") + + @staticmethod + def _sign( + method: str, path: str, query_params: dict, + headers: dict, body: bytes, ak: str, sk: str, + ) -> dict: + now = datetime.now(timezone.utc) + x_date = now.strftime("%Y%m%dT%H%M%SZ") + short_date = x_date[:8] + + body_hash = hashlib.sha256(body).hexdigest() + headers = dict(headers) + headers["Host"] = _HOST + headers["X-Date"] = x_date + headers["X-Content-Sha256"] = body_hash + headers["Content-Type"] = "application/json" + + lower_headers = {k.lower(): v.strip() for k, v in headers.items()} + signed_names = sorted(lower_headers) + canonical_headers = "".join( + f"{k}:{lower_headers[k]}\n" + for k in signed_names + ) + signed_str = ";".join(signed_names) + + canonical_query = "&".join( + f"{urllib.parse.quote(str(k), safe='')}={urllib.parse.quote(str(v), safe='')}" + for k, v in sorted(query_params.items()) + ) + + canonical_request = "\n".join([ + method.upper(), path, canonical_query, + canonical_headers, signed_str, body_hash, + ]) + + credential_scope = f"{short_date}/{_REGION}/{_SERVICE}/request" + string_to_sign = "\n".join([ + _ALGORITHM, x_date, credential_scope, + hashlib.sha256(canonical_request.encode("utf-8")).hexdigest(), + ]) + + k_date = hmac.new(sk.encode("utf-8"), short_date.encode("utf-8"), hashlib.sha256).digest() + k_region = hmac.new(k_date, _REGION.encode("utf-8"), hashlib.sha256).digest() + k_service = hmac.new(k_region, _SERVICE.encode("utf-8"), hashlib.sha256).digest() + k_signing = hmac.new(k_service, b"request", hashlib.sha256).digest() + + signature = hmac.new(k_signing, string_to_sign.encode("utf-8"), hashlib.sha256).hexdigest() + + headers["Authorization"] = ( + f"{_ALGORITHM} Credential={ak}/{credential_scope}, " + f"SignedHeaders={signed_str}, Signature={signature}" + ) + return headers + + @staticmethod + def _safe_error(exc: Exception) -> str: + msg = str(exc) + for var in ("VOLC_SECRETKEY", "VOLC_ACCESSKEY"): + val = os.environ.get(var, "") + if val: + msg = msg.replace(val, "[redacted]") + return msg + + @staticmethod + def _json_or_raise(response: Any) -> dict[str, Any]: + try: + return response.json() + except ValueError as exc: + raise RuntimeError( + f"Non-JSON response from Jimeng API: HTTP {response.status_code}" + ) from exc + + @staticmethod + def _check_code(http_status: int, payload: dict[str, Any]) -> None: + if http_status < 400: + code = payload.get("code", 10000) + if code == 10000: + return + msg = payload.get("message", "unknown error") + raise RuntimeError(f"Jimeng API error: code={code}, msg={msg}") + code = payload.get("code", "unknown") + msg = payload.get("message", "unknown error") + raise RuntimeError(f"Jimeng API error: HTTP {http_status}, code={code}, msg={msg}") diff --git a/tools/video/veo_video.py b/tools/video/veo_video.py index de4efea2..11606d7b 100644 --- a/tools/video/veo_video.py +++ b/tools/video/veo_video.py @@ -308,13 +308,6 @@ class VeoVideo(BaseTool): client._api_client, "vertexai", False ) - if is_vertex: - return ToolResult( - success=False, - error="Google Veo video generation via google-genai is only supported using the Gemini Developer API (API key) backend. " - "Please configure GEMINI_API_KEY/GOOGLE_API_KEY or use the FAL.ai backend.", - ) - prompt = inputs["prompt"] operation = inputs.get("operation", "text_to_video") model_variant = inputs.get("model_variant", "veo3.1") @@ -501,7 +494,19 @@ class VeoVideo(BaseTool): success=False, error="No video asset returned in the response.", ) - client.files.download(file=video_asset) + if not is_vertex: + # The Files service is a Gemini Developer API feature. Vertex + # returns bytes inline when no GCS output URI is requested. + client.files.download(file=video_asset) + elif not getattr(video_asset, "video_bytes", None): + return ToolResult( + success=False, + error=( + "Vertex AI returned a video without inline bytes " + f"(uri={getattr(video_asset, 'uri', None)!r}). Configure " + "the request without an output GCS URI so bytes are returned inline." + ), + ) output_path = Path(inputs.get("output_path", "veo_output.mp4")) output_path.parent.mkdir(parents=True, exist_ok=True) diff --git a/tools/video/video_compose.py b/tools/video/video_compose.py index 976a2e55..7bf3ff90 100644 --- a/tools/video/video_compose.py +++ b/tools/video/video_compose.py @@ -31,11 +31,14 @@ the agent to re-ask the user rather than substituting a different engine. from __future__ import annotations import json +import hashlib import logging +import shutil import subprocess import time from pathlib import Path from typing import Any, Optional +from urllib.parse import unquote, urlsplit from tools.base_tool import ( BaseTool, @@ -387,6 +390,49 @@ class VideoCompose(BaseTool): except Exception: return False + def _mux_external_audio(self, video_path: Path, audio_path: str | Path) -> ToolResult: + """Atomically replace a rendered video's audio with the approved mix.""" + + audio = Path(audio_path).resolve() + if not audio.is_file(): + return ToolResult(success=False, error=f"Mixed audio not found: {audio}") + + temp_output = video_path.with_name( + f".{video_path.stem}.audio-mux-{time.time_ns()}{video_path.suffix}" + ) + try: + self.run_command([ + "ffmpeg", "-y", + "-i", str(video_path), + "-i", str(audio), + "-map", "0:v:0", + "-map", "1:a:0", + "-c:v", "copy", + "-c:a", "aac", + "-b:a", "192k", + "-af", "apad", + "-shortest", + "-movflags", "+faststart", + str(temp_output), + ]) + if not temp_output.is_file(): + return ToolResult( + success=False, + error=f"Audio mux completed but output file is missing: {temp_output}", + ) + temp_output.replace(video_path) + except Exception as exc: + return ToolResult(success=False, error=f"Could not mux mixed audio: {exc}") + finally: + if temp_output.exists(): + temp_output.unlink() + + return ToolResult( + success=True, + data={"output": str(video_path), "has_mixed_audio": True}, + artifacts=[str(video_path)], + ) + def _compose(self, inputs: dict[str, Any]) -> ToolResult: """FFmpeg composition: concat video cuts, add audio, burn subtitles. @@ -714,6 +760,124 @@ class VideoCompose(BaseTool): ) return comp + @staticmethod + def _cuts_to_cinematic_scenes(cuts: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Adapt canonical sequential cuts to CinematicRenderer's scene contract.""" + + scenes: list[dict[str, Any]] = [] + timeline_cursor = 0.0 + hard_transitions = {"cut", "none"} + title_types = {"hero_title", "text_card", "title"} + + for index, cut in enumerate(cuts): + try: + source_in = float(cut.get("in_seconds", 0)) + source_out = float(cut.get("out_seconds", source_in)) + speed = max(float(cut.get("speed", 1.0)), 0.1) + except (TypeError, ValueError): + continue + duration = max(0.0, (source_out - source_in) / speed) + if duration <= 0: + continue + + scene_id = str(cut.get("id") or f"cut-{index + 1}") + source = str(cut.get("source") or "") + cut_type = str(cut.get("type") or "").lower() + common = { + "id": scene_id, + "startSeconds": timeline_cursor, + "durationSeconds": duration, + } + + if cut_type in title_types or not source: + scene: dict[str, Any] = { + **common, + "kind": "title", + "text": str( + cut.get("text") + or cut.get("title") + or cut.get("reason") + or scene_id + ), + } + if source: + scene["backgroundSrc"] = source + scene["backgroundTrimBeforeSeconds"] = source_in + scene["backgroundTrimAfterSeconds"] = source_out + else: + scene = { + **common, + "kind": "video", + "src": source, + "trimBeforeSeconds": source_in, + "trimAfterSeconds": source_out, + "playbackRate": speed, + } + if str(cut.get("transition_in") or "").lower() in hard_transitions: + scene["fadeInFrames"] = 0 + if str(cut.get("transition_out") or "").lower() in hard_transitions: + scene["fadeOutFrames"] = 0 + + scenes.append(scene) + timeline_cursor += duration + + return scenes + + @staticmethod + def _stage_remotion_media(value: Any, public_dir: Path) -> int: + """Copy local media references into a Remotion public dir in-place. + + OffthreadVideo's compositor rejects ``file://`` sources. Rewriting + staged files to relative ``staticFile()`` paths works for video and + image components on every platform. + """ + + staged_by_source: dict[Path, str] = {} + media_keys = {"source", "src", "backgroundSrc"} + + def visit(node: Any, parent_key: str | None = None) -> Any: + if isinstance(node, dict): + for key, child in list(node.items()): + node[key] = visit(child, key) + return node + if isinstance(node, list): + for index, child in enumerate(node): + node[index] = visit(child, parent_key) + return node + if not isinstance(node, str) or parent_key not in media_keys: + return node + if node.startswith(("http://", "https://", "data:")): + return node + + if node.lower().startswith("file://"): + parsed = urlsplit(node) + decoded_path = unquote(parsed.path) + if len(parsed.netloc) == 2 and parsed.netloc[1] == ":": + raw_path = f"{parsed.netloc}{decoded_path}" + elif parsed.netloc and parsed.netloc.lower() != "localhost": + raw_path = f"//{parsed.netloc}{decoded_path}" + else: + raw_path = decoded_path + # Standard Windows file URIs use file:///C:/...; pathlib on + # Windows needs the drive path without the URI's leading slash. + if len(raw_path) >= 3 and raw_path[0] == "/" and raw_path[2] == ":": + raw_path = raw_path[1:] + else: + raw_path = node + source = Path(raw_path).resolve() + if not source.is_file(): + return node + if source not in staged_by_source: + digest = hashlib.sha256(str(source).encode("utf-8")).hexdigest()[:12] + name = f"{digest}-{source.name}" + public_dir.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, public_dir / name) + staged_by_source[source] = name + return staged_by_source[source] + + visit(value) + return len(staged_by_source) + def _render_via_atelier( self, inputs: dict[str, Any], @@ -840,6 +1004,11 @@ class VideoCompose(BaseTool): error=f"Atelier render completed but output file missing: {output_path}", ) + if inputs.get("audio_path"): + mux_result = self._mux_external_audio(output_path, inputs["audio_path"]) + if not mux_result.success: + return mux_result + # --- Atelier post-render review ------------------------------------- # The cut-schema paths run _run_final_review (technical/visual/audio # probes + transcript-vs-script). Atelier MUST do the same so hero @@ -1063,8 +1232,12 @@ class VideoCompose(BaseTool): try: from styles.playbook_loader import load_playbook playbook = load_playbook(playbook_name) - except Exception: - pass + except Exception as exc: + logging.getLogger(__name__).warning( + "Could not load style playbook %r for Remotion theme: %s", + playbook_name, + exc, + ) if playbook: vl = playbook.get("visual_language", {}) @@ -1356,6 +1529,31 @@ class VideoCompose(BaseTool): if render_runtime == "remotion" and remotion_atelier_requested: return self._render_via_atelier(inputs, edit_decisions) + # HyperFrames is HTML-first and therefore atelier by default for hero + # work. When a project-local authored workspace already exists, route + # before the stock cut/asset requirements so hyperframes_compose can + # validate and render it without overwriting index.html. + hyperframes_atelier_requested = ( + render_runtime == "hyperframes" + and ( + edit_decisions.get("composition_mode") == "atelier" + or edit_decisions.get("renderer_family") == "bespoke" + or bool(edit_decisions.get("bespoke", {}).get("entry")) + ) + ) + if hyperframes_atelier_requested: + output_path = Path(inputs.get("output_path", "renders/output.mp4")) + output_path.parent.mkdir(parents=True, exist_ok=True) + profile = inputs.get("profile") or inputs.get("output_profile") + return self._render_via_hyperframes( + inputs=inputs, + edit_decisions=edit_decisions, + asset_manifest=asset_manifest or {"version": "1.0", "assets": []}, + resolved_cuts=list(edit_decisions.get("cuts") or []), + output_path=output_path, + profile=profile, + ) + if not asset_manifest: return ToolResult(success=False, error="asset_manifest required for render") @@ -1418,6 +1616,8 @@ class VideoCompose(BaseTool): # would only take effect on a direct _remotion_render() call. if inputs.get("remotion_timeout_ms") is not None: remotion_inputs["remotion_timeout_ms"] = inputs["remotion_timeout_ms"] + if inputs.get("public_dir") is not None: + remotion_inputs["public_dir"] = inputs["public_dir"] render_result = self._remotion_render(remotion_inputs) # Governance: NEVER silently fall back to FFmpeg when Remotion fails. @@ -1437,6 +1637,11 @@ class VideoCompose(BaseTool): f"Per governance: renderer downgrade requires user approval." ), ) + if inputs.get("audio_path"): + mux_result = self._mux_external_audio(output_path, inputs["audio_path"]) + if not mux_result.success: + return mux_result + render_result.data["has_mixed_audio"] = True else: # --- FFmpeg fallback: only when Remotion is unavailable --- options = inputs.get("options", {}) @@ -1546,11 +1751,21 @@ class VideoCompose(BaseTool): try: from styles.playbook_loader import load_playbook # type: ignore playbook_data = load_playbook(playbook_name) - except Exception: + except Exception as exc: + logging.getLogger(__name__).warning( + "Could not load style playbook %r for HyperFrames bridge: %s", + playbook_name, + exc, + ) playbook_data = None + authored_workspace = ( + edit_decisions.get("composition_mode") == "atelier" + or edit_decisions.get("renderer_family") == "bespoke" + or bool(edit_decisions.get("bespoke", {}).get("entry")) + ) hf_inputs: dict[str, Any] = { - "operation": "render", + "operation": "render_existing" if authored_workspace else "render", "workspace_path": workspace_path, "output_path": str(output_path), "edit_decisions": dict(edit_decisions, cuts=resolved_cuts), @@ -1568,6 +1783,10 @@ class VideoCompose(BaseTool): hf_inputs["strict"] = inputs["strict"] if "skip_contrast" in inputs: hf_inputs["skip_contrast"] = inputs["skip_contrast"] + if "strict_check" in inputs: + hf_inputs["strict_check"] = inputs["strict_check"] + if "snapshots" in inputs: + hf_inputs["snapshots"] = inputs["snapshots"] render_result = HyperFramesCompose().execute(hf_inputs) @@ -1677,8 +1896,6 @@ class VideoCompose(BaseTool): types, and transitions using React-based frame-accurate rendering. Accepts edit_decisions (with resolved file paths) or raw composition_data. """ - import shutil - if not shutil.which("npx"): return ToolResult( success=False, @@ -1700,16 +1917,6 @@ class VideoCompose(BaseTool): # Deep-copy props so we don't mutate the original props = json.loads(json.dumps(composition_data)) - # Convert absolute file paths to file:// URIs for Remotion's - # Img and OffthreadVideo components - for cut in props.get("cuts", []): - source = cut.get("source", "") - if source and not source.startswith(("http://", "https://", "file://")): - resolved = Path(source).resolve() - if resolved.exists(): - posix = resolved.as_posix() - cut["source"] = f"file:///{posix}" if not posix.startswith("/") else f"file://{posix}" - # Build a custom themeConfig from the playbook's actual colors. # This ensures every video gets a unique visual identity derived # from its production decisions — not picked from a preset menu. @@ -1723,11 +1930,6 @@ class VideoCompose(BaseTool): if theme_config: props["themeConfig"] = theme_config - # Write props to temp file for Remotion CLI - props_path = output_path.parent / ".remotion_props.json" - with open(props_path, "w", encoding="utf-8") as f: - json.dump(props, f) - # remotion-composer lives at project root composer_dir = Path(__file__).resolve().parent.parent.parent / "remotion-composer" if not composer_dir.exists(): @@ -1741,6 +1943,39 @@ class VideoCompose(BaseTool): renderer_family = (composition_data or {}).get("renderer_family", "explainer-data") composition_id = self._get_composition_id(renderer_family) + if composition_id == "CinematicRenderer": + if not props.get("scenes") and props.get("cuts"): + props["scenes"] = self._cuts_to_cinematic_scenes(props["cuts"]) + props.pop("cuts", None) + if not props.get("scenes"): + return ToolResult( + success=False, + error="CinematicRenderer received cuts but none could be adapted into scenes.", + ) + + requested_public_dir = inputs.get("public_dir") + cleanup_public_dir = False + public_dir: Path | None = None + if requested_public_dir: + public_dir = Path(requested_public_dir).resolve() + if not public_dir.is_dir(): + return ToolResult( + success=False, + error=f"Remotion public_dir does not exist or is not a directory: {public_dir}", + ) + else: + public_dir = output_path.parent / f".remotion-public-{output_path.stem}" + cleanup_public_dir = True + + staged_count = self._stage_remotion_media(props, public_dir) + if not staged_count and cleanup_public_dir: + public_dir = None + + # Write the fully adapted/staged props, never the original cut payload. + props_path = output_path.parent / ".remotion_props.json" + with open(props_path, "w", encoding="utf-8") as f: + json.dump(props, f) + cmd = [ "npx", "remotion", "render", str(composer_dir / "src" / "index.tsx"), @@ -1753,6 +1988,8 @@ class VideoCompose(BaseTool): # API Remotion recommends for file paths and is cross-platform safe. f"--props={props_path}", ] + if public_dir is not None: + cmd.append(f"--public-dir={public_dir}") # Apply media profile dimensions profile_name = inputs.get("profile") @@ -1770,7 +2007,8 @@ class VideoCompose(BaseTool): # opaque failure. Pass it through and give the subprocess enough headroom # so run_command() does not kill Remotion before its own timeout fires. remotion_timeout_ms = inputs.get("remotion_timeout_ms") - subprocess_timeout = 600 + scene_count = len(props.get("scenes") or props.get("cuts") or []) + subprocess_timeout = max(600, scene_count * 15) if remotion_timeout_ms: try: ms = int(remotion_timeout_ms) @@ -1808,6 +2046,8 @@ class VideoCompose(BaseTool): finally: if props_path.exists(): props_path.unlink() + if cleanup_public_dir and public_dir is not None and public_dir.exists(): + shutil.rmtree(public_dir, ignore_errors=True) if not output_path.exists(): return ToolResult( @@ -1821,6 +2061,7 @@ class VideoCompose(BaseTool): "operation": "remotion_render", "output": str(output_path), "profile": profile_name, + "staged_media_count": staged_count, }, artifacts=[str(output_path)], )
BloomeBloome lets multiple AI agents (Claude, ChatGPT, DeepSeek, and more) collaborate in one conversation for agentic video pipelines. It has zero setup, runs in the cloud, works on web and mobile, and lets you share a configured agent with your whole team. Try Bloome.BloomeBloome lets multiple AI agents (Claude, ChatGPT, DeepSeek, and more) collaborate in one conversation for agentic video pipelines. It has zero setup, runs in the cloud, works on web and mobile, and lets you share a configured agent with your whole team. Try Bloome.
Atlas Cloud