mirror of
https://github.com/calesthio/OpenMontage.git
synced 2026-08-16 05:26:13 +08:00
feat: add Gemini Omni, Seedance 2.5, and MiniMax H3 routes
This commit is contained in:
calesthio
@@ -16,6 +16,17 @@ Use this skill before calling `comfyui_image`, `comfyui_video`, or `comfyui_musi
|
||||
- Long waits (video, music) prefer ComfyUI's websocket feed for immediate completion/error detection and transparently fall back to REST polling if `websocket-client` isn't installed. Either way, a timeout is recoverable: pass the error's `prompt_id` back in as `resume_prompt_id` to resume waiting on the same job instead of resubmitting it.
|
||||
- Export workflows with ComfyUI's API-format JSON, not the UI layout format. If a downloaded workflow will not submit, re-export it from ComfyUI with API format enabled.
|
||||
|
||||
### Partner Nodes are hosted
|
||||
|
||||
- `gemini_omni_flash`, `seedance_2.5`, and `minimax_h3_api` in
|
||||
`comfyui_video` are official ComfyUI Partner Nodes. They call hosted APIs and
|
||||
require network access, a logged-in Comfy account, and prepaid credits.
|
||||
- Do not describe Partner Nodes as local, offline, or free merely because the
|
||||
graph runs in a local ComfyUI process.
|
||||
- `minimax_h3_local` is a separate open-weight path. It requires the official
|
||||
MiniMax H3 workflow exported in API format, its `output_node`, and the model
|
||||
stack reported by the tool.
|
||||
|
||||
## Choosing a Workflow
|
||||
|
||||
- Use bundled workflows when the requested operation matches and the local machine has the required models and VRAM.
|
||||
|
||||
@@ -17,6 +17,20 @@ Gemini Omni is Google DeepMind's video generation **and editing** model family,
|
||||
|
||||
OpenMontage wraps it as `gemini_omni_video` (native Gemini API, no gateway). It shares `GOOGLE_API_KEY`/`GEMINI_API_KEY` with `google_imagen` and `google_tts` — one key, three capabilities. Paid tier only: ~$0.10 per second of output video (billed as 5,792 output tokens/sec at $17.50/1M).
|
||||
|
||||
Other documented routes are available when the direct Google key is not the
|
||||
chosen provider:
|
||||
|
||||
| Route | OpenMontage call | Important limitation |
|
||||
|-------|------------------|----------------------|
|
||||
| fal.ai | `gemini_omni_fal` | T2V, I2V, reference video, and edit endpoints; no Google interaction ID is returned |
|
||||
| Runway | `runway_video`, `model: "gemini_omni_flash"` | T2V/I2V/V2V; video edits accept up to five image references |
|
||||
| ComfyUI Partner Node | `comfyui_video`, `model_family: "gemini_omni_flash"` | Hosted paid node; requires network, Comfy login, and credits |
|
||||
|
||||
Use the direct `gemini_omni_video` route for stateful conversational editing.
|
||||
Gateway routes return ordinary provider tasks and cannot preserve Google's
|
||||
`previous_interaction_id` workflow. The fal edit endpoint can still be iterated
|
||||
by feeding each output video URL into the next edit call.
|
||||
|
||||
## When to pick it (and when not)
|
||||
|
||||
| Use it for | Prefer another provider for |
|
||||
|
||||
47
.agents/skills/minimax-h3/SKILL.md
Normal file
47
.agents/skills/minimax-h3/SKILL.md
Normal file
@@ -0,0 +1,47 @@
|
||||
---
|
||||
name: minimax-h3
|
||||
description: |
|
||||
Generate MiniMax H3 (Hailuo 3.0) video through the official MiniMax v2 API, fal.ai, Runway, ComfyUI Partner Nodes, or local open weights in ComfyUI. Use for 4-15 second 2K clips, first/last-frame animation, and image/video/audio reference-conditioned video.
|
||||
---
|
||||
|
||||
# MiniMax H3
|
||||
|
||||
MiniMax H3 is the Hailuo 3.0 family. The first-party API identifier is
|
||||
`MiniMax-H3`; fal.ai exposes the family as `hailuo-03`, and Runway uses
|
||||
`hailuo3`. Do not substitute one provider's identifier into another API.
|
||||
|
||||
## Choose a route
|
||||
|
||||
| Route | Tool call | Execution |
|
||||
|-------|-----------|-----------|
|
||||
| MiniMax direct | `minimax_video`, `model: "MiniMax-H3"` | Hosted first-party v2 API; global or mainland-China region |
|
||||
| fal.ai | `minimax_fal_video` | Hosted gateway; T2V, I2V, reference-to-video |
|
||||
| Runway | `runway_video`, `model: "hailuo3"` | Hosted; 768P or 2K, 5–15 seconds |
|
||||
| ComfyUI Partner Node | `comfyui_video`, `model_family: "minimax_h3_api"` | Hosted and billed in Comfy credits |
|
||||
| ComfyUI open weights | `comfyui_video`, `model_family: "minimax_h3_local"` | Local GPU with official workflow and model stack |
|
||||
|
||||
For local ComfyUI, export the official workflow in API format and pass
|
||||
`workflow_json` or `workflow_path` plus `output_node`. OpenMontage reports the
|
||||
required diffusion model, Qwen3-VL text encoder, video VAE, and audio VAE; it
|
||||
does not silently download large weights.
|
||||
|
||||
## Operations and prompting
|
||||
|
||||
- Text-to-video: concrete ratio; do not use `adaptive` without visual input.
|
||||
- Image-to-video: provide a first frame.
|
||||
- First/last-frame: provide both images and describe the motion between them.
|
||||
- Reference-to-video: images, videos, and audio can be combined. Audio needs at
|
||||
least one visual reference.
|
||||
|
||||
Write prompts as subject + action + camera path + environment + lighting +
|
||||
audio intent. MiniMax responds well to explicit camera direction. Keep the
|
||||
requested motion achievable within 4–15 seconds and inspect native audio as
|
||||
carefully as the image track.
|
||||
|
||||
## Provider differences
|
||||
|
||||
The direct MiniMax v2 route currently outputs 2K and supports 4–15 seconds.
|
||||
Runway's Hailuo 3.0 route supports 768P/2K and documents 5–15 seconds. Partner
|
||||
Nodes require network access and credits. Only the open-weight ComfyUI route is
|
||||
local/offline after all models are installed.
|
||||
|
||||
53
.agents/skills/seedance-2-5/SKILL.md
Normal file
53
.agents/skills/seedance-2-5/SKILL.md
Normal file
@@ -0,0 +1,53 @@
|
||||
---
|
||||
name: seedance-2-5
|
||||
description: |
|
||||
Generate 4-30 second cinematic video with ByteDance Seedance 2.5 through fal.ai, Volcengine Ark, Runway, or ComfyUI Partner Nodes. Use for long single generations, synchronized audio, and large multimodal reference sets (up to 30 images, 10 videos, and 10 audio clips).
|
||||
---
|
||||
|
||||
# Seedance 2.5
|
||||
|
||||
Seedance 2.5 extends the Seedance 2 family to 4–30 second 480p/720p clips and
|
||||
larger multimodal reference sets. It is hosted; there are no local model
|
||||
weights in OpenMontage.
|
||||
|
||||
## Choose a supported route
|
||||
|
||||
| Route | Tool call | Notes |
|
||||
|-------|-----------|-------|
|
||||
| fal.ai | `seedance_video`, `model_version: "2.5"` | T2V, I2V, and reference-to-video |
|
||||
| Volcengine Ark | `seedance_ark`, `model: "2.5"` | First-party model ID `doubao-seedance-2-5-260628`; custom token price required for cost estimates |
|
||||
| Runway | `runway_video`, `model: "seedance2_5"` | T2V, I2V, V2V; 480p/720p |
|
||||
| ComfyUI Partner Node | `comfyui_video`, `model_family: "seedance_2.5"` | Hosted and paid despite running in a ComfyUI graph |
|
||||
|
||||
Do not invent a Replicate, HeyGen, or Higgsfield identifier when their current
|
||||
public API schema does not list Seedance 2.5.
|
||||
|
||||
## Reference limits
|
||||
|
||||
- Up to 30 reference images.
|
||||
- Up to 10 reference videos.
|
||||
- Up to 10 reference audio clips.
|
||||
- Keep combined reference video/audio duration within the provider's documented
|
||||
ceiling; Runway caps it at 30 seconds.
|
||||
- For Runway video-to-video, the source video consumes one video slot.
|
||||
|
||||
Reference inputs are provider-specific. fal.ai uses `image_urls`, `video_urls`,
|
||||
and `audio_urls` internally. Runway uses `references`, `referenceVideos`, and
|
||||
`referenceAudio`. Ark uses typed content entries with roles. Always call the
|
||||
OpenMontage tool instead of constructing a provider payload manually.
|
||||
|
||||
## Prompting
|
||||
|
||||
Lead with the shot structure, then subject, action, camera, lighting, and audio.
|
||||
For multi-shot work, give each beat an explicit time range and use quoted text
|
||||
for dialogue. Reference each supplied asset by a stable role in the prompt.
|
||||
Thirty seconds is a ceiling, not a target: use a shorter generation when the
|
||||
scene has only one meaningful action.
|
||||
|
||||
## Cost and verification
|
||||
|
||||
All supported routes are paid. Confirm the exact provider/model before calling
|
||||
and review the result for identity continuity, cuts, lip sync, audio artifacts,
|
||||
and prompt adherence. ComfyUI Partner Nodes use prepaid Comfy credits and are
|
||||
not an offline fallback.
|
||||
|
||||
13
.env.example
13
.env.example
@@ -9,11 +9,13 @@ FAL_KEY=
|
||||
FAL_AI_API_KEY=
|
||||
|
||||
# --- MiniMax official direct API ---
|
||||
# First-party image generation (image-01 / image-01-live).
|
||||
# First-party image generation plus MiniMax H3 / older Hailuo video generation.
|
||||
# Get one at https://platform.minimax.io/user-center/basic-information/interface-key
|
||||
MINIMAX_API_KEY=
|
||||
# Optional: global (default) or cn.
|
||||
MINIMAX_REGION=global
|
||||
# Optional endpoint override; normally leave blank so MINIMAX_REGION selects it.
|
||||
# MINIMAX_BASE_URL=
|
||||
|
||||
# --- Replicate ---
|
||||
# Replicate-hosted video gen (seedance_replicate). Needed to make the
|
||||
@@ -81,11 +83,11 @@ TENCENT_TOKENHUB_API_KEY=
|
||||
SUNO_API_KEY=
|
||||
|
||||
# --- Video Generation ---
|
||||
# Volcengine Ark direct Seedance 2.0 API key body (without the "Bearer " prefix).
|
||||
# Volcengine Ark direct Seedance 2.0 / 2.5 API key body (without the "Bearer " prefix).
|
||||
# Get one at https://console.volcengine.com/ark/region:cn-beijing/apiKey
|
||||
ARK_API_KEY=
|
||||
# Optional overrides; uncomment only when needed.
|
||||
# ARK_SEEDANCE_MODEL=doubao-seedance-2-0-260128
|
||||
# ARK_SEEDANCE_MODEL=doubao-seedance-2-5-260628
|
||||
# ARK_BASE_URL=https://ark.cn-beijing.volces.com/api/v3
|
||||
# ARK_CNY_PER_USD=7.2
|
||||
# HeyGen API (VEO, Sora, Runway, Kling, Seedance via single key).
|
||||
@@ -103,6 +105,11 @@ VIDEO_GEN_LOCAL_MODEL=
|
||||
# Modal self-hosted LTX-2 endpoint (optional).
|
||||
MODAL_LTX2_ENDPOINT_URL=
|
||||
|
||||
# ComfyUI server overrides (optional; default shared server is localhost:8188).
|
||||
# Partner Nodes still require network access, a logged-in Comfy account, and credits.
|
||||
COMFYUI_SERVER_URL=
|
||||
COMFYUI_VIDEO_SERVER_URL=
|
||||
|
||||
# --- Stock Media ---
|
||||
# Pexels stock footage/images (free).
|
||||
PEXELS_API_KEY=
|
||||
|
||||
@@ -53,18 +53,18 @@ AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
|
||||
|
||||
# MULTI-MODEL GATEWAY (one key, 6+ tools)
|
||||
FAL_KEY= # FLUX, Recraft, Kling, Veo, MiniMax video
|
||||
MINIMAX_API_KEY= # MiniMax first-party image generation
|
||||
MINIMAX_API_KEY= # MiniMax first-party image + MiniMax H3 video generation
|
||||
|
||||
# KLING OFFICIAL DIRECT API
|
||||
KLING_API_KEY= # Official Kling video, image, TTS, avatar, lip sync
|
||||
KLING_API_BASE_URL= # Optional; default https://api-singapore.klingai.com
|
||||
|
||||
# VOLCENGINE ARK DIRECT SEEDANCE 2.0 API
|
||||
# VOLCENGINE ARK DIRECT SEEDANCE 2.0 / 2.5 API
|
||||
ARK_API_KEY= # API key body only; do not include the "Bearer " prefix
|
||||
|
||||
# VIDEO
|
||||
HEYGEN_API_KEY= # HeyGen avatar video gateway
|
||||
RUNWAY_API_KEY= # Runway Gen-4 video (direct)
|
||||
RUNWAY_API_KEY= # Runway native + Seedance 2.5, Gemini Omni, MiniMax H3
|
||||
SUNO_API_KEY= # Suno music generation
|
||||
|
||||
# TENCLOUD HUNYUAN VIDEO
|
||||
@@ -73,10 +73,36 @@ TENCENT_TOKENHUB_API_KEY= # Tencent Hunyuan cloud video via TokenHub API
|
||||
# LOCAL (no keys needed — just GPU + install)
|
||||
VIDEO_GEN_LOCAL_ENABLED= # Set to "true" for local video gen
|
||||
VIDEO_GEN_LOCAL_MODEL= # wan2.1-1.3b, wan2.1-14b, hunyuan-1.5, ltx2-local, cogvideo-5b
|
||||
|
||||
# COMFYUI (optional overrides; localhost:8188 is the default)
|
||||
COMFYUI_SERVER_URL= # Local ComfyUI server for shared workflows
|
||||
COMFYUI_VIDEO_SERVER_URL= # Optional video-specific ComfyUI server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Current Video Model Coverage
|
||||
|
||||
The following integrations are based on documented, currently exposed model
|
||||
identifiers. Provider pages without a public API contract were not given
|
||||
speculative model strings.
|
||||
|
||||
| Model | Direct provider | fal.ai | Runway | ComfyUI Partner Nodes | Local ComfyUI |
|
||||
|-------|-----------------|--------|--------|-----------------------|---------------|
|
||||
| **Gemini Omni Flash** | Google `gemini_omni_video` | `gemini_omni_fal` (T2V, I2V, references, editing) | `runway_video` model `gemini_omni_flash` | `GeminiVideoOmni` (hosted, paid credits) | Not available as local weights |
|
||||
| **Seedance 2.5** | Volcengine `seedance_ark` model variant `2.5` | `seedance_video` model version `2.5` | `runway_video` model `seedance2_5` | `ByteDance2TextToVideoNode` (hosted, paid credits) | Not available as local weights |
|
||||
| **MiniMax H3** | `minimax_video` model `MiniMax-H3` | `minimax_fal_video` (`hailuo-03`) | `runway_video` model `hailuo3` | `MinimaxHailuo03TextToVideoNode` (hosted, paid credits) | Supported with official open weights and an exported API workflow |
|
||||
|
||||
ComfyUI Partner Nodes run inside the ComfyUI graph but call hosted services;
|
||||
they require network access, a logged-in Comfy account, and prepaid credits.
|
||||
Only the MiniMax H3 open-weight workflow in this table is a local model path.
|
||||
|
||||
Replicate, HeyGen, and Higgsfield were not updated for these exact model
|
||||
versions because their public API documentation did not expose a current,
|
||||
stable contract for them at the time of this update.
|
||||
|
||||
---
|
||||
|
||||
## Cloud Providers
|
||||
|
||||
### xAI — Grok Image + Video
|
||||
@@ -161,7 +187,7 @@ The `req_key` for video is `jimeng_ti2v_v30_pro`. Success code is `10000`. Task
|
||||
|
||||
---
|
||||
|
||||
### Volcengine Ark — Direct Seedance 2.0 Video Generation
|
||||
### Volcengine Ark — Direct Seedance 2.0 and 2.5 Video Generation
|
||||
|
||||
> **Official direct Seedance API.** Calls Volcengine Ark without routing through fal.ai or Replicate, while keeping those existing provider paths available as independent fallbacks.
|
||||
|
||||
@@ -172,7 +198,7 @@ The `req_key` for video is `jimeng_ti2v_v30_pro`. Success code is `10000`. Task
|
||||
#### Setup
|
||||
|
||||
1. Open the [Volcengine Ark API key console](https://console.volcengine.com/ark/region:cn-beijing/apiKey)
|
||||
2. Enable the Seedance 2.0 model family and confirm that the account has balance or a valid resource package
|
||||
2. Enable the Seedance model family and confirm that the account has balance or a valid resource package
|
||||
3. Create a long-lived API key
|
||||
4. Add the key body to `.env`: `ARK_API_KEY=...`
|
||||
|
||||
@@ -190,6 +216,7 @@ ARK_CNY_PER_USD=7.2
|
||||
|
||||
| Variant | Default model ID | Output |
|
||||
|---------|------------------|--------|
|
||||
| 2.5 | `doubao-seedance-2-5-260628` | 480p or 720p; 4–30 seconds |
|
||||
| Standard | `doubao-seedance-2-0-260128` | 480p, 720p, 1080p, or 4K |
|
||||
| Fast | `doubao-seedance-2-0-fast-260128` | 480p or 720p |
|
||||
| Mini | `doubao-seedance-2-0-mini-260615` | 480p or 720p |
|
||||
@@ -203,6 +230,12 @@ The adapter supports:
|
||||
- synchronized audio, optional last-frame return, web search for text-only requests, and output download
|
||||
- pre-submit dry-run and token-based cost estimates
|
||||
|
||||
Seedance 2.5 accepts up to 30 image, 10 video, and 10 audio references.
|
||||
Select it with `model: "2.5"` (or its exact model ID). Because the public
|
||||
documentation does not establish a stable default token price for this model,
|
||||
OpenMontage requires `custom_price_cny_per_million_tokens` before presenting a
|
||||
cost estimate; unknown pricing is never reported as free.
|
||||
|
||||
Local reference videos are intentionally rejected because the public API does not document video Data URI support. Use a provider-accessible HTTPS URL or an Ark asset reference instead.
|
||||
|
||||
#### API and billing notes
|
||||
@@ -213,9 +246,9 @@ The asynchronous API flow is:
|
||||
|
||||
Queued tasks can be cancelled with `DELETE /contents/generations/tasks/{id}`. Task records are retained for a limited period, and successful result URLs are short-lived, so the tool downloads outputs promptly.
|
||||
|
||||
Ark bills Seedance 2.0 by completion tokens. Rates vary by model, resolution, and whether the request includes reference video. OpenMontage estimates cost before submission and reconciles against provider-returned usage when available. Check the Ark console for current rates before a paid run; custom endpoint IDs require an explicit custom price so unknown pricing is never treated as free.
|
||||
Ark bills Seedance by completion tokens. Rates vary by model, resolution, and whether the request includes reference video. OpenMontage estimates cost before submission and reconciles against provider-returned usage when available. Check the Ark console for current rates before a paid run; custom endpoint IDs and Seedance 2.5 require an explicit custom price so unknown pricing is never treated as free.
|
||||
|
||||
Official references: [model list](https://www.volcengine.com/docs/82379/1330310?lang=zh), [create task](https://www.volcengine.com/docs/82379/1520757?lang=zh), [query task](https://www.volcengine.com/docs/82379/1521309?lang=zh).
|
||||
Official references: [Seedance model list](https://www.volcengine.com/docs/82379/1366799), [create task](https://www.volcengine.com/docs/82379/1520757?lang=zh), [query task](https://www.volcengine.com/docs/82379/1521309?lang=zh).
|
||||
|
||||
---
|
||||
|
||||
@@ -278,7 +311,9 @@ reference-image inputs are normalized to the provider's `images` array.
|
||||
|
||||
> **Broad single-key coverage.** One API key unlocks image and video providers across multiple models.
|
||||
|
||||
**Tools unlocked:** `flux_image`, `recraft_image`, `seedream_image`, `kling_video`, `veo_video`, `minimax_video`, `fal_elevenlabs_tts`, `fal_elevenlabs_music`
|
||||
**Tools unlocked:** `flux_image`, `recraft_image`, `seedream_image`,
|
||||
`kling_video`, `veo_video`, `seedance_video`, `gemini_omni_fal`,
|
||||
`minimax_fal_video`, `fal_elevenlabs_tts`, `fal_elevenlabs_music`
|
||||
**Env var:** `FAL_KEY`
|
||||
|
||||
#### Setup
|
||||
@@ -307,7 +342,9 @@ No subscription — pure pay-as-you-go, no minimum spend.
|
||||
| Model | Price | Per $1 |
|
||||
|-------|-------|--------|
|
||||
| Kling 2.5 Turbo Pro | $0.07/sec | 14 seconds |
|
||||
| MiniMax | ~$0.05/sec | 20 seconds |
|
||||
| Seedance 2.5 | endpoint-dependent | 4–30 seconds |
|
||||
| Gemini Omni Flash | endpoint-dependent | 3–10 seconds |
|
||||
| MiniMax H3 (`hailuo-03`) | endpoint-dependent | 4–15 seconds |
|
||||
| Veo 3 | $0.40/sec | 2.5 seconds |
|
||||
| WAN 2.5 | $0.05/sec | 20 seconds |
|
||||
|
||||
@@ -319,13 +356,13 @@ select it through `tts_selector` with `preferred_provider: "fal.ai"`.
|
||||
|
||||
---
|
||||
|
||||
### MiniMax — Official Direct Image API
|
||||
### MiniMax — Official Direct Image and Video API
|
||||
|
||||
> **Low-cost first-party image generation.** The direct MiniMax API supports
|
||||
> seeded text-to-image, character subject references, custom dimensions, and
|
||||
> global or mainland-China routing without a gateway.
|
||||
> **First-party image and video generation.** The direct MiniMax API supports
|
||||
> seeded image generation plus MiniMax H3 video generation with text, first/last
|
||||
> frames, image/video/audio references, and global or mainland-China routing.
|
||||
|
||||
**Tool unlocked:** `minimax_image`
|
||||
**Tools unlocked:** `minimax_image`, `minimax_video`
|
||||
|
||||
**Env var:** `MINIMAX_API_KEY`
|
||||
|
||||
@@ -338,6 +375,19 @@ select it through `tts_selector` with `preferred_provider: "fal.ai"`.
|
||||
3. Add `MINIMAX_API_KEY=...` to `.env`.
|
||||
4. For a mainland-China account, also set `MINIMAX_REGION=cn`.
|
||||
|
||||
`MINIMAX_BASE_URL` may be used for a documented private/enterprise endpoint
|
||||
override. The default global and mainland-China hosts are selected from
|
||||
`MINIMAX_REGION`.
|
||||
|
||||
#### MiniMax H3 video
|
||||
|
||||
Use `minimax_video` with `model: "MiniMax-H3"`. The tool uses the v2 task
|
||||
contract (`POST /v2/video_generation`, then
|
||||
`GET /v2/query/video_generation/{task_id}`) and supports 4–15 second 2K clips.
|
||||
Older Hailuo models continue to use the v1 API. MiniMax H3 reference generation
|
||||
can combine images, video, and audio; reference audio requires a visual
|
||||
reference.
|
||||
|
||||
#### Pricing
|
||||
|
||||
| Models | Global pay-as-you-go price |
|
||||
@@ -348,8 +398,8 @@ MiniMax also offers subscription token plans with included daily image quota.
|
||||
OpenMontage conservatively reports the standard pay-as-you-go amount in cost
|
||||
estimates and generation results.
|
||||
|
||||
The tool is automatically discoverable through `image_selector`; choose it
|
||||
with `preferred_provider: "minimax"`.
|
||||
The tools are automatically discoverable through the image and video selectors;
|
||||
choose them with `preferred_provider: "minimax"`.
|
||||
|
||||
---
|
||||
|
||||
@@ -812,9 +862,11 @@ Google TTS offers 700+ voices across 50+ languages. Voice names follow the patte
|
||||
|
||||
---
|
||||
|
||||
### Runway — Gen-3/Gen-4 Video
|
||||
### Runway — Native and Third-Party Video Models
|
||||
|
||||
> **Highest-rated AI video quality.** #1 on Elo rankings. Professional-grade video generation with Gen-3 Alpha Turbo, Gen-4 Turbo, and Gen-4 Aleph models.
|
||||
> **Multi-model production API.** OpenMontage supports current Runway-native
|
||||
> models plus documented third-party Seedance 2.5, Gemini Omni Flash, and
|
||||
> MiniMax H3/Hailuo 3.0 routes.
|
||||
|
||||
**Tools unlocked:** `runway_video`
|
||||
**Env var:** `RUNWAY_API_KEY`
|
||||
@@ -835,13 +887,25 @@ Google TTS offers 700+ voices across 50+ languages. Voice names follow the patte
|
||||
| Pro | $28/mo | 2,250 | ~90 seconds Gen-4 |
|
||||
| Unlimited | $76/mo | Unlimited (Explore Mode) | Unlimited Gen-4 Turbo |
|
||||
|
||||
**API pricing (approximate):**
|
||||
**API pricing (Runway credits are $0.01 each):**
|
||||
|
||||
| Model | Price per second |
|
||||
|-------|-----------------|
|
||||
| Gen-3 Alpha Turbo | ~$0.05 |
|
||||
| Gen-4 Turbo | ~$0.05 |
|
||||
| Gen-4 Aleph | ~$0.15 |
|
||||
| Gen-4.5 | ~$0.12 |
|
||||
| Seedance 2.5 | ~$0.20 at 480p / ~$0.30 at 720p |
|
||||
| Gemini Omni Flash | ~$0.10 generation / ~$0.11 video editing |
|
||||
| MiniMax H3 (`hailuo3`) | ~$0.10 at 768P / ~$0.15 at 2K |
|
||||
|
||||
Seedance 2.5 supports text, image, and video inputs, 4–30 second outputs, and
|
||||
up to 30 image, 10 video, and 10 audio references. Gemini Omni Flash supports
|
||||
3–10 second text/image generation plus video editing with up to five image
|
||||
references. Hailuo 3.0 is Runway's MiniMax H3 route and supports 5–15 second
|
||||
outputs at 768P or 2K. The adapter maps each model to its exact request field
|
||||
names instead of sending a generic payload.
|
||||
|
||||
Gen-3 Alpha Turbo and Gen-4 Aleph were removed from the Runway API on
|
||||
2026-07-30 and are not offered by the tool.
|
||||
|
||||
**Free tier:** 125 one-time credits (no monthly renewal). Enough for about 5 seconds of Gen-4 video. API access requires a paid subscription.
|
||||
|
||||
@@ -1113,6 +1177,43 @@ piper --download-dir ~/.piper/models --model en_US-lessac-medium
|
||||
|
||||
---
|
||||
|
||||
### ComfyUI Video — Local Workflows and Hosted Partner Nodes
|
||||
|
||||
**Tool:** `comfyui_video`
|
||||
|
||||
**Optional env vars:** `COMFYUI_SERVER_URL` (default
|
||||
`http://localhost:8188`) and `COMFYUI_VIDEO_SERVER_URL` (video-specific
|
||||
override).
|
||||
|
||||
The bundled WAN 2.2 workflows and caller-supplied local workflows execute on
|
||||
the ComfyUI machine. MiniMax H3 is available as an official open-weight local
|
||||
workflow; pass the official workflow exported in API format using
|
||||
`workflow_json` or `workflow_path`, plus its `output_node`.
|
||||
|
||||
The MiniMax H3 local stack includes the pruned INT8 diffusion model, Qwen3-VL
|
||||
text encoder, video VAE, and audio VAE. OpenMontage exposes the official
|
||||
download URLs and destination folders in tool metadata rather than silently
|
||||
downloading large weights.
|
||||
|
||||
The same tool also supports these ComfyUI Partner Nodes:
|
||||
|
||||
| `model_family` | Node | Execution | Approximate cost |
|
||||
|----------------|------|-----------|------------------|
|
||||
| `gemini_omni_flash` | `GeminiVideoOmni` | Hosted Partner Node | ~$0.146/sec |
|
||||
| `seedance_2.5` | `ByteDance2TextToVideoNode` | Hosted Partner Node | ~$0.148/sec 480p; ~$0.333/sec 720p |
|
||||
| `minimax_h3_api` | `MinimaxHailuo03TextToVideoNode` | Hosted Partner Node | ~$0.129/sec 768P; ~$0.186/sec 2K |
|
||||
| `minimax_h3_local` | official MiniMax H3 graph | Local GPU | No API charge |
|
||||
|
||||
Partner Nodes are not offline: they require current ComfyUI, network access, a
|
||||
logged-in Comfy account, and prepaid credits. Prices are estimates converted
|
||||
from Comfy credits (211 credits = $1); actual metered usage is authoritative.
|
||||
|
||||
Official references: [Partner Node overview](https://docs.comfy.org/tutorials/partner-nodes/overview),
|
||||
[pricing](https://docs.comfy.org/tutorials/partner-nodes/pricing), and
|
||||
[MiniMax H3 local tutorial](https://docs.comfy.org/tutorials/video/minimax/minimax-h3).
|
||||
|
||||
---
|
||||
|
||||
### Local Video Generation (GPU Required)
|
||||
|
||||
> **Free AI video generation.** Requires an NVIDIA GPU with sufficient VRAM.
|
||||
@@ -1228,9 +1329,10 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
|
||||
| **Google** | `GOOGLE_API_KEY` (or `GEMINI_API_KEY`) | `google_tts`, `google_imagen`, `google_music`, `gemini_omni_video`, `veo_video` | Free tier (TTS) + paid |
|
||||
| **ElevenLabs** | `ELEVENLABS_API_KEY` | `elevenlabs_tts`, `music_gen` | Free tier + paid |
|
||||
| **fish.audio** | `FISH_AUDIO_API_KEY` | `fish_audio_tts` | Free tier (s2.1-pro-free) + paid |
|
||||
| **fal.ai** | `FAL_KEY` | `flux_image`, `recraft_image`, `kling_video`, `veo_video`, `minimax_video` | Pay-as-you-go |
|
||||
| **fal.ai** | `FAL_KEY` | `flux_image`, `recraft_image`, `kling_video`, `veo_video`, `seedance_video`, `gemini_omni_fal`, `minimax_fal_video` | Pay-as-you-go |
|
||||
| **Kling Official** | `KLING_API_KEY` | `kling_official_video`, `kling_official_image`, `kling_tts`, `kling_avatar`, `kling_lip_sync` | Pay-as-you-go |
|
||||
| **Volcengine Ark** | `ARK_API_KEY` | `seedance_ark` | Pay-as-you-go |
|
||||
| **MiniMax direct** | `MINIMAX_API_KEY` | `minimax_image`, `minimax_video` | Pay-as-you-go |
|
||||
| **OpenAI** | `OPENAI_API_KEY` | `openai_tts`, `openai_image` | Paid only |
|
||||
| **xAI** | `XAI_API_KEY` | `grok_image`, `grok_video` | Paid only |
|
||||
| **Runway** | `RUNWAY_API_KEY` | `runway_video` | Free trial + paid |
|
||||
@@ -1241,6 +1343,7 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
|
||||
| **Local GPU** | `VIDEO_GEN_LOCAL_ENABLED` | `wan_video`, `hunyuan_video`, `cogvideo_video`, `ltx_video_local` | Free (GPU required) |
|
||||
| **Local Diffusion** | — (install only) | `local_diffusion` | Free (GPU required) |
|
||||
| **Modal** | `MODAL_LTX2_ENDPOINT_URL` | `ltx_video_modal` | Self-hosted cloud |
|
||||
| **ComfyUI** | optional server URL overrides | `comfyui_video` | Local GPU, or paid Partner Node credits |
|
||||
|
||||
---
|
||||
|
||||
@@ -1251,7 +1354,7 @@ How many providers cover each capability:
|
||||
| Capability | Cloud Providers | Local Providers | Free Options |
|
||||
|-----------|----------------|-----------------|--------------|
|
||||
| **Image Generation** | FLUX, Kling Official, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
|
||||
| **Video Generation** | Grok, Kling Official, Kling via fal.ai, Seedance via Volcengine Ark, Runway, Veo, Gemini Omni, Higgsfield, MiniMax, HeyGen, Tencent Hunyuan | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
|
||||
| **Video Generation** | Grok, Kling Official, fal.ai, Seedance via Volcengine Ark, Runway, Veo, Gemini Omni, Higgsfield, MiniMax, HeyGen, Tencent Hunyuan, ComfyUI Partner Nodes | WAN, Hunyuan, CogVideo, LTX, ComfyUI WAN, ComfyUI MiniMax H3 | Pexels, Pixabay (stock) |
|
||||
| **Text-to-Speech** | Azure AI Speech, ElevenLabs, fish.audio, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier, Azure free tier, fish.audio s2.1-pro-free |
|
||||
| **Music Generation** | ElevenLabs, Suno, Google Lyria | — | ElevenLabs free tier |
|
||||
| **Post-Production** | — | FFmpeg (compose, stitch, trim, mix, enhance, grade) | All free |
|
||||
|
||||
@@ -55,11 +55,30 @@ class TestContract:
|
||||
|
||||
def test_official_model_ids(self):
|
||||
assert SeedanceArkVideo.MODEL_IDS == {
|
||||
"2.5": "doubao-seedance-2-5-260628",
|
||||
"standard": "doubao-seedance-2-0-260128",
|
||||
"fast": "doubao-seedance-2-0-fast-260128",
|
||||
"mini": "doubao-seedance-2-0-mini-260615",
|
||||
}
|
||||
|
||||
def test_seedance_25_contract_and_limits(self):
|
||||
tool = SeedanceArkVideo()
|
||||
payload = tool._build_payload(
|
||||
{
|
||||
"prompt": "A continuous cinematic chase with synchronized sound",
|
||||
"model_variant": "2.5",
|
||||
"duration": 30,
|
||||
"resolution": "720p",
|
||||
"aspect_ratio": "16:9",
|
||||
}
|
||||
)
|
||||
assert payload["model"] == "doubao-seedance-2-5-260628"
|
||||
assert payload["duration"] == 30
|
||||
assert payload["generate_audio"] is True
|
||||
|
||||
with pytest.raises(ValueError, match="custom_price_cny_per_million_tokens"):
|
||||
tool.estimate_cost(payload)
|
||||
|
||||
|
||||
class TestTaskActions:
|
||||
def test_create_uses_official_endpoint_bearer_auth_and_body(self, monkeypatch):
|
||||
@@ -310,9 +329,7 @@ class TestInputSafety:
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_invalid_input_never_reaches_network(
|
||||
self, inputs, message, monkeypatch
|
||||
):
|
||||
def test_invalid_input_never_reaches_network(self, inputs, message, monkeypatch):
|
||||
monkeypatch.setenv("ARK_API_KEY", "fake-ark-key")
|
||||
|
||||
def fail_network(*args, **kwargs):
|
||||
@@ -333,16 +350,12 @@ class TestInputSafety:
|
||||
raise RuntimeError(f"request failed using {api_key}")
|
||||
|
||||
monkeypatch.setattr("requests.post", failing_post)
|
||||
result = SeedanceArkVideo().execute(
|
||||
{"task_action": "create", "prompt": "x"}
|
||||
)
|
||||
result = SeedanceArkVideo().execute({"task_action": "create", "prompt": "x"})
|
||||
assert result.success is False
|
||||
assert api_key not in result.error
|
||||
assert "[redacted]" in result.error
|
||||
|
||||
def test_download_errors_redact_signed_url_and_preserve_task_id(
|
||||
self, monkeypatch
|
||||
):
|
||||
def test_download_errors_redact_signed_url_and_preserve_task_id(self, monkeypatch):
|
||||
monkeypatch.setenv("ARK_API_KEY", "fake-ark-key")
|
||||
|
||||
def fake_post(*args, **kwargs):
|
||||
@@ -439,9 +452,7 @@ class TestInputSafety:
|
||||
assert tiny_result.success is False
|
||||
assert "300 to 6000" in tiny_result.error
|
||||
|
||||
def test_corrupt_or_tiny_image_data_uri_never_reaches_network(
|
||||
self, monkeypatch
|
||||
):
|
||||
def test_corrupt_or_tiny_image_data_uri_never_reaches_network(self, monkeypatch):
|
||||
from io import BytesIO
|
||||
|
||||
from PIL import Image
|
||||
@@ -465,10 +476,9 @@ class TestInputSafety:
|
||||
|
||||
buffer = BytesIO()
|
||||
Image.new("RGB", (1, 1), "white").save(buffer, format="PNG")
|
||||
tiny_uri = (
|
||||
"data:image/png;base64,"
|
||||
+ base64.b64encode(buffer.getvalue()).decode("ascii")
|
||||
)
|
||||
tiny_uri = "data:image/png;base64," + base64.b64encode(
|
||||
buffer.getvalue()
|
||||
).decode("ascii")
|
||||
tiny = SeedanceArkVideo().execute(
|
||||
{
|
||||
"task_action": "create",
|
||||
@@ -487,15 +497,11 @@ class TestInputSafety:
|
||||
raise AssertionError("invalid API Key config must not call network")
|
||||
|
||||
monkeypatch.setattr("requests.post", fail_network)
|
||||
result = SeedanceArkVideo().execute(
|
||||
{"task_action": "create", "prompt": "x"}
|
||||
)
|
||||
result = SeedanceArkVideo().execute({"task_action": "create", "prompt": "x"})
|
||||
assert result.success is False
|
||||
assert "remove the 'Bearer ' prefix" in result.error
|
||||
|
||||
def test_rejects_local_reference_video_before_network(
|
||||
self, monkeypatch, tmp_path
|
||||
):
|
||||
def test_rejects_local_reference_video_before_network(self, monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("ARK_API_KEY", "fake-ark-key")
|
||||
video = tmp_path / "reference.mp4"
|
||||
video.write_bytes(b"not-a-real-video")
|
||||
@@ -533,9 +539,7 @@ class TestInputSafety:
|
||||
assert result.success is False
|
||||
assert "only 480p or 720p" in result.error
|
||||
|
||||
def test_short_local_audio_never_reaches_network(
|
||||
self, monkeypatch, tmp_path
|
||||
):
|
||||
def test_short_local_audio_never_reaches_network(self, monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("ARK_API_KEY", "fake-ark-key")
|
||||
audio = tmp_path / "short.wav"
|
||||
with wave.open(str(audio), "wb") as writer:
|
||||
@@ -568,9 +572,7 @@ class TestInputSafety:
|
||||
raise AssertionError("local cost config error must precede paid POST")
|
||||
|
||||
monkeypatch.setattr("requests.post", fail_network)
|
||||
result = SeedanceArkVideo().execute(
|
||||
{"task_action": "create", "prompt": "x"}
|
||||
)
|
||||
result = SeedanceArkVideo().execute({"task_action": "create", "prompt": "x"})
|
||||
assert result.success is False
|
||||
assert result.data == {}
|
||||
|
||||
@@ -585,9 +587,7 @@ class TestInputSafety:
|
||||
raise AssertionError("non-finite exchange rate reached paid POST")
|
||||
|
||||
monkeypatch.setattr("requests.post", fail_network)
|
||||
result = SeedanceArkVideo().execute(
|
||||
{"task_action": "create", "prompt": "x"}
|
||||
)
|
||||
result = SeedanceArkVideo().execute({"task_action": "create", "prompt": "x"})
|
||||
assert result.success is False
|
||||
assert "finite" in result.error
|
||||
|
||||
@@ -640,9 +640,7 @@ class TestInputSafety:
|
||||
assert result.success is False
|
||||
assert "finite" in result.error
|
||||
|
||||
def test_custom_query_without_price_reports_unknown_not_zero(
|
||||
self, monkeypatch
|
||||
):
|
||||
def test_custom_query_without_price_reports_unknown_not_zero(self, monkeypatch):
|
||||
monkeypatch.setenv("ARK_API_KEY", "fake-ark-key")
|
||||
monkeypatch.setattr(
|
||||
"requests.get",
|
||||
@@ -679,9 +677,7 @@ class TestInputSafety:
|
||||
{"error": {"code": "InternalError"}},
|
||||
status_code=503,
|
||||
)
|
||||
return _FakeResponse(
|
||||
{"id": "cgt-test-123", "status": "running"}
|
||||
)
|
||||
return _FakeResponse({"id": "cgt-test-123", "status": "running"})
|
||||
|
||||
monkeypatch.setattr("requests.get", fake_get)
|
||||
monkeypatch.setattr("time.sleep", lambda *_: None)
|
||||
@@ -782,9 +778,7 @@ class TestOfficialCostFormula:
|
||||
assert dry["valid"] is False
|
||||
assert "pricing is unknown" in dry["error"]
|
||||
|
||||
def test_custom_endpoint_requires_price_before_paid_post(
|
||||
self, monkeypatch
|
||||
):
|
||||
def test_custom_endpoint_requires_price_before_paid_post(self, monkeypatch):
|
||||
monkeypatch.setenv("ARK_API_KEY", "fake-ark-key")
|
||||
|
||||
def fail_network(*args, **kwargs):
|
||||
@@ -836,9 +830,7 @@ class TestOfficialCostFormula:
|
||||
assert result.success is False
|
||||
assert "finite" in result.error
|
||||
|
||||
def test_valid_audio_data_uri_reaches_mocked_create(
|
||||
self, monkeypatch, tmp_path
|
||||
):
|
||||
def test_valid_audio_data_uri_reaches_mocked_create(self, monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("ARK_API_KEY", "fake-ark-key")
|
||||
audio = tmp_path / "two-seconds.wav"
|
||||
with wave.open(str(audio), "wb") as writer:
|
||||
@@ -846,10 +838,9 @@ class TestOfficialCostFormula:
|
||||
writer.setsampwidth(2)
|
||||
writer.setframerate(8_000)
|
||||
writer.writeframes(b"\x00\x00" * 16_000)
|
||||
audio_uri = (
|
||||
"data:audio/wav;base64,"
|
||||
+ base64.b64encode(audio.read_bytes()).decode("ascii")
|
||||
)
|
||||
audio_uri = "data:audio/wav;base64," + base64.b64encode(
|
||||
audio.read_bytes()
|
||||
).decode("ascii")
|
||||
observed = {}
|
||||
|
||||
def fake_post(url, **kwargs):
|
||||
|
||||
506
tests/tools/test_minimax_video.py
Normal file
506
tests/tools/test_minimax_video.py
Normal file
@@ -0,0 +1,506 @@
|
||||
"""Contract coverage for the first-party MiniMax video provider.
|
||||
|
||||
These tests patch the live ``requests`` module so the shared import remains
|
||||
available to the rest of the test suite.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from tools.base_tool import ToolStatus
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
def __init__(self, *, json_data=None, content=b""):
|
||||
self._json = json_data or {}
|
||||
self.content = content
|
||||
|
||||
def raise_for_status(self):
|
||||
return None
|
||||
|
||||
def json(self):
|
||||
return self._json
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_minimax_environment(monkeypatch):
|
||||
monkeypatch.delenv("MINIMAX_API_KEY", raising=False)
|
||||
monkeypatch.delenv("MINIMAX_REGION", raising=False)
|
||||
monkeypatch.delenv("MINIMAX_BASE_URL", raising=False)
|
||||
|
||||
|
||||
def _h3_success_task(task_id="task-h3"):
|
||||
return {
|
||||
"task": {
|
||||
"id": task_id,
|
||||
"model": "MiniMax-H3",
|
||||
"status": "succeeded",
|
||||
"error": None,
|
||||
"content": {"url": "https://cdn.example/h3.mp4"},
|
||||
"resolution": "2K",
|
||||
"duration": 5,
|
||||
"usage": {
|
||||
"total_seconds": 5,
|
||||
"input_seconds": 0,
|
||||
"output_seconds": 5,
|
||||
"image_count": 0,
|
||||
},
|
||||
"ratio": "16:9",
|
||||
"task_type": "video_generation",
|
||||
"modality": "text_to_video",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def test_minimax_video_is_discovered_as_direct_provider():
|
||||
from tools.tool_registry import ToolRegistry
|
||||
from tools.video.minimax_video import DEFAULT_MODEL, MODELS
|
||||
|
||||
registry = ToolRegistry()
|
||||
registry.discover()
|
||||
|
||||
tool = registry.get("minimax_video")
|
||||
assert tool is not None
|
||||
assert tool.provider == "minimax"
|
||||
assert tool.capability == "video_generation"
|
||||
assert tool.dependencies == ["env:MINIMAX_API_KEY"]
|
||||
assert DEFAULT_MODEL == "MiniMax-H3"
|
||||
assert MODELS == [
|
||||
"MiniMax-H3",
|
||||
"MiniMax-Hailuo-2.3",
|
||||
"MiniMax-Hailuo-2.3-Fast",
|
||||
"MiniMax-Hailuo-02",
|
||||
"T2V-01-Director",
|
||||
"T2V-01",
|
||||
"I2V-01-Director",
|
||||
"I2V-01-live",
|
||||
"I2V-01",
|
||||
]
|
||||
|
||||
|
||||
def test_minimax_video_unavailable_without_key(monkeypatch):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.delenv("MINIMAX_API_KEY", raising=False)
|
||||
assert MiniMaxVideo().get_status() == ToolStatus.UNAVAILABLE
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
assert MiniMaxVideo().get_status() == ToolStatus.AVAILABLE
|
||||
|
||||
|
||||
def test_minimax_video_region_routing(monkeypatch):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
tool = MiniMaxVideo()
|
||||
monkeypatch.delenv("MINIMAX_BASE_URL", raising=False)
|
||||
monkeypatch.delenv("MINIMAX_REGION", raising=False)
|
||||
assert tool._base_url() == "https://api.minimax.io"
|
||||
|
||||
monkeypatch.setenv("MINIMAX_REGION", "global_en")
|
||||
assert tool._base_url() == "https://api.minimax.io"
|
||||
|
||||
monkeypatch.setenv("MINIMAX_REGION", "cn_zh")
|
||||
assert tool._base_url() == "https://api.minimaxi.com"
|
||||
|
||||
monkeypatch.setenv("MINIMAX_BASE_URL", "https://proxy.example.com/")
|
||||
assert tool._base_url() == "https://proxy.example.com"
|
||||
|
||||
|
||||
def test_minimax_h3_text_to_video_v2_contract(monkeypatch, tmp_path):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
monkeypatch.setenv("MINIMAX_REGION", "global")
|
||||
monkeypatch.setattr("time.sleep", lambda _seconds: None)
|
||||
calls = {}
|
||||
|
||||
def fake_post(url, headers=None, json=None, timeout=None):
|
||||
calls["post_url"] = url
|
||||
calls["post_headers"] = headers
|
||||
calls["post_payload"] = json
|
||||
return _FakeResponse(json_data={"task_id": "task-h3"})
|
||||
|
||||
def fake_get(url, headers=None, params=None, timeout=None):
|
||||
calls.setdefault("get_urls", []).append(url)
|
||||
if url.endswith("/v2/query/video_generation/task-h3"):
|
||||
return _FakeResponse(json_data=_h3_success_task())
|
||||
return _FakeResponse(content=b"h3 video bytes")
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
monkeypatch.setattr(requests, "get", fake_get)
|
||||
|
||||
output_path = tmp_path / "h3.mp4"
|
||||
result = MiniMaxVideo().execute(
|
||||
{
|
||||
"prompt": "A calm product shot, slow dolly-in",
|
||||
"output_path": str(output_path),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success, result.error
|
||||
assert calls["post_url"] == "https://api.minimax.io/v2/video_generation"
|
||||
assert calls["post_headers"]["Authorization"] == "Bearer test-minimax-key"
|
||||
assert calls["post_payload"] == {
|
||||
"model": "MiniMax-H3",
|
||||
"content": [
|
||||
{"type": "text", "text": "A calm product shot, slow dolly-in"},
|
||||
],
|
||||
"resolution": "2K",
|
||||
"duration": 5,
|
||||
"ratio": "16:9",
|
||||
}
|
||||
assert (
|
||||
"https://api.minimax.io/v2/query/video_generation/task-h3" in calls["get_urls"]
|
||||
)
|
||||
assert output_path.read_bytes() == b"h3 video bytes"
|
||||
assert result.data["api_version"] == "v2"
|
||||
assert result.data["task"]["status"] == "succeeded"
|
||||
assert result.model == "MiniMax-H3"
|
||||
|
||||
|
||||
def test_minimax_h3_reference_content_and_cn_watermark(monkeypatch, tmp_path):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
monkeypatch.setenv("MINIMAX_REGION", "cn")
|
||||
monkeypatch.setattr("time.sleep", lambda _seconds: None)
|
||||
calls = {}
|
||||
|
||||
def fake_post(url, headers=None, json=None, timeout=None):
|
||||
calls["url"] = url
|
||||
calls["payload"] = json
|
||||
return _FakeResponse(json_data={"task_id": "task-reference"})
|
||||
|
||||
def fake_get(url, headers=None, params=None, timeout=None):
|
||||
if url.endswith("/v2/query/video_generation/task-reference"):
|
||||
task = _h3_success_task("task-reference")
|
||||
task["task"]["ratio"] = "adaptive"
|
||||
task["task"]["modality"] = "reference_to_video"
|
||||
return _FakeResponse(json_data=task)
|
||||
return _FakeResponse(content=b"reference video bytes")
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
monkeypatch.setattr(requests, "get", fake_get)
|
||||
|
||||
result = MiniMaxVideo().execute(
|
||||
{
|
||||
"prompt": "Keep the character and camera rhythm consistent",
|
||||
"operation": "reference_to_video",
|
||||
"reference_image_urls": ["https://cdn.example/reference.png"],
|
||||
"reference_video_url": "https://cdn.example/reference.mp4",
|
||||
"reference_audio_urls": ["https://cdn.example/reference.wav"],
|
||||
"aigc_watermark": True,
|
||||
"output_path": str(tmp_path / "reference.mp4"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success, result.error
|
||||
assert calls["url"] == "https://api.minimaxi.com/v2/video_generation"
|
||||
assert calls["payload"]["aigc_watermark"] is True
|
||||
assert calls["payload"]["ratio"] == "adaptive"
|
||||
assert [
|
||||
(item["type"], item.get("role")) for item in calls["payload"]["content"]
|
||||
] == [
|
||||
("text", None),
|
||||
("image_url", "reference_image"),
|
||||
("video_url", "reference_video"),
|
||||
("audio_url", "reference_audio"),
|
||||
]
|
||||
|
||||
|
||||
def test_minimax_h3_validates_frame_and_audio_rules(monkeypatch):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
tool = MiniMaxVideo()
|
||||
|
||||
missing_last = tool.execute(
|
||||
{
|
||||
"prompt": "Transition between these frames",
|
||||
"operation": "first_last_frame_to_video",
|
||||
"first_frame_image": "https://cdn.example/first.png",
|
||||
}
|
||||
)
|
||||
assert not missing_last.success
|
||||
assert "last_frame_image" in missing_last.error
|
||||
|
||||
audio_only = tool.execute(
|
||||
{
|
||||
"prompt": "Follow this rhythm",
|
||||
"operation": "reference_to_video",
|
||||
"reference_audio_urls": ["https://cdn.example/reference.wav"],
|
||||
}
|
||||
)
|
||||
assert not audio_only.success
|
||||
assert "requires at least one reference image or video" in audio_only.error
|
||||
|
||||
|
||||
def test_minimax_h3_normalizes_selector_duration_and_first_last_frames(monkeypatch):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
tool = MiniMaxVideo()
|
||||
payload, error = tool._build_v2_payload(
|
||||
{
|
||||
"prompt": "Move smoothly between these frames",
|
||||
"operation": "first_last_frame_to_video",
|
||||
"first_frame_image": "https://cdn.example/first.png",
|
||||
"last_frame_image": "https://cdn.example/last.png",
|
||||
"duration": "10",
|
||||
"aspect_ratio": "9:16",
|
||||
},
|
||||
"https://api.minimax.io",
|
||||
)
|
||||
|
||||
assert error is None
|
||||
assert payload["duration"] == 10
|
||||
assert payload["ratio"] == "9:16"
|
||||
assert [item.get("role") for item in payload["content"]] == [
|
||||
None,
|
||||
"first_frame",
|
||||
"last_frame",
|
||||
]
|
||||
assert (
|
||||
tool.estimate_cost(
|
||||
{
|
||||
"duration": "10",
|
||||
"reference_image_urls": [
|
||||
f"https://cdn.example/{index}.png" for index in range(6)
|
||||
],
|
||||
}
|
||||
)
|
||||
== 1.33
|
||||
)
|
||||
|
||||
|
||||
def test_minimax_hailuo_text_to_video_v1_contract(monkeypatch, tmp_path):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
monkeypatch.setenv("MINIMAX_REGION", "global")
|
||||
monkeypatch.setattr("time.sleep", lambda _seconds: None)
|
||||
calls = {}
|
||||
|
||||
def fake_post(url, headers=None, json=None, timeout=None):
|
||||
calls["post_url"] = url
|
||||
calls["post_payload"] = json
|
||||
return _FakeResponse(
|
||||
json_data={"task_id": "task-v1", "base_resp": {"status_code": 0}}
|
||||
)
|
||||
|
||||
def fake_get(url, headers=None, params=None, timeout=None):
|
||||
calls.setdefault("get", []).append((url, params))
|
||||
if url.endswith("/v1/query/video_generation"):
|
||||
return _FakeResponse(
|
||||
json_data={
|
||||
"status": "Success",
|
||||
"file_id": "file-v1",
|
||||
"base_resp": {"status_code": 0},
|
||||
}
|
||||
)
|
||||
if url.endswith("/v1/files/retrieve"):
|
||||
return _FakeResponse(
|
||||
json_data={
|
||||
"file": {"download_url": "https://cdn.example/v1.mp4"},
|
||||
"base_resp": {"status_code": 0},
|
||||
}
|
||||
)
|
||||
return _FakeResponse(content=b"v1 video bytes")
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
monkeypatch.setattr(requests, "get", fake_get)
|
||||
|
||||
output_path = tmp_path / "v1.mp4"
|
||||
result = MiniMaxVideo().execute(
|
||||
{
|
||||
"prompt": "A calm product shot",
|
||||
"model": "MiniMax-Hailuo-2.3",
|
||||
"duration": 6,
|
||||
"resolution": "1080P",
|
||||
"output_path": str(output_path),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success, result.error
|
||||
assert calls["post_url"] == "https://api.minimax.io/v1/video_generation"
|
||||
assert calls["post_payload"] == {
|
||||
"model": "MiniMax-Hailuo-2.3",
|
||||
"prompt": "A calm product shot",
|
||||
"duration": 6,
|
||||
"resolution": "1080P",
|
||||
}
|
||||
get_urls = [url for url, _params in calls["get"]]
|
||||
assert "https://api.minimax.io/v1/query/video_generation" in get_urls
|
||||
assert "https://api.minimax.io/v1/files/retrieve" in get_urls
|
||||
assert output_path.read_bytes() == b"v1 video bytes"
|
||||
assert result.data["api_version"] == "v1"
|
||||
assert result.data["file_id"] == "file-v1"
|
||||
|
||||
|
||||
def test_minimax_hailuo_image_to_video_prompt_is_optional(monkeypatch, tmp_path):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
monkeypatch.setattr("time.sleep", lambda _seconds: None)
|
||||
calls = {}
|
||||
|
||||
def fake_post(url, headers=None, json=None, timeout=None):
|
||||
calls["payload"] = json
|
||||
return _FakeResponse(
|
||||
json_data={"task_id": "task-v1-image", "base_resp": {"status_code": 0}}
|
||||
)
|
||||
|
||||
def fake_get(url, headers=None, params=None, timeout=None):
|
||||
if url.endswith("/v1/query/video_generation"):
|
||||
return _FakeResponse(
|
||||
json_data={
|
||||
"status": "Success",
|
||||
"file_id": "file-v1-image",
|
||||
"base_resp": {"status_code": 0},
|
||||
}
|
||||
)
|
||||
if url.endswith("/v1/files/retrieve"):
|
||||
return _FakeResponse(
|
||||
json_data={
|
||||
"file": {"download_url": "https://cdn.example/v1-image.mp4"},
|
||||
"base_resp": {"status_code": 0},
|
||||
}
|
||||
)
|
||||
return _FakeResponse(content=b"v1 image video bytes")
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
monkeypatch.setattr(requests, "get", fake_get)
|
||||
|
||||
result = MiniMaxVideo().execute(
|
||||
{
|
||||
"operation": "image_to_video",
|
||||
"model": "I2V-01",
|
||||
"first_frame_image": "https://cdn.example/first.png",
|
||||
"output_path": str(tmp_path / "v1-image.mp4"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success, result.error
|
||||
assert calls["payload"] == {
|
||||
"model": "I2V-01",
|
||||
"first_frame_image": "https://cdn.example/first.png",
|
||||
}
|
||||
|
||||
|
||||
def test_video_selector_routes_reference_image_to_minimax_h3(monkeypatch, tmp_path):
|
||||
from tools.video import _shared
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
from tools.video.video_selector import VideoSelector
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
monkeypatch.setattr("time.sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(
|
||||
_shared,
|
||||
"upload_image_fal",
|
||||
lambda _path: "https://cdn.example/reference.png",
|
||||
)
|
||||
calls = {}
|
||||
|
||||
def fake_post(url, headers=None, json=None, timeout=None):
|
||||
calls["payload"] = json
|
||||
return _FakeResponse(json_data={"task_id": "task-selector"})
|
||||
|
||||
def fake_get(url, headers=None, params=None, timeout=None):
|
||||
if url.endswith("/v2/query/video_generation/task-selector"):
|
||||
return _FakeResponse(json_data=_h3_success_task("task-selector"))
|
||||
return _FakeResponse(content=b"selector video bytes")
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
monkeypatch.setattr(requests, "get", fake_get)
|
||||
|
||||
tool = MiniMaxVideo()
|
||||
selector = VideoSelector()
|
||||
monkeypatch.setattr(selector, "_providers", lambda: [tool])
|
||||
monkeypatch.setattr(
|
||||
selector,
|
||||
"_select_best_tool",
|
||||
lambda _inputs, _candidates, _context: (tool, None),
|
||||
)
|
||||
|
||||
result = selector.execute(
|
||||
{
|
||||
"prompt": "A calm product shot",
|
||||
"operation": "image_to_video",
|
||||
"reference_image_path": str(tmp_path / "reference.png"),
|
||||
"output_path": str(tmp_path / "selector.mp4"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success, result.error
|
||||
first_frame = calls["payload"]["content"][1]
|
||||
assert first_frame == {
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://cdn.example/reference.png"},
|
||||
"role": "first_frame",
|
||||
}
|
||||
assert result.data["selected_tool"] == "minimax_video"
|
||||
|
||||
|
||||
def test_minimax_video_surfaces_v1_base_resp_error(monkeypatch):
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
|
||||
def fake_post(url, headers=None, json=None, timeout=None):
|
||||
return _FakeResponse(
|
||||
json_data={"base_resp": {"status_code": 1004, "status_msg": "auth failed"}}
|
||||
)
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
|
||||
result = MiniMaxVideo().execute({"prompt": "hi", "model": "MiniMax-Hailuo-2.3"})
|
||||
assert not result.success
|
||||
assert "1004" in result.error
|
||||
|
||||
|
||||
@pytest.mark.parametrize("model", ["MiniMax-H3", "MiniMax-Hailuo-2.3"])
|
||||
def test_polling_timeout_is_bounded_and_preserves_task_id(monkeypatch, model):
|
||||
import tools.video.minimax_video as minimax_module
|
||||
from tools.video.minimax_video import MiniMaxVideo
|
||||
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key")
|
||||
monkeypatch.setattr(
|
||||
requests,
|
||||
"post",
|
||||
lambda *args, **kwargs: _FakeResponse(
|
||||
json_data={"task_id": "recoverable-task", "base_resp": {"status_code": 0}}
|
||||
),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
requests,
|
||||
"get",
|
||||
lambda *args, **kwargs: pytest.fail("deadline must stop polling"),
|
||||
)
|
||||
|
||||
class _ExpiredClock:
|
||||
def __init__(self):
|
||||
self._ticks = iter([0.0, 2.0])
|
||||
|
||||
@staticmethod
|
||||
def time():
|
||||
return 0.0
|
||||
|
||||
def monotonic(self):
|
||||
return next(self._ticks)
|
||||
|
||||
@staticmethod
|
||||
def sleep(_seconds):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(minimax_module, "time", _ExpiredClock())
|
||||
|
||||
result = MiniMaxVideo().execute(
|
||||
{"prompt": "A camera move", "model": model, "timeout_seconds": 1}
|
||||
)
|
||||
|
||||
assert not result.success
|
||||
assert "timed out" in (result.error or "")
|
||||
assert result.data["task_id"] == "recoverable-task"
|
||||
assert result.data["status"] == "timed_out"
|
||||
264
tests/tools/test_new_video_model_support.py
Normal file
264
tests/tools/test_new_video_model_support.py
Normal file
@@ -0,0 +1,264 @@
|
||||
"""Contracts for the August 2026 video-model provider refresh."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import requests
|
||||
|
||||
|
||||
class _Response:
|
||||
def __init__(self, payload=None, content=b"video", status_code=200):
|
||||
self.payload = payload or {}
|
||||
self.content = content
|
||||
self.status_code = status_code
|
||||
self.ok = status_code < 400
|
||||
self.text = str(self.payload)
|
||||
|
||||
def json(self):
|
||||
return self.payload
|
||||
|
||||
def raise_for_status(self):
|
||||
if not self.ok:
|
||||
raise RuntimeError(f"HTTP {self.status_code}")
|
||||
|
||||
|
||||
def _queue_mocks(monkeypatch):
|
||||
calls = {"posts": []}
|
||||
|
||||
def post(url, headers=None, json=None, timeout=None):
|
||||
calls["posts"].append((url, json))
|
||||
return _Response(
|
||||
{"status_url": "https://status", "response_url": "https://result"}
|
||||
)
|
||||
|
||||
def get(url, headers=None, timeout=None, params=None):
|
||||
if url == "https://status":
|
||||
return _Response({"status": "COMPLETED"})
|
||||
if url == "https://result":
|
||||
return _Response({"video": {"url": "https://video"}, "seed": 9})
|
||||
return _Response(content=b"fake mp4")
|
||||
|
||||
monkeypatch.setattr(requests, "post", post)
|
||||
monkeypatch.setattr(requests, "get", get)
|
||||
monkeypatch.setattr("time.sleep", lambda _: None)
|
||||
return calls
|
||||
|
||||
|
||||
def test_fal_seedance_25_uses_current_endpoint_and_reference_fields(
|
||||
monkeypatch, tmp_path
|
||||
):
|
||||
from tools.video.seedance_video import SeedanceVideo
|
||||
|
||||
monkeypatch.setenv("FAL_KEY", "test")
|
||||
calls = _queue_mocks(monkeypatch)
|
||||
result = SeedanceVideo().execute(
|
||||
{
|
||||
"prompt": "Use @Image1 as the hero",
|
||||
"model_version": "2.5",
|
||||
"operation": "reference_to_video",
|
||||
"duration": "30",
|
||||
"reference_image_urls": ["https://image"],
|
||||
"reference_video_urls": ["https://motion"],
|
||||
"reference_audio_urls": ["https://voice"],
|
||||
"output_path": str(tmp_path / "seedance25.mp4"),
|
||||
}
|
||||
)
|
||||
assert result.success, result.error
|
||||
url, payload = calls["posts"][0]
|
||||
assert url.endswith("/bytedance/seedance-2.5/reference-to-video")
|
||||
assert payload["image_urls"] == ["https://image"]
|
||||
assert payload["video_urls"] == ["https://motion"]
|
||||
assert payload["audio_urls"] == ["https://voice"]
|
||||
assert payload["duration"] == "30"
|
||||
|
||||
|
||||
def test_fal_gemini_omni_and_minimax_h3_are_discovered_and_submit(
|
||||
monkeypatch, tmp_path
|
||||
):
|
||||
from tools.tool_registry import ToolRegistry
|
||||
from tools.video.gemini_omni_fal import GeminiOmniFalVideo
|
||||
from tools.video.minimax_fal_video import MiniMaxFalVideo
|
||||
|
||||
monkeypatch.setenv("FAL_KEY", "test")
|
||||
registry = ToolRegistry()
|
||||
registry.discover()
|
||||
assert registry.get("gemini_omni_fal") is not None
|
||||
assert registry.get("minimax_fal_video") is not None
|
||||
|
||||
calls = _queue_mocks(monkeypatch)
|
||||
gemini = GeminiOmniFalVideo().execute(
|
||||
{
|
||||
"prompt": "Animate <IMAGE_REF_0>",
|
||||
"operation": "image_to_video",
|
||||
"image_url": "https://image",
|
||||
"output_path": str(tmp_path / "omni.mp4"),
|
||||
}
|
||||
)
|
||||
assert gemini.success, gemini.error
|
||||
assert calls["posts"][0][0].endswith("/google/gemini-omni-flash/image-to-video")
|
||||
|
||||
calls = _queue_mocks(monkeypatch)
|
||||
edited = GeminiOmniFalVideo().execute(
|
||||
{
|
||||
"prompt": "Remove the sign",
|
||||
"operation": "edit_video",
|
||||
"video_url": "https://video-input",
|
||||
"output_path": str(tmp_path / "omni-edit.mp4"),
|
||||
}
|
||||
)
|
||||
assert edited.success, edited.error
|
||||
assert calls["posts"][0][0].endswith("/google/gemini-omni-flash/edit")
|
||||
assert calls["posts"][0][1] == {
|
||||
"prompt": "Remove the sign",
|
||||
"video_url": "https://video-input",
|
||||
}
|
||||
|
||||
calls = _queue_mocks(monkeypatch)
|
||||
minimax = MiniMaxFalVideo().execute(
|
||||
{
|
||||
"prompt": "A dolly shot",
|
||||
"duration": 15,
|
||||
"output_path": str(tmp_path / "h3.mp4"),
|
||||
}
|
||||
)
|
||||
assert minimax.success, minimax.error
|
||||
assert calls["posts"][0][0].endswith("/fal-ai/minimax/hailuo-03/text-to-video")
|
||||
|
||||
|
||||
def test_runway_supports_all_three_current_model_identifiers(monkeypatch, tmp_path):
|
||||
from tools.video.runway_video import RunwayVideo
|
||||
|
||||
monkeypatch.setenv("RUNWAY_API_KEY", "test")
|
||||
calls = {"posts": []}
|
||||
|
||||
def post(url, headers=None, json=None, timeout=None):
|
||||
calls["posts"].append((url, json))
|
||||
return _Response({"id": "task"})
|
||||
|
||||
def get(url, headers=None, timeout=None, params=None):
|
||||
if url.endswith("/tasks/task"):
|
||||
return _Response({"status": "SUCCEEDED", "output": ["https://video"]})
|
||||
return _Response(content=b"fake mp4")
|
||||
|
||||
monkeypatch.setattr(requests, "post", post)
|
||||
monkeypatch.setattr(requests, "get", get)
|
||||
monkeypatch.setattr("time.sleep", lambda _: None)
|
||||
for model, duration in (
|
||||
("seedance2_5", 30),
|
||||
("gemini_omni_flash", 10),
|
||||
("hailuo3", 15),
|
||||
):
|
||||
result = RunwayVideo().execute(
|
||||
{
|
||||
"prompt": "A cinematic shot",
|
||||
"model": model,
|
||||
"duration": duration,
|
||||
"output_path": str(tmp_path / f"{model}.mp4"),
|
||||
}
|
||||
)
|
||||
assert result.success, result.error
|
||||
assert calls["posts"][-1][1]["model"] == model
|
||||
|
||||
|
||||
def test_runway_uses_each_models_official_request_shape(monkeypatch, tmp_path):
|
||||
from tools.video.runway_video import RunwayVideo
|
||||
|
||||
monkeypatch.setenv("RUNWAY_API_KEY", "test")
|
||||
calls = {"posts": []}
|
||||
|
||||
def post(url, headers=None, json=None, timeout=None):
|
||||
calls["posts"].append((url, json))
|
||||
return _Response({"id": "task"})
|
||||
|
||||
def get(url, headers=None, timeout=None, params=None):
|
||||
if url.endswith("/tasks/task"):
|
||||
return _Response({"status": "SUCCEEDED", "output": ["https://video"]})
|
||||
return _Response(content=b"fake mp4")
|
||||
|
||||
monkeypatch.setattr(requests, "post", post)
|
||||
monkeypatch.setattr(requests, "get", get)
|
||||
monkeypatch.setattr("time.sleep", lambda _: None)
|
||||
tool = RunwayVideo()
|
||||
|
||||
result = tool.execute(
|
||||
{
|
||||
"prompt": "Use the references",
|
||||
"model": "seedance2_5",
|
||||
"duration": 12,
|
||||
"resolution": "480p",
|
||||
"ratio": "9:16",
|
||||
"generate_audio": True,
|
||||
"reference_image_urls": ["https://image"],
|
||||
"reference_video_urls": ["https://motion"],
|
||||
"reference_audio_urls": ["https://audio"],
|
||||
"output_path": str(tmp_path / "seedance.mp4"),
|
||||
}
|
||||
)
|
||||
assert result.success, result.error
|
||||
payload = calls["posts"][-1][1]
|
||||
assert payload["ratio"] == "480:854"
|
||||
assert payload["audio"] is True
|
||||
assert payload["references"] == [{"uri": "https://image"}]
|
||||
assert payload["referenceVideos"] == [{"type": "video", "uri": "https://motion"}]
|
||||
assert payload["referenceAudio"] == [{"type": "audio", "uri": "https://audio"}]
|
||||
assert "resolution" not in payload
|
||||
|
||||
result = tool.execute(
|
||||
{
|
||||
"prompt": "Restyle the motion",
|
||||
"model": "hailuo3",
|
||||
"operation": "video_to_video",
|
||||
"duration": 8,
|
||||
"resolution": "768P",
|
||||
"video_url": "https://source",
|
||||
"reference_image_urls": ["https://image"],
|
||||
"output_path": str(tmp_path / "hailuo.mp4"),
|
||||
}
|
||||
)
|
||||
assert result.success, result.error
|
||||
payload = calls["posts"][-1][1]
|
||||
assert payload["promptVideo"] == "https://source"
|
||||
assert payload["resolution"] == "768P"
|
||||
assert payload["references"] == [{"uri": "https://image"}]
|
||||
assert "videoUri" not in payload
|
||||
|
||||
result = tool.execute(
|
||||
{
|
||||
"prompt": "Remove the sign",
|
||||
"model": "gemini_omni_flash",
|
||||
"operation": "video_to_video",
|
||||
"video_url": "https://source",
|
||||
"reference_image_urls": ["https://image"],
|
||||
"output_path": str(tmp_path / "gemini.mp4"),
|
||||
}
|
||||
)
|
||||
assert result.success, result.error
|
||||
payload = calls["posts"][-1][1]
|
||||
assert payload["videoUri"] == "https://source"
|
||||
assert payload["references"] == [{"uri": "https://image"}]
|
||||
assert "duration" not in payload
|
||||
assert "ratio" not in payload
|
||||
|
||||
|
||||
def test_comfyui_partner_workflows_and_local_h3_metadata():
|
||||
from tools.video.comfyui_video import ComfyUIVideo
|
||||
|
||||
tool = ComfyUIVideo()
|
||||
for family, node_class in (
|
||||
("gemini_omni_flash", "GeminiVideoOmni"),
|
||||
("seedance_2.5", "ByteDance2TextToVideoNode"),
|
||||
("minimax_h3_api", "MinimaxHailuo03TextToVideoNode"),
|
||||
):
|
||||
workflow, output_node = tool._build_partner_t2v(
|
||||
{"prompt": "A product reveal", "duration": 5},
|
||||
42,
|
||||
__import__("pathlib").Path("clip.mp4"),
|
||||
family,
|
||||
)
|
||||
assert workflow["1"]["class_type"] == node_class
|
||||
assert workflow["2"]["class_type"] == "SaveVideo"
|
||||
assert output_node == "2"
|
||||
|
||||
local = tool.execute({"prompt": "x", "model_family": "minimax_h3_local"})
|
||||
assert not local.success
|
||||
assert local.data["model"] == "MiniMax-H3"
|
||||
assert len(local.data["model_stack"]) == 4
|
||||
@@ -56,8 +56,8 @@ class ComfyUIClient:
|
||||
self._capability_env_var = (
|
||||
f"COMFYUI_{capability.upper()}_SERVER_URL" if capability else None
|
||||
)
|
||||
resolved = server_url or self._capability_url() or os.environ.get(
|
||||
"COMFYUI_SERVER_URL"
|
||||
resolved = (
|
||||
server_url or self._capability_url() or os.environ.get("COMFYUI_SERVER_URL")
|
||||
)
|
||||
self.server_url = (resolved or "http://localhost:8188").rstrip("/")
|
||||
# Scopes websocket execution events to this client (see wait_ws) and
|
||||
@@ -98,9 +98,7 @@ class ComfyUIClient:
|
||||
def is_available(self) -> bool:
|
||||
"""Return True if the ComfyUI server is reachable."""
|
||||
try:
|
||||
resp = requests.get(
|
||||
f"{self.server_url}/system_stats", timeout=5
|
||||
)
|
||||
resp = requests.get(f"{self.server_url}/system_stats", timeout=5)
|
||||
return resp.status_code == 200
|
||||
except Exception:
|
||||
return False
|
||||
@@ -149,9 +147,7 @@ class ComfyUIClient:
|
||||
result[group] = []
|
||||
return result
|
||||
|
||||
def check_models(
|
||||
self, required: list[str]
|
||||
) -> tuple[list[str], list[str]]:
|
||||
def check_models(self, required: list[str]) -> tuple[list[str], list[str]]:
|
||||
"""Check which of *required* model filenames are available.
|
||||
|
||||
Returns ``(found, missing)`` — two lists of filenames.
|
||||
@@ -164,6 +160,17 @@ class ComfyUIClient:
|
||||
missing = [m for m in required if m not in all_models]
|
||||
return found, missing
|
||||
|
||||
def has_node(self, node_class: str) -> bool:
|
||||
"""Return whether the connected server exposes a node class."""
|
||||
try:
|
||||
response = requests.get(
|
||||
f"{self.server_url}/object_info/{node_class}", timeout=10
|
||||
)
|
||||
response.raise_for_status()
|
||||
return node_class in response.json()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Core cycle
|
||||
# ------------------------------------------------------------------
|
||||
@@ -307,9 +314,7 @@ class ComfyUIClient:
|
||||
if on_progress:
|
||||
on_progress(data)
|
||||
elif msg_type == "execution_error":
|
||||
raise ComfyUIError(
|
||||
f"Execution error: {data}", prompt_id=prompt_id
|
||||
)
|
||||
raise ComfyUIError(f"Execution error: {data}", prompt_id=prompt_id)
|
||||
elif msg_type == "executing" and data.get("node") is None:
|
||||
finished = True
|
||||
break
|
||||
@@ -445,6 +450,7 @@ class ComfyUIClient:
|
||||
node_output.get("images", [])
|
||||
or node_output.get("gifs", [])
|
||||
or node_output.get("audio", [])
|
||||
or node_output.get("video", [])
|
||||
)
|
||||
if not items:
|
||||
raise ComfyUIError(
|
||||
@@ -479,9 +485,7 @@ class ComfyUIClient:
|
||||
return json.load(f)
|
||||
|
||||
@staticmethod
|
||||
def patch_workflow(
|
||||
workflow: dict, patches: dict[str, dict[str, Any]]
|
||||
) -> dict:
|
||||
def patch_workflow(workflow: dict, patches: dict[str, dict[str, Any]]) -> dict:
|
||||
"""Deep-copy *workflow* and apply *patches*.
|
||||
|
||||
*patches* maps ``node_id`` → ``{input_name: value, ...}``.
|
||||
|
||||
@@ -30,6 +30,34 @@ COMFYUI_SETUP_OFFER: dict[str, Any] = {
|
||||
|
||||
|
||||
BUNDLED_MODEL_STACKS: dict[str, list[dict[str, Any]]] = {
|
||||
"minimax-h3-local": [
|
||||
{
|
||||
"role": "diffusion_model",
|
||||
"name": "minimax_h3_fl2va_pruned_int8_convrot.safetensors",
|
||||
"quantization": "INT8 ConvRot",
|
||||
"destination_hint": "ComfyUI/models/diffusion_models/",
|
||||
"download_url": "https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models",
|
||||
},
|
||||
{
|
||||
"role": "text_encoder",
|
||||
"name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
|
||||
"quantization": "NVFP4 AWQ",
|
||||
"destination_hint": "ComfyUI/models/text_encoders/",
|
||||
"download_url": "https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders",
|
||||
},
|
||||
{
|
||||
"role": "video_vae",
|
||||
"name": "minimax_h3_video_vae_fp16.safetensors",
|
||||
"destination_hint": "ComfyUI/models/vae/",
|
||||
"download_url": "https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/vae",
|
||||
},
|
||||
{
|
||||
"role": "audio_vae",
|
||||
"name": "minimax_h3_audio_vae_fp32.safetensors",
|
||||
"destination_hint": "ComfyUI/models/vae/",
|
||||
"download_url": "https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/vae",
|
||||
},
|
||||
],
|
||||
"flux2-txt2img": [
|
||||
{
|
||||
"role": "diffusion_model",
|
||||
@@ -204,7 +232,9 @@ def workflow_hash(workflow: dict[str, Any]) -> str:
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def model_stack(workflow_key: str | None, inputs: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
def model_stack(
|
||||
workflow_key: str | None, inputs: dict[str, Any]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Return bundled or caller-supplied model stack metadata."""
|
||||
if workflow_key:
|
||||
return [dict(item) for item in BUNDLED_MODEL_STACKS[workflow_key]]
|
||||
@@ -228,7 +258,9 @@ def missing_models_payload(
|
||||
meta = dict(stack_by_name.get(name, {}))
|
||||
meta.setdefault("name", name)
|
||||
meta.setdefault("role", "unknown")
|
||||
meta.setdefault("destination_hint", "ComfyUI/models/ matching the workflow node")
|
||||
meta.setdefault(
|
||||
"destination_hint", "ComfyUI/models/ matching the workflow node"
|
||||
)
|
||||
meta.setdefault("download_url", None)
|
||||
items.append(meta)
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@ _RESOURCE_PROFILES = {
|
||||
|
||||
class ComfyUIVideo(BaseTool):
|
||||
name = "comfyui_video"
|
||||
version = "0.1.0"
|
||||
version = "0.2.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "comfyui"
|
||||
@@ -107,11 +107,21 @@ class ComfyUIVideo(BaseTool):
|
||||
install_instructions = (
|
||||
"Start a ComfyUI server and set COMFYUI_SERVER_URL "
|
||||
"(default http://localhost:8188).\n"
|
||||
"Requires WAN 2.2 models and LightX2V LoRAs in ComfyUI's model directory.\n"
|
||||
"Bundled local WAN requires WAN 2.2 models and LightX2V LoRAs. Local "
|
||||
"MiniMax H3 requires its official model stack and exported API workflow.\n"
|
||||
"Gemini Omni, Seedance 2.5, and MiniMax H3 Partner Nodes require a "
|
||||
"logged-in Comfy account, credits, and network access.\n"
|
||||
"Running a separate ComfyUI instance for video? Set COMFYUI_VIDEO_SERVER_URL "
|
||||
"instead -- it takes priority over COMFYUI_SERVER_URL for this tool only."
|
||||
)
|
||||
agent_skills = ["comfyui", "ai-video-gen", "ltx2"]
|
||||
agent_skills = [
|
||||
"comfyui",
|
||||
"gemini-omni",
|
||||
"seedance-2-5",
|
||||
"minimax-h3",
|
||||
"ai-video-gen",
|
||||
"ltx2",
|
||||
]
|
||||
|
||||
capabilities = ["text_to_video", "image_to_video"]
|
||||
supports = {
|
||||
@@ -120,6 +130,10 @@ class ComfyUIVideo(BaseTool):
|
||||
"custom_workflow": True,
|
||||
"custom_output_node": True,
|
||||
"offline": True,
|
||||
"gemini_omni_flash_partner_node": True,
|
||||
"seedance_2_5_partner_node": True,
|
||||
"minimax_h3_partner_node": True,
|
||||
"minimax_h3_local_weights": True,
|
||||
}
|
||||
best_for = [
|
||||
"local GPU video generation without API costs",
|
||||
@@ -127,6 +141,8 @@ class ComfyUIVideo(BaseTool):
|
||||
"image-to-video with WAN 2.2 14B (4-step accelerated)",
|
||||
"text-to-video with WAN 2.2 14B (4-step accelerated)",
|
||||
"custom low-VRAM ComfyUI workflows on 8GB-12GB GPUs",
|
||||
"ComfyUI Partner Node workflows for Gemini Omni Flash, Seedance 2.5, and MiniMax H3",
|
||||
"open-weight MiniMax H3 custom workflows with native stereo audio",
|
||||
]
|
||||
not_good_for = [
|
||||
"setups without a running ComfyUI server",
|
||||
@@ -140,12 +156,31 @@ class ComfyUIVideo(BaseTool):
|
||||
"type": "object",
|
||||
"required": ["prompt"],
|
||||
"properties": {
|
||||
"prompt": {"type": "string", "description": "Text prompt for video generation"},
|
||||
"prompt": {
|
||||
"type": "string",
|
||||
"description": "Text prompt for video generation",
|
||||
},
|
||||
"operation": {
|
||||
"type": "string",
|
||||
"enum": ["text_to_video", "image_to_video"],
|
||||
"default": "text_to_video",
|
||||
},
|
||||
"model_family": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"wan2.2",
|
||||
"gemini_omni_flash",
|
||||
"seedance_2.5",
|
||||
"minimax_h3_api",
|
||||
"minimax_h3_local",
|
||||
],
|
||||
"default": "wan2.2",
|
||||
"description": (
|
||||
"Built-in execution family. The three *_api/hosted families use "
|
||||
"ComfyUI Partner Nodes and credits; minimax_h3_local uses open weights "
|
||||
"through a caller-supplied official/custom workflow."
|
||||
),
|
||||
},
|
||||
"reference_image_path": {
|
||||
"type": "string",
|
||||
"description": "Local path to reference image (for image_to_video)",
|
||||
@@ -154,9 +189,39 @@ class ComfyUIVideo(BaseTool):
|
||||
"type": "string",
|
||||
"description": "URL of reference image (for image_to_video, downloaded first)",
|
||||
},
|
||||
"width": {"type": "integer", "default": 832, "description": "T2V default 832, I2V default 640"},
|
||||
"height": {"type": "integer", "default": 480, "description": "T2V default 480, I2V default 640"},
|
||||
"num_frames": {"type": "integer", "default": 81, "description": "81 frames = 5s at 16fps"},
|
||||
"width": {
|
||||
"type": "integer",
|
||||
"default": 832,
|
||||
"description": "T2V default 832, I2V default 640",
|
||||
},
|
||||
"height": {
|
||||
"type": "integer",
|
||||
"default": 480,
|
||||
"description": "T2V default 480, I2V default 640",
|
||||
},
|
||||
"num_frames": {
|
||||
"type": "integer",
|
||||
"default": 81,
|
||||
"description": "81 frames = 5s at 16fps",
|
||||
},
|
||||
"duration": {
|
||||
"type": "integer",
|
||||
"minimum": 3,
|
||||
"maximum": 30,
|
||||
"default": 5,
|
||||
"description": "Partner Node duration; bundled WAN still uses num_frames.",
|
||||
},
|
||||
"aspect_ratio": {
|
||||
"type": "string",
|
||||
"enum": ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"],
|
||||
"default": "16:9",
|
||||
},
|
||||
"resolution": {
|
||||
"type": "string",
|
||||
"enum": ["480p", "720p", "768P", "2K"],
|
||||
"default": "720p",
|
||||
},
|
||||
"generate_audio": {"type": "boolean", "default": True},
|
||||
"seed": {"type": "integer", "description": "Random if omitted"},
|
||||
"output_path": {"type": "string", "description": "Where to save the video"},
|
||||
"workflow_json": {
|
||||
@@ -210,12 +275,25 @@ class ComfyUIVideo(BaseTool):
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=2, ram_mb=16000, vram_mb=8000, disk_mb=2000, network_required=False,
|
||||
cpu_cores=2,
|
||||
ram_mb=16000,
|
||||
vram_mb=8000,
|
||||
disk_mb=2000,
|
||||
network_required=False,
|
||||
)
|
||||
retry_policy = RetryPolicy(max_retries=1, retryable_errors=["timeout"])
|
||||
idempotency_key_fields = ["prompt", "operation", "width", "height", "num_frames", "seed"]
|
||||
idempotency_key_fields = [
|
||||
"prompt",
|
||||
"operation",
|
||||
"width",
|
||||
"height",
|
||||
"num_frames",
|
||||
"seed",
|
||||
]
|
||||
side_effects = ["writes video file to output_path"]
|
||||
user_visible_verification = ["Watch generated clip for motion coherence and artifacts"]
|
||||
user_visible_verification = [
|
||||
"Watch generated clip for motion coherence and artifacts"
|
||||
]
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._client = ComfyUIClient(capability="video")
|
||||
@@ -282,9 +360,46 @@ class ComfyUIVideo(BaseTool):
|
||||
"workflows recommend 16GB VRAM; custom low-VRAM workflows can target "
|
||||
"8GB-12GB depending on model, quantization, resolution, and frame count."
|
||||
)
|
||||
info["execution_modes"] = {
|
||||
"wan2.2": {
|
||||
"hosted": False,
|
||||
"network_required": False,
|
||||
"cost": "local compute",
|
||||
},
|
||||
"minimax_h3_local": {
|
||||
"hosted": False,
|
||||
"network_required": False,
|
||||
"cost": "local compute",
|
||||
},
|
||||
"gemini_omni_flash": {
|
||||
"hosted": True,
|
||||
"network_required": True,
|
||||
"billing": "ComfyUI Partner Node credits",
|
||||
},
|
||||
"seedance_2.5": {
|
||||
"hosted": True,
|
||||
"network_required": True,
|
||||
"billing": "ComfyUI Partner Node credits",
|
||||
},
|
||||
"minimax_h3_api": {
|
||||
"hosted": True,
|
||||
"network_required": True,
|
||||
"billing": "ComfyUI Partner Node credits",
|
||||
},
|
||||
}
|
||||
return info
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
family = str(inputs.get("model_family", "wan2.2"))
|
||||
duration = float(inputs.get("duration", 5))
|
||||
if family == "gemini_omni_flash":
|
||||
return round(0.146 * duration, 4)
|
||||
if family == "seedance_2.5":
|
||||
rate = 0.1483 if inputs.get("resolution", "720p") == "480p" else 0.3333
|
||||
return round(rate * duration, 4)
|
||||
if family == "minimax_h3_api":
|
||||
rate = 0.1287 if inputs.get("resolution", "768P") == "768P" else 0.1859
|
||||
return round(rate * duration, 4)
|
||||
return 0.0
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
@@ -294,7 +409,40 @@ class ComfyUIVideo(BaseTool):
|
||||
return 240.0 # ~4 min
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
custom_workflow = bool(inputs.get("workflow_json") or inputs.get("workflow_path"))
|
||||
custom_workflow = bool(
|
||||
inputs.get("workflow_json") or inputs.get("workflow_path")
|
||||
)
|
||||
model_family = str(inputs.get("model_family", "wan2.2"))
|
||||
partner_nodes = {
|
||||
"gemini_omni_flash": "GeminiVideoOmni",
|
||||
"seedance_2.5": "ByteDance2TextToVideoNode",
|
||||
"minimax_h3_api": "MinimaxHailuo03TextToVideoNode",
|
||||
}
|
||||
if model_family == "minimax_h3_local" and not custom_workflow:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
data={
|
||||
"provider": "comfyui",
|
||||
"model": "MiniMax-H3",
|
||||
"workflow_template": "https://github.com/Comfy-Org/workflow_templates/blob/main/templates/video_minimax_h3_t2v.json",
|
||||
"model_stack": BUNDLED_MODEL_STACKS["minimax-h3-local"],
|
||||
},
|
||||
error=(
|
||||
"minimax_h3_local requires the official MiniMax H3 ComfyUI workflow "
|
||||
"exported in API format via workflow_json/workflow_path plus output_node."
|
||||
),
|
||||
)
|
||||
if (
|
||||
model_family in partner_nodes
|
||||
and inputs.get("operation", "text_to_video") != "text_to_video"
|
||||
):
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=(
|
||||
f"Built-in {model_family} Partner Node execution currently supports "
|
||||
"text_to_video; use an official ComfyUI custom workflow for other modes."
|
||||
),
|
||||
)
|
||||
if custom_workflow and not inputs.get("output_node"):
|
||||
return ToolResult(
|
||||
success=False,
|
||||
@@ -312,8 +460,12 @@ class ComfyUIVideo(BaseTool):
|
||||
|
||||
operation = inputs.get("operation", "text_to_video")
|
||||
|
||||
if not custom_workflow:
|
||||
required = _REQUIRED_MODELS_I2V if operation == "image_to_video" else _REQUIRED_MODELS_T2V
|
||||
if not custom_workflow and model_family == "wan2.2":
|
||||
required = (
|
||||
_REQUIRED_MODELS_I2V
|
||||
if operation == "image_to_video"
|
||||
else _REQUIRED_MODELS_T2V
|
||||
)
|
||||
_, missing = self._client.check_models(required)
|
||||
if missing:
|
||||
workflow_key = (
|
||||
@@ -345,13 +497,23 @@ class ComfyUIVideo(BaseTool):
|
||||
if custom_workflow:
|
||||
workflow = self._load_custom_workflow(inputs)
|
||||
output_node = str(inputs["output_node"])
|
||||
elif model_family in partner_nodes:
|
||||
node_class = partner_nodes[model_family]
|
||||
if not self._client.has_node(node_class):
|
||||
raise ComfyUIError(
|
||||
f"ComfyUI server does not expose {node_class}; update ComfyUI "
|
||||
"and enable Partner Nodes. These are hosted API nodes, not offline models."
|
||||
)
|
||||
workflow, output_node = self._build_partner_t2v(
|
||||
inputs, seed, output_path, model_family
|
||||
)
|
||||
elif operation == "image_to_video":
|
||||
workflow, output_node = self._build_i2v(inputs, seed, output_path)
|
||||
else:
|
||||
workflow, output_node = self._build_t2v(inputs, seed, output_path)
|
||||
|
||||
provenance = self._workflow_provenance(
|
||||
inputs, custom_workflow, output_node, operation, workflow
|
||||
inputs, custom_workflow, output_node, operation, workflow, model_family
|
||||
)
|
||||
paths = self._client.generate(
|
||||
workflow,
|
||||
@@ -377,31 +539,51 @@ class ComfyUIVideo(BaseTool):
|
||||
error_msg = str(exc)
|
||||
return ToolResult(success=False, error=error_msg, data=data)
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, error=f"ComfyUI video generation failed: {exc}")
|
||||
|
||||
width = inputs.get("width", 832 if operation == "text_to_video" else 640)
|
||||
height = inputs.get("height", 480 if operation == "text_to_video" else 640)
|
||||
num_frames = inputs.get("num_frames", 81)
|
||||
return ToolResult(
|
||||
success=False, error=f"ComfyUI video generation failed: {exc}"
|
||||
)
|
||||
|
||||
model_name = self._model_name(inputs, custom_workflow)
|
||||
partner_execution = model_family in partner_nodes and not custom_workflow
|
||||
result_data: dict[str, Any] = {
|
||||
"provider": "comfyui",
|
||||
"model": model_name,
|
||||
"prompt": inputs["prompt"],
|
||||
"operation": operation,
|
||||
"output": str(paths[0]),
|
||||
"format": "mp4",
|
||||
"workflow_provenance": provenance,
|
||||
"hosted": partner_execution,
|
||||
"network_required": partner_execution,
|
||||
}
|
||||
if partner_execution:
|
||||
result_data.update(
|
||||
{
|
||||
"duration_seconds": int(inputs.get("duration", 5)),
|
||||
"fps": 24,
|
||||
"aspect_ratio": inputs.get("aspect_ratio", "16:9"),
|
||||
"resolution": inputs.get("resolution", "720p"),
|
||||
"billing": "ComfyUI Partner Node credits",
|
||||
}
|
||||
)
|
||||
else:
|
||||
width = inputs.get("width", 832 if operation == "text_to_video" else 640)
|
||||
height = inputs.get("height", 480 if operation == "text_to_video" else 640)
|
||||
num_frames = inputs.get("num_frames", 81)
|
||||
result_data.update(
|
||||
{
|
||||
"width": width,
|
||||
"height": height,
|
||||
"num_frames": num_frames,
|
||||
"fps": 16,
|
||||
"duration_seconds": round(num_frames / 16, 2),
|
||||
}
|
||||
)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": "comfyui",
|
||||
"model": model_name,
|
||||
"prompt": inputs["prompt"],
|
||||
"operation": operation,
|
||||
"width": width,
|
||||
"height": height,
|
||||
"num_frames": num_frames,
|
||||
"fps": 16,
|
||||
"duration_seconds": round(num_frames / 16, 2),
|
||||
"output": str(paths[0]),
|
||||
"format": "mp4",
|
||||
"workflow_provenance": provenance,
|
||||
},
|
||||
data=result_data,
|
||||
artifacts=[str(p) for p in paths],
|
||||
cost_usd=0.0,
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
seed=seed,
|
||||
model=model_name,
|
||||
@@ -419,12 +601,15 @@ class ComfyUIVideo(BaseTool):
|
||||
num_frames = inputs.get("num_frames", 81)
|
||||
|
||||
workflow = ComfyUIClient.load_workflow(_WORKFLOWS / "wan22-t2v-4step.json")
|
||||
workflow = ComfyUIClient.patch_workflow(workflow, {
|
||||
"2": {"text": inputs["prompt"]},
|
||||
"11": {"width": width, "height": height, "batch_size": num_frames},
|
||||
"12": {"noise_seed": seed},
|
||||
"16": {"filename_prefix": output_path.stem},
|
||||
})
|
||||
workflow = ComfyUIClient.patch_workflow(
|
||||
workflow,
|
||||
{
|
||||
"2": {"text": inputs["prompt"]},
|
||||
"11": {"width": width, "height": height, "batch_size": num_frames},
|
||||
"12": {"noise_seed": seed},
|
||||
"16": {"filename_prefix": output_path.stem},
|
||||
},
|
||||
)
|
||||
return workflow, _T2V_OUTPUT_NODE
|
||||
|
||||
def _build_i2v(
|
||||
@@ -456,13 +641,16 @@ class ComfyUIVideo(BaseTool):
|
||||
server_name = self._client.upload_image(Path(ref_path), upload_name)
|
||||
|
||||
workflow = ComfyUIClient.load_workflow(_WORKFLOWS / "wan22-i2v-4step.json")
|
||||
workflow = ComfyUIClient.patch_workflow(workflow, {
|
||||
"93": {"text": inputs["prompt"]},
|
||||
"97": {"image": server_name},
|
||||
"98": {"width": width, "height": height, "length": num_frames},
|
||||
"86": {"noise_seed": seed},
|
||||
"108": {"filename_prefix": output_path.stem},
|
||||
})
|
||||
workflow = ComfyUIClient.patch_workflow(
|
||||
workflow,
|
||||
{
|
||||
"93": {"text": inputs["prompt"]},
|
||||
"97": {"image": server_name},
|
||||
"98": {"width": width, "height": height, "length": num_frames},
|
||||
"86": {"noise_seed": seed},
|
||||
"108": {"filename_prefix": output_path.stem},
|
||||
},
|
||||
)
|
||||
return workflow, _I2V_OUTPUT_NODE
|
||||
|
||||
@staticmethod
|
||||
@@ -474,7 +662,14 @@ class ComfyUIVideo(BaseTool):
|
||||
@staticmethod
|
||||
def _model_name(inputs: dict[str, Any], custom_workflow: bool) -> str:
|
||||
if not custom_workflow:
|
||||
return "wan2.2-14b-fp8-4step"
|
||||
return {
|
||||
"wan2.2": "wan2.2-14b-fp8-4step",
|
||||
"gemini_omni_flash": "gemini-omni-flash-preview (ComfyUI Partner Node)",
|
||||
"seedance_2.5": "Seedance 2.5 (ComfyUI Partner Node)",
|
||||
"minimax_h3_api": "MiniMax-H3 (ComfyUI Partner Node)",
|
||||
}.get(str(inputs.get("model_family", "wan2.2")), "custom-comfyui-model")
|
||||
if str(inputs.get("model_family")) == "minimax_h3_local":
|
||||
return inputs.get("workflow_model") or "MiniMax-H3 (local ComfyUI workflow)"
|
||||
return (
|
||||
inputs.get("workflow_model")
|
||||
or inputs.get("model")
|
||||
@@ -489,7 +684,23 @@ class ComfyUIVideo(BaseTool):
|
||||
output_node: str,
|
||||
operation: str,
|
||||
workflow: dict[str, Any],
|
||||
model_family: str,
|
||||
) -> dict[str, Any]:
|
||||
partner_nodes = {
|
||||
"gemini_omni_flash": "GeminiVideoOmni",
|
||||
"seedance_2.5": "ByteDance2TextToVideoNode",
|
||||
"minimax_h3_api": "MinimaxHailuo03TextToVideoNode",
|
||||
}
|
||||
if not custom_workflow and model_family in partner_nodes:
|
||||
return {
|
||||
"source": "comfyui_partner_node",
|
||||
"node_class": partner_nodes[model_family],
|
||||
"hosted": True,
|
||||
"network_required": True,
|
||||
"billing": "ComfyUI Partner Node credits",
|
||||
"workflow_hash_sha256": workflow_hash(workflow),
|
||||
"output_node": output_node,
|
||||
}
|
||||
if not custom_workflow:
|
||||
workflow_key = (
|
||||
"wan22-i2v-4step"
|
||||
@@ -507,17 +718,81 @@ class ComfyUIVideo(BaseTool):
|
||||
"model_stack": model_stack(workflow_key, inputs),
|
||||
"output_node": output_node,
|
||||
}
|
||||
local_h3 = model_family == "minimax_h3_local"
|
||||
return {
|
||||
"source": "user_supplied",
|
||||
"workflow_name": inputs.get("workflow_name"),
|
||||
"workflow_path": inputs.get("workflow_path"),
|
||||
"model": inputs.get("workflow_model") or inputs.get("model"),
|
||||
"workflow_hash_sha256": workflow_hash(workflow),
|
||||
"model_stack": model_stack(None, inputs),
|
||||
"model_stack": (
|
||||
BUNDLED_MODEL_STACKS["minimax-h3-local"]
|
||||
if local_h3 and not inputs.get("workflow_model_stack")
|
||||
else model_stack(None, inputs)
|
||||
),
|
||||
"model_stack_source": (
|
||||
"caller_supplied"
|
||||
if inputs.get("workflow_model_stack")
|
||||
else "official_minimax_h3_stack"
|
||||
if local_h3
|
||||
else "unknown_custom_workflow"
|
||||
),
|
||||
"output_node": output_node,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _build_partner_t2v(
|
||||
inputs: dict[str, Any], seed: int, output_path: Path, model_family: str
|
||||
) -> tuple[dict[str, Any], str]:
|
||||
duration = int(inputs.get("duration", 5))
|
||||
ratio = inputs.get("aspect_ratio", "16:9")
|
||||
resolution = inputs.get("resolution", "720p")
|
||||
if model_family == "gemini_omni_flash":
|
||||
node_class = "GeminiVideoOmni"
|
||||
model = {
|
||||
"model": "Omni Flash",
|
||||
"prompt": (
|
||||
f"{inputs['prompt']}\nGenerate a {duration}-second {ratio} video."
|
||||
),
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
}
|
||||
elif model_family == "seedance_2.5":
|
||||
node_class = "ByteDance2TextToVideoNode"
|
||||
model = {
|
||||
"model": "Seedance 2.5",
|
||||
"prompt": inputs["prompt"],
|
||||
"resolution": resolution if resolution in {"480p", "720p"} else "720p",
|
||||
"ratio": ratio,
|
||||
"duration": duration,
|
||||
"generate_audio": bool(inputs.get("generate_audio", True)),
|
||||
}
|
||||
else:
|
||||
node_class = "MinimaxHailuo03TextToVideoNode"
|
||||
model = {
|
||||
"model": "MiniMax H3",
|
||||
"prompt": inputs["prompt"],
|
||||
"resolution": "768P"
|
||||
if resolution not in {"768P", "2K"}
|
||||
else resolution,
|
||||
"ratio": ratio,
|
||||
"duration": duration,
|
||||
}
|
||||
workflow = {
|
||||
"1": {
|
||||
"class_type": node_class,
|
||||
"inputs": {"model": model, "seed": seed, "watermark": False},
|
||||
},
|
||||
"2": {
|
||||
"class_type": "SaveVideo",
|
||||
"inputs": {
|
||||
"video": ["1", 0],
|
||||
"filename_prefix": f"video/{output_path.stem}",
|
||||
"format": "auto",
|
||||
"codec": {"codec": "auto"},
|
||||
},
|
||||
},
|
||||
}
|
||||
if model_family == "gemini_omni_flash":
|
||||
workflow["1"]["inputs"].pop("watermark", None)
|
||||
return workflow, "2"
|
||||
|
||||
229
tools/video/gemini_omni_fal.py
Normal file
229
tools/video/gemini_omni_fal.py
Normal file
@@ -0,0 +1,229 @@
|
||||
"""Gemini Omni Flash generation and editing through fal.ai."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolStatus,
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
|
||||
class GeminiOmniFalVideo(BaseTool):
|
||||
name = "gemini_omni_fal"
|
||||
version = "0.2.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "gemini_omni"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
agent_skills = ["gemini-omni", "ai-video-gen"]
|
||||
capabilities = [
|
||||
"text_to_video",
|
||||
"image_to_video",
|
||||
"reference_to_video",
|
||||
"edit_video",
|
||||
]
|
||||
supports = {
|
||||
"text_to_video": True,
|
||||
"image_to_video": True,
|
||||
"reference_to_video": True,
|
||||
"video_to_video": True,
|
||||
"multiple_reference_images": True,
|
||||
"native_audio": True,
|
||||
}
|
||||
best_for = [
|
||||
"Gemini Omni Flash with a fal.ai key",
|
||||
"reference-image-driven 3-10 second video with synchronized audio",
|
||||
]
|
||||
not_good_for = ["Google interaction-id workflows", "offline generation"]
|
||||
fallback_tools = ["gemini_omni_video", "runway_video", "veo_video"]
|
||||
dependencies = ["env:FAL_KEY"]
|
||||
install_instructions = (
|
||||
"Set FAL_KEY (or FAL_AI_API_KEY) from https://fal.ai/dashboard/keys."
|
||||
)
|
||||
quality_score = 0.85
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["prompt"],
|
||||
"properties": {
|
||||
"prompt": {"type": "string"},
|
||||
"operation": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"text_to_video",
|
||||
"image_to_video",
|
||||
"reference_to_video",
|
||||
"edit_video",
|
||||
],
|
||||
"default": "text_to_video",
|
||||
},
|
||||
"image_url": {"type": "string"},
|
||||
"image_path": {"type": "string"},
|
||||
"reference_image_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"reference_image_paths": {"type": "array", "items": {"type": "string"}},
|
||||
"video_url": {
|
||||
"type": "string",
|
||||
"description": "Source clip for edit_video",
|
||||
},
|
||||
"aspect_ratio": {
|
||||
"type": "string",
|
||||
"enum": ["16:9", "9:16"],
|
||||
"default": "16:9",
|
||||
},
|
||||
"duration": {"type": "integer", "minimum": 3, "maximum": 10, "default": 8},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2, retryable_errors=["rate_limit", "timeout"]
|
||||
)
|
||||
idempotency_key_fields = [
|
||||
"prompt",
|
||||
"operation",
|
||||
"aspect_ratio",
|
||||
"duration",
|
||||
"reference_image_urls",
|
||||
]
|
||||
side_effects = ["writes video file to output_path", "calls fal.ai API"]
|
||||
user_visible_verification = ["Watch the clip and listen for synchronized audio"]
|
||||
|
||||
@staticmethod
|
||||
def _api_key() -> str | None:
|
||||
return os.environ.get("FAL_KEY") or os.environ.get("FAL_AI_API_KEY")
|
||||
|
||||
def get_status(self) -> ToolStatus:
|
||||
return ToolStatus.AVAILABLE if self._api_key() else ToolStatus.UNAVAILABLE
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
return round(0.13 * int(inputs.get("duration", 8)), 2)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
return 90.0
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
api_key = self._api_key()
|
||||
if not api_key:
|
||||
return ToolResult(
|
||||
success=False, error="FAL_KEY not set. " + self.install_instructions
|
||||
)
|
||||
import requests
|
||||
from tools.video._shared import probe_output, upload_image_fal
|
||||
|
||||
operation = inputs.get("operation", "text_to_video")
|
||||
urls = list(inputs.get("reference_image_urls") or [])
|
||||
for local in inputs.get("reference_image_paths") or []:
|
||||
urls.append(upload_image_fal(local))
|
||||
if inputs.get("image_url"):
|
||||
urls.insert(0, inputs["image_url"])
|
||||
elif inputs.get("image_path"):
|
||||
urls.insert(0, upload_image_fal(inputs["image_path"]))
|
||||
if operation in {"image_to_video", "reference_to_video"} and not urls:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"{operation} requires at least one reference image",
|
||||
)
|
||||
|
||||
endpoints = {
|
||||
"text_to_video": "google/gemini-omni-flash",
|
||||
"image_to_video": "google/gemini-omni-flash/image-to-video",
|
||||
"reference_to_video": "google/gemini-omni-flash/reference-to-video",
|
||||
"edit_video": "google/gemini-omni-flash/edit",
|
||||
}
|
||||
if operation not in endpoints:
|
||||
return ToolResult(
|
||||
success=False, error=f"unsupported Gemini Omni operation: {operation}"
|
||||
)
|
||||
if operation in {"text_to_video", "edit_video"} and urls:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"{operation} does not accept reference images on fal.ai",
|
||||
)
|
||||
endpoint = endpoints[operation]
|
||||
payload: dict[str, Any] = {
|
||||
"prompt": inputs["prompt"],
|
||||
"aspect_ratio": inputs.get("aspect_ratio", "16:9"),
|
||||
"duration": int(inputs.get("duration", 8)),
|
||||
}
|
||||
if operation == "image_to_video":
|
||||
payload["image_url"] = urls[0]
|
||||
elif operation == "reference_to_video":
|
||||
payload["image_urls"] = urls
|
||||
elif operation == "edit_video":
|
||||
if not inputs.get("video_url"):
|
||||
return ToolResult(success=False, error="edit_video requires video_url")
|
||||
payload = {"prompt": inputs["prompt"], "video_url": inputs["video_url"]}
|
||||
headers = {
|
||||
"Authorization": f"Key {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
started = time.time()
|
||||
try:
|
||||
submit = requests.post(
|
||||
f"https://queue.fal.run/{endpoint}",
|
||||
headers=headers,
|
||||
json=payload,
|
||||
timeout=30,
|
||||
)
|
||||
submit.raise_for_status()
|
||||
queued = submit.json()
|
||||
while True:
|
||||
time.sleep(5)
|
||||
status_response = requests.get(
|
||||
queued["status_url"], headers=headers, timeout=15
|
||||
)
|
||||
status_response.raise_for_status()
|
||||
status = status_response.json().get("status")
|
||||
if status == "COMPLETED":
|
||||
break
|
||||
if status in {"FAILED", "CANCELLED"}:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"fal.ai Gemini Omni generation {status.lower()}",
|
||||
)
|
||||
result_response = requests.get(
|
||||
queued["response_url"], headers=headers, timeout=30
|
||||
)
|
||||
result_response.raise_for_status()
|
||||
video_url = result_response.json()["video"]["url"]
|
||||
download = requests.get(video_url, timeout=120)
|
||||
download.raise_for_status()
|
||||
output_path = Path(inputs.get("output_path", "gemini_omni_fal_output.mp4"))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_bytes(download.content)
|
||||
except Exception as exc:
|
||||
return ToolResult(
|
||||
success=False, error=f"fal.ai Gemini Omni generation failed: {exc}"
|
||||
)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": "gemini_omni",
|
||||
"gateway": "fal.ai",
|
||||
"model": endpoint,
|
||||
"operation": operation,
|
||||
"output": str(output_path),
|
||||
**probe_output(output_path),
|
||||
},
|
||||
artifacts=[str(output_path)],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - started, 2),
|
||||
model=endpoint,
|
||||
)
|
||||
201
tools/video/minimax_fal_video.py
Normal file
201
tools/video/minimax_fal_video.py
Normal file
@@ -0,0 +1,201 @@
|
||||
"""MiniMax H3 (Hailuo 03) generation through fal.ai."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolStatus,
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
|
||||
class MiniMaxFalVideo(BaseTool):
|
||||
name = "minimax_fal_video"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "minimax"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
dependencies = ["env:FAL_KEY"]
|
||||
install_instructions = (
|
||||
"Set FAL_KEY (or FAL_AI_API_KEY) from https://fal.ai/dashboard/keys."
|
||||
)
|
||||
agent_skills = ["minimax-h3", "ai-video-gen"]
|
||||
capabilities = ["text_to_video", "image_to_video", "reference_to_video"]
|
||||
supports = {
|
||||
"text_to_video": True,
|
||||
"image_to_video": True,
|
||||
"reference_to_video": True,
|
||||
"multiple_reference_images": True,
|
||||
"reference_video": True,
|
||||
"reference_audio": True,
|
||||
"native_audio": True,
|
||||
}
|
||||
best_for = [
|
||||
"MiniMax H3 through an existing fal.ai account",
|
||||
"2K multimodal reference video",
|
||||
]
|
||||
not_good_for = ["offline generation"]
|
||||
fallback_tools = ["minimax_video", "runway_video", "comfyui_video"]
|
||||
quality_score = 0.95
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["prompt"],
|
||||
"properties": {
|
||||
"prompt": {"type": "string"},
|
||||
"operation": {
|
||||
"type": "string",
|
||||
"enum": ["text_to_video", "image_to_video", "reference_to_video"],
|
||||
"default": "text_to_video",
|
||||
},
|
||||
"duration": {"type": "integer", "minimum": 5, "maximum": 15, "default": 5},
|
||||
"aspect_ratio": {
|
||||
"type": "string",
|
||||
"enum": ["adaptive", "21:9", "16:9", "4:3", "1:1", "3:4", "9:16"],
|
||||
"default": "16:9",
|
||||
},
|
||||
"image_url": {"type": "string"},
|
||||
"end_image_url": {"type": "string"},
|
||||
"reference_image_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"reference_video_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"reference_audio_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2, retryable_errors=["rate_limit", "timeout"]
|
||||
)
|
||||
idempotency_key_fields = ["prompt", "operation", "duration", "aspect_ratio"]
|
||||
side_effects = ["writes video file to output_path", "calls fal.ai API"]
|
||||
user_visible_verification = ["Watch the result and verify native audio"]
|
||||
|
||||
@staticmethod
|
||||
def _api_key() -> str | None:
|
||||
return os.environ.get("FAL_KEY") or os.environ.get("FAL_AI_API_KEY")
|
||||
|
||||
def get_status(self) -> ToolStatus:
|
||||
return ToolStatus.AVAILABLE if self._api_key() else ToolStatus.UNAVAILABLE
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
return round(0.19 * int(inputs.get("duration", 5)), 2)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
return 120.0
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
api_key = self._api_key()
|
||||
if not api_key:
|
||||
return ToolResult(
|
||||
success=False, error="FAL_KEY not set. " + self.install_instructions
|
||||
)
|
||||
import requests
|
||||
from tools.video._shared import probe_output
|
||||
|
||||
operation = inputs.get("operation", "text_to_video")
|
||||
payload: dict[str, Any] = {
|
||||
"prompt": inputs["prompt"],
|
||||
"duration": int(inputs.get("duration", 5)),
|
||||
"resolution": "2K",
|
||||
}
|
||||
if operation != "image_to_video":
|
||||
payload["aspect_ratio"] = inputs.get("aspect_ratio", "16:9")
|
||||
if operation == "image_to_video":
|
||||
if not inputs.get("image_url"):
|
||||
return ToolResult(
|
||||
success=False, error="image_to_video requires image_url"
|
||||
)
|
||||
payload["image_url"] = inputs["image_url"]
|
||||
if inputs.get("end_image_url"):
|
||||
payload["end_image_url"] = inputs["end_image_url"]
|
||||
elif operation == "reference_to_video":
|
||||
for key in (
|
||||
"reference_image_urls",
|
||||
"reference_video_urls",
|
||||
"reference_audio_urls",
|
||||
):
|
||||
if inputs.get(key):
|
||||
payload[key] = inputs[key]
|
||||
if not any(
|
||||
payload.get(key)
|
||||
for key in ("reference_image_urls", "reference_video_urls")
|
||||
):
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="reference_to_video requires a reference image or video",
|
||||
)
|
||||
endpoint = f"fal-ai/minimax/hailuo-03/{operation.replace('_', '-')}"
|
||||
headers = {
|
||||
"Authorization": f"Key {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
started = time.time()
|
||||
try:
|
||||
submit = requests.post(
|
||||
f"https://queue.fal.run/{endpoint}",
|
||||
headers=headers,
|
||||
json=payload,
|
||||
timeout=30,
|
||||
)
|
||||
submit.raise_for_status()
|
||||
queued = submit.json()
|
||||
while True:
|
||||
time.sleep(5)
|
||||
status_response = requests.get(
|
||||
queued["status_url"], headers=headers, timeout=15
|
||||
)
|
||||
status_response.raise_for_status()
|
||||
status = status_response.json().get("status")
|
||||
if status == "COMPLETED":
|
||||
break
|
||||
if status in {"FAILED", "CANCELLED"}:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"fal.ai MiniMax H3 generation {status.lower()}",
|
||||
)
|
||||
result_response = requests.get(
|
||||
queued["response_url"], headers=headers, timeout=30
|
||||
)
|
||||
result_response.raise_for_status()
|
||||
video_url = result_response.json()["video"]["url"]
|
||||
download = requests.get(video_url, timeout=180)
|
||||
download.raise_for_status()
|
||||
output_path = Path(inputs.get("output_path", "minimax_h3_fal_output.mp4"))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_bytes(download.content)
|
||||
except Exception as exc:
|
||||
return ToolResult(
|
||||
success=False, error=f"fal.ai MiniMax H3 generation failed: {exc}"
|
||||
)
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": "minimax",
|
||||
"gateway": "fal.ai",
|
||||
"model": endpoint,
|
||||
"operation": operation,
|
||||
"output": str(output_path),
|
||||
**probe_output(output_path),
|
||||
},
|
||||
artifacts=[str(output_path)],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - started, 2),
|
||||
model=endpoint,
|
||||
)
|
||||
@@ -1,6 +1,8 @@
|
||||
"""MiniMax (Hailuo AI) video generation via fal.ai API.
|
||||
"""MiniMax video generation via the official first-party API.
|
||||
|
||||
Rewards prompt craft — follows camera directions well and produces high-texture footage.
|
||||
MiniMax-H3 uses the v2 content-based API. Hailuo 2.3 and earlier models keep
|
||||
using the v1 generation, query, and file-retrieval flow. Both API versions
|
||||
support the global and mainland China hosts through ``MINIMAX_REGION``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -23,10 +25,39 @@ from tools.base_tool import (
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
REGION_BASE_URLS = {
|
||||
"global": "https://api.minimax.io",
|
||||
"global_en": "https://api.minimax.io",
|
||||
"cn": "https://api.minimaxi.com",
|
||||
"cn_zh": "https://api.minimaxi.com",
|
||||
}
|
||||
DEFAULT_REGION = "global"
|
||||
|
||||
V2_MODELS = ["MiniMax-H3"]
|
||||
V1_MODELS = [
|
||||
"MiniMax-Hailuo-2.3",
|
||||
"MiniMax-Hailuo-2.3-Fast",
|
||||
"MiniMax-Hailuo-02",
|
||||
"T2V-01-Director",
|
||||
"T2V-01",
|
||||
"I2V-01-Director",
|
||||
"I2V-01-live",
|
||||
"I2V-01",
|
||||
]
|
||||
MODELS = V2_MODELS + V1_MODELS
|
||||
DEFAULT_MODEL = "MiniMax-H3"
|
||||
|
||||
V2_RATIO_VALUES = ["adaptive", "21:9", "16:9", "4:3", "1:1", "3:4", "9:16"]
|
||||
_V2_IN_PROGRESS = {"queued", "running"}
|
||||
_V2_SUCCESS = "succeeded"
|
||||
_V2_FAILURES = {"failed", "cancelled"}
|
||||
_V1_SUCCESS = "Success"
|
||||
_V1_FAILURE = "Fail"
|
||||
|
||||
|
||||
class MiniMaxVideo(BaseTool):
|
||||
name = "minimax_video"
|
||||
version = "0.1.0"
|
||||
version = "0.3.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "minimax"
|
||||
@@ -35,46 +66,149 @@ class MiniMaxVideo(BaseTool):
|
||||
determinism = Determinism.STOCHASTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
dependencies = []
|
||||
dependencies = ["env:MINIMAX_API_KEY"]
|
||||
install_instructions = (
|
||||
"Set FAL_KEY to your fal.ai API key.\n"
|
||||
" Get one at https://fal.ai/dashboard/keys"
|
||||
"Set MINIMAX_API_KEY to your MiniMax API key.\n"
|
||||
" Get one at https://platform.minimax.io/ (global) or "
|
||||
"https://platform.minimaxi.com/ (CN).\n"
|
||||
" Optionally set MINIMAX_REGION=cn to route to the mainland China host, "
|
||||
"or MINIMAX_BASE_URL to override the endpoint entirely."
|
||||
)
|
||||
agent_skills = ["ai-video-gen"]
|
||||
agent_skills = ["minimax-h3", "ai-video-gen"]
|
||||
|
||||
capabilities = ["text_to_video", "image_to_video"]
|
||||
capabilities = [
|
||||
"text_to_video",
|
||||
"image_to_video",
|
||||
"first_last_frame_to_video",
|
||||
"reference_to_video",
|
||||
]
|
||||
supports = {
|
||||
"text_to_video": True,
|
||||
"image_to_video": True,
|
||||
"first_last_frame_to_video": True,
|
||||
"reference_to_video": True,
|
||||
"reference_image": True,
|
||||
"reference_video": True,
|
||||
"reference_audio": True,
|
||||
"native_audio": True,
|
||||
"camera_direction": True,
|
||||
}
|
||||
best_for = [
|
||||
"prompt-following with camera directions (framing, motion, composition)",
|
||||
"high-texture footage with minimal hallucination",
|
||||
"cost-effective video generation",
|
||||
(
|
||||
"2K text, image, first/last-frame, and reference video generation "
|
||||
"with MiniMax-H3"
|
||||
),
|
||||
"prompt-following with camera directions and high-texture footage",
|
||||
"direct first-party API access with global and mainland China routing",
|
||||
]
|
||||
not_good_for = ["offline generation", "very long clips"]
|
||||
not_good_for = ["offline generation", "clips longer than 15 seconds"]
|
||||
fallback_tools = ["kling_video", "veo_video", "wan_video"]
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["prompt"],
|
||||
"required": [],
|
||||
"properties": {
|
||||
"prompt": {"type": "string"},
|
||||
"prompt": {"type": "string", "maxLength": 7000},
|
||||
"operation": {
|
||||
"type": "string",
|
||||
"enum": ["text_to_video", "image_to_video"],
|
||||
"enum": [
|
||||
"text_to_video",
|
||||
"image_to_video",
|
||||
"first_last_frame_to_video",
|
||||
"reference_to_video",
|
||||
],
|
||||
"default": "text_to_video",
|
||||
},
|
||||
"model_variant": {
|
||||
"model": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"video-01", "hailuo-02/pro", "hailuo-02/standard",
|
||||
"hailuo-2.3-fast/pro", "hailuo-2.3-fast/standard",
|
||||
],
|
||||
"default": "hailuo-02/pro",
|
||||
"enum": MODELS,
|
||||
"default": DEFAULT_MODEL,
|
||||
},
|
||||
"first_frame_image": {
|
||||
"type": "string",
|
||||
"description": "Public URL or data URI used as the first frame.",
|
||||
},
|
||||
"last_frame_image": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Public URL or data URI used as the last frame by MiniMax-H3."
|
||||
),
|
||||
},
|
||||
"end_image_url": {
|
||||
"type": "string",
|
||||
"description": "Alias for last_frame_image.",
|
||||
},
|
||||
"image_url": {
|
||||
"type": "string",
|
||||
"description": "Selector-compatible alias for first_frame_image.",
|
||||
},
|
||||
"reference_image_url": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"First-frame alias or a reference image for reference_to_video."
|
||||
),
|
||||
},
|
||||
"reference_image_urls": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
},
|
||||
"reference_video_url": {
|
||||
"type": "string",
|
||||
"description": "Reference video URL for MiniMax-H3.",
|
||||
},
|
||||
"reference_video_urls": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
},
|
||||
"reference_audio_urls": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
},
|
||||
"prompt_optimizer": {"type": "boolean", "default": True},
|
||||
"fast_pretreatment": {"type": "boolean"},
|
||||
"duration": {
|
||||
"type": "integer",
|
||||
"minimum": 4,
|
||||
"maximum": 15,
|
||||
"description": (
|
||||
"Clip length in seconds; MiniMax-H3 accepts integers from 4 to 15."
|
||||
),
|
||||
},
|
||||
"resolution": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"MiniMax-H3 requires 2K; v1 models accept their documented "
|
||||
"resolutions."
|
||||
),
|
||||
},
|
||||
"ratio": {"type": "string", "enum": V2_RATIO_VALUES},
|
||||
"aspect_ratio": {
|
||||
"type": "string",
|
||||
"enum": V2_RATIO_VALUES,
|
||||
"description": "Selector-compatible alias for ratio.",
|
||||
},
|
||||
"callback_url": {"type": "string"},
|
||||
"aigc_watermark": {
|
||||
"type": "boolean",
|
||||
"description": "Mainland China MiniMax-H3 watermark option.",
|
||||
},
|
||||
"poll_interval_seconds": {
|
||||
"type": "number",
|
||||
"minimum": 0.1,
|
||||
"maximum": 60,
|
||||
"default": 5,
|
||||
"description": "Seconds between task-status requests.",
|
||||
},
|
||||
"timeout_seconds": {
|
||||
"type": "number",
|
||||
"minimum": 1,
|
||||
"maximum": 3600,
|
||||
"default": 900,
|
||||
"description": (
|
||||
"Maximum time to wait for remote generation. Timeout errors "
|
||||
"include task_id so the job can be recovered manually."
|
||||
),
|
||||
},
|
||||
"image_url": {"type": "string", "description": "Reference image URL for image_to_video"},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
@@ -82,13 +216,52 @@ class MiniMaxVideo(BaseTool):
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout"])
|
||||
idempotency_key_fields = ["prompt", "model_variant", "operation"]
|
||||
side_effects = ["writes video file to output_path", "calls fal.ai API"]
|
||||
user_visible_verification = ["Watch generated clip for motion coherence and prompt adherence"]
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2, retryable_errors=["rate_limit", "timeout"]
|
||||
)
|
||||
idempotency_key_fields = [
|
||||
"prompt",
|
||||
"model",
|
||||
"operation",
|
||||
"first_frame_image",
|
||||
"last_frame_image",
|
||||
"end_image_url",
|
||||
"image_url",
|
||||
"reference_image_url",
|
||||
"reference_image_urls",
|
||||
"reference_video_url",
|
||||
"reference_video_urls",
|
||||
"reference_audio_urls",
|
||||
"prompt_optimizer",
|
||||
"fast_pretreatment",
|
||||
"duration",
|
||||
"resolution",
|
||||
"ratio",
|
||||
"aspect_ratio",
|
||||
"callback_url",
|
||||
"aigc_watermark",
|
||||
]
|
||||
side_effects = ["writes video file to output_path", "calls MiniMax video API"]
|
||||
user_visible_verification = [
|
||||
"Watch generated clip for motion coherence and prompt adherence"
|
||||
]
|
||||
|
||||
def _get_api_key(self) -> str | None:
|
||||
return os.environ.get("FAL_KEY") or os.environ.get("FAL_AI_API_KEY")
|
||||
return os.environ.get("MINIMAX_API_KEY")
|
||||
|
||||
def _region(self) -> str:
|
||||
region = os.environ.get("MINIMAX_REGION", DEFAULT_REGION).strip().lower()
|
||||
return "cn" if region in {"cn", "cn_zh"} else "global"
|
||||
|
||||
def _base_url(self) -> str:
|
||||
override = os.environ.get("MINIMAX_BASE_URL")
|
||||
if override:
|
||||
return override.rstrip("/")
|
||||
region = os.environ.get("MINIMAX_REGION", DEFAULT_REGION).strip().lower()
|
||||
return REGION_BASE_URLS.get(region, REGION_BASE_URLS[DEFAULT_REGION])
|
||||
|
||||
def _uses_cn_api(self, base_url: str) -> bool:
|
||||
return self._region() == "cn" or "api.minimaxi.com" in base_url
|
||||
|
||||
def get_status(self) -> ToolStatus:
|
||||
if self._get_api_key():
|
||||
@@ -96,105 +269,395 @@ class MiniMaxVideo(BaseTool):
|
||||
return ToolStatus.UNAVAILABLE
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
variant = inputs.get("model_variant", "hailuo-02/pro")
|
||||
if "pro" in variant:
|
||||
return 0.15
|
||||
if "fast" in variant:
|
||||
model = inputs.get("model", DEFAULT_MODEL)
|
||||
if model == "MiniMax-H3":
|
||||
duration = inputs.get("duration", 5)
|
||||
if isinstance(duration, str) and duration.isdigit():
|
||||
duration = int(duration)
|
||||
seconds = (
|
||||
duration
|
||||
if isinstance(duration, int) and not isinstance(duration, bool)
|
||||
else 5
|
||||
)
|
||||
reference_images = len(inputs.get("reference_image_urls") or [])
|
||||
additional_images = max(0, reference_images - 5)
|
||||
return round((seconds * 0.13) + (additional_images * 0.03), 2)
|
||||
if "Fast" in model:
|
||||
return 0.08
|
||||
return 0.10 # standard
|
||||
return 0.15
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
variant = inputs.get("model_variant", "hailuo-02/pro")
|
||||
if "fast" in variant:
|
||||
model = inputs.get("model", DEFAULT_MODEL)
|
||||
if model == "MiniMax-H3":
|
||||
return 90.0
|
||||
if "Fast" in model:
|
||||
return 30.0
|
||||
return 60.0
|
||||
|
||||
@staticmethod
|
||||
def _base_resp_error(payload: dict[str, Any]) -> str | None:
|
||||
"""Return an error string when a v1 base_resp signals failure."""
|
||||
base_resp = payload.get("base_resp") or {}
|
||||
status_code = base_resp.get("status_code")
|
||||
if status_code in (None, 0):
|
||||
return None
|
||||
status_msg = base_resp.get("status_msg", "")
|
||||
return f"MiniMax API error {status_code}: {status_msg}".strip()
|
||||
|
||||
@staticmethod
|
||||
def _media_content(content_type: str, url: str, role: str) -> dict[str, Any]:
|
||||
return {
|
||||
"type": content_type,
|
||||
content_type: {"url": url},
|
||||
"role": role,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _url_values(inputs: dict[str, Any], singular: str, plural: str) -> list[str]:
|
||||
values: list[str] = []
|
||||
if inputs.get(singular):
|
||||
values.append(str(inputs[singular]))
|
||||
values.extend(str(value) for value in (inputs.get(plural) or []) if value)
|
||||
return values
|
||||
|
||||
def _build_v2_payload(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
base_url: str,
|
||||
) -> tuple[dict[str, Any] | None, str | None]:
|
||||
operation = inputs.get("operation", "text_to_video")
|
||||
prompt = str(inputs.get("prompt") or "")
|
||||
if not prompt:
|
||||
return None, "MiniMax-H3 requires 'prompt'."
|
||||
if len(prompt) > 7000:
|
||||
return None, "MiniMax-H3 prompt must not exceed 7000 characters."
|
||||
|
||||
content: list[dict[str, Any]] = [{"type": "text", "text": prompt}]
|
||||
first_frame = (
|
||||
inputs.get("first_frame_image")
|
||||
or inputs.get("image_url")
|
||||
or inputs.get("reference_image_url")
|
||||
)
|
||||
last_frame = inputs.get("last_frame_image") or inputs.get("end_image_url")
|
||||
|
||||
if operation in {"image_to_video", "first_last_frame_to_video"}:
|
||||
if not first_frame:
|
||||
return None, f"{operation} requires 'first_frame_image'."
|
||||
content.append(
|
||||
self._media_content("image_url", str(first_frame), "first_frame")
|
||||
)
|
||||
if operation == "first_last_frame_to_video" and not last_frame:
|
||||
return None, "first_last_frame_to_video requires 'last_frame_image'."
|
||||
if last_frame:
|
||||
content.append(
|
||||
self._media_content("image_url", str(last_frame), "last_frame")
|
||||
)
|
||||
elif operation == "reference_to_video":
|
||||
reference_images = self._url_values(
|
||||
inputs, "reference_image_url", "reference_image_urls"
|
||||
)
|
||||
reference_videos = self._url_values(
|
||||
inputs, "reference_video_url", "reference_video_urls"
|
||||
)
|
||||
reference_audio = [
|
||||
str(value)
|
||||
for value in (inputs.get("reference_audio_urls") or [])
|
||||
if value
|
||||
]
|
||||
if not reference_images and not reference_videos and not reference_audio:
|
||||
return None, "reference_to_video requires at least one reference URL."
|
||||
if reference_audio and not (reference_images or reference_videos):
|
||||
return (
|
||||
None,
|
||||
"Reference audio requires at least one reference image or video.",
|
||||
)
|
||||
content.extend(
|
||||
self._media_content("image_url", url, "reference_image")
|
||||
for url in reference_images
|
||||
)
|
||||
content.extend(
|
||||
self._media_content("video_url", url, "reference_video")
|
||||
for url in reference_videos
|
||||
)
|
||||
content.extend(
|
||||
self._media_content("audio_url", url, "reference_audio")
|
||||
for url in reference_audio
|
||||
)
|
||||
elif operation != "text_to_video":
|
||||
return None, f"MiniMax-H3 does not support operation '{operation}'."
|
||||
|
||||
duration = inputs.get("duration", 5)
|
||||
if isinstance(duration, str) and duration.isdigit():
|
||||
duration = int(duration)
|
||||
if (
|
||||
isinstance(duration, bool)
|
||||
or not isinstance(duration, int)
|
||||
or not 4 <= duration <= 15
|
||||
):
|
||||
return None, "MiniMax-H3 duration must be an integer from 4 to 15 seconds."
|
||||
|
||||
resolution = inputs.get("resolution", "2K")
|
||||
if resolution != "2K":
|
||||
return None, "MiniMax-H3 resolution must be '2K'."
|
||||
|
||||
ratio = inputs.get("ratio") or inputs.get("aspect_ratio")
|
||||
if not ratio:
|
||||
ratio = "16:9" if operation == "text_to_video" else "adaptive"
|
||||
if ratio not in V2_RATIO_VALUES:
|
||||
return None, f"Unsupported MiniMax-H3 ratio '{ratio}'."
|
||||
if operation == "text_to_video" and ratio == "adaptive":
|
||||
return None, "MiniMax-H3 text_to_video does not support the adaptive ratio."
|
||||
|
||||
payload: dict[str, Any] = {
|
||||
"model": "MiniMax-H3",
|
||||
"content": content,
|
||||
"resolution": resolution,
|
||||
"duration": duration,
|
||||
"ratio": ratio,
|
||||
}
|
||||
if inputs.get("callback_url"):
|
||||
payload["callback_url"] = inputs["callback_url"]
|
||||
if self._uses_cn_api(base_url) and "aigc_watermark" in inputs:
|
||||
payload["aigc_watermark"] = inputs["aigc_watermark"]
|
||||
return payload, None
|
||||
|
||||
@staticmethod
|
||||
def _build_v1_payload(
|
||||
inputs: dict[str, Any], model: str
|
||||
) -> tuple[dict[str, Any] | None, str | None]:
|
||||
operation = inputs.get("operation", "text_to_video")
|
||||
if operation not in {"text_to_video", "image_to_video"}:
|
||||
return None, f"{model} does not support operation '{operation}'."
|
||||
|
||||
payload: dict[str, Any] = {"model": model}
|
||||
if operation == "image_to_video":
|
||||
first_frame = (
|
||||
inputs.get("first_frame_image")
|
||||
or inputs.get("reference_image_url")
|
||||
or inputs.get("image_url")
|
||||
)
|
||||
if not first_frame:
|
||||
return None, "image_to_video requires 'first_frame_image'."
|
||||
payload["first_frame_image"] = first_frame
|
||||
if inputs.get("prompt"):
|
||||
payload["prompt"] = inputs["prompt"]
|
||||
else:
|
||||
if not inputs.get("prompt"):
|
||||
return None, "text_to_video requires 'prompt'."
|
||||
payload["prompt"] = inputs["prompt"]
|
||||
|
||||
if "prompt_optimizer" in inputs:
|
||||
payload["prompt_optimizer"] = inputs["prompt_optimizer"]
|
||||
for field in ("fast_pretreatment", "duration", "resolution", "callback_url"):
|
||||
if inputs.get(field) is not None:
|
||||
payload[field] = inputs[field]
|
||||
return payload, None
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
api_key = self._get_api_key()
|
||||
if not api_key:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="FAL_KEY not set. " + self.install_instructions,
|
||||
error="MINIMAX_API_KEY not set. " + self.install_instructions,
|
||||
)
|
||||
|
||||
import requests
|
||||
|
||||
start = time.time()
|
||||
operation = inputs.get("operation", "text_to_video")
|
||||
variant = inputs.get("model_variant", "hailuo-02/pro")
|
||||
base_url = self._base_url()
|
||||
model = inputs.get("model", DEFAULT_MODEL)
|
||||
if model not in MODELS:
|
||||
return ToolResult(
|
||||
success=False, error=f"Unsupported MiniMax model '{model}'."
|
||||
)
|
||||
|
||||
# Build fal.ai model path
|
||||
if operation == "text_to_video":
|
||||
model_path = f"minimax/{variant}/text-to-video"
|
||||
if variant == "video-01":
|
||||
model_path = "minimax/video-01"
|
||||
if model in V2_MODELS:
|
||||
payload, validation_error = self._build_v2_payload(inputs, base_url)
|
||||
api_version = "v2"
|
||||
else:
|
||||
model_path = f"minimax/{variant}/image-to-video"
|
||||
if variant == "video-01":
|
||||
model_path = "minimax/video-01/image-to-video"
|
||||
|
||||
payload: dict[str, Any] = {"prompt": inputs["prompt"]}
|
||||
if operation == "image_to_video" and inputs.get("image_url"):
|
||||
payload["image_url"] = inputs["image_url"]
|
||||
payload, validation_error = self._build_v1_payload(inputs, model)
|
||||
api_version = "v1"
|
||||
if validation_error:
|
||||
return ToolResult(success=False, error=validation_error)
|
||||
assert payload is not None
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Key {api_key}",
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
def _redact(text: str) -> str:
|
||||
return text.replace(api_key, "***") if api_key else text
|
||||
|
||||
task_id: str | None = None
|
||||
file_id: str | None = None
|
||||
task_data: dict[str, Any] = {}
|
||||
poll_interval = max(float(inputs.get("poll_interval_seconds", 5)), 0.1)
|
||||
timeout_seconds = max(float(inputs.get("timeout_seconds", 900)), 1.0)
|
||||
poll_deadline = time.monotonic() + timeout_seconds
|
||||
|
||||
def _timeout_result() -> ToolResult:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=(
|
||||
f"MiniMax video generation timed out after {timeout_seconds:g}s; "
|
||||
f"the remote task may still complete. Resume or inspect task_id "
|
||||
f"'{task_id}'."
|
||||
),
|
||||
data={
|
||||
"provider": "minimax",
|
||||
"model": model,
|
||||
"api_version": api_version,
|
||||
"task_id": task_id,
|
||||
"status": "timed_out",
|
||||
},
|
||||
model=model,
|
||||
)
|
||||
|
||||
try:
|
||||
# Submit to queue API (async) — sync endpoint times out for video gen
|
||||
submit_resp = requests.post(
|
||||
f"https://queue.fal.run/fal-ai/{model_path}",
|
||||
f"{base_url}/{api_version}/video_generation",
|
||||
headers=headers,
|
||||
json=payload,
|
||||
timeout=30,
|
||||
)
|
||||
submit_resp.raise_for_status()
|
||||
queue_data = submit_resp.json()
|
||||
status_url = queue_data["status_url"]
|
||||
response_url = queue_data["response_url"]
|
||||
submit_data = submit_resp.json()
|
||||
if api_version == "v1":
|
||||
base_err = self._base_resp_error(submit_data)
|
||||
if base_err:
|
||||
return ToolResult(success=False, error=base_err)
|
||||
task_id = submit_data.get("task_id")
|
||||
if not task_id:
|
||||
return ToolResult(
|
||||
success=False, error="MiniMax API did not return a task_id."
|
||||
)
|
||||
|
||||
# Poll until complete
|
||||
while True:
|
||||
time.sleep(5)
|
||||
status_resp = requests.get(status_url, headers=headers, timeout=15)
|
||||
status_resp.raise_for_status()
|
||||
status = status_resp.json().get("status", "UNKNOWN")
|
||||
if status == "COMPLETED":
|
||||
break
|
||||
if status in ("FAILED", "CANCELLED"):
|
||||
download_url: str | None = None
|
||||
if api_version == "v2":
|
||||
while True:
|
||||
if time.monotonic() >= poll_deadline:
|
||||
return _timeout_result()
|
||||
time.sleep(
|
||||
min(poll_interval, max(poll_deadline - time.monotonic(), 0))
|
||||
)
|
||||
if time.monotonic() >= poll_deadline:
|
||||
return _timeout_result()
|
||||
status_resp = requests.get(
|
||||
f"{base_url}/v2/query/video_generation/{task_id}",
|
||||
headers=headers,
|
||||
timeout=15,
|
||||
)
|
||||
status_resp.raise_for_status()
|
||||
task_data = status_resp.json().get("task") or {}
|
||||
status = task_data.get("status")
|
||||
if status == _V2_SUCCESS:
|
||||
download_url = (task_data.get("content") or {}).get("url")
|
||||
break
|
||||
if status in _V2_FAILURES:
|
||||
error = task_data.get("error") or "unknown error"
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"MiniMax-H3 video generation {status}: {error}",
|
||||
)
|
||||
if status not in _V2_IN_PROGRESS:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=(
|
||||
f"MiniMax-H3 returned unknown task status '{status}'."
|
||||
),
|
||||
)
|
||||
else:
|
||||
while True:
|
||||
if time.monotonic() >= poll_deadline:
|
||||
return _timeout_result()
|
||||
time.sleep(
|
||||
min(poll_interval, max(poll_deadline - time.monotonic(), 0))
|
||||
)
|
||||
if time.monotonic() >= poll_deadline:
|
||||
return _timeout_result()
|
||||
status_resp = requests.get(
|
||||
f"{base_url}/v1/query/video_generation",
|
||||
headers=headers,
|
||||
params={"task_id": task_id},
|
||||
timeout=15,
|
||||
)
|
||||
status_resp.raise_for_status()
|
||||
status_data = status_resp.json()
|
||||
base_err = self._base_resp_error(status_data)
|
||||
if base_err:
|
||||
return ToolResult(success=False, error=base_err)
|
||||
status = status_data.get("status", "")
|
||||
if status == _V1_SUCCESS:
|
||||
file_id = status_data.get("file_id")
|
||||
break
|
||||
if status == _V1_FAILURE:
|
||||
return ToolResult(
|
||||
success=False, error="MiniMax video generation failed."
|
||||
)
|
||||
|
||||
if not file_id:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"MiniMax video generation {status.lower()}",
|
||||
error="MiniMax API did not return a file_id on success.",
|
||||
)
|
||||
file_resp = requests.get(
|
||||
f"{base_url}/v1/files/retrieve",
|
||||
headers=headers,
|
||||
params={"file_id": file_id},
|
||||
timeout=30,
|
||||
)
|
||||
file_resp.raise_for_status()
|
||||
file_data = file_resp.json()
|
||||
base_err = self._base_resp_error(file_data)
|
||||
if base_err:
|
||||
return ToolResult(success=False, error=base_err)
|
||||
download_url = (file_data.get("file") or {}).get("download_url")
|
||||
|
||||
# Fetch result
|
||||
result_resp = requests.get(response_url, headers=headers, timeout=30)
|
||||
result_resp.raise_for_status()
|
||||
data = result_resp.json()
|
||||
if not download_url:
|
||||
return ToolResult(
|
||||
success=False, error="MiniMax API did not return a download URL."
|
||||
)
|
||||
|
||||
video_url = data["video"]["url"]
|
||||
video_response = requests.get(video_url, timeout=120)
|
||||
video_response = requests.get(download_url, timeout=120)
|
||||
video_response.raise_for_status()
|
||||
|
||||
output_path = Path(inputs.get("output_path", "minimax_output.mp4"))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_bytes(video_response.content)
|
||||
except Exception as e: # noqa: BLE001 - surface a redacted error to the caller
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"MiniMax video generation failed: {_redact(str(e))}",
|
||||
data={
|
||||
"provider": "minimax",
|
||||
"model": model,
|
||||
"api_version": api_version,
|
||||
"task_id": task_id,
|
||||
}
|
||||
if task_id
|
||||
else {},
|
||||
model=model,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
return ToolResult(success=False, error=f"MiniMax video generation failed: {e}")
|
||||
data: dict[str, Any] = {
|
||||
"provider": "minimax",
|
||||
"model": model,
|
||||
"api_version": api_version,
|
||||
"region": base_url,
|
||||
"task_id": task_id,
|
||||
"prompt": inputs.get("prompt", ""),
|
||||
"output": str(output_path),
|
||||
}
|
||||
if file_id:
|
||||
data["file_id"] = file_id
|
||||
if task_data:
|
||||
data["task"] = task_data
|
||||
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": "minimax",
|
||||
"model": f"fal-ai/{model_path}",
|
||||
"prompt": inputs["prompt"],
|
||||
"output": str(output_path),
|
||||
},
|
||||
data=data,
|
||||
artifacts=[str(output_path)],
|
||||
cost_usd=self.estimate_cost(inputs),
|
||||
duration_seconds=round(time.time() - start, 2),
|
||||
model=f"fal-ai/{model_path}",
|
||||
model=model,
|
||||
)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Runway Gen-4 video generation via Runway API.
|
||||
"""Runway and third-party video generation through the Runway API.
|
||||
|
||||
Highest Elo-rated video generation model — professional quality and control.
|
||||
Supports Gen-3 Alpha Turbo, Gen-4 Turbo, and Gen-4 Aleph (highest fidelity).
|
||||
Supports Runway-native models plus the current Seedance, Gemini Omni Flash,
|
||||
and MiniMax Hailuo 3 partner models exposed by Runway's API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -24,38 +24,62 @@ from tools.base_tool import (
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
_RATIO_MAP = {
|
||||
_LEGACY_RATIO_MAP = {
|
||||
"16:9": "1280:720",
|
||||
"9:16": "720:1280",
|
||||
"1:1": "720:720",
|
||||
}
|
||||
|
||||
_SEEDANCE_25_RATIOS = {
|
||||
"480p": {
|
||||
"21:9": "992:432",
|
||||
"16:9": "854:480",
|
||||
"4:3": "752:560",
|
||||
"1:1": "640:640",
|
||||
"3:4": "560:752",
|
||||
"9:16": "480:854",
|
||||
},
|
||||
"720p": {
|
||||
"21:9": "1470:630",
|
||||
"16:9": "1280:720",
|
||||
"4:3": "1112:834",
|
||||
"1:1": "960:960",
|
||||
"3:4": "834:1112",
|
||||
"9:16": "720:1280",
|
||||
},
|
||||
}
|
||||
|
||||
_COST_PER_SECOND = {
|
||||
"gen3a_turbo": 0.05,
|
||||
"gen4_turbo": 0.05,
|
||||
"gen4_aleph": 0.15,
|
||||
# Third-party Seedance 2.0 inside Runway (Enterprise/Unlimited, non-US).
|
||||
"seedance_2.0": 0.30,
|
||||
"seedance_2.0_fast": 0.24,
|
||||
"gen4.5": 0.12,
|
||||
"seedance2": 0.36,
|
||||
"seedance2_fast": 0.29,
|
||||
"seedance2_mini": 0.16,
|
||||
"seedance2_5": 0.30,
|
||||
"gemini_omni_flash": 0.10,
|
||||
"hailuo3": 0.15,
|
||||
}
|
||||
|
||||
_RUNTIME_SECONDS = {
|
||||
"gen3a_turbo": 25.0,
|
||||
"gen4_turbo": 30.0,
|
||||
"gen4_aleph": 60.0,
|
||||
"seedance_2.0": 120.0,
|
||||
"seedance_2.0_fast": 60.0,
|
||||
"gen4.5": 60.0,
|
||||
"seedance2": 120.0,
|
||||
"seedance2_fast": 60.0,
|
||||
"seedance2_mini": 60.0,
|
||||
"seedance2_5": 150.0,
|
||||
"gemini_omni_flash": 90.0,
|
||||
"hailuo3": 120.0,
|
||||
}
|
||||
|
||||
# Single source of truth for the default model. Referenced by both the input
|
||||
# schema and every code path that reads `model`, so estimate_cost / estimate_runtime
|
||||
# / execute can never silently diverge from the advertised default again.
|
||||
_DEFAULT_MODEL = "seedance_2.0"
|
||||
_DEFAULT_MODEL = "seedance2"
|
||||
|
||||
|
||||
class RunwayVideo(BaseTool):
|
||||
name = "runway_video"
|
||||
version = "0.2.0"
|
||||
version = "0.3.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "runway"
|
||||
@@ -69,12 +93,22 @@ class RunwayVideo(BaseTool):
|
||||
"Set RUNWAY_API_KEY to your Runway API secret.\n"
|
||||
" Get one at https://dev.runwayml.com/"
|
||||
)
|
||||
agent_skills = ["seedance-2-0", "ai-video-gen"]
|
||||
agent_skills = [
|
||||
"seedance-2-0",
|
||||
"seedance-2-5",
|
||||
"gemini-omni",
|
||||
"minimax-h3",
|
||||
"ai-video-gen",
|
||||
]
|
||||
|
||||
capabilities = ["text_to_video", "image_to_video"]
|
||||
capabilities = ["text_to_video", "image_to_video", "video_to_video"]
|
||||
supports = {
|
||||
"text_to_video": True,
|
||||
"image_to_video": True,
|
||||
"video_to_video": True,
|
||||
"reference_image": True,
|
||||
"reference_video": True,
|
||||
"reference_audio": True,
|
||||
"professional_control": True,
|
||||
"native_audio": True,
|
||||
"cinematic_quality": True,
|
||||
@@ -83,14 +117,20 @@ class RunwayVideo(BaseTool):
|
||||
"multi_shot": True,
|
||||
}
|
||||
best_for = [
|
||||
"preferred premium video gen on Runway when Seedance 2.0 model is selected",
|
||||
"cinematic trailers, teasers, and high-fidelity clips with native synchronized audio (Seedance 2.0 path)",
|
||||
"director-level camera control and multi-shot editing (Seedance 2.0) or Runway Gen-4 professional control",
|
||||
"lip-sync from quoted dialogue in prompts (Seedance 2.0)",
|
||||
"Seedance 2.5 clips up to 30 seconds with large multimodal reference sets",
|
||||
"Gemini Omni Flash generation and video editing",
|
||||
"MiniMax H3 / Hailuo 3.0 generation through Runway",
|
||||
"professional video production",
|
||||
]
|
||||
not_good_for = ["budget projects", "offline generation", "very long clips"]
|
||||
fallback_tools = ["seedance_video", "seedance_replicate", "kling_video", "veo_video", "minimax_video", "wan_video"]
|
||||
fallback_tools = [
|
||||
"seedance_video",
|
||||
"seedance_replicate",
|
||||
"kling_video",
|
||||
"veo_video",
|
||||
"minimax_video",
|
||||
"wan_video",
|
||||
]
|
||||
quality_score = 0.9
|
||||
|
||||
input_schema = {
|
||||
@@ -100,39 +140,62 @@ class RunwayVideo(BaseTool):
|
||||
"prompt": {"type": "string"},
|
||||
"operation": {
|
||||
"type": "string",
|
||||
"enum": ["text_to_video", "image_to_video"],
|
||||
"enum": ["text_to_video", "image_to_video", "video_to_video"],
|
||||
"default": "text_to_video",
|
||||
},
|
||||
"model": {
|
||||
"type": "string",
|
||||
"enum": ["seedance_2.0", "seedance_2.0_fast", "gen4_turbo", "gen4_aleph", "gen3a_turbo"],
|
||||
"enum": [
|
||||
"seedance2_5",
|
||||
"gemini_omni_flash",
|
||||
"hailuo3",
|
||||
"seedance2",
|
||||
"seedance2_fast",
|
||||
"seedance2_mini",
|
||||
"gen4.5",
|
||||
"gen4_turbo",
|
||||
],
|
||||
"default": _DEFAULT_MODEL,
|
||||
"description": (
|
||||
"seedance_2.0 = preferred premium default (single-pass synced audio, multi-shot, lip-sync — "
|
||||
"Runway Unlimited/Enterprise plan, non-US only). "
|
||||
"seedance_2.0_fast = lower-cost Seedance variant. "
|
||||
"gen4_aleph = Runway's highest-fidelity native model. "
|
||||
"gen4_turbo = balanced Runway native. "
|
||||
"gen3a_turbo = cheapest Runway native."
|
||||
"Current Runway model identifiers. seedance2_5 supports 4-30s "
|
||||
"and large reference sets; gemini_omni_flash supports 3-10s "
|
||||
"generation/editing; hailuo3 is MiniMax H3 with 5-15s output."
|
||||
),
|
||||
},
|
||||
"duration": {
|
||||
"type": "integer",
|
||||
"enum": [5, 10],
|
||||
"minimum": 2,
|
||||
"maximum": 30,
|
||||
"default": 5,
|
||||
"description": "Duration in seconds",
|
||||
},
|
||||
"ratio": {
|
||||
"type": "string",
|
||||
"enum": ["16:9", "9:16", "1:1"],
|
||||
"enum": ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"],
|
||||
"default": "16:9",
|
||||
},
|
||||
"watermark": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Include Runway watermark on output",
|
||||
"image_url": {
|
||||
"type": "string",
|
||||
"description": "First-frame image URL for image_to_video",
|
||||
},
|
||||
"video_url": {
|
||||
"type": "string",
|
||||
"description": "Source video URL for video_to_video",
|
||||
},
|
||||
"reference_image_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"reference_video_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"reference_audio_urls": {"type": "array", "items": {"type": "string"}},
|
||||
"resolution": {
|
||||
"type": "string",
|
||||
"enum": ["480p", "720p", "768P", "2K"],
|
||||
"description": "Seedance 2.5: 480p/720p; Hailuo 3: 768P/2K.",
|
||||
},
|
||||
"generate_audio": {"type": "boolean", "default": True},
|
||||
"mode": {
|
||||
"type": "string",
|
||||
"enum": ["reference", "extend"],
|
||||
"default": "reference",
|
||||
},
|
||||
"image_url": {"type": "string", "description": "Reference image URL for image_to_video"},
|
||||
"output_path": {"type": "string"},
|
||||
},
|
||||
}
|
||||
@@ -140,10 +203,14 @@ class RunwayVideo(BaseTool):
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout", "THROTTLED"])
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2, retryable_errors=["rate_limit", "timeout", "THROTTLED"]
|
||||
)
|
||||
idempotency_key_fields = ["prompt", "model", "operation", "duration"]
|
||||
side_effects = ["writes video file to output_path", "calls Runway API"]
|
||||
user_visible_verification = ["Watch generated clip for visual quality and motion coherence"]
|
||||
user_visible_verification = [
|
||||
"Watch generated clip for visual quality and motion coherence"
|
||||
]
|
||||
|
||||
def get_status(self) -> ToolStatus:
|
||||
if os.environ.get("RUNWAY_API_KEY") or os.environ.get("RUNWAYML_API_SECRET"):
|
||||
@@ -156,7 +223,14 @@ class RunwayVideo(BaseTool):
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
model = inputs.get("model", _DEFAULT_MODEL)
|
||||
duration = inputs.get("duration", 5)
|
||||
return _COST_PER_SECOND.get(model, 0.05) * duration
|
||||
rate = _COST_PER_SECOND.get(model, 0.05)
|
||||
if model == "gemini_omni_flash" and inputs.get("operation") == "video_to_video":
|
||||
rate = 0.11
|
||||
if model == "seedance2_5" and inputs.get("resolution", "720p") == "480p":
|
||||
rate = 0.20
|
||||
if model == "hailuo3" and inputs.get("resolution", "2K") == "768P":
|
||||
rate = 0.10
|
||||
return rate * duration
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
model = inputs.get("model", _DEFAULT_MODEL)
|
||||
@@ -176,24 +250,149 @@ class RunwayVideo(BaseTool):
|
||||
model = inputs.get("model", _DEFAULT_MODEL)
|
||||
operation = inputs.get("operation", "text_to_video")
|
||||
ratio_friendly = inputs.get("ratio", "16:9")
|
||||
ratio_pixels = _RATIO_MAP.get(ratio_friendly, "1280:720")
|
||||
duration = inputs.get("duration", 5)
|
||||
prompt = inputs.get("prompt", "")
|
||||
image_refs = list(inputs.get("reference_image_urls") or [])
|
||||
video_refs = list(inputs.get("reference_video_urls") or [])
|
||||
audio_refs = list(inputs.get("reference_audio_urls") or [])
|
||||
|
||||
task_payload: dict[str, Any] = {
|
||||
"model": model,
|
||||
"promptText": inputs["prompt"],
|
||||
"duration": inputs.get("duration", 5),
|
||||
"ratio": ratio_pixels,
|
||||
"watermark": inputs.get("watermark", False),
|
||||
task_payload: dict[str, Any] = {"model": model, "duration": duration}
|
||||
if prompt:
|
||||
task_payload["promptText"] = prompt
|
||||
|
||||
seedance_models = {
|
||||
"seedance2",
|
||||
"seedance2_fast",
|
||||
"seedance2_mini",
|
||||
"seedance2_5",
|
||||
}
|
||||
if operation == "image_to_video" and inputs.get("image_url"):
|
||||
if model == "seedance2_5":
|
||||
resolution = inputs.get("resolution", "720p")
|
||||
if resolution not in _SEEDANCE_25_RATIOS:
|
||||
return ToolResult(
|
||||
success=False, error="seedance2_5 resolution must be 480p or 720p"
|
||||
)
|
||||
if ratio_friendly not in _SEEDANCE_25_RATIOS[resolution]:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"unsupported Seedance 2.5 ratio: {ratio_friendly}",
|
||||
)
|
||||
task_payload["ratio"] = _SEEDANCE_25_RATIOS[resolution][ratio_friendly]
|
||||
elif model == "hailuo3":
|
||||
task_payload["ratio"] = ratio_friendly
|
||||
task_payload["resolution"] = inputs.get("resolution", "2K")
|
||||
elif model == "gemini_omni_flash":
|
||||
if ratio_friendly not in {"16:9", "9:16"}:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="gemini_omni_flash supports only 16:9 or 9:16",
|
||||
)
|
||||
task_payload["ratio"] = _LEGACY_RATIO_MAP[ratio_friendly]
|
||||
else:
|
||||
task_payload["ratio"] = _LEGACY_RATIO_MAP.get(ratio_friendly, "1280:720")
|
||||
|
||||
if model == "gemini_omni_flash" and not 3 <= task_payload["duration"] <= 10:
|
||||
return ToolResult(
|
||||
success=False, error="gemini_omni_flash duration must be 3-10 seconds"
|
||||
)
|
||||
if model == "seedance2_5" and not 4 <= task_payload["duration"] <= 30:
|
||||
return ToolResult(
|
||||
success=False, error="seedance2_5 duration must be 4-30 seconds"
|
||||
)
|
||||
if model == "hailuo3" and not 5 <= task_payload["duration"] <= 15:
|
||||
return ToolResult(
|
||||
success=False, error="hailuo3 duration must be 5-15 seconds"
|
||||
)
|
||||
|
||||
if model in seedance_models or model == "hailuo3":
|
||||
if model in seedance_models and "generate_audio" in inputs:
|
||||
task_payload["audio"] = inputs["generate_audio"]
|
||||
if operation != "image_to_video":
|
||||
if image_refs:
|
||||
task_payload["references"] = [{"uri": url} for url in image_refs]
|
||||
if video_refs:
|
||||
task_payload["referenceVideos"] = [
|
||||
{"type": "video", "uri": url} for url in video_refs
|
||||
]
|
||||
elif image_refs or video_refs:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=(
|
||||
f"{model} image_to_video accepts a first frame and optional "
|
||||
"audio references, not additional image/video references"
|
||||
),
|
||||
)
|
||||
if audio_refs:
|
||||
task_payload["referenceAudio"] = [
|
||||
{"type": "audio", "uri": url} for url in audio_refs
|
||||
]
|
||||
elif model == "gemini_omni_flash" and operation != "video_to_video":
|
||||
if image_refs or video_refs or audio_refs:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=(
|
||||
"gemini_omni_flash text/image generation through Runway does "
|
||||
"not accept extra references"
|
||||
),
|
||||
)
|
||||
|
||||
if (
|
||||
model in {"gemini_omni_flash", "hailuo3"}
|
||||
and inputs.get("generate_audio") is False
|
||||
):
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"Runway's {model} schema does not expose an audio-disable parameter",
|
||||
)
|
||||
|
||||
if operation == "image_to_video":
|
||||
if not inputs.get("image_url"):
|
||||
return ToolResult(
|
||||
success=False, error="image_to_video requires image_url"
|
||||
)
|
||||
task_payload["promptImage"] = inputs["image_url"]
|
||||
if operation == "video_to_video":
|
||||
if not inputs.get("video_url"):
|
||||
return ToolResult(
|
||||
success=False, error="video_to_video requires video_url"
|
||||
)
|
||||
source_field = "videoUri" if model == "gemini_omni_flash" else "promptVideo"
|
||||
task_payload[source_field] = inputs["video_url"]
|
||||
if model == "gemini_omni_flash":
|
||||
task_payload.pop("duration", None)
|
||||
task_payload.pop("ratio", None)
|
||||
if video_refs or audio_refs:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="gemini_omni_flash video editing accepts image references only",
|
||||
)
|
||||
if image_refs:
|
||||
task_payload["references"] = [{"uri": url} for url in image_refs]
|
||||
if model == "seedance2_5":
|
||||
task_payload["mode"] = inputs.get("mode", "reference")
|
||||
if task_payload["mode"] == "extend":
|
||||
task_payload.pop("ratio", None)
|
||||
|
||||
if operation == "text_to_video" and model == "seedance2_5":
|
||||
if not prompt and not (image_refs or video_refs or audio_refs):
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="seedance2_5 text_to_video requires prompt or references",
|
||||
)
|
||||
elif not prompt and model != "gen4_turbo":
|
||||
return ToolResult(success=False, error=f"{model} requires prompt")
|
||||
|
||||
if model == "gen4_turbo" and operation != "image_to_video":
|
||||
return ToolResult(
|
||||
success=False, error="gen4_turbo supports image_to_video only"
|
||||
)
|
||||
if model in {"gen4.5", "gen4_turbo"} and operation == "video_to_video":
|
||||
return ToolResult(
|
||||
success=False, error=f"{model} does not support video_to_video"
|
||||
)
|
||||
|
||||
# Choose endpoint based on operation
|
||||
endpoint = (
|
||||
"https://api.dev.runwayml.com/v1/image_to_video"
|
||||
if operation == "image_to_video"
|
||||
else "https://api.dev.runwayml.com/v1/text_to_video"
|
||||
)
|
||||
endpoint = f"https://api.dev.runwayml.com/v1/{operation}"
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
@@ -237,7 +436,9 @@ class RunwayVideo(BaseTool):
|
||||
# PENDING, THROTTLED, RUNNING — keep polling
|
||||
|
||||
if not video_url:
|
||||
return ToolResult(success=False, error="Runway generation timed out after 5 minutes.")
|
||||
return ToolResult(
|
||||
success=False, error="Runway generation timed out after 5 minutes."
|
||||
)
|
||||
|
||||
# Download video — URLs are ephemeral (expire in 24-48h)
|
||||
video_response = requests.get(video_url, timeout=120)
|
||||
@@ -248,7 +449,9 @@ class RunwayVideo(BaseTool):
|
||||
output_path.write_bytes(video_response.content)
|
||||
|
||||
except Exception as e:
|
||||
return ToolResult(success=False, error=f"Runway video generation failed: {e}")
|
||||
return ToolResult(
|
||||
success=False, error=f"Runway video generation failed: {e}"
|
||||
)
|
||||
|
||||
from tools.video._shared import probe_output
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Direct Volcengine Ark adapter for the Seedance 2.0 model family.
|
||||
"""Direct Volcengine Ark adapter for the Seedance 2.0 and 2.5 model families.
|
||||
|
||||
The Ark API is asynchronous: create a task, poll its status, then download the
|
||||
24-hour result URL immediately. This provider is intentionally independent
|
||||
@@ -40,7 +40,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
"""Generate Seedance 2.0 video through Volcengine Ark's official REST API."""
|
||||
|
||||
name = "seedance_ark"
|
||||
version = "0.1.0"
|
||||
version = "0.2.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "ark"
|
||||
@@ -51,14 +51,13 @@ class SeedanceArkVideo(BaseTool):
|
||||
|
||||
BASE_URL = "https://ark.cn-beijing.volces.com/api/v3"
|
||||
MODEL_IDS = {
|
||||
"2.5": "doubao-seedance-2-5-260628",
|
||||
"standard": "doubao-seedance-2-0-260128",
|
||||
"fast": "doubao-seedance-2-0-fast-260128",
|
||||
"mini": "doubao-seedance-2-0-mini-260615",
|
||||
}
|
||||
TASK_ID_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$")
|
||||
TERMINAL_STATUSES = frozenset(
|
||||
{"succeeded", "failed", "cancelled", "expired"}
|
||||
)
|
||||
TERMINAL_STATUSES = frozenset({"succeeded", "failed", "cancelled", "expired"})
|
||||
IMAGE_SUFFIX_TO_MIME = {
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
@@ -142,7 +141,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
"Set ARK_API_KEY to the API Key body from Volcengine Ark (without "
|
||||
"the 'Bearer ' prefix). Optional: ARK_SEEDANCE_MODEL and ARK_BASE_URL."
|
||||
)
|
||||
agent_skills = ["seedance-2-0", "ai-video-gen"]
|
||||
agent_skills = ["seedance-2-0", "seedance-2-5", "ai-video-gen"]
|
||||
|
||||
capabilities = [
|
||||
"text_to_video",
|
||||
@@ -200,7 +199,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
},
|
||||
"model_variant": {
|
||||
"type": "string",
|
||||
"enum": ["standard", "fast", "mini"],
|
||||
"enum": ["2.5", "standard", "fast", "mini"],
|
||||
"default": "standard",
|
||||
},
|
||||
"model": {
|
||||
@@ -369,14 +368,14 @@ class SeedanceArkVideo(BaseTool):
|
||||
|
||||
def estimate_token_usage(self, inputs: dict[str, Any]) -> int:
|
||||
"""Estimate billable completion tokens using Ark's published formula."""
|
||||
duration = self._normalize_duration(inputs.get("duration", 5))
|
||||
output_seconds = 15 if duration == -1 else duration
|
||||
_, variant = self._resolve_model(inputs)
|
||||
max_duration = 30 if variant == "2.5" else 15
|
||||
duration = self._normalize_duration(inputs.get("duration", 5), max_duration)
|
||||
output_seconds = max_duration if duration == -1 else duration
|
||||
video_refs = list(inputs.get("reference_video_urls") or [])
|
||||
if inputs.get("reference_video_url"):
|
||||
video_refs.append(inputs["reference_video_url"])
|
||||
video_durations = list(
|
||||
inputs.get("reference_video_durations") or []
|
||||
)
|
||||
video_durations = list(inputs.get("reference_video_durations") or [])
|
||||
# A video reference changes both the token formula and the price tier.
|
||||
# When duration metadata is absent, use the official 15-second combined
|
||||
# maximum as a conservative preflight upper bound instead of reporting
|
||||
@@ -391,17 +390,13 @@ class SeedanceArkVideo(BaseTool):
|
||||
ratio = "16:9"
|
||||
width, height = self.OUTPUT_DIMENSIONS[resolution][ratio]
|
||||
return round(
|
||||
(input_video_seconds + output_seconds)
|
||||
* width
|
||||
* height
|
||||
* 24
|
||||
/ 1024
|
||||
(input_video_seconds + output_seconds) * width * height * 24 / 1024
|
||||
)
|
||||
|
||||
def estimate_cost_cny(self, inputs: dict[str, Any]) -> float:
|
||||
model, variant = self._resolve_model(inputs)
|
||||
del model
|
||||
if variant is None:
|
||||
if variant is None or variant == "2.5":
|
||||
rate = self._get_custom_price(inputs, required=True)
|
||||
return round(
|
||||
self.estimate_token_usage(inputs) * rate / 1_000_000,
|
||||
@@ -409,8 +404,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
)
|
||||
resolution = str(inputs.get("resolution", "720p")).lower()
|
||||
with_video = bool(
|
||||
inputs.get("reference_video_url")
|
||||
or inputs.get("reference_video_urls")
|
||||
inputs.get("reference_video_url") or inputs.get("reference_video_urls")
|
||||
)
|
||||
condition = "with_video" if with_video else "without_video"
|
||||
try:
|
||||
@@ -426,9 +420,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
return round(self.estimate_cost_cny(inputs) / cny_per_usd, 4)
|
||||
|
||||
@staticmethod
|
||||
def _get_custom_price(
|
||||
inputs: dict[str, Any], *, required: bool
|
||||
) -> float:
|
||||
def _get_custom_price(inputs: dict[str, Any], *, required: bool) -> float:
|
||||
try:
|
||||
raw = inputs["custom_price_cny_per_million_tokens"]
|
||||
except KeyError as exc:
|
||||
@@ -461,9 +453,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
"ARK_CNY_PER_USD must be a finite number greater than 0"
|
||||
) from exc
|
||||
if not math.isfinite(value) or value <= 0:
|
||||
raise ValueError(
|
||||
"ARK_CNY_PER_USD must be a finite number greater than 0"
|
||||
)
|
||||
raise ValueError("ARK_CNY_PER_USD must be a finite number greater than 0")
|
||||
return value
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
@@ -482,12 +472,10 @@ class SeedanceArkVideo(BaseTool):
|
||||
"api_contract": {
|
||||
"create": f"POST {self.BASE_URL}/contents/generations/tasks",
|
||||
"query": (
|
||||
f"GET {self.BASE_URL}/contents/generations/tasks/"
|
||||
"{task_id}"
|
||||
f"GET {self.BASE_URL}/contents/generations/tasks/{{task_id}}"
|
||||
),
|
||||
"cancel": (
|
||||
f"DELETE {self.BASE_URL}/contents/generations/tasks/"
|
||||
"{task_id}"
|
||||
f"DELETE {self.BASE_URL}/contents/generations/tasks/{{task_id}}"
|
||||
),
|
||||
},
|
||||
}
|
||||
@@ -500,17 +488,9 @@ class SeedanceArkVideo(BaseTool):
|
||||
"the 'Bearer ' prefix"
|
||||
)
|
||||
result["api_contract"] = {
|
||||
"create": (
|
||||
f"POST {base_url}/contents/generations/tasks"
|
||||
),
|
||||
"query": (
|
||||
f"GET {base_url}/contents/generations/tasks/"
|
||||
"{task_id}"
|
||||
),
|
||||
"cancel": (
|
||||
f"DELETE {base_url}/contents/generations/tasks/"
|
||||
"{task_id}"
|
||||
),
|
||||
"create": (f"POST {base_url}/contents/generations/tasks"),
|
||||
"query": (f"GET {base_url}/contents/generations/tasks/{{task_id}}"),
|
||||
"cancel": (f"DELETE {base_url}/contents/generations/tasks/{{task_id}}"),
|
||||
}
|
||||
if action in {"query", "cancel"}:
|
||||
self._validate_task_id(inputs.get("task_id"))
|
||||
@@ -519,15 +499,11 @@ class SeedanceArkVideo(BaseTool):
|
||||
result.update(
|
||||
{
|
||||
"model": payload["model"],
|
||||
"operation": inputs.get(
|
||||
"operation", "text_to_video"
|
||||
),
|
||||
"operation": inputs.get("operation", "text_to_video"),
|
||||
"resolution": payload.get("resolution", "720p"),
|
||||
"ratio": payload.get("ratio", "16:9"),
|
||||
"duration": payload.get("duration", 5),
|
||||
"generate_audio": payload.get(
|
||||
"generate_audio", True
|
||||
),
|
||||
"generate_audio": payload.get("generate_audio", True),
|
||||
"media_counts": self._media_counts(payload["content"]),
|
||||
"estimated_tokens": self.estimate_token_usage(inputs),
|
||||
"estimated_cost_cny": self.estimate_cost_cny(inputs),
|
||||
@@ -638,9 +614,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
if status != "succeeded":
|
||||
detail = self._task_error(task)
|
||||
safe_detail = (
|
||||
self._safe_error(RuntimeError(detail), api_key)
|
||||
if detail
|
||||
else ""
|
||||
self._safe_error(RuntimeError(detail), api_key) if detail else ""
|
||||
)
|
||||
return ToolResult(
|
||||
success=False,
|
||||
@@ -656,12 +630,8 @@ class SeedanceArkVideo(BaseTool):
|
||||
content = task.get("content") or {}
|
||||
video_url = content.get("video_url")
|
||||
if not video_url:
|
||||
raise RuntimeError(
|
||||
"Ark task succeeded without content.video_url"
|
||||
)
|
||||
output_path = Path(
|
||||
inputs.get("output_path", "seedance_ark_output.mp4")
|
||||
)
|
||||
raise RuntimeError("Ark task succeeded without content.video_url")
|
||||
output_path = Path(inputs.get("output_path", "seedance_ark_output.mp4"))
|
||||
self._download_video(str(video_url), output_path)
|
||||
|
||||
from tools.video._shared import probe_output
|
||||
@@ -676,28 +646,18 @@ class SeedanceArkVideo(BaseTool):
|
||||
"status": status,
|
||||
"model": task.get("model") or model,
|
||||
"prompt": inputs.get("prompt"),
|
||||
"operation": inputs.get(
|
||||
"operation", "text_to_video"
|
||||
),
|
||||
"operation": inputs.get("operation", "text_to_video"),
|
||||
"video_url": video_url,
|
||||
"last_frame_url": content.get("last_frame_url"),
|
||||
"output": str(output_path),
|
||||
"output_path": str(output_path),
|
||||
"format": "mp4",
|
||||
"resolution": task.get(
|
||||
"resolution", payload.get("resolution")
|
||||
),
|
||||
"aspect_ratio": task.get(
|
||||
"ratio", payload.get("ratio")
|
||||
),
|
||||
"duration": task.get(
|
||||
"duration", payload.get("duration")
|
||||
),
|
||||
"resolution": task.get("resolution", payload.get("resolution")),
|
||||
"aspect_ratio": task.get("ratio", payload.get("ratio")),
|
||||
"duration": task.get("duration", payload.get("duration")),
|
||||
"generate_audio": task.get("generate_audio"),
|
||||
"usage": task.get("usage") or {},
|
||||
"estimated_cost_cny": self._cost_from_task_cny(
|
||||
task, inputs
|
||||
),
|
||||
"estimated_cost_cny": self._cost_from_task_cny(task, inputs),
|
||||
**probed,
|
||||
},
|
||||
artifacts=[str(output_path)],
|
||||
@@ -719,8 +679,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
success=False,
|
||||
data=error_data,
|
||||
error=(
|
||||
"Ark Seedance request failed: "
|
||||
f"{self._safe_error(exc, api_key)}"
|
||||
f"Ark Seedance request failed: {self._safe_error(exc, api_key)}"
|
||||
),
|
||||
duration_seconds=round(time.time() - started, 2),
|
||||
)
|
||||
@@ -733,23 +692,18 @@ class SeedanceArkVideo(BaseTool):
|
||||
"reference_to_video",
|
||||
}:
|
||||
raise ValueError(
|
||||
"operation must be text_to_video, image_to_video, or "
|
||||
"reference_to_video"
|
||||
"operation must be text_to_video, image_to_video, or reference_to_video"
|
||||
)
|
||||
|
||||
model, variant = self._resolve_model(inputs)
|
||||
resolution = str(inputs.get("resolution", "720p")).lower()
|
||||
if resolution not in self.OUTPUT_DIMENSIONS:
|
||||
raise ValueError(
|
||||
"resolution must be 480p, 720p, 1080p, or 4k"
|
||||
)
|
||||
if variant in {"fast", "mini"} and resolution not in {
|
||||
raise ValueError("resolution must be 480p, 720p, 1080p, or 4k")
|
||||
if variant in {"2.5", "fast", "mini"} and resolution not in {
|
||||
"480p",
|
||||
"720p",
|
||||
}:
|
||||
raise ValueError(
|
||||
f"{variant} supports only 480p or 720p resolution"
|
||||
)
|
||||
raise ValueError(f"{variant} supports only 480p or 720p resolution")
|
||||
|
||||
ratio = str(inputs.get("aspect_ratio", "16:9"))
|
||||
valid_ratios = {
|
||||
@@ -763,11 +717,11 @@ class SeedanceArkVideo(BaseTool):
|
||||
}
|
||||
if ratio not in valid_ratios:
|
||||
raise ValueError(
|
||||
"aspect_ratio must be adaptive, 21:9, 16:9, 4:3, "
|
||||
"1:1, 3:4, or 9:16"
|
||||
"aspect_ratio must be adaptive, 21:9, 16:9, 4:3, 1:1, 3:4, or 9:16"
|
||||
)
|
||||
|
||||
duration = self._normalize_duration(inputs.get("duration", 5))
|
||||
max_duration = 30 if variant == "2.5" else 15
|
||||
duration = self._normalize_duration(inputs.get("duration", 5), max_duration)
|
||||
prompt = str(inputs.get("prompt") or "").strip()
|
||||
content: list[dict[str, Any]] = []
|
||||
if prompt:
|
||||
@@ -790,12 +744,8 @@ class SeedanceArkVideo(BaseTool):
|
||||
elif operation == "image_to_video":
|
||||
first_refs = self._single_image_refs(inputs)
|
||||
if len(first_refs) != 1:
|
||||
raise ValueError(
|
||||
"image_to_video requires exactly one reference image"
|
||||
)
|
||||
content.append(
|
||||
self._image_content(first_refs[0], role="first_frame")
|
||||
)
|
||||
raise ValueError("image_to_video requires exactly one reference image")
|
||||
content.append(self._image_content(first_refs[0], role="first_frame"))
|
||||
end_refs = [
|
||||
value
|
||||
for value in (
|
||||
@@ -805,13 +755,9 @@ class SeedanceArkVideo(BaseTool):
|
||||
if value
|
||||
]
|
||||
if len(end_refs) > 1:
|
||||
raise ValueError(
|
||||
"provide only one of end_image_url/end_image_path"
|
||||
)
|
||||
raise ValueError("provide only one of end_image_url/end_image_path")
|
||||
if end_refs:
|
||||
content.append(
|
||||
self._image_content(end_refs[0], role="last_frame")
|
||||
)
|
||||
content.append(self._image_content(end_refs[0], role="last_frame"))
|
||||
else:
|
||||
image_refs = list(inputs.get("reference_image_urls") or [])
|
||||
image_refs.extend(inputs.get("reference_image_paths") or [])
|
||||
@@ -819,22 +765,24 @@ class SeedanceArkVideo(BaseTool):
|
||||
image_refs.append(inputs["reference_image_url"])
|
||||
if inputs.get("reference_image_path"):
|
||||
image_refs.append(inputs["reference_image_path"])
|
||||
if len(image_refs) > 9:
|
||||
max_images = 30 if variant == "2.5" else 9
|
||||
max_videos = 10 if variant == "2.5" else 3
|
||||
max_audios = 10 if variant == "2.5" else 3
|
||||
max_reference_seconds = 30 if variant == "2.5" else 15
|
||||
if len(image_refs) > max_images:
|
||||
raise ValueError(
|
||||
"reference_to_video accepts at most 9 reference images"
|
||||
f"reference_to_video accepts at most {max_images} reference images"
|
||||
)
|
||||
|
||||
video_refs = list(inputs.get("reference_video_urls") or [])
|
||||
if inputs.get("reference_video_url"):
|
||||
video_refs.append(inputs["reference_video_url"])
|
||||
if len(video_refs) > 3:
|
||||
if len(video_refs) > max_videos:
|
||||
raise ValueError(
|
||||
"reference_to_video accepts at most 3 reference videos"
|
||||
f"reference_to_video accepts at most {max_videos} reference videos"
|
||||
)
|
||||
self._validate_remote_refs(video_refs, "reference video")
|
||||
video_durations = list(
|
||||
inputs.get("reference_video_durations") or []
|
||||
)
|
||||
video_durations = list(inputs.get("reference_video_durations") or [])
|
||||
if video_durations:
|
||||
if len(video_durations) != len(video_refs):
|
||||
raise ValueError(
|
||||
@@ -842,16 +790,19 @@ class SeedanceArkVideo(BaseTool):
|
||||
"reference videos"
|
||||
)
|
||||
if any(
|
||||
float(value) < 2 or float(value) > 15
|
||||
float(value) < 2 or float(value) > max_reference_seconds
|
||||
for value in video_durations
|
||||
):
|
||||
raise ValueError(
|
||||
"each reference video duration must be 2 to 15 seconds"
|
||||
f"each reference video duration must be 2 to {max_reference_seconds} seconds"
|
||||
)
|
||||
if sum(float(value) for value in video_durations) > 15:
|
||||
if (
|
||||
sum(float(value) for value in video_durations)
|
||||
> max_reference_seconds
|
||||
):
|
||||
raise ValueError(
|
||||
"all reference videos together must be at most "
|
||||
"15 seconds"
|
||||
f"{max_reference_seconds} seconds"
|
||||
)
|
||||
|
||||
audio_refs = list(inputs.get("reference_audio_urls") or [])
|
||||
@@ -860,14 +811,12 @@ class SeedanceArkVideo(BaseTool):
|
||||
audio_refs.append(inputs["reference_audio_url"])
|
||||
if inputs.get("reference_audio_path"):
|
||||
audio_refs.append(inputs["reference_audio_path"])
|
||||
if len(audio_refs) > 3:
|
||||
if len(audio_refs) > max_audios:
|
||||
raise ValueError(
|
||||
"reference_to_video accepts at most 3 reference audio "
|
||||
"clips"
|
||||
f"clips (maximum {max_audios})"
|
||||
)
|
||||
audio_durations = list(
|
||||
inputs.get("reference_audio_durations") or []
|
||||
)
|
||||
audio_durations = list(inputs.get("reference_audio_durations") or [])
|
||||
if audio_durations:
|
||||
if len(audio_durations) != len(audio_refs):
|
||||
raise ValueError(
|
||||
@@ -875,34 +824,38 @@ class SeedanceArkVideo(BaseTool):
|
||||
"reference audio clips"
|
||||
)
|
||||
if any(
|
||||
float(value) < 2 or float(value) > 15
|
||||
float(value) < 2 or float(value) > max_reference_seconds
|
||||
for value in audio_durations
|
||||
):
|
||||
raise ValueError(
|
||||
"each reference audio duration must be 2 to 15 seconds"
|
||||
f"each reference audio duration must be 2 to {max_reference_seconds} seconds"
|
||||
)
|
||||
if sum(float(value) for value in audio_durations) > 15:
|
||||
if (
|
||||
sum(float(value) for value in audio_durations)
|
||||
> max_reference_seconds
|
||||
):
|
||||
raise ValueError(
|
||||
"all reference audio clips together must be at most "
|
||||
"15 seconds"
|
||||
f"{max_reference_seconds} seconds"
|
||||
)
|
||||
local_audio_durations = [
|
||||
duration
|
||||
for ref in audio_refs
|
||||
if (
|
||||
duration := self._local_or_data_audio_duration(str(ref))
|
||||
duration := self._local_or_data_audio_duration(
|
||||
str(ref), max_seconds=max_reference_seconds
|
||||
)
|
||||
)
|
||||
is not None
|
||||
]
|
||||
if sum(local_audio_durations) > 15:
|
||||
if sum(local_audio_durations) > max_reference_seconds:
|
||||
raise ValueError(
|
||||
"all local reference audio clips together must be at "
|
||||
"most 15 seconds"
|
||||
f"most {max_reference_seconds} seconds"
|
||||
)
|
||||
if audio_refs and not (image_refs or video_refs):
|
||||
raise ValueError(
|
||||
"reference audio requires at least one reference image "
|
||||
"or video"
|
||||
"reference audio requires at least one reference image or video"
|
||||
)
|
||||
if not (image_refs or video_refs):
|
||||
raise ValueError(
|
||||
@@ -910,8 +863,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
)
|
||||
|
||||
content.extend(
|
||||
self._image_content(ref, role="reference_image")
|
||||
for ref in image_refs
|
||||
self._image_content(ref, role="reference_image") for ref in image_refs
|
||||
)
|
||||
content.extend(
|
||||
{
|
||||
@@ -922,14 +874,11 @@ class SeedanceArkVideo(BaseTool):
|
||||
for ref in video_refs
|
||||
)
|
||||
content.extend(
|
||||
self._audio_content(ref, role="reference_audio")
|
||||
for ref in audio_refs
|
||||
self._audio_content(ref, role="reference_audio") for ref in audio_refs
|
||||
)
|
||||
|
||||
if inputs.get("web_search") and len(content) != 1:
|
||||
raise ValueError(
|
||||
"web_search is supported only for pure text input"
|
||||
)
|
||||
raise ValueError("web_search is supported only for pure text input")
|
||||
|
||||
payload: dict[str, Any] = {
|
||||
"model": model,
|
||||
@@ -939,9 +888,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
"resolution": resolution,
|
||||
"generate_audio": bool(inputs.get("generate_audio", True)),
|
||||
"watermark": bool(inputs.get("watermark", False)),
|
||||
"return_last_frame": bool(
|
||||
inputs.get("return_last_frame", False)
|
||||
),
|
||||
"return_last_frame": bool(inputs.get("return_last_frame", False)),
|
||||
}
|
||||
optional = (
|
||||
"callback_url",
|
||||
@@ -959,12 +906,10 @@ class SeedanceArkVideo(BaseTool):
|
||||
self._validate_request_size(payload)
|
||||
return payload
|
||||
|
||||
def _resolve_model(
|
||||
self, inputs: dict[str, Any]
|
||||
) -> tuple[str, str | None]:
|
||||
def _resolve_model(self, inputs: dict[str, Any]) -> tuple[str, str | None]:
|
||||
variant = str(inputs.get("model_variant", "standard")).lower()
|
||||
if variant not in self.MODEL_IDS:
|
||||
raise ValueError("model_variant must be standard, fast, or mini")
|
||||
raise ValueError("model_variant must be 2.5, standard, fast, or mini")
|
||||
model = str(
|
||||
inputs.get("model")
|
||||
or os.environ.get("ARK_SEEDANCE_MODEL")
|
||||
@@ -981,23 +926,25 @@ class SeedanceArkVideo(BaseTool):
|
||||
return model, None
|
||||
|
||||
@staticmethod
|
||||
def _normalize_duration(value: Any) -> int:
|
||||
def _normalize_duration(value: Any, max_seconds: int = 15) -> int:
|
||||
if value == "auto":
|
||||
return -1
|
||||
if isinstance(value, bool):
|
||||
raise ValueError("duration must be an integer from 4 to 15 or -1")
|
||||
raise ValueError(
|
||||
f"duration must be an integer from 4 to {max_seconds} or -1"
|
||||
)
|
||||
try:
|
||||
duration = int(value)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError(
|
||||
"duration must be an integer from 4 to 15 or -1"
|
||||
f"duration must be an integer from 4 to {max_seconds} or -1"
|
||||
) from exc
|
||||
if str(value).strip() not in {str(duration), "auto"}:
|
||||
raise ValueError(
|
||||
"duration must be an integer from 4 to 15 or -1"
|
||||
f"duration must be an integer from 4 to {max_seconds} or -1"
|
||||
)
|
||||
if duration != -1 and not 4 <= duration <= 15:
|
||||
raise ValueError("duration must be between 4 and 15 or -1")
|
||||
if duration != -1 and not 4 <= duration <= max_seconds:
|
||||
raise ValueError(f"duration must be between 4 and {max_seconds} or -1")
|
||||
return duration
|
||||
|
||||
@staticmethod
|
||||
@@ -1082,8 +1029,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
size = path.stat().st_size
|
||||
if size >= max_bytes:
|
||||
raise ValueError(
|
||||
f"local {label} must be smaller than "
|
||||
f"{max_bytes // (1024 * 1024)} MB"
|
||||
f"local {label} must be smaller than {max_bytes // (1024 * 1024)} MB"
|
||||
)
|
||||
suffix = path.suffix.lower()
|
||||
mime = suffix_to_mime.get(suffix)
|
||||
@@ -1114,9 +1060,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
width, height = image.size
|
||||
image.verify()
|
||||
except Exception as exc:
|
||||
raise ValueError(
|
||||
f"image is unreadable or corrupt: {source}"
|
||||
) from exc
|
||||
raise ValueError(f"image is unreadable or corrupt: {source}") from exc
|
||||
if not (300 <= width <= 6000 and 300 <= height <= 6000):
|
||||
raise ValueError(
|
||||
"local image width and height must each be 300 to 6000 pixels"
|
||||
@@ -1179,7 +1123,9 @@ class SeedanceArkVideo(BaseTool):
|
||||
)
|
||||
return mime, decoded
|
||||
|
||||
def _local_or_data_audio_duration(self, value: str) -> float | None:
|
||||
def _local_or_data_audio_duration(
|
||||
self, value: str, *, max_seconds: int = 15
|
||||
) -> float | None:
|
||||
if value.startswith("data:"):
|
||||
mime, decoded = self._decode_data_uri(
|
||||
value,
|
||||
@@ -1187,12 +1133,16 @@ class SeedanceArkVideo(BaseTool):
|
||||
max_bytes=self.MAX_AUDIO_BYTES,
|
||||
label="audio",
|
||||
)
|
||||
return self._probe_audio_bytes(decoded, mime)
|
||||
return self._probe_audio_bytes(decoded, mime, max_seconds=max_seconds)
|
||||
if self._is_remote_or_asset(value):
|
||||
return None
|
||||
return self._probe_local_audio_duration(Path(value).expanduser())
|
||||
return self._probe_local_audio_duration(
|
||||
Path(value).expanduser(), max_seconds=max_seconds
|
||||
)
|
||||
|
||||
def _probe_audio_bytes(self, decoded: bytes, mime: str) -> float:
|
||||
def _probe_audio_bytes(
|
||||
self, decoded: bytes, mime: str, *, max_seconds: int = 15
|
||||
) -> float:
|
||||
suffix = ".wav" if mime == "audio/wav" else ".mp3"
|
||||
# Windows does not allow ffprobe to reopen a NamedTemporaryFile while
|
||||
# Python still holds the file handle. Close it before probing, then
|
||||
@@ -1202,19 +1152,17 @@ class SeedanceArkVideo(BaseTool):
|
||||
temp.flush()
|
||||
temp_path = Path(temp.name)
|
||||
try:
|
||||
return self._probe_local_audio_duration(temp_path)
|
||||
return self._probe_local_audio_duration(temp_path, max_seconds=max_seconds)
|
||||
finally:
|
||||
temp_path.unlink(missing_ok=True)
|
||||
|
||||
@staticmethod
|
||||
def _probe_local_audio_duration(path: Path) -> float:
|
||||
def _probe_local_audio_duration(path: Path, *, max_seconds: int = 15) -> float:
|
||||
if not path.is_file():
|
||||
raise ValueError(f"local audio file does not exist: {path}")
|
||||
ffprobe = shutil.which("ffprobe")
|
||||
if not ffprobe:
|
||||
raise ValueError(
|
||||
"ffprobe is required to validate local reference audio"
|
||||
)
|
||||
raise ValueError("ffprobe is required to validate local reference audio")
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[
|
||||
@@ -1234,12 +1182,10 @@ class SeedanceArkVideo(BaseTool):
|
||||
)
|
||||
duration = float(proc.stdout.strip()) if proc.returncode == 0 else 0
|
||||
except (OSError, ValueError, subprocess.SubprocessError) as exc:
|
||||
raise ValueError(f"failed to probe local reference audio: {path}") from exc
|
||||
if not 2 <= duration <= max_seconds:
|
||||
raise ValueError(
|
||||
f"failed to probe local reference audio: {path}"
|
||||
) from exc
|
||||
if not 2 <= duration <= 15:
|
||||
raise ValueError(
|
||||
"each local reference audio clip must be 2 to 15 seconds"
|
||||
f"each local reference audio clip must be 2 to {max_seconds} seconds"
|
||||
)
|
||||
return duration
|
||||
|
||||
@@ -1247,9 +1193,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
def _is_remote_or_asset(value: str) -> bool:
|
||||
return value.startswith(("https://", "http://", "asset://"))
|
||||
|
||||
def _validate_remote_refs(
|
||||
self, refs: list[Any], label: str
|
||||
) -> None:
|
||||
def _validate_remote_refs(self, refs: list[Any], label: str) -> None:
|
||||
for ref in refs:
|
||||
value = str(ref)
|
||||
if not value.startswith(("https://", "http://", "asset://")):
|
||||
@@ -1258,9 +1202,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
"Ark does not document video Base64 or local paths"
|
||||
)
|
||||
|
||||
def _validate_optional_parameters(
|
||||
self, payload: dict[str, Any]
|
||||
) -> None:
|
||||
def _validate_optional_parameters(self, payload: dict[str, Any]) -> None:
|
||||
callback = payload.get("callback_url")
|
||||
if callback is not None and not str(callback).startswith(
|
||||
("https://", "http://")
|
||||
@@ -1268,9 +1210,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
raise ValueError("callback_url must be an http(s) URL")
|
||||
expires = payload.get("execution_expires_after")
|
||||
if expires is not None and not 3600 <= int(expires) <= 259200:
|
||||
raise ValueError(
|
||||
"execution_expires_after must be between 3600 and 259200"
|
||||
)
|
||||
raise ValueError("execution_expires_after must be between 3600 and 259200")
|
||||
priority = payload.get("priority")
|
||||
if priority is not None and not 0 <= int(priority) <= 9:
|
||||
raise ValueError("priority must be between 0 and 9")
|
||||
@@ -1308,9 +1248,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
def _create_task(
|
||||
self, payload: dict[str, Any], api_key: str
|
||||
) -> str:
|
||||
def _create_task(self, payload: dict[str, Any], api_key: str) -> str:
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
@@ -1338,16 +1276,10 @@ class SeedanceArkVideo(BaseTool):
|
||||
timeout=30,
|
||||
)
|
||||
retryable_status = (
|
||||
response.status_code == 429
|
||||
or response.status_code >= 500
|
||||
response.status_code == 429 or response.status_code >= 500
|
||||
)
|
||||
if (
|
||||
retryable_status
|
||||
and attempt < self.retry_policy.max_retries
|
||||
):
|
||||
time.sleep(
|
||||
self.retry_policy.backoff_seconds * (2**attempt)
|
||||
)
|
||||
if retryable_status and attempt < self.retry_policy.max_retries:
|
||||
time.sleep(self.retry_policy.backoff_seconds * (2**attempt))
|
||||
continue
|
||||
self._raise_for_status(response)
|
||||
break
|
||||
@@ -1435,18 +1367,14 @@ class SeedanceArkVideo(BaseTool):
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
raise RuntimeError(
|
||||
f"{exc}" + (f"; {detail}" if detail else "")
|
||||
) from exc
|
||||
raise RuntimeError(f"{exc}" + (f"; {detail}" if detail else "")) from exc
|
||||
|
||||
@staticmethod
|
||||
def _task_error(task: dict[str, Any]) -> str:
|
||||
error = task.get("error")
|
||||
if isinstance(error, dict):
|
||||
return ": ".join(
|
||||
str(error.get(key))
|
||||
for key in ("code", "message")
|
||||
if error.get(key)
|
||||
str(error.get(key)) for key in ("code", "message") if error.get(key)
|
||||
)
|
||||
return str(error or "")
|
||||
|
||||
@@ -1466,8 +1394,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
model_inputs["model"] = task["model"]
|
||||
_, variant = self._resolve_model(model_inputs)
|
||||
resolution = str(
|
||||
task.get("resolution")
|
||||
or inputs.get("resolution", "720p")
|
||||
task.get("resolution") or inputs.get("resolution", "720p")
|
||||
).lower()
|
||||
condition = (
|
||||
"with_video"
|
||||
@@ -1482,9 +1409,7 @@ class SeedanceArkVideo(BaseTool):
|
||||
rate = self._get_custom_price(inputs, required=False)
|
||||
else:
|
||||
try:
|
||||
rate = self.PRICE_CNY_PER_MILLION[variant][condition][
|
||||
resolution
|
||||
]
|
||||
rate = self.PRICE_CNY_PER_MILLION[variant][condition][resolution]
|
||||
except KeyError:
|
||||
return None
|
||||
return round(float(tokens) * rate / 1_000_000, 4)
|
||||
@@ -1498,15 +1423,9 @@ class SeedanceArkVideo(BaseTool):
|
||||
cny_per_usd = self._get_cny_per_usd()
|
||||
return round(cost_cny / cny_per_usd, 4)
|
||||
|
||||
def _safe_error(
|
||||
self, exc: Exception, api_key: str | None = None
|
||||
) -> str:
|
||||
def _safe_error(self, exc: Exception, api_key: str | None = None) -> str:
|
||||
message = str(exc)
|
||||
secrets = {
|
||||
value
|
||||
for value in (api_key, self._get_api_key())
|
||||
if value
|
||||
}
|
||||
secrets = {value for value in (api_key, self._get_api_key()) if value}
|
||||
for secret in secrets:
|
||||
message = message.replace(secret, "[redacted]")
|
||||
message = re.sub(
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Seedance 2.0 (ByteDance) video generation via fal.ai API.
|
||||
"""Seedance 2.0 and 2.5 (ByteDance) video generation via fal.ai API.
|
||||
|
||||
Best for cinematic clips with native audio, director-level camera control,
|
||||
and lip-sync from quoted dialogue in prompts.
|
||||
@@ -27,7 +27,7 @@ from tools.base_tool import (
|
||||
|
||||
class SeedanceVideo(BaseTool):
|
||||
name = "seedance_video"
|
||||
version = "0.2.0"
|
||||
version = "0.3.0"
|
||||
tier = ToolTier.GENERATE
|
||||
capability = "video_generation"
|
||||
provider = "seedance"
|
||||
@@ -41,7 +41,7 @@ class SeedanceVideo(BaseTool):
|
||||
"Set FAL_KEY to your fal.ai API key.\n"
|
||||
" Get one at https://fal.ai/dashboard/keys"
|
||||
)
|
||||
agent_skills = ["seedance-2-0", "ai-video-gen"]
|
||||
agent_skills = ["seedance-2-0", "seedance-2-5", "ai-video-gen"]
|
||||
|
||||
capabilities = ["text_to_video", "image_to_video", "reference_to_video"]
|
||||
supports = {
|
||||
@@ -63,7 +63,7 @@ class SeedanceVideo(BaseTool):
|
||||
"cinematic trailers, teasers, and high-fidelity clips with native synchronized audio",
|
||||
"director-level camera control and multi-shot editing in a single generation",
|
||||
"lip-sync from quoted dialogue in prompts",
|
||||
"reference-conditioned generation (up to 9 images + 3 video clips + 3 audio clips)",
|
||||
"Seedance 2.5 reference generation (up to 30 images + 10 video + 10 audio clips)",
|
||||
"consistent character identity across shots",
|
||||
]
|
||||
not_good_for = ["offline generation", "budget-constrained projects"]
|
||||
@@ -88,9 +88,44 @@ class SeedanceVideo(BaseTool):
|
||||
"default": "standard",
|
||||
"description": "standard = highest quality, fast = lower latency and cost",
|
||||
},
|
||||
"model_version": {
|
||||
"type": "string",
|
||||
"enum": ["2.0", "2.5"],
|
||||
"default": "2.0",
|
||||
"description": "Seedance 2.5 is the current high-quality model; 2.0 retains fast-tier access.",
|
||||
},
|
||||
"duration": {
|
||||
"type": "string",
|
||||
"enum": ["auto", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15"],
|
||||
"enum": [
|
||||
"auto",
|
||||
"4",
|
||||
"5",
|
||||
"6",
|
||||
"7",
|
||||
"8",
|
||||
"9",
|
||||
"10",
|
||||
"11",
|
||||
"12",
|
||||
"13",
|
||||
"14",
|
||||
"15",
|
||||
"16",
|
||||
"17",
|
||||
"18",
|
||||
"19",
|
||||
"20",
|
||||
"21",
|
||||
"22",
|
||||
"23",
|
||||
"24",
|
||||
"25",
|
||||
"26",
|
||||
"27",
|
||||
"28",
|
||||
"29",
|
||||
"30",
|
||||
],
|
||||
"default": "5",
|
||||
"description": "Duration in seconds. 'auto' lets the model decide.",
|
||||
},
|
||||
@@ -152,8 +187,17 @@ class SeedanceVideo(BaseTool):
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout"])
|
||||
idempotency_key_fields = ["prompt", "model_variant", "operation", "duration", "seed"]
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2, retryable_errors=["rate_limit", "timeout"]
|
||||
)
|
||||
idempotency_key_fields = [
|
||||
"prompt",
|
||||
"model_version",
|
||||
"model_variant",
|
||||
"operation",
|
||||
"duration",
|
||||
"seed",
|
||||
]
|
||||
side_effects = ["writes video file to output_path", "calls fal.ai API"]
|
||||
user_visible_verification = [
|
||||
"Watch generated clip for motion coherence, audio sync, and visual quality"
|
||||
@@ -168,6 +212,16 @@ class SeedanceVideo(BaseTool):
|
||||
return ToolStatus.UNAVAILABLE
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
if inputs.get("model_version", "2.0") == "2.5":
|
||||
return round(
|
||||
0.30
|
||||
* (
|
||||
5
|
||||
if inputs.get("duration", "5") == "auto"
|
||||
else int(inputs.get("duration", "5"))
|
||||
),
|
||||
2,
|
||||
)
|
||||
variant = inputs.get("model_variant", "standard")
|
||||
duration = inputs.get("duration", "5")
|
||||
secs = 5 if duration == "auto" else int(duration)
|
||||
@@ -175,6 +229,8 @@ class SeedanceVideo(BaseTool):
|
||||
return round(rate * secs, 2)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
if inputs.get("model_version", "2.0") == "2.5":
|
||||
return 150.0
|
||||
variant = inputs.get("model_variant", "standard")
|
||||
return 60.0 if variant == "fast" else 120.0
|
||||
|
||||
@@ -190,10 +246,18 @@ class SeedanceVideo(BaseTool):
|
||||
|
||||
start = time.time()
|
||||
operation = inputs.get("operation", "text_to_video")
|
||||
model_version = inputs.get("model_version", "2.0")
|
||||
variant = inputs.get("model_variant", "standard")
|
||||
operation_path = operation.replace("_", "-")
|
||||
|
||||
if variant == "fast":
|
||||
if model_version == "2.5":
|
||||
if variant == "fast":
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="Seedance 2.5 on fal.ai has no fast endpoint; use model_variant='standard'.",
|
||||
)
|
||||
model_path = f"bytedance/seedance-2.5/{operation_path}"
|
||||
elif variant == "fast":
|
||||
model_path = f"bytedance/seedance-2.0/fast/{operation_path}"
|
||||
else:
|
||||
model_path = f"bytedance/seedance-2.0/{operation_path}"
|
||||
@@ -216,39 +280,51 @@ class SeedanceVideo(BaseTool):
|
||||
payload["image_url"] = inputs["image_url"]
|
||||
elif inputs.get("image_path"):
|
||||
from tools.video._shared import upload_image_fal
|
||||
|
||||
payload["image_url"] = upload_image_fal(inputs["image_path"])
|
||||
if inputs.get("end_image_url"):
|
||||
payload["end_image_url"] = inputs["end_image_url"]
|
||||
if model_version == "2.5":
|
||||
payload["aspect_ratio"] = "auto"
|
||||
|
||||
if operation == "reference_to_video":
|
||||
ref_image_urls = list(inputs.get("reference_image_urls") or [])
|
||||
for local_path in inputs.get("reference_image_paths") or []:
|
||||
from tools.video._shared import upload_image_fal
|
||||
|
||||
ref_image_urls.append(upload_image_fal(local_path))
|
||||
# Seedance 2.0 reference-to-video ceilings: 9 images + 3 video + 3 audio.
|
||||
if len(ref_image_urls) > 9:
|
||||
max_images = 30 if model_version == "2.5" else 9
|
||||
max_videos = 10 if model_version == "2.5" else 3
|
||||
max_audios = 10 if model_version == "2.5" else 3
|
||||
if len(ref_image_urls) > max_images:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"Seedance 2.0 reference_to_video accepts at most 9 reference images; got {len(ref_image_urls)}",
|
||||
error=f"Seedance {model_version} reference_to_video accepts at most {max_images} reference images; got {len(ref_image_urls)}",
|
||||
)
|
||||
ref_video_urls = list(inputs.get("reference_video_urls") or [])
|
||||
if len(ref_video_urls) > 3:
|
||||
if len(ref_video_urls) > max_videos:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"Seedance 2.0 reference_to_video accepts at most 3 reference videos; got {len(ref_video_urls)}",
|
||||
error=f"Seedance {model_version} reference_to_video accepts at most {max_videos} reference videos; got {len(ref_video_urls)}",
|
||||
)
|
||||
ref_audio_urls = list(inputs.get("reference_audio_urls") or [])
|
||||
if len(ref_audio_urls) > 3:
|
||||
if len(ref_audio_urls) > max_audios:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"Seedance 2.0 reference_to_video accepts at most 3 reference audio clips; got {len(ref_audio_urls)}",
|
||||
error=f"Seedance {model_version} reference_to_video accepts at most {max_audios} reference audio clips; got {len(ref_audio_urls)}",
|
||||
)
|
||||
if ref_image_urls:
|
||||
payload["reference_image_urls"] = ref_image_urls
|
||||
payload[
|
||||
"image_urls" if model_version == "2.5" else "reference_image_urls"
|
||||
] = ref_image_urls
|
||||
if ref_video_urls:
|
||||
payload["reference_video_urls"] = ref_video_urls
|
||||
payload[
|
||||
"video_urls" if model_version == "2.5" else "reference_video_urls"
|
||||
] = ref_video_urls
|
||||
if ref_audio_urls:
|
||||
payload["reference_audio_urls"] = ref_audio_urls
|
||||
payload[
|
||||
"audio_urls" if model_version == "2.5" else "reference_audio_urls"
|
||||
] = ref_audio_urls
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Key {api_key}",
|
||||
@@ -277,7 +353,7 @@ class SeedanceVideo(BaseTool):
|
||||
if status in ("FAILED", "CANCELLED"):
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"Seedance 2.0 video generation {status.lower()}",
|
||||
error=f"Seedance {model_version} video generation {status.lower()}",
|
||||
)
|
||||
|
||||
result_resp = requests.get(response_url, headers=headers, timeout=30)
|
||||
@@ -295,7 +371,7 @@ class SeedanceVideo(BaseTool):
|
||||
except Exception as e:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"Seedance 2.0 video generation failed: {e}",
|
||||
error=f"Seedance {model_version} video generation failed: {e}",
|
||||
)
|
||||
|
||||
from tools.video._shared import probe_output
|
||||
@@ -309,6 +385,7 @@ class SeedanceVideo(BaseTool):
|
||||
"prompt": inputs["prompt"],
|
||||
"operation": operation,
|
||||
"variant": variant,
|
||||
"model_version": model_version,
|
||||
"aspect_ratio": inputs.get("aspect_ratio", "16:9"),
|
||||
"resolution": inputs.get("resolution", "720p"),
|
||||
"generate_audio": inputs.get("generate_audio", True),
|
||||
|
||||
Reference in New Issue
Block a user