Eleven vendor-shaped Protocols are replaced by one IntegrationsDomain with
`call(integration, operation, **params)` and `describe()`. The published API
was growing by one class per third-party service while the enforcement point
— `integrations.<name>`, checked at the broker — never varied. The hand-kept
public stub had already fallen behind that growth: Luma, ImgBB and SenseNova
existed in the implementation but had never been published.
Three defects surfaced and are fixed here:
- LlamaCppModelRef.generate reached the vendor by attribute, so every pack
using the returned handle broke at the wire, not at the call site.
- The in-process path exposed vendors as public attributes and had no
`call`, so a node could work unsandboxed and fail once sandboxed. The
vendors are now private and reached only by name, giving both paths the
same surface.
- `describe` was declared sync in the Protocol but implemented async in the
guest, which must round-trip to the host.
Also removes a test for load_onnx_image_classifier, deleted from core in
2076700f without its tests; the surviving _validate_onnx_weight_file test
is kept.
The twenty handlers for IP-Adapter, SAM, CLIPSeg, image classification,
advanced ControlNet, transparent VAE, segmentation, inpainting, image
preprocessing and interpolation state used no instance state: they were
module functions parked on a class. They now live in _vendor_ops and
register through the same table, which leaves InProcessOps holding the
engine primitives and collects every pack-specific operation in one file
that can move out to the packs that own them.
They reach the SDK through the module rather than by importing names, so
substitution still resolves at call time.
A stock install — no overlay, no SDK markers on the node — now takes the
original invocation path: no ExecutionPlan, no ref table, no runtime
binding, and the pre-existing async task semantics. The seam engages per
node when it declares SDK_REFS/SDK_PERMISSIONS/SDK_REQUIRED_WEIGHTS, or
globally once any default provider is replaced, so a registered backend
still sees every node and provenance-based sandboxing cannot be bypassed
by a node simply declaring nothing.
The pipeline patches a live model, so its operations stay in core. What
leaves is the typed facade over them: IpAdapterRef and IpAdapterEmbedsRef
were published contract, so every one of their methods was a permanent
commitment. The same ergonomics now ship with the packs that want them,
built on the generic dispatch, and core keeps the operations without
keeping the surface.
ipadapter.apply, apply_tiled, encode and ipadapter_embeds.combine all
remain available by name. Handles cross as generic Refs carrying the
IPADAPTER_PIPE and IPADAPTER_EMBEDS kinds, which is what the validation and
the marshaller now check.
Ref gains _wrap so a handle with no dedicated class can still be re-typed
by the marshaller; the base keeps the resolver's kind while a subclass that
declares KIND asserts its own.
op() was defined only on ImageRef, so a pack holding any other handle had
no way to call a named operation on it and had to wait for core to grow a
typed method. The dispatch itself was always generic: the broker's wire
parameter is named image but passes straight through to
ops.apply(op, subject, params), and core's own vendor wrappers already send
non-image handles along it.
Moving op() to the base Ref is what lets a pack build its own typed
accessor over a capability core knows nothing about, so the operation
vocabulary can keep growing while the API does not. ImageRef keeps its
override to narrow the return type.
Removes OnnxDetectorRef, ctx.models.load_onnx_detector, the
onnx_detector.detect operation, its entry type, loader and cache, and the
secure_kind inference branch that mapped to the removed type.
The detector is one model family, and running an untrusted ONNX graph
belongs in the sandbox rather than in the trusted host process, so the packs
now load and execute it themselves through the generic asset broker.
Completes the removal: the hand-maintained public surface still declared
MattingModelRef and load_vitmatte after the implementation moved to the
LayerStyle pack.
ViTMatte is one pack's model family, so the architecture, the loader and the
refinement move to the pack that uses it. Removes MattingModelRef,
ctx.models.load_vitmatte, the matting.refine operation, the ViTMatte entry
type, its loader and its cache from core.
The pack asks only for generic primitives: resolve a declared weight and read
its state dict. It selects and offloads its own device, because the model is
its own and runs in its own process.
Ports 31d76988 from api-v2-runtime so both cores carry the same cache
behaviour. Seven of the thirteen copy-pasted caches evicted with
dict.popitem(), dropping the most recently inserted entry rather than the
least recently used, which pinned whichever model loaded first.
Ports the core half of the LayerStyle BLIP move to this branch, which had
only received the pack half. Removes VqaModelRef, ctx.models.load_vqa, the
VQA cache and the 30,522-token BERT vocabulary from core; LayerStyle now
carries that implementation and its own vocabulary and asks for weights
through the generic asset broker.
Brings this core back into step with api-v2-runtime, where this landed as
f1510546.
Preserves in-progress work so it is not lost: a new _cloud_media module
plus the SDK, public-surface, and model-transform changes that reference
it. All four files compile; committed as a recovery point rather than a
validated release.
Pack schemas use it for input bounds, and the module that has always
defined it (nodes.py) is a host module a sandboxed pack cannot import.
Same value by definition: 16384 is frozen into every workflow that
ever serialized a bound.
The V2 additions the KJNodes completion needed on the core side:
- _model_transforms: the closed, core-owned transform vocabulary
behind ModelRef.patch — 29 named transforms, declaratively
parameterized, validated host-side, immutable and stacking. No
function ever crosses the boundary; a pack cannot register one.
- structured-vs-live split: value()/from_value() only on structured
data refs (LATENT, AUDIO, TRACKS...). MODEL/CLIP/VAE/asset refs are
handles in every execution mode — in-process identity resolution no
longer hands a live model to node code.
- preview overrides (tiny-VAE, LTX factors), triton VAE seam, memory
attention, and profiling surfaces backing the corresponding closed
brokers in the overlay.
- torch_compile/model_patcher/model_management: compiled-view
aliasing recognized by the model manager (no double-counted
weights); shared state-dict loading path so the native loader and
the V2 broker cannot drift.
VaeRef.decode/encode, ClipRef.tokenize/encode_from_tokens_scheduled/encode,
CondRef.combine/concat, with in-process implementations behind the existing
named-op registry so an overlay extends the vocabulary without touching the
contract.
These keep the old API's shape on purpose — you still write vae.decode(latent)
— and change only what the call means: the node holds a handle, awaits, and the
decode runs on the trusted plane against weights it never sees. That is what
lets a node DECLARE a VAE input and still be sandboxable. Compatible in shape so
conversion is mechanical across a corpus nobody here maintains; different in
substance so a converted node is sandboxable by construction.
Two mirroring defects fixed before shipping: encode() sliced its input to three
channels (core's VAEEncode does not, so it silently dropped alpha), and ClipRef
offered only a combined encode(text) (CLIPTextEncodeSDXL builds one token dict
from two prompts, and the ACE nodes pass a dozen tokenizer kwargs, so the
collapse made both inexpressible).
InProcessRefResolver now records a ref's kind at creation and checks it at
resolve. Ref tokens cross as {kind, id, cls} and the host rebuilt from what
arrived, so the holder chose its own ref's type: an ImageRef id could be
presented as a VaeRef. Unguessable ids already stopped a guest reaching a handle
it was never given; possessing a handle is not the same as labelling it.
Release is explicit, not collected. InProcessRefResolver.clear() drops the
table's strong references at a known point, called from execution.py in a
finally so it also covers the path where the node raised or its guest died —
which is exactly when nothing else will run. A ref table can hold multi-gigabyte
tensors, and refcount timing neither crosses a process boundary nor is bounded
under reference cycles.
OpsProvider.apply is annotated (op, subject: Ref, params) -> Any; it claimed
ImageRef in and out while handle ops take a VaeRef and may return a LatentRef,
a CondRef, or a plain token dict.
Two generic gaps that blocked an out-of-process node from being a sampler.
1. wrap_inputs only wrapped tensors and latents. A MODEL or CONDITIONING is a
live engine object, so it passed straight through — and an out-of-process
backend cannot serialize a ModelPatcher, so any node taking one simply could
not run in a guest. The rule is now by capability rather than an enumerated
type list: a value that can cross as data does, and anything else becomes a
handle. That is the correct boundary rule anyway — objects do not cross,
handles do — and it is what lets a node take a MODEL and still run isolated.
2. ExecutionPlan.permissions was never populated, so the seam could not tell a
backend what a node needs. A node now declares SDK_PERMISSIONS and the seam
copies it onto the plan. Declaring is not granting: the backend decides, and
an out-of-process one still gates every call at the wire. Nodes that declare
nothing — the overwhelming majority — get nothing.
Behaviour-preserving in-process: 54 core seam tests and 26 overlay tests pass.
`unwrap_outputs` rebuilds a node's NodeOutput in order to swap output refs back
for real objects. It rebuilt it from the results alone — `NodeOutput(*resolved)`
— silently discarding `ui`, `expand` and `block_execution`.
The practical effect: no SDK_REFS node could be an output node. ComfyUI only
emits the `executed` websocket event, the one that delivers a node's results to
the frontend, for nodes that return ui data (`if len(output_ui) > 0`). So a
converted PreviewImage-style node executed perfectly and then displayed
nothing, with no error anywhere to explain it. `expand` (subgraph expansion)
and `block_execution` were lost the same way.
Resolving refs is a transport concern and has no business changing what the
node said. Generic fix, not specific to any backend: it applies equally to the
in-process path.
Replaces enumerated invert/scale methods on OpsProvider with generic dispatch:
apply(op, image, params) + supports(op) + a built-in registry {invert, scale}
+ register_op. ImageRef.op(name, **params) is the untyped transport seam;
invert()/scale() remain as built-in convenience. Adds OpNotSupported (carries
the capability name) so a node can fall back to the raw tier. An overlay now
extends the op vocabulary without touching OSS core.
Statically-typed op methods live on the secure-lib side (per guidance), not
here — core stays a generic seam.
Generic out-of-process enablement: the ExecutionPlan now ships the node's
module spec and ref-wrapped inputs, and dispatch receives the per-node host
runtime (refs/ctx/ops) so an external backend can execute the node elsewhere
and broker guest calls against the same ref table. In-process default ignores
all of it; legacy nodes unaffected (seam tests green).
Nodes operate on assets (image.invert()) and never receive buffers; compute
runs on the trusted plane via the OpsProvider seam. Raw buffer access becomes
a permissioned, discouraged escape hatch (raw(); forces dedicated tier under
the overlay). The execution seam wraps heavy inputs as refs for SDK_REFS
nodes and resolves output refs for downstream legacy nodes. The .pyi contract
no longer imports torch.
POC stand-ins (interface debt, ledgered in the overlay repo DEBT.md):
invert/scale enumerated on OpsProvider; duck-typed wrap_inputs; SDK_REFS
class-attr opt-in.
- execution.py: route V3 node dispatch through providers.execution_backend
with a behavior-preserving local_call closure (exact sync/async-task
semantics retained); bind per-node ctx+refs inside the invocation scope so
the concurrent-async path is correct. V1 nodes untouched. Default backend =
in-process => byte-identical to today.
- comfy_api/latest/_sdk_public.pyi: authoritative type contract for the secure
SDK (backend analog of the frontend comfy-api.d.ts): refs, ctx + domains,
ctx() accessor; host/overlay seam separated.
- custom_nodes/comfy_sdk_poc: SandboxInvert POC node authored to the SDK.
- tests-unit: seam regression (sync+async SDK nodes through the real engine;
provider-swap intercept). Verified PASS.
Establishes the open-source 'key Python API' for secure custom nodes:
- comfy_api/latest/_sdk.py: opaque typed refs (ImageRef/ModelRef/AssetRef...),
a brokered ctx surface (assets/progress/scratch/events/storage + stubs),
and a Providers registry (ExecutionBackend/CtxProvider/RefResolver) with
in-process DEFAULTS so OSS behaves exactly as today (a ref wraps the real
object; ctx is a passthrough; zero-copy, zero-overhead).
- load_overlay(): env-var (COMFY_OVERLAY_MODULE) path loader that lets a
separable, proprietary cloud overlay register isolated implementations at
the seam. Unset => pure OSS. Wired guarded into main.startup.
- comfy_api/v0_0_3: new API version exposing sdk; registered in version_list.
Nothing isolation-specific lives in OSS: this is the seam, not the engine.
* feat(assets): add namespaced model type tags
* fix(assets): mark path-derived upload tags automatic
* fix(assets): merge duplicate scan specs
* test(assets): make duplicate path normalization portable
* feat(assets): add loader_path as the authoritative loader locator (#14796)
* fix(assets): filter model_type tags by bucket extension sets
Buckets sharing a base directory (e.g. diffusion_models and a custom
unet_gguf) tagged every file in the directory regardless of whether the
bucket could load it, so .safetensors files were tagged
model_type:unet_gguf and vice versa. Carry each bucket's registered
extension set through get_comfy_models_folders and only emit a
model_type tag when the file extension matches, keeping the empty-set
match-all convention from folder_paths.filter_files_extensions.
Files under a model base matching no bucket now keep only the models
tag instead of every directory-matching model_type tag.
* feat(assets): replace response file_path with persisted loader_path
The old file_path response field was a namespaced storage locator
(models/checkpoints/foo.safetensors): not an absolute path, not unique
identity, and not the value a loader consumes. Nothing needs that shape
on the wire (hash/ID-based locating is the long-term direction), so it
is dropped rather than renamed; the storage-root matching stays internal,
powering display_name.
What loaders DO need is the in-root loader path (category dropped:
models/checkpoints/foo/bar.safetensors -> foo/bar.safetensors). Serve it
as a first-class loader_path field, persisted on asset_references
(migration 0006) and written by every ingest pipeline at insert, so
responses read the column verbatim.
Like the model_type tags, loader_path is a seed-time derivative of the
model folder registry, maintained by the same scan lifecycle (new files seed
fresh values, pruning retires rows whose bucket disappeared). Rows
predating the column serve a null loader_path; databases from before
this stack already need recreating for the base branch's tag changes.
loader_path resolves every registered base including extra_model_paths
entries; display_name only the canonical storage roots. A file can
therefore be loadable with no display name (extra-path models) or the
reverse (unregistered files under the models root), and loader_path is
null exactly when no loader can resolve the file.
* test(assets): lock loader_path matrix (asymmetry, null, persist/read)
Cover the behaviour that has no production change but is easy to regress:
the extra-path asymmetry (loadable but no storage namespace), null
loader_path persistence for orphan files, and the response reading the
stored column with a compute fallback for un-backfilled rows.
* fix(assets): persist subfolder-qualified loader_path for ingested outputs
ingest_existing_file built its seed spec with the file's basename, so
outputs saved into a subfolder persisted loader_path (and the
user_metadata filename that preview URLs split for their subfolder
param) as just the basename: the served locator pointed at a file that
does not exist at that path. Scanner and seeder specs already derive
fname via compute_loader_path; use the same derivation here.
* fix(assets): only extension-matching buckets contribute a loader_path
The model-base match in get_asset_category_and_relative_path ignored
each bucket's extension set, so a file inside a registered base whose
extension the bucket cannot load (e.g. a .txt uploaded into
model_type:checkpoints) advertised a loader_path that no loader list
would ever resolve, while the tag side of the same stack already
excluded it. Apply the extension check used for backend tags (empty set
accepts any extension), keeping loader_path null exactly when no loader
can resolve the file.
* fix(assets): refresh loader_path when re-ingesting an existing reference
upsert_reference only wrote loader_path on the INSERT branch, so
re-ingesting an existing reference (an output overwritten in place, or a
file re-registered after its loader_path derivation changed) kept the
stale or NULL value forever. Write it on the UPDATE branch too, with a
null-safe change guard so a loader_path difference alone is enough to
trigger the update, and identical values stay a no-op.
* fix(assets): repair semantic merge breakage from #14796 and master
Two textually-clean but semantically-broken merges:
- routes.py lost its folder_paths import when #14796's import block
superseded the base's, while the content-type hardening added via the
base's master merge still calls folder_paths.is_dangerous_content_type.
- master's SVG download-hardening test uploads with the pre-namespacing
bare checkpoints tag, which this branch's destination validation
rejects; use model_type:checkpoints.
---------
Co-authored-by: guill <jacob.e.segal@gmail.com>
a1d95f3f padded the decode width to the next multiple of 32 with the pad filter to fix libswscale's float YUV->GBR edge corruption, but kept the pad target height equal to the source height. The pad filter requires the target height to be a multiple of the input's vertical chroma subsampling factor, so a chroma-subsampled input such as yuv420p (the format the gbrpf32le float branch decodes) with an odd height makes the filter round the target below the input height and fail to configure: 'Padded dimensions cannot be smaller than input dimensions' (Errno 22). This is reachable from LoadImage, which routes static images through VideoFromFile, on a lossy WebP whose width is not a multiple of 32 and whose height is odd.
The pad filter also fills the added border with black, and chroma upsampling bleeds that black into the cropped edge of every unaligned-width subsampled decode.
Pad both axes to the next multiple of 32 (32 is a multiple of every vertical subsampling factor, including yuv410p's 4 that a plain even rounding misses) and run fillborders mode=smear to replicate the real edge into the padding so it never bleeds into the cropped output, then crop both axes back to the source size. Aligned-width and uint8 paths run the identical to_ndarray call as before and are byte-identical to master; only unaligned-width subsampled inputs change, from a crash or edge artifact to a clean, deterministic decode.
Create Video gets a bit_depth option (8-bit/10-bit); the selected depth is carried by the video and applied when it gets encoded. Save Video and Video Slice now keep the source bit depth instead of always quantizing to 8-bit, so 10-bit videos stay 10-bit. 10-bit uses h264 with the yuv420p10le pixel format,so there's no new codec or container.
Signed-off-by: bigcat88 <bigcat88@icloud.com>
benjcooley