mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-09-14 13:59:18 +08:00
44f6312cd8
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud needs a verified image for each merged source commit. > - Fresh builders restore compiled native dependencies from registry caches. > - The current workflow imports up to eleven historical cache manifests at once. > - Live builds missed native layers that a fresh builder reused from one manifest. > - This PR selects the nearest available cache and tests reuse across fresh builders. ## Linked Issues or Issue Description Refs #13329 and #13330. A search of open cache PRs found no duplicate of this change. **What existing behavior does this improve?** Remote Docker cache reuse on fresh Cloud image builders. **Current behavior** [Cloud run 34714483272](https://github.com/paperclipai/paperclip/actions/runs/34714483272/job/103609096836) imported the previous cache manifest successfully but rebuilt `cargo-chef` and Rust dependencies. The dependency compile took 3m43s. The preceding image build had already exported those layers. A controlled [fresh-builder diagnostic](https://github.com/paperclipai/paperclip/actions/runs/34715336530) used the same source and registry cache. The single-manifest job reused both layers immediately. The multiple-manifest job rebuilt them and failed the cache assertion. Both jobs used GitHub-hosted runners with read-only access. **Proposed behavior** Inspect cache manifests in first-parent order and import only the nearest available one. Keep full-SHA cache exports, the ten-commit search bound, and the legacy fallback. If caches cannot be read, permit a cold build. **Reason and benefit** Avoid the observed cache misses without changing image contents or builder sizes. Expected savings include about four minutes of native tool/dependency compilation when those inputs are unchanged. The final merge-to-deployable gain still needs a post-merge measurement. **Breaking changes** No image, artifact, deployment, or runner-routing contract changes. ## What Changed - Select one available ancestor cache after Docker login and Buildx setup. - Preserve separate writable cache tags for each full source SHA. - Test cache ordering, missing caches, registry errors, and workflow integration. - Add the selector tests to the existing release-registry suite. - Export a local test cache, remove the first builder, and verify a source rebuild on a fresh builder. - Document cache selection and the stronger Docker check. ## Verification - Passed 456 focused workflow, routing, readiness, preview-artifact, and cache-selector tests. - Passed shell syntax, ShellCheck for the changed probe, actionlint workflow validation, and `git diff --check`. actionlint's shell checks were disabled for the workflow validation because unchanged migration-label commands trigger existing SC2012 notes. - The fresh-builder registry diagnostic proves the single-cache behavior. The [permanent two-builder probe passed](https://github.com/paperclipai/paperclip/actions/runs/34715771048/job/103612624090), including a changed real binary and dependency-declaration invalidation. - Passed all 35 latest-head checks (green or intentionally skipped), including full typecheck, test, build, and browser suites in [PR CI run 34715771217](https://github.com/paperclipai/paperclip/actions/runs/34715771217). - The real selector CLI inspected registry metadata and chose the nearest available ancestor cache. - Fresh Greptile review is 5/5 with no open findings. The PR title was corrected to meet the source-change naming rule; the review check passed after that correction. - Local full-suite runs and Docker builds are unavailable because the local Docker daemon is unresponsive after disk exhaustion. CI provides the Linux verification. ## Risks - Missing or unreadable caches cause a slower cold build. The selector logs that condition and preserves image publication. - Inspecting several missing ancestors adds lookup time. Each lookup has a ten-second timeout and the search is bounded. - The Docker test now exports a local cache. It removes the first builder before starting the second to release disk space, then cleans up its builders and files. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
67 lines
2.4 KiB
JavaScript
67 lines
2.4 KiB
JavaScript
#!/usr/bin/env node
|
|
import { execFileSync } from "node:child_process";
|
|
import { appendFileSync } from "node:fs";
|
|
import { pathToFileURL } from "node:url";
|
|
|
|
export function cloudCacheCandidates(image, commits) {
|
|
if (!/^ghcr\.io\/[a-z0-9._-]+\/[a-z0-9._-]+$/.test(image ?? "")) {
|
|
throw new Error("Expected a GHCR owner/repository cache image.");
|
|
}
|
|
if (!Array.isArray(commits) || commits.length === 0 || commits.some((sha) => !/^[a-f0-9]{40}$/.test(sha))) {
|
|
throw new Error("Cloud cache ancestry requires full commit SHAs.");
|
|
}
|
|
return [
|
|
...[...new Set(commits)].slice(0, 10).map((sha) => `${image}:buildcache-cloud-${sha}`),
|
|
`${image}:buildcache-cloud`,
|
|
];
|
|
}
|
|
|
|
export async function selectCloudCache(image, commits, {
|
|
exists = registryCacheExists,
|
|
log = console.log,
|
|
} = {}) {
|
|
for (const ref of cloudCacheCandidates(image, commits)) {
|
|
try {
|
|
if (!await exists(ref)) continue;
|
|
log(`Using cloud cache: ${ref}`);
|
|
return `type=registry,ref=${ref}`;
|
|
} catch {
|
|
// Cache availability must not turn an otherwise valid build into a
|
|
// failure. A later ancestor may still be available during a rollout.
|
|
log(`Could not inspect cloud cache ${ref}; trying the next ancestor.`);
|
|
}
|
|
}
|
|
log("No cloud cache is available; this build will populate one.");
|
|
return "";
|
|
}
|
|
|
|
function registryCacheExists(ref) {
|
|
try {
|
|
// Use the preceding Docker login, including for private registry caches.
|
|
// Inspect metadata only: no layer download and no image execution.
|
|
execFileSync("docker", ["buildx", "imagetools", "inspect", "--raw", ref], {
|
|
timeout: 10_000,
|
|
maxBuffer: 1024 * 1024,
|
|
stdio: ["ignore", "pipe", "pipe"],
|
|
});
|
|
return true;
|
|
} catch (error) {
|
|
if (/manifest unknown|not found|NAME_UNKNOWN/i.test(String(error.stderr ?? ""))) return false;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
|
try {
|
|
if (!process.env.GITHUB_OUTPUT) throw new Error("GITHUB_OUTPUT is required.");
|
|
const commits = execFileSync("git", ["rev-list", "--first-parent", "--max-count=10", "HEAD"], {
|
|
encoding: "utf8",
|
|
}).trim().split("\n");
|
|
const source = await selectCloudCache(process.env.CACHE_IMAGE, commits, { exists: registryCacheExists });
|
|
appendFileSync(process.env.GITHUB_OUTPUT, `source=${source}\n`);
|
|
} catch (error) {
|
|
console.error(error.message);
|
|
process.exitCode = 1;
|
|
}
|
|
}
|