civitai__civitai

skills/civitai__civitai

Fork 0

mirror of https://github.com/civitai/civitai.git synced 2026-09-20 22:08:18 +08:00

Commit Graph

Author	SHA1	Message	Date
Justin Maier	98f42aab65	fix(3d-models): correct the cutover constant and drop the superseded backfill (#4054 ) * fix(3d-models): correct the cutover constant and drop the superseded backfill The constant said 2026-08-18, picked by hand before the deploy. Tracking actually started 2026-08-17 17:21:31 UTC, so the Creator Studio would have marked the boundary a day late. Now read off production, with a note that the day is MIXED - backfilled before 17:21:31, live beacons after - so a consumer marks that day and not the one after it. Removes scripts/oneoffs/backfill-model3d-views.{ts,helpers.ts} and its test. The backfill now lives in civitai-scripts (backfill/model3d-views.js), and the copy here still carried the design that shipped wrong: an --until DATE cannot express the real boundary, because tracking starts mid-day. Any date either drops that day's pre-deploy views or double-counts its post-deploy ones - measured on the real cutover, 1,488 views before the first beacon and 296 after. The replacement derives the boundary as min(time) of the first tracked day, so the two sources abut exactly and there is no value left to typo. Leaving a second, wrong copy in this repo is worse than having none: it is the one a reader finds by grepping. The DDL file is kept as the record of what ran, now marked applied. Backfill result, verified independently: 11,336 rows / 175,964 views across 476 models, 2026-06-19 to the cutover. The boundary day reads 1,650 = 162 beacons + 1,488 backfilled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS * refactor(3d-models): keep the cutover constant hardcoded, trim comments The constant stays and stays hardcoded - the backfill derives its own boundary from data and never reads it, so its only job is captioning the Studio chart, and a hand-set date is fine for that. Corrected to 2026-08-17, the day tracking actually began (17:21:31 UTC). The comment now says the thing a reader needs and nothing else: the two spans are different measurements, page loads ran ~1.5x the beacons that replaced them, so a chart crossing that day steps down for reasons unrelated to the creator. SQL comments cut 60 lines to 19, keeping only what a future editor would get wrong without them - that MODIFY COLUMN replaces rather than appends, that entityType is a sorting key so renumbering means a table rewrite, and why the MV goes last. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-17 21:41:09 -06:00
Justin Maier	eb848582ef	feat(3d-models): track Model3D views, with a pageViews backfill (#3997 ) * feat(3d-models): track Model3D views, with a pageViews backfill 3D models had no entity view tracking: no TrackView on any /3d-models route, no Model3D arm in the ClickHouse enums, nothing in daily_views. The history exists in pageViews but nothing reads it. Adds one TrackView on the detail page, reusing the existing TrackView -> /api/internal/pulse -> Tracker.view() path. No new component, no new endpoint, no new rollup table, no materialized view: daily_views is already ORDER BY (entityType, entityId, createdDate), so a per-creator query over at most 539 ids is a primary-key prefix seek. Ownership resolves from Postgres. A view is one load of a model's public detail page. /edit and /reviews are excluded by name rather than by path shape, because /3d-models/481/edit and /3d-models/481/my-slug are structurally identical - an anchored id regex matches the edit page too. Verified in ClickHouse's own RE2 against 30 days of prod pageViews: 82,319 counted, 399 edit+reviews excluded, 21 junk excluded. The DDL is not applied. It must run after the comics DDL (PR #3993), and it restates all twelve enum arms because MODIFY COLUMN replaces an Enum8 rather than appending to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS * fix(3d-models): address adversarial + simplification review Backfill correctness: - The shared ClickHouse client sets wait_for_async_insert: 0, so insert() resolved before the rows were queryable and the read-back raced the flush - a successful run could report "boundary mismatch, found 0" and then be blocked from retrying by its own guard. Wait on this one insert. - Verify the whole written range (count + sum) instead of one boundary day. A partial write that lost June and July but landed the last day passed before; a boundary day with zero rows verified nothing at all. - Assert the cutover constant against live data: the first day `views` holds a Model3DView IS the cutover. Nothing tied the constant to the deploy, so a deploy landing a day early or late left a doubled or permanently empty day that nothing detected. - Derive the id-extraction pattern and the filter from one constant. They had to agree or extract() returns '' and toUInt32('') throws mid-query. - The documented `npx ts-node` command could not resolve the ~/ imports. DDL: - Say to apply before deploying the emitting code, and to run steps 1-3 without a pause. Between step 2 and step 3 a Model3D row can reach an MV whose declared header is still nine arms, and every view type on the site shares that insert path. - State what step 0 expects rather than asking the operator to confirm against nothing. - Note that metadata-only holds only because every existing name->index pair is preserved; entityType is a sorting key, so renumbering means a rewrite. Local dev ClickHouse (containers/clickhouse/docker-init/init.sh) declared the nine-arm enums, so a fresh container silently dropped every Model3DView row. Tests: delete two DETAIL_PREDICATE assertions that could not fail on a revert - both matched substrings present in the surviving half of the predicate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS * fix(3d-models): harden the cutover guard against incidental tracking rows Applying the enum DDL removed an accidental protection: before it, a dev server or preview deploy on this branch writing a Model3DView row was rejected by the nine-arm enum. Now it succeeds. Local dev points at the same ClickHouse as prod, so one developer opening a 3D model page writes a real row - and the cutover guard took a bare min(createdDate) over them, which would move the detected cutover back to that day and silently drop every day between it and the real one. The cutover is now the first day clearing a floor no incidental page load reaches (50 rows against ~2,700/day of real traffic). Days below it that fall before the cutover are reported rather than ignored silently. The mismatch error also prescribed a lossy repair as if it were the fix: the mismatched day is almost always partial, since tracking starts mid-day. It now names both options and says not to take the first by default. Also: - select_sequential_consistency on the readback. Prod is a two-replica SharedMergeTree behind a load balancer, so the insert and the read can land on different replicas; wait_for_async_insert guarantees the write committed, not that the next query sees it. A stale read reports a false failure and the pre-flight guard then blocks the retry. - Restore the Number() coercion dropped in the last commit. It was what kept the script from depending on a client-level output format setting. - --dry-run works again before the deploy, which the live check had broken. - Restore an assertion on the edit/reviews exclusion. The previous commit claimed both deleted tests were vacuous; only one was. The other failed on a real revert, and it was dropped because ID_PATTERN broke its regex, not because it caught nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-16 20:14:06 -06:00

Author

SHA1

Message

Date

Justin Maier

98f42aab65

fix(3d-models): correct the cutover constant and drop the superseded backfill (#4054 )

* fix(3d-models): correct the cutover constant and drop the superseded backfill

The constant said 2026-08-18, picked by hand before the deploy. Tracking actually
started 2026-08-17 17:21:31 UTC, so the Creator Studio would have marked the
boundary a day late. Now read off production, with a note that the day is MIXED -
backfilled before 17:21:31, live beacons after - so a consumer marks that day and
not the one after it.

Removes scripts/oneoffs/backfill-model3d-views.{ts,helpers.ts} and its test. The
backfill now lives in civitai-scripts (backfill/model3d-views.js), and the copy
here still carried the design that shipped wrong: an --until DATE cannot express
the real boundary, because tracking starts mid-day. Any date either drops that
day's pre-deploy views or double-counts its post-deploy ones - measured on the
real cutover, 1,488 views before the first beacon and 296 after. The replacement
derives the boundary as min(time) of the first tracked day, so the two sources
abut exactly and there is no value left to typo.

Leaving a second, wrong copy in this repo is worse than having none: it is the
one a reader finds by grepping.

The DDL file is kept as the record of what ran, now marked applied.

Backfill result, verified independently: 11,336 rows / 175,964 views across 476
models, 2026-06-19 to the cutover. The boundary day reads 1,650 = 162 beacons +
1,488 backfilled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS

* refactor(3d-models): keep the cutover constant hardcoded, trim comments

The constant stays and stays hardcoded - the backfill derives its own boundary
from data and never reads it, so its only job is captioning the Studio chart,
and a hand-set date is fine for that.

Corrected to 2026-08-17, the day tracking actually began (17:21:31 UTC). The
comment now says the thing a reader needs and nothing else: the two spans are
different measurements, page loads ran ~1.5x the beacons that replaced them, so
a chart crossing that day steps down for reasons unrelated to the creator.

SQL comments cut 60 lines to 19, keeping only what a future editor would get
wrong without them - that MODIFY COLUMN replaces rather than appends, that
entityType is a sorting key so renumbering means a table rewrite, and why the
MV goes last.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-17 21:41:09 -06:00

Justin Maier

eb848582ef

feat(3d-models): track Model3D views, with a pageViews backfill (#3997 )

* feat(3d-models): track Model3D views, with a pageViews backfill

3D models had no entity view tracking: no TrackView on any /3d-models route,
no Model3D arm in the ClickHouse enums, nothing in daily_views. The history
exists in pageViews but nothing reads it.

Adds one TrackView on the detail page, reusing the existing
TrackView -> /api/internal/pulse -> Tracker.view() path. No new component, no
new endpoint, no new rollup table, no materialized view: daily_views is already
ORDER BY (entityType, entityId, createdDate), so a per-creator query over at
most 539 ids is a primary-key prefix seek. Ownership resolves from Postgres.

A view is one load of a model's public detail page. /edit and /reviews are
excluded by name rather than by path shape, because /3d-models/481/edit and
/3d-models/481/my-slug are structurally identical - an anchored id regex
matches the edit page too. Verified in ClickHouse's own RE2 against 30 days of
prod pageViews: 82,319 counted, 399 edit+reviews excluded, 21 junk excluded.

The DDL is not applied. It must run after the comics DDL (PR #3993), and it
restates all twelve enum arms because MODIFY COLUMN replaces an Enum8 rather
than appending to it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS

* fix(3d-models): address adversarial + simplification review

Backfill correctness:
- The shared ClickHouse client sets wait_for_async_insert: 0, so insert()
  resolved before the rows were queryable and the read-back raced the flush -
  a successful run could report "boundary mismatch, found 0" and then be
  blocked from retrying by its own guard. Wait on this one insert.
- Verify the whole written range (count + sum) instead of one boundary day.
  A partial write that lost June and July but landed the last day passed
  before; a boundary day with zero rows verified nothing at all.
- Assert the cutover constant against live data: the first day `views` holds a
  Model3DView IS the cutover. Nothing tied the constant to the deploy, so a
  deploy landing a day early or late left a doubled or permanently empty day
  that nothing detected.
- Derive the id-extraction pattern and the filter from one constant. They had
  to agree or extract() returns '' and toUInt32('') throws mid-query.
- The documented `npx ts-node` command could not resolve the ~/ imports.

DDL:
- Say to apply before deploying the emitting code, and to run steps 1-3
  without a pause. Between step 2 and step 3 a Model3D row can reach an MV
  whose declared header is still nine arms, and every view type on the site
  shares that insert path.
- State what step 0 expects rather than asking the operator to confirm against
  nothing.
- Note that metadata-only holds only because every existing name->index pair
  is preserved; entityType is a sorting key, so renumbering means a rewrite.

Local dev ClickHouse (containers/clickhouse/docker-init/init.sh) declared the
nine-arm enums, so a fresh container silently dropped every Model3DView row.

Tests: delete two DETAIL_PREDICATE assertions that could not fail on a revert -
both matched substrings present in the surviving half of the predicate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS

* fix(3d-models): harden the cutover guard against incidental tracking rows

Applying the enum DDL removed an accidental protection: before it, a dev
server or preview deploy on this branch writing a Model3DView row was
rejected by the nine-arm enum. Now it succeeds. Local dev points at the same
ClickHouse as prod, so one developer opening a 3D model page writes a real
row - and the cutover guard took a bare min(createdDate) over them, which
would move the detected cutover back to that day and silently drop every day
between it and the real one.

The cutover is now the first day clearing a floor no incidental page load
reaches (50 rows against ~2,700/day of real traffic). Days below it that fall
before the cutover are reported rather than ignored silently.

The mismatch error also prescribed a lossy repair as if it were the fix: the
mismatched day is almost always partial, since tracking starts mid-day. It now
names both options and says not to take the first by default.

Also:
- select_sequential_consistency on the readback. Prod is a two-replica
  SharedMergeTree behind a load balancer, so the insert and the read can land
  on different replicas; wait_for_async_insert guarantees the write committed,
  not that the next query sees it. A stale read reports a false failure and the
  pre-flight guard then blocks the retry.
- Restore the Number() coercion dropped in the last commit. It was what kept
  the script from depending on a client-level output format setting.
- --dry-run works again before the deploy, which the live check had broken.
- Restore an assertion on the edit/reviews exclusion. The previous commit
  claimed both deleted tests were vacuous; only one was. The other failed on a
  real revert, and it was dropped because ID_PATTERN broke its regex, not
  because it caught nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0125bNeEK6nqERsbALjiDaUS

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-16 20:14:06 -06:00

2 Commits