Misc input validation improvements, sanitizing path segments in both SQL
and JSON queries, standardizing the processing of column and JSON paths
across different adapters, and making adjustments to traversal and alias
generation to align behavior across components.
This PR properly runs `sanitizeWhereQuery` for joined collections which
is important specifically for virtual relationship fields path
transformation.
Fixes https://github.com/payloadcms/payload/issues/15332
### What
Ensures unknown query operators are properly rejected during query
validation.
### Why
Previously, unrecognized operators in where clauses were silently
ignored. This could lead to unexpected behavior. Query validation should
fail-closed and only accept known operators.
### How
Added an `else` in validateQueryPaths.ts to push an error for any
operator not in the valid operator set.
Test added to `joins > int`
This PR adds `customID` argument to `payload.db.create` which can be
used to ensure that a document is created with the provided ID, for
example:
```ts
payload.db.create({ collection: 'posts', customID: 'ce98d6c4-c3ab-45de-9dfc-bf33d94cc941', data: { } })
```
This does not require to have a custom ID field in `posts` collection,
as the following:
```ts
payload.db.create({ collection: 'posts', data: { customID: 'ce98d6c4-c3ab-45de-9dfc-bf33d94cc941' } })
```
Can be used only when you define a custom ID field
This PR adds ability to migrate an existing project to use the
`blocksAsJSON` postgres/sqlite property.
https://github.com/payloadcms/payload/pull/12750
Usage: `pnpm payload migrate:create --file
@payloadcms/db-postgres/blocks-as-json`.
Example of a generated migration for the website template:
```ts
import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'
import { getBlocksToJsonMigrator } from '@payloadcms/db-postgres/migration-utils'
import { fileURLToPath } from 'url'
import path from 'path'
const filename = fileURLToPath(import.meta.url)
const dirname = path.dirname(filename)
// Configure migration options (optional)
const BATCH_SIZE = 100 // Number of entities to process per batch
const TEMP_FOLDER = path.resolve(dirname, '.payload-blocks-migration') // Folder path to store migration batch
export async function up({ db, payload, req }: MigrateUpArgs): Promise<void> {
const migrator = getBlocksToJsonMigrator(payload)
migrator.setTempFolder(TEMP_FOLDER)
await migrator.collectAndSaveEntitiesToBatches(req, { batchSize: BATCH_SIZE })
await db.execute(sql`
ALTER TABLE "pages_blocks_cta_links" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "pages_blocks_cta" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "pages_blocks_content_columns" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "pages_blocks_content" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "pages_blocks_media_block" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "pages_blocks_archive" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "pages_blocks_form_block" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "_pages_v_blocks_cta_links" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "_pages_v_blocks_cta" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "_pages_v_blocks_content_columns" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "_pages_v_blocks_content" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "_pages_v_blocks_media_block" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "_pages_v_blocks_archive" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "_pages_v_blocks_form_block" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_checkbox" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_country" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_email" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_message" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_number" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_select_options" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_select" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_state" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_text" DISABLE ROW LEVEL SECURITY;
ALTER TABLE "forms_blocks_textarea" DISABLE ROW LEVEL SECURITY;
DROP TABLE "pages_blocks_cta_links" CASCADE;
DROP TABLE "pages_blocks_cta" CASCADE;
DROP TABLE "pages_blocks_content_columns" CASCADE;
DROP TABLE "pages_blocks_content" CASCADE;
DROP TABLE "pages_blocks_media_block" CASCADE;
DROP TABLE "pages_blocks_archive" CASCADE;
DROP TABLE "pages_blocks_form_block" CASCADE;
DROP TABLE "_pages_v_blocks_cta_links" CASCADE;
DROP TABLE "_pages_v_blocks_cta" CASCADE;
DROP TABLE "_pages_v_blocks_content_columns" CASCADE;
DROP TABLE "_pages_v_blocks_content" CASCADE;
DROP TABLE "_pages_v_blocks_media_block" CASCADE;
DROP TABLE "_pages_v_blocks_archive" CASCADE;
DROP TABLE "_pages_v_blocks_form_block" CASCADE;
DROP TABLE "forms_blocks_checkbox" CASCADE;
DROP TABLE "forms_blocks_country" CASCADE;
DROP TABLE "forms_blocks_email" CASCADE;
DROP TABLE "forms_blocks_message" CASCADE;
DROP TABLE "forms_blocks_number" CASCADE;
DROP TABLE "forms_blocks_select_options" CASCADE;
DROP TABLE "forms_blocks_select" CASCADE;
DROP TABLE "forms_blocks_state" CASCADE;
DROP TABLE "forms_blocks_text" CASCADE;
DROP TABLE "forms_blocks_textarea" CASCADE;
ALTER TABLE "pages_rels" DROP CONSTRAINT "pages_rels_categories_fk";
ALTER TABLE "_pages_v_rels" DROP CONSTRAINT "_pages_v_rels_categories_fk";
DROP INDEX "pages_rels_categories_id_idx";
DROP INDEX "_pages_v_rels_categories_id_idx";
ALTER TABLE "pages" ADD COLUMN "layout" jsonb;
ALTER TABLE "_pages_v" ADD COLUMN "version_layout" jsonb;
ALTER TABLE "forms" ADD COLUMN "fields" jsonb;
ALTER TABLE "pages_rels" DROP COLUMN "categories_id";
ALTER TABLE "_pages_v_rels" DROP COLUMN "categories_id";
DROP TYPE "public"."enum_pages_blocks_cta_links_link_type";
DROP TYPE "public"."enum_pages_blocks_cta_links_link_appearance";
DROP TYPE "public"."enum_pages_blocks_content_columns_size";
DROP TYPE "public"."enum_pages_blocks_content_columns_link_type";
DROP TYPE "public"."enum_pages_blocks_content_columns_link_appearance";
DROP TYPE "public"."enum_pages_blocks_archive_populate_by";
DROP TYPE "public"."enum_pages_blocks_archive_relation_to";
DROP TYPE "public"."enum__pages_v_blocks_cta_links_link_type";
DROP TYPE "public"."enum__pages_v_blocks_cta_links_link_appearance";
DROP TYPE "public"."enum__pages_v_blocks_content_columns_size";
DROP TYPE "public"."enum__pages_v_blocks_content_columns_link_type";
DROP TYPE "public"."enum__pages_v_blocks_content_columns_link_appearance";
DROP TYPE "public"."enum__pages_v_blocks_archive_populate_by";
DROP TYPE "public"."enum__pages_v_blocks_archive_relation_to";`)
payload.logger.info("Executed blocks to JSON migration statements.")
await migrator.migrateEntitiesFromTempFolder(req, { clearBatches: true })
}
export async function down({ db, payload, req }: MigrateDownArgs): Promise<void> {
// Migration code
}
```
The migration when created will automatically update the payload config
file (if one is found) to insert `blocksAsJSON: true`.
Additionally, extends the predefined migration API with `dynamic` to
dynamically build the migration file.
## Localized Status (Experimental)
This PR introduces a new **experimental** option that allows each locale
to track and manage its own publication status independently, for
collection docs and globals.
### Configuration
To enable this feature, you need **two** configurations:
1. Enable the experimental flag in your `payload.config`:
```ts
experimental: {
localizeStatus: true, // default: `false`
}
```
2. Enable it on specific collections/globals:
```ts
export const Posts: CollectionConfig = {
slug: 'posts',
versions: {
drafts: {
localizeStatus: true, // default: false
},
},
// ...
}
```
## Key Changes
When enabled:
- **Per-locale status tracking** - Each locale maintains its own
published/draft status
- **Independent publishing** - Publish or unpublish individual locales
without affecting others
- **Locale-aware UI** - Admin panel shows status for the currently
active locale
- **Localized version history** - Versions list reflects the current
locale's status
## Improved Behavior
- Creating a document in one locale sets that locale to the specified
status and other locales default to draft
- You can publish/unpublish specific locales independently
- Collection list views show status for the currently active locale
- Document edit view displays the current locale's status
## Migration Required (If you already have version data)
> If this is a new project then you only need to enable the flags. If
you have existing data you will want to continue with the migration
guide below.
⚠️ Breaking Change: When `localizeStatus` is enabled, existing `_status`
fields will need to be migrated from strings to locale objects.
### ➡️ Step 1
Before doing anything, **take a backup of your current database**.
### ➡️ Step 2
Stop your dev server if it is running
### ➡️ Step 3
**Create migration file**
```ts
// run the following to create a blank
// migration file named `localize_status`
payload migrate:create localize_status
```
**Add migration code**
🔵 **PostgreSQL / SQLite**:
```ts
import type { MigrateDownArgs, MigrateUpArgs } from '@payloadcms/db-postgres'
import { sql } from '@payloadcms/db-postgres'
import { localizeStatus } from 'payload/migrations'
export async function up({ db, payload }: MigrateUpArgs): Promise<void> {
await localizeStatus.up({
collectionSlug: 'posts', // 👈 Change to your collection
db,
payload,
sql,
})
}
export async function down({ db, payload }: MigrateDownArgs): Promise<void> {
await localizeStatus.down({
collectionSlug: 'posts', // 👈 Change to your collection
db,
payload,
sql,
})
}
```
🟢 **MongoDB**:
```ts
import type { MigrateDownArgs, MigrateUpArgs } from '@payloadcms/db-mongodb'
import { localizeStatus } from 'payload/migrations'
export async function up({ payload }: MigrateUpArgs): Promise<void> {
await localizeStatus.up({
collectionSlug: 'posts', // 👈 Change to your collection
payload,
})
}
export async function down({ payload }: MigrateDownArgs): Promise<void> {
await localizeStatus.down({
collectionSlug: 'posts', // 👈 Change to your collection
payload,
})
}
```
### ➡️ Step 4
Run the migration
```ts
payload migrate
```
### ➡️ Step 5
Set `localizeStatus: true`
**Payload Config**
```ts
// payload config file
experimental: {
localizeStatus: true, // <-- add this
}
```
**Collection Config**
```ts
// collection you want to migrate
export const Posts: CollectionConfig = {
slug: 'posts',
versions: {
drafts: {
localizeStatus: true, // <-- add this
},
},
// ...
}
```
### ➡️ Step 6
Run `pnpm dev` and test out the feature.
## Related PRs in this feature
- [feat: adds versions.drafts.localizeStatus and allows unpublish
per‑locale #14667](https://github.com/payloadcms/payload/pull/14667)
- [feat(ui): experimental localize metadata UI
#14699](https://github.com/payloadcms/payload/pull/14699)
- [chore: localize status migration work
#14862](https://github.com/payloadcms/payload/pull/14862)
---------
Co-authored-by: Jessica Chowdhury <jessica@trbl.design>
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
Co-authored-by: Jessica Rynkar <67977755+jessrynkar@users.noreply.github.com>
`skip` is an internal API. It is used only by the database adapters and
is not publicly exposed through the local API, only when accessing the
database directly. It does exactly the same thing as `page`, but with
different semantics. In this PR I am deprecating it because it adds
nothing but complexity. It is used only once in the entire repository
and conflicts with the `page` parameter.
## Summary
Adds a `bulkOperationsSingleTransaction` config option to the MongoDB
adapter to handle database transaction limitations when processing large
numbers of documents in bulk operations.
## Problem
Some databases (e.g., DocumentDB, Cosmos DB) have transaction
limitations when working with large datasets. DocumentDB throws an error
when a transaction attempts to use a cursor for more than 100 documents:
`Feature not supported: cursor within transaction. Try increasing the
batchsize.`
## Solution
- Adds `bulkOperationsSingleTransaction` boolean option to MongoDB
adapter (`@payloadcms/db-mongodb`)
- Added to `BaseDatabaseAdapter` interface for cross-adapter
compatibility
- Included in `compatibilityOptions` for DocumentDB and Cosmos DB
(default: `true`)
- When enabled, bulk update and delete operations process documents
sequentially in separate transactions
- When disabled (default: `false`), maintains current behavior with
parallel processing in a single transaction
- Fully backward compatible - no changes to existing behavior unless
explicitly configured
## Changes
- **MongoDB Adapter**: Add `bulkOperationsSingleTransaction` config
option
- **Base Adapter Interface**: Add optional property to
`BaseDatabaseAdapter`
- **Compatibility Options**: Enable by default for DocumentDB and Cosmos
DB
- **Core Operations**: Modify `updateOperation` and `deleteOperation` to
support per-document transactions
- **Documentation**: Add to MongoDB adapter options documentation
- **Tests**: Add integration tests for both update and delete operations
## Testing
Tested with both `bulkOperationsSingleTransaction: true` and `false` to
ensure:
- Documents are correctly updated/deleted in both modes
- Sequential processing works without transaction conflicts
- Parallel processing (default) maintains optimal performance
Fixes https://github.com/payloadcms/payload/issues/15006https://github.com/payloadcms/payload/pull/14337 introduced a next build
warning, because the `eval` around `await import` statements has been
removed.
Unlike jest, vitest is fine with it. But during Next.js build, the
bundler will throw a `"Critical dependency: the request of a dependency
is an expression"` warning.
This PR fixes this warning by adding back the eval statement around the
import. It does not add back the conditional require statement - that
part was only necessary to appease jest.
For vitest, we have to directly call `await import()`. Using eval in
vitest will throw an `ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING` error.
## Preventing this regression in the future
This PR adds a comment above the eval explaining why it's needed.
Previously this was missing, and we only had a comment explaining why
the `require` was needed.
Additionally, the reason this was not caught in CI was because warnings
during next build did not fail the build script. This PR adds an
additional check in `build-template-with-local-pkgs.ts` that throws an
error if a warning was detected during next build. You can find the CI
failure here:
https://github.com/payloadcms/payload/actions/runs/20402288354/job/58626594247?pr=15007
(CI run from commit that added this check, before the commit that fixed
it)
Now, if this ever happens again, our CI will fail and prevent a
regression.
## More Tests
This PR adds additional database and job queue tests that verify that
the job queue handler and migration file imports work correctly.
In large projects, calculating the `permissions` object is one of the
slowest parts of Payload. This PR completely rewrites the permission
calculation system to make it significantly faster and more reliable.
The permissions object is calculated multiple times throughout Payload's
lifecycle: for the entire config every time we load or navigate to any
admin panel page, on every API endpoint call, and as part of query path
validation when calling any Payload operation.
This is done by `getAccessResults` without any document data, and in
large configs can be one of the slowest operations, noticeably slowing
down admin panel navigation.
After this rewrite, **speed improvements in permissions calculation
without data range from 5.3% (access-control test suite) to 74% (fields
test suite - our largest config)**. The larger the config, the more
noticeable these improvements become. These performance gains apply to
both the Payload Admin Panel and API requests.
---
The `permissions` object can also be calculated WITH document data,
which triggers evaluation of `Where` conditions in
collection/global-level access control and may fetch the document to
pass to access control functions.
This has been heavily optimized with **speed improvements for
permissions calculation with data ranging between 75% and 5138%** (if I
craft a collection config that excessively uses aspects this PR
improves) in the benchmarks. This optimization affects:
- Loading any document (collection/global edit view or drawer)
- Saving a document
- Loading the list view with query presets applied
- Bulk upload initialization (runs twice)
These improvements are most noticeable when navigating _to_ documents in
the admin panel.
## What's Fixed
In addition to performance improvements, the new function is now
cleaner, easier to understand and works more reliably. I specifically
wanna call out two issues that were fixed:
**Field access control data bug**: Added a new e2e test that was
previously failing - when saving a document, the data object passed to
field-level update access control functions had the wrong shape (or in
some cases did not exist at all). This caused fields to show incorrect
`readOnly` states after save. The new implementation properly passes
document data through all access control checks.
**Where query validation**: The old implementation sometimes skipped
validating `Where` queries for collections when `req.data` existed,
potentially granting incorrect access. The new version always validates
where queries correctly when `fetchData: true`.
**Global Where queries:**: Where queries were _never_ executed for
globals. Instead, [we were just checking if the global existed in
general](https://github.com/payloadcms/payload/blob/main/packages/payload/src/utilities/getEntityPolicies.ts#L59).
This PR ensures that `Where` queries returned from global access control
are respected.
## What's Faster (and Why)
**Where query caching:** When multiple operations return identical where
queries (common pattern), we now cache the result. Instead of 3-4
separate DB calls, we make 1 call and reuse it.
**Parallel execution:** All `Where` query evaluations and async access
control functions (for both collections and fields) now execute
concurrently with maximum parallelism, rather than sequentially.
**Optimized DB operations**: When evaluating returned `Where` queries,
we now use direct database `db.count()` queries instead of
`payload.find()` operations. This is much faster (especially on
Postgres).
**Synchronous tree traversal**: The entire field permission tree is now
built synchronously with all async work collected and executed in
parallel at the end, rather than cascading await calls through each
nesting level. This eliminates sequential bottlenecks in deeply nested
field structures.
## Benchmarks
### Admin Panel
- Fields test suite with access control added
- 1000 blocks added to the blocks collection. This is not as excessive
as you would expect:
- most Payload apps do not run on an M3 Max. CPU/DB is often much, much
slower
- this is all local, with a local DB
- a lot of projects accumulate a huge amount of blocks if you multiply
blocks x block fields. They can definitely reach 1000 total blocks
- each block is simple - only one text field per block. In real
projects, blocks are usually a lot more complex
**Before:**
https://github.com/user-attachments/assets/37b55f02-6dbc-4005-9da6-d7cd4bfdc925
**After:**
https://github.com/user-attachments/assets/8dd796dd-66ed-4d32-a688-27b4169577c6
Branch:
https://github.com/payloadcms/payload/tree/fix/update-field-access-control-after-save-benchmarks
### Modified Access-Control Test Suite
`cd test && pnpm payload run access-control/benchmark-permissions.ts`
```md
📊 Benchmark 1: getAccessResults (all collections + globals)
──────────────────────────────────────────────────────────────────────
┌─────────┬───────────────────┬─────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼───────────────────┼─────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (optimized)' │ '16.91' │ '59203.986' │ '±0.52%' │
│ 1 │ 'OLD (previous)' │ '16.07' │ '62323.953' │ '±0.57%' │
└─────────┴───────────────────┴─────────┴───────────────────┴──────────┘
⚡ Speedup: +5.3% 🚀📊 DB Calls per operation (across all collections + globals):
NEW: 0.0 total (0.0 data, 0.0 where)
OLD: 0.0 total (0.0 data, 0.0 where)
📊 Benchmark 2: docAccessOperation (with fetchData)
──────────────────────────────────────────────────────────────────────
Collection: where-cache-same (same where queries)
┌─────────┬────────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (with cache)' │ '2272.81' │ '442.319' │ '±0.11%' │
│ 1 │ 'OLD (no cache)' │ '1019.72' │ '983.347' │ '±0.12%' │
└─────────┴────────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +122.9% 🚀📊 DB Calls per operation:
NEW: 2.0 total (1.0 data, 1.0 where)
OLD: 4.0 total (1.0 data, 3.0 where)
📊 Benchmark 3: docAccessOperation (with data passed)
──────────────────────────────────────────────────────────────────────
Collection: where-cache-same (same where queries, no DB fetch)
┌─────────┬────────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (with cache)' │ '4945.43' │ '203.252' │ '±0.11%' │
│ 1 │ 'OLD (no cache)' │ '1023.75' │ '979.898' │ '±0.14%' │
└─────────┴────────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +383.1% 🚀📊 DB Calls per operation:
NEW: 1.0 total (0.0 data, 1.0 where)
OLD: 4.0 total (1.0 data, 3.0 where)
📊 Benchmark 4: docAccessOperation (unique where queries)
──────────────────────────────────────────────────────────────────────
Collection: where-cache-unique (unique where queries per operation)
┌─────────┬────────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (parallel)' │ '1693.67' │ '599.091' │ '±0.29%' │
│ 1 │ 'OLD (sequential)' │ '970.37' │ '1036.268' │ '±0.20%' │
└─────────┴────────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +74.5% 🚀📊 DB Calls per operation:
NEW: 4.0 total (1.0 data, 3.0 where)
OLD: 4.0 total (1.0 data, 3.0 where)
📊 Benchmark 5: Complex Collection (async access, nested blocks, field access)
──────────────────────────────────────────────────────────────────────
Collection: complex-content (stress test)
┌─────────┬────────────────────┬──────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼──────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (optimized)' │ '133.17' │ '7683.346' │ '±0.83%' │
│ 1 │ 'OLD (sequential)' │ '72.86' │ '13901.616' │ '±0.82%' │
└─────────┴────────────────────┴──────────┴───────────────────┴──────────┘
⚡ Speedup: +82.8% 🚀📊 DB Calls per operation:
NEW: 2.0 total (1.0 data, 1.0 where)
OLD: 2.0 total (1.0 data, 1.0 where)
📊 Benchmark 6: Sync-Heavy Collection (same where, many sync field access)
──────────────────────────────────────────────────────────────────────
Collection: sync-heavy (where cache + field access)
┌─────────┬────────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (with cache)' │ '3607.72' │ '279.559' │ '±0.15%' │
│ 1 │ 'OLD (no cache)' │ '68.87' │ '15136.557' │ '±1.44%' │
└─────────┴────────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +5138.5% 🚀📊 DB Calls per operation:
NEW: 1.0 total (0.0 data, 1.0 where)
OLD: 5.0 total (1.0 data, 4.0 where)
══════════════════════════════════════════════════════════════════════
📈 Summary:
══════════════════════════════════════════════════════════════════════
1. getAccessResults: (see above)
2. docAccessOperation (with fetchData): (see above)
3. docAccessOperation (with data passed): (see above)
4. docAccessOperation (unique where): (see above)
5. Complex collection (async + nested): (see above)
6. Sync-heavy (where cache + fields): (see above)
══════════════════════════════════════════════════════════════════════
```
### Fields Test Suite
`cd test && pnpm payload run fields/benchmark-getAccessResults.ts`
```md
📊 Benchmark: getAccessResults (all fields test collections + globals)
──────────────────────────────────────────────────────────────────────
┌─────────┬───────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼───────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (optimized)' │ '2104.06' │ '478.382' │ '±0.18%' │
│ 1 │ 'OLD (previous)' │ '1208.14' │ '834.974' │ '±0.21%' │
└─────────┴───────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +74.2% 🚀📊 DB Calls per operation (across all collections + globals):
NEW: 0.0 total (0.0 data, 0.0 where)
OLD: 0.0 total (0.0 data, 0.0 where)
```
### What?
Ensures `getLocalizedPaths` doesn't falsely detect fields with
locale-like names as a localized field.
### Why?
In the function we are determining if the next segment is a locale based
on whether the segment of the field path matches a locale:
```ts
const nextSegmentIsLocale = localizationConfig && localizationConfig.localeCodes.includes(nextSegment)
```
If you have locale `en` and a nested field with `name: 'en'`, the
function will mistake it for a locale key regardless of whether the
field is localized.
### How?
Uses the existing `fieldShouldBeLocalized` function to check whether the
field is localized in addition to the other checks.
---------
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Fixes https://github.com/payloadcms/payload/issues/14379
Additionally, removes duplication of `findMigrationDir` between DB
adapters, deprecates the export from `@payloadcms/drizzle` and adds unit
testing to this function.
- Adds documentdb and cosmosdb to CI test matrix
- Adds missing generateDatabaseAdapter configs
- Adjust generateDatabaseAdapter firestore config to make it pure to the
compatibilityOptions we provide as an export
Creating as draft because I expect a few tests to fail based on previous
comments.
---------
Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
### What?
Adds a new `experimental.localizeStatus` config option, set to `false`
by default. When `true`, the admin panel will display the document
status based on the *current locale* instead of the _latest_ overall
status. Also updates the edit view to only show a `changed` status when
`autosave` is enabled.
### Why?
Showing the status for the current locale is more accurate and useful in
multi-locale setups. This update will become default behavior, able to
be opted in by setting `experimental.localizeStatus: true` in the
Payload config. This option will become depreciated in V4.
### How?
When `localizeStatus` is `true`, we store the localized status in a new
`localeStatus` field group within version data. The admin panel then
reads from this field to display the correct status for the current
locale.
---------
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
Catches list filter errors and prevents the list view from crashing when
attempting to search on fields the user does not have access to. Instead
just shows the default "no results found" message.
Previously, filtering by a polymorphic relationship inside an array /
group (unless the `name` is `version`) / tab caused `QueryError: The
following path cannot be queried:`.
Adds a new `schedule` property to workflow and task configs that can be
used to have Payload automatically _queue_ jobs following a certain
_schedule_.
Docs:
https://payloadcms.com/docs/dynamic/jobs-queue/schedules?branch=feat/schedule-jobs
## API Example
```ts
export default buildConfig({
// ...
jobs: {
// ...
scheduler: 'manual', // Or `cron` if you're not using serverless. If `manual` is used, then user needs to set up running /api/payload-jobs/handleSchedules or payload.jobs.handleSchedules in regular intervals
tasks: [
{
schedule: [
{
cron: '* * * * * *',
queue: 'autorunSecond',
// Hooks are optional
hooks: {
// Not an array, as providing and calling `defaultBeforeSchedule` would be more error-prone if this was an array
beforeSchedule: async (args) => {
// Handles verifying that there are no jobs already scheduled or processing.
// You can override this behavior by not calling defaultBeforeSchedule, e.g. if you wanted
// to allow a maximum of 3 scheduled jobs in the queue instead of 1, or add any additional conditions
const result = await args.defaultBeforeSchedule(args)
return {
...result,
input: {
message: 'This task runs every second',
},
}
},
afterSchedule: async (args) => {
await args.defaultAfterSchedule(args) // Handles updating the payload-jobs-stats global
args.req.payload.logger.info(
'EverySecond task scheduled: ' +
(args.status === 'success' ? args.job.id : 'skipped or failed to schedule'),
)
},
},
},
],
slug: 'EverySecond',
inputSchema: [
{
name: 'message',
type: 'text',
required: true,
},
],
handler: ({ input, req }) => {
req.payload.logger.info(input.message)
return {
output: {},
}
},
}
]
}
})
```
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210495300843759
Adds a new operation findDistinct that can give you distinct values of a
field for a given collection
Example:
Assume you have a collection posts with multiple documents, and some of
them share the same title:
```js
// Example dataset (some titles appear multiple times)
[
{ title: 'title-1' },
{ title: 'title-2' },
{ title: 'title-1' },
{ title: 'title-3' },
{ title: 'title-2' },
{ title: 'title-4' },
{ title: 'title-5' },
{ title: 'title-6' },
{ title: 'title-7' },
{ title: 'title-8' },
{ title: 'title-9' },
]
```
You can now retrieve all unique title values using findDistinct:
```js
const result = await payload.findDistinct({
collection: 'posts',
field: 'title',
})
console.log(result.values)
// Output:
// [
// 'title-1',
// 'title-2',
// 'title-3',
// 'title-4',
// 'title-5',
// 'title-6',
// 'title-7',
// 'title-8',
// 'title-9'
// ]
```
You can also limit the number of distinct results:
```js
const limitedResult = await payload.findDistinct({
collection: 'posts',
field: 'title',
sortOrder: 'desc',
limit: 3,
})
console.log(limitedResult.values)
// Output:
// [
// 'title-1',
// 'title-2',
// 'title-3'
// ]
```
You can also pass a `where` query to filter the documents.
Fixes an issue when querying deeply new relationship virtual fields with
`draft: true`. Changes the method for `where` sanitization, before it
was done in `validateSearchParam` which didn't work with versions
properly, now there's a separate `sanitizeWhereQuery` function that does
this.
Previously, there were multiple ways to type a running job:
- `GeneratedTypes['payload-jobs']` - only works in an installed project
- is `any` in monorepo
- `BaseJob` - works everywhere, but does not incorporate generated types
which may include type for custom fields added to the jobs collection
- `RunningJob<>` - more accurate version of `BaseJob`, but same problem
This PR deprecated all those types in favor of a new `Job` type.
Benefits:
- Works in both monorepo and installed projects. If no generated types
exist, it will automatically fall back to `BaseJob`
- Comes with an optional generic that can be used to narrow down
`job.input` based on the task / workflow slug. No need to use a separate
type helper like `RunningJob<>`
With this new type, I was able to replace every usage of
`GeneratedTypes['payload-jobs']`, `BaseJob` and `RunningJob<>` with the
simple `Job` type.
Additionally, this PR simplifies some of the logic used to run jobs
Important: An intentional effort is being made during migration to not
modify runtime behavior. This implies that there will be several
assertions, non-null assertions, and @ts-expect-error. This philosophy
applies only to migrating old code to TypeScript strict, not to writing
new code. For a more detailed justification for this reasoning, see
#11840 (comment).
In this PR, instead of following the approach of migrating a subset of
files, I'm migrating all files by disabling specific rules. The first
commits are named after the rule being disabled.
With this PR, the migration of the payload package is complete 🚀
Important: An intentional effort is being made during migration to not
modify runtime behavior. This implies that there will be several
assertions, non-null assertions, and @ts-expect-error. This philosophy
applies only to migrating old code to TypeScript strict, not to writing
new code. For a more detailed justification for this reasoning,
https://github.com/payloadcms/payload/pull/11840#discussion_r2021975897.
In this PR, instead of following the approach of migrating a subset of
files, I'm migrating all files by disabling a specific rule. In this
case, `strictNullChecks`.
`strictNullChecks` is a good rule to start the migration with because
it's easy to silence with non-null assertions or optional chainings.
Additionally, almost all ts strict errors are due to this rule.
This PR improves 200+ files, leaving only 68 remaining to migrate to
strict mode in the payload package.
This PR makes it possible to do polymorphic join querying by fields that
don't exist in all collections specified in `field.collection`, for
example:
```
const result = await payload.find({
collection: 'payload-folders',
joins: {
documentsAndFolders: {
where: {
and: [
{
relationTo: {
in: ['folderPoly1', 'folderPoly2'],
},
},
{
folderPoly2Title: { // this field exists only in the folderPoly2 collection, before it'd throw a query error.
equals: 'Poly 2 Title',
},
},
],
},
},
},
})
```
---------
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
This PR adds an ability to specify a virtual field in this way
```js
{
slug: 'posts',
fields: [
{
name: 'title',
type: 'text',
required: true,
},
],
},
{
slug: 'virtual-relations',
fields: [
{
name: 'postTitle',
type: 'text',
virtual: 'post.title',
},
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
},
],
},
```
Then, every time you query `virtual-relations`, `postTitle` will be
automatically populated (even if using `depth: 0`) on the db level. This
field also, unlike `virtual: true` is available for querying / sorting /
`useAsTitle`.
Also, the field can be deeply nested to 2 or more relationships, for
example:
```
{
name: 'postCategoryTitle',
type: 'text',
virtual: 'post.category.title',
},
```
Where the current collection has `post` - a relationship to `posts`, the
collection `posts` has `category` that's a relationship to `categories`
and finally `categories` has `title`.
### What?
This PR adds support for `where` querying by the join field (don't
confuse with `where` querying of related docs via `joins.where`)
Previously, this didn't work:
```
const categories = await payload.find({
collection: 'categories',
where: { 'relatedPosts.title': { equals: 'my-title' } },
})
```
### Why?
This is crucial for bi-directional relationships, can be used for access
control.
### How?
Implements `where` handling for join fields the same as we do for
relationships. In MongoDB it's not as efficient as it can be, the old PR
that improves it and can be updated later is here
https://github.com/payloadcms/payload/pull/8858
Fixes https://github.com/payloadcms/payload/discussions/9683
Previously, querying by polymorphic joins `relationTo` with
`overrideAccess: false` caused an error:
```
QueryError: The following paths cannot be queried: relationTo
```
As this field actually doesn't exist in the schema. Now, under condition
that the query comes from a polymorphic join we skip checking
`relationTo` field access.
This adds support for running multiple job queue tasks in parallel
within the same workflow while preventing conflicts. Previously, this
would have caused the following issues:
- Job log entries get lost - the final job log is incomplete, despite
all tasks having been executed
- Write conflicts in postgres, leading to unique constraint violation
errors
The solution involves handling job log data updates in a way that avoids
overwriting, and ensuring the final update reflects the latest job log
data. Each job log entry now initializes its own ID, so a given job log
entry’s ID remains the same across multiple, parallel task executions.
## Postgres
In Postgres, we need to enable transactions for the
`payload.db.updateJobs` operation; otherwise, two tasks updating the
same job in parallel can conflict. This happens because Postgres handles
array rows by deleting them all, then re-inserting (rather than
upserting). The rows are stored in a separate table, and the following
scenario can occur:
Op 1: deletes all job log rows
Op 2: deletes all job log rows
Op 1: inserts 200 job log rows
Op 2: insert the same 200 job log rows again => `error: “duplicate key
value violates unique constraint "payload_jobs_log_pkey”`
Because transactions were not used, the rows inserted by Op 1
immediately became visible to Op 2, causing the conflict. Enabling
transactions fixes this. In theory, it can still happen if Op 1 commits
before Op 2 starts inserting (due to the read committed isolation
level), but it should occur far less frequently.
Alongside this change, we should consider inserting the rows using an
upsert (update on conflict), which will get rid of this error
completely. That way, if the insertion of Op 1 is visible to Op 2, Op 2
will simply overwrite it, rather than erroring. Individual job entries
are immutable and job entries cannot be deleted, thus this shouldn't
corrupt any data.
## Mongo
In Mongo, the issue is addressed by ensuring that log row deletions
caused due to different log states in concurrent operations are not
merged back to the client job log, and by making sure the final update
includes all job logs.
There is no duplicate key error in Mongo because the array log resides
in the same document and duplicates are simply upserted. We cannot use
transactions in Mongo, as it appears to lock the document in a way that
prevents reliable parallel updates, leading to:
`MongoServerError: WriteConflict error: this operation conflicted with
another operation. Please retry your operation or multi-document
transaction`
You can access the database name from `sanitizedConfig.db.name`. But
currently, it' not possible to access the db name from the unsanitized
config.
Plugins only have access to the unsanitized config. This change allows
db adapters to return the db name early, which will allow plugins to
conditionally initialize db-specific functionality
Continuation of #11489. This adds a new, optional `updateJobs` db
adapter method that reduces the amount of database calls for the jobs
queue.
## MongoDB
### Previous: running a set of 50 queued jobs
- 1x db.find (= 1x `Model.paginate`)
- 50x db.updateOne (= 50x `Model.findOneAndUpdate`)
### Now: running a set of 50 queued jobs
- 1x db.updateJobs (= 1x `Model.find` and 1x `Model.updateMany`)
**=> 51 db round trips before, 2 db round trips after**
### Previous: upon task completion
- 1x db.find (= 1x `Model.paginate`)
- 1x db.updateOne (= 1x `Model.findOneAndUpdate`)
### Now: upon task completion
- 1x db.updateJobs (= 1x `Model.findOneAndUpdate`)
**=> 2 db round trips before, 1 db round trip after**
## Drizzle (e.g. Postgres)
### running a set of 50 queued jobs
- 1x db.query[tablename].findMany
- 50x db.select
- 50x upsertRow
This is unaffected by this PR and will be addressed in a future PR
Previously, if you were querying a collection that has a join field with
`draft: true`, and the join field's collection also has
`versions.drafts: true` our db adapter would still query the original
SQL table / mongodb collection instead of the versions one which isn't
quite right since we respect `draft: true` when populating relationships
Fixes https://github.com/payloadcms/payload/issues/6884
Adds a new flag `acceptIDOnCreate` that allows you to thread your own
`id` to `payload.create` `data`, for example:
```ts
// doc created with id 1
const doc = await payload.create({ collection: 'posts', data: {id: 1, title: "my title"}})
```
```ts
import { Types } from 'mongoose'
const id = new Types.ObjectId().toHexString()
const doc = await payload.create({ collection: 'posts', data: {id, title: "my title"}})
```
This PR adds a new `limit` property to `payload.db.updateMany`. This functionality is required for [migrating our job system to use faster, direct db adapter calls](https://github.com/payloadcms/payload/pull/11489)
This PR adds a new `returning` option to various db adapter methods. Setting it to `false` where the return value is not used will lead to performance gains, as we don't have to do additional db calls to fetch the updated document and then sanitize it.
In the `findOne` db operation, we return `null` if the document was not
found.
For single-document delete and update operations, if the document you
wanted to update is not found, the following runtime error is thrown
instead: `Cannot read properties of null (reading '_id')`.
This PR correctly handles these cases and returns `null` from the db
method, just like the `findOne` operation.
This feature allows you to specify `collection` for the join field as
array.
This can be useful for example to describe relationship linking like
this:
```ts
{
slug: 'folders',
fields: [
{
type: 'join',
on: 'folder',
collection: ['files', 'documents', 'folders'],
name: 'children',
},
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
{
slug: 'files',
upload: true,
fields: [
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
{
slug: 'documents',
fields: [
{
type: 'relationship',
relationTo: 'folders',
name: 'folder',
},
],
},
```
Documents and files can be placed to folders and folders themselves can
be nested to other folders (root folders just have `folder` as `null`).
Output type of `Folder`:
```ts
export interface Folder {
id: string;
children?: {
docs?:
| (
| {
relationTo?: 'files';
value: string | File;
}
| {
relationTo?: 'documents';
value: string | Document;
}
| {
relationTo?: 'folders';
value: string | Folder;
}
)[]
| null;
hasNextPage?: boolean | null;
} | null;
folder?: (string | null) | Folder;
updatedAt: string;
createdAt: string;
}
```
While you could instead have many join fields (for example
`childrenFolders`, `childrenFiles`) etc - this doesn't allow you to
sort/filter and paginate things across many collections, which isn't
trivial. With SQL we use `UNION ALL` query to achieve that.
---------
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
This PR optimizes permissions calculation for block references, by
calculating them only once per block reference config, instead of once
every single time the blocks are referenced.
This will lead to significant performance improvements in Payload
Configs with a lot of duplicative block references, as permissions are
calculated every time you navigate from page to page.
# Benchmarks
Tested using `pnpm dev benchmark-blocks`.
## Before - ~ 6 seconds
https://github.com/user-attachments/assets/85cac698-3120-414f-91d3-608a404a3a5f
## After - ~ 2 seconds
https://github.com/user-attachments/assets/0c3642f6-6001-41ae-a7cd-f30b24362e9b
Currently, the join field outputs to its result `hasNextPage: boolean`
and have the `limit` query parameter but lacks `page` which can be
useful. This PR adds it.
Previously, we forgot to add `depth: 0` to our `findGlobal` call in `getEntityPolicies`. This PR adds `depth: 0` which will be faster.
It also cleans up the `getEntityPolicies` function in general by adding missing types, JSDocs and improving code readability.
This was part of https://github.com/payloadcms/payload/pull/11236 and has been extracted into this separate PR, to make it easier to review
Jessica Rynkar