Misc input validation improvements, sanitizing path segments in both SQL
and JSON queries, standardizing the processing of column and JSON paths
across different adapters, and making adjustments to traversal and alias
generation to align behavior across components.
### What
Ensures unknown query operators are properly rejected during query
validation.
### Why
Previously, unrecognized operators in where clauses were silently
ignored. This could lead to unexpected behavior. Query validation should
fail-closed and only accept known operators.
### How
Added an `else` in validateQueryPaths.ts to push an error for any
operator not in the valid operator set.
Test added to `joins > int`
In large projects, calculating the `permissions` object is one of the
slowest parts of Payload. This PR completely rewrites the permission
calculation system to make it significantly faster and more reliable.
The permissions object is calculated multiple times throughout Payload's
lifecycle: for the entire config every time we load or navigate to any
admin panel page, on every API endpoint call, and as part of query path
validation when calling any Payload operation.
This is done by `getAccessResults` without any document data, and in
large configs can be one of the slowest operations, noticeably slowing
down admin panel navigation.
After this rewrite, **speed improvements in permissions calculation
without data range from 5.3% (access-control test suite) to 74% (fields
test suite - our largest config)**. The larger the config, the more
noticeable these improvements become. These performance gains apply to
both the Payload Admin Panel and API requests.
---
The `permissions` object can also be calculated WITH document data,
which triggers evaluation of `Where` conditions in
collection/global-level access control and may fetch the document to
pass to access control functions.
This has been heavily optimized with **speed improvements for
permissions calculation with data ranging between 75% and 5138%** (if I
craft a collection config that excessively uses aspects this PR
improves) in the benchmarks. This optimization affects:
- Loading any document (collection/global edit view or drawer)
- Saving a document
- Loading the list view with query presets applied
- Bulk upload initialization (runs twice)
These improvements are most noticeable when navigating _to_ documents in
the admin panel.
## What's Fixed
In addition to performance improvements, the new function is now
cleaner, easier to understand and works more reliably. I specifically
wanna call out two issues that were fixed:
**Field access control data bug**: Added a new e2e test that was
previously failing - when saving a document, the data object passed to
field-level update access control functions had the wrong shape (or in
some cases did not exist at all). This caused fields to show incorrect
`readOnly` states after save. The new implementation properly passes
document data through all access control checks.
**Where query validation**: The old implementation sometimes skipped
validating `Where` queries for collections when `req.data` existed,
potentially granting incorrect access. The new version always validates
where queries correctly when `fetchData: true`.
**Global Where queries:**: Where queries were _never_ executed for
globals. Instead, [we were just checking if the global existed in
general](https://github.com/payloadcms/payload/blob/main/packages/payload/src/utilities/getEntityPolicies.ts#L59).
This PR ensures that `Where` queries returned from global access control
are respected.
## What's Faster (and Why)
**Where query caching:** When multiple operations return identical where
queries (common pattern), we now cache the result. Instead of 3-4
separate DB calls, we make 1 call and reuse it.
**Parallel execution:** All `Where` query evaluations and async access
control functions (for both collections and fields) now execute
concurrently with maximum parallelism, rather than sequentially.
**Optimized DB operations**: When evaluating returned `Where` queries,
we now use direct database `db.count()` queries instead of
`payload.find()` operations. This is much faster (especially on
Postgres).
**Synchronous tree traversal**: The entire field permission tree is now
built synchronously with all async work collected and executed in
parallel at the end, rather than cascading await calls through each
nesting level. This eliminates sequential bottlenecks in deeply nested
field structures.
## Benchmarks
### Admin Panel
- Fields test suite with access control added
- 1000 blocks added to the blocks collection. This is not as excessive
as you would expect:
- most Payload apps do not run on an M3 Max. CPU/DB is often much, much
slower
- this is all local, with a local DB
- a lot of projects accumulate a huge amount of blocks if you multiply
blocks x block fields. They can definitely reach 1000 total blocks
- each block is simple - only one text field per block. In real
projects, blocks are usually a lot more complex
**Before:**
https://github.com/user-attachments/assets/37b55f02-6dbc-4005-9da6-d7cd4bfdc925
**After:**
https://github.com/user-attachments/assets/8dd796dd-66ed-4d32-a688-27b4169577c6
Branch:
https://github.com/payloadcms/payload/tree/fix/update-field-access-control-after-save-benchmarks
### Modified Access-Control Test Suite
`cd test && pnpm payload run access-control/benchmark-permissions.ts`
```md
📊 Benchmark 1: getAccessResults (all collections + globals)
──────────────────────────────────────────────────────────────────────
┌─────────┬───────────────────┬─────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼───────────────────┼─────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (optimized)' │ '16.91' │ '59203.986' │ '±0.52%' │
│ 1 │ 'OLD (previous)' │ '16.07' │ '62323.953' │ '±0.57%' │
└─────────┴───────────────────┴─────────┴───────────────────┴──────────┘
⚡ Speedup: +5.3% 🚀📊 DB Calls per operation (across all collections + globals):
NEW: 0.0 total (0.0 data, 0.0 where)
OLD: 0.0 total (0.0 data, 0.0 where)
📊 Benchmark 2: docAccessOperation (with fetchData)
──────────────────────────────────────────────────────────────────────
Collection: where-cache-same (same where queries)
┌─────────┬────────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (with cache)' │ '2272.81' │ '442.319' │ '±0.11%' │
│ 1 │ 'OLD (no cache)' │ '1019.72' │ '983.347' │ '±0.12%' │
└─────────┴────────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +122.9% 🚀📊 DB Calls per operation:
NEW: 2.0 total (1.0 data, 1.0 where)
OLD: 4.0 total (1.0 data, 3.0 where)
📊 Benchmark 3: docAccessOperation (with data passed)
──────────────────────────────────────────────────────────────────────
Collection: where-cache-same (same where queries, no DB fetch)
┌─────────┬────────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (with cache)' │ '4945.43' │ '203.252' │ '±0.11%' │
│ 1 │ 'OLD (no cache)' │ '1023.75' │ '979.898' │ '±0.14%' │
└─────────┴────────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +383.1% 🚀📊 DB Calls per operation:
NEW: 1.0 total (0.0 data, 1.0 where)
OLD: 4.0 total (1.0 data, 3.0 where)
📊 Benchmark 4: docAccessOperation (unique where queries)
──────────────────────────────────────────────────────────────────────
Collection: where-cache-unique (unique where queries per operation)
┌─────────┬────────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (parallel)' │ '1693.67' │ '599.091' │ '±0.29%' │
│ 1 │ 'OLD (sequential)' │ '970.37' │ '1036.268' │ '±0.20%' │
└─────────┴────────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +74.5% 🚀📊 DB Calls per operation:
NEW: 4.0 total (1.0 data, 3.0 where)
OLD: 4.0 total (1.0 data, 3.0 where)
📊 Benchmark 5: Complex Collection (async access, nested blocks, field access)
──────────────────────────────────────────────────────────────────────
Collection: complex-content (stress test)
┌─────────┬────────────────────┬──────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼──────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (optimized)' │ '133.17' │ '7683.346' │ '±0.83%' │
│ 1 │ 'OLD (sequential)' │ '72.86' │ '13901.616' │ '±0.82%' │
└─────────┴────────────────────┴──────────┴───────────────────┴──────────┘
⚡ Speedup: +82.8% 🚀📊 DB Calls per operation:
NEW: 2.0 total (1.0 data, 1.0 where)
OLD: 2.0 total (1.0 data, 1.0 where)
📊 Benchmark 6: Sync-Heavy Collection (same where, many sync field access)
──────────────────────────────────────────────────────────────────────
Collection: sync-heavy (where cache + field access)
┌─────────┬────────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼────────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (with cache)' │ '3607.72' │ '279.559' │ '±0.15%' │
│ 1 │ 'OLD (no cache)' │ '68.87' │ '15136.557' │ '±1.44%' │
└─────────┴────────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +5138.5% 🚀📊 DB Calls per operation:
NEW: 1.0 total (0.0 data, 1.0 where)
OLD: 5.0 total (1.0 data, 4.0 where)
══════════════════════════════════════════════════════════════════════
📈 Summary:
══════════════════════════════════════════════════════════════════════
1. getAccessResults: (see above)
2. docAccessOperation (with fetchData): (see above)
3. docAccessOperation (with data passed): (see above)
4. docAccessOperation (unique where): (see above)
5. Complex collection (async + nested): (see above)
6. Sync-heavy (where cache + fields): (see above)
══════════════════════════════════════════════════════════════════════
```
### Fields Test Suite
`cd test && pnpm payload run fields/benchmark-getAccessResults.ts`
```md
📊 Benchmark: getAccessResults (all fields test collections + globals)
──────────────────────────────────────────────────────────────────────
┌─────────┬───────────────────┬───────────┬───────────────────┬──────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ms) │ Margin │
├─────────┼───────────────────┼───────────┼───────────────────┼──────────┤
│ 0 │ 'NEW (optimized)' │ '2104.06' │ '478.382' │ '±0.18%' │
│ 1 │ 'OLD (previous)' │ '1208.14' │ '834.974' │ '±0.21%' │
└─────────┴───────────────────┴───────────┴───────────────────┴──────────┘
⚡ Speedup: +74.2% 🚀📊 DB Calls per operation (across all collections + globals):
NEW: 0.0 total (0.0 data, 0.0 where)
OLD: 0.0 total (0.0 data, 0.0 where)
```
Catches list filter errors and prevents the list view from crashing when
attempting to search on fields the user does not have access to. Instead
just shows the default "no results found" message.
Fixes an issue when querying deeply new relationship virtual fields with
`draft: true`. Changes the method for `where` sanitization, before it
was done in `validateSearchParam` which didn't work with versions
properly, now there's a separate `sanitizeWhereQuery` function that does
this.
Important: An intentional effort is being made during migration to not
modify runtime behavior. This implies that there will be several
assertions, non-null assertions, and @ts-expect-error. This philosophy
applies only to migrating old code to TypeScript strict, not to writing
new code. For a more detailed justification for this reasoning, see
#11840 (comment).
In this PR, instead of following the approach of migrating a subset of
files, I'm migrating all files by disabling specific rules. The first
commits are named after the rule being disabled.
With this PR, the migration of the payload package is complete 🚀
Important: An intentional effort is being made during migration to not
modify runtime behavior. This implies that there will be several
assertions, non-null assertions, and @ts-expect-error. This philosophy
applies only to migrating old code to TypeScript strict, not to writing
new code. For a more detailed justification for this reasoning,
https://github.com/payloadcms/payload/pull/11840#discussion_r2021975897.
In this PR, instead of following the approach of migrating a subset of
files, I'm migrating all files by disabling a specific rule. In this
case, `strictNullChecks`.
`strictNullChecks` is a good rule to start the migration with because
it's easy to silence with non-null assertions or optional chainings.
Additionally, almost all ts strict errors are due to this rule.
This PR improves 200+ files, leaving only 68 remaining to migrate to
strict mode in the payload package.
This PR makes it possible to do polymorphic join querying by fields that
don't exist in all collections specified in `field.collection`, for
example:
```
const result = await payload.find({
collection: 'payload-folders',
joins: {
documentsAndFolders: {
where: {
and: [
{
relationTo: {
in: ['folderPoly1', 'folderPoly2'],
},
},
{
folderPoly2Title: { // this field exists only in the folderPoly2 collection, before it'd throw a query error.
equals: 'Poly 2 Title',
},
},
],
},
},
},
})
```
---------
Co-authored-by: Jarrod Flesch <jarrodmflesch@gmail.com>
This PR adds an ability to specify a virtual field in this way
```js
{
slug: 'posts',
fields: [
{
name: 'title',
type: 'text',
required: true,
},
],
},
{
slug: 'virtual-relations',
fields: [
{
name: 'postTitle',
type: 'text',
virtual: 'post.title',
},
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
},
],
},
```
Then, every time you query `virtual-relations`, `postTitle` will be
automatically populated (even if using `depth: 0`) on the db level. This
field also, unlike `virtual: true` is available for querying / sorting /
`useAsTitle`.
Also, the field can be deeply nested to 2 or more relationships, for
example:
```
{
name: 'postCategoryTitle',
type: 'text',
virtual: 'post.category.title',
},
```
Where the current collection has `post` - a relationship to `posts`, the
collection `posts` has `category` that's a relationship to `categories`
and finally `categories` has `title`.
Previously, querying by polymorphic joins `relationTo` with
`overrideAccess: false` caused an error:
```
QueryError: The following paths cannot be queried: relationTo
```
As this field actually doesn't exist in the schema. Now, under condition
that the query comes from a polymorphic join we skip checking
`relationTo` field access.
This PR optimizes permissions calculation for block references, by
calculating them only once per block reference config, instead of once
every single time the blocks are referenced.
This will lead to significant performance improvements in Payload
Configs with a lot of duplicative block references, as permissions are
calculated every time you navigate from page to page.
# Benchmarks
Tested using `pnpm dev benchmark-blocks`.
## Before - ~ 6 seconds
https://github.com/user-attachments/assets/85cac698-3120-414f-91d3-608a404a3a5f
## After - ~ 2 seconds
https://github.com/user-attachments/assets/0c3642f6-6001-41ae-a7cd-f30b24362e9b
The `localized` properly was not stripped out of referenced block fields, if any parent was localized. For normal fields, this is done in sanitizeConfig. As the same referenced block config can be used in both a localized and non-localized config, we are not able to strip it out inside sanitizeConfig by modifying the block config.
Instead, this PR had to bring back tedious logic to handle it everywhere the `field.localized` property is accessed. For backwards-compatibility, we need to keep the existing sanitizeConfig logic. In 4.0, we should remove it to benefit from better test coverage of runtime field.localized handling - for now, this is done for our test suite using the `PAYLOAD_DO_NOT_SANITIZE_LOCALIZED_PROPERTY` flag.
If you have multiple blocks that are used in multiple places, this can quickly blow up the size of your Payload Config. This will incur a performance hit, as more data is
1. sent to the client (=> bloated `ClientConfig` and large initial html) and
2. processed on the server (permissions are calculated every single time you navigate to a page - this iterates through all blocks you have defined, even if they're duplicative)
This can be optimized by defining your block **once** in your Payload Config, and just referencing the block slug whenever it's used, instead of passing the entire block config. To do this, the block can be defined in the `blocks` array of the Payload Config. The slug can then be passed to the `blockReferences` array in the Blocks Field - the `blocks` array has to be empty for compatibility reasons.
```ts
import { buildConfig } from 'payload'
import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'
// Payload Config
const config = buildConfig({
// Define the block once
blocks: [
{
slug: 'TextBlock',
fields: [
{
name: 'text',
type: 'text',
},
],
},
],
collections: [
{
slug: 'collection1',
fields: [
{
name: 'content',
type: 'blocks',
// Reference the block by slug
blockReferences: ['TextBlock'],
blocks: [], // Required to be empty, for compatibility reasons
},
],
},
{
slug: 'collection2',
fields: [
{
name: 'editor',
type: 'richText',
editor: lexicalEditor({
BlocksFeature({
// Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
blocks: ['TextBlock'],
})
})
},
],
},
],
})
```
## v4.0 Plans
In 4.0, we will remove the `blockReferences` property, and allow string block references to be passed directly to the blocks `property`. Essentially, we'd remove the `blocks` property and rename `blockReferences` to `blocks`.
The reason we opted to a new property in this PR is to avoid breaking changes. Allowing strings to be passed to the `blocks` property will prevent plugins that iterate through fields / blocks from compiling.
## PR Changes
- Testing: This PR introduces a plugin that automatically converts blocks to block references. This is done in the fields__blocks test suite, to run our existing test suite using block references.
- Block References support: Most changes are similar. Everywhere we iterate through blocks, we have to now do the following:
1. Check if `field.blockReferences` is provided. If so, only iterate through that.
2. Check if the block is an object (= actual block), or string
3. If it's a string, pull the actual block from the Payload Config or from `payload.blocks`.
The exception is config sanitization and block type generations. This PR optimizes them so that each block is only handled once, instead of every time the block is referenced.
## Benchmarks
60 Block fields, each block field having the same 600 Blocks.
### Before:
**Initial HTML:** 195 kB
**Generated types:** takes 11 minutes, 461,209 lines
https://github.com/user-attachments/assets/11d49a4e-5414-4579-8050-e6346e552f56
### After:
**Initial HTML:** 73.6 kB
**Generated types:** takes 2 seconds, 35,810 lines
https://github.com/user-attachments/assets/3eab1a99-6c29-489d-add5-698df67780a3
### After Permissions Optimization (follow-up PR)
Initial HTML: 73.6 kB
https://github.com/user-attachments/assets/a909202e-45a8-4bf6-9a38-8c85813f1312
## Future Plans
1. This PR does not yet deduplicate block references during permissions calculation. We'll optimize that in a separate PR, as this one is already large enough
2. The same optimization can be done to deduplicate fields. One common use-case would be link field groups that may be referenced in multiple entities, outside of blocks. We might explore adding a new `fieldReferences` property, that allows you to reference those same `config.blocks`.
### What?
Implement the
[typescript-strict-plugin](https://github.com/allegro/typescript-strict-plugin)
plugin in the payload (core) package.
### Why?
1. One strategy for incremental migration is to enable strictness rules
in tsconfig, fix some errors, and push them without committing the
changes to tsconfig.json. However, this is not feasible for a package as
large as Payload that has over 1000 typescript errors. Until the work is
done, new contributions would undo the work being done.
2. Even if no migration work is done after this PR, this change already
improves the strictness of the package. 89 of the 311 files within the
package already satisfy strict mode. This PR only adds a comment
`@ts-strict-ignore` to files that had at least one compilation error.
This way, the propagation of errors in those files is stopped.
3. New files created in the package are strict by default (this was the
main improvement in version 2 of `typescript-strict-plugin`).
I recommend starting the migration with this package because it is the
one that almost all the others depend on. Once we finish this package,
we can repeat the same strategy on another one, or use the strategy I
mentioned in point 1 if the package is small.
### Note
If you don't see errors in the IDE when you uncomment `//
@ts-strict-ignore`, try restarting the typescript server or VSCode
### How to contribute to the migration ❤️
1. Remove `// @ts-strict-ignore` comments from 1 or more files
2. Fix the pending errors (they should appear in your IDE's intellisense
or when running `cd packages/payload` + `pnpm build:types`
3. Submit your PR!
Important: You don't need to fix everything at once! Furthermore, I
recommend breaking this down into very small PRs to trace potential
issues later if there are any. So if you have 5 minutes, tackle a small
file—every bit counts! 🤗
- reduces unnecessary shallow copying within operations by removing
unnecessary spreads or .map()'s
- removes unnecessary `deleteMany` call in `deleteUserPreferences` for
auth-enabled collections
- replaces all instances of `validOperators.includes` with
`validOperatorMap[]`. O(n) => O(1)
- optimizes the `sanitizeInternalFields` function. Previously, it was
doing a **lot** of shallow copying
### What?
Fixes the issue with querying by `id` from REST / `overrideAccess:
false`.
For example, this didn't work:
`/api/loans?where[book.bibliography.id][equals]=67224d74257b3f2acddc75f4`
```
QueryError: The following path cannot be queried: id
```
### Why?
We support this syntax within the Local API.
### How?
Now, for simplicity we sanitize everything like
`relation.otherRelation.id` to `relation.otherRelation`
Fixes https://github.com/payloadcms/payload/issues/9008
## Description
Adds `virtual` property to the fields config. Providing `true`
completely disables the field in the DB, which is useful for [Virtual
Fields](https://payloadcms.com/blog/learn-how-virtual-fields-can-help-solve-common-cms-challenges)
Disables abillity to query by a field with `virtual: true`.
Currently, they bloat the DB with unused tables / columns, which may as
well introduce additional joins.
Discussion https://github.com/payloadcms/payload/discussions/6270
Prev PR (this one contains only this feature):
https://github.com/payloadcms/payload/pull/6983
- [x] I have read and understand the
[CONTRIBUTING.md](https://github.com/payloadcms/payload/blob/main/CONTRIBUTING.md)
document in this repository.
## Type of change
<!-- Please delete options that are not relevant. -->
- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update
## Checklist:
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] Existing test suite passes locally with my changes
- [x] I have made corresponding changes to the documentation
Supports `hasMany` upload fields, similar to how `hasMany` works in
other fields, i.e.:
```ts
{
type: 'upload',
relationTo: 'media',
hasMany: true
}
```
---------
Co-authored-by: Jacob Fletcher <jacobsfletch@gmail.com>
Co-authored-by: James <james@trbl.design>
Removes PayloadRequestWithData in favour of just PayloadRequest with
optional types for `data` and `locale`
`addDataAndFileToRequest` and `addLocalesToRequestFromData` now takes in
a single argument instead of an object
```ts
// before
await addDataAndFileToRequest({ request: req })
addLocalesToRequestFromData({ request: req })
// current
await addDataAndFileToRequest(req)
addLocalesToRequestFromData(req)
```
---------
Co-authored-by: Paul Popus <paul@nouance.io>
Jessica Rynkar