Merge first: https://github.com/payloadcms/payload/pull/15177
Adds `supersedes` option to concurrency controls allowing newer jobs to
automatically delete older pending jobs with the same concurrency key.
This enables "last queued wins" behavior for scenarios where only the
latest state matters.
## Usage
```typescript
concurrency: {
key: ({ input }) => `generate:${input.documentId}`,
exclusive: true,
supersedes: true, // Newer jobs delete older pending ones (not yet completed and did not start processing yet)
}
```
**Example scenario:** User rapidly edits a document triggering 5
regeneration jobs. Only the last job runs - intermediate pending jobs
are automatically deleted.
## Important
Only **pending** jobs are deleted. Running jobs complete normally and
the new job waits.
Adds exclusive concurrency controls to prevent race conditions when
multiple jobs operate on the same resource. Jobs with the same
concurrency key will not run in parallel.
## Key Changes
- Added `concurrency` option to workflows and tasks for defining
resource keys
- Added `enableConcurrencyControl` feature flag (defaults to `false`,
will be `true` in 4.0)
- Added indexed `concurrencyKey` field to jobs collection when
`enableConcurrencyControl` is set to `true`
## Usage Example
```typescript
export default buildConfig({
jobs: {
enableConcurrencyControl: true,
workflows: [{
slug: 'syncDocument',
concurrency: ({ input }) => `sync:${input.documentId}`,
handler: async ({ job }) => {
// Only one job per documentId runs at a time
}
}]
}
})
```
Adds access control for jobs `queue` and `cancel` operations, similar to
what we have for `run` operations.
**Implementation:**
Both operations now support `overrideAccess` parameter (defaults to
`true`) and respect `jobsConfig.access.queue` and
`jobsConfig.access.cancel` functions. When `overrideAccess: false` is
passed without an authenticated request, operations throw `Forbidden`
error.
**Configuration:**
Access control functions can be defined in the Jobs Config under
`jobs.access.queue`, `jobs.access.run`, and `jobs.access.cancel`. Each
function receives `{ req }` and returns a boolean. If no custom access
control is defined, the default allows any authenticated user to perform
the operation.
**Example:**
```ts
// Configure access control
jobs: {
access: {
queue: ({ req }) => req.user?.roles?.includes('admin'),
cancel: ({ req }) => req.user?.roles?.includes('admin'),
}
}
// Use in Local API
await payload.jobs.cancel({
where: { workflowSlug: { equals: 'sync' } },
overrideAccess: false,
req,
})
```
By default, `payload.jobs.run` only runs jobs from the `default` queue
(since https://github.com/payloadcms/payload/pull/12799). It exposes an
`allQueues` property to run jobs from all queues.
For handling schedules (`payload.jobs.handleSchedules` and
`config.jobs.autoRun`), this behaves differently - jobs are run from all
queues by default, and no `allQueues` property exists.
This PR adds an `allQueues` property to scheduling, as well as changes
the default behavior to only handle schedules for the `default` queue.
That way, the behavior of running and scheduling jobs matches.
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210982048221260
Adds a new `schedule` property to workflow and task configs that can be
used to have Payload automatically _queue_ jobs following a certain
_schedule_.
Docs:
https://payloadcms.com/docs/dynamic/jobs-queue/schedules?branch=feat/schedule-jobs
## API Example
```ts
export default buildConfig({
// ...
jobs: {
// ...
scheduler: 'manual', // Or `cron` if you're not using serverless. If `manual` is used, then user needs to set up running /api/payload-jobs/handleSchedules or payload.jobs.handleSchedules in regular intervals
tasks: [
{
schedule: [
{
cron: '* * * * * *',
queue: 'autorunSecond',
// Hooks are optional
hooks: {
// Not an array, as providing and calling `defaultBeforeSchedule` would be more error-prone if this was an array
beforeSchedule: async (args) => {
// Handles verifying that there are no jobs already scheduled or processing.
// You can override this behavior by not calling defaultBeforeSchedule, e.g. if you wanted
// to allow a maximum of 3 scheduled jobs in the queue instead of 1, or add any additional conditions
const result = await args.defaultBeforeSchedule(args)
return {
...result,
input: {
message: 'This task runs every second',
},
}
},
afterSchedule: async (args) => {
await args.defaultAfterSchedule(args) // Handles updating the payload-jobs-stats global
args.req.payload.logger.info(
'EverySecond task scheduled: ' +
(args.status === 'success' ? args.job.id : 'skipped or failed to schedule'),
)
},
},
},
],
slug: 'EverySecond',
inputSchema: [
{
name: 'message',
type: 'text',
required: true,
},
],
handler: ({ input, req }) => {
req.payload.logger.info(input.message)
return {
output: {},
}
},
}
]
}
})
```
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1210495300843759
Previously, there were multiple ways to type a running job:
- `GeneratedTypes['payload-jobs']` - only works in an installed project
- is `any` in monorepo
- `BaseJob` - works everywhere, but does not incorporate generated types
which may include type for custom fields added to the jobs collection
- `RunningJob<>` - more accurate version of `BaseJob`, but same problem
This PR deprecated all those types in favor of a new `Job` type.
Benefits:
- Works in both monorepo and installed projects. If no generated types
exist, it will automatically fall back to `BaseJob`
- Comes with an optional generic that can be used to narrow down
`job.input` based on the task / workflow slug. No need to use a separate
type helper like `RunningJob<>`
With this new type, I was able to replace every usage of
`GeneratedTypes['payload-jobs']`, `BaseJob` and `RunningJob<>` with the
simple `Job` type.
Additionally, this PR simplifies some of the logic used to run jobs
By default, calling `payload.jobs.run()` will incorrectly run all jobs
from all queues. The `npx payload jobs:run` bin script behaves the same
way.
The `payload-jobs/run` endpoint runs jobs from the `default` queue,
which is the correct behavior.
This PR does the following:
- Change `payload.jobs.run()` to only runs jobs from the `default` queue
by default
- Change `npx payload jobs:run` bin script to only runs jobs from the
`default` queue by default
- Add new allQueues / --all-queues arguments/queryparams/flags for the
local API, rest API and bin script to allow you to run all jobs from all
queues
- Clarify the docs
Previously, jobs were executed in FIFO order on MongoDB, and LIFO on
Postgres, with no way to configure this behavior.
This PR makes FIFO the default on both MongoDB and Postgres and
introduces the following new options to configure the processing order
globally or on a queue-by-queue basis:
- a `processingOrder` property to the jobs config
- a `processingOrder` argument to `payload.jobs.run()` to override
what's set in the jobs config
It also adds a new `sequential` option to `payload.jobs.run()`, which
can be useful for debugging.
Previously, our job queue system relied on `payload.*` operations, which
ran very frequently:
- whenever job execution starts, as all jobs need to be set to
`processing: true`
- every single time a task completes or fails, as the job log needs to
be updated
- whenever job execution stops, to mark it as completed and to delete it
(if `deleteJobOnComplete` is set)
This PR replaces these with direct `payload.db.*` calls, which are
significantly faster than payload operations. Given how often the job
queue system communicates with the database, this should be a massive
performance improvement.
## How it affects running hooks
To generate the task status, we previously used an `afterRead` hook.
Since direct db adapter calls no longer execute hooks, this PR
introduces new `updateJob` and `updateJobs` helpers to handle task
status generation outside the normal payload hook lifecycle.
Additionally, a new `runHooks` property has been added to the global job
configuration. While setting this to `true` can be useful if custom
hooks were added to the `payload-jobs` collection config, this will
revert the job system to use normal payload operations.
This should be avoided as it degrades performance. In most cases, the
`onSuccess` or `onFail` properties in the job config will be sufficient
and much faster.
Furthermore, if the `depth` property is set in the global job
configuration, the job queue system will also fall back to the slower,
normal payload operations.
---------
Co-authored-by: Dan Ribbens <DanRibbens@users.noreply.github.com>
Cleans up various Payload-generated configs, namely:
- Renames config entry files from `preferencesCollection.ts`,
`lockedDocumentsCollection.ts`, and `jobsCollection.ts` to `config.ts`
- Standardizes collection slugs for `payload-preferences`,
`payload-locked-documents`, and `payload-jobs` and reuses everywhere
- Renames camel-cased `payloadPreferences` directory to kebab case, i.e.
`payload-preferences`
This adds new `payload.jobs.cancel` and `payload.jobs.cancelByID` methods that allow you to cancel already-running jobs, or prevent queued jobs from running.
While it's not possible to cancel a function mid-execution, this will stop job execution the next time the job makes a request to the db, which happens after every task.
### What?
Implement the
[typescript-strict-plugin](https://github.com/allegro/typescript-strict-plugin)
plugin in the payload (core) package.
### Why?
1. One strategy for incremental migration is to enable strictness rules
in tsconfig, fix some errors, and push them without committing the
changes to tsconfig.json. However, this is not feasible for a package as
large as Payload that has over 1000 typescript errors. Until the work is
done, new contributions would undo the work being done.
2. Even if no migration work is done after this PR, this change already
improves the strictness of the package. 89 of the 311 files within the
package already satisfy strict mode. This PR only adds a comment
`@ts-strict-ignore` to files that had at least one compilation error.
This way, the propagation of errors in those files is stopped.
3. New files created in the package are strict by default (this was the
main improvement in version 2 of `typescript-strict-plugin`).
I recommend starting the migration with this package because it is the
one that almost all the others depend on. Once we finish this package,
we can repeat the same strategy on another one, or use the strategy I
mentioned in point 1 if the package is small.
### Note
If you don't see errors in the IDE when you uncomment `//
@ts-strict-ignore`, try restarting the typescript server or VSCode
### How to contribute to the migration ❤️
1. Remove `// @ts-strict-ignore` comments from 1 or more files
2. Fix the pending errors (they should appear in your IDE's intellisense
or when running `cd packages/payload` + `pnpm build:types`
3. Submit your PR!
Important: You don't need to fix everything at once! Furthermore, I
recommend breaking this down into very small PRs to trace potential
issues later if there are any. So if you have 5 minutes, tackle a small
file—every bit counts! 🤗
Currently, custom components do not respect `admin.condition` unless
manually wrapped with the `withCondition` HOC, like all default fields
currently do. This should not be a requirement of component authors.
Instead, we can automatically detect custom client and server fields and
wrap them with the underlying `WatchCondition` component which will
subscribe to the `passesCondition` property within client-side form
state.
For my future self: there are potentially multiple instances where
fields subscribe to conditions duplicately, such as when rendering a
default Payload field within a custom field component. This was always a
problem and it is non-breaking, but needs to be reevaluated and removed
in the future for performance. Only the default fields that Payload
renders client-side need to subscribe to field conditions in this way.
When importing a Payload field into your custom field component, for
example, it should not include the HOC, because custom components now
watch conditions themselves.
Adds a jobs queue to Payload.
- [x] Docs, w/ examples for Vercel Cron, additional services
- [x] Type the `job` using GeneratedTypes in `JobRunnerArgs`
(@AlessioGr)
- [x] Write the `runJobs` function
- [x] Allow for some type of `payload.runTask`
- [x] Open up a new bin script for running jobs
- [x] Determine strategy for runner endpoint to either await jobs
successfully or return early and stay open until job work completes
(serverless ramifications here)
- [x] Allow for job runner to accept how many jobs to run in one
invocation
- [x] Make a Payload local API method for creating a new job easily
(payload.createJob) or similar which is strongly typed (@AlessioGr)
- [x] Make `payload.runJobs` or similar (@AlessioGr)
- [x] Write tests for retrying up to max retries for a given step
- [x] Write tests for dynamic import of a runner
The shape of the config should permit the definition of steps separate
from the job workflows themselves.
```js
const config = {
// Not sure if we need this property anymore
queues: {
},
// A job is an instance of a workflow, stored in DB
// and triggered by something at some point
jobs: {
// Be able to override the jobs collection
collectionOverrides: () => {},
// Workflows are groups of tasks that handle
// the flow from task to task.
// When defined on the config, they are considered as predefined workflows
// BUT - in the future, we'll allow for UI-based workflow definition as well.
workflows: [
{
slug: 'job-name',
// Temporary name for this
// should be able to pass function
// or path to it for Node to dynamically import
controlFlowInJS: '/my-runner.js',
// Temporary name as well
// should be able to eventually define workflows
// in UI (meaning they need to be serialized in JSON)
// Should not be able to define both control flows
controlFlowInJSON: [
{
task: 'myTask',
next: {
// etc
}
}
],
// Workflows take input
// which are a group of fields
input: [
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
maxDepth: 0,
required: true,
},
{
name: 'message',
type: 'text',
required: true,
},
],
},
],
// Tasks are defined separately as isolated functions
// that can be retried on fail
tasks: [
{
slug: 'myTask',
retries: 2,
// Each task takes input
// Used to auto-type the task func args
input: [
{
name: 'post',
type: 'relationship',
relationTo: 'posts',
maxDepth: 0,
required: true,
},
{
name: 'message',
type: 'text',
required: true,
},
],
// Each task takes output
// Used to auto-type the function signature
output: [
{
name: 'success',
type: 'checkbox',
}
],
onSuccess: () => {},
onFail: () => {},
run: myRunner,
},
]
}
}
```
### `payload.createJob`
This function should allow for the creation of jobs based on either a
workflow (group of tasks) or an individual task.
To create a job using a workflow:
```js
const job = await payload.createJob({
// Accept the `name` of a workflow so we can match to either a
// code-based workflow OR a workflow defined in the DB
// Should auto-type the input
workflowName: 'myWorkflow',
input: {
// typed to the args of the workflow by name
}
})
```
To create a job using a task:
```js
const job = await payload.createJob({
// Accept the `name` of a task
task: 'myTask',
input: {
// typed to the args of the task by name
}
})
```
---------
Co-authored-by: Alessio Gravili <alessio@gravili.de>
Co-authored-by: Dan Ribbens <dan.ribbens@gmail.com>
Alessio Gravili