Background Jobs

Queue Across Servers

Share one @axiljs/queue between processes and servers with SqlQueueBackend on PostgreSQL, MySQL 8 or SQLite: atomic claims with FOR UPDATE SKIP LOCKED, heartbeats, stale recovery and fencing.

8 min readUpdated DocumentationEdit this page
On this page16

InMemoryQueueStore and FileQueueStore belong to one process. To let several processes or servers share one queue without Redis, give the queue a backend instead of a store. SqlQueueBackend keeps jobs in a table in PostgreSQL, MySQL 8 or SQLite, and hands each job to exactly one worker.

Note

New in 0.3.5: the backend option, SqlQueueBackend, InMemoryQueueBackend and the async accessors.

Usage

TypeScript
TS
1import { Queue, SqlQueueBackend } from '@axiljs/queue'
2import { createConnection } from '@axiljs/orm'
3
4const conn = await createConnection({ driver: 'postgres', url: process.env.DATABASE_URL })
5const driver = conn.getDriver()
6
7const backend = new SqlQueueBackend({
8  table: 'axil_jobs', // default; created on first use
9  executor: {
10    dialect: 'postgres', // the same value as `driver` in the connection config
11    query: (sql, params) => driver.query(sql, params),
12    transaction: (fn) => driver.transaction((trx) => fn(trx)) // used by mysql only
13  }
14})
15
16const queue = new Queue('emails', { backend, concurrency: 5, pollIntervalMs: 500 })
17queue.process('send-welcome', async (job) => sendEmail(job.data.to))
18queue.start()

Run the same code on every server. Each job is processed by exactly one of them.

If you already use @axiljs/stores, its fromConnection(db) executor has the same shape and can be passed directly:

TypeScript
TS
import { fromConnection } from '@axiljs/stores'

const queue = new Queue('jobs', { backend: new SqlQueueBackend({ executor: fromConnection(db) }) })

Warning

Register a handler for every job name on every server that runs the queue. A server that claims a job it has no handler for marks it failed with No handler registered for job '<name>'.

Options

SqlQueueBackend

OptionDefaultDescription
executorrequiredRuns parameterized SQL (see below)
table'axil_jobs'Letters, digits and underscores, optionally schema-qualified (jobs.axil_jobs)
autoCreatetrueCreate the table and indexes on first use

The executor

MemberRequiredDescription
dialectyes'postgres' | 'mysql' | 'sqlite'
query(sql, params?)yesResolve with { rows, affectedRows? }. SQL is written with ? placeholders and rewritten to $1, $2, ... for postgres
transaction(fn)noRun fn(trx) in one transaction on one connection, committing on success and rolling back on error. Used by MySQL only

On MySQL, which has no RETURNING, the executor must report affectedRows for writes. An executor that does not report it cannot fence: the backend then assumes the worker still owns the job. @axiljs/orm drivers report it.

Queue options in backend mode

OptionDefaultDescription
backendA QueueBackend. Mutually exclusive with store; passing both throws
workerIdhost:pid:randomThis worker's lock owner id
visibilityTimeoutMs30000An active job whose lock has not been renewed for this long is treated as abandoned and returned to pending
concurrency, pollIntervalMs, backoff, loggerAs in store mode

How it works

Claiming

claim() moves due jobs (pending or delayed with runAt in the past) to active in one atomic step, increments their attempt count and locks them to the worker. Highest priority first, then insertion order.

DialectClaim
PostgreSQLUPDATE ... WHERE id = ANY(ARRAY(SELECT ... FOR UPDATE SKIP LOCKED)) RETURNING *
MySQL 8SELECT ... FOR UPDATE SKIP LOCKED and UPDATE in one transaction (executor.transaction). Without transaction, one compare-and-set UPDATE per candidate row, which is also safe but needs more round trips
SQLiteUPDATE ... WHERE id IN (SELECT ... LIMIT n) RETURNING *. SQLite has a single writer, so this suits several processes on one machine sharing a database file, not several servers

Polling

Each worker polls every pollIntervalMs, and also as soon as a slot frees up or a job is added locally. Concurrency, priority, delay / runAt, maxAttempts with backoff, timeoutMs, cancellation and the added, completed, retry, failed and cancelled events behave as with the in-memory store.

Heartbeats and stale recovery

Every poll renews the lock of the jobs the worker is running. If a server crashes, its jobs stop being renewed. Every visibilityTimeoutMs / 2, each worker looks for jobs of its queue whose lock is older than visibilityTimeoutMs and:

  • returns them to pending with the attempt count kept, or
  • marks them failed if they have no attempts left, with the error Job stalled (its worker stopped renewing the lock) and has no attempts left.

Set visibilityTimeoutMs well above pollIntervalMs, since locks are only renewed on a poll.

Fencing

Completing, retrying or failing a job only succeeds while the worker still owns it (the job is active and locked by its workerId). If the job was cancelled from any server, or recovered as stale, the worker aborts the job's AbortSignal on its next poll and discards the result without emitting events.

Errors

Backend errors, such as an unreachable database, are emitted as error events and logged once per outage. Polling continues, and a backend recovered message is logged when a poll succeeds again. If writing a job's outcome fails, the job stays active in the table and is recovered once its lock expires.

TypeScript
TS
queue.on('error', (err) => metrics.inc('queue.backend_error'))

Delivery guarantee

Delivery is at least once. A job whose server died mid-run, or whose lock was not renewed within visibilityTimeoutMs (for example because that worker's event loop was blocked), is run again by another worker. Make handlers idempotent, for example by keying side effects on job.id:

TypeScript
TS
queue.process('charge', async (job) => {
  await payments.charge(job.data.orderId, { idempotencyKey: job.id })
})

Async accessors

The synchronous query methods cannot reach a database, so in backend mode they throw Queue '<name>' uses an async backend; use <method>Async() instead of <method>(). Use the async variants, which also work in store mode:

Store modeBackend mode
getJobs(status?)await getJobsAsync(status?)
getJob(id)await getJobAsync(id)
getStats()await getStatsAsync()
cancel(id)await cancelAsync(id)
purge()await purgeAsync()

The Job returned by add() is a snapshot; read the current state with getJobAsync(job.id). cancelAsync() marks a job that is active on another server as cancelled at once; that server aborts it on its next poll. purgeAsync() deletes the queue's completed, failed and cancelled jobs, except ones this worker is still winding down.

The table

init(), called lazily before the first query, runs CREATE TABLE IF NOT EXISTS plus two indexes: (queue, status, priority DESC, run_at) for claiming and (status, locked_at) for stale recovery. Call it yourself to fail fast at startup. If two servers create the table at the same moment and one fails, the error is ignored as long as the table exists.

To manage the schema in a migration instead, take the DDL from schemaStatements() and pass autoCreate: false:

TypeScript
TS
const backend = new SqlQueueBackend({ executor, autoCreate: false })
console.log(backend.schemaStatements().join(';\n'))
  • Times are BIGINT epoch milliseconds.
  • data and result are JSON text, so they must be JSON-serializable.
  • Several queues can share one table; they are separated by the queue column.
  • Queries run outside @axiljs/runtime's tReadOnly / tPure checks, so enqueuing from a read-only method is not reported as a database write.

Custom backends

SqlQueueBackend implements the QueueBackend interface. Implement it to put jobs anywhere that supports an atomic claim:

MethodContract
enqueue(job)Persist a new job
claim(queue, workerId, limit, now)Atomically move up to limit due jobs to active, locked by workerId; increment attempts, set startedAt; return them in priority, then insertion order
complete(job, workerId), fail(...), retry(...)Write the attempt's final state and release the lock, only while the job is still active and locked by workerId; return false otherwise
update(job)Overwrite the job's mutable fields unconditionally
cancel(id, now)Set a pending, delayed or active job to cancelled and return it; undefined if missing or finished
heartbeat(workerId, ids, now)Refresh the locks of the given jobs held by workerId; return the ids it still owns
recoverStale(olderThanMs, now?, queue?)Return stale active jobs to pending (or failed when out of attempts); return how many changed
get(id), list(filter?), counts(queue), remove(id)Queries. list accepts { queue?, status?, ids?, limit?, offset? }
purge(queue, keepIds?)Delete finished jobs of queue except keepIds; return how many were deleted

InMemoryQueueBackend is a reference implementation that keeps jobs in process memory. It behaves like the SQL backend (claiming, fencing, stale recovery) but is only shared by Queue instances in the same process, which makes it useful in tests:

TypeScript
TS
1import { Queue, InMemoryQueueBackend } from '@axiljs/queue'
2
3const backend = new InMemoryQueueBackend()
4const a = new Queue('jobs', { backend, workerId: 'a' })
5const b = new Queue('jobs', { backend, workerId: 'b' })

Was this page helpful?

Help improve the documentation

AxilJS is open source and documentation improvements are welcome.

AxilJS DocumentationMIT License · Built by SyntaxilitY