Queue Across Servers
Share one @axiljs/queue between processes and servers with SqlQueueBackend on PostgreSQL, MySQL 8 or SQLite: atomic claims with FOR UPDATE SKIP LOCKED, heartbeats, stale recovery and fencing.
On this page16
InMemoryQueueStore and FileQueueStore belong to one process. To let several processes or servers share one queue without Redis, give the queue a backend instead of a store. SqlQueueBackend keeps jobs in a table in PostgreSQL, MySQL 8 or SQLite, and hands each job to exactly one worker.
Note
New in 0.3.5: the backend option, SqlQueueBackend, InMemoryQueueBackend and the async accessors.
Usage
1import { Queue, SqlQueueBackend } from '@axiljs/queue'
2import { createConnection } from '@axiljs/orm'
3
4const conn = await createConnection({ driver: 'postgres', url: process.env.DATABASE_URL })
5const driver = conn.getDriver()
6
7const backend = new SqlQueueBackend({
8 table: 'axil_jobs', // default; created on first use
9 executor: {
10 dialect: 'postgres', // the same value as `driver` in the connection config
11 query: (sql, params) => driver.query(sql, params),
12 transaction: (fn) => driver.transaction((trx) => fn(trx)) // used by mysql only
13 }
14})
15
16const queue = new Queue('emails', { backend, concurrency: 5, pollIntervalMs: 500 })
17queue.process('send-welcome', async (job) => sendEmail(job.data.to))
18queue.start()Run the same code on every server. Each job is processed by exactly one of them.
If you already use @axiljs/stores, its fromConnection(db) executor has the same shape and can be passed directly:
import { fromConnection } from '@axiljs/stores'
const queue = new Queue('jobs', { backend: new SqlQueueBackend({ executor: fromConnection(db) }) })Warning
Register a handler for every job name on every server that runs the queue. A server that claims a job it has no handler for marks it failed with No handler registered for job '<name>'.
Options
SqlQueueBackend
| Option | Default | Description |
|---|---|---|
executor | required | Runs parameterized SQL (see below) |
table | 'axil_jobs' | Letters, digits and underscores, optionally schema-qualified (jobs.axil_jobs) |
autoCreate | true | Create the table and indexes on first use |
The executor
| Member | Required | Description |
|---|---|---|
dialect | yes | 'postgres' | 'mysql' | 'sqlite' |
query(sql, params?) | yes | Resolve with { rows, affectedRows? }. SQL is written with ? placeholders and rewritten to $1, $2, ... for postgres |
transaction(fn) | no | Run fn(trx) in one transaction on one connection, committing on success and rolling back on error. Used by MySQL only |
On MySQL, which has no RETURNING, the executor must report affectedRows for writes. An executor that does not report it cannot fence: the backend then assumes the worker still owns the job. @axiljs/orm drivers report it.
Queue options in backend mode
| Option | Default | Description |
|---|---|---|
backend | A QueueBackend. Mutually exclusive with store; passing both throws | |
workerId | host:pid:random | This worker's lock owner id |
visibilityTimeoutMs | 30000 | An active job whose lock has not been renewed for this long is treated as abandoned and returned to pending |
concurrency, pollIntervalMs, backoff, logger | As in store mode |
How it works
Claiming
claim() moves due jobs (pending or delayed with runAt in the past) to active in one atomic step, increments their attempt count and locks them to the worker. Highest priority first, then insertion order.
| Dialect | Claim |
|---|---|
| PostgreSQL | UPDATE ... WHERE id = ANY(ARRAY(SELECT ... FOR UPDATE SKIP LOCKED)) RETURNING * |
| MySQL 8 | SELECT ... FOR UPDATE SKIP LOCKED and UPDATE in one transaction (executor.transaction). Without transaction, one compare-and-set UPDATE per candidate row, which is also safe but needs more round trips |
| SQLite | UPDATE ... WHERE id IN (SELECT ... LIMIT n) RETURNING *. SQLite has a single writer, so this suits several processes on one machine sharing a database file, not several servers |
Polling
Each worker polls every pollIntervalMs, and also as soon as a slot frees up or a job is added locally. Concurrency, priority, delay / runAt, maxAttempts with backoff, timeoutMs, cancellation and the added, completed, retry, failed and cancelled events behave as with the in-memory store.
Heartbeats and stale recovery
Every poll renews the lock of the jobs the worker is running. If a server crashes, its jobs stop being renewed. Every visibilityTimeoutMs / 2, each worker looks for jobs of its queue whose lock is older than visibilityTimeoutMs and:
- returns them to
pendingwith the attempt count kept, or - marks them
failedif they have no attempts left, with the errorJob stalled (its worker stopped renewing the lock) and has no attempts left.
Set visibilityTimeoutMs well above pollIntervalMs, since locks are only renewed on a poll.
Fencing
Completing, retrying or failing a job only succeeds while the worker still owns it (the job is active and locked by its workerId). If the job was cancelled from any server, or recovered as stale, the worker aborts the job's AbortSignal on its next poll and discards the result without emitting events.
Errors
Backend errors, such as an unreachable database, are emitted as error events and logged once per outage. Polling continues, and a backend recovered message is logged when a poll succeeds again. If writing a job's outcome fails, the job stays active in the table and is recovered once its lock expires.
queue.on('error', (err) => metrics.inc('queue.backend_error'))Delivery guarantee
Delivery is at least once. A job whose server died mid-run, or whose lock was not renewed within visibilityTimeoutMs (for example because that worker's event loop was blocked), is run again by another worker. Make handlers idempotent, for example by keying side effects on job.id:
queue.process('charge', async (job) => {
await payments.charge(job.data.orderId, { idempotencyKey: job.id })
})Async accessors
The synchronous query methods cannot reach a database, so in backend mode they throw Queue '<name>' uses an async backend; use <method>Async() instead of <method>(). Use the async variants, which also work in store mode:
| Store mode | Backend mode |
|---|---|
getJobs(status?) | await getJobsAsync(status?) |
getJob(id) | await getJobAsync(id) |
getStats() | await getStatsAsync() |
cancel(id) | await cancelAsync(id) |
purge() | await purgeAsync() |
The Job returned by add() is a snapshot; read the current state with getJobAsync(job.id). cancelAsync() marks a job that is active on another server as cancelled at once; that server aborts it on its next poll. purgeAsync() deletes the queue's completed, failed and cancelled jobs, except ones this worker is still winding down.
The table
init(), called lazily before the first query, runs CREATE TABLE IF NOT EXISTS plus two indexes: (queue, status, priority DESC, run_at) for claiming and (status, locked_at) for stale recovery. Call it yourself to fail fast at startup. If two servers create the table at the same moment and one fails, the error is ignored as long as the table exists.
To manage the schema in a migration instead, take the DDL from schemaStatements() and pass autoCreate: false:
const backend = new SqlQueueBackend({ executor, autoCreate: false })
console.log(backend.schemaStatements().join(';\n'))- Times are
BIGINTepoch milliseconds. dataandresultare JSON text, so they must be JSON-serializable.- Several queues can share one table; they are separated by the
queuecolumn. - Queries run outside
@axiljs/runtime'stReadOnly/tPurechecks, so enqueuing from a read-only method is not reported as a database write.
Custom backends
SqlQueueBackend implements the QueueBackend interface. Implement it to put jobs anywhere that supports an atomic claim:
| Method | Contract |
|---|---|
enqueue(job) | Persist a new job |
claim(queue, workerId, limit, now) | Atomically move up to limit due jobs to active, locked by workerId; increment attempts, set startedAt; return them in priority, then insertion order |
complete(job, workerId), fail(...), retry(...) | Write the attempt's final state and release the lock, only while the job is still active and locked by workerId; return false otherwise |
update(job) | Overwrite the job's mutable fields unconditionally |
cancel(id, now) | Set a pending, delayed or active job to cancelled and return it; undefined if missing or finished |
heartbeat(workerId, ids, now) | Refresh the locks of the given jobs held by workerId; return the ids it still owns |
recoverStale(olderThanMs, now?, queue?) | Return stale active jobs to pending (or failed when out of attempts); return how many changed |
get(id), list(filter?), counts(queue), remove(id) | Queries. list accepts { queue?, status?, ids?, limit?, offset? } |
purge(queue, keepIds?) | Delete finished jobs of queue except keepIds; return how many were deleted |
InMemoryQueueBackend is a reference implementation that keeps jobs in process memory. It behaves like the SQL backend (claiming, fencing, stale recovery) but is only shared by Queue instances in the same process, which makes it useful in tests:
1import { Queue, InMemoryQueueBackend } from '@axiljs/queue'
2
3const backend = new InMemoryQueueBackend()
4const a = new Queue('jobs', { backend, workerId: 'a' })
5const b = new Queue('jobs', { backend, workerId: 'b' })Related
Was this page helpful?