Background Jobs

Queue Storage

Persist @axiljs/queue jobs to disk with FileQueueStore: an append-only journal plus atomic snapshots, recovery of jobs that were running at a crash, and its limits.

4 min readUpdated DocumentationEdit this page
On this page8

A Queue keeps its jobs in a QueueStore. The default, InMemoryQueueStore, loses every job when the process exits. FileQueueStore persists them to disk with synchronous node:fs calls and needs no extra dependencies.

Note

New in 0.3.4: FileQueueStore.

Usage

TypeScript
TS
1import { Queue, FileQueueStore } from '@axiljs/queue'
2
3const store = new FileQueueStore({ path: './data/emails.queue', compactEvery: 1000 })
4const queue = new Queue('emails', { store })
5
6queue.process('send-welcome', async (job) => sendEmail(job.data.to))
7queue.start()
8
9process.on('SIGTERM', async () => {
10  await queue.stop()
11  store.close() // writes a final snapshot
12})

Stop the queue (await queue.stop()) before closing its store. After close(), writes throw FileQueueStore(<path>) is closed.

Options

OptionDefaultDescription
pathrequiredSnapshot file path. Resolved to an absolute path; its directory is created if missing
compactEvery1000Compact after this many journal entries. 0 disables automatic compaction; it still happens on open and on close()

On-disk layout

For path: './data/emails.queue':

FileContent
emails.queueSnapshot: JSON { version: 1, jobs: [...] } in queue order
emails.queue.journalAppend-only JSON lines, one per change: {"op":"put","job":{...}} or {"op":"del","id":"..."}
emails.queue.tmpTransient: the next snapshot, before it is renamed into place

Every change (add, update, remove) is appended to the journal synchronously. Every compactEvery journal entries, on open and on close(), the full state is written to the .tmp file and atomically renamed over the snapshot, then the journal is emptied. compact() can also be called directly.

Recovery

On open, the snapshot is loaded and the journal replayed on top of it. Then:

  • Jobs that were active when the process died return to pending, with their attempt count kept, so they are retried within maxAttempts.
  • A truncated final journal line (a crash in the middle of a write) is ignored. A corrupt line anywhere else, or a corrupt snapshot, throws.
  • The recovered state is written as a new snapshot and the journal starts empty.

Warning

Because a job that was running at the crash runs again, handlers should be idempotent: running the same job twice must not, for example, send two e-mails or charge a card twice.

Ordering is the same as the in-memory store: higher priority first, then first in, first out.

Data rules

  • Job data must be JSON-serializable. Date values inside data come back as strings. A job whose data cannot be serialized is rejected and the store is left unchanged.
  • A handler result that cannot be serialized is not persisted; the job's status still is.
  • The job's own createdAt, startedAt, finishedAt and runAt are restored as Date objects.

Limits

  • Process crashes, not power loss. Writes are not fsynced. Data survives a process crash because the operating system still holds the written pages, but the most recent changes can be lost if the machine itself goes down.
  • One process per file. Use one store per file, owned by a single process. Two processes sharing a file would overwrite each other's snapshots. In cluster mode, give each worker its own file, or use a SQL backend so all workers share one queue.
  • Synchronous I/O. Every change is a synchronous file append that runs on the event loop.

Custom stores

A store implements QueueStore: add(job), get(id), update(job), list() and remove(id). The interface is synchronous, so it fits in-memory or synchronous embedded stores, not network databases. For a database shared by several processes or servers, give the queue a backend instead of a store.

Was this page helpful?

Help improve the documentation

AxilJS is open source and documentation improvements are welcome.

AxilJS DocumentationMIT License · Built by SyntaxilitY