Queue Storage
Persist @axiljs/queue jobs to disk with FileQueueStore: an append-only journal plus atomic snapshots, recovery of jobs that were running at a crash, and its limits.
A Queue keeps its jobs in a QueueStore. The default, InMemoryQueueStore, loses every job when the process exits. FileQueueStore persists them to disk with synchronous node:fs calls and needs no extra dependencies.
Note
New in 0.3.4: FileQueueStore.
Usage
1import { Queue, FileQueueStore } from '@axiljs/queue'
2
3const store = new FileQueueStore({ path: './data/emails.queue', compactEvery: 1000 })
4const queue = new Queue('emails', { store })
5
6queue.process('send-welcome', async (job) => sendEmail(job.data.to))
7queue.start()
8
9process.on('SIGTERM', async () => {
10 await queue.stop()
11 store.close() // writes a final snapshot
12})Stop the queue (await queue.stop()) before closing its store. After close(), writes throw FileQueueStore(<path>) is closed.
Options
| Option | Default | Description |
|---|---|---|
path | required | Snapshot file path. Resolved to an absolute path; its directory is created if missing |
compactEvery | 1000 | Compact after this many journal entries. 0 disables automatic compaction; it still happens on open and on close() |
On-disk layout
For path: './data/emails.queue':
| File | Content |
|---|---|
emails.queue | Snapshot: JSON { version: 1, jobs: [...] } in queue order |
emails.queue.journal | Append-only JSON lines, one per change: {"op":"put","job":{...}} or {"op":"del","id":"..."} |
emails.queue.tmp | Transient: the next snapshot, before it is renamed into place |
Every change (add, update, remove) is appended to the journal synchronously. Every compactEvery journal entries, on open and on close(), the full state is written to the .tmp file and atomically renamed over the snapshot, then the journal is emptied. compact() can also be called directly.
Recovery
On open, the snapshot is loaded and the journal replayed on top of it. Then:
- Jobs that were
activewhen the process died return topending, with their attempt count kept, so they are retried withinmaxAttempts. - A truncated final journal line (a crash in the middle of a write) is ignored. A corrupt line anywhere else, or a corrupt snapshot, throws.
- The recovered state is written as a new snapshot and the journal starts empty.
Warning
Because a job that was running at the crash runs again, handlers should be idempotent: running the same job twice must not, for example, send two e-mails or charge a card twice.
Ordering is the same as the in-memory store: higher priority first, then first in, first out.
Data rules
- Job
datamust be JSON-serializable.Datevalues insidedatacome back as strings. A job whose data cannot be serialized is rejected and the store is left unchanged. - A handler result that cannot be serialized is not persisted; the job's status still is.
- The job's own
createdAt,startedAt,finishedAtandrunAtare restored asDateobjects.
Limits
- Process crashes, not power loss. Writes are not
fsynced. Data survives a process crash because the operating system still holds the written pages, but the most recent changes can be lost if the machine itself goes down. - One process per file. Use one store per file, owned by a single process. Two processes sharing a file would overwrite each other's snapshots. In cluster mode, give each worker its own file, or use a SQL backend so all workers share one queue.
- Synchronous I/O. Every change is a synchronous file append that runs on the event loop.
Custom stores
A store implements QueueStore: add(job), get(id), update(job), list() and remove(id). The interface is synchronous, so it fits in-memory or synchronous embedded stores, not network databases. For a database shared by several processes or servers, give the queue a backend instead of a store.
Related
Was this page helpful?