AI Governance
Token, cost and request budgets, PII and secret redaction, and recorded offline tests for any AxilJS AI provider: withBudget, withRedaction, replayProvider.
On this page27
- Composing the wrappers
- Budgets with `withBudget`
- Options
- Limits and windows
- How a call is checked
- Scope
- Pricing
- Fallback model
- Usage
- `AIBudgetExceededError`
- Budget stores
- Redaction with `withRedaction`
- Options
- Detectors
- Modes
- `onRedact` events
- `AIPolicyViolationError`
- `redactText()`
- False positives and gaps
- Recorded tests with `recordingProvider` and `replayProvider`
- `replayProvider` options
- Request keys
- Streaming
- Fixture format
- `AIReplayMissError`
- HTTP status codes in the AI middleware
- Related
@axiljs/ai includes three provider wrappers that add policy to any AIProvider:
withBudget: token, cost and request limits per scope (typically a tenant) and time window, checked before the provider is called.withRedaction: masks, tokenizes or blocks e-mail addresses, phone numbers, card numbers, IBANs, API keys, JWTs and custom patterns before they reach the model, and can restore them in the answer.recordingProvider/replayProvider: record real responses once, then replay them in tests with no network and no API key.
Added in 0.3.5. Each wrapper returns an AIProvider again, so they compose, and everything built on providers (Agent, RAGPipeline, aiChat and aiChatStream) works unchanged.
Composing the wrappers
1import { OpenAIProvider, withBudget, withRedaction, recordingProvider, replayProvider, aiChat } from '@axiljs/ai'
2import { getContext } from '@axiljs/runtime'
3
4const apiKey = process.env.OPENAI_API_KEY!
5
6const base = process.env.CI
7 ? replayProvider({ file: 'tests/fixtures/ai.json' }) // offline, deterministic
8 : recordingProvider(new OpenAIProvider({ apiKey }), { file: 'tests/fixtures/ai.json' })
9
10const ai = withBudget(
11 withRedaction(base, {
12 mode: 'tokenize',
13 restore: true,
14 onRedact: e => console.log('ai.redact', e.counts)
15 }),
16 {
17 limits: [
18 { window: 'minute', requests: 30 },
19 { window: 'month', costUsd: 25 }
20 ],
21 scope: () => getContext()?.tenantId,
22 pricing: { 'gpt-4o-mini': { inputPerMTok: 0.15, outputPerMTok: 0.6 } },
23 onExceeded: 'fallback-model',
24 fallbackModel: 'gpt-4o-mini'
25 }
26)
27
28app.post('/chat', aiChat({ provider: ai }))Order matters: the outermost wrapper sees the original request. With the order above, budgets are checked first, personal data is removed next, and only the redacted request is recorded, so fixtures never contain personal data.
Budgets with withBudget
1import { OpenAIProvider, withBudget } from '@axiljs/ai'
2
3const ai = withBudget(new OpenAIProvider({ apiKey: process.env.OPENAI_API_KEY! }), {
4 limits: [
5 { window: 'minute', requests: 60 },
6 { window: 'day', tokens: 2_000_000 },
7 { window: 'month', costUsd: 50 }
8 ],
9 pricing: { 'gpt-4o-mini': { inputPerMTok: 0.15, outputPerMTok: 0.6 } },
10 defaultMaxTokens: 1024
11})Keep pricing in line with your provider's current prices; AxilJS does not ship a price table.
Options
| Option | Default | Description |
|---|---|---|
limits | required | One BudgetLimit or an array. At least one is required |
scope | runtime tenant, else 'global' | () => string | undefined: the budget key for the current call, typically a tenant id |
pricing | none | USD per million tokens, keyed by model: { inputPerMTok, outputPerMTok } |
store | InMemoryBudgetStore | Where counters live. The default is per process |
onExceeded | 'reject' | 'reject' throws AIBudgetExceededError; 'fallback-model' retries the estimate with fallbackModel |
fallbackModel | none | Required when onExceeded is 'fallback-model' |
defaultMaxTokens | 0 | Output tokens to assume when a call has no maxTokens |
now | Date.now | Clock, for tests |
Limits and windows
A limit is { window, tokens?, costUsd?, requests? }. Omit a metric to leave it unlimited. Every limit is checked on every call.
window | Boundaries |
|---|---|
'minute' | UTC-aligned minute |
'hour' | UTC-aligned hour |
'day' | 00:00 to 24:00 UTC |
'month' | Calendar month, UTC |
How a call is checked
- Estimate. Prompt tokens are estimated at about 4 characters per token (system prompt, message contents, tool calls and tool definitions). Output tokens are taken from
maxTokens, ordefaultMaxTokens. The cost comes frompricing. Each call counts as one request. - Reserve. The estimate is added to every window atomically. If any limit would be exceeded, the reservation is rolled back and the call is rejected before the provider is invoked. Concurrent calls therefore cannot jointly overrun a limit.
- Settle. When the provider answers, the reservation is replaced by the usage it reported. A failed call releases the reservation.
Streams are settled from the final chunk. A stream that fails releases its reservation; a stream abandoned by the consumer keeps the estimate, because tokens were likely generated. Embedding calls are estimated from their input and count as one request each.
Scope
scope returns the budget key for each call. When it is omitted or returns undefined, withBudget uses the tenantId of the active @axiljs/runtime request context, and 'global' when there is none. @axiljs/runtime is loaded with a lazy require() inside try/catch; @axiljs/ai does not depend on it.
Pricing
Price lookup for a model is: the exact model name, then the longest key that prefixes the model name ('gpt-4o' matches 'gpt-4o-2024-08-06'), then '*'. Models without a price cost 0 USD, so a costUsd limit never triggers for them.
Fallback model
With onExceeded: 'fallback-model', a call that does not fit is evaluated again with fallbackModel and sent with that model if it fits. This helps with cost limits. Token and request limits apply to the scope regardless of model, so a call over those limits is still rejected.
Usage
The wrapped provider has getUsage(scope?). It returns one entry per limit for the current window, for the given scope or the scope of the current call:
1const usage = await ai.getUsage('acme')
2// [{ scope: 'acme', window: 'month', start, end,
3// used: { tokens, costUsd, requests },
4// limits: { costUsd: 50 }, remaining: { costUsd: 41.2 } }, ...]limits and remaining contain only the metrics the limit sets.
AIBudgetExceededError
A rejected call throws AIBudgetExceededError. It extends axilError with status 429 and code AI_BUDGET_EXCEEDED, so the AxilJS error handler turns it into a 429 response. retryAfterMs is the time until the window resets, and details contains:
| Field | Description |
|---|---|
scope | The scope that ran out, for example a tenant id or 'global' |
window | The window whose limit would be exceeded |
metric | 'tokens', 'costUsd' or 'requests' |
limit | The configured limit for that metric and window |
used | Amount already used in the window, before this request |
requested | Amount this request was estimated to need |
retryAfterMs | Milliseconds until the window resets |
model | The model the request was evaluated against |
Budget stores
A BudgetStore has two methods, both sync or async:
1interface BudgetStore {
2 add(scope: string, window: BudgetWindowRef, amounts: BudgetAmounts): Promise<BudgetAmounts> | BudgetAmounts
3 get(scope: string, window: BudgetWindowRef): Promise<BudgetAmounts> | BudgetAmounts
4}A row is identified by (scope, window.name, window.start). add must be atomic and return the totals after the increment; amounts can be negative, to release a reservation. window.end is there for expiry.
The default InMemoryBudgetStore keeps counters per process. With several processes or servers, each would enforce its own budget, so a limit would effectively be multiplied by the number of instances.
SqlBudgetStore
Since 0.3.6, @axiljs/stores provides SqlBudgetStore, which keeps the counters in PostgreSQL, MySQL 8 or SQLite. Every server that shares the table enforces one budget:
1import { OpenAIProvider, withBudget } from '@axiljs/ai'
2import { createConnection } from '@axiljs/orm'
3import { SqlBudgetStore, fromConnection } from '@axiljs/stores'
4
5const db = await createConnection({ driver: 'postgres', url: process.env.DATABASE_URL })
6
7const ai = withBudget(new OpenAIProvider({ apiKey: process.env.OPENAI_API_KEY! }), {
8 limits: [
9 { window: 'minute', requests: 60 },
10 { window: 'month', costUsd: 50 }
11 ],
12 pricing: { 'gpt-4o-mini': { inputPerMTok: 0.15, outputPerMTok: 0.6 } },
13 store: new SqlBudgetStore(fromConnection(db), { cleanupIntervalMs: 3_600_000 })
14})With createSqlStores(), which builds every SQL store at once, use store: stores.budget. See the SqlBudgetStore reference for the table layout.
| Option | Default | Description |
|---|---|---|
table | axil_ai_budget | Table name. Must match /^[A-Za-z_][A-Za-z0-9_]{0,62}$/ |
cleanupIntervalMs | none | Delete expired rows every N ms. The timer is unref'd |
retainMs | 0 | Keep rows this long after their window ends before cleanup() removes them, for usage reports |
now | Date.now | Clock. All servers should agree on time |
- The table is created on first use, or explicitly with
ensureSchema(). It has one row per(scope, window_name, window_start). - PostgreSQL and SQLite add with one
INSERT ... ON CONFLICT ... DO UPDATE ... RETURNING. MySQL usesON DUPLICATE KEY UPDATEplus aSELECTin one transaction. - Costs are stored as an exact
DECIMAL(30, 12)(NUMERICon PostgreSQL), so a reservation and its release cancel exactly. SQLite keeps costs asREALand rounds totals to 12 decimal places on read. Tokens and requests areBIGINTand are rounded to integers. cleanup()deletes rows whose window ended more thanretainMsago.close()stops the cleanup timer and does not close the database connection.
Redaction with withRedaction
withRedaction scans every outgoing message (all roles), the system prompt, tool-call arguments and embedding inputs before the provider sees them.
1import { withRedaction } from '@axiljs/ai'
2
3const ai = withRedaction(provider, {
4 mode: 'tokenize',
5 restore: true,
6 detect: ['email', 'phone', 'card', 'iban'],
7 custom: [/\bCUST-\d{6}\b/],
8 onRedact: e => audit('ai.redact', e.counts)
9})Options
| Option | Default | Description |
|---|---|---|
mode | required | 'mask', 'tokenize' or 'block' |
detect | all detectors | Detectors to run. An unknown name throws |
custom | none | Extra RegExps, reported as custom and masked as [REDACTED] |
restore | false | tokenize mode only: map placeholders in the response back to the original values |
onRedact | none | Called whenever something was found, also in block mode before throwing |
Detectors
| Detector | Matches |
|---|---|
email | Addresses with a dotted domain and an alphabetic TLD |
phone | +CC numbers and grouped national formats such as (415) 555-2671; 8 to 15 digits. Bare digit runs are ignored |
card | 13 to 19 digits, optional spaces or dashes, network prefix 2 to 6, must pass the Luhn check |
iban | Uppercase IBANs that pass the mod-97 check |
ssn | US SSNs in 123-45-6789 form; invalid ranges are excluded |
apiKey | Well-known key formats: OpenAI and Anthropic sk-, Stripe, AWS AKIA, GitHub, Slack, Google AIza |
jwt | Three-part eyJ... tokens |
ipv4 | Dotted-quad addresses with valid octets |
Modes
| Mode | Behaviour |
|---|---|
'mask' | Replaces matches with a label: [EMAIL], [PHONE], [CARD], [IBAN], [SSN], [API_KEY], [JWT], [IP], [REDACTED] for custom patterns |
'tokenize' | Replaces matches with placeholders such as ‹EMAIL_1›. The same value gets the same placeholder within a request, and numbering follows the order of appearance, so identical requests stay identical and replayable |
'block' | Rejects the call with AIPolicyViolationError; the provider is not called |
With mode: 'tokenize' and restore: true, placeholders in the response are mapped back: response text, tool-call arguments and streamed deltas, even when a placeholder is split across chunks. Users see their data while the model never does.
onRedact events
onRedact(event) receives:
1{
2 mode: 'tokenize',
3 findings: [{ type: 'email', location: 'messages[2]' }, { type: 'card', location: 'system' }],
4 counts: { email: 1, card: 1 }
5}Locations are 'system', 'messages[i]', 'messages[i].toolCalls', 'input' or 'input[i]'. Matched values are never included.
AIPolicyViolationError
In block mode a match throws AIPolicyViolationError, which extends axilError with status 422 and code AI_POLICY_VIOLATION. details contains types (the detector types that matched) and locations, never the matched values.
redactText()
redactText(text, options?) applies the same detectors to any string and returns { text, findings }. The default is { mode: 'mask' }; block is treated as mask. Use it for logs or for data that never goes through a provider.
import { redactText } from '@axiljs/ai'
redactText('Contact jane@example.com').text // 'Contact [EMAIL]'False positives and gaps
Detection is pattern-based and deliberately conservative to keep false positives low: card numbers must pass Luhn and start with 2 to 6 (which rules out, for example, millisecond timestamps), IBANs must pass mod-97, and phone numbers need separators or a leading +. Some matches can still be unintended: any valid dotted quad, such as a four-part version string, matches ipv4, and a grouped number of 8 to 15 digits can match phone. Narrow detect to the detectors you need.
Warning
Redaction does not find names, postal addresses or free-form identifiers. Add custom patterns for domain-specific data such as customer or account numbers.
Recorded tests with recordingProvider and replayProvider
1import { AnthropicProvider, recordingProvider, replayProvider } from '@axiljs/ai'
2
3// Record once against the real API
4const ai = recordingProvider(new AnthropicProvider({ apiKey }), { file: 'tests/fixtures/agent.json' })
5
6// Replay in tests: no network, no API key
7const ai = replayProvider({ file: 'tests/fixtures/agent.json' }) // strict by default
8const ai = replayProvider({ file, strict: false, fallback: realProvider }) // record misses as they happenrecordingProvider(provider, { file }) passes every call through and saves each request and response to file. The file and its parent directories are created when missing, and existing entries are kept. Recording the same request again overwrites its entry.
replayProvider options
| Option | Default | Description |
|---|---|---|
file | required | The JSON fixture |
strict | true | An unrecorded request throws AIReplayMissError |
fallback | none | With strict: false, misses go to this provider and are recorded into the file |
name | 'replay' | Reported as the provider's name |
defaultModel | the fallback's, else 'replay' | Reported as the provider's defaultModel |
With strict: false and no fallback, a miss returns an empty response with finishReason: 'stop' (for streams, a single final chunk; for embeddings, empty vectors).
Request keys
Requests are keyed by a SHA-256 hash of the normalized request: the kind (complete, stream or embed), model, system, messages, tools, toolChoice, temperature, maxTokens, stopSequences and topP. Embeddings are keyed by model and input.
- Object key order,
signalandtimeoutMsdo not affect the key. - Requests without an explicit
modelare keyed withmodel: null. completeandstreamcalls with the same options have different keys.requestKey(kind, options)is exported if you need the key yourself.
Streaming
A stream is recorded as its complete chunk list, and only when it finishes: a consumer that stops early, or a provider error, leaves the fixture untouched. Replay yields the recorded chunks as an async iterable.
Fixture format
The fixture is plain JSON, written atomically (to a temporary file, then renamed), and is meant to be committed:
1{
2 "version": 1,
3 "entries": {
4 "<request key>": {
5 "kind": "complete",
6 "request": { "kind": "complete", "model": null, "messages": [ ... ] },
7 "response": { "content": "...", "finishReason": "stop", "usage": { ... }, "model": "..." },
8 "recordedAt": "2026-11-17T10:00:00.000Z"
9 }
10 }
11}Stream entries have chunks instead of response.
AIReplayMissError
A strict replay miss throws AIReplayMissError with code AI_REPLAY_MISS, the request key and the fixture file. The message names the request kind, hash, model, the last user message and the fixture file, and suggests recording the request or replaying with strict: false. It is a plain Error, not an HTTP error.
HTTP status codes in the AI middleware
aiChat, aiEmbed and aiAgent answer governance errors with their own status and code instead of a generic 500. Any axilError with a 4xx status keeps its status and code; everything else is still 500 with code AI_ERROR.
| Error | Status | Code | Extra |
|---|---|---|---|
AIBudgetExceededError | 429 | AI_BUDGET_EXCEEDED | Retry-After header in seconds, from retryAfterMs |
AIPolicyViolationError | 422 | AI_POLICY_VIOLATION | |
| Other errors | 500 | AI_ERROR |
The body has the usual shape: { "error": { "code", "message" } }.
Note
aiChatStream has already started a Server-Sent Events response when the provider is called, so the status cannot change. It sends an error event with the same code and message instead. The streaming form of aiAgent sends an error event with the message only.
Related
Was this page helpful?