AI Platform

AI Governance

Token, cost and request budgets, PII and secret redaction, and recorded offline tests for any AxilJS AI provider: withBudget, withRedaction, replayProvider.

14 min readUpdated DocumentationEdit this page
On this page27

@axiljs/ai includes three provider wrappers that add policy to any AIProvider:

  • withBudget: token, cost and request limits per scope (typically a tenant) and time window, checked before the provider is called.
  • withRedaction: masks, tokenizes or blocks e-mail addresses, phone numbers, card numbers, IBANs, API keys, JWTs and custom patterns before they reach the model, and can restore them in the answer.
  • recordingProvider / replayProvider: record real responses once, then replay them in tests with no network and no API key.

Added in 0.3.5. Each wrapper returns an AIProvider again, so they compose, and everything built on providers (Agent, RAGPipeline, aiChat and aiChatStream) works unchanged.

Composing the wrappers

TypeScript
TS
1import { OpenAIProvider, withBudget, withRedaction, recordingProvider, replayProvider, aiChat } from '@axiljs/ai'
2import { getContext } from '@axiljs/runtime'
3
4const apiKey = process.env.OPENAI_API_KEY!
5
6const base = process.env.CI
7  ? replayProvider({ file: 'tests/fixtures/ai.json' })              // offline, deterministic
8  : recordingProvider(new OpenAIProvider({ apiKey }), { file: 'tests/fixtures/ai.json' })
9
10const ai = withBudget(
11  withRedaction(base, {
12    mode: 'tokenize',
13    restore: true,
14    onRedact: e => console.log('ai.redact', e.counts)
15  }),
16  {
17    limits: [
18      { window: 'minute', requests: 30 },
19      { window: 'month', costUsd: 25 }
20    ],
21    scope: () => getContext()?.tenantId,
22    pricing: { 'gpt-4o-mini': { inputPerMTok: 0.15, outputPerMTok: 0.6 } },
23    onExceeded: 'fallback-model',
24    fallbackModel: 'gpt-4o-mini'
25  }
26)
27
28app.post('/chat', aiChat({ provider: ai }))

Order matters: the outermost wrapper sees the original request. With the order above, budgets are checked first, personal data is removed next, and only the redacted request is recorded, so fixtures never contain personal data.

Budgets with withBudget

TypeScript
TS
1import { OpenAIProvider, withBudget } from '@axiljs/ai'
2
3const ai = withBudget(new OpenAIProvider({ apiKey: process.env.OPENAI_API_KEY! }), {
4  limits: [
5    { window: 'minute', requests: 60 },
6    { window: 'day', tokens: 2_000_000 },
7    { window: 'month', costUsd: 50 }
8  ],
9  pricing: { 'gpt-4o-mini': { inputPerMTok: 0.15, outputPerMTok: 0.6 } },
10  defaultMaxTokens: 1024
11})

Keep pricing in line with your provider's current prices; AxilJS does not ship a price table.

Options

OptionDefaultDescription
limitsrequiredOne BudgetLimit or an array. At least one is required
scoperuntime tenant, else 'global'() => string | undefined: the budget key for the current call, typically a tenant id
pricingnoneUSD per million tokens, keyed by model: { inputPerMTok, outputPerMTok }
storeInMemoryBudgetStoreWhere counters live. The default is per process
onExceeded'reject''reject' throws AIBudgetExceededError; 'fallback-model' retries the estimate with fallbackModel
fallbackModelnoneRequired when onExceeded is 'fallback-model'
defaultMaxTokens0Output tokens to assume when a call has no maxTokens
nowDate.nowClock, for tests

Limits and windows

A limit is { window, tokens?, costUsd?, requests? }. Omit a metric to leave it unlimited. Every limit is checked on every call.

windowBoundaries
'minute'UTC-aligned minute
'hour'UTC-aligned hour
'day'00:00 to 24:00 UTC
'month'Calendar month, UTC

How a call is checked

  1. Estimate. Prompt tokens are estimated at about 4 characters per token (system prompt, message contents, tool calls and tool definitions). Output tokens are taken from maxTokens, or defaultMaxTokens. The cost comes from pricing. Each call counts as one request.
  2. Reserve. The estimate is added to every window atomically. If any limit would be exceeded, the reservation is rolled back and the call is rejected before the provider is invoked. Concurrent calls therefore cannot jointly overrun a limit.
  3. Settle. When the provider answers, the reservation is replaced by the usage it reported. A failed call releases the reservation.

Streams are settled from the final chunk. A stream that fails releases its reservation; a stream abandoned by the consumer keeps the estimate, because tokens were likely generated. Embedding calls are estimated from their input and count as one request each.

Scope

scope returns the budget key for each call. When it is omitted or returns undefined, withBudget uses the tenantId of the active @axiljs/runtime request context, and 'global' when there is none. @axiljs/runtime is loaded with a lazy require() inside try/catch; @axiljs/ai does not depend on it.

Pricing

Price lookup for a model is: the exact model name, then the longest key that prefixes the model name ('gpt-4o' matches 'gpt-4o-2024-08-06'), then '*'. Models without a price cost 0 USD, so a costUsd limit never triggers for them.

Fallback model

With onExceeded: 'fallback-model', a call that does not fit is evaluated again with fallbackModel and sent with that model if it fits. This helps with cost limits. Token and request limits apply to the scope regardless of model, so a call over those limits is still rejected.

Usage

The wrapped provider has getUsage(scope?). It returns one entry per limit for the current window, for the given scope or the scope of the current call:

TypeScript
TS
1const usage = await ai.getUsage('acme')
2// [{ scope: 'acme', window: 'month', start, end,
3//    used: { tokens, costUsd, requests },
4//    limits: { costUsd: 50 }, remaining: { costUsd: 41.2 } }, ...]

limits and remaining contain only the metrics the limit sets.

AIBudgetExceededError

A rejected call throws AIBudgetExceededError. It extends axilError with status 429 and code AI_BUDGET_EXCEEDED, so the AxilJS error handler turns it into a 429 response. retryAfterMs is the time until the window resets, and details contains:

FieldDescription
scopeThe scope that ran out, for example a tenant id or 'global'
windowThe window whose limit would be exceeded
metric'tokens', 'costUsd' or 'requests'
limitThe configured limit for that metric and window
usedAmount already used in the window, before this request
requestedAmount this request was estimated to need
retryAfterMsMilliseconds until the window resets
modelThe model the request was evaluated against

Budget stores

A BudgetStore has two methods, both sync or async:

TypeScript
TS
1interface BudgetStore {
2  add(scope: string, window: BudgetWindowRef, amounts: BudgetAmounts): Promise<BudgetAmounts> | BudgetAmounts
3  get(scope: string, window: BudgetWindowRef): Promise<BudgetAmounts> | BudgetAmounts
4}

A row is identified by (scope, window.name, window.start). add must be atomic and return the totals after the increment; amounts can be negative, to release a reservation. window.end is there for expiry.

The default InMemoryBudgetStore keeps counters per process. With several processes or servers, each would enforce its own budget, so a limit would effectively be multiplied by the number of instances.

SqlBudgetStore

Since 0.3.6, @axiljs/stores provides SqlBudgetStore, which keeps the counters in PostgreSQL, MySQL 8 or SQLite. Every server that shares the table enforces one budget:

TypeScript
TS
1import { OpenAIProvider, withBudget } from '@axiljs/ai'
2import { createConnection } from '@axiljs/orm'
3import { SqlBudgetStore, fromConnection } from '@axiljs/stores'
4
5const db = await createConnection({ driver: 'postgres', url: process.env.DATABASE_URL })
6
7const ai = withBudget(new OpenAIProvider({ apiKey: process.env.OPENAI_API_KEY! }), {
8  limits: [
9    { window: 'minute', requests: 60 },
10    { window: 'month', costUsd: 50 }
11  ],
12  pricing: { 'gpt-4o-mini': { inputPerMTok: 0.15, outputPerMTok: 0.6 } },
13  store: new SqlBudgetStore(fromConnection(db), { cleanupIntervalMs: 3_600_000 })
14})

With createSqlStores(), which builds every SQL store at once, use store: stores.budget. See the SqlBudgetStore reference for the table layout.

OptionDefaultDescription
tableaxil_ai_budgetTable name. Must match /^[A-Za-z_][A-Za-z0-9_]{0,62}$/
cleanupIntervalMsnoneDelete expired rows every N ms. The timer is unref'd
retainMs0Keep rows this long after their window ends before cleanup() removes them, for usage reports
nowDate.nowClock. All servers should agree on time
  • The table is created on first use, or explicitly with ensureSchema(). It has one row per (scope, window_name, window_start).
  • PostgreSQL and SQLite add with one INSERT ... ON CONFLICT ... DO UPDATE ... RETURNING. MySQL uses ON DUPLICATE KEY UPDATE plus a SELECT in one transaction.
  • Costs are stored as an exact DECIMAL(30, 12) (NUMERIC on PostgreSQL), so a reservation and its release cancel exactly. SQLite keeps costs as REAL and rounds totals to 12 decimal places on read. Tokens and requests are BIGINT and are rounded to integers.
  • cleanup() deletes rows whose window ended more than retainMs ago. close() stops the cleanup timer and does not close the database connection.

Redaction with withRedaction

withRedaction scans every outgoing message (all roles), the system prompt, tool-call arguments and embedding inputs before the provider sees them.

TypeScript
TS
1import { withRedaction } from '@axiljs/ai'
2
3const ai = withRedaction(provider, {
4  mode: 'tokenize',
5  restore: true,
6  detect: ['email', 'phone', 'card', 'iban'],
7  custom: [/\bCUST-\d{6}\b/],
8  onRedact: e => audit('ai.redact', e.counts)
9})

Options

OptionDefaultDescription
moderequired'mask', 'tokenize' or 'block'
detectall detectorsDetectors to run. An unknown name throws
customnoneExtra RegExps, reported as custom and masked as [REDACTED]
restorefalsetokenize mode only: map placeholders in the response back to the original values
onRedactnoneCalled whenever something was found, also in block mode before throwing

Detectors

DetectorMatches
emailAddresses with a dotted domain and an alphabetic TLD
phone+CC numbers and grouped national formats such as (415) 555-2671; 8 to 15 digits. Bare digit runs are ignored
card13 to 19 digits, optional spaces or dashes, network prefix 2 to 6, must pass the Luhn check
ibanUppercase IBANs that pass the mod-97 check
ssnUS SSNs in 123-45-6789 form; invalid ranges are excluded
apiKeyWell-known key formats: OpenAI and Anthropic sk-, Stripe, AWS AKIA, GitHub, Slack, Google AIza
jwtThree-part eyJ... tokens
ipv4Dotted-quad addresses with valid octets

Modes

ModeBehaviour
'mask'Replaces matches with a label: [EMAIL], [PHONE], [CARD], [IBAN], [SSN], [API_KEY], [JWT], [IP], [REDACTED] for custom patterns
'tokenize'Replaces matches with placeholders such as ‹EMAIL_1›. The same value gets the same placeholder within a request, and numbering follows the order of appearance, so identical requests stay identical and replayable
'block'Rejects the call with AIPolicyViolationError; the provider is not called

With mode: 'tokenize' and restore: true, placeholders in the response are mapped back: response text, tool-call arguments and streamed deltas, even when a placeholder is split across chunks. Users see their data while the model never does.

onRedact events

onRedact(event) receives:

TypeScript
TS
1{
2  mode: 'tokenize',
3  findings: [{ type: 'email', location: 'messages[2]' }, { type: 'card', location: 'system' }],
4  counts: { email: 1, card: 1 }
5}

Locations are 'system', 'messages[i]', 'messages[i].toolCalls', 'input' or 'input[i]'. Matched values are never included.

AIPolicyViolationError

In block mode a match throws AIPolicyViolationError, which extends axilError with status 422 and code AI_POLICY_VIOLATION. details contains types (the detector types that matched) and locations, never the matched values.

redactText()

redactText(text, options?) applies the same detectors to any string and returns { text, findings }. The default is { mode: 'mask' }; block is treated as mask. Use it for logs or for data that never goes through a provider.

TypeScript
TS
import { redactText } from '@axiljs/ai'

redactText('Contact jane@example.com').text   // 'Contact [EMAIL]'

False positives and gaps

Detection is pattern-based and deliberately conservative to keep false positives low: card numbers must pass Luhn and start with 2 to 6 (which rules out, for example, millisecond timestamps), IBANs must pass mod-97, and phone numbers need separators or a leading +. Some matches can still be unintended: any valid dotted quad, such as a four-part version string, matches ipv4, and a grouped number of 8 to 15 digits can match phone. Narrow detect to the detectors you need.

Warning

Redaction does not find names, postal addresses or free-form identifiers. Add custom patterns for domain-specific data such as customer or account numbers.

Recorded tests with recordingProvider and replayProvider

TypeScript
TS
1import { AnthropicProvider, recordingProvider, replayProvider } from '@axiljs/ai'
2
3// Record once against the real API
4const ai = recordingProvider(new AnthropicProvider({ apiKey }), { file: 'tests/fixtures/agent.json' })
5
6// Replay in tests: no network, no API key
7const ai = replayProvider({ file: 'tests/fixtures/agent.json' })            // strict by default
8const ai = replayProvider({ file, strict: false, fallback: realProvider })  // record misses as they happen

recordingProvider(provider, { file }) passes every call through and saves each request and response to file. The file and its parent directories are created when missing, and existing entries are kept. Recording the same request again overwrites its entry.

replayProvider options

OptionDefaultDescription
filerequiredThe JSON fixture
stricttrueAn unrecorded request throws AIReplayMissError
fallbacknoneWith strict: false, misses go to this provider and are recorded into the file
name'replay'Reported as the provider's name
defaultModelthe fallback's, else 'replay'Reported as the provider's defaultModel

With strict: false and no fallback, a miss returns an empty response with finishReason: 'stop' (for streams, a single final chunk; for embeddings, empty vectors).

Request keys

Requests are keyed by a SHA-256 hash of the normalized request: the kind (complete, stream or embed), model, system, messages, tools, toolChoice, temperature, maxTokens, stopSequences and topP. Embeddings are keyed by model and input.

  • Object key order, signal and timeoutMs do not affect the key.
  • Requests without an explicit model are keyed with model: null.
  • complete and stream calls with the same options have different keys.
  • requestKey(kind, options) is exported if you need the key yourself.

Streaming

A stream is recorded as its complete chunk list, and only when it finishes: a consumer that stops early, or a provider error, leaves the fixture untouched. Replay yields the recorded chunks as an async iterable.

Fixture format

The fixture is plain JSON, written atomically (to a temporary file, then renamed), and is meant to be committed:

JSON
JSON
1{
2  "version": 1,
3  "entries": {
4    "<request key>": {
5      "kind": "complete",
6      "request": { "kind": "complete", "model": null, "messages": [ ... ] },
7      "response": { "content": "...", "finishReason": "stop", "usage": { ... }, "model": "..." },
8      "recordedAt": "2026-11-17T10:00:00.000Z"
9    }
10  }
11}

Stream entries have chunks instead of response.

AIReplayMissError

A strict replay miss throws AIReplayMissError with code AI_REPLAY_MISS, the request key and the fixture file. The message names the request kind, hash, model, the last user message and the fixture file, and suggests recording the request or replaying with strict: false. It is a plain Error, not an HTTP error.

HTTP status codes in the AI middleware

aiChat, aiEmbed and aiAgent answer governance errors with their own status and code instead of a generic 500. Any axilError with a 4xx status keeps its status and code; everything else is still 500 with code AI_ERROR.

ErrorStatusCodeExtra
AIBudgetExceededError429AI_BUDGET_EXCEEDEDRetry-After header in seconds, from retryAfterMs
AIPolicyViolationError422AI_POLICY_VIOLATION
Other errors500AI_ERROR

The body has the usual shape: { "error": { "code", "message" } }.

Note

aiChatStream has already started a Server-Sent Events response when the provider is called, so the status cannot change. It sends an error event with the same code and message instead. The streaming form of aiAgent sends an error event with the message only.

Was this page helpful?

Help improve the documentation

AxilJS is open source and documentation improvements are welcome.

AxilJS DocumentationMIT License · Built by SyntaxilitY