Governed inference API

What it is#

POST /api/v1/inference makes one typed model call and returns the answer as one JSON body. With stream: true it returns a text/event-stream instead. You name a capability_tier label. You never name a provider, a model, a base URL or a provider key, and you never hold a provider key.

The endpoint uses the same controls as the rest of LumenFlow Cloud: a credential, limits, residency checks, a receipt for every call, and typed refusals. A refusal is never a fallback. A call never moves to a cheaper or less governed route.

Credentials#

Call the endpoint with an inference-only workspace key. Its prefix is lf_inf_ and it carries exactly one scope, inference:invoke. A primary workspace key, an org service token and an app key are refused with 403 insufficient_scope. Issue and revoke inference keys as described in Credentials and scopes.

curl -X POST https://<your-cloud-host>/api/v1/inference \
  -H "Authorization: Bearer lf_inf_..." \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: 7f3c2a10-5b1e-4d8a-9c2e-1a2b3c4d5e6f" \
  -d '{
    "capability_tier": "standard",
    "messages": [
      { "role": "system", "content": "You summarise support tickets in one sentence." },
      { "role": "user", "content": "The export button stopped working after the 3 October release." }
    ],
    "max_output_tokens": 200,
    "app_id": "support-desk",
    "caller_reference": "ticket-10482"
  }'

Request fields#

The request body is a closed schema. Any field not listed here is refused with 400 invalid_request. A field that names a provider, model, base URL or key is listed in error.details.fields.

Send Idempotency-Key as a header to make a retry safe. The same key with the same payload returns the stored result with the header Idempotent-Replayed: true and makes no second model call. The same key with a different payload returns 409 idempotency_conflict. A call that is still running returns 409 idempotency_in_progress. A stored result is kept for the retention limit, which is 24 hours by default, and is then cleared.

Capability tiers and routes#

A tier label is resolved to a route before any provider call. The route is the workspace's own model registration that declares that tier. A workspace with no registration of its own for a tier can use a managed route for that tier where the platform offers one. Whether managed routes are enabled for a given workspace is not visible from this documentation. Check the routes listing (below) for the tiers your key can reach.

Messages, images and cache hints#

Image parts are accepted in user messages. Content is either a string or an array of parts. A part is { "type": "text", "text": "..." } or { "type": "image", "media_type": "image/png", "data": "<base64>" }.

• Media types are image/png, image/jpeg and image/webp.
• data is canonical base64 with no line breaks.
• A request carries at most 8 images. Each image decodes to at most 5 MiB and each side is at most 4096 pixels. The dimensions are read from the image header, not from the request.
• A message carries at most 64 parts.

Cache hints let a long, repeated prefix, such as a system prompt and a tool set, be cached by the route when the route supports it. Mark a message or a tool definition with "cache": "ephemeral". The marker covers everything before it. A request carries at most 4 markers across messages and tools together. More than 4 is refused with 400 invalid_request.

A route that cannot cache accepts the markers, ignores them and serves the call normally. When the request has a marker, the response includes cache_hint_applied (true or false). Without markers the field is absent. Usage reports cached_input_tokens and cache_write_tokens, which are null when the route reports neither.

Tool calling#

The endpoint carries the model side of an agent loop. Cloud never runs a tool.

• Declare. tools is an array of { "name", "description", "input_schema" }. A name is 1 to 64 letters, digits, _ or -, starts with a letter, and is unique. input_schema is a JSON Schema object with "type": "object".
• Respond. When the model asks for tools, the 200 response carries tool_calls (each { id, name, arguments }) and finish_reason: "tool_calls". text may be empty. Run the tool in your own code.
• Send the result. Send the next request with the assistant message's tool_calls and a tool message per call: { "role": "tool", "tool_call_id": "...", "content": "...", "is_error": false }. Every result must answer exactly one earlier call. An orphan, a duplicate or a result that comes before its call is refused with 400 invalid_request.
• Invalid calls. A model call with an unknown name, unparseable arguments or a bad id is never returned as a tool call. It is counted in invalid_tool_call_count. If no valid call remains the response has finish_reason: "tool_calls_invalid" and no tool_calls.
• Bounds. input_schema and arguments are limited to 32 levels of nesting, 2,000 keys and 1,000 enum entries.

Tool names, schemas, arguments and results are never written to a receipt. The receipt records only counts.

Streaming#

With stream: true the response is text/event-stream (Content-Type: text/event-stream, Cache-Control: no-store, no-transform). It is the same governed call as a single response: the same refusal order, one model call, one hold, one receipt and one settlement.

• Before the first byte. Every refusal that is known before the response opens is an ordinary JSON error with its status code.
• Frames. Each frame is event: <name>, one data: <json> line and a blank line. The events are:
  • delta with { "text": "..." }, zero or more times.
  • tool_call with { "id", "name", "arguments" }, one whole assembled call. Partial argument text is never sent.
  • exactly one terminal event, either usage (the call completed, carrying the response fields other than the content, including receipt) or error (the call failed after the first byte, with code, message and request_id).
• Heartbeat. While the model is silent for 15 seconds the stream sends a comment line : keepalive. Ignore it.
• Idle and total limits. A stream ends with an error event if it runs longer than the stream duration limit or sits silent past the idle limit. Output of any kind, including a tool argument being written, resets the idle clock. The routes listing reports both limits for your key.
• Disconnect. If your client closes the connection, the upstream call stops and is charged for the output produced, estimated from the text sent. The receipt is marked stream.termination: "client_closed" and stream.usage_estimated: true.
• Replay. A retry with the same Idempotency-Key after a stream has ended replays the stored result, with no model call and no charge. A retry while the first stream is still running returns 409 idempotency_in_progress.

Response#

A single 200 response carries:

• text and finish_reason.
• usage with input_tokens, output_tokens, total_tokens, cached_input_tokens and cache_write_tokens.
• route with capability_tier, provider_kind and provider_family. provider_kind is a label for the kind of route: a hosted route, or a customer-addressed endpoint. No provider or model name is returned.
• cost with amount (a six-decimal string), currency (USD) and basis. basis is managed for a wallet-debited call, byok_declared or byok_estimate for a call on your own route where the provider bills you, or null when the route has no known price.
• receipt with id, recorded and signed.
• caller_reference, when you sent one.

A receipt is written for every admitted, refused or replayed call. receipt.recorded is false when a call on your own route went ahead but its receipt could not be written. Such a gap is flagged, not hidden. A call whose cost is debited from a managed wallet is refused with 503 receipt_unavailable instead, and nothing is charged. receipt.signed is true only when the stored signature is present.

Typed refusals#

Every refusal has the body { "request_id", "error": { "code", "message", "details?" } }.

Each provider call makes one generation attempt. The SDK does not retry it, and a provider failure never triggers another route.

Routes listing#

GET /api/v1/inference/routes tells a client what it can reach before it calls. Authenticate with an inference key, or with a primary workspace key holding read. An app key and an org service token are refused.

The reply has three parts:

• tiers: one entry per capability tier with capability_tier, reachable, source_kind, priced, input_modalities and max_output_tokens.
• limits: the effective limits for the calling key: tokens_per_call, tokens_per_day, spend_usd_per_day, requests_per_minute, concurrent_calls, stream_max_duration_seconds and stream_idle_timeout_seconds.
• ceilings: the most the key or its workspace may set the two stream limits to.

It never returns a provider, a model, an endpoint, a region or a credential. Errors are 503 route_unavailable with a retry message.

Keys and limits#

Inference keys belong to a workspace. A workspace owner or admin session can manage them. A primary workspace key with the write scope can also mint and revoke keys, but only for its own workspace and only the keys it minted.

The plaintext key is returned once, at mint, with a notice that it cannot be retrieved again. Only its hash is stored. A primary workspace key that mints is refused with 409 inference_key_cap_reached when the workspace already holds its limit of active keys. The reply names the limit.

Limits resolve in order: the key, then the workspace, then the organisation, then the tier and the platform. Each value comes with its source and its ceiling.

A PUT body is { "limits": { "<key>": <value> }, "reason": "..." }. The reason is up to 500 characters. A value above its ceiling is clamped, and the reply lists every clamp. Each change is audited with the actor and the old and new values.

The self-service limit keys are inference.tokens_per_call, inference.message_chars, inference.request_body_bytes, inference.tokens_per_day, inference.spend_usd_per_day, inference.stream_max_duration_seconds, inference.stream_idle_timeout_seconds and inference.max_output_tokens. Other limits are set by the platform operator.

A per-subject budget applies to one consumer-asserted subject. It limits inference.tenant.tokens_per_day and inference.tenant.spend_usd_per_day for that subject. The subject is never used to decide who may call the endpoint. A subject with no budget of its own reads its source as default.

Receipts and cost#

Every call writes one receipt. A receipt holds the calling key's id (never the key), the resolved labels, token counts, latency, the gate outcome (permit, refuse or replay), the refusal code if there was one, the routing decision, app_id and end_user_id when you sent them, and a cost block for admitted calls. Your prompt, your completion, your credentials and any provider key are never written to a receipt, a log or an error body.

Cost is reported per call on the response (cost). The receipts are the record of spend. Spend is grouped by day or by caller_reference through the costs endpoint.