Jobs lifecycle: queue, retries and dead letters

The statuses#

A job reads one of six statuses from GET /api/v1/sidekick/jobs/{id}, or from the status events of its stream. The status is the caller-facing vocabulary. Internal states are mapped onto it.

A terminal job sends one result to its callback, if it has one, and its result_envelope is set. Before a job is terminal, result_envelope is null.

Queueing and position#

Creating a job with POST /api/v1/jobs returns 202 and the job starts as queued. The same Idempotency-Key with the same body returns 200 with the original job_id. The same key with a different body is refused.

Jobs wait in one queue per tenant and take turns with other tenants, so one tenant's backlog does not hold back another. While a job is queued its status read carries:

• queue_position: its place in its own tenant's queue, where 1 is next. It is null when the job is not queued.
• jobs_ahead: an estimate of how many claimable jobs the claim would hand out before this one, across the whole claim pool the job competes in. It is a count only and names nothing of any other tenant. It is capped at a platform limit. Above the cap, jobs_ahead reads the cap and jobs_ahead_capped is true.
• started_at: when the job first started. It is kept when a lost worker returns the job to the queue.

The stream sends a new status event each time the position moves.

Concurrency, tenants and exclusive groups#

• Tenant concurrency. A job over its tenant's concurrent-run cap is accepted and reads as queued. It starts when a slot frees.
• The only queue refusal. A workspace whose queue has reached its ceiling is refused with 429 queue_saturated and a Retry-After header. A tenant whose own queue is full gets the same status with a tenant scope.
• Exclusive groups. Set exclusive_group on create to name work that must not run at once. It is an opaque label of 1 to 200 printable ASCII characters. When the workspace limit jobs.exclusive_group_max is set, a job whose group already has that many running is skipped by the claim. It is never waited on, so jobs in other groups, and jobs with no group, are handed out past it. Unset is unlimited.
• Limits. A cap that is not set is unlimited. A job whose max_cost_usd is above the platform cap is refused with max_cost_usd_exceeds_cap.

Retries#

A job created with a retry policy runs again after a retryable failure. The policy is:

{
  "retry": {
    "max_attempts": 4,
    "base_delay_s": 5,
    "max_delay_s": 300,
    "retry_on": ["provider_timeout"],
    "deadline_s": 3600
  }
}

• max_attempts is at least 1. It may not exceed the platform cap for the job's kind.
• base_delay_s is the first backoff. It may not be below the platform minimum, which is 1 second unless the platform sets it higher. A policy that names no delay waits 5 seconds, doubling, and never more than 5 minutes.
• max_delay_s caps the backoff. It may not be below base_delay_s.
• retry_on limits retries to the failure codes you name in error.reason_code. When it is absent, any retryable failure is retried.
• deadline_at (an ISO 8601 time) or deadline_s (seconds after the first attempt starts) sets an overall deadline. Not both. The deadline may not be further ahead than the platform maximum, which is 7 days by default.

After failed attempt n, the job waits min(max_delay_s, base_delay_s * 2^(n-1)) seconds, jittered to between half of that and the whole of it. The claim never hands the job out before then. The status shows next_attempt_at, and attempt_count counts how many times the job has been started in its current attempt chain.

These failures are terminal and are never retried: budget_exhausted, secret_in_diff, duration_exceeded, and any reason code that ends in _refused.

A policy above the platform caps is refused with 400 retry_policy_exceeds_cap. A malformed policy, or a base_delay_s below the minimum, is refused with 400 invalid_retry_policy.

A failed job with a retry policy is retried until it succeeds or uses its attempts. A tenant may use a limited number of retries per window, set by the platform. Past that budget a failed job is dead-lettered with reason retry_budget_exhausted, and no other tenant is affected.

Deadlines#

When the deadline has passed, or when the next backoff would start a retry after it, a failing job is dead-lettered with reason deadline_exceeded, whatever attempts are left. An attempt already running at the deadline finishes or fails normally. An expired lease counts as a failing job.

Dead letters#

A dead letter is a job that used its retry attempts, or failed terminally under a retry policy, and is waiting for its owner to requeue or discard it. Its status stays failed.

GET /api/v1/jobs/dead-letters lists them, newest first. Each entry carries reason, attempt_count and last_error.reason_code. The reason is a free string. The reasons in use are attempts_exhausted, terminal_error, retry_not_allowed, retry_budget_exhausted and deadline_exceeded, and more may be added. Page with limit (1 to 200, default 50) and the opaque cursor.

A workspace key sees only the dead letters of the tenant it names, in the X-Tenant-Subject header or the tenant_subject query parameter. A tenant-scoped key that names no tenant is refused with 400 tenant_subject_required. A dead letter of a job with no tenant is listed to a key that names no tenant, but that key cannot requeue or discard it. A workspace member can.

Resolved dead letters, and their attempt records, are deleted after the dead-letter retention period, 30 days by default.

Requeue#

POST /api/v1/jobs/{id}/requeue puts a dead letter back on the queue as a new attempt chain. The attempt count starts again. The job's last checkpoint is kept, so a step it already finished is not repeated.

From a workspace key the body must carry tenant_subject, actor_kind and actor_ref. Without them the request is refused with 403 actor_attestation_required. Another tenant's job, or a job that is not a dead letter, is 404. A tenant that has used its retries for the window is refused with 429 rate_limited and Retry-After. No other tenant is affected.

Discard#

DELETE /api/v1/jobs/{id}/dead-letter discards a dead letter. It leaves the list and the job stays failed. The same tenant and actor rules apply as for requeue.

Stopping a job#

POST /api/v1/jobs/{id}/controls with { "action": "cancel" } stops a job. A cancelled job reads cancelled, with cancelled_at set once. It is never retried, dead-lettered or claimed again. Its callback and its stream carry cancelled.

Runner claims, waits and delays#

A runner gets work with POST /api/v1/runtime/claim.

• Waiting for work. The body may carry wait_seconds, a whole number of seconds, 0 or more. The default is 0, which is a plain claim. When nothing is claimable the request is held up to that many seconds and returns the first job that becomes claimable. The wait is capped by the jobs.claim_max_wait_seconds limit, 20 seconds by default. A value that is not a whole number is refused with 400 invalid_wait_seconds.
• Giving the claim back. A runner that cannot proceed yet calls POST /api/v1/assignments/{id}/release with not_before (an ISO 8601 time) or delay_s (seconds from now), never both. The lease is freed at once. The job returns to the queue as queued, with next_attempt_at set, and it is not claimable before then. A release does not count as an attempt, and the last checkpoint is kept. The time must be in the future, no further ahead than the jobs.release_max_delay_s limit (86400 seconds by default), and before the job's retry deadline. A job released too often is refused with 429 rate_limited. A release never dead-letters a job.

Leases, heartbeats and lapse#

A runner holds a lease on the job it runs. It keeps the lease with POST /api/v1/assignments/{id}/extend, a heartbeat that may carry a checkpoint with seq, step_id and completed_step_ids. A re-claimed job resumes after its last recorded checkpoint.

A run longer than the limit for its job kind is refused with 409 duration_exceeded. The job ends with that reason and keeps its last checkpoint.

A lease that is not extended lapses. The job returns to the queue within the lapse bound:

• on the next claim in the same workspace once the lease expired more than a short grace period ago;
• with no claims at all, through a periodic reconciler.

A lapse counts an attempt, like any other failure. A late extend, complete or fail from the lapsed holder is refused, or changes nothing.

Workspace limit#

Your workspace can read and change this limit through the job limits route, GET and PUT /api/v1/workspaces/{workspaceId}/job-limits, bounded by the platform ceiling for it.