Batch format and client example
The API follows the OpenAI batch workflow for text chat completions. Only the subset documented here is supported.
Input file
UTF-8 JSONL, one request per line. Each line needs a unique custom_id, method POST, url /v1/chat/completions and a body. Every line in a batch uses the same model.
{"custom_id": "q-0", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "Qwen/Qwen3-0.6B", "max_tokens": 64, "temperature": 0, "messages": [{"role": "user", "content": "What is a batch API?"}]}}Supported request fields
model, messages (system, user and assistant roles with text content), max_tokens or max_completion_tokens, temperature, top_p, stop, seed, presence_penalty, frequency_penalty, and chat_template_kwargs.enable_thinking for Qwen3 models.
Rejected, never ignored
Images, audio, tools and function calling, streaming, n greater than 1, log probabilities and structured output are rejected with an error that names the field. A request longer than the model context is rejected on its own row; nothing is truncated.
Endpoints
POST /v1/files upload a JSONL file (purpose "batch")
GET /v1/files/{id} describe a file
GET /v1/files/{id}/content download input, output or error file
DELETE /v1/files/{id} delete the content now
POST /v1/batches create (optional Idempotency-Key header)
GET /v1/batches | /v1/batches/{id} list or poll
POST /v1/batches/{id}/cancel cancelComplete Python example
Uses the official openai package with the same base URL and API key as real-time inference. Create a key on the API keys page of your console.
"""Upload a JSONL batch, create it, wait for it, and download results - with the official openai package.
Environment: BATCH_API_KEY (an API key from your console), BATCH_API_BASE (default: this site's API gateway),
BATCH_MODEL (a model listed on the Batches page of your console).
"""
import json
import os
import time
from openai import OpenAI
client = OpenAI(base_url=os.environ.get("BATCH_API_BASE", "https://api.reductionofstates.com/v1"),
api_key=os.environ["BATCH_API_KEY"])
model = os.environ.get("BATCH_MODEL", "Qwen/Qwen3-0.6B")
questions = ["What is a batch API?", "Name three rivers in Europe.", "Explain JSONL in one sentence."]
with open("requests.jsonl", "w") as f:
for i, question in enumerate(questions):
f.write(json.dumps({"custom_id": f"q-{i}", "method": "POST", "url": "/v1/chat/completions",
"body": {"model": model, "max_tokens": 64, "temperature": 0,
"messages": [{"role": "user", "content": question}]}}) + "\n")
batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=batch_file.id, endpoint="/v1/chat/completions",
completion_window="24h",
extra_headers={"Idempotency-Key": f"example-{batch_file.id}"})
while batch.status not in ("completed", "failed", "cancelled", "expired"):
time.sleep(2)
batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)
if batch.errors:
for e in batch.errors.data:
print("validation error:", e.line, e.code, e.message)
if batch.output_file_id:
for line in client.files.content(batch.output_file_id).text.splitlines():
row = json.loads(line)
print(row["custom_id"], "->", row["response"]["body"]["choices"][0]["message"]["content"])
if batch.error_file_id:
for line in client.files.content(batch.error_file_id).text.splitlines():
row = json.loads(line)
print(row["custom_id"], "error:", row["error"]["code"], row["error"]["message"])
Results
Successful rows are in the output file. Failed, cancelled and expired rows are in the error file with a typed code. Rows are matched by custom_id; their order is not guaranteed.
Differences from the OpenAI Batch API
Only /v1/chat/completions batches are supported. Completion windows are set by the operator. Batch objects add x_ fields with the internal state and token usage, and a queued batch reports in_progress. Each batch's charge appears on the Batches page of your console.
Local evaluation
Pilot participants can run the complete service and a small open model on one Apple Silicon Mac, without our infrastructure, to check the workflow before sending data.