1
Upload a JSONL file
One request per line: a unique custom_id and a text chat completion body.
Pilot · invitation only
Batch inference for open models.
Choose a supported model, submit your requests, and retrieve results and usage in one batch workflow.
No model is offered in the pilot yet. Each model below shows its exact status and what still blocks it.
Model availability
Statuses come from the service catalog. A model is listed as available only after it has been validated on its production hardware.
Target models available in the pilot: 0 / 6
| Model | Status | Execution | What it still needs |
|---|---|---|---|
DeepSeek V4 Flashdeepseek-ai/DeepSeek-V4-Flash | Planned | Native only (optimization not applicable to this architecture) | needs a multi-GPU host (none authorized); encoder parity test with the official encoding not run |
Llama 3.3 70B Instructmeta-llama/Llama-3.3-70B-Instruct | Validated on GPU, not yet offered | Native only (optimization not established) | no price set; no hosted capacity. Validated topology is a tight fit (8k context); the 4x80 GB profile with 32k context is not run |
Qwen3.6 27B (text-only)Qwen/Qwen3.6-27B | Validated on GPU, not yet offered | Native only (optimization not applicable to this architecture) | no price set; no hosted capacity. The 1x80 GB profile is not run (validated topology: 4x40 GB) |
Qwen3 4BQwen/Qwen3-4B | Validated on GPU, not yet offered | Native; an optimized profile runs on GPU, its quality scope is not yet published | no price set; no hosted capacity (validated on 1x A100-SXM4-40GB; Metal profile not run) |
Qwen3 8BQwen/Qwen3-8B | Validated on GPU, not yet offered | Native; an optimized profile runs on GPU, its quality scope is not yet published | no price set; no hosted capacity |
Qwen3 32BQwen/Qwen3-32B | Validated on GPU, not yet offered | Native; an optimized profile runs on GPU, its quality scope is not yet published | no price set; no hosted capacity. The 2x80 GB TP=2 profile is not run (validated topologies: 4x40 GB TP=4; 1x80 GB at 8k context) |
How a batch works
2
Create the batch
We validate the whole file before anything runs, pin the model version and execution profile, and reserve the maximum usage.
3
We run it
Rows run concurrently on vLLM. Each result is saved as soon as it finishes, so a restart never repeats a delivered result.
4
Download results and usage
Results are matched to your custom_id, failures are typed per row, and your console shows each batch's token usage and charge.
Delivery and cancellation
Each batch has a completion window set by the service operator. Rows unfinished at the deadline are returned as expired; finished rows are delivered.
Cancelling stops new work at once. Finished rows are kept and charged; rows in progress are stopped and not charged.
If a worker fails, unfinished rows run again. You receive and pay for at most one result per row.
Pricing and quotes
Prices are per million input and output tokens, set per model in a versioned price list, with a minimum charge per batch. Current prices and the minimum are on the home page, and only requests that succeed are billed.
Pilot customers receive a written quote and a spending authorization. A batch that would exceed it is refused before it runs, and a model without a price is never run at no charge.
Usage statements come from recorded usage. Payment processing is not live in the pilot.
Data handling
Your prompts and results are used only to run your batch. They are not used to train, tune or select models.
Uploaded files and results are deleted when you delete them or when the retention period ends. A usage record without prompt or output text is kept for billing.
Each customer has a separate prompt-cache namespace on the model server.
Native and optimized execution
Every model is offered first with native vLLM execution. An optimized profile, such as our KV-cache retention method, is a separate opt-in offering. It has run on GPU, but it is offered only with a published quality scope that you accept, because its outputs can differ from native execution.
Pilot onboarding
- Tell us the model, typical prompt and output lengths, volume, and the delivery window you need.
- We confirm availability, send a quote, and enable batch inference for your workspace. You create API keys in the console.
- Run a small sample batch, check the results and the receipt, then send production batches.
Later
Fine-tuning of supported models and a separate optimizer product are planned. Neither is available today.