Inference optimization
Compare baseline, KV compression and routing.
Use the existing cache-compression service and per-model router. Inspect what ran, what was measured and where the policy applies.
Explore inferenceBatch inference · pilot
Batch inference for open models
Send many chat requests as one JSONL file, run them on a supported open model, and download the results when the batch finishes. Batches use the same OpenAI-compatible API and keys as real-time inference, and only the requests that succeed are billed.
No model is offered in the pilot yet. Each model below shows its exact status and what still blocks it.
Recorded examples
Inspect what the platform has already run.
Training optimization
Compare training work and actual cost.
Inspect baseline and optimized procedures from the existing training harness, including validation, preparation and execution costs.
Explore trainingTry your own prompt
Executable API exampleTry your own prompt
Live inference
POST /v1/demo/inference
import json
from pyodide.http import pyfetch
response = await pyfetch(
"https://api.reductionofstates.com/v1/demo/inference",
method="POST",
headers={"Content-Type": "application/json"},
body=json.dumps({"prompt": "Explain KV caching."}),
credentials="omit",
)
response.raise_for_status()
result = await response.json()
print(result["choices"][0]["message"]["content"])const response = await fetch(
"https://api.reductionofstates.com/v1/demo/inference",
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt: "Explain KV caching." }),
credentials: "omit",
},
);
if (!response.ok) throw new Error(`Gateway returned ${response.status}`);
const result = await response.json();
console.log(result.choices[0].message.content);curl https://api.reductionofstates.com/v1/demo/inference \
-H 'Content-Type: application/json' \
-d '{"prompt":"Explain KV caching."}'Model response
A single inference request. Savings require a workload-specific comparison.
Scoped evidence
The result and its limits, together.
These are registered experimental results. They do not authorize a different model, workload, backend, or hardware configuration. Your workload still needs its own measurement.
Example retention · With early stopping
9.40% less job time
Keeping 6 of every 8 examples after a 64-update full-data warmup reduced complete-job time by a paired geometric mean of 9.40% across 24 paired Qwen2.5-1.5B LoRA jobs on one H100 PCIe, with held-out loss inside the registered quality margin.
The one-sided 95% lower bound is 6.01%. That misses the 10% lower bound the study's frozen protocol set as its benchmark target; it does not mean the saving is absent. Held-out loss was slightly higher than the full run's. Pairs whose arms stopped at different updates account for most of the spread.
Where it did not help
The same selection policy showed no net benefit on two other workloads: a 23.2M-parameter model trained from scratch on an Apple M5 Pro, where held-out loss worsened beyond the quality margin, and a Phi-4 adapter fine-tune on an A100, where the reduced run took longer end to end. Results are model-, workload-, recipe- and hardware-specific.
Example retention · Fixed-length jobs
13.82% less job time
Twelve paired one-epoch Qwen2.5-1.5B LoRA jobs on one H100 NVL, retaining 1,536 of 2,048 examples, finished with 13.82% less complete-job time (geometric mean; two-sided 90% interval 13.00%–14.64%), and the saved checkpoints passed the registered quality rule.
An earlier, separate study with a different recipe and GPU, run without early stopping. It does not establish savings for other models, full fine-tuning, pretraining, energy or invoices.
ML-T6 · Bounded adaptation
50% fewer backward examples
The evaluated adaptation pool fell from 4,800 to 2,400 backward examples, with zero observed failures in 149 certification evaluations and a one-sided 95% failure-risk upper bound of 1.99048%.
The registered fully charged wall-time reduction was 1.69%. This does not establish a 50% reduction in training time or cost. End-to-end savings must be measured separately on each training system.
About the pre-cache follow-up
A later pre-cache implementation reported a larger CPU wall-time improvement. It remains follow-up engineering evidence: cache-construction cost and amortization across repeated jobs must be evaluated before drawing a production cost conclusion.
Privacy boundary
Local-first by design.
Run profiling in your own environment. Keep model weights local. Share only the report data you choose. This statement covers the local-first profiling flow. The optional hosted inference gateway is a separate surface whose managed-provider path sends prompts and responses to the configured provider.
Evaluations and Enterprise
Need help with a model and workload?
Leave a work email and we will follow up about scoping an evaluation or an Enterprise agreement. A one-time benchmark does not need a subscription or a conversation.