Full KV cache
The reference execution retains the full cache. It supplies the matched baseline for the selected workload.
Inference demo
See what KV compression changes.
Compare baseline, fixed cache compression and router-selected execution within a supported deployment scope.
Supported model and execution scope
- Model
- Qwen/Qwen3-8B
- Frozen model revision
b968826d9c46dd6066d109eabc6255188de91218- Policy settings
- Fixed-policy paired harness: greedy decoding, 12 new tokens, batch size 1. Router contract: greedy decoding, at most 16 new tokens.
- Comparison protocol
- Sequential arms on a compatible, exclusive backend. The located fixed-policy scope is NVIDIA A100/CUDA; the learned router scope is Apple MPS. These require separate qualified comparisons. Cross-hardware timings cannot establish a speed comparison.
Fixed SnapKV policy
The existing white-box runtime physically evicts cache positions. Measured resident bytes are distinct from total device memory, latency and cost.
Per-model cache allocation
The calibrated router selects a supported allocation or falls back to full cache. Recorded decisions explain what ran; they do not certify an individual answer.
Compare an approved workload
Live comparison is not configured for this public deployment. No benchmark has been launched.
The comparison launcher requires a configured public preset, compatible harness protocol and bounded operator-owned capacity. The existing live endpoint remains available through its separately configured snippet below.
Try a single live query
This calls the existing sponsored endpoint. It produces an answer, not a matched benchmark or quality certificate. The returned execution label identifies whether the request used an optimized policy or baseline.
Executable API exampleTry your own prompt
Live inference
POST /v1/demo/inference
import json
from pyodide.http import pyfetch
response = await pyfetch(
"https://api.reductionofstates.com/v1/demo/inference",
method="POST",
headers={"Content-Type": "application/json"},
body=json.dumps({"prompt": "Explain KV caching."}),
credentials="omit",
)
response.raise_for_status()
result = await response.json()
print(result["choices"][0]["message"]["content"])const response = await fetch(
"https://api.reductionofstates.com/v1/demo/inference",
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt: "Explain KV caching." }),
credentials: "omit",
},
);
if (!response.ok) throw new Error(`Gateway returned ${response.status}`);
const result = await response.json();
console.log(result.choices[0].message.content);curl https://api.reductionofstates.com/v1/demo/inference \
-H 'Content-Type: application/json' \
-d '{"prompt":"Explain KV caching."}'Model response
A single inference request. Savings require a workload-specific comparison.
When approval no longer matches
Existing enforcement binds approval to the exact deployed identity. Revoked, expired or mismatched approvals cannot authorize an optimized policy. The runtime uses the full baseline when safe fallback is available and rejects the request otherwise.
Inspect the evidence and enforcement workflow