Training demo
See which training work was removed.
Inspect completed baseline-versus-optimized runs from the integrated training harness. Compare work, validation and measured execution costs.
Qwen/Qwen2.5-1.5B
- Method
- LoRA continued pretraining
- Dataset
- HuggingFaceFW/fineweb-edu
- Hardware
- NVIDIA H100 NVL 94 GB
- Scope
- Frozen confirmatory seeds · sequence length 256
Model revision, initialization, optimizer and evaluator settings
{
"dataset": {
"development_sequences": 512,
"domains": 4,
"evaluation_sequences": 256,
"expected_sha256": "b393f51fefab26cd6f4c8f65707c1924f6666c4961a0ebebe04bb57f7ec832de",
"filename": "sample/10BT/013_00000.parquet",
"minimum_document_tokens": 320,
"monitor_sequences": 128,
"repo_id": "HuggingFaceFW/fineweb-edu",
"revision": "05c1931294b0d1379055d1f802d369f2c3bb2f4b",
"sentinel_sequences": 64,
"train_sequences": 2048
},
"exact_targets_online_permitted": false,
"gates": {
"dynamic_turnover_lower95": 0.2,
"dynamic_turnover_median": 0.3,
"economic_flop_reduction": 0.05,
"economic_quality_vs_cheap": 0.005,
"economic_quality_vs_full": 0.02,
"economic_wall_reduction": 0.05,
"maximum_static_failure_rate": 0.125,
"quality_margin_nats": 0.02,
"regret_advantage": 0.0001,
"spearman_advantage": 0.05
},
"model": {
"attention_implementation": "sdpa",
"dtype": "bfloat16",
"lora_alpha": 16,
"lora_dropout": 0.0,
"lora_rank": 8,
"repo_id": "Qwen/Qwen2.5-1.5B",
"revision": "cf341214a7c12d36db6c7fbbf8b113c7ed61502f",
"sequence_length": 256,
"target_modules": [
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj"
],
"tiny_heads": 0,
"tiny_hidden_size": 0,
"tiny_layers": 0,
"vocab_size": 0
},
"optimization": {
"betas": [
0.9,
0.95
],
"epochs": 1,
"epsilon": 1e-08,
"gradient_accumulation_steps": 1,
"gradient_clip_norm": 1.0,
"learning_rate": 0.0002,
"micro_batch_size": 32,
"use_gradient_checkpointing": true,
"use_torch_compile": false,
"warmup_steps": 8,
"weight_decay": 0.01
},
"primary_dynamic_panel": "constant_mixture",
"sample_sizes": {
"bootstrap_repetitions": 5000,
"confirmatory_seeds": [
710001,
710002,
710003,
710004,
710005,
710006,
710007,
710008
],
"development_seeds": [
410001,
410002,
410003,
410004
]
},
"selection": {
"audit_candidates": 24,
"cheap_methods": [
"random",
"loss",
"metadata",
"diversity"
],
"learned_removal_fractions": [
0.25,
0.33,
0.4,
0.5
],
"long_target_horizon": 8,
"online_monitor_sequences": 8,
"online_probe_batch_size": 4,
"online_probe_candidates": 3,
"ordinary_methods": [
"random",
"loss",
"metadata",
"diversity",
"grad_norm",
"influence",
"adam_influence"
],
"primary_removal_fraction": 0.25,
"projection_dimension": 128,
"refresh_checkpoints": [
0.0,
0.5,
0.8
],
"removal_fractions": [
0.1,
0.15,
0.2,
0.25
],
"ridge_l2_grid": [
0.0001,
0.001,
0.01,
0.1,
1.0,
10.0,
100.0
],
"short_probe_horizon": 2
}
}Live training is not enabled for this public demo. The optimized runner's bounded execution contract and an operator-approved compute budget must be connected before a public run can start. Completed harness evidence is available below.
No job is queued by this page. Inspect a completed run below.
Choose a recorded comparison
Each candidate is paired with the saved full baseline from the same confirmatory seed, initialization and workload. The baseline is reused, not rerun.
What the full experiment established
T3_DYNAMIC_REDUNDANCY_NO_COUNTERFACTUAL_ADVANTAGE
The registered economic gate did not pass. Example removal, wall time and dollar savings are different measurements.
H1 · Static redundancy — passed
A cheap static selector achieved registered physical removal within the quality envelope.
{
"failure_count": 0,
"failure_rate": 0.0,
"method": "random",
"quality_delta": {
"lower_95": -0.00020351035922194782,
"observed": 0.0005031062107461182,
"upper_95": 0.0011676563936120488
},
"realized_removal_fraction": {
"lower_95": 0.25,
"observed": 0.25,
"upper_95": 0.25
},
"registered_fraction": 0.25
}H2 · Dynamic redundancy — passed
The exact dispensable set changed materially during constant-mixture training.
{
"primary_panel": "constant_mixture",
"turnover": {
"lower_95": 0.6666666666666666,
"observed": 0.7916666666666666,
"upper_95": 0.8333333333333334
}
}H3 · Long-horizon counterfactual value — did not pass
Incremental long-horizon value beyond the strongest ordinary baseline was not established.
{
"baseline": "adam_influence",
"regret_advantage": {
"lower_95": -0.00030220380795527757,
"observed": 1.0012957005608222e-05,
"upper_95": 0.0002456764769707871
},
"spearman_advantage": {
"lower_95": -0.3550724637681159,
"observed": -0.037681159420289864,
"upper_95": 0.08405797101449279
}
}H4 · Fully charged economics — did not pass
Fully charged economic superiority over the best cheap policy was not established.
{
"cheap_policy": {
"eligible": true,
"median_flops": 912387238133760.0,
"median_quality_delta": -0.0015877368403416359,
"median_wall_seconds": 6.803864163695835,
"method": "random",
"mode": "dynamic",
"removal_fraction": 0.25
},
"flop_reduction_vs_cheap": {
"lower_95": 0.21419270833333337,
"observed": 0.21419270833333337,
"upper_95": 0.21419270833333337
},
"learned_policy": {
"complete": true,
"eligible": true,
"failure_count": 0,
"failure_rate": 0.0,
"median_flops": 1043067701928960.0,
"median_quality_delta_vs_cheap": -0.0006051605822994244,
"median_quality_delta_vs_full": -0.0022033093022364447,
"median_realized_removal_fraction": 0.5,
"median_wall_seconds": 12.834182432619855,
"method": "counterfactual_surrogate",
"mode": "learned",
"removal_fraction": 0.5
},
"quality_delta_vs_cheap": {
"lower_95": 0.0009249818091299211,
"observed": 0.0017759584913068505,
"upper_95": 0.0024974748200063956
},
"quality_delta_vs_full": {
"lower_95": 0.0014373480104934444,
"observed": 0.002526885387944322,
"upper_95": 0.0030464022767313637
},
"wall_reduction_vs_cheap": {
"lower_95": 0.025741425728295964,
"observed": 0.02781383793775022,
"upper_95": 0.031607205848334374
}
}