Requirements you must meet¶

Every binding constraint, grouped by what it constrains, with a note on how to tell whether you comply. Rules marked have a failure action of reject the submission.

The policy repository wins

This page summarises policy in submitter language. It does not restate it. Where this page and mlcommons/endpoints_policies disagree, the policy repository is correct and this page is a bug. Please report it.

Frontier structure¶

From §5.3, §5.4 and §5.6, with failure actions from §9.1.

Requirement	How to check
≥ 8 measurement points including a dedicated Offline run; ≥ 7 if you elect `C_max` as the Offline result, or for an agentic benchmark	`endpoints-submission-cli check-submission` — rule `point-count`
≤ 32 measurement points, Offline included	rule `point-cap`
Exactly one Offline point for a non-agentic benchmark, none for an agentic one	rule `offline-point-present`
≥ 1 point in Ultra Low Concurrency (1–`C_min`, `C_min` ≤ 32)	rule `ultra-low-concurrency-coverage`
`c_min_override`, if used: an integer in 1–32 that matches a submitted point, the same on every point	rule `c-min-override` (checker `v1.5.1.0` or later)
≥ 1 point in each of Low, Medium, High Concurrency	rules `low-` / `med-` / `high-concurrency-coverage`
`C_max` declared and > 32	rules `system-description-valid` (missing) and `region-computation` (≤ 32)
Every concurrency falls in a valid region (a dedicated Offline run is exempt)	rule `concurrency-in-range`

Regions are computed in log-2 space from your own C_min and C_max. See Plan your Pareto frontier and Metrics and regions.

How the checker knows a benchmark is agentic

From the load pattern. A frontier whose points all use agentic_inference is checked as agentic; anything else is checked as single-turn, where a missing Offline point fails.

The Offline point¶

Required for non-agentic benchmarks, and not allowed for agentic ones (§5.7). You either make a dedicated Offline run or elect your C_max point; both are defined in §5.7.2, and what an Offline run is (dataset-sized concurrency, no TTFT requirement, reordering only within a pass) is in §5.7.1. What the checker does with it:

Requirement	How to check
Declaration: `offline: dedicated` or `offline: elected` in that point's `point.yaml`; `elected` only on the `C_max` point	rule `offline-point-present`
Dedicated run: `system_tps` ≥ 0.98 × the `C_max` point's, and concurrency ≥ `C_max`	rule `offline-ordering` (flagged). The throughput half is skipped silently if no point sits at exactly `C_max`
Dedicated run uses the Offline load pattern	Not checked: `load-pattern` exempts a dedicated Offline point

A dedicated run counts toward no region. An elected point keeps its own region and reports latency like any other point.

Per-point run requirements¶

Defined in §6 of the rules. The values at a glance:

Requirement	Value	How to check
Load pattern (§6.1)	Fixed-concurrency for every point except a dedicated Offline run. `poisson` is invalid	rule `load-pattern`
Steady-state duration (§6.2)	600 s at concurrency 1–32. Above 32: 1,200 s, or 2,400 s for agentic benchmarks	rule `point-duration`, from checker `v1.4.1.0`. Earlier releases go by region and don't check the agentic 2,400 s
Completed queries (§6.4)	At least one pass over the dataset, or the approved subset at concurrency 1–32	rule `min-query-count`
Samples issued (§6.4)	A whole-number multiple of the dataset size	—
Streaming (§6.5)	Streaming responses on every fixed-concurrency point; a dedicated Offline run is exempt. `stream_all_chunks` may be either value	rule `streaming-config`, which still requires `stream_all_chunks: true`
Sampling order (§6.5)	Performance: with replacement. Accuracy: without replacement	—

Section 6 is pending ratification

The duration and query-count values are marked as example values subject to working-group ratification, though they are stated as locked and Task Force-approved for the v0.7 round. Check Open questions before planning around them.

Warmup¶

Optional, capped at 24 hours per point, and it must not use performance-dataset samples; if the client does use the performance dataset, salting must be enabled (§6.3.1). Every point declares six warmup fields (§6.3.3), checked by rule warmup-present. Salting is not on by default in the client, so if your warmup uses the performance dataset you have to turn it on yourself; rule warmup-salt then warns, and reviewers may ask about it. Incomplete warmup documentation is grounds for a Methodology objection.

Accuracy¶

Accuracy validation runs are needed at the four mandatory region points plus the Offline point: 5 for non-agentic benchmarks, 4 for agentic (§5.3). How they're judged, single-turn per point or multi-turn on the average, is in §4.3. That count is fixed: submit exactly one validation result per mandatory point, none at the submitter's-choice points, because extra results are rejected at automated compliance (§4.3, §9.1). If you elect your C_max point as the Offline result, that point's one run covers both (§5.7.2). Accuracy runs use the same configuration and stack as the performance runs, on the un-salted accuracy dataset, and there is no tolerance (Reproducibility Expectations).

With c_min_override, the Ultra Low validation result can come from any point you designate in 1–C_min; without it, it comes from the point at C_min (§4.3).

Agentic: inline accuracy at every point

Every agentic point, submitter's-choice points included, also reports an inline score from its own performance run, and each must pass on its own. Inline scores don't count toward the four validation runs and aren't part of the SWE-bench mean (§4.3).

Requirement	How to check
Accuracy results at every required point	rule `accuracy-coverage`
The results pass the quality target	rule `accuracy-gate`, or for the agentic models `agentic-accuracy-inline`, `agentic-accuracy-swebench` and `agentic-osl-range`

The targets are not published

The per-benchmark targets are marked [WIP] in §2.9.8. For the legacy benchmarks the checker gates against the MLPerf Inference targets (see Benchmarks and models). For the agentic benchmarks it uses the targets in the client's agentic README. Tracked as C2 in Open questions.

Seeds¶

Every point records the same published seed set, adopted within its window and bound for the life of the submission (Submission Rules §4.6). The checker tests this as seed-set-consistency and seed-set-membership; the exact test, including the rule for amendments, is the Seed-set validity row of §9.1. The per-query salt is seeded from the same set and has its own requirements in §2.9.5.1.

Consistency across the frontier¶

Same model, endpoint configuration, software stack and seed set at every point (Configuration consistency in §9.1). The checker also requires the same dataset (config-consistency-dataset). Every point's system_desc.json must describe the same system, and the whole frontier uses one system_power.json, and only the per-point node count may vary (§4.5.3). If you use speculative decoding, it's the same drafter at every point. Freeze the stack before the first run.

Power normalization¶

Standardized results, CoP and CoN, are normalised by provisioned power, estimated from the rated power of CPUs, accelerators, scale-up switches and, for multi-node systems, scale-out NICs, plus a fixed overhead, with scale-out switch power added outside that overhead. The model is in Power Model, the evidence standard and what MLCommons does with gaps in Methodology, partial racks, mixed racks and node rounding in §4.5.2.1, and the per-point metric in §4.5.3. RDI may report it; Serviced is deferred.

Each point is normalized per node set. Run every point with as much of the provisioned hardware engaged as fits (data-parallel replication, DP); a shortfall goes in dp_shortfall in point.yaml (§4.5.3). A multi-node, rack-scale or heterogeneous system needs a complete system_power.json.

Requirement	How to check
`system_power.json` for each system, at `results/<system>/system_power.json`	rule `power-descriptor`
`system_tps_per_kw = system_tps / point_power_kw`	rule `metric-consistency-tps-per-kw`; the CLI release this page was verified against checks `system_tps / provisioned_power_kw`
Values you leave out are estimated by MLCommons and the result tagged "MLC Estimated Power"	rule `power-estimated` warns

Field names and an example: system_power.json.

Section 4.5 is pending ratification

The tiers, overhead fractions, reference components and even the metric's name and units are marked as subject to change (§4.5).

Disclosure by division¶

What each division has to disclose (source code, hardware, parallelism mapping, software stack) is tabulated in §2.7 Transparency Requirements. The minimum software list for Standardized and RDI is in §8.4. Serviced additionally discloses pricing and rate limits (§2.3.1).

Reproducibility¶

Standardized must be reproducible by a third party on a third-party system; Serviced on the public endpoint; RDI has no requirement (§2.6). The variability margins reviewers apply (10% independent re-run, 5% same system, throughput only) are in Reproducibility Expectations.

Tokenization¶

Official output token counts come from the client-side reference tokenizer, applied once to the reconstructed response through the model's reference chat template, not from your serving stack (§2.8). Visible output, tool calls and reasoning all count.

Padding and duplication invalidate the point

Duplicating content across response fields, or padding any field, to inflate token counts invalidates the affected measurement point (§2.8). Reviewers also look for leading content emitted only to stop the TTFT clock (§9.2).