Compliance checks¶
Every automated check, cross-walked from checker rule ID to the rules clause it enforces and the failure action the rules assign. Use it to work out what a failed check actually means.
Run locally with endpoints-submission-cli check-submission (documented upstream as
submission-checker,
see B16); run server-side during Week 0. This page describes checker v1.1.0.0.
Versions before v1.0.1.0 lack the Offline, power, accuracy-coverage, steady-state and drafter
rules, and versions before v1.1.0.0 don't know the agentic models or their accuracy targets.
Week 0 failures reject the submission
A submission that fails any automated check by the end of Week 0 is rejected (§6.1 of the Submission Rules). You resubmit as a new submission — there is no in-place patching, and you lose your cohort slot.
How to read the severity column¶
| Severity | Meaning |
|---|---|
| Reject | The rules assign Reject submission to this check |
| Reject points | Non-conforming points are rejected |
| Flag | Flagged for reviewer attention — does not block automatically, but is objection material |
| Warn | Checker-level warning; becomes an error under --strict |
Local severity can be stricter than §9.1
The icons show the failure action in the rules. The checker sets its own severity, and most
Flag rows are errors locally, which stops submissions create:
system-description-consistency, model-name-consistency, tps-utilization,
concurrency-in-range, streaming-config, min-query-count, warmup-present,
benchmark-type-consistency, config-consistency-dataset, config-consistency-model and every
metric-consistency-* rule. Of the Warn rows,
steady-state-valid, steady-state-consistency, seed-config-legacy (a seed other than 42)
and region-basis (no parsable point.yaml) can also fail as errors.
Structure¶
| Rule ID | Clause | Checks | Severity |
|---|---|---|---|
path-exists |
§1 | Submission root directory exists | |
required-dir |
§1 | results/ and docs/ present |
|
src-dir |
§2.2.1 | src/ present with at least one implementation directory |
|
src-readme |
§2.2.1 | Each src/<implementation>/ has a README.md |
|
system-results-dir |
§1 | At least one results/<system>/ exists |
|
benchmark-model-dir |
§1 | At least one model directory per system | |
point-dirs |
§1 | At least one r<N>/ point directory per model |
|
measurement-points-present |
§1 | Every r<N>/ carries a point.yaml |
|
result-summary-present |
§1 | result_summary.json exists for each point |
|
shared-path-resolution |
§9.1 | shared_src / shared_docs resolve under the submission root |
|
point-dirname-concurrency |
§1 | r<N>/ name matches the declared concurrency |
System description¶
| Rule ID | Clause | Checks | Severity |
|---|---|---|---|
system-description-present |
§8.2 | Every point has a system_desc.json |
|
system-description-valid |
§8.2 | Parses against the SystemDescription schema |
|
system-description-consistency |
§9.1 | Every point of a curve describes the same system | |
model-name-valid |
§3.2 | model_name is one of the round's supported models |
|
model-name-consistency |
§8.2 | Matches the results directory name | |
max-concurrency-declared |
§9.1 | Reports max_supported_concurrency. A missing value fails system-description-valid, and a value ≤ 32 fails region-computation |
|
tps-utilization |
§8.2 | Equals system_tps / max(system_tps) over the point's own curve |
|
power-descriptor |
§4.5.2, §9.1 | results/<system>/system_power.json exists and a total power can be derived from it |
|
power-estimated |
§4.5.2 | Names component groups left for MLCommons to fill in — the result will carry "MLC Estimated Power" |
Regions and curve structure¶
| Rule ID | Clause | Checks | Severity |
|---|---|---|---|
region-basis |
§5.4 | Reports the derived C_min and how many points it came from |
|
region-computation |
§5.5 | (C_max, C_min) is a valid input to the reference algorithm |
|
concurrency-in-range |
§9.1 | Each concurrency falls in a valid region, margin included. A dedicated Offline run is exempt | |
region-declared |
§8.3 | Declared region is one of the permitted values |
|
region-placement |
§8.3 | Declared region matches the computed one | |
offline-declared |
§8.3 | offline is dedicated, elected or none |
|
offline-point-present |
§5.7, §9.1 | Single-turn: exactly one Offline point, and elected only on the C_max point. Agentic: none |
|
benchmark-type-consistency |
§6.1, §9.1 | Every point of the curve uses the same load pattern, so the curve is either agentic or single-turn | |
offline-ordering |
§5.7.2, §9.1 | Dedicated Offline run: system_tps ≥ 0.98 × the C_max point's, and concurrency ≥ C_max |
|
ultra-low-concurrency-coverage |
§5.4, §9.1 | At least one point at concurrency ≤ 32 | |
low-concurrency-coverage |
§9.1 | At least one point in Low Concurrency | |
med-concurrency-coverage |
§9.1 | At least one point in Medium Concurrency | |
high-concurrency-coverage |
§9.1 | At least one point in High Concurrency | |
point-count |
§5.3, §9.1 | At least 8 points when one declares offline: dedicated, otherwise at least 7 |
|
point-cap |
§5.6, §9.1 | Does not exceed 32 points, Offline included |
The 10% margin does not satisfy High Concurrency
A point in C_max + 1 … ceil(1.10 × C_max) is in its own region. It passes
concurrency-in-range but does not count toward high-concurrency-coverage. The rules say
a dedicated Offline run counts toward no region (§5.7.2), but the checker's
coverage checks don't exclude it: a dedicated run at exactly C_max is counted as High
Concurrency.
How the checker tells agentic from single-turn
Nothing in point.yaml names the benchmark type, so the checker reads it from the load pattern.
A curve whose points all use agentic_inference is agentic: it may not have an Offline point,
and the minimum is 7 points. Anything else is single-turn, and a missing Offline point fails
offline-point-present. If the points disagree, benchmark-type-consistency fails and the
curve is checked as single-turn.
offline-ordering needs a point at exactly C_max
The throughput half of the check looks up the point whose concurrency equals your declared
C_max. If there isn't one, it skips the comparison without reporting anything.
Measurement points¶
| Rule ID | Clause | Checks | Severity |
|---|---|---|---|
point-config-valid |
§8.3 | point.yaml parses against the PointConfig schema |
|
point-disclosure-complete |
§8.3 | Every required disclosure field is present | |
load-pattern |
§6.1 | load_pattern is concurrency, or agentic_inference for an agentic benchmark, with a positive level. A point declaring offline: dedicated is exempt, and no Offline pattern name is tested |
|
streaming-config |
§6.5, §9.1 | stream_all_chunks is True |
|
point-duration |
§6.2 | The steady-state window's issue-time span meets the region's minimum | |
steady-state-valid |
§4.4, §8.3 | status, verdict and gating state values are from the permitted vocabulary |
|
steady-state-consistency |
§4.4 | The declared status agrees with the window it describes |
|
steady-state-basis |
§4.4 | Reports whether steady-state or total supplies the official result; warns on fallback and drift |
|
min-query-count |
§6.4 | n_samples_completed meets the dataset minimum |
|
warmup-present |
§6.3.3 | Warmup declaration present | |
warmup-logs-retained |
§6.3.2 | Log retention declared | |
warmup-salt |
§6.3.3 | Warns when the warmup salt is enabled | |
config-consistency-dataset |
§9.1 | All points use the same dataset | |
config-consistency-model |
§8.5, §9.1 | All points declare the same model_name in point.yaml |
The rules no longer require stream_all_chunks: true
Rules §6.5 now requires streaming responses (model_params.streaming resolving to on)
on fixed-concurrency performance runs, exempts a dedicated Offline run, and allows either value
of stream_all_chunks. Checker v1.1.0.0 hasn't caught up: streaming-config still fails a
point whose point.yaml records stream_all_chunks: false, and nothing checks
model_params.streaming. Run with stream_all_chunks: true until the checker changes.
Seed binding¶
| Rule ID | Clause | Checks | Severity |
|---|---|---|---|
seed-set-consistency |
§9.1 | Every point records the same seed set | |
seed-set-membership |
§9.1 | The bound set is one MLCommons published | |
seed-runtime-match |
§2.1.1 | The RNG seeds equal the bound set's values | |
target-cohort |
§4.6 | target_cohort matches YYYY-MM-C0 / YYYY-MM-C1 |
|
seed-set-adoption |
§4.6 | Set published for the target cohort or the three before it | |
seed-config-legacy |
§4.6 | v0.7 fallback — seeds equal 42 when no seed_set is declared |
|
seed-set-registry |
§4.6 | Warns when the seed-set file cannot be read |
seed-set-adoption now runs
From v1.0.1.0 the bundled file mirrors the published seedset.yaml, cohort key included, and
the adoption window is derived from it. Older checkers report SKIP here; upgrade rather than
override. See step 4.
Speculative decoding¶
| Rule ID | Clause | Checks | Severity |
|---|---|---|---|
approved-drafter |
§2.9.4, §9.1 | The declared drafter is on the benchmark's published list, by weight checksum or by target checksum plus configuration | |
drafter-approval-lead-time |
§2.9.4, §9.1 | It was approved at least two cohorts before target_cohort |
|
drafter-list-registry |
§2.9.4 | Warns when the drafter list itself can't be read |
The bundled list is empty
The reference repository lists approved heads for the agentic benchmarks only, and the checker
hasn't picked them up: it ships an empty list and rejects any point that uses speculative
decoding. Point --approved-drafters FILE or $MLPERF_ENDPOINTS_APPROVED_DRAFTERS at a
published list once one exists.
Amendments are not re-tested for adoption
For an amendment, every new or replacement point must match the original submission's bound seed set. The four-cohort adoption test is not reapplied using the amendment's later cohort.
Metrics¶
| Rule ID | Clause | Checks | Severity |
|---|---|---|---|
result-file-valid |
§8.3 | result_summary.json parses against PointSummary |
|
metric-consistency-duration |
§9.1 | duration_ns > 0 |
|
metric-consistency-accounting |
§9.1 | completed + failed == issued |
|
metric-consistency-output-tokens |
§9.1 | total_output_tokens ≥ 0 |
|
metric-consistency-system-tps |
§9.1 | Stored system_tps matches the derived value |
|
metric-consistency-tpot-p90 |
§9.1 | Reported TPOT P90 present, finite, strictly positive | |
metric-consistency-tps-per-user |
§9.1 | Stored tps_per_user matches 1000 / tpot_p90_ms |
|
metric-consistency-tps-per-kw |
§4.5.3 | A stored system_tps_per_kw matches system_tps / provisioned_power_kw |
|
agentic-metric-consistency |
§4.1, §9.1 | e2e_avg_interactivity is derivable from its reported inputs |
Accuracy¶
| Rule ID | Clause | Checks | Severity |
|---|---|---|---|
accuracy-present |
§6.6, §9.1 | At least one model carries accuracy results | |
accuracy-coverage |
§5.3, §9.1 | Accuracy results at a point in each of Ultra Low, Low, Medium and High Concurrency, and at the Offline point | |
accuracy-valid |
§6.6 | accuracy_results.json parses correctly |
|
accuracy-sample-count |
§6.6 | Issued sample count meets the model's minimum | |
accuracy-gate |
§9.1 | Score meets the benchmark quality target | |
agentic-accuracy-inline |
§4.3, §9.1 | Agentic models: inline accuracy meets the model's threshold at every point | |
agentic-accuracy-swebench |
§4.3, §9.1 | Agentic models: the mean of the four mandatory-region SWE-bench scores meets the threshold | |
agentic-osl-range |
§4.3, §9.1 | Agentic models: the full-run mean output length per turn is inside the model's range |
Accuracy has no variability allowance at any stage (Reproducibility Expectations in the Submission Rules).
Agentic models have their own accuracy rules
For Kimi K3, Qwen3.6-35B-A3B and DeepSeek-V4.1-Flash, accuracy-gate stands aside and the three
agentic-* rules apply the thresholds from the client's agentic
README.
SWE-bench is judged on the mean of four results, as §4.3 requires for multi-turn benchmarks.
For the other models, accuracy-gate checks each score.
What automation does not check¶
Manual reviewers focus on what the checker cannot see. These are not rule IDs — they are objection grounds. See Why submissions get rejected.
Clause-numbering note¶
Rule IDs cite some clause numbers that do not exist
The checker's own documentation cites §14, §15 and §16 for the metrics, accuracy and consistency families. Those sections are not present in the current rules document — the corresponding content is in §6.6, §8.5 and §9.1. The clause column above maps to the rules as they actually are. Tracked as B3 in Open questions.
Last verified against: mlcommons/endpoints_policies@v1.0_rules_dev (d2d9da6) and
mlcommons/endpoints-submission-cli@main (a42a056, v1.1.0.0), 2026-10-02.