Skip to content

FAQ

Process and policy questions that are not errors. For errors, see Troubleshooting.

An FAQ entry is a documentation bug

If a question keeps getting asked here, the workflow page that should have answered it is at fault. The fix is to improve that page and delete the entry, not to grow this one.

Getting started

Do I have to be an MLCommons member to submit?
Unconfirmed. The rules require the MLCommons CLA from the individual making the submission (Submission Rules §5.1), but no clause makes organisational membership a precondition. Ask MLCommons before planning around either answer. See Membership, PRISM and eligibility and A1 in Open questions.
Does everyone in my organisation need to sign the CLA?
No. Only the individual making the submission signs it (Submission Rules §5.1).
Is there a registration deadline?
No. The general MLPerf eight-week advance registration doesn't apply (Submission Rules §5.1). Registration is having a PRISM API key.
When is the submission deadline?
There isn't one; submissions are rolling. What matters is the cohort cutoff: automated checks must pass at least one business day before a publication date (Submission Rules §4.3).

Scope of a submission

Can I submit more than one model?
One result is one Pareto curve: one system, one benchmark model, one dataset. A single submission can carry several systems and models, and each becomes its own result.
How many points do I need?
For a non-agentic benchmark, 8 to 32: 1 + 3 + 3 fixed-concurrency points plus an Offline point. If you elect your C_max point as the Offline result, 7 is enough. Agentic benchmarks need 7 and have no Offline point. See §5.3 of the rules and Plan your Pareto curve.
Do I really need a separate Offline run?
No. You can elect your C_max point as the Offline result instead, provided you have a point at exactly C_max. You give up whatever extra throughput an unpaced run might have shown, and save a run plus its accuracy validation. See §5.7.2 of the rules and step 3.
Can I add points after submitting?
No. The post-submission window for adding measurement points was removed. There is no add-run, and submissions update --run-ids rejects any list that would add one. The technical rules still mention the old window; see B2 in Open questions.
Can I remove a bad point?
Yes, with submissions remove-run during peer review. But withdrawn points do not count toward the minimum (Submission Rules §8.1) and the shortfall cannot be repaired by adding a replacement. This is why planning a spare point matters.
Do I need an accuracy run per point?
No. You need one at five points: the four mandatory region points and the Offline point, or four for agentic benchmarks (§5.3 of the rules). How each must pass, single-turn versus multi-turn, is in §4.3.

Rules

Is response caching allowed?
No. Every request must execute the forward pass (§2.9.9 Q1 of the rules).
Is KV-cache reuse across queries allowed?
Yes, on a per-query salted token stream. This is the main difference from MLPerf Inference. See §2.9.5 of the rules and Model equivalence.
Can I use a different serving framework than the reference?
Yes, if it conforms to the rest of the rules and meets the Available definition (§2.9.9 Q3 of the rules).
Can I quantize?
Yes. Post-training quantization is permitted under the conditions in §2.9.3 of the rules.
Can I use speculative decoding?
Only with a drafter on the benchmark's approved list. The reference repository lists approved speculative-decoding heads for the agentic benchmarks only, in the agentic example README. There is none for Llama 3.1 8B, GPT-OSS 120B or DeepSeek-R1. The checker's bundled list is still empty, so for now it rejects any point that uses speculative decoding. See §2.9.4 of the rules and C10 in Open questions.
Can I disable speculative decoding on some points?
Yes. The drafter is fixed across the curve, but its configuration can vary per point, including off (§2.9.4). A drafter that ships in the canonical checkpoint must stay in your derived checkpoint (§2.9.3).
Do I have to report power?
For Standardized, yes, as provisioned power rather than a meter reading (§4.5 of the rules). The checker wants a system_power.json for every system whatever your division (B11 in Open questions). See Power normalization.
Does passing the accuracy gate make an optimisation legal?
No. Accuracy is necessary, not sufficient (§2.9.8 of the rules).

Publication

Can I publish before review finishes?
Yes, by opting in to provisional publication at submission time. The choice is irrevocable (Submission Rules §6.2). Provisional publication itself is in §6.2.3.
Can I delay publication?
Yes, with an embargo date declared at submission. What it holds back depends on your publication mode; see Submission Rules §4.2 and §6.2.
Can I be audited?
Yes. From the moment you're nominated you must keep the system as submitted. See Submission Rules §10 and Audits.
What happens if my Preview system does not become available in time?
The result is invalidated and removed at the next cohort, unless you got the one extension allowed. See Submission Rules §7.3.5 and §7.3.6.
Can I relabel a Preview result as Available without re-running?
Only if the hardware and software configuration did not change materially. If it did, you must re-run (Submission Rules §7.3.7).
Can I fix an error in a published result?
No. The affected points or the whole submission are invalidated instead (Submission Rules §8.1).

Review

Who reviews my submission?
The review committee for your cohort, plus one designated reviewer assigned at random from outside your organisation. See Submission Rules §2.1 and §2.6.
Is a competitor reviewing me a conflict of interest?
No, and neither are CSP/OEM/ODM partnerships (Submission Rules §2.4).
How fast do I have to respond to an objection?
3 business days, with local public holidays exempt. Penalties escalate to withdrawal at 10 business days and are non-reversible (Submission Rules §6.3).
Can I improve my results during review?
Only when the review committee requests it, and any improvement to performance metrics must be justified (Submission Rules §6.3).
What if someone objects after review closes?
Only on limited grounds, and reproducibility isn't one of them: a reproducibility concern becomes an audit nomination instead. See Submission Rules §6.6.
How long does my result stay challengeable?
Until the later of the next audit vote or 90 days after finalization. You must keep the system available for audit until then, and longer if you're nominated (Submission Rules §6.6).

Naming and messaging

What do I call my result?
Unqualified "MLPerf Endpoints" means Standardized. Serviced, RDI and Preview results carry a qualifier. See §2.2.3, §2.3.2 and §2.4.2 of the rules, and Submission Rules §7.3.8.
Do I have to say anything when quoting a provisional result?
Yes, the MLCommons footnote saying results are preliminary (Submission Rules §6.2.3).