Skip to content

7. Validate locally

Produces: a clean checker report.

Before you begin

You only get one attempt at Week 0

The same checks run on the server after you submit, and a submission that fails any of them by the end of Week 0 is rejected (Submission Rules §6.1). You can't patch it in place: you resubmit as an entirely new submission. Local validation is the cheap version of that.

What you'll do

  • Run submissions create --dry-run, which builds the bundle from your runs and checks it without creating anything
  • Or run endpoints-submission-cli check-submission against a submission directory you assembled yourself
  • Fix, re-register, re-run, repeat until clean

Steps

1. Dry-run the real pipeline

Run submissions create with --dry-run, the same classification you'll use in step 8 (division, scenario, availability and publication mode, which you decided in Before you begin), and the run IDs from step 6. It takes the same path step 8 will, without creating anything, so it's the closest local approximation to what Week 0 will do.

The full report, warnings included, goes to a submission_checker_<timestamp>.log file in the current directory. If a check fails, fix the run folder, register it again, and re-run with the new run ID.

2. Or check a directory you assembled

Use this for CI, or when you've laid out a submission yourself following Submission package layout. The command, its flags and exit codes are in the checker's README.

3. Read the report properly

Errors and warnings are not the same thing. The checker sets the severity, and it mostly follows the failure actions in §9.1 of the rules. Some rules §9.1 only flags are errors locally, listed in Why submissions get rejected.

Meaning Consequence
Error A check that fails Blocks submissions create
Warning A check that is flagged rather than fatal Does not block locally, but reviewers see it and may object

Read the warnings, not just the exit code

The dry-run fails only on errors, so warnings pass silently unless you read the log. submissions create --dry-run has no strict mode; endpoints-submission-cli check-submission <dir> --strict treats warnings as errors. Warnings are where methodology objections come from. A point that merely warns on duration or region placement is exactly the kind of thing a reviewer files an objection about in Weeks 1–3, and an objection costs you far more than a re-run does now.

4. Know what is being checked

What each check looks at is in the checker's README. Which of them reject rather than flag is in §9.1 of the rules, and the cross-walk from checker rule ID to clause is in Compliance checks.

Use checker v1.1.0.0 or later

The Offline, power, accuracy-coverage, steady-state and drafter checks arrived in v1.0.1.0 (2026-09-23). v1.1.0.0 (2026-10-01) added the agentic models and their accuracy gates, fails a single-turn curve with no Offline point, and follows the rules' power model. An older checker passes submissions the server will reject. Check with endpoints-submission-cli --version, and upgrade with pip install -U endpoints-submission-cli.

5. Use the programmatic API for CI

from pathlib import Path
from submission_checker import SubmissionChecker

report = SubmissionChecker(Path("/submissions/acme_corp")).run()

if not report.passed:
    for result in report.errors:
        print(f"[{result.rule}] {result.message}")

report.warnings and report.model_dump_json() are also available. Wiring this into CI so every change to your disclosure files is checked is worth the hour it takes.

6. Look at your curves (optional)

The checker tells you whether the submission is valid, not whether the curve looks the way you expect. To see it, render the same folder in a local dashboard:

pip install mlperf-viz
mlperf-viz /path/to/submission

It opens the Pareto Explorer in your browser; nothing is uploaded. Each <System>/<Model> pair is one curve. Flags and formats: Results visualizer.

Verify

submission-checker check /path/to/submission --strict --output checker.json
echo "exit: $?"

You're ready to submit when the exit code is 0. Keep checker.json, which is useful evidence if a reviewer later questions something the checker already passed.

Next

→ 8. Submit

Problems? See Troubleshooting and Why submissions get rejected.