Troubleshooting

Indexed by what you see. Search this page for a fragment of your error message.

Derived from documented behaviour, not from observed support cases

MLPerf Endpoints has not completed a submission round under these rules. Every entry is derived from a documented failure mode in the tooling or a stated rule, not from failures seen in the wild. Once real support threads exist, this page should be rewritten from them.

Authentication

Every command fails with an authentication error

Cause: the token is missing, mistyped, or not scoped correctly.

Fix:

1. Confirm the variable is set and starts with mlc_:

   echo "${PRISM_USER_API_TOKEN:0:4}"
   

2. Confirm the key was created with Service Scope: MLPerf Endpoints. A key scoped to another service authenticates against a different surface.
3. Try --token explicitly to rule out a shell-profile problem.

See step 1.

Connection errors, or requests going somewhere unexpected

Cause: MLPERF_API_BASE_URL is set.

Fix: unset it. It defaults to https://api.mlcommons.org and should only be overridden for dev or staging environments.

unset MLPERF_API_BASE_URL

Installation

inference-endpoint: command not found

Cause: the venv is not activated, or you installed with uv sync and dropped the prefix.

Fix: either source .venv/bin/activate, or prefix every command with uv run.

Dependency versions differ from what the project expects

Cause: you installed with pip rather than uv sync. The pip path does not use uv.lock.

Fix: for a submission, prefer uv sync so your dependency set matches the lockfile. Record the commit SHA you built from either way, since you need it for disclosure.

Submission commands fail mentioning GitHub

Cause: gh is missing or unauthenticated. It is required for creating, updating and withdrawing submissions.

Fix: install the gh CLI and run gh auth login.

Running the benchmark

result_summary.json shows complete: false

Cause: the run drained out or was interrupted. The metrics are partial.

Fix: the point is not usable — re-run it. Check whether you hit run_timeout_s, a drain timeout, or endpoint_response_idle_timeout_s.

A state of INTERRUPTED means the run was aborted; state: complete with pending tasks means a drain timeout.

The run ends far sooner than the region minimum

Cause: sample-count sizing. With no explicit count and no minimum issue duration, the client issues the dataset once and stops.

Fix: set settings.runtime.n_samples_to_issue, or min_issue_duration_ms, so the run sustains 600 s (Ultra Low) or 1,200 s (other regions) of steady state.

Priority order: n_samples_to_issue > Poisson QPS × min issue duration > dataset size.

A percentile lookup in result_summary.json returns nothing

Cause: percentile keys are decimal strings — "50.0", "90.0", "99.9", not "50" or "90".

Fix: use the decimal form.

Dataset validation fails saying samples cannot be salted

Cause: salt requires a dict sample with a text prompt field. A dataset whose samples carry messages, or multimodal content parts, cannot be salted, and the client validates every sample up front rather than shipping an unsalted payload.

Fix: use a dataset with a text prompt, or disable the warmup salt, but if warmup uses the performance dataset, salting is mandatory, so the dataset must support it.

The endpoint stops responding and the run hangs

Cause: no liveness deadline set.

Fix:

settings:
  timeouts:
    endpoint_response_idle_timeout_s: 300   # >= 300; raise for long requests

from-config ignores --report-dir

Cause: not a bug. from-config accepts only --config, --timeout and --mode.

Fix: set report_dir in the YAML.

Disclosure files

The checker reports missing point.yaml or system_desc.json

Cause: you expected a tool to generate them. Nothing does.

Fix: author both by hand and place them at the top level of every run folder. See step 5, point.yaml and system_desc.json.

Fields you set in config.yaml do not appear in the submission

Cause: point.yaml is copied verbatim. The CLI does not derive it from config.yaml and does not fill in missing fields.

Fix: put every required disclosure field in point.yaml itself.

shared-path-resolution fails

Cause: shared_src or shared_docs does not resolve to an existing directory under the submission root.

Fix: correct the pointers in every point.yaml. This check rejects the submission.

tps-utilization fails

Cause: the value is not system_tps / max(system_tps) across your own curve.

Fix: recompute after every point has run. You cannot fill it in before then.

Validation failures

point-count fails: fewer than 7 points

Cause: too few points, or points were withdrawn.

Fix: run more. Note withdrawn points do not count toward the minimum and cannot be replaced, because there's no add-run. If the curve needs a different set of runs, create a new submission.

A concurrency-coverage check fails

Cause: no point in one of Low, Medium or High Concurrency — often because C_min changed.

Fix:

submission-checker regions --max-concurrency <C_max> --min-concurrency <your lowest point>

Remember C_min is derived from your own lowest point, so dropping that point moves every other boundary. And a point in the 10% margin does not satisfy High Concurrency.

region-placement warns

Cause: the region you declared in point.yaml disagrees with the computed one.

Fix: correct the declared value. It is only a warning, but it is exactly the kind of thing a reviewer files a methodology objection about.

seed-set-adoption reports SKIP

Cause: the bundled seed-set file carries no cohort keys, so the adoption test cannot run.

Fix: none available locally. Confirm the correct seed set with MLCommons, and point --seed-sets FILE or $MLPERF_ENDPOINTS_SEED_SETS at a newer file if one exists. Tracked as B4 in Open questions.

accuracy-gate fails

Cause: the accuracy run missed the benchmark quality target.

Fix: there is no tolerance — this rejects the submission. Check whether an approximation permitted under model equivalence pushed you below the target; dynamic approximate sparsity and aggressive PTQ are both gated on exactly this.

Submission failures

Run folder error: … is missing required file(s): performance/result_summary.json

Cause: a flat run folder with the summary at the top level.

Fix: use the layout the reference client writes — the summary belongs under performance/. Flat layouts are not accepted. See Submission package layout.

The build fails naming a specific run

Cause: the builder cannot tell whether the run is an accuracy or a performance run. It reads datasets[].type from config.yaml, then falls back to point.yaml's dataset_type, but only when that is exactly Accuracy or Performance.

Fix: supply config.yaml, or set dataset_type to exactly Accuracy or Performance. Accuracy + Performance describes the dataset, not the run, and is rejected on purpose: guessing used to file accuracy runs as performance runs and silently drop the accuracy results.

submissions update --run-ids is rejected

Cause: the list would add a run. The post-submission window for adding points was removed.

Fix: the list may only shrink. To remove one point, use submissions remove-run. For a different set of runs, create a new submission.

A run cannot be deleted

Cause: it belongs to an active submission.

Fix: submissions withdraw first, then runs delete.

The upload failed and you are unsure of the state

Cause: partial failure. The CLI rolls back automatically — a failed run-archive upload deletes the run record; a failed bundle upload withdraws the submission.

Fix: if rollback also failed, the CLI prints the orphaned ID. Clean it up:

endpoints-submission-cli runs delete --run-id <orphaned-id>
endpoints-submission-cli submissions withdraw --submission-id <orphaned-id>

pr_url is empty after a successful submission

Cause: not a bug. The CLI no longer opens the review pull request.

Fix: nothing. pr_url and pr_number populate once whatever opens it has set them.

submissions create succeeded but status is not REVIEW_PENDING

Cause: the final PATCH step failed. Both submission and bundle exist — the CLI treats this as a warning, not a fatal error.

Fix: the status can be set manually. Confirm with submissions get.

During review

You missed the 3-business-day response window

Cause: no one was watching the review thread.

Fix: respond immediately. Penalties are cumulative and non-reversible — responding does not undo a penalty already incurred, but it prevents further escalation. At 10 business days the submission is withdrawn.

See After you submit.

A measurement point turns out to be wrong during review

Cause: an error found after compliance passed.

Fix: results may not be changed during peer review. Your options are to withdraw the point (submissions remove-run) or withdraw the submission. Withdrawn points do not count toward the 7-point minimum and cannot be replaced.

Still stuck?

Getting support.