Submission package layout¶

Two layouts matter: the run folder the reference client writes, and the submission bundle the CLI assembles from run folders. Authored in step 5.

The run folder¶

What mlcommons/endpoints writes to report_dir for one benchmark run.

<report_dir>/
├── config.yaml                       # resolved config as run          [always]
├── report.txt                        # human-readable summary          [always]
├── events.jsonl                      # one JSON object per event       [always]
├── sample_idx_map.json               # sample index → dataset row      [always]
├── performance/
│   └── result_summary.json           # performance metrics             [performance phase]
├── accuracy/
│   └── accuracy_results.json         # per-dataset accuracy scores     [accuracy phase]
└── metrics/
    ├── final_snapshot.json           # aggregator end-of-run snapshot  [always]
    └── .ready                        # zero-byte completion sentinel   [always]

Phase directories exist only for phases that ran:

Invocation	`performance/`	`accuracy/`
`--mode perf` (default)	yes	no
`--accuracy-only`	no	yes
`--mode both`	yes	yes

Plus what you author¶

Two required files are not written by any tool

runs create additionally requires system_desc.json and point.yaml in the run folder. Neither is an endpoints artifact — you author both and drop them in before upload.

<run-folder>/
├── system_desc.json        # you author — hardware/software description
├── point.yaml              # you author — per-point disclosure
├── src/<implementation>/   # merged into the bundle's shared src/ (README.md required)
└── documentation/          # merged into the bundle's shared docs/

This is the only accepted layout

A flat folder with result_summary.json at the top level is rejected: "Run folder error: … is missing required file(s): performance/result_summary.json".

Sizes¶

From a 60-second, 16-concurrency run issuing 1,200 samples:

File	Size	Scales with
`events.jsonl`	46 MB	samples × events per sample
`sample_idx_map.json`	100 KB	samples issued
`metrics/final_snapshot.json`	28 KB	fixed
`performance/result_summary.json`	20 KB	fixed (histogram buckets)
`accuracy/accuracy_results.json`	8 KB	datasets scored
`config.yaml`, `report.txt`	4–8 KB	fixed

events.jsonl dominates. A 600-second Pareto point produces several hundred megabytes, so run archives are large — plan disk and upload bandwidth accordingly.

Key fields in `result_summary.json`¶

version, git_sha, test_started_at, n_samples_issued, n_samples_completed,
n_samples_failed, n_samples_succeeded, n_samples_dropped, duration_ns, state,
complete, ttft, tpot, latency, output_sequence_lengths, input_sequence_lengths,
legacy_loadgen_window_duration_ns, qps, tps, finish_reason_counts, run_config

ttft, tpot, latency and the sequence-length entries are stat blocks of {total, min, max, median, avg, std_dev, percentiles, histogram}.

Percentile keys are decimal strings

"50.0", "90.0", "99.9" — not "50" / "90". A lookup by integer string returns nothing.

`accuracy_results.json`¶

{"osl_tokenization_s": 0.0,
 "accuracy_scores": [
   {"dataset_name": "...", "extractor": "...", "ground_truth_column": "...",
    "score": 0.0, "unit_samples": 0, "num_repeats": 0, "total_samples": 0,
    "duration_s": 0.0, "complete": true, "dataset_type": "...",
    "response_counts": {}, "output_sequence_lengths": {}, "osl_tokenize_s": 0.0}
 ]}

accuracy_scores is a list of per-dataset entries — index it by dataset_name, not by position.

The submission bundle¶

What endpoints-submission-cli assembles and uploads.

<submitting_organization>/
└── <submission_id>/                  # assigned by MLCommons; one per submission
    ├── cli_metadata.json             # which CLI shaped this bundle
    │
    ├── src/                          # SHARED across the whole submission
    │   └── <implementation>/         # e.g. trtllm/, vllm/, sglang/
    │       ├── README.md             # REQUIRED — how to build/launch and reproduce a point
    │       └── <endpoint interface code, infra setup, client harness>
    │
    ├── docs/                         # SHARED across the whole submission
    │   ├── calibration.adoc          # if weight transformations were applied
    │   ├── software_disclosure.md
    │   └── <additional documentation>
    │
    └── results/
        └── <system>/                 # e.g. H200-SXM-141GBx8_TRT/
            └── <model_name>/         # e.g. deepseek-r1/, gpt-oss-120b/
                └── r<N>/             # one PARETO POINT per concurrency (r1, r32, r256, …)
                    ├── point.yaml
                    ├── system_desc.json
                    ├── result_summary.json
                    ├── accuracy_results.json
                    ├── config.yaml            # OPTIONAL as of v1.0
                    └── server_configs/        # OPTIONAL, point-specific, submitter-defined

Shared versus per-point¶

Shared (src/, docs/) is written once per submission. You don't repeat infrastructure code or documentation for each Pareto point. If you need different code for different systems or models, add another src/<implementation>/ or a subdirectory under docs/.

Per-point covers only what changes with concurrency: point.yaml, the result and metadata JSON files, and the optional server_configs/.

Adding or withdrawing a point must not touch src/ or docs/

That's why each point names the shared content it used through the shared_src and shared_docs pointers in its point.yaml. If those pointers don't resolve to a directory under the submission root, the point is incomplete and the submission is rejected.

`system_desc.json` is per point¶

Since policies PR #119 there is no per-system file — every Pareto point carries its own system_desc.json. The checker verifies that all points of a curve describe the same system.

`cli_metadata.json`¶

Written inside <submission_id>/, not at the organisation level.

{
  "command": "remove-run",
  "cli_version": "1.0.0.0",
  "most_recent_cli_used": "1.2.0.0",
  "created_at": "2026-08-20T17:08:02.936070Z"
}

Key	Meaning
`command`	The command that assembled this bundle
`cli_version`	The CLI that created the submission
`most_recent_cli_used`	The CLI that ran this command
`created_at`	Submission creation time — not rebuild time

Known gaps¶

Things the tooling does not do that you might expect

run_metadata.json is never written by the reference client. build_archive(..., run_date=…) injects run_date into that file, so for a client-written run folder the injection is a silent no-op. The field only takes effect if you author the file.
test_started_at is 0. The real wall-clock start is not recorded in result_summary.json, so started_at / finished_at are reconstructed as now() - duration_ns. The window length is correct; its absolute position is the upload time, not the run time.
config.yaml is sanitized. Report directories contain a config.yaml with credentials and other secrets replaced by <redacted>. Restore them before reusing that file as benchmark input.

Last verified against: mlcommons/endpoints-submission-cli@main (f48ca84) and mlcommons/endpoints_policies@v1.0_rules_dev (a7ec3cc), 2026-09-19.