Skip to content

Submission Generation

Click here to view the proposal slide for Common Automation for MLPerf Inference Submission Generation through MLCFlow.

Please refer to the installation page to install MLCFlow for automating the submission generation. In a typical development environment pip install mlc-scripts should be enough.

If you have not followed the mlcr commands under the individual model pages in the benchmarks directory, please make sure that the result directory is structured in the following way. You can see the real examples for the expected folder structure here. 

└── System description ID(SUT Name)
    ├── system_meta.json
    └── Benchmark
        └── Scenario
            ├── Performance
            |   └── run_1 run for all scenarios
            |       ├── mlperf_log_summary.txt
            |       └── mlperf_log_detail.txt
            ├── Accuracy
            |   ├── mlperf_log_summary.txt
            |   ├── mlperf_log_detail.txt
            |   ├── mlperf_log_accuracy.json
            |   └── accuracy.txt
            |── Compliance_Test_ID
            |   ├── Performance
            |   |   └── run_x/#1 run for all scenarios
            |   |       ├── mlperf_log_summary.txt
            |   |       └── mlperf_log_detail.txt
            |   ├── Accuracy # for TEST01 only
            |   |   ├── baseline_accuracy.txt (if test fails in deterministic mode)
            |   |   ├── compliance_accuracy.txt (if test fails in deterministic mode)
            |   |   ├── mlperf_log_accuracy.json
            |   |   └── accuracy.txt
            |   ├── verify_performance.txt
            |   └── verify_accuracy.txt # for TEST01 only
            |── user.conf
            └── measurements.json

Click here if you are submitting in open division

  • The model_mapping.json should be included inside the SUT folder which is used to map the custom model full name to the official model name. The format of json file is:

    {
        "custom_model_name_for_model1":"official_model_name_for_model1",
        "custom_model_name_for_model2":"official_model_name_for_model2",

    }

If you have followed the mlcr commands under the individual model pages in the benchmarks directory, all the valid results will get aggregated to the mlc cache folder. The following command could be used to browse the structure of inference results folder generated by MLCFlow.

Get results folder structure

mlc find cache --tags=get,mlperf,inference,results,dir | xargs tree

Once all the results across all the models are ready you can use the following the below section to generate a valid submission tree compliant with the MLPerf requirements.

Generate submission folder

The submission generation flow is explained in the below diagram

flowchart LR
    subgraph Generation [Submission Generation SUT1]
      direction TB
      A[populate system details] --> B[generate submission structure]
      B --> C[truncate-accuracy-logs]
      C --> D{Infer low latency results <br>and/or<br> filter out invalid results}
      D --> yes --> E[preprocess-mlperf-inference-submission]
      D --> no --> F[run-mlperf-inference-submission-checker]
      E --> F
    end
    Input((Results SUT1)) --> Generation
    Generation --> Output((Submission Folder <br> SUT1))

Command to generate submission folder

mlcr generate,inference,submission \
  --clean \
  --preprocess_submission=yes \
  --run_checker=yes \
  --submitter=MLCommons \
  --division=closed \
  --env.MLC_DETERMINE_MEMORY_CONFIGURATION=yes \
  --quiet

Tip

  • Use --hw_name="My system name" to give a meaningful system name. Examples can be seen here

  • Use --submitter=<Your name> if your organization is an official MLCommons member and would like to submit under your organization

  • Use --hw_notes_extra option to add additional notes like --hw_notes_extra="Result taken by NAME"

  • Use --results_dir option to specify the results folder. It is automatically taken from MLC cache for MLPerf automation based runs

  • Use --submission_dir option to specify the submission folder. (You can avoid this if you're pushing to github or only running a single SUT and MLC will use its cache folder)

  • Use --division=open for open division submission

  • Use --category option to specify the category for which submission is generated(datacenter/edge). By default, the category is taken from system_meta.json file located in the SUT root directory.

  • Use --submission_base_dir to specify the directory to which the outputs from preprocess submission script and final submission is added. No need to provide --submission_dir along with this. For docker run, use --submission_base_dir instead of --submission_dir.

If there are multiple systems where MLPerf results are collected, the same process needs to be repeated on each of them. One we have submission folders on all the SUTs, we need to sync them to make a single submission folder

If you are having results in multiple systems, you need to merge them to one system. You can use rsync for this. For example, the below command will sync the submission folder from SUT2 to the one in SUT1. 

rsync -avz username@host1:<path_to_submission_folder2>/ <path_to_submission_folder1>/
Same needs to be repeated for all other SUTs so that we have the full submissions in SUT1.

    flowchart LR
        subgraph SUT1 [Submission Generation SUT1]
          A[Submission Folder SUT1]
        end
        subgraph SUT2 [Submission Generation SUT2]
          B[Submission Folder SUT2]
        end
        subgraph SUT3 [Submission Generation SUT3]
          C[Submission Folder SUT3]
        end
        subgraph SUTN [Submission Generation SUTN]
          D[Submission Folder SUTN]
        end
        SUT2 --> SUT1
        SUT3 --> SUT1
        SUTN --> SUT1

If you are collecting results across multiple systems you can generate different submissions and aggregate all of them to a GitHub repository (can be private) and use it to generate a single tar ball which can be uploaded to the MLCommons Submission UI.

Run the following command after replacing --repo_url with your GitHub repository URL.

mlcr push,github,mlperf,inference,submission \
   --repo_url=https://github.com/mlcommons/mlperf_inference_submissions_v5.0 \
   --commit_message="Results on <HW name> added by <Name>" \
   --quiet

    flowchart LR
        subgraph SUT1 [Submission Generation SUT1]
          A[Submission Folder SUT1]
        end
        subgraph SUT2 [Submission Generation SUT2]
          B[Submission Folder SUT2]
        end
        subgraph SUT3 [Submission Generation SUT3]
          C[Submission Folder SUT3]
        end
        subgraph SUTN [Submission Generation SUTN]
          D[Submission Folder SUTN]
        end
    SUT2 -- git sync and push --> G[Github Repo]
    SUT3 -- git sync and push --> G[Github Repo]
    SUTN -- git sync and push --> G[Github Repo]
    SUT1 -- git sync and push --> G[Github Repo]

Upload the final submission

Warning

If you are using GitHub for consolidating your results, make sure that you have run the push-to-github command on the same system to ensure results are synced as is on the GitHub repository.

Once you have all the results on the system, you can upload them to the MLCommons submission server as follows:

You can do the following command which will run the submission checker and upload the results to the MLCommons submission server 

mlcr run,mlperf,submission,checker,inference \
--submitter_id=<> \
--submission_dir=<Path to the submission folder>

You can do the following command to generate the final submission tar file and then upload to the MLCommons Submission UI. 

mlcr run,mlperf,submission,checker,inference \
--submission_dir=<Path to the submission folder> \
--tar=yes \
--submission_tar_file=mysubmission.tar.gz


        flowchart LR
            subgraph SUT [Combined Submissions]
              A[Combined Submission Folder in SUT1]
            end
        SUT --> B[Run submission checker]
        B --> C[Upload to MLC Submission server]
        C --> D[Receive validation email]