Skip to main content

Data Diagnosis

Introduction#

This tool is to filter the defective machines automatically from thousands of benchmarking results according to rules defined in rule file.

Usage#

  1. Install SuperBench on the local machine.

  2. Prepare the raw data, rule file, baseline file under current path or somewhere on the local machine.

  3. After installing the Superbnech and the files are ready, you can start to filter the defective machines automatically using sb result diagnosis command. The detailed command can be found from SuperBench CLI.

    sb result diagnosis --data-file ./results-summary.jsonl --rule-file ./rule.yaml --baseline-file ./baseline.json --output-file-format excel --output-dir ${output-dir}

  4. After the command finished, you can find the output result file named 'diagnosis_summary.xlsx' / 'diagnosis_summary.json' under ${output_dir}.

Input#

The input mainly includes 3 files:

  • raw data: jsonl file including multiple nodes' results automatically generated by SuperBench runner.

    Tips: this file can be found at ${output-dir}/results-summary.jsonl after each successful run.

  • rule file: It uses YAML format and includes each metrics' rules to filter defective machines for diagnosis.

  • baseline file (optional): json file including the baseline values for the metrics.

    Tips: this file for some representative machine types will be published in SuperBench Results Repo with the release of Superbench.

rule file#

This section describes how to write rules in rule file.

The convention is the same with SuperBench Config File, please view it first.

Here is an overview of the rule file structure:

scheme:

version: stringsuperbench:  var:    ${var_name}: dict  rules:    ${rule_name}:      function: (optional)string      criteria: (optional)string      store: (optional)bool      categories: string      metrics:        - ${benchmark_name}/regex        - ${benchmark_name}/regex        ...

example:

# SuperBench rulesversion: v0.11superbench:  rules:    failure-rule:      function: failure_check      criteria: lambda x:x>0      categories: Failed      metrics:        - kernel-launch/return_code        - mem-bw/return_code        - nccl-bw/return_code        - ib-loopback/return_code    rule0:    # Rule 0: If KernelLaunch suffers > 5% downgrade, label it as defective      function: variance      criteria: lambda x:x>0.05      categories: KernelLaunch      metrics:        - kernel-launch/event_time:\d+        - kernel-launch/wall_time:\d+    rule1:    # Rule 1: If H2D_Mem_BW or D2H_Mem_BW test suffers > 5% downgrade, label it as defective      function: variance      criteria: lambda x:x<-0.05      categories: Mem      metrics:        - mem-bw/H2D_Mem_BW:\d+        - mem-bw/D2H_Mem_BW:\d+    rule2:    # Rule 2: If NCCL_BW suffers > 5% downgrade, label it as defective      function: variance      criteria: lambda x:x<-0.05      categories: NCCL      metircs:        - nccl-bw/allreduce_8589934592_busbw:0    rule3:    # Rule 3: If GPT-2, BERT suffers > 5% downgrade, label it as defective      function: variance      criteria: lambda x:x<-0.05      categories: Model      metrics:        - bert_models/pytorch-bert-base/throughput_train_float(32|16)        - bert_models/pytorch-bert-large/throughput_train_float(32|16)        - gpt_models/pytorch-gpt-large/throughput_train_float(32|16)    rule4:      function: variance      criteria: "lambda x:x<-0.05"      store: True      categories: CNN      metrics:        - resnet_models/pytorch-resnet.*/throughput_train_.*    rule5:      function: variance      criteria: "lambda x:x<-0.05"      store: True      categories: CNN      metrics:        - vgg_models/pytorch-vgg.*/throughput_train_.*\    rule6:      function: multi_rules      criteria: 'lambda label: bool(label["rule4"]+label["rule5"]>=2)'      categories: CNN    rule7:      categories: MODEL_DIST      store: True      metrics:        - model-benchmarks:stress-run.*/pytorch-gpt2-large/fp32_train_throughput    rule8:      function: multi_rules      criteria: 'lambda label: bool(min(label["rule7"].values()))<1)'      categories: MODEL_DIST

This rule file describes the rules used for data diagnosis.

They are firstly organized by the rule name, and each rule mainly includes several elements:

metrics#

The list of metrics for this rule. Each metric is in the format of ${benchmark_name}/regex, you can use regex after the first '/', but to be noticed, the benchmark name can not be a regex.

categories#

The categories belong to this rule.

criteria#

The criterion used for this rule, which indicates how to compare the data with the baseline value for each metric. The format should be a lambda function supported by Python.

store#

  • True: this rule is used to store metrics which will be used by other subsequent rules.
    • If store is True and criteria/function are not None in the rule, it will store how many metrics in this rule meet the criteria into lable["rule_name"], for example lable["rule_name"]=2 means 2 metrics are identified as defective in this rule;
    • If store is True and criteria/function are None, it will store the dict of {metric_name: values} of the metrics into lable["rule_name"]
  • False (default): this rule is used to label the defective machine directly.

function#

The function used for this rule.

The supported functions are listed as follows:

  • variance: the rule is to check if the variance between raw data and baseline violates the criteria. variance = (raw data - baseline) / baseline

    For example, if the 'criteria' is lambda x:x>0.05, the rule is that if the variance is larger than 5%, it should be defective.

  • value: the rule is to check if the raw data violate the criteria.

    For example, if the 'criteria' is lambda x:x>0, the rule is that if the raw data is larger than the 0, it should be defective.

  • multi_rules: the rule is to check if the combined results of multiple previous rules and metrics violate the criteria. We would like to list several examples as follows:

    • criteria: lambda label: bool(label["rule4"]+label["rule5"]>=2) means that this rule will be triggered if the sum of labeled metrics in rule4 and rule5 is larger than 2
    • criteria: lambda label: bool(min(label["rule7"].values()))<1) means that if the minimum of the metrics' values in rule6 is smaller than 1, it should be defective.
      • If you reference a non-existent rule, it will raise exception.
      • If the test in the referenced rule failed or not run resulting in exception in creteria, it will not raise exception since it will be checked in failure_rule.

  • failure_check: the rule is to check if any metric in this rule fail or miss the test. The metrics in this rule should be like {benchmark_name}/.*:return_code used to identify the failure.

    • If any item is never matched with the metrics of the raw data, the rule will identify it as miss test.
    • If any metric violate the value criteria which means return_code is not success(0), the rule will identify it as failed test.

Tips: you must contain a default rule for ${benchmark_name}/return_code as the above in the example, which is used to identify failed tests.

Output#

We support different output formats for filtering the defective machines including json, jsonl, excel, md and html.

The output includes all defective machines' information including index, failure category, failure details, and detailed metrics by default.

  • index: the name of defective machines.

  • Category (diagnosis/category in json format): categories defined in the rule.

  • Defective Details (diagnosis/issue_details in json format): all violated metrics including metric data and related rule.

  • ${metric}: the data of the metrics defined in the rule file. If the rule is variance, the form of the data is variance in percentage; if the rule is value, the form of the data is raw data.

    • 'N/A' indicates a empty value for the metric in output files.

If you specify '--output-all' in the command, the output includes all machines' information and an extra field to indicate if the machines is defective.

  • Accept (diagnosis/accept in json format): False if the machine is defective, otherwise True.