Genie Accuracy JSON configuration string

The following sections contain information that pertain to the format of the JSON configuration string that is supplied to GenieAccuracyConfig_createFromJson. This JSON configuration can be supplied to genie-app tool for accuracy evaluation.

General configuration schema

The following provides the schema of the JSON configuration format that is provided to GenieAccuracyConfig_createFromJson.

{
  "dataset": {"type": "string"},
  "type": {"type": "string", "enum": ["perplexity"]},
  "engine": {
    "type": "object",
    "properties": {
      "convert-to-basic": {"type": "boolean"},
      "role": {"type": "string"}
    }
  }
}

Option	Required	Description
dataset	Yes	Path to the dataset file used for accuracy evaluation.
type	Yes	Type of accuracy metric to compute. Currently only “perplexity” is supported.
engine	No	Optional engine configuration for accuracy evaluation.
engine::convert-to-basic	No	If present, must be set to true. Converts the dialog to basic mode for accuracy computation. Required for non-basic dialogs.
engine::role	No	Specifies the engine role to use for accuracy evaluation.

Accuracy configuration example

The following is an example configuration for accuracy evaluation.

{
  "dataset": "/path/to/dataset.txt",
  "type": "perplexity",
  "engine": {
    "convert-to-basic": true,
    "role": "target"
  }
}