Genie Accuracy

Genie Accuracy APIs provide a method to evaluate and measure the accuracy of LLM inference results. The accuracy feature supports computing perplexity metrics on a given dataset to assess model performance.

Accuracy evaluation can be performed using the genie-app tool. After computing the accuracy metrics, the results will be returned as a JSON object containing the perplexity value.

Note

The accuracy computation resets the dialog and associated engine. It is recommended to run accuracy evaluation on a newly created dialog rather than in between dialog queries.

Example on how to use Accuracy APIs with Dialog

The following example demonstrates how to integrate GenieAccuracy APIs with an existing dialog to compute perplexity metrics.

// Create Dialog Config
GenieDialogConfig_Handle_t dialogConfigHandle = NULL;
GenieDialogConfig_createFromJson(dialogConfigStr, &dialogConfigHandle);

// Create Dialog
GenieDialog_Handle_t dialogHandle = NULL;
GenieDialog_create(dialogConfigHandle, &dialogHandle);

// Create Accuracy Config
GenieAccuracyConfig_Handle_t accuracyConfigHandle = NULL;
GenieAccuracyConfig_createFromJson(accuracyConfigStr, &accuracyConfigHandle);

// Create Accuracy Handle from Dialog
GenieAccuracy_Handle_t accuracyHandle = NULL;
GenieAccuracy_createFromDialog(accuracyConfigHandle, dialogHandle, &accuracyHandle);

// Define allocation callback for accuracy results
auto allocCallback = [](uint32_t size, void** buffer) -> Genie_Status_t {
    *buffer = reinterpret_cast<char*>(malloc(size));
    return (*buffer != nullptr) ? GENIE_STATUS_SUCCESS : GENIE_STATUS_ERROR_MEM_ALLOC;
};

// Compute accuracy metrics
const char* jsonData = nullptr;
Genie_Status_t status = GenieAccuracy_compute(accuracyHandle, allocCallback, &jsonData);

if (status == GENIE_STATUS_SUCCESS && jsonData != nullptr) {
    // Process the accuracy results (JSON format)
    // Example output: {"perplexity": 5.234}
    std::cout << "Accuracy Results: " << jsonData << std::endl;
}

// Free the allocated memory for JSON data
if (jsonData) {
    free(const_cast<char*>(jsonData));
}

// Free Accuracy Handle
GenieAccuracy_free(accuracyHandle);

// Free Accuracy Config
GenieAccuracyConfig_free(accuracyConfigHandle);

// Free Dialog
GenieDialog_free(dialogHandle);

// Free Dialog Config
GenieDialogConfig_free(dialogConfigHandle);

Example on how to use Accuracy APIs with Node

The following example demonstrates how to integrate GenieAccuracy APIs with a node (TextGenerator) to compute perplexity metrics.

// Create Node Config
GenieNodeConfig_Handle_t nodeConfigHandle = NULL;
GenieNodeConfig_createFromJson(nodeConfigStr, &nodeConfigHandle);

// Create Node (TextGenerator)
GenieNode_Handle_t nodeHandle = NULL;
GenieNode_create(nodeConfigHandle, &nodeHandle);

// Create Accuracy Config (can be NULL for default perplexity calculation)
GenieAccuracyConfig_Handle_t accuracyConfigHandle = NULL;
GenieAccuracyConfig_createFromJson(accuracyConfigStr, &accuracyConfigHandle);

// Create Accuracy Handle from Node
GenieAccuracy_Handle_t accuracyHandle = NULL;
GenieAccuracy_createFromNode(accuracyConfigHandle, nodeHandle, &accuracyHandle);

// Define allocation callback for accuracy results
auto allocCallback = [](uint32_t size, void** buffer) -> Genie_Status_t {
    *buffer = reinterpret_cast<char*>(malloc(size));
    return (*buffer != nullptr) ? GENIE_STATUS_SUCCESS : GENIE_STATUS_ERROR_MEM_ALLOC;
};

// Compute accuracy metrics
const char* jsonData = nullptr;
Genie_Status_t status = GenieAccuracy_compute(accuracyHandle, allocCallback, &jsonData);

if (status == GENIE_STATUS_SUCCESS && jsonData != nullptr) {
    // Process the accuracy results (JSON format)
    std::cout << "Accuracy Results: " << jsonData << std::endl;

    // Free the allocated memory for JSON data
    free(const_cast<char*>(jsonData));
}

// Free Accuracy Handle
GenieAccuracy_free(accuracyHandle);

// Free Accuracy Config (if created)
if (accuracyConfigHandle != NULL) {
    GenieAccuracyConfig_free(accuracyConfigHandle);
}

// Free Node
GenieNode_free(nodeHandle);

// Free Node Config
GenieNodeConfig_free(nodeConfigHandle);

Example genie-app script

The following is an example script for utilizing accuracy features with genie-app

# Version
version

# Dialog creation
dialog config create config1 dialogConfig.json
dialog create dialog1 config1

# Accuracy Handle creation
accuracy config create accConfig accConfig.json
accuracy create accHandle1 accConfig dialog dialog1
accuracy compute accHandle1 OUTPUT_FILE.json

# Clean up
accuracy config free accConfig
accuracy free accHandle1
dialog config free config1
dialog free dialog1
ls

# Exit
exit

Accuracy Configuration

The accuracy configuration JSON should specify the dataset path and the type of accuracy metric to compute. For more details on the accuracy configuration format, please refer to the Genie Accuracy JSON Configuration documentation.