Convert to QNN for Linux Host on LPAI Backend

Note

This is Part 3 of the Convert to QNN tutorial for Linux host machines. If you have not completed Part 2, please do so here.

Warning

LPAI Backend for model offline preparation was introduced at the time of v4 (eNPU hardware version 4). This means you must build the model first, then transfer it to the target device. The offline prepared model was executed by LPAI SDK released separatelly, but starting v5 (eNPU hardware version 5) the execution of serialized model is supported by QNN SDK and additional LPAI SDK is not required.

The offline generated model for v4 must be executed via the LPAI SDK. You will have to sign in to access the LPAI SDK, and it may be dependent on filling out specific paperwork for your account to access the SDK.

Preparing LPAI Configuration Files for Model Preparation

EXAMPLE of config.json file:

{
   "backend_extensions": {
      "shared_library_path": "${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnLpaiNetRunExtensions.so",
      "config_file_path": "./lpaiParams.conf"
   }
}

EXAMPLE of lpaiParams.conf file that includes only preparation parameters:

{
   "lpai_backend": {
      "target_env": "adsp",
      "enable_hw_ver": "v5"
   },
   "lpai_graph": {
      "prepare": {
         "enable_batchnorm_fold": true,
         "exclude_io": false
      }
   }
}

To configure lpaiParams.conf, consider using the following optional settings:

lpai_backend
   "target_env"              "arm/adsp/x86/tensilica, default adsp"
   "enable_hw_ver"           "v4,v5 default v5"
lpai_graph
   prepare
      "enable_batchnorm_fold"   "true/false,     default false"
      "exclude_io"              "true/false,     default false"

Using the above config.json and lpaiParams.conf you can use qnn-context-binary-generator to build the LPAI offline model.

When files are mentioned, ensure that they have the relative or absolute path to that value.

cd ${QNN_SDK_ROOT}/examples/QNN/converter/models
$LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${QNN_SDK_ROOT}/lib/x86_64-linux-clang \
& ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-context-binary-generator \
              --backend ${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnLpai.so \
              --model ${QNN_SDK_ROOT}/examples/Models/InceptionV3/model_libs/x86_64-linux-clang/<libQnnModel.so> \
              --config_file <config.json> \
              --log_level verbose \
              --backend_binary <output_graph.eai> \
              --binary_file <lpai_graph_serialized>

Note

• Use generated output_graph.eai file in the format of eai model to be executed by LPAI SDK for v4 version only.

• Use generated lpai_graph_serialized.bin file in QNN format to be executed directly by QNN SDK for v5 and higher versions.

Running LPAI Emulation Backend on Linux x86

While LPAI Backend on x86_64 Linux CPU is designed for offline model generation as described above, it still can run in HW simulation mode where internally it executes prepare and execution steps together.

EXAMPLE of config.json file for ARM:

{
   "backend_extensions": {
      "shared_library_path": "${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnLpaiNetRunExtensions.so",
      "config_file_path": "./lpaiParams.conf"
   }
}

EXAMPLE of lpaiParams.conf file that includes preparation and execution parameters:

{
   "lpai_backend": {
      "target_env": "x86",
      "enable_hw_ver": "v5"
   },
   "lpai_graph": {
      "prepare": {
         "enable_batchnorm_fold": false,
         "exclude_io": false
      },
      "execute": {
         "fps": 1,
         "ftrt_ratio": 10,
         "client_type": "real_time",
         "affinity": "soft",
         "core_selection": 0
      }
   }
}

To configure lpaiParams.conf, consider using the following optional settings:

lpai_backend
   "target_env"              "arm/adsp/x86/tensilica, default adsp"
   "enable_hw_ver"           "v4,v5 default v5"
lpai_graph
   prepare
      "enable_batchnorm_fold"   "true/false,     default false"
      "exclude_io"              "true/false,     default false"
   execute
      "fps"                     "Specify the fps rate number, used for clock voting, default 1"
      "ftrt_ratio"              "Specify the ftrt_ratio number, default 10"
      "client_type"             "real_time/non_real_time, defult real_time"
      "affinity"                "soft/hard, default soft"
      "core_selection"          "Specify the core number, default 0"

Using the above config.json and lpaiParams.conf you can use qnn-net-run to directly execute LPAI backend in simulation mode.

With the appropriate libraries compiled, qnn-net-run is used with the following:

Note

If full paths are not given to qnn-net-run, all libraries must be added to LD_LIBRARY_PATH and be discoverable by the system library loader.

$ cd ${QNN_SDK_ROOT}/examples/QNN/converter/models
$ ${QNN_SDK_ROOT}/bin/x86_64-linux-clang/qnn-net-run \
              --backend ${QNN_SDK_ROOT}/lib/x86_64-linux-clang/libQnnLpai.so \
              --model ${QNN_SDK_ROOT}/examples/QNN/example_libs/x86_64-linux-clang/libqnn_model_8bit_quantized.so \
              --input_list ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt \
              --config_file <config.json>

Outputs from the run will be located at the default ./output directory.

Running LPAI on Target Platform via ARM Backend using offline prepared graph

While graph prepare is predefined by offline generation step the Execution paramaters still can be changed at the execution time.

EXAMPLE of config.json file:

{
   "backend_extensions": {
      "shared_library_path": "./libQnnLpaiNetRunExtensions.so",
      "config_file_path": "./lpaiParams.conf"
   }
}

EXAMPLE of lpaiParams.conf file that includes only execution parameters:

{
   "lpai_backend": {
      "target_env": "adsp",
      "enable_hw_ver": "v5"
   },
   "lpai_graph": {
      "execute": {
         "fps": 1,
         "ftrt_ratio": 10,
         "client_type": "real_time",
         "affinity": "soft",
         "core_selection": 0
      }
   }
}

To configure lpaiParams.conf, consider using the following optional settings:

lpai_graph
   execute
      "fps"                     "Specify the fps rate number, used for clock voting, default 1"
      "ftrt_ratio"              "Specify the ftrt_ratio number, default 10"
      "client_type"             "real_time/non_real_time, defult real_time"
      "affinity"                "soft/hard, default soft"
      "core_selection"          "Specify the core number, default 0"

Using the above config.json and lpaiParams.conf you can use by qnn-net-run to directly execute LPAI backend on target.

Note

Running the LPAI Backend on an Android via ARM target is supported only for offline prepared graphs. Follow “Preparing LPAI Configuration Files Model Preparation” paragraph first to prepare the graph on x86_64 host and then push the serialized context binary to the device.

In order to run on a particular target platform, the libraries compiled for that target must be used. Below are examples. The QNN_TARGET_ARCH variable can be used to specify the appropriate library for the target.

$ export QNN_TARGET_ARCH=aarch64-android
$ export DSP_ARCH=hexagon-v79
$ export DSP_VER=V79
$ export HW_VER=v5

To execute the LPAI backend on Android device the following conditions must be met:

1. ${QNN_SDK_ROOT}/lib/lpai-${HW_VER}/unsigned/libQnnLpaiSkel.so has to be signed by client
2. qnn-net-run to be executed with root permissions

After offline model preparation of ./output/lpai_graph_serialized.bin push the serialized model to device: .. code-block:

$ adb push ./output/lpai_graph_serialized.bin /data/local/tmp/LPAI

Push configuration files to device:

$ adb push ./config.json /data/local/tmp/LPAI
$ adb push ./lpaiParams.conf /data/local/tmp/LPAI

Push LPAI artifacts to device:

$ adb push ${QNN_SDK_ROOT}/lib/lpai-${HW_VER}/libQnnLpai.so /data/local/tmp/LPAI
$ adb push ${QNN_SDK_ROOT}/lib/lpai-${HW_VER}/libQnnLpaiNetRunExtensions.so /data/local/tmp/LPAI
$ adb push ${QNN_SDK_ROOT}/lib/lpai-${HW_VER}/libQnnLpaiStub.so /data/local/tmp/LPAI
$ # Additionally, the LPAI requires Hexagon specific libraries
$ adb push ${QNN_SDK_ROOT}/lib/lpai-${HW_VER}/unsigned/libQnnLpaiSkel.so /data/local/tmp/LPAI

Push the input data and input lists to device:

$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_data_float /data/local/tmp/LPAI
$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt /data/local/tmp/LPAI

Push the qnn-net-run tool:

$ adb push ${QNN_SDK_ROOT}/bin/aarch64-android/qnn-net-run /data/local/tmp/LPAI

Setup the environment on device:

$ adb shell
$ cd /data/local/tmp/LPAI
$ export LD_LIBRARY_PATH=/data/local/tmp/LPAI
$ export ADSP_LIBRARY_PATH="/data/local/tmp/LPAI"

Finally, use qnn-net-run for execution with the following:

$ ./qnn-net-run --backend libQnnLpai.so --retrieve_context lpai_graph_serialized.bin --input_list input_list_float.txt --config_file <config.json>

Running LPAI on Target Platform via Native aDSP Backend using offline prepared graph

While graph prepare is predefined by offline generation step the Execution paramaters still can be changed at the execution time.

EXAMPLE of config.json file:

{
   "backend_extensions": {
      "shared_library_path": "./libQnnLpaiNetRunExtensions.so",
      "config_file_path": "./lpaiParams.conf"
   }
}

EXAMPLE of lpaiParams.conf file that includes only execution parameters:

{
   "lpai_backend": {
      "target_env": "adsp",
      "enable_hw_ver": "v5"
   },
   "lpai_graph": {
      "execute": {
         "fps": 1,
         "ftrt_ratio": 10,
         "client_type": "real_time",
         "affinity": "soft",
         "core_selection": 0
      }
   }
}

To configure lpaiParams.conf, consider using the following optional settings:

lpai_graph
   execute
      "fps"                     "Specify the fps rate number, used for clock voting, default 1"
      "ftrt_ratio"              "Specify the ftrt_ratio number, default 10"
      "client_type"             "real_time/non_real_time, defult real_time"
      "affinity"                "soft/hard, default soft"
      "core_selection"          "Specify the core number, default 0"

Using the above config.json and lpaiParams.conf you can use by qnn-net-run to directly execute LPAI backend on target.

Note

Running the LPAI Backend on an Android via native aDSP target is supported only for offline prepared graphs. Follow “Preparing LPAI Configuration Files Model Preparation” paragraph first to prepare the graph on x86_64 host and then push the serialized context binary to the device.

In order to run on a particular target platform, the libraries compiled for that target must be used. Below are examples. The QNN_TARGET_ARCH variable can be used to specify the appropriate library for the target.

$ export QNN_TARGET_ARCH=aarch64-android
$ export DSP_ARCH=hexagon-v79
$ export DSP_VER=V79
$ export HW_VER=v5

To execute the LPAI backend on Android device the following conditions must be met:

1. The following artifacts in ${QNN_SDK_ROOT}/lib/lpai-${HW_VER}/unsigned  has to be signed by client
   a. libQnnLpai.so
   b. libQnnLpaiNetRunExtensions.so
2. The following artifacts in ${QNN_SDK_ROOT}/lib/${DSP_ARCH}/unsigned  has to be signed by client
   a. libQnnHexagonSkel_dspApp.so
   b. libQnnNetRunDirect${DSP_VER}Skel.so
2. qnn-net-run to be executed with root permissions

After offline model preparation of ./output/lpai_graph_serialized.bin push the serialized model to device: .. code-block:

$ adb push ./output/lpai_graph_serialized.bin /data/local/tmp/LPAI

Push configuration files to device:

$ adb push ./config.json /data/local/tmp/LPAI
$ adb push ./lpaiParams.conf /data/local/tmp/LPAI

Push the necessary libraries to device:

$ adb push ${QNN_SDK_ROOT}/lib/lpai-${HW_VER}/unsigned/libQnnLpai.so /data/local/tmp/LPAI
$ adb push ${QNN_SDK_ROOT}/lib/lpai-${HW_VER}/unsigned/libQnnLpaiNetRunExtensions.so /data/local/tmp/LPAI
$ adb push ${QNN_SDK_ROOT}/lib/${DSP_ARCH}/unsigned/libQnnHexagonSkel_dspApp.so /data/local/tmp/LPAI
$ adb push ${QNN_SDK_ROOT}/lib/${DSP_ARCH}/unsigned/libQnnNetRunDirect${DSP_VER}Skel.so /data/local/tmp/LPAI

Push the input data and input lists to device:

$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_data_float /data/local/tmp/LPAI
$ adb push ${QNN_SDK_ROOT}/examples/QNN/converter/models/input_list_float.txt /data/local/tmp/LPAI

Push the qnn-net-run tool and its dependent libraries:

$ adb push ${QNN_SDK_ROOT}/bin/${QNN_TARGET_ARCH}/qnn-net-run /data/local/tmp/LPAI
$ adb push ${QNN_SDK_ROOT}/lib/${QNN_TARGET_ARCH}/libQnnNetRunDirect${DSP_VER}Stub.so /data/local/tmp/LPAI

Setup the environment on device:

$ adb shell
$ cd /data/local/tmp/LPAI
$ export LD_LIBRARY_PATH=/data/local/tmp/LPAI
$ export ADSP_LIBRARY_PATH="/data/local/tmp/LPAI"
$ export DSP_VER=V79

Finally, use qnn-net-run for execution with the following:

$ ./qnn-net-run --backend libQnnLpai.so --retrieve_context qnn_model_8bit_quantized.serialized.bin --input_list input_list_float.txt --config_file <config.json>