Qualcomm Model Deployment

Qualcomm provides the Python library for qai_hub_madels, which allows for easy conversion, quantization, and export of models into BIN files that can be loaded by Qualcomm NPU. It also supports the use of Qualcomm online virtual devices for inference and validation of models.

Preparation work

1. Install qai_hub_models

pip3 install qai_hub_models

2. Configure API Token

export PATH=~/.local/bin/:$PATH
qai-hub configure --api_token <API_TOKEN>

Model Core Information

The models supported by qai_hub_madels include Compute Vision, Multinodal, Audio, and Generative AI, which are the main categories. Please refer to the table below for details

Computer Vision

Multimodal

Audio

Generative AI

Model compilation

Taking real_esrgan-x4plus as an example

export PRODUCT_CHIP=qualcomm-qcs6490-proxy
python3 -m qai_hub_models.models.real_esrgan_x4plus.export --chipset ${PRODUCT_CHIP} \
        --target-runtime qnn_context_binary  --height 128 --width 128 --quantize w8a8 \
        --num-calibration-samples 10

• --chipset: Specifies the target chip to run on
• --target: Runtime specifies the target runtime
• --Height: Target Model Input Height
• --width: Target model input width
• --quantize: Specifies the quantization method
• --num-calibration-samples: Specifies the number of quantization calibration set images

The above command generates a mode id and a file named real_ esrgan-x4plus. qnd_text-binary, which is the file for running the model on the target chip.

Run demo

python3 -m qai_hub_models.models.real_esrgan_x4plus.demo --eval-mode on-device --hub-model-id <mode-id> --chipset ${PRODUCT_CHIP}

• Please modify the hub-model-id parameter to the final printed hub-model-id of the model compilation result

Use the NPU of the local device for inference verification

Reference NPU Development Guide

Q&A

• How to use models to develop app ？

  Qualcomm provides the ai-engine-direct-helper SDK, which provides Python and C++ interfaces for users to develop apps, load models, and run inference. For details, please refer to ai-engine-direct-helper。