RKLLM

• https://github.com/airockchip/rknn-llm

模型转换（X64）

• https://github.com/airockchip/rknn-llm/tree/main/doc

参考上面文档：

1. 创建环境conda create -n rkllm python=3.12
2. 安装工具pip install rkllm_toolkit-1.3.0-cp312-cp312-linux_x86_64.whl
3. 下载模型modelscope download --model deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --local_dir ./DeepSeek-R1-Distill-Qwen-7B
4. 转换模型/data/rkllm/rknn-llm-release-v1.3.0/examples/rkllm_api_demo/export

• 生成数据python generate_data_quant.py -m /data/rkllm/DeepSeek-R1-Distill-Qwen-7B
• 转换模型python export_rkllm.py

部署模型（aarch64）

/data/rkllm/rknn-llm-release-v1.3.0/examples/rkllm_server_demo/rkllm_server

安装依赖

pip install flask gradio

server

python3 flask_server.py --rkllm_model_path ./DeepSeek-R1-Distill-Qwen-7B_W8A8_RK3588.rkllm --target_platform rk3588

gradio

python3 gradio_server.py --rkllm_model_path ./DeepSeek-R1-Distill-Qwen-7B_W8A8_RK3588.rkllm --target_platform rk3588

NPU

cat /proc/meminfo | grep -E "CmaTotal|CmaFree"
cat /sys/kernel/debug/rknpu/load