MongoDB - Build AI That Scales
Mini toolsRecomputer Rk Llm
This repository utilizes Docker to package large language models and multimodal models optimized for Rockchip platforms. It provides a unified calling interface that is compatible with the OpenAI API, making it easy for users to integrate and use these models.
README
Introduction
This repository utilizes Docker to package large language models and multimodal models optimized for Rockchip platforms. It provides a unified calling interface that is compatible with the OpenAI API, making it easy for users to integrate and use these models.
Hardware Prepare
For reComputer RK3588 and reComputer RK3576.
LLM
VLM
Speed test
Note: A rough estimate of a model's inference speed includes both TTFT and TPOT.
Note: You can usepython test_inference_speed.py --helpto view the help function.
python -m venv .env && source .env/bin/activate
pip install requests
python llm_speed_test.py💞 Top contributors:
<a href="https://github.com/Seeed-Projects/reComputer-RK-LLM/graphs/contributors">
<img src="https://contrib.rocks/image?repo=Seeed-Projects/reComputer-RK-LLM" alt="contrib.rocks image" />
</a>
🌟 Star History
Reference: rknn-llm
