Local LLM Chatbots on Embedded Hardware
Bachelor's thesis: Efficient local execution of LLM-based chatbots on the Raspberry Pi 5 – comparing llama.cpp and TinyChatEngine (AWQ) across inference speed, memory, power consumption, and model quality.
Project Description
The thesis investigates whether local LLM inference on CPU-based embedded hardware is practically viable. TinyChatEngine (MIT Han Lab, AWQ quantization) and llama.cpp (GGUF, Q4_K_M) are compared head-to-head. Since the Raspberry Pi 5 has no built-in power measurement interface, an external measurement setup was developed using an ESP32 microcontroller and two INA219 current sensors. A FastAPI proxy (LLMProxy) abstracts both backends behind an OpenAI-compatible interface and records per-request metrics. Model quality is assessed using MMLU, MATH500, and TinyBenchmarks.
Highlights
- llama.cpp (Q4_K_M) achieves 2.1–2.3 tok/s on the Raspberry Pi 5
- TinyChatEngine (AWQ) achieves 0.5–1.6 tok/s — consistently slower
- 1B and 3–5B models exceed the 3 tok/s interactive dialogue threshold
- External power measurement via ESP32 + INA219 + InfluxDB/Grafana
- OpenAI-compatible LLMProxy for automated benchmarking (FastAPI)
- Evaluation: MMLU, MATH500, TinyBenchmarks, LocalScore
Technologies
Category
AI / ML
Period
2025–2026
Bachelor's Thesis
PDF · 2.8 MB