eIQ® GenAI Flow : Conversational AI Software Pipeline on Edge Devices
eIQ Gen-AI Flow is a modular, end-to-end software pipeline optimized to bring large and small language models (LLMs and SLMs) to NXP applications processors like the i.MX 95, i.MX 93 and i.MX 8M Plus. The pipeline includes wake word, speech-to-text, text-to-speech and Gen-AI models to run on device CPU and/or NPU. Gen-AI Flow also includes retrieval augmented generation (RAG) to securely and easily fine-tune LLMs with custom, private knowledge sources like PDFs.
Software Details
Diagram
Choose a diagram:
Features
Wake Word Engine (WWE)
- Recognize key phrases and triggers to wake up the conversational AI system. This feature is enabled by NXP Voice Intelligent Technology (VIT).
Speech-To-Text (STT)
- Convert human generated audio into text, intent (Speech to Intent) and tokens for further processing. ASR uses Whisper_small and Moonshine AI models for mutlilingual speech-to-text.
Large Language Models (LLMs)
- Support for leading open foundation models like Llama, Qwen and Danube with pre-optimized ONNX models quantized and able to run on CPU or NPU.
Text-to-Speech (TTS)
- Convert text and LLM outputs into AI-generated speech for a fully conversational interface. TTS uses the VITS model available in ONNX.
Retrieval Augmented Generation (RAG)
- Fine tune LLMs and SLMs with private and domain-specific knowledge sources. Simply input a PDF and get a compact database out. The database is stored on-device and accessed by the AI model on-demand for efficiency and data privacy.
System Requirements
eIQ Gen-AI Flow Demonstrator
- This demo requires a Linux BSP and Python 3.13
eIQ RAG Database Generator
- Linux PC environment. Python 3
eIQ® GenAI Flow Benchmark
| Performance | Platform | Configuration | TTFA Avg (s) | CPU Avg (%) | Memory Avg (MB) | STT WER* (%) | LLM TTFT (s) | LLM TPS (tok/s) | TTS RTF |
|---|---|---|---|---|---|---|---|---|---|
| High Performance | NXP i.MX95 19X19 EVK | whisper-small.en + rag + danube-500M-q8 (with NPU) + tts | 3.27 | 41.48 | 4794 | 0 | 0.28 | 11.55 | 0.36 |
| moonshine-base+ rag + tts | 1.04 | 27.76 | 1103 | 2.12 | N/A | N/A | 0.31 | ||
| Lightweight | NXP i.MX93 11X11 EVK | moonshine-base + rag + danube-500M-q4 + tts | 4.98 | 79.06 | 1533 | 2.12 | 1.56 | 6.2 | 0.74 |
| moonshine-tiny + rag + tts | 1.32 | 40.05 | 783 | 3.17 | N/A | N/A | 0.43 | ||
| Minimal | NXP i.MX8MM EVK | moonshine-base + rag + no_llm + tts | 1.99 | 33.12 | 986 | 2.12 | N/A | N/A | 0.77 |
| moonshine-tiny + rag + no_llm + tts | 1.55 | 27.69 | 926 | 3.17 | N/A | N/A | 0.78 | ||
| Ultra-Light | NXP i.MX91 11X11 EVK | moonshine-tiny + rag + no_llm + text (no TTS) | N/A | 55.97 | 723 | 3.17 | N/A | N/A | N/A |
Metrics Legend:
Downloads
Quick reference to our software types.
3 downloads
-
Application Example Software
eIQ RAG Database Generator
-
Application Example Software
eIQ VLM Demonstratornew
-
Libraries
eIQ GenAI Flow Demonstrator
Note: For better experience, software downloads are recommended on desktop.
