- Voice Processing
- Multimodal GenAI on Edge Devices Made Possible
Simplified and Optimized Generative AI at the Edge with eIQ GenAI Flow
eIQ Gen-AI Flow is a modular, end-to-end software pipeline optimized to bring large and small language models (LLMs and SLMs) to NXP applications processors like the i.MX 95, i.MX 93 and i.MX 8M Plus. The pipeline includes wake word, speech-to-text, text-to-speech and Gen-AI models to run on device CPU and/or NPU. Gen-AI Flow also includes retrieval augmented generation (RAG) to securely and easily fine-tune LLMs with custom, private knowledge sources like PDFs.
Software Details
Features
Wake Word Engine (WWE)
- Recognize key phrases and triggers to wake up the conversational AI system. This feature is enabled by NXP Voice Intelligent Technology (VIT).
Speech-To-Text (STT)
- Convert human generated audio into text, intent (Speech to Intent) and tokens for further processing. ASR uses Whisper_small and Moonshine AI models for mutlilingual speech-to-text.
Large Language Models (LLMs)
- Support for leading open foundation models like Llama, Qwen and Danube with pre-optimized ONNX models quantized and able to run on CPU or NPU.
Text-to-Speech (TTS)
- Convert text and LLM outputs into AI-generated speech for a fully conversational interface. TTS uses the VITS model available in ONNX.
Retrieval Augmented Generation (RAG)
- Fine tune LLMs and SLMs with private and domain-specific knowledge sources. Simply input a PDF and get a compact database out. The database is stored on-device and accessed by the AI model on-demand for efficiency and data privacy.
System Requirements
eIQ Gen-AI Flow Demonstrator
- This demo requires a Linux BSPand Python 3.13
eIQ RAG Database Generator
- Linux PC environment. Python 3
eIQ® GenAI Flow Benchmark
| Performance | Platform | Configuration | TTFA Avg (s) | CPU Avg (%) | Memory Avg (MB) | STT WER* (%) | LLM TTFT (s) | LLM TPS (tok/s) | TTS RTF |
|---|---|---|---|---|---|---|---|---|---|
| High Performance | NXP i.MX95 19X19 EVK | whisper-small.en + rag + danube-500M-q8 (with NPU) + tts | 3.27 | 41.48 | 4794 | 0 | 0.28 | 11.55 | 0.36 |
| moonshine-base+ rag + tts | 1.04 | 27.76 | 1103 | 2.12 | N/A | N/A | 0.31 | ||
| Lightweight | NXP i.MX93 11X11 EVK | moonshine-base + rag + danube-500M-q4 + tts | 4.98 | 79.06 | 1533 | 2.12 | 1.56 | 6.2 | 0.74 |
| moonshine-tiny + rag + tts | 1.32 | 40.05 | 783 | 3.17 | N/A | N/A | 0.43 | ||
| Minimal | NXP i.MX8MM EVK | moonshine-base + rag + no_llm + tts | 1.99 | 33.12 | 986 | 2.12 | N/A | N/A | 0.77 |
| moonshine-tiny + rag + no_llm + tts | 1.55 | 27.69 | 926 | 3.17 | N/A | N/A | 0.78 | ||
| Ultra-Light | NXP i.MX91 11X11 EVK | moonshine-tiny + rag + no_llm + text (no TTS) | N/A | 55.97 | 723 | 3.17 | N/A | N/A | N/A |
Metrics Legend:
- TTFA Avg: Time To First Audio (seconds), the lower the better
- CPU Avg: Average CPU utilization (%), the lower the better
- Memory Avg: Average memory usage (MB)
- STT WER: Automatic Speech Recognition Word Error Rate (%), the lower the better
- LLM TTFT: Large Language Model Time To First Token (seconds), the lower the better
- LLM TPS: Large Language Model Tokens Per Second, the higher the better
- TTS RTF: Text-To-Speech Real Time Factor, the lower the better
Downloads
Quick reference to our software types.
2 downloads
-
Application Example Software
eIQ RAG Database Generator
-
Libraries
eIQ GenAI Flow Demonstrator
Note: For better experience, software downloads are recommended on desktop.
