Embedded systems (e.g., microprocessors in home appliances, industrial equipment, automobiles, etc.) need to accommodate limited computing power and memory availability under cost and power constraints. This makes it challenging to deploy high-precision and high-performance language models on edge devices.

Recent periodic AI-related hot topics have almost always been centered around large language models (LLMs) and generative AI models, a trend that reflects the increasing influence and popularity of these topics in recent years. Applications related to large language models and generative AI models cover a wide range of areas, from open chatbots to task-based assistants. While LLMs have mainly focused on cloud-based and server-side applications, there is also growing interest in deploying these models in embedded systems and edge devices.

Embedded systems (e.g., microprocessors in home appliances, industrial equipment, automobiles, etc.) need to accommodate limited computing power and memory availability under cost and power constraints. This makes it challenging to deploy high-precision and high-performance language models on edge devices.

Deploying LLMs on Edge Devices

In embedded solutions, a key area for leveraging LLMs is natural conversational interaction between operators and machines, i.e., human-machine interfaces (HMIs). Embedded systems can simplify various input options such as microphones, cameras or other sensors, but most systems will not have a full keyboard to interact with LLM models like PCs, laptops and mobile phones. Therefore, embedded systems must be practical when using audio and vision as LLM inputs. This requires a pre-processing module for automatic speech recognition (ASR) or image recognition and classification. Similarly, the output options for interaction are limited. Embedded solutions may not have a screen or it may not be convenient for users to read the screen information. Therefore, after the generative AI model, a post-processing step is required to convert the model output into audio using a text-to-speech (TTS) algorithm. NXP is building eIQ® GenAI Flow to make edge generative AI more practical by adding the necessary pre-processing and post-processing modules to make it a modular flow.

Revolutionizing Applications with LLM

By integrating LLM-based speech recognition, natural language understanding and text generation capabilities, embedded devices are able to provide a more intuitive and conversational user experience. This includes smart home devices that respond to voice commands, industrial machinery controlled via natural language, and car infotainment systems that enable hands-free conversations to guide users or operate in-car features.

LLMs also play a role in embedded predictive analytics and decision support systems in health applications. Devices can embed language models trained with domain-specific data, which then use natural language processing to analyze sensor data, identify patterns, and generate insights, all while running in real time at the edge and protecting patient privacy without sending data to the cloud.

Addressing Generative AI Challenges

There are many challenges to deploying accurate and powerful generative AI models in embedded environments. Model size and memory usage need to be optimized so that LLMs can fit within the resource constraints of the target hardware. Models with billions of parameters require gigabytes of storage, which can be costly and difficult to implement in edge systems. Model optimization techniques such as quantization and pruning apply not only to convolutional neural networks but also to transformer models—an important way for generative AI to overcome model size issues.

Generative AI models like LLMs also have knowledge limitations. For example, their understanding is limited, often providing inconsistent answers, also known as “hallucination,” and their knowledge is limited by the timeliness of their training data. Training models or fine-tuning them through retraining can improve accuracy and context awareness, but this can be costly in terms of data collection and required training compute. Fortunately, where there is demand, there is innovation; this problem can be solved through retrieval augmented generation (RAG). The RAG approach creates a knowledge database with context-specific data that LLMs can reference at runtime to accurately answer queries.

The eIQ GenAI Flow applies the benefits of generative AI and LLMs to edge scenarios in a practical way. By incorporating RAG into the flow, we provide embedded devices with domain-specific knowledge without exposing user data to the original AI model’s training data. This ensures that any changes to the LLM are private and only used locally at the edge.