| Maestro Audio Framework v 1.0 NXP Semiconductors

Getting started with Maestro framework

# Introduction

This document describes basic usage of Maestro Audio Framework. The intent of this framework is to enable chaining of basic audio processing blocks (called "elements"). These blocks then form stream processing object ("pipeline"). This pipeline can be used for multiple audio processing use cases. The processing blocks can include (but are not limited to) different audio sources (for example file or microphone), decoders or encoders, filters or effects and audio sinks.

Framework overview is depicted on the following picture:

# Supported examples

- maestro_demo
- maestro_usb_mic
- maestro_usb_speaker

The examples can be found in the audio_examples folder of the desired board. The demo applications are based on FreeRTOS and use multiple tasks to form application functionality.

# Streamer setup

In order to setup the audio framework properly, it is necessary to create a streamer with `streamer_create` api. It is also essential to setup the desired hardware peripherals using the functions described in `streamer_pcm.h`. The Maestro example projects consist of several files with reference to audio framework. The initial file is `main.c` with code to create multiple tasks. For features including SD card (in the maestro_demo example, reading a file from SD card or writing to SD card is currently supported), the `APP_SDCARD_Task` is created. The command prompt and connected functionalities are handled by `APP_Shell_Task`. User commands are handled by functions in `cmd.c`. Five main commands are:

- version: prints component version
- file: performs audio file decode and playback
- record_mic: records MIC audio and either:
  - perform voice recognition (VIT)
  - playback on a codec
  - store samples to a file
- usb_mic: record MIC audio and playback to USB port as an audio 2.0 microphone device
- usb_speaker: play data from USB port as an audio 2.0 speaker device

One of the most important parts of the configuration is the `streamer_pcm.c` where can be found the initialization of the hardware peripherals, input and output buffer management. For further information please see also `streamer_pcm.h`

In the Maestro USB examples (maestro_usb_mic and maestro_usb_speaker), the USB configuration is located in the `usb_device_descriptor.c` and `audio_speaker.c` files. For further information please see also `usb_device_descriptor.h` and `audio_speaker.h`.

With creation of streamer object, the message thread is also created. The message thread is placed in `app_streamer.c` file, reads message queue and reacts on following messages:

- STREAM_MSG_ERROR
- STREAM_MSG_EOS
- STREAM_MSG_UPDATE_POSITION
- STREAM_MSG_CLOSE_TASK

The `STREAM_MSG_UPDATE_POSITION` prints info about current stream position, `STREAM_MSG_EOS` stops streamer, `STREAM_MSG_CLOSE_TASK` closes streamer thread and `STREAM_MSG_ERROR` stops streamer and closes thread.

# Commands

## file

File command is only supported in the maestro_demo example. The command creates a streamer pipeline described in function `STREAMER_file_Create` in file `app_streamer.c`. The crucial part in this is defining pipeline type as `STREAM_PIPELINE_FILESYSTEM` or `STREAM_PIPELINE_EAP`. Depending on whether the `EAP_PROC` is defined and on the command option, that may be either `<audio_file>`, `all` or `eap`. This sets up the processing chain from file source to audio sink with decoder and potentially other processing in between. Playback itself is started in `STREAMER_Start` function. The elements that pipeline consists of are:

- TYPE_ELEMENT_FILE_SRC
- TYPE_ELEMENT_DECODER
- TYPE_ELEMENT_EAP (if `EAP_PROC` is defined and `eap` option is used)
- TYPE_ELEMENT_AUDIO_SINK

Other possible element types could be found in `streamer_api.h`.

Each of the elements have several properties that can be accessed using `streamer_get_property`. These properties allow user to change the values of the appropriate elements. The list of properties can be found in `streamer_element_properties.h`. See the example of setting property value in the following piece of code from `app_streamer.c`:

ELEMENT_PROPERTY_T prop;
eap_args.preset_num = *eap_par;
POST_PROCESS_DESC_T eap_proc = {EAP_Initialize_func, EAP_Execute_func, EAP_Deinit_func, &eap_args, 0};
prop.prop = PROP_AUDIOSINK_ADD_POST_PROCESS;
prop.val = (uintptr_t)&eap_proc;
streamer_set_property(handle->streamer, prop, true);
prop.prop = PROP_FILESRC_SET_CHUNK_SIZE;
prop.val = 960;
streamer_set_property(handle->streamer, prop, true);
ELEMENT_PROPERTY_T
Element Property structure.
Definition: streamer_api.h:477

Some of the predefined values can be found in `streamer_api.h`.

## record_mic

Record_mic command is only supported in the maestro_demo example. The command creates pipeline described in function `STREAMER_mic_Create` in file `app_streamer.c`. The pipeline is selected with `STREAM_PIPELINE_PCM`, `STREAM_PIPELINE_MIC2FILE` or `STREAM_PIPELINE_VIT` in `pipeline_type` parameter. Depending on the command option, that may be either `audio`, `file` or `vit`. This configuration takes samples from microphone input and send it to audio sink (speaker(s) or filesystem) with optional processing in between. The elements that pipeline consists of are:

- TYPE_ELEMENT_AUDIO_SRC
- TYPE_ELEMENT_AUDIO_SINK or TYPE_ELEMENT_FILE_SINK or TYPE_ELEMENT_VIT_SINK

Other possible element types and some of the predefined values could be found in `streamer_api.h`.

## usb_mic

Usb_mic command is only supported in the maestro_usb_mic example. The command creates pipeline described in function `STREAMER_mic_Create` in file `app_streamer.c`. The pipeline is selected with `STREAM_PIPELINE_PCM` in `pipeline_type` parameter. This configuration takes samples from microphone input and send it to audio sink (USB port). The elements that pipeline consists of are:

- TYPE_ELEMENT_AUDIO_SRC
- TYPE_ELEMENT_AUDIO_SINK

Other possible element types and some of the predefined values could be found in `streamer_api.h`.

## usb_speaker

Usb_speaker command is only supported in the maestro_usb_speaker example. The command creates pipeline described in function `STREAMER_speaker_Create` in file `app_streamer.c`. The pipeline is selected with `STREAM_PIPELINE_PCM` in `pipeline_type` parameter. This configuration takes samples from USB port and send it to audio sink (speaker output). The elements that pipeline consists of are:

- TYPE_ELEMENT_AUDIO_SRC
- TYPE_ELEMENT_AUDIO_SINK

Other possible element types and some of the predefined values could be found in `streamer_api.h`.

# Configuration options

User can change the pipeline type when creating the streamer object. Currently tested options are: `STREAM_PIPELINE_PCM`, `STREAM_PIPELINE_FILESYSTEM`, `STREAM_PIPELINE_MIC2FILE`, `STREAM_PIPELINE_VIT` and `STREAM_PIPELINE_EAP`. There are more pipelines defined, but supporting them is planned for future.

# Supported features

Current version of the audio framework supports some optional features. These can be limited for some MCU cores or boards variants.

## Codecs

The maestro_demo can play multiple files with the `file` option (`STREAM_PIPELINE_FILESYSTEM`). The `opus` (as standalone or as a part of `ogg` encapsulation) and `mp3` codecs are supported for now. For detailed code handling this file extension please check `cmd.c` file and `shellFile()` function. Supported codecs and its option are:

Decoder | Sample rates [kHz] | Number of channels | Bit depth
--- | --- | --- | ---
MP3 | 8, 16, 24, 32, 48, 11.025, 22.050, 44.1 | 1, 2 (mono/stereo) | 16
opus | 8, 16, 24, 48 | 1, 2 (mono/stereo) | 16

## VIT

VIT can be enabled for the `record_mic` command using the `vit` option. In the `STREAMER_mic_Create` function in the `app_streamer.c` file, the `VIT_PROC` definition ensures the creation of pipeline with VIT (`STREAM_PIPELINE_VIT`). More details about VIT are available in the VIT package, which is located in `middleware\vit\{Cortex_M7, HIFI4}\Doc\`(depending on the platform).

## EAP

EAP can be enabled for the `file` command using the `eap` option. In the `STREAMER_file_Create` function in the `app_streamer.c` file, the `EAP_PROC` definition ensures the creation of pipeline with EAP (`STREAMER_PIPELINE_EAP`). More details about EAP are available in the EAP package, which is located in `middleware\EAP\Doc\`.