Audio Library Development Roadmap
Even though this audio library already does far more than any other audio library for Arduino compatible boards, so much remains to be done.Here is a list of the long-term plans for the Teensy audio library.
24 Bit Data & Faster Sample Rates
This feature is listed first, even though it's the one item on this list that is not planned, because requests for 24+ bits and higher sample rates are by far the most time consuming forum topics & questions. Please, refrain from arguing (without code contribution) the merits of higher resolution and faster sampling.The Teensy Audio Library is designed for 16 bit data, because the ARM Cortex-M4 processor on Teensy 3.1 has special "DSP instructions" and features that accelerate processing 16 bit signals. Adding a 17th bit incurs a heavy performance penalty. More bits also require the code to be rewritten. It's not simply a matter of changing variable definitions from 16 to 32 bits.
Within the filters and synthesis objects, 32, 48 and 64 bit intermediate results are used extensively to achieve good quality results. Not everything is 16 bits, but the samples in and out of every object are.
Even though the Cortex-M4 ARM is a very capable chip, it's still meant to be a low power microcontroller. This audio library attempts to provide great audio, targeted at low power projects. The choice of 16 bits and 44.1 kHz allows great sound for most uses, with a pretty good trade-off in hardware utilization for this single-chip microcontroller.
However, at some point, a limited set of 32 bit objects will likely be added to the library. These will use a pair of 16 bit ports, to easily integrate with the rest of the library. Useful ways to contribute to this effort involve actually testing other DACs and ADCs with the existing I2S master & slave mode objects, or submitting pull requests, or (maybe) sending test hardware.
Faster sample rates may be possible using the I2S slave mode objects. So far, nobody has really tested this. If you actually do give it a try, please post on the forum with your findings.
However, the primary focus of the Teensy Audio Library is 16 bit, 44.1 kHz data.
Update: higher performance audio hardware is in the experimental phase...
Update: 192 kHz experiments.
MP3 & AAC Playing
FrankB has already done some work to play MP3 and AAC files. Eventually, this needs to be integrated nicely with the library.SPI Flash Memory
The Teensy Audio Shield has a location to solder a 16 MByte SPI flash memory. Pete (El Supremo) wrote some initial code to play (and maybe record) using this chip. So far, it hasn't been integrated into the library. Obviously, the plan is to bring this code in soon...My plan for SPI flash support involves a simple partition scheme. The memory should be usable as one big chunk, or for several smaller samples. One of the partitions might later be used to support wave tables for synthesis.
Update: raw file playing from flash is now supported
Recording To SD Cards
Continuous recording to SD cards remains an elusive feature.A record queue object has recently been added to the library, to allow you to try building this in your Arduino sketch. The queue allows about 150 ms of audio to buffer while your sketch does other work (eg, waiting for the SD card to no longer be busy with actually writing to the media).
Sadly, the Arduino SD library is just not up to the task of making this easy. Eventually, perhaps as Teensyduino adds support for Arduino 1.5.X, we can transition away from the old SD library to Bill's newer SdFat library?
Improved Simultaneous SD Playback
Again, the Arduino SD library and limitations of SD card media are problematic.Many "class 10" SD cards are only fast for sequential access. When 3 or 4 files are read, one block at a time, the controller inside the card can add terrible latency, which stalls the audio library.
Future versions of the WAV play object might support upsampling from 11 and 22 kHz, as the memory play object does.
An option may someday be added to coordinate SD card access between multiple instances of this object. For example, if 6 instances all read 44 kHz mono steams, each needs to read a block from the SD card on every other audio interrupt (128 samples).
More experimentation is needed with more SD cards. This issue is absolutely on my radar. Polyphonic playback from SD card media is critically important. The library needs to do more, automatically, to deal with the latency issues of common SD cards.
FFT CPU Usage Improvement
The 1024 point FFT consumes 52% CPU time on 1 of every 4 audio updates. On the other 3 of 4 updates, it merely stores the incoming audio, to be analyzed on the 4th update.While this technically isn't a bug, a better design would distribute the CPU load over all 4 updates. Easier said than done, but I do want to improve this someday.
SoundFont & Wave Table Synthesis
Currently, the waveform synth object supports arbitrary waveforms, so in theory wave table synthesis is available in the library today.In practice, much more is needed to make great-sounding wave table synthesis easy to use. SoundFont is an open format with a huge base of existing, free and inexpensive samples. My hope is to build basic SoundFont synthesis into a future version of the library, and a simple GUI-based tool to import SoundFont files and export to a format that be dropped into an Arduino sketch or perhaps the SPI flash memory.
Update: Wavetable-Synthesis project.
Clipping Detecton & Reporting
Several objects can cause clipping if their input signal is too large and they have gain.The library should do more to help report and troubleshooting clipping. I'll probably add functions similar to the CPU usage reporting, for clipping.
Maybe the library should also have an option to turn on a LED for certain length of time when ANY object has any internal clipping? I believe there are actually pretty efficient ways to implement this. It's really a feature that's needed.
Reverb
Mixer with Automatic Limiting and/or Soft Clipping
A more sophisticated mixer that handles signals adding to greater than 1.0 in more graceful ways would really help projects that run into clipping issues. It should be a simple drop-in replacement for the normal 4-channel mixer, with good default settings.Volume Analysis
Volume analysis is needed, with adjustable attack and decay response times. Currently the library has waveform peak analysis, which works very well, but doesn't reproduce the response speed of a traditional analog VU meter.Javascript Implementation of wav2sketch
The wav2sketch program is a command line tool. Ideally, this should be implemented in client-side Javascript, so it's easier to convert short sound clips for use with the memory player.More Waveform Synthesis Options
Today the only waveform which allows another signal to modulate its frequency is a sine wave. Many synthesis applications need frequency modulated sawtooth, ramp, triangle or other waveforms.More Waveform Control Signals
Some synthesis applications need special control signals, usually as inputs to the frequency modulated sine wave, multiplier object (amplitude modulation) or variable filter.These can be created using the DC object, fed into the envelope object, and perhaps modified by other signals using the mixer and multiplier for math operations. At the needs of these synthesis applications become clearer, special objects for directly (and very efficiently) creating these control waveforms should be added to the library. The coding is pretty simple, but figuring out elegant ways to meet such a wide range of needs with just a small number of objects having a manageable number of functions requires more investigation.
Note Onset Detection & Beat Estimation
So many awesome wearable LED and Burning Man art installation projects want great beat detection, often from a very low quality microphone signal!This sort of analysis could probably be built on top of the 1024 point FFT object, with a LOT of code. It could be done more efficiently within the library.
I plan to work on this. Like many things, it's easy to imagine, and probably possible to make work easily over a narrow range of inputs, but not so easy to really make highly reliable for lots of music types and without pristine quality audio input.
Karplus-Strong & Other Physical Model Synthesis
Many physical synthesis models exist. Some are very efficient. I'm personally very interested to implement these, eventually.Update: Karplus-Strong is now supported.
Dynamic Updates & Web Designer Control
Today, the audio objects really need to be statically allocated. Some distant future version will feature working object destructors, and constructors capable of adding objects and connections into a live system.Eventually, I want to create an example that listens for commands to create & delete objects, and control many of their functions, and of course corresponding code in the GUI design tool. As you drag objects onto the canvas, and connect them, messages would be sent to your Teensy running that example, to actually implement as you draw.
This wonderful future would also have GUI control for the selected object in the right-side panel, so you'd be able to control the objects and cause them to make sound, see the live output of analysis objects in your browser, and so on. Just a "small matter" of more programming.....
USB Audio Interface
A (likely distant) future version of Teensyduino should have an option in Tools > USB Type to create an audio device. Of course, it would provide input and output objects you could connect into the rest of the library, so your computer could stream audio in and out of the system, while also simultaneously streaming in and out of all the other supported objects.Update: USB Audio now works, select it in Tools > USB Type.
Ethernet or Wifi (CC3000) Streaming
Likewise, streaming over network communication is an awesome feature I really want to implement someday. Ideally, it would work with Apple's device streaming, so an iPhone or Mac could stream in to the library, and maybe the library could sent to an airplay device?The CC3000 and Arduino Ethernet libraries probably need performance improvements to be able to handle this application.