Finally, I’d like to present  the most sophisticated project I’ve done so far, build around the idea turning Arduino board into a DSP.  The results are really impressive for small microprocessor, with low memory size and low MIPS. IMHO, arduino provides better results, than Windows Vista VR system, with 1 GB / 2.2 GHz  hardware, for short one-two words commands, of course.
No HMM, neural networks, or other very popular and “scientifically sounding” theories, were considered to be implemented in the algorithm. Google brings up  millions links on a topic, just ask, but only few of them are designed on really scientific concept, rather than dumb data base “sharpening”. I’m not saying they are completely wrong, and I’m not an expert in the field, but they are not smart ether. My decision is simple 2D cross-correlation. Basically, the heart of the recognition algorithm is similar to an image matching program, which works the same way for voice/sound.  To create a Spectrogram image, arduino is continuously monitoring sound level via microphone, and start capturing data when VOX threshold is exceeded. After input array “X” filled up, data transfered on next level to calculate FFT. The same “conveyor belt” works between FFT and Filtering, flags raised when data is ready, and flags lowered when process finished. The only difference is a speed, conveyor belt is running faster passing data ADC-FFT, and slower at Filter-Correlation stage, as it requires 64 regular cycles to complete spectrogram image in one SuperCycle.  The most time consuming part is Edge Enhancement / HPF Filtering of the spectrogram. I’m still looking around to improve performance of this stage, as it holds all process back from to be fully “Real Time”.
 Specification:
-  4 kHz sampling rate:  2 kHz voice freq. range;
-  64 FFT subroutine,    62.5 Hz spectral resolution;
-  16 x 64 Spectrogram Image, around 1 second max voice password;
-  duration of the Cross-Correlation < 5 milliseconds;
-  duration of the FFT+SQRT+Compression < 4 milliseconds;
-  duration of the Edge Enhancement ~ 35 milliseconds;Main cycle time frame is 16 milliseconds, it’s defined by sampling rate x FFT size, 0.25 x 64 = 64 millisecond. Super-cycle 1.024 is needed only because EE prevents all processes to be completed in less than 16 milliseconds. There is a resources left, to increase sampling up to 8 or even 12 kHz, I just had no time to conduct experiments if it is beneficial.

There is a Command Line Interface, built-in the software, which control “record” and debug “print” functions, 7 commands for now:
if (incomingByte == ‘x’) {           // INPUT ADC DATA
if (incomingByte == ‘f’) {           // FFT OUTPUT
if (incomingByte == ‘s’) {           // SPECROGRAMM PRE  FILTERED
if (incomingByte == ‘g’) {           // SPECROGRAMM POST FILTERED
if (incomingByte == ‘r’) {           // RECORD SPECROGRAMM TO EEPROM
if (incomingByte == ‘p’) {           // PLAY SPECROGRAMM FROM EEPROM
if (incomingByte == ‘m’) {           // FREE MEMORY BYTES

Software is written for AtMega328p microprocessor, Arduino Uno board or similar. For others, all referenced registers has to be replaced with appropriate names for microprocessor.Compiles on 022 IDE, there are some conflicts with 1.0 IDE, that I was not feel myself right to troubleshoot yet. For better understanding some math background, have a look at my previous posts.

Link to download a sketch:   Voice_Recognition_24_01

Analog front-end is the same, as I used in my first project: Color Ogran
There is not much could be improved on this part, and I again used both inputs – from microphone to do tests with my own voice, and also from “line” input, for single tone test generated by computer during debugging. Next picture shows “s” command print-out in the serial monitor window, after I pronounce a word : “Spectrogram” . Due limited size of the window, data printed with 90 degree rotation, left-right is frequencies bands direction, and up-down is time. Lower freq. on left side (60 Hz) and higher (2 kHz) on the right.  The same time 3D images generated in right view angle.

This is how spectrogram looks like after “g” command entered in serial monitor and word sounds just right after that:

Next couple images created with single tone frequency  (320 Hz), just to show more clear “internal properties” of the filtering, again “s” and “g” commands were entered:

Well, as tone sounds continuously, it shows filtering in one direction only, and not the best tutorial on edge-enhancement theory. (“Home brew” lab limits). The same time last picture shows, that each “peek” on the original spectrogram, become surrounded by negative smaller peeks, resulting in “0″ overall sum  on 3×3 foot-print, and consequently on the whole map. In electronics it goes under HPF name, and essence of process is to remove DC component, plus attenuate  Low Frequencies.
Excelent on-line book

Short manual:
to be completed later

This blog considered to be next stage in the series published earlier, concentrated around the idea tracking object in the space. There are an enormous quantity of similar projects could be developed on this platform. I’ll name a few:

- star / rocket / vehicle tracking;
- follow me / robot / hands / cap etc;
- navigation to charging station / landing pad / around area;
- contact-less measurements rotational speed / angle to surface / shifting.

All of this on a few dollars micro-controller and cheap CMOS camera! Real-time, up to 60 Hz update rate!
Please, check on the first and second version, as I’d skip explanation basic design concept here.

  Most important features in this series of projects:

* 1. LOCALIZATION XY.
* 2. RANGE FINDER Z.
* 3. TRACKING 3-D.
* 4. TRACE TOOL.
* 5. TRACKING 6-D.
* 6. OPTICAL MAGNET.

Feature considered to be independent, so you can star  from  project 1:
http://fftarduino.blogspot.com/2011/12/arduino-laser-3d-tracking-range-finder.html
than move on next stage, and so on depends on a budget, parts availability or your interest!

This version of hardware/software system design capable to track object in

6 – D ++ space:

*   -  Linear motion along X, Y, Z coordinates (3D);
*   -  Rotation around fixed axis (6D);

I put two ++ plus signs, in order to underline capability of the hardware design to track Rotation of the object based not only on distance measurements, but also Reflectivity. As Power Control Loop strictly hold lasers radiation under control, simple calculation in periodicity of the emitted power would provide information about angular speed round / cylindrical object. Phase difference in 4 signals gives rotational center for Z. It’s also apply for linear motion, tracking of the object could be based on reflectivity or distance or BOTH simultaneously ,  which opens enormously great amount of possibilities.

Optical Magnet:
*  – Attract closest surface;
*  – Repel;
*  – Attract or repel surface with specific reflectivity (BLACK, WHITE, OR COLOR)!!!
*    * work in progress, Reflectivity math are not implemented yet       *
*    * algorithm to track rotation is not included, this is version for demonstration purposes  mainly.*

8 January, 2012.
Release notes of the version 3.2 software:

-  Video is De-interlaced, full image size 512 (active 492) lines;
-  digitalWrite, analogWrite functions of the arduino IDE were replaced by direct port manipulation, in order  to improve time performance of critical section of the code – interrupt subroutine and to avoid blocking interruption call (functions have locking mechanism in theirs body);
- minor changes in Power Control Loop algorithm, to prevent oscillation;

Link to download Arduino Uno sketch:  Optical_Magnet_6D_V3

 finished…

 In order to minimize error rate of this process, Masking Shadow Theory (MST) was developed. Masking shadow for each note is calculated in several steps and result is compared with notes magnitude in the cycle. If magnitude is less than shadow, than led corresponding to this note wouldn’t  lights up. There are five steps for now, but as I say, work in progress. 
Step 1 (masking shadow 0):  noise floor, which is common for all notes. 
Step 2 (masking shadow 1):  shadow from neighboring notes, that includes 8   notes on left and right side, in inverse proportion to their distances. 
Step 3 (masking shadow 2):  shadow from the note, which is located 1 octave below. Or in other words, cross check if current bins value isn’t second harmonic from sounding note 1 octave below  it. 
Step 4 (masking shadow 3):  similar to step 3, the only difference is, cross check if current bin (note) isn’t third harmonic from sounding note below  it. 
Step 5 (masking shadow 5):  similar to step 3 and 4,  cross check if current note isn’t fifth harmonic from sounding note below  it.

Masking shadow is multiplied by notes specific coefficients after steps 3 – 5, to accommodate significantly richer spectral content for lower octaves.  Formula for calculation of the coefficients, is the trickiest   part of all project. What I’ve discovered, coefficient not just varying between notes / tones, they dynamically varying during “life-time” of the tone.

25 Sept. 2011 

There are two variables have been considered: speed, octave range. Third one, “not technical” – is a price for the project. 
 Octave range is defined by  RAM memory available on chip. 2K on UNO. As maximum processing array size is must be power of 2 ( FFT Radix-2 ), array couldn’t be more than  1024 bytes, next value – 2048 is size of all RAM, that obviously couldn’t be taken. Size of array defines maximum quantity of frequency bins at the output 1024 / 4 = 256. Divided by 4 as there is real / imaginary part, and only half real part is present data. What is interesting, that musical octave is nothing else than doubling of the tones frequency, so it follows binary arithmetic rules… If I will count from high side, the upper octave would occupied half of all 256 bins, from bin 256 to bin 128. Simply because frequency / musical tone spaced logarithmically (LOG_2), and bins spaced equally. Next octave takes half what left over, from bin 128 to bin 64. Third octave 64 to 32, and fourth 32 to 16. Can I go more down ? No. There are 12 notes in each octave. It means, that after fourth octave counting down , I arrived to location where bins and tones spaced almost in sequence, bin 16 – C3, bin 17 – C3#, bin 18 – D3. Well there is four more ( 15, 14, 13, and 12 ), but it doesn’t change much, as it only 1/3 of octave and only would complicate multiplexing LED display.
This is why variable octave range = 4. Summing up, increasing octave range by 1 ( to 5 octaves ) would require double memory size (2K processing array, still possible with chip 4K), by 2 ( to 6 octaves ) – four times more memory (4K processing array).

Arduino mega board has 8K RAM, would it be better to design project with it? In first, it cost more money. In second, it has the same CPU performance, and as you will see below, to have more memory w/o faster CPU doesn’t make any sense. CPU wouldn’t be able to process bigger volume of data in time.

Now lets have a look at speed variable. Following math (and design itself), is greatly depends on it. In order to get at least 1/16 note to be visually “alive”, all cycle ( sampling , pre-processing, FFT, post-processing ) has to be completed for less than 64 msec. My impression is, when timing a little bit longer, LED display looks like it shows something, that was played last Saturday night. Invisible real-time connection between “light” and “music” become broken.
Octave range variable ( 4 octave to be specific ), especially low notes starting from C3, begging for 8 Hz resolution, as it equals to distance between C3 and C3#. And consequently, for 128 msec sampling frame duration. So, there is a contradiction. To solved this , zero padding was introduced, which help to keep sampling window down to 64 msec, the same time frequency resolution not very far from 8 Hz. To make real-time life show, sampling must continue w/o interruption. Even more, it has to be “overlapped”, as pre-processing (windowing) would cut off beginning and ending of the sampling pull. All three other functions ( FFT, pre- and post-processing ) have to go in parallel and must be completed in the same time frame 64 msec. Arduino platform has 8-bit microprocessor, with low horse power engine under hood. This is why 8-bit math was selected instead of 16-bits ( which would save me a lot of troubles ). Troubles, I’m talking about, are very low dynamic range when integer math and 8 – bit comes together in FFT. Integer math, which gives nice time performance, puts really hard constrain on dynamic range, just because it performs “scaling” before and after every “butterfly”. And every scaling procedure brings in rounding error, which grows enormously, as there are 2304 butterfly (9216 round operation) for N = 512 FFT. Special attention must be payed, to keep rounding error under control, the same time not to increase calculation time too much or integer math would not make any sense. What I find out, there is  an excellent algorithm to make “symmetrical” 1/2 bit rounding, but it almost doubles calculation FFT timing, which I obviously, could not afford. So, I choose other path, to increase dynamic range. Compression algorithm on the input data. The easiest way to do it, is “clipping”. Set couple lines in the code:
             if ( x[i] >  127 )  x[i] =  127; 
             if ( x[i] < -127 )  x[i] = -127; 
and all good. Not quite. It will do a great job for any other signal (vibration from accelerometer for example), except music…..
Clipping generates a very high level of harmonics, and ones again , it couldn’t be afford, as it just undermine basic idea of the project – MUSICAL note recognition.
 
 Summary: Scaling extends dynamic range of integer FFT on 24 dB ( 4 bits ).

8- bit FFT dynamic range is +36 dB;
scaling                               +24 dB;
noise                                 -   3 dB;     
———————————————————–
Overall                                 57 dB.   

Link to download a scketch:


Arduino_Musical_Notes_Recognition