8.3 KiB
libsonic Home Page
Download the latest tar-ball from here.
The source code repository can be cloned using git:
$ git clone git://github.com/waywardgeek/sonic.git
The source code for the Android version, sonic-ndk, can be cloned with:
$ git clone git://github.com/waywardgeek/sonic-ndk.git
There is a simple test app for android that demos capabilities. You can install the Android application from here
There is a new native Java port, which is very fast! Checkout Sonic.java and Main.java in the latest tar-ball, or get the code from git.
Overview
Sonic is free software for speeding up or slowing down speech. While similar to other algorithms that came before, Sonic is optimized for speed ups of over 2X. There is a simple sonic library in ANSI C, and one in pure Java. Both are designed to easily be integrated into streaming voice applications, like TTS back ends. While a very new project, it is already integrated into:
- espeak
- Debian Sid as package libsonic
- Android Astro Player Nova
- Android Osplayer
- Multiple closed source TTS engines
The primary motivation behind sonic is to enable the blind and visually impaired to improve their productivity with free software speech engines, like espeak. Sonic can also be used by the sighted. For example, sonic can improve the experience of listening to an audio book on an Android phone.
Sonic is Copyright 2010, 2011, Bill Cox, all rights reserved. It is released as under the Apache 2.0 license. Feel free to contact me at waywardgeek@gmail.com. One user was concerned about patents. I believe the sonic algorithms do not violate any patents, as most of it is very old, based on PICOLA, and the new part, for greater than 2X speed up, is clearly a capability most developers ignore, and would not bother to patent.
Comparison to Other Solutions
In short, Sonic is better for speech, while WSOLA is better for music.
A popular alternative is SoundTouch. SoundTouch uses WSOLA, an algorithm optimized for changing the tempo of music. No WSOLA based program performs well for speech (contrary to the inventor's estimate of WSOLA). Listen to this soundstretch sample, which uses SoundTouch, and compare it to this sonic sample. Both are sped up by 2X. WSOLA introduces unacceptable levels of distortion, making speech impossible to understand at high speed (over 2.5X) by blind speed listeners.
However, there are decent free software algorithms for speeding up speech. They are all in the TD-PSOLA family. For speech rates below 2X, sonic uses PICOLA, which I find to be the best algorithm available. A slightly buggy implementation of PICOLA is available in the spandsp library. I find the one in RockBox quite good, though it's limited to 2X speed up. So far as I know, only sonic is optimized for speed factors needed by the blind, up to 6X.
Sonic does all of it's CPU intensive work with integer math, and works well on ARM CPUs without FPUs. It supports multiple channels (stereo), and is also able to change the pitch of a voice. It works well in streaming audio applications, and can deal with sound streams in 16-bit signed integer, 32-bit floating point, or 8-bit unsigned formats. The source code is in plain ANSI C. In short, it's production ready.
Using libsonic in your program
Sonic is still a new library, but is in Debian Sid. It will take a while for it to filter out into all the other distros. For now, feel free to simply add sonic.c and sonic.h to your application (or Sonic.java), but consider switching to -lsonic once the library is available on your distro.
The file main.c is the source code for the sonic command-line application. It is meant to be useful as example code. Feel free to copy directly from main.c into your application, as main.c is in the public domain. Dependencies listed in debian/control like libsndfile are there to compile the sonic command-line application. Libsonic has no external dependencies.
There are basically two ways to use sonic: batch or stream mode. The simplest is batch mode where you pass an entire sound sample to sonic. All you do is call one function, like this:
sonicChangeShortSpeed(samples, numSamples, speed, pitch, rate, volume, useChordPitch, sampleRate, numChannels);
This will change the speed and pitch of the sound samples pointed to by samples, which should be 16-bit signed integers. Stereo mode is supported, as is any arbitrary number of channels. Samples for each channel should be adjacent in the input array. Because the samples are modified in-place, be sure that there is room in the samples array for the speed-changed samples. In general, if you are speeding up, rather than slowing down, it will be safe to have no extra padding. If your sound samples are mono, and you don't want to scale volume or playback rate, and if you want normal pitch scaling, then call it like this:
sonicChangeShortSpeed(samples, numSamples, speed, pitch, 1.0f, 1.0f, 0, sampleRate, 1);
The other way to use libsonic is in stream mode. This is more complex, but allows sonic to be inserted into a sound stream with fairly low latency. The current maximum latency in sonic is 31 milliseconds, which is enough to process two pitch periods of voice as low as 65 Hz. In general, the latency is equal to two pitch periods, which is typically closer to 20 milliseconds.
To process a sound stream, you must create a sonicStream object, which contains all of the state used by sonic. Sonic should be thread safe, and multiple sonicStream objects can be used at the same time. You create a sonicStream object like this:
sonicStream stream = sonicCreateStream(sampleRate, numChannels);
When you're done with a sonic stream, you can free it's memory with:
sonicDestroyStream(stream);
By default, a sonic stream sets the speed, pitch, rate, and volume to 1.0, which means no change at all to the sound stream. Sonic detects this case, and simply copies the input to the output to reduce CPU load. To change the speed, pitch, rate, or volume, set the parameters using:
sonicSetSpeed(stream, speed);
sonicSetPitch(stream, pitch);
sonicSetRate(stream, rate);
sonicSetVolume(stream, volume);
These four parameters are floating point numbers. A speed of 2.0 means to double speed of speech. A pitch of 0.95 means to lower the pitch by about 5%, and a volume of 1.4 means to multiply the sound samples by 1.4, clipping if we exceed the maximum range of a 16-bit integer. Speech rate scales how fast speech is played. A 2.0 value will make you sound like a chipmunk talking very fast. A 0.7 value will make you sound like a giant talking slowly.
By default, pitch is modified by changing the rate, and then using speed modification to bring the speed back to normal. This allows for a wide range of pitch changes, but changing the pitch makes the speaker sound larger or smaller, too. If you want to make the person sound like the same person, but talking at a higher or lower pitch, then enable the vocal chord emulation mode for pitch scaling, using:
sonicSetChordPitch(stream, 1);
However, only small changes to pitch should be used in this mode, as it introduces significant distortion otherwise.
After setting the sound parameters, you write to the stream like this:
sonicWriteShortToStream(stream, samples, numSamples);
You read the sped up speech samples from sonic like this:
samplesRead = sonicReadShortFromStream(stream, outBuffer, maxBufferSize);
if(samplesRead > 0) {
/* Do something with the output samples in outBuffer, like send them to
* the sound device. */
}
You may change the speed, pitch, rate, and volume parameters at any time, without having to flush or create a new sonic stream.
When your sound stream ends, there may be several milliseconds of sound data in the sonic stream's buffers. To force sonic to process those samples use:
sonicFlushStream(stream);
Then, read those samples as above. That's about all there is to using libsonic. There are some more functions as a convenience for the user, like sonicGetSpeed. Other sound data formats are supported: signed char and float. If float, the sound data should be between -1.0 and 1.0. Internally, all sound data is converted to 16-bit integers for processing.