Real-time pitchshifting performance

Android
1

In my journey to use Juce in my music app (Android only) I am trying to see if real-time pitchshifting is feasible.

I’ve tried a few things but all of them result in the audio thread running on > 70% with ‘just’ pitchshifting. Since I would like to add more audio effects in the future I think >70% is too much.

Below is what I tried so far. It’s all working fine, except for it taking too much out of the CPU for my needs. Is this just the way it is because pitchshifting is such a heavy operation? Or am I missing something that would improve performance?

I’ve read about SoundTouch as well. Since that is running in the time-domain I expect it to be better for performance. I prefer the options listed below since they are in the frequency-domain, but maybe that’s just not an option?

I’m using deviceManager.getCpuUsage() to check CPU usage.

Audio-Effects/Pitch Shift at master · juandagilc/Audio-Effects · GitHub
I implemented this algorithm in my app and made sure Juce is using FFTW or PFFFT. No luck here.

Rubberband
Below you can find the code used to run

void MainComponent::prepareToPlay (int samplesPerBlockExpected, double sampleRate)
{
    rubberBandStretcher = new RubberBand::RubberBandStretcher(sampleRate,
                                                              2,
                                                              RubberBand::RubberBandStretcher::OptionProcessRealTime,
                                                              1.0,
                                                              1.5);

    rubberBandStretcher->setMaxProcessSize(samplesPerBlockExpected);
}

void MainComponent::getNextAudioBlock (const juce::AudioSourceChannelInfo& bufferToFill)
{
    rubberBandStretcher->process((const float* const*)bufferToFill.buffer->getArrayOfReadPointers(), bufferToFill.buffer->getNumSamples(), false);
    rubberBandStretcher->retrieve((float* const*)bufferToFill.buffer->getArrayOfWritePointers(), bufferToFill.buffer->getNumSamples());
}

The cmakelists.txt file used to build:

project(rubberband)
cmake_minimum_required(VERSION 2.8)

file(GLOB rubberband_src
    "src/audiocurves/*.*"
    "src/base/*.*"
    "src/dsp/*.*"
    "src/float_cast/*.*"
    "src/kissfft/*.*"
    "src/pommier/*.*"
    "src/speex/*.*"
    "src/system/*.*"
    "src/*.*"
)

add_library(rubberband ${rubberband_src})

target_include_directories(rubberband
        PRIVATE src
        PUBLIC include)

# Macro definitions
add_definitions(-DUSE_PTHREADS -DPROCESS_SAMPLE_TYPE=float)
add_definitions(-D_LIB -D_USE_MATH_DEFINES -DHAVE_KISSFFT -DUSE_SPEEX)

# CXX_FLAGS
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -DNDEBUG -DNO_THREAD_CHECKS -DNO_TIMING")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -ffast-math -mfpmath=sse -msse -msse2 -O3 -ftree-vectorize")

install(TARGETS rubberband
        LIBRARY DESTINATION lib/${ANDROID_ABI}
        ARCHIVE DESTINATION lib/${ANDROID_ABI})

install(DIRECTORY include DESTINATION include)
2

The state of the art for realtime pitch shifting is PSOLA, which is an entirely time domain, granular-based approach.

Here’s some example code - this may not be production ready, it’s still under development - Limes/analyzer.h at main · benthevining/Limes · GitHub

1 Like
3

PSOLA is good if a singular pitch exists, but for more complex signals frequency domain are superior.

4

Thanks for the PSOLA suggestion! As @stenzel mentioned and from what I read about it it’s mainly focussed on speech pitch shifting.

In my app (a loopstation app) I intend to allow multiple audio effects (like pitch shifting, reverb, echo etc…) on self-recorded music. Therefor it seems PSOLA won’t be a good fit. If I misunderstand please let me know!

5

This is true. If you’re expecting polyphonic input, a resampling-based approach may be your best bet.

6

i wouldn’t worry about that too much. pitchshifting is creative work and these types of effects always come with compromises anyway. people would rather have a weird sound than weird limitations, i think

7

That’s true as well. I will check it out anyway, thanks for the responses!

In the mean time, if anyone knows about a high-performant frequency-domain pitchshifter (or any other tips) please let me know!