How to handle latency in pitch shifter (rubberband)

1

Hi guys,

Im trying to implement a pitch shifter but I’m facing some difficulties regarding the real time processing.

I’m using RubberBand which sound great when I change the pitch scale but I’ve some question about the latency.

Rubberband give 2 functions:

/**
* Return the processing latency of the stretcher.  This is the
* number of audio samples that one would have to discard at the
* start of the output in order to ensure that the resulting audio
* aligned with the input audio at the start.
*/
getLatency();

and

/**
* Ask the stretcher how many audio sample frames should be
* provided as input in order to ensure that some more output
* becomes available.
* 
* If your application has no particular constraint on processing
* block size and you are able to provide any block size as input
* for each cycle, then your normal mode of operation would be to
* loop querying this function; providing that number of samples
* to process(); and reading the output using available() and
* retrieve().  See setMaxProcessSize() for a more suitable
* operating mode for applications that do have external block
* size constraints.
*/
getSamplesRequired();

However to get a processing without have some empty buffer in the middle of the signal from the start of the stream, I need to have a total latency of getLatency() + getSamplesRequired() + processBlockSize

So for my maximum required sample case (lowest pitch scale possible at 0.5), I’ve 2049 + 2048 + 512 latency sample.
But If I look only at the getLatency() or getSamplesRequired() I should have 2049 or 2048.

Note that I use setMaxProcessSize(spec.maximumBlockSize) which is recomanded when w

Here is my code with the prepare function:

void PitchShifter::prepareRubberBand(const juce::dsp::ProcessSpec& spec) noexcept {
  // Used to transfert data between context and rubberband
  rubberBandBuffer_ = 
  juce::AudioBuffer<float>(spec.numChannels, spec.maximumBlockSize);

// Init rubberband with options
  auto stretcherOptions = 0;
  stretcherOptions |= RubberBand::RubberBandStretcher::OptionProcessRealTime;
  rubberBandStretcher_ =
  std::make_unique<RubberBand::RubberBandStretcher>(spec.sampleRate,
                                                    spec.numChannels,
                                                    stretcherOptions);

  rubberBandStretcher_->setPitchOption(RubberBand::RubberBandStretcher::OptionPitchHighConsistency);
  rubberBandStretcher_->setMaxProcessSize(spec.maximumBlockSize);

  // Set the lowest pitch scale in order to get the max (i.e worst) required
  // number of sample.
  rubberBandStretcher_->setPitchScale(0.5);
  const auto stretcherLatency = rubberBandStretcher_->getLatency();
  const auto requiredSamples = rubberBandStretcher_->getSamplesRequired();
  
  // Get back to the default pitch scale.
  rubberBandStretcher_->setPitchScale(1);

  const auto latency = static_cast<int>(stretcherLatency)
  + static_cast<int>(requiredSamples)
  + spec.maximumBlockSize;

  auto silenceBuffer = juce::AudioBuffer<float>(2, spec.maximumBlockSize);
  auto silenceBlock = juce::dsp::AudioBlock<float>(silenceBuffer);
  silenceBlock.fill(0);

  // Fill the buffer here to avoid empty buffer during the process.
  auto processedSampleCount = 0;
  while(processedSampleCount < latency) {
    rubberBandStretcher_->process(silenceBuffer.getArrayOfReadPointers(),
                                  silenceBuffer.getNumSamples(),
                                  false);
    processedSampleCount += silenceBuffer.getNumSamples();
  }
}

and the process function:

void PitchShifter::processRubberBand(const ContextReplacing& context) noexcept {
  auto ioBlock = context.getOutputBlock();
  auto sampleCount = static_cast<int>(inputBlock.getNumSamples());
  auto channelCount = inputBlock.getNumChannels();

  for (int channelIndex = 0; channelIndex < channelCount; channelIndex++) {
    auto source = ioBlock.getChannelPointer(channelIndex);
    rubberBandBuffer_.copyFrom(channelIndex, 0, source, sampleCount);
  }

  const auto required = rubberBandStretcher_->getSamplesRequired();
  const auto availableSampleCount = rubberBandStretcher_->available();

  rubberBandStretcher_->process(rubberBandBuffer_.getArrayOfReadPointers(),
                                sampleCount,
                                false);

  if (availableSampleCount > sampleCount) {
    auto retrievedSamples =
    rubberBandStretcher_->retrieve(rubberBandBuffer_.getArrayOfWritePointers(),
                                   sampleCount);
  } else {
    // If I don't process in the prepareRubberBand function,
    // I go here and get empty buffer in the real time at the begining and
    // 3 times after few processing in the begining then it's ok.
    rubberBandResultBuffer_.clear();
  }

  ioBlock.copyFrom(rubberBandBuffer_);
}

Is it normal that I have to use the getLatency() + getSamplesRequired() + processBlockSize number of sample to have the right process latency?

Thanks

2

I don’t know about rubberband, but if I compare this code to soundtouch it seems a bit over complicated. I don’t think you should be using anything else other then process(), available() and retrieve() for basic shifting but that’s just from reading the docs, not from actual practice. I tried setting up rubberband but got tired from all the tweaks and mods needed to get it up and running. Is there a simple guide for Windows users on how to set up rubberband?

3

Sorry for the late reply @durvrin

In fact I do what you mention. It’s just that I added few extra things related to JUCE.

The core code for Rubberband I used is what you said:

const auto required = rubberBandStretcher_->getSamplesRequired();
  const auto availableSampleCount = rubberBandStretcher_->available();

  rubberBandStretcher_->process(rubberBandBuffer_.getArrayOfReadPointers(),
                                sampleCount,
                                false);

  if (availableSampleCount > sampleCount) {
    auto retrievedSamples =
    rubberBandStretcher_->retrieve(rubberBandBuffer_.getArrayOfWritePointers(),
                                   sampleCount);
  } else {
    // If I don't process in the prepareRubberBand function,
    // I go here and get empty buffer in the real time at the begining and
    // 3 times after few processing in the begining then it's ok.
    rubberBandResultBuffer_.clear();
  }

We have the available(), process() and retrieve() functions.

Unfortunately I’m currently working on macOS but the usage should be the same on Windows. What’s your issue?

4

Eventually I managed to install RB. All I needed was to link the library as usual and NOT open the readme file to avoid confusion.

If I understand your question, in order to know if the current processBlock will have enough output samples then you just need if (availableSampleCount > sampleCount).

5

@DEADBEEF

Were you actually able to get shifting to occur using only those three functions? I’ve been trying to get RB to shift some input buffer and write the output to another buffer but I never have any actual shifting or stretching.

pitchRatio seems to only cause the output to be resampled (setting it to 2 causes an octave shift but half the time) and timeScale causes no change in output regardless of its setting.

6

Yes I’ve managed to shift since the publication of my last message using only these function. However they were a lot of artefact. I use the more complex solution (the ladspa example: rubberband/ladspa at default · breakfastquay/rubberband · GitHub ) which seems to work better.

I’ve never used the time stretching tho.

7

I’ve recently been playing with the Rubberband library for real-time pitch-shifting and have found the latency to also be way higher than reported. The RubberBandStretcher will report latencies below 1k samples, but in my tests the latency is generally over 10k samples. Changing the buffer size of my host made no changes to the latency of the process.
I’ve been experimenting with various combinations of setMaxProcessSize(), getPrefferedStartPad(), and getStartDelay(), but nothing seems to make a real impact.
Did anybody manage to find anything to help with this, or is the Rubberband library just significantly slower than it claims?

8

A quick update:
After talking to somebody at RubberBand - the library isn’t intended for real-time processing. The real-time engine is more for streaming from disk in something like a DJ setup, rather than live input. The best latency you can hope for from the library is 5000 samples and there’s no way of retrieving this latency from the RubberBandStretcher class. My 10k sample delay was probably the best I could get, which means that the library is simply not suitable for my project or anything processing live audio.

2 Likes
9

Thanks you very much for the update @icebreakeraudio , it’s really sad to hear this :confused: it was one of the best candidate. I guess we have to pay the high price for real time processing…

10

See this thread. I think this is good for an audio plugin provided you report some latency to the host.

Although, live use may be problematic for some. I have no problems adapting to the latency but have made a low-latency (non pitch shift) version due to user requests.