Resampling every buffer

I am working on a project that requires resampling every buffer from a source that can change sample rates (hint: network audio).

I’m doing the following for the conversion and it does not sound good.

AudioFrame audioFrame; //I didn't write this type.

//the buffer from the network has some unknown samplerate.
auto buffer = getBufferFromNetwork(&audioFrame);

if( sampleRate.load() != audioFrame.sample_rate )
           resample from the sample rate of the network sender to the sample rate of the host.
    DBG( "resampling buffer from " << audioFrame.sample_rate << "hz to " << sampleRate.load() << "hz");
    juce::MemoryAudioSource memoryAudioSource(buffer, true);
    auto preferredRatio = static_cast<double>(audioFrame.sample_rate) / static_cast<double>(sampleRate.load());
    juce::ResamplingAudioSource resampler(&memoryAudioSource, false, audioFrame.no_channels);
    juce::AudioBuffer<float> tempBuffer(audioFrame.no_channels, audioFrame.no_samples);
    juce::AudioSourceChannelInfo asci(&tempBuffer, 0, audioFrame.no_samples);
    buffer = tempBuffer;

//now do whatever I need to do with buffer, since it now 
//contains audio at the host's samplerate

Any idea what is the problem or where to start looking?

One problem might be that the resampler is most likely stateful (haven’t checked the ResamplingAudioSource, bit most interpolators are)