Can I send my input channelData to an external API?

Audio Plugins
1

Hi All, I’m a developer with expertise in Python, C++ and Machine Learning. I have the knowledge of DSP as well. But I’m totally new to Audio Programming.At the moment, I’m looking to implement an API that will send my audio channelData to an external API. Is it possible to do in JUCE?

void NewProjectAudioProcessor::processBlock (juce::AudioBuffer<float>& buffer, juce::MidiBuffer& midiMessages)
{
    juce::ScopedNoDenormals noDenormals;
    auto totalNumInputChannels  = getTotalNumInputChannels();
    auto totalNumOutputChannels = getTotalNumOutputChannels();

   
    for (auto i = totalNumInputChannels; i < totalNumOutputChannels; ++i)
        buffer.clear (i, 0, buffer.getNumSamples());

    for (int channel = 0; channel < totalNumInputChannels; ++channel)
    {
        auto* channelData = buffer.getWritePointer (channel);
        //send this channelData over to an API
    }
}
2

yes-ish… :slight_smile: you can call any code you want, but you have to be aware of the realtime constraints in processBlock, and ensure that the called API doesn’t violate those constraints.

3

Thank you for replying. What do you suggest if I’m looking to send my vocals sang in a DAW to an external API using a VST plugin? I want to do some processing on the vocals with the help of that external API and then get those vocals back in DAW.

4

I don’t understand your question? How do you mean ‘suggest’? You will have a buffer of samples which you pass to the API. I assume it will return a modified buffer? If the sample buffer formats differ, you would have to convert back and forth. Also, when you say ‘external API’, do you just mean a function call to a 3rd party library? or sending the data to another application, or service elsewhere?

5

first of all, you can just use buffer.getArrayOfWritePointers() to get the whole audio instead of per channel. it is given as 2d pointer of float (or double) and whatever you do with it will be output. the host calls processBlock in quick intervals, so you need stuff to perform fast. if the host doesn’t get a buffer back in time there will be discontinuities due to skipped blocks. if you need to implement a library that was not designed for realtime purposes, like many timestretch ones, you have to either look for a better library or implement workarounds, for example by making sure to already call the allocating library functions in prepareToPlay or by implementing a fifo for the types of dsp processors that need a certain amount of samples to do their thing. check out juce’ fft tutorial as an example

6

What I mean is, is there an alternative method to transmit your vocal audio data from a VST plugin to a Django service for processing? The Django service will carry out its processing, which takes approximately 30 seconds for every 20 seconds of vocal data. Given the real-time constraints of the processBlock function in the VST plugin, are there alternative approaches to handle this delay and data transfer more efficiently?If I’m still not clear please let me know.

7

Is there data being sent back to the plugin?

If not, it’s not that difficult of a problem. You will copy your audio buffer to another buffer for your transmission thread to eventually send. It sounds like you want to send a larger chunk than you receive in the audio callback, so after a number of audio callbacks, the outbound buffer will be full, at which point your other thread will send the buffer. You will likely want to do double buffering, so the collector and sender can run simultaneously. Since you aren’t reading/writing to the same buffer, you shouldn’t need any locking (which you shouldn’t do in the audio callback). Updating the ‘current buffer’ for the audio callback to use should just be a pointer update, and thus atomic, also not needing any locking.

8

Yes, data is being sent back to the plugin .I want to return those vocals to the plugin after the processing is done by django service.Django service performs some processing and it sure takes a lot of time (e.g 20sec vocals will take about 30sec). Can I use anything other than processBlock to do it? Since processBlock have real-time constraints.

9

Hey! Thank you for contributing.Actually I’ve implemented a django service and I want to receive vocals from vst to the django service. Django service performs some processing and it sure takes a lot of time (e.g 20sec vocals will take about 30sec). Can I use anything other than processBlock to do it? Since processBlock have real-time constraints.

10

You could try ARA to just read once on the track. Similar to what Melodyne would do, if you know that plugin.

2 Likes
11

There will be at least a 50 second delay between the first sample in your big block being returned after the django processing. Just curious, how are you planning to deal with that in your plugin?

12

I believe there is a fundamental problem: the processing of the audio takes longer than the duration of the audio itself. You said that for 20 sec of audio you need at least 30 sec of processing.
In the conventional realtime usage, the plug-in will always lag behind and will never be able to return the processed audio in time for the DAW to reproduce it.
I’m afraid your only chance to have something working is to rely on offline processing or some ARA access to the track data (I’m not an expert on ARA so I don’t know how to do that)

1 Like
13

Just adding to it, while I love django and I used it to create web apps and even a software activation service, I’m afraid it is not the right tool for your use case.
The arguments against are multiple and are already hinted in this thread:

  • the webserver running your django application is not designed to be used in realtime
  • python that is used by django is not designed to work in realtime
  • sending it through an unknown transport layer is not realtime safe (even local sockets may be risky)

So it might be sensible to lose django from that equation and see, if there is a more direct way to integrate your processing into a plugin.

14

Putting aside the real-time consideration momentarily, I have an audio recording that I wish to send to my API. This API hosts a machine learning model that has been trained to apply specific processing to the audio. Following this, my objective is to fetch the processed audio and seamlessly incorporate it back into the Digital Audio Workstation (DAW). Please note that I’m open to exploring different frameworks, and the emphasis is on establishing effective communication with the machine learning model. The specific technology, such as Django, is not a strict requirement; I’m seeking guidance on the best approach for achieving this goal.

15

You can implement a record process in the plugin, that writes the channelData to a file. Once done, you can send it to your server and eventually let the user drag the generated audio back onto the timeline.

Like @Mrugalla pointed out, that is the original Melodyne workflow.
Because that is awkward for the user, they invented the mentioned ARA interface, which is your best option, as others already pointed out.

2 Likes