Resources to learn to code vsts (c++) and implement machine learning (python)

Getting Started
1

Hello

I am currently learning Python. My next goal is to start learning more about machine learning / ai / neural networks.

Afterwards, I want to start learning C++.

My goal is to be able to combine both tools with the help of juce to create machine trained audio plugins someday.

I know that this will need years, but I am willing to invest time.

I just cant find many in depth resources on VSTs and machine learning / AI and how to combine Python with C++ to finish such projects.

So I want to ask, if you know some reliable resources, preferably book or in depth courses / videos on that topic. I am looking for basic resources covering general vst coding, but also resources that cover the amchine learning part and its implementation too.

I could think of different kinds of projects in the future, even if I have nothing detailed in mind right now (it is difficult to make detailed plans, when you don’t know whats even possible).

But I could think of machine learned midi generation, an ai compressor trained with files processed by hardware, where the “ai” compressor applies similar behavior (transient response, noise level, stereo field, attack/release), machine learned eq etc.

I know some of my ideas already exist in some ways, but I have fun creating things on my own.

Thanks a lot.

2

The reason you can’t find many in-depth resources is that not a lot of people are doing this kind of thing yet. The field is much smaller than computer-vision, for example.

Some resources:

1 Like
3

Thanks! Thats at least something helpful to start with.
I will take a close look at those links.

4

Of course it’s completely up to you but imo learning python first is a mistake. When you switch over to c your brain will be completely fried.

Python is extreme basics syntax, you don’t need to specify anything. You just say

x = 2

Whereas c will read this as

Syntax error; “x” is undefined, WTF is x?

You need to specify that it is of type type and whether or not the value can be changed like

// the value is set in stone as 2. X will always be 2
const int x = 2;

// the value can change
int x = 2;

There’s also dynamically typed issues where c++ read top to bottom so

const float getX()
{
    return x; // error; what is x?
}

const float x = 2.0f;

Or in python;

// the variable shifts are stupid
x = 0 -> integer
x = 2.05 -> double?
x = 5 -> back to integer?

C would give an error like

int x = 0;
x = 1.0f; // wrong type error
int x = 5; // redefinition error

// you can do it like 
float y = static_cast<float>(x);
y = 1.0f

Also the libraries available and ways to implement are vastly different. With python there is an ocean of “AI” libraries most popular being tensorflow. In cpp there isn’t such a big basket to pick from. And you have to set up paths and link everything yourself.
But on the flip side, c++ is a billion times faster than python and is capable of far more.

I’m no pro so take everything I say with a grain of salt, it’s just my opinion.

1 Like
5

Hey thanks.

I understand your point. The only reason I start with Python is, because I already have to learn it for my university studies. I just dont want the struggle to learn two languages at the same time. So I will focus on C afterwards.

6

Python is the de facto language for machine learning these days (or at least for deep learning) so it makes perfect sense to start with that.

I do a lot of my DSP prototyping in Python notebooks and then afterwards convert it to C++. Especially for doing your research, Python provides a much nicer set of tools than C++. For turning the research into an actual product, that’s where C++ comes in.

7

I’m working at a company that actually builds plugins that use AI, so I have a good idea at least about our technology stack. I think that there is no specific literature out there that describes exactly how to build an AI driven audio plugin. It’s more a matter of building up skills in multiple areas: Have a good understanding of “conventional” digital signal processing, know the challenges and solutions for realtime capable audio processing (this is where C++ comes in) and have a solid understanding of AI.

For us this means that our developer team consists of multiple people with various skills in the areas outlined above – and more like e.g. building great user interfaces and devops. I’d tend to say that no team member has all the skills to get the job done alone, simply because building AI driven plugins that bring real benefit to the user is a pretty complex task. Still, this should not demotivate you getting started, just be prepared that there is a lot to learn and that you need to search for knowledge in various contexts and that you’ll have to put the pieces together yourself.

Regarding languages: Yes, Python is an excellent choice for both, prototyping dsp algorithms and all kinds of machine learning related stuff. It’s used on a daily basis by our algorithm and AI experts. On the other hand, we don’t use any piece of Python in our actual plugin code, it’s all translated to C++ to achieve the runtime performance that is expected from an audio plugin. All popular machine learning frameworks out there offer you some C/C++ interface. Of course they are in no way designed to be used directly from the audio thread, so you’ll have to figure out how to run your machine learning tasks on background threads that are fed with samples from the audio thread in a lock-free way in case you want to perform analysis of the incoming audio stream, which is again something that needs a bit of knowledge on how to build real-time safe C++ code.

Of course, this is a super broad overview, but I hope that it will help you finding some good starting points. Good luck and have fun on that journey!

3 Likes