Automatic Synthesiser Programmer


#1

Hey all and the ever present Jules and the rest of the team. I’ve been lurking and searching these forums for knowledge as I’ve used JUCE a few times for pet projects like synthesisers using the Maximilian library for sound generation and RapidMix lib for parameter mapping.

I study at Goldsmiths University of London and I’m entering my final year of Undergraduate study and I’ve elected to work on creating an automatic synthesiser programmer - i.e you give the JUCE program the sound, and it will return the parameters for the most similar sound it can create with a given synth - using JUCE to host the VSTs.

For the project I have to keep frequent logs of my activities, and for some reason they (my University) wants me to use tumblr. I thought at best this would be the most relevant place to leave the progress as there might be helpful feedback and at worst, the code will be left here for some new lurker like I was to cobble up and use in some manner!

The plugin host example is a bit overkill for my needs, so I’ve cobbled together code I could find and scrape from my thick head. What I am building here is a terminal VST host similar to Mrs Watson. Seriously, I know you all know what you are doing here, so apologies to any JUCE gods that are offended in the manner I use C++ and JUCE in general!

I’ve got a few days start on this so will add in what I have already and will add semi-frequent updates in the upcoming few months.

#include "../JuceLibraryCode/JuceHeader.h"
#include <thread>
#include <chrono>

using namespace juce;

//=============================================================================
AudioPluginInstance* createSynthInstance(int    _samplerate,
                                         int    _buffersize,
                                         String _pluginPath = "")
{
    OwnedArray<PluginDescription> pluginDescriptions;
    KnownPluginList pluginList;
    AudioPluginFormatManager pluginFormatManager;
    pluginFormatManager.addDefaultFormats();
    
    for (int i = pluginFormatManager.getNumFormats(); --i >= 0;)
        pluginList.scanAndAddFile(_pluginPath,
                                  true,
                                  pluginDescriptions,
                                  *pluginFormatManager.getFormat(i));
    
    jassert(pluginDescriptions.size() > 0);
    
    String errorMessage;
    auto plugin = pluginFormatManager.createPluginInstance(*pluginDescriptions[0],
                                                           _samplerate,
                                                           _buffersize,
                                                           errorMessage);
    if (plugin == nullptr)
    {
        std::cout << "createSynthInstance - Error: "
                  << errorMessage.toStdString()
                  << std::endl;
        exit(1);
    }
    
    plugin->prepareToPlay(_samplerate, _buffersize);
    plugin->setNonRealtime(true);

    return plugin;
}

//=============================================================================
MidiBuffer getMidiNote(int    _midiChannel,
                       uint8  _midiPitch,
                       uint8  _midiVelocity,
                       double _noteSeconds,
                       double _sampleRate)
{
    MidiMessage onMessage    = MidiMessage::noteOn (_midiChannel,
                                                    _midiPitch,
                                                    _midiVelocity);

    MidiMessage offMessage   = MidiMessage::noteOff(onMessage.getChannel(),
                                                    onMessage.getNoteNumber(),
                                                    onMessage.getVelocity());
    
    onMessage.setTimeStamp(0);
    offMessage.setTimeStamp(onMessage.getTimeStamp() + _noteSeconds * _sampleRate);
    
    MidiBuffer midiBuffer;
    midiBuffer.addEvent(onMessage, onMessage.getTimeStamp());
    midiBuffer.addEvent(offMessage, offMessage.getTimeStamp());
    
    return midiBuffer;
}

//=============================================================================
int main (int argc, char* argv[])
{
    double sampleRate      = 44100;
    int    bufferSize      = 512;
    int    numberChannels  = 2;
    String vstpath         = "/Users/admin/Downloads/MrTramp2_VST_AU_Win_OSX_UB_21/OSX/MrTramp2.vst";
    String audioFile       = "test";
    String audioPath       = "/Users/admin/Desktop/VstHost/Source/";
    String audioType       = ".wav";
    double noteLength      = 8;
    int    midiChannel     = 2;
    int    midiPitch       = 59;
    int    midiVelocity    = 127;
    int    bitsPerSample   = 16;
    double renderLength    = noteLength * 2;
    int    numberOfBuffers = int(std::ceil(renderLength * sampleRate / bufferSize));
    
    AudioSampleBuffer audioBuffer(numberChannels, bufferSize);
    
    auto midiNote = getMidiNote(midiChannel,
                                midiPitch,
                                midiVelocity,
                                noteLength,
                                sampleRate);
    
    auto vst = ScopedPointer<AudioPluginInstance>(createSynthInstance(int(sampleRate),
                                                                      bufferSize,
                                                                      vstpath));
    for (int i = 0; i < vst->getNumParameters(); ++i)
        std::cout << vst->getParameterName(i)
                  << ": " << vst->getParameter(i)
                  << std::endl;
    
    std::cout << "\nAmount Parameters: "
              << vst->getNumParameters()
              << std::endl;
    
    const File file((File(audioPath)).getNonexistentChildFile(audioFile, audioType));

    ScopedPointer<FileOutputStream> fileStream(file.createOutputStream());
    WavAudioFormat wavFormat;
    ScopedPointer<AudioFormatWriter> wavWriter = wavFormat.createWriterFor(fileStream,
                                                                           sampleRate,
                                                                           numberChannels,
                                                                           bitsPerSample,
                                                                           StringPairArray(),
                                                                           0);
    for (int i = 0; i < numberOfBuffers; ++i)
    {
        
        vst->processBlock(audioBuffer,
                          midiNote);
        
        wavWriter->writeFromAudioSampleBuffer(audioBuffer, 0, bufferSize);
    }
    
    wavWriter->flush();
    wavWriter = nullptr;
    
    return 0;
}


#2

One thing is bugging me at the moment, aside from a memory leak issue I think involved with the VST instance, is that I can’t get the midi buffer to work properly at the moment. The note off MidiMessage just doesn’t seem to happen at the correct sample so the only way to get a full note is to not have it at all have the synth render ‘note on’ audio right up to the end of the amount of render samples.


#3

So I’m really scratching my head over this. I have slightly modified the priorly posted code. I am creating a midi buffer like this. For the values passed in, look to the next code box.

MidiBuffer getMidiNote(int    _midiChannel,
                       uint8  _midiPitch,
                       uint8  _midiVelocity,
                       double _noteSeconds,
                       double _sampleRate)
{
    MidiMessage onMessage    = MidiMessage::noteOn (_midiChannel,
                                                    _midiPitch,
                                                    _midiVelocity);

    MidiMessage offMessage   = MidiMessage::noteOff(onMessage.getChannel(),
                                                    onMessage.getNoteNumber(),
                                                    onMessage.getVelocity());
    
    double noteSampleLength  = _noteSeconds * _sampleRate;
    
    onMessage.setTimeStamp(0);
    offMessage.setTimeStamp(onMessage.getTimeStamp() + noteSampleLength);
    
    MidiBuffer midiBuffer;
    midiBuffer.addEvent(onMessage, onMessage.getTimeStamp());
    midiBuffer.addEvent(offMessage, offMessage.getTimeStamp());
    
    return midiBuffer;
}

Then, I assign it here, the values being in the comments:

auto midiNote = getMidiNote(midiChannel,  // 2
                            midiPitch,    // 60
                            midiVelocity, // 127
                            noteLength,   // 4
                            sampleRate);  // 44100

Finally I process it here and after the first offline block of audio is processed, the values in the comments are what are present after a break point is placed after the ‘bempty’ line.

auto afirst = midiNote.getFirstEventTime();     // 0
auto alast  = midiNote.getLastEventTime();      // 176400
auto anum   = midiNote.getNumEvents();          // 2
auto aempty = midiNote.isEmpty();               // false

for (int i = 0; i < numberOfBuffers; ++i)       // (i = 0)
{
    vst->processBlock(audioBuffer, midiNote);
     
    auto bfirst = midiNote.getFirstEventTime(); // 0
    auto blast  = midiNote.getLastEventTime();  // 0
    auto bnum   = midiNote.getNumEvents();      // 0
    auto bempty = midiNote.isEmpty();           // true
        
    /* BREAKPOINT SET HERE */

    wavWriter->writeFromAudioSampleBuffer(audioBuffer, 0, bufferSize);
}

Why is the midiBuffer’s MidiMessage events being wiped after the first block is processed? The juce::VSTPluginInstance is looking suspect to me, or far more likely, I am using it incorrectly.


#4

Todays update:

So I still have no idea whats going on with the midi buffer. I’m currently hacking my way around it by having an infinite noteOn; the noteOff occurs before the noteOn event does. This obviously isn’t ideal because it means the envelope generator isn’t going to work correctly. I will look into MidiMessages perhaps next to see if there is an alternative, but from looking in the forums that seems to be more oriented towards real-time processing, and may still be overkill for a solitary note that I need. In the mean time today, I decided to press on with other matters.

Essentially I am going to to hook up a deep net to audio features and the given parameters for the VST to establish a mapping.

I have made it so I can print out CSV files for the parameters and the features, which are currently MFCCS. I will probably end up using something like Yaafe for feature extraction but for now and ease of use I am using Maximilian. A major change is being able to create lots of synth patches and print/render out the appropriate files thanks to the while() loop in main().

On the subject of creating synth patches, There seems to be an interesting problem of creating every possible synth patch. I’m not great at Maths, let alone Combinatorics, but I will probably try to figure out an algorithm that will generate every possible parameter layout given a default parameter step size. I imagine that number must be huge, but there would be a way to cut it down. Currently to generate synth patches I am using a uniform real PRNG, which works fine provided the discrete state space (or parameter size) isn’t too small otherwise there will be many repeated values and ultimately similar sounds!

I need to add command line options so the code doesn’t need to be recompiled each time I need to change the variables.

Oh and I added a rad ASCII text title into the program so it’s like the source I’ve been reading all of these years.

/*
 ==============================================================================
              ______               _          ___  ___
              | ___ \             | |         |  \/  |
              | |_/ /___ _ __   __| | ___ _ __| .  . | __ _ _ __
              |    // _ \ '_ \ / _` |/ _ \ '__| |\/| |/ _` | '_ \
              | |\ \  __/ | | | (_| |  __/ |  | |  | | (_| | | | |
              \_| \_\___|_| |_|\__,_|\___|_|  \_|  |_/\__,_|_| |_|
 
       * *  Command Line VST Audio, Features and Parameter Renderer  * *
 
 ==============================================================================
 */

#include <array>
#include <thread>
#include <chrono>
#include <random>

#include "Maximilian/maximilian.h"
#include "Maximilian/libs/maxiFFT.h"
#include "Maximilian/libs/maxiMFCC.h"

#include "../JuceLibraryCode/JuceHeader.h"

using namespace juce;

//=============================================================================
AudioPluginInstance* createSynthInstance(int    _samplerate,
                                         int    _buffersize,
                                         String _pluginPath = "")
{
    OwnedArray<PluginDescription> pluginDescriptions;
    KnownPluginList pluginList;
    AudioPluginFormatManager pluginFormatManager;
    pluginFormatManager.addDefaultFormats();
    
    for (int i = pluginFormatManager.getNumFormats(); --i >= 0;)
    {
        pluginList.scanAndAddFile(_pluginPath,
                                  true,
                                  pluginDescriptions,
                                  *pluginFormatManager.getFormat(i));
    }
    
    jassert(pluginDescriptions.size() > 0);
    
    String errorMessage;
    auto plugin = pluginFormatManager.createPluginInstance(*pluginDescriptions[0],
                                                           _samplerate,
                                                           _buffersize,
                                                           errorMessage);
    if (plugin == nullptr)
    {
        std::cout << "createSynthInstance - Error: "
        << errorMessage.toStdString()
        << std::endl;
        exit(1);
    }
    
    plugin->prepareToPlay(_samplerate, _buffersize);
    plugin->setNonRealtime(true);
    
    return plugin;
}

//=============================================================================
MidiBuffer getMidiNote(int    _midiChannel,
                       uint8  _midiPitch,
                       uint8  _midiVelocity,
                       double _noteSeconds,
                       double _sampleRate)
{
    MidiMessage onMessage    = MidiMessage::noteOn (_midiChannel,
                                                    _midiPitch,
                                                    _midiVelocity);
    
    MidiMessage offMessage   = MidiMessage::noteOff(onMessage.getChannel(),
                                                    onMessage.getNoteNumber(),
                                                    onMessage.getVelocity());
    
    double noteSampleLength  = -1; //_noteSeconds * _sampleRate;
    
    onMessage.setTimeStamp(0);
    offMessage.setTimeStamp(onMessage.getTimeStamp() + noteSampleLength);
    
    MidiBuffer midiBuffer;
    midiBuffer.addEvent(onMessage, onMessage.getTimeStamp());
    midiBuffer.addEvent(offMessage, offMessage.getTimeStamp());
    
    return midiBuffer;
}

//=============================================================================
bool isUnusedParameter(const String& _parameterName)
{
    return _parameterName == "Param";
}

//=============================================================================
void fillAudioFeatures(const AudioSampleBuffer&             _data,
                       maxiFFT&                             _fft,
                       maxiMFCC&                            _mfcc,
                       std::vector<std::array<double, 13>>& _features)
{
    // Keep it auto as it may or may not be double precision.
    auto readptrs = _data.getArrayOfReadPointers();
    for (int i = 0; i < _data.getNumSamples(); ++i)
    {
        // Mono the frame.
        int channel = 0;
        auto currentFrame = readptrs[channel][i];
        while (++channel < _data.getNumChannels())
        {
            currentFrame += readptrs[channel][i];
        }
        currentFrame /= _data.getNumChannels();
        
        // Extract features.
        if (_fft.process(currentFrame))
        {
            // This isn't real-time so I can take the luxuary of allocating
            // heap memory here.
            double* mfccs = new double[13];
            _mfcc.mfcc(_fft.magnitudes, mfccs);
            
            std::array<double, 13> mfccsFrame;
            std::memcpy(mfccsFrame.data(), mfccs, sizeof(double) * 13);
            _features.push_back(mfccsFrame);
            
            delete[] mfccs;
        }
    }
}

//=============================================================================
void writeFeaturesToCSV(std::vector<std::array<double, 13>>& _features,
                        const String&                        _csvPath,
                        const String&                        _csvName)
{
    File csvFile((File(_csvPath)).getNonexistentChildFile(_csvName,
                                                          ".csv",
                                                          false));
    
    String fileText = "";
    for (size_t i = 0; i < _features.size(); ++i)
    {
        for (size_t j = 0; j < _features[i].size() - 1; ++j)
        {
            fileText += String(_features[i][j]) + ',';
        }
        fileText += String(_features[i][_features[i].size() - 1]) + '\n';
    }
    
    csvFile.replaceWithText(fileText);
    
    _features.clear();
}

//=============================================================================
void writeParametersToCSV(std::vector<std::pair<int, float>> _parameters,
                          const String&                      _csvPath,
                          const String&                      _csvName)
{
    File csvFile((File(_csvPath)).getNonexistentChildFile(_csvName,
                                                          ".csv",
                                                          false));
    String fileText = "";
    for (size_t i = 0; i < _parameters.size(); ++i)
    {
        fileText += String(_parameters[i].first) + ',' + String(_parameters[i].second) + '\n';
    }
    
    csvFile.replaceWithText(fileText);
}

//=============================================================================
int main (int argc, char* argv[])
{
    // Variable declaration and assignment.
    double sampleRate         = 44100;
    int    bufferSize         = 512;
    int    numberChannels     = 2;
    String fileName           = "_TAL-U-No-62_";
    String vstpath            = "/Users/admin/Desktop/VstHost/Plugins/Mac/32Bit/TAL-U-No-62.vst";
    String audioPath          = "/Users/admin/Desktop/VstHost/Renders/";
    String audioFile          = "audio" + fileName;
    String featuresPath       = "/Users/admin/Desktop/VstHost/Features/";
    String featuresFileName   = "features" + fileName;
    String parametersPath     = "/Users/admin/Desktop/VstHost/Parameters/";
    String parametersFileName = "parameters" + fileName;
    double noteLength         = 4;
    int    midiChannel        = 2;
    int    midiPitch          = 60;
    int    midiVelocity       = 127;
    int    bitsPerSample      = 16;
    double renderLength       = noteLength * 2;
    int    numberOfBuffers    = int(std::ceil(renderLength * sampleRate / bufferSize));
    long   amountOfPresets    = 1;
    int    fftSize            = 16384;
    
    // Maximilian setup.
    maxiSettings::setup(sampleRate, numberChannels, bufferSize);
    maxiFFT  fft;
    maxiMFCC mfcc;
    fft.setup(fftSize, fftSize / 2, fftSize / 4);
    mfcc.setup(512, 42, 13, 20, int(sampleRate / 2), sampleRate);
    
    // The container for the features pulled from the synth's output audio.
    std::vector<std::array<double, 13>> features;
    
    // Add parameters indices to the initialiser list to force them to stay
    // at the supplied float. Could be useful for sustain or gain parameters.
    std::vector<std::pair<int, float>> overridenParameters{ std::make_pair(1, 10000.f) };
    
    // Data structure to hold multi-channel audio data.
    AudioSampleBuffer audioBuffer(numberChannels, bufferSize);
    
    // The midi note that goes into the vst.
    auto midiNote = getMidiNote(midiChannel,
                                midiPitch,
                                midiVelocity,
                                noteLength,
                                sampleRate);
    
    // Our vst instance.
    auto vst = ScopedPointer<AudioPluginInstance>(createSynthInstance(int(sampleRate),
                                                                      bufferSize,
                                                                      vstpath));
    
    // The indices of the parameter's that are used. They are stored as pairs
    // with a float intended to be the parameters value.
    std::vector<std::pair<int, float>> parameterIndexValuePairs;
    parameterIndexValuePairs.reserve(vst->getNumParameters());
    
    // This loop gets all the parameters that are ACTUALLY being used.
    int usedParameterAmount = 0;
    for (int i = 0; i < vst->getNumParameters(); ++i)
    {
        if (!isUnusedParameter(vst->getParameterName(i)))
        {
            ++usedParameterAmount;
            parameterIndexValuePairs.push_back(std::make_pair(i, 0.0f));
        }
    }
    parameterIndexValuePairs.shrink_to_fit();
    
    // This will be used to create random parameter samples.
    std::random_device random;
    std::mt19937_64 generator;
    std::uniform_real_distribution<> distribution(0, 1);
    
    // Generate and record random presets.
    while (--amountOfPresets >= 0)
    {
        // Generate a random synth patch from a uniform distribution. Keep
        // overriden parameters the way they were set in the initialiser
        // list.
        std::pair<int, float> copy;
        for (auto& pair : parameterIndexValuePairs)
        {
            // Should we have overriden this parameter's index...
            if (std::any_of(overridenParameters.begin(), overridenParameters.end(),
                            [pair, &copy] (std::pair<int, float> p)
                            {
                                copy = p;
                                return p.first == pair.first;
                            }))
            {
                // Assign the current pair the 'copy' pair so the set of
                // parameters are consistent when it comes to writing the
                // parameters to file.
                pair = copy;
                vst->setParameter(copy.first, copy.second);
            }
            else
            {
                // Else create random value for the parameter.
                pair.second = distribution(generator);
                vst->setParameter(pair.first, pair.second);
            }
        }
    
        // Open a unique .wav in the given path with the given name.
        // There must be a nicer way of writing this..
        const File wavFile((File(audioPath)).getNonexistentChildFile(audioFile,
                                                                     ".wav",
                                                                     false));

        // The objects used to stream the channel data in the audio buffer to
        // the aforementioned unique file.
        ScopedPointer<FileOutputStream> fileStream(wavFile.createOutputStream());
        WavAudioFormat wavFormat;
        AudioFormatWriter* wavWriter = wavFormat.createWriterFor(fileStream,
                                                                 sampleRate,
                                                                 numberChannels,
                                                                 bitsPerSample,
                                                                 StringPairArray(),
                                                                 0);
    
        // Loop through all the buffers and create the audio from the midi data.
        for (int i = 0; i < numberOfBuffers; ++i)
        {
            vst->processBlock(audioBuffer, midiNote);
            
            fillAudioFeatures(audioBuffer, fft, mfcc, features);
  
            wavWriter->writeFromAudioSampleBuffer(audioBuffer, 0, bufferSize);
        }
    
        // Write the .wav, features and parameters files to disk.
        wavWriter->flush();
        writeFeaturesToCSV(features, featuresPath, featuresFileName);
        writeParametersToCSV(parameterIndexValuePairs, parametersPath, parametersFileName);
    }
    
    return 0;
}

#5

Well, because that’s how it works!

Good luck with your project, it looks interesting! And your code looks OK, but I would suggest avoiding leading underscrores for variable names, it’ll make most experienced C++ people cringe! Apart from just being ugly, leading underscores (particularly double underscores) are generally reserved for the language and standard library, so you should never use them in your own code.

One other thought: many plugins won’t work without an event loop running - they may rely on timers to update things, or they may do some initialisation on the event loop. If you’re only planning to use a few plugins that you’ve tested and know to work, then that’s fine, but if you’re planning on letting other people use this with random plugins, then you’d probably need to redesign it to be asynchronous.


#6

Thank you for the tips Jules. I’ve removed the leading underscores, and never plan to use them again.

I am currently looking into the AudioBuffer now, I have managed to make a god awful noise by copying the MidiBuffer in each loop to fill the block to try and prevent it being wiped, so that will need some more consideration.

More importantly, I take it by asynchronous you don’t mean loading the plugin asynchronously, as in AudioPluginFormat::createPluginInstanceAsync, but likely more like in how the plugin host example manages hosts with the graph? Or perhaps the AsyncUpdater class.


#7

I mean structuring your app as an event-loop driven one, not as a traditional single-threaded synchronous command-line thing.


#8

Thanks for the elaboration, I’ll look into that. Also I appreciate your help with the MidiBuffer, it is working as desired now!


#9

Wow Code::Blocks sucks! Would rather work without a debugger in commandline :smile: I have spent the day generating 1000’s of random presets for various VSTs on my Mac. Regarding Linux, I presume there is a reason QT-Creator isn’t supported. I am now making the code play nice with Linux but am having problems with both the plugin host example and my code when loading .so files. I’m hoping compiling 32-bit apps should help. But it also could be how I am using the .so libs.

It’s not really essential to have this program working in Linux because I can generate all the data I need to perform training of neural networks on a Mac and then whack it into python but for completeness’s sake it would be nice to not have to be dependant on a given OS.

One thing is when I compile my code for a 32-bit archetecture I get this:

Compiling juce_graphics.cpp
In file included from /usr/include/freetype2/ft2build.h:37:0,
                 from ../../../JUCE/modules/juce_graphics/juce_graphics.cpp:93,
                 from ../../JuceLibraryCode/juce_graphics.cpp:9:
/usr/include/freetype2/freetype/config/ftheader.h:7:65: fatal error: i386-linux-gnu/freetype2/freetype/config/ftheader.h: No such file or directory
 #  include <i386-linux-gnu/freetype2/freetype/config/ftheader.h>

Perhaps I could change something I pass to the compiler or install a library? Not sure what though, I’m no pro with Linux…


#10

Many updates:

This project is all about using deep learning to map audio features to synthesiser parameters. In order to start with something incredibly simple that will be a lot easier to work with initially, I have spent time writing an incredibly simple FM synthesiser.

The FM plugin has just two parameters; the modulation index and a scalar which affects the frequency of the modulator signal. The scalar dictates what ratio of the current midi note’s frequency is passed to the modulator.

The advantage here is that in order to generate training data from a given synthesiser or plugin, you have to sample it with different parameter configurations. But how many possible parameter configurations are there? It firstly depends on the parameter step size - how many possible increments for each parameter from 0 - 1. I opted for 128, as that felt intuitively correct for midi messages, but I would love to know if I was right or wrong there if anyone knows.

Some of the vsts I was playing with earlier easily had anything between 20 - 60 parameters. Given our 0 - 127 step size for each parameter, how many possible patches could there be? As we are generating data based on our parameters it would be nice to have a uniform representation of every possible parameter.

If we had 20 parameters, the amount of patches that would be need to be generated would be 128^20, or:

1393796574908163946345982392040522594123776

If we had 40:

1942668892225729070919461906823518906642406839052139521251812409738904285205208498176

And if we had 60:

2707685248164858261307045101702230179137145581421695874189921465443966120903931272499975005961073806735733604454495675614232576

Suddenly when looking at these kind of numbers reducing the amount of parameters for an initial first step into deep learning looks quite enticing. I might also add it reminds me somewhat of the AlphaGo game, in that like the amount of possible board states, there are more possible synth patches than atoms in the universe.

My two parameter FM synth has a total of 16384 possible patches with a given parameter step size of 128. That looks like a dataset that is fortunately far more manageable.

After a few small modifications to RenderMan, as I type this he is now crunching through every possible state of the FM vst. After this I will have a folder of features, parameter settings and bounces to work with.

Looking to the future, what is next?

In the training stage of the Neural Network, when the Network is fed a training example which comprises of audio features and their label, the audio parameters. It it will take the audio features and output a set of parameters, and it will need to check how ‘correct’ they are compared to the expected parameters which belong to the given training example. A naive way (and probably the way I’ll start - to again keep things simple) is to take the Euclidean distance from each set of parameters.

There is an issue here however; this assumes that the features (and the sound) obtained from the given set of parameters is unique - no other parameter combination could create that sound from this given engine. It is very likely that many different combinations can produce similar audio (and therefore features,) so becuase I am doing Deep Learning on Linux RenderMan needs to compile on Linux and work properly like it does on OS X. Once this is done, when parameters are generated, they can be fed into RenderMan to get the real features from those parameters, and instead of taking the distance from the training set’s parameters and the Neural Net’s outputted parameters, the distance can be taken from the newly rendered features and the features initially fed into the network for the training example.

Another issue to start thinking about, perhaps after the initial foray into Deep Learning is finished is how to generate the most varying - in terms of audio features - training data from a synth with say 60+ parameters. I am currently doing a random sample, but who is to say that a random sample of even 100,000,000 parameter sets would uniformly represent the range of sounds that synth could make. Currently I am thinking of something like a Genetic Algorithm, in where the fitness function would be greater should the current gene’s output feature be farther from the mean. But more thought will have to go into generating varying datasets.


#11

Here is the code for RenderMan and the FM Synth for those curious few (note you’ll need to include Maximilian files to build both projects):

RenderMan:

Main.cpp

/*
 ==============================================================================
              ______               _          ___  ___
              | ___ \             | |         |  \/  |
              | |_/ /___ _ __   __| | ___ _ __| .  . | __ _ _ __
              |    // _ \ '_ \ / _` |/ _ \ '__| |\/| |/ _` | '_ \
              | |\ \  __/ | | | (_| |  __/ |  | |  | | (_| | | | |
              \_| \_\___|_| |_|\__,_|\___|_|  \_|  |_/\__,_|_| |_|

       * *  Command Line VST Audio, Features and Parameter Renderer  * *

             Copyright (C) 2017  Leon Fedden - All Rights Reserved

 ==============================================================================
 */

#include "Utils.h"
#include "ParameterGenerator.h"

using namespace juce;

//=============================================================================
int main (int argc, char* argv[])
{
    // Convenient class to parse arguments.
    ArgumentsParser parser(argc, argv);
    String arg;

    // If user wants help.
    if (parser.flagExists("-h"))
    {
        printHelp();
        exit(0);
    }

#ifdef __APPLE__
    String defaultVstPath     = "/Users/admin/Library/Audio/Plug-Ins/VST/IncrediblySimpleFMSynth.vst";
#else
    String defaultVstPath     = "/home/tollie/Development/FINAL_PROJECT/plugins/amsynth";
#endif


    // Variable declaration and assignment.
    double sampleRate         = ((arg = parser.getArgument("-s")) != "") ? arg.getDoubleValue() : 44100;
    int    bufferSize         = ((arg = parser.getArgument("-b")) != "") ? arg.getIntValue() : 512;
    String vstPath            = ((arg = parser.getArgument("-v")) != "") ? arg : defaultVstPath;
    String name               = getName(vstPath);
    String audioPath          = "../../../../Renders/";
    String audioFile          = name + "_audio_";
    String featuresPath       = "../../../../Features/";
    String featuresFileName   = name + "_features_";
    String parametersPath     = "../../../../Parameters/";
    String parametersFileName = name + "_parameters_";
    double noteLength         = ((arg = parser.getArgument("-n")) != "") ? arg.getDoubleValue() : 4;
    uint8  midiChannel        = 2;
    uint8  midiPitch          = ((arg = parser.getArgument("-p")) != "") ? uint8(arg.getIntValue()) : 60;
    uint8  midiVelocity       = 127;
    int    bitsPerSample      = 16;
    double renderLength       = ((arg = parser.getArgument("-r")) != "") ? arg.getDoubleValue() : noteLength * 2;
    int    numberOfBuffers    = int(std::ceil(renderLength * sampleRate / bufferSize));
    long   amountOfPresets    = ((arg = parser.getArgument("-a")) != "") ? arg.getIntValue() : 20;
    int    fftSize            = 16384;
    bool   listParameters     = (parser.flagExists("-l")) ? true : false;
    bool   iterateAllPatches  = (parser.flagExists("-i")) ? true : false;

    // Our vst instance.
    auto vst = ScopedPointer<AudioPluginInstance>(createSynthInstance(int(sampleRate),
                                                                      bufferSize,
                                                                      vstPath));
    const int numberChannels  = vst->getTotalNumOutputChannels();

    // Add parameters indices to the initialiser list to force them to stay
    // at the supplied float. Could be useful for sustain or gain parameters.
    std::vector<std::pair<int, float>> overridenParameters;
    getOverridenParameters(overridenParameters,
                           vst->getNumParameters(),
                           parser);

    // Maximilian setup.
    maxiSettings::setup(sampleRate, numberChannels, bufferSize);
    maxiFFT  fft;
    maxiMFCC mfcc;
    fft.setup(fftSize, fftSize / 2, fftSize / 4);
    mfcc.setup(512, 42, 13, 20, int(sampleRate / 2), sampleRate);

    // The container for the features pulled from the synth's output audio.
    std::vector<std::array<double, 13>> features;

    // Data structure to hold multi-channel audio data.
    AudioSampleBuffer audioBuffer(numberChannels, bufferSize);

    // The midi note that goes into the vst.
    auto masterNote = getMidiNote(midiChannel,
                                  midiPitch,
                                  midiVelocity,
                                  noteLength,
                                  sampleRate);

    // The indices of the parameter's that are used. They are stored as pairs
    // with a float intended to be the parameters value.
    std::vector<std::pair<int, float>> parameterIndexValuePairs;
    parameterIndexValuePairs.reserve(vst->getNumParameters());

    // This loop gets all the parameters that are ACTUALLY being used.
    int usedParameterAmount = 0;
    for (int i = 0; i < vst->getNumParameters(); ++i)
    {
        if (!isUnusedParameter(vst->getParameterName(i)))
        {
            ++usedParameterAmount;
            parameterIndexValuePairs.push_back(std::make_pair(i, 0.0f));
        }
    }
    parameterIndexValuePairs.shrink_to_fit();

    // List all available parameters
    String parameterListString("");
    if (listParameters)
    {
        for (auto& pair : parameterIndexValuePairs)
        {
            String leadingSpace = (pair.first < 10) ? " " : "";
            parameterListString = parameterListString +
                                  "Index: " + leadingSpace + String(pair.first) +
                                  ", Name: " + vst->getParameterName(pair.first) + "\n";
        }

        std::cout << parameterListString.toStdString() << std::endl;

        exit(0);
    }
    
    // Used to generate random (or every single) sample of parameters from the
    // parameter space.
    ParameterGenerator parameterGenerator;
    
    // If we are generating all possible parameters, then fill the
    // ParameterMatrix (a vector of arrays of pairs) with all the combinations
    // of parameter values. Reset amount of presets too so we can loop through
    // every one too.
    ParameterMatrix    allParameters;
    if (iterateAllPatches)
    {
        parameterGenerator.fillAllCombinationsOfParameters(128, allParameters);
        amountOfPresets = allParameters.size();
    }
    
    const long totalPatchesSampled = amountOfPresets;

    // Sample all of the desired presets.
    while (--amountOfPresets >= 0)
    {
        // Access the elements from the start, not the end.
        const std::size_t indexFromStart = totalPatchesSampled - 1 - amountOfPresets;
        
        // Print progress
        printProgress(indexFromStart, totalPatchesSampled);
        
        if (iterateAllPatches)
        {
            // Get the current set of parameters.
            const auto currentPatch = allParameters[indexFromStart];
            for (int i = 0; i < currentPatch.size(); ++i)
            {
                // Set the vst parameter with the parameter given.
                const auto currentParameter = currentPatch[i];
                vst->setParameter(currentParameter.first, currentParameter.second);
            }
        }
        else
        {
            // Generate a random synth patch from a uniform distribution. Keep
            // overriden parameters the way they were set in the initialiser
            // list.
            std::pair<int, float> copy;
            for (auto& pair : parameterIndexValuePairs)
            {
                // Should we have overriden this parameter's index...
                if (std::any_of(overridenParameters.begin(), overridenParameters.end(),
                                [pair, &copy] (std::pair<int, float> p)
                                {
                                    copy = p;
                                    return p.first == pair.first;
                                }))
                {
                    // Assign the current pair the 'copy' pair so the set of
                    // parameters are consistent when it comes to writing the
                    // parameters to file.
                    pair = copy;
                    vst->setParameter(copy.first, copy.second);
                }
                else
                {
                    // Else create random value for the parameter.
                    pair = parameterGenerator.getRandomParameter(pair.first);
                    vst->setParameter(pair.first, pair.second);
                }
            }
        }
        
        // Open a unique .wav in the given path with the given name.
        // There must be a nicer way of writing this..
        const File wavFile((File(audioPath)).getNonexistentChildFile(audioFile,
                                                                     ".wav",
                                                                     false));

        // The objects used to stream the channel data in the audio buffer to
        // the aforementioned unique file.
        ScopedPointer<FileOutputStream> fileStream(wavFile.createOutputStream());
        WavAudioFormat wavFormat;
        AudioFormatWriter* wavWriter = wavFormat.createWriterFor(fileStream,
                                                                 sampleRate,
                                                                 numberChannels,
                                                                 bitsPerSample,
                                                                 StringPairArray(),
                                                                 0);

        // Get the note on midiBuffer.
        auto midiNote = getInitialNoteOn(midiChannel,
                                         midiPitch,
                                         midiVelocity);

        // Loop through all the buffers and create the audio from the midi data.
        for (int i = 0; i < numberOfBuffers; ++i)
        {
            // Trigger note off if in the correct audio buffer.
            ifTimeSetNoteOff(noteLength,
                             sampleRate,
                             bufferSize,
                             midiChannel,
                             midiPitch,
                             midiVelocity,
                             i,
                             midiNote);

            // Turn Midi to audio via the vst.
            vst->processBlock(audioBuffer, midiNote);

            // Get audio features and fill the datastructure.
            fillAudioFeatures(audioBuffer, fft, mfcc, features);

            // Write the audio data to a buffer to be flushed.
            wavWriter->writeFromAudioSampleBuffer(audioBuffer, 0, bufferSize);
        }

        // Write the .wav, features and parameters files to disk.
        wavWriter->flush();
        writeFeaturesToCSV(features, featuresPath, featuresFileName);
        writeParametersToCSV(parameterIndexValuePairs, parametersPath, parametersFileName);
    }
    
    std::cout << std::endl;

    vst->releaseResources();

    return 0;
}

ParameterGenerator.h

/*
  ==============================================================================

    ParameterGenerator.h
    Created: 1 Feb 2017 11:17:22pm
    Author:  admin

  ==============================================================================
*/

#ifndef PARAMETERGENERATOR_H_INCLUDED
#define PARAMETERGENERATOR_H_INCLUDED

#include <random>
#include <array>
#include "../JuceLibraryCode/JuceHeader.h"

typedef std::vector<std::array<std::pair<int, float>, 2>> ParameterMatrix;

class ParameterGenerator
{

public:

    ParameterGenerator()
    {
        distribution = new std::uniform_real_distribution<>(0, 1);
    }
    
    ~ParameterGenerator()
    {
        delete distribution;
    }
    
    std::pair<int, float> getRandomParameter(int index)
    {
        float randomValue = (*distribution)(generator);
        return std::make_pair(index, randomValue);
    }
    
    void fillAllCombinationsOfParameters(int              stepSize,
                                         ParameterMatrix& matrixToFill)
    {
        // The amount of parameter sets we are making.
        std::size_t amountToMake = std::size_t(std::pow(stepSize, 2));
        
        // Reserve the room to prevent unessessary allocations
        matrixToFill.reserve(amountToMake);
        for (int i = 0; i < stepSize; ++i)
        {
            // Our set of parameters.
            std::array<std::pair<int, float>, 2> parameters;
            
            // The first parameter.
            parameters[0] = std::make_pair(int(0), float(i) / float(stepSize));
            
            // Loop through again to get every possible combination of the
            // second parameter with the first.
            for (int j = 0; j < stepSize; ++j)
            {
                // The second parameter
                parameters[1] = std::make_pair(int(1), float(j) / float(stepSize));
                matrixToFill.push_back(parameters);
            }
        }
        // Shrink the vector to the correct size.
        matrixToFill.shrink_to_fit();
        
        // This shouldn't fire unless we are getting the combinations wrong.
        jassert(amountToMake == matrixToFill.size());
    }
    
private:
    
    // This will be used to create random parameter samples.
    std::random_device random;
    std::mt19937_64 generator;
    std::uniform_real_distribution<>* distribution;
};


#endif  // PARAMETERGENERATOR_H_INCLUDED

Utils.cpp

/*
  ==============================================================================

    Utils.cpp
    Created: 29 Jan 2017 5:45:03pm
    Author:  Leon Fedden

  ==============================================================================
*/

#include "Utils.h"

//=============================================================================
void printHelp()
{
    String helpString = "\nRenderMan Options:\n\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -s <arg> : Sample rate. Double\n";
    helpString += "               argument which\n";
    helpString += "               shouldn't need setting.\n";
    helpString += "\n";
    helpString += "               Default: 44100\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -b <arg> : Buffer size. Integer\n";
    helpString += "               argument and should\n";
    helpString += "               be power of two.\n";
    helpString += "\n";
    helpString += "               Default: 512\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -v <arg> : VST name. Supply the\n";
    helpString += "               full file path and\n";
    helpString += "               extension. String\n";
    helpString += "               argument e.g:\n";
    helpString += "               '/Path/To/My/VstName.vst'\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -n <arg> : Note length in seconds\n";
    helpString += "               of how long the midi\n";
    helpString += "               note should play.\n";
    helpString += "               Integer argument.\n";
    helpString += "\n";
    helpString += "               Default: 4\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -p <arg> : Pitch of the midi note\n";
    helpString += "               that is passed to the\n";
    helpString += "               vst. Integer argument\n";
    helpString += "               of the midi note.\n";
    helpString += "\n";
    helpString += "               Default: 60 (Middle C)\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -r <arg> : Render length in\n";
    helpString += "               seconds of how long\n";
    helpString += "               the .wav file will be.\n";
    helpString += "\n";
    helpString += "               Default: Twice the midi\n";
    helpString += "                        note length (8)\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -a <arg> : Amount of presets to\n";
    helpString += "               generate. Integer\n";
    helpString += "               argument.\n";
    helpString += "\n";
    helpString += "               Default: 1\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -i       : Iterate over every\n";
    helpString += "               possible combination of\n";
    helpString += "               set of parameters.\n";
    helpString += "               Warning, this is only.\n";
    helpString += "               working for synths with.\n";
    helpString += "               two parameters currently.\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "   -p1 <arg> : You can override\n";
    helpString += "   -p2       : any parameter the synth\n";
    helpString += "   ...       : offers to the supplied\n";
    helpString += "   -pN       : value which is a float\n";
    helpString += "               between 0 and 1 for min\n";
    helpString += "               and max. To override\n";
    helpString += "               parameter 35 to 0.6, you\n";
    helpString += "               would type -p35 0.6.\n";
    helpString += "\n";
    helpString += "               Default: No parameters\n";
    helpString += "                        are overriden!\n";
    helpString += "\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    helpString += "    -l       : List all parameters\n";
    helpString += "               used in the VST with\n";
    helpString += "               with indices. No\n";
    helpString += "               argument needed.\n\n";
    helpString += "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n";
    
    std::cout << helpString.toStdString() << std::endl;
}

//=============================================================================
AudioPluginInstance* createSynthInstance(int    samplerate,
                                         int    buffersize,
                                         String pluginPath)
{
    OwnedArray<PluginDescription> pluginDescriptions;
    KnownPluginList pluginList;
    AudioPluginFormatManager pluginFormatManager;
    pluginFormatManager.addDefaultFormats();
    
    for (int i = pluginFormatManager.getNumFormats(); --i >= 0;)
    {
        pluginList.scanAndAddFile(pluginPath,
                                  true,
                                  pluginDescriptions,
                                  *pluginFormatManager.getFormat(i));
    }
    
    jassert(pluginDescriptions.size() > 0);
    
    String errorMessage;
    auto plugin = pluginFormatManager.createPluginInstance(*pluginDescriptions[0],
                                                           samplerate,
                                                           buffersize,
                                                           errorMessage);
    if (plugin == nullptr)
    {
        std::cout << "createSynthInstance - Error: "
        << errorMessage.toStdString()
        << std::endl;
        exit(1);
    }
    
    plugin->prepareToPlay(samplerate, buffersize);
    plugin->setNonRealtime(true);
    
    return plugin;
}

//=============================================================================
MidiBuffer getMidiNote(int    midiChannel,
                       uint8  midiPitch,
                       uint8  midiVelocity,
                       double noteSeconds,
                       double sampleRate)
{
    MidiMessage onMessage    = MidiMessage::noteOn (midiChannel,
                                                    midiPitch,
                                                    midiVelocity);
    
    MidiMessage offMessage   = MidiMessage::noteOff(onMessage.getChannel(),
                                                    onMessage.getNoteNumber(),
                                                    onMessage.getVelocity());
    
    double noteSampleLength  = noteSeconds * sampleRate;
    
    onMessage.setTimeStamp(0);
    offMessage.setTimeStamp(onMessage.getTimeStamp() + noteSampleLength);
    
    MidiBuffer midiBuffer;
    midiBuffer.addEvent(onMessage, onMessage.getTimeStamp());
    midiBuffer.addEvent(offMessage, offMessage.getTimeStamp());
    
    return midiBuffer;
}

//=============================================================================
void ifTimeSetNoteOff(const double& noteLength,
                      const double& sampleRate,
                      const int&    bufferSize,
                      const uint8&  midiChannel,
                      const uint8&  midiPitch,
                      const uint8&  midiVelocity,
                      const int&    currentBufferIndex,
                      MidiBuffer&   bufferToNoteOff)
{
    double eventFrame = noteLength * sampleRate;
    bool bufferBeginIsBeforeEvent = currentBufferIndex * bufferSize < eventFrame;
    bool bufferEndIsAfterEvent = (currentBufferIndex + 1) * bufferSize >= eventFrame;
    bool noteOffEvent = bufferBeginIsBeforeEvent && bufferEndIsAfterEvent;
    if (noteOffEvent)
    {
        MidiBuffer midiOffBuffer;
        MidiMessage offMessage = MidiMessage::noteOff(midiChannel,
                                                      midiPitch,
                                                      midiVelocity);
        offMessage.setTimeStamp(eventFrame);
        midiOffBuffer.addEvent(offMessage, offMessage.getTimeStamp());
        bufferToNoteOff = midiOffBuffer;
    }
}

//=============================================================================
MidiBuffer getInitialNoteOn(const uint8&  midiChannel,
                            const uint8&  midiPitch,
                            const uint8&  midiVelocity)
{
    MidiBuffer midiOnBuffer;
    MidiMessage onMessage = MidiMessage::noteOn(midiChannel,
                                                midiPitch,
                                                midiVelocity);
    onMessage.setTimeStamp(0);
    midiOnBuffer.addEvent(onMessage, onMessage.getTimeStamp());
    
    return midiOnBuffer;
}

//=============================================================================
bool isUnusedParameter(const String& parameterName)
{
    return parameterName == "Param";
}

//=============================================================================
void fillAudioFeatures(const AudioSampleBuffer&             data,
                       maxiFFT&                             fft,
                       maxiMFCC&                            mfcc,
                       std::vector<std::array<double, 13>>& features)
{
    // Keep it auto as it may or may not be double precision.
    auto readptrs = data.getArrayOfReadPointers();
    for (int i = 0; i < data.getNumSamples(); ++i)
    {
        // Mono the frame.
        int channel = 0;
        auto currentFrame = readptrs[channel][i];
        while (++channel < data.getNumChannels())
        {
            currentFrame += readptrs[channel][i];
        }
        currentFrame /= data.getNumChannels();
        
        // Extract features.
        if (fft.process(currentFrame))
        {
            // This isn't real-time so I can take the luxuary of allocating
            // heap memory here.
            double* mfccs = new double[13];
            mfcc.mfcc(fft.magnitudes, mfccs);
            
            std::array<double, 13> mfccsFrame;
            std::memcpy(mfccsFrame.data(), mfccs, sizeof(double) * 13);
            features.push_back(mfccsFrame);
            
            delete[] mfccs;
        }
    }
}

//=============================================================================
void writeFeaturesToCSV(std::vector<std::array<double, 13>>& features,
                        const String&                        csvPath,
                        const String&                        csvName)
{
    File csvFile((File(csvPath)).getNonexistentChildFile(csvName,
                                                         ".csv",
                                                         false));
    
    String fileText = "";
    for (size_t i = 0; i < features.size(); ++i)
    {
        for (size_t j = 0; j < features[i].size() - 1; ++j)
        {
            fileText += String(features[i][j]) + ',';
        }
        fileText += String(features[i][features[i].size() - 1]) + '\n';
    }
    
    csvFile.replaceWithText(fileText);
    
    features.clear();
}

//=============================================================================
void writeParametersToCSV(std::vector<std::pair<int, float>> parameters,
                          const String&                      csvPath,
                          const String&                      csvName)
{
    File csvFile((File(csvPath)).getNonexistentChildFile(csvName,
                                                         ".csv",
                                                         false));
    String fileText = "";
    for (size_t i = 0; i < parameters.size(); ++i)
    {
        fileText += String(parameters[i].first) + ',' + String(parameters[i].second) + '\n';
    }
    
    csvFile.replaceWithText(fileText);
}

//=============================================================================
String getName(const String& vstPath)
{
    String noExtension;
    if (vstPath.containsWholeWord(".vst"))
        noExtension = vstPath.dropLastCharacters(4);
    else
        noExtension = vstPath.dropLastCharacters(3);
    int i = noExtension.length();
    String name("");
    while (noExtension[--i] != '/')
        name = noExtension[i] + name;
    return name;
}

//=============================================================================
void getOverridenParameters(std::vector<std::pair<int, float>>& overridenParameters,
                            int                                 vstAmountOfParameters,
                            ArgumentsParser&                    parser)
{
    String arg;
    for (int i = 0; i < vstAmountOfParameters; ++i)
    {
        String parameterFlagName = "-p" + String(i);
        if ((arg = parser.getArgument(parameterFlagName)) != "")
        {
            overridenParameters.push_back(std::make_pair(i, arg.getFloatValue()));
        }
    }
}

//=============================================================================
// Stack overflow - search for how to display a progress indictor in pure C/C++
void printProgress(long currentPatch, long amountOfPatches)
{
    int barWidth = 50;
    float progress = float(currentPatch) / float(amountOfPatches);
    
    std::cout << "[";
    int position = int(progress * barWidth);
    
    for (int i = 0; i < barWidth; ++i)
    {
        if (i < position) std::cout << "=";
        else if (i == position) std::cout << ">";
        else std::cout << " ";
    }
    
    std::cout << "] " << int(progress * 100.0) << " %\r";
    std::cout.flush();
}

Utils.h

/*
  ==============================================================================

    Utils.h
    Created: 29 Jan 2017 5:47:49pm
    Author:  Leon Fedden

  ==============================================================================
*/

#ifndef UTILS_H_INCLUDED
#define UTILS_H_INCLUDED

#include <array>
#include <thread>
#include <chrono>
#include <random>

#include "Maximilian/maximilian.h"
#include "Maximilian/libs/maxiFFT.h"
#include "Maximilian/libs/maxiMFCC.h"

#include "../JuceLibraryCode/JuceHeader.h"

//=============================================================================
// http://stackoverflow.com/questions/865668/how-to-parse-command-line-arguments-in-c
// Minimal changes to the work from the above link. Praise the lord for Stack.

class ArgumentsParser
{
public:
    // Store all the command line arguments.
    ArgumentsParser(int &argc, char *argv[]) : emptyString("")
    {
        for (int i = 1; i < argc; ++i)
            arguments.push_back(String(argv[i]));
    }
    
    // Get the argument supplied after a given flag.
    const String& getArgument(const String& flag)
    {
        auto it = std::find(arguments.begin(), arguments.end(), flag);
        if (it != arguments.end() && ++it != arguments.end())
            return *it;
        return emptyString;
    }
    
    // Has a given flag been supplied as an argument?
    bool flagExists(const String &flag)
    {
        return std::find(arguments.begin(), arguments.end(), flag) != arguments.end();
    }
private:
    std::vector<String> arguments;
    String              emptyString;
};

//=============================================================================
void printHelp();

//=============================================================================

AudioPluginInstance* createSynthInstance(int    samplerate,
                                         int    buffersize,
                                         String pluginPath = "");

//=============================================================================
MidiBuffer getMidiNote(int    midiChannel,
                       uint8  midiPitch,
                       uint8  midiVelocity,
                       double noteSeconds,
                       double sampleRate);

//=============================================================================
void ifTimeSetNoteOff(const double& noteLength,
                      const double& sampleRate,
                      const int&    bufferSize,
                      const uint8&  midiChannel,
                      const uint8&  midiPitch,
                      const uint8&  midiVelocity,
                      const int&    currentBufferIndex,
                      MidiBuffer&   bufferToNoteOff);

//=============================================================================
MidiBuffer getInitialNoteOn(const uint8&  midiChannel,
                            const uint8&  midiPitch,
                            const uint8&  midiVelocity);

//=============================================================================
bool isUnusedParameter(const String& parameterName);

//=============================================================================
void fillAudioFeatures(const AudioSampleBuffer&             data,
                       maxiFFT&                             fft,
                       maxiMFCC&                            mfcc,
                       std::vector<std::array<double, 13>>& features);

//=============================================================================
void writeFeaturesToCSV(std::vector<std::array<double, 13>>& features,
                        const String&                        csvPath,
                        const String&                        csvName);

//=============================================================================
void writeParametersToCSV(std::vector<std::pair<int, float>> parameters,
                          const String&                      csvPath,
                          const String&                      csvName);
//=============================================================================
String getName(const String& vstPath);

//=============================================================================
void getOverridenParameters(std::vector<std::pair<int, float>>& overridenParameters,
                            int                                 vstAmountOfParameters,
                            ArgumentsParser&                    parser);

//=============================================================================
void printProgress(long currentPatch, long amountOfPatches);


#endif  // UTILS_H_INCLUDED


#12

And the FM vst here:

PluginEditor.h

/*
  ==============================================================================

    This file was auto-generated!

    It contains the basic framework code for a JUCE plugin editor.

  ==============================================================================
*/

#ifndef PLUGINEDITOR_H_INCLUDED
#define PLUGINEDITOR_H_INCLUDED

#include "../JuceLibraryCode/JuceHeader.h"
#include "PluginProcessor.h"


//==============================================================================
/**
*/
class IncrediblySimpleFmsynthAudioProcessorEditor  : public AudioProcessorEditor,
                                                     public Slider::Listener,
                                                     private Timer
{
public:
    IncrediblySimpleFmsynthAudioProcessorEditor (IncrediblySimpleFmsynthAudioProcessor&);
    ~IncrediblySimpleFmsynthAudioProcessorEditor();

    //==============================================================================
    void                     paint (Graphics&) override;
    void                     resized() override;
    void                     sliderValueChanged (Slider*) override;
private:
    void                     timerCallback() override;
    AudioProcessorParameter* getParameter (const String& paramId);
    float                    getParameterValue (const String& paramId);
    void                     setParameterValue (const String& paramId, float value);
    
    Slider                   modulationIndex;
    Slider                   modulationFrequencyRatio;
    
    IncrediblySimpleFmsynthAudioProcessor& processor;

    JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (IncrediblySimpleFmsynthAudioProcessorEditor)
};


#endif  // PLUGINEDITOR_H_INCLUDED

PluginEditor.cpp

/*
  ==============================================================================

    This file was auto-generated!

    It contains the basic framework code for a JUCE plugin editor.

  ==============================================================================
*/

#include "PluginProcessor.h"
#include "PluginEditor.h"


//==============================================================================
IncrediblySimpleFmsynthAudioProcessorEditor::IncrediblySimpleFmsynthAudioProcessorEditor (IncrediblySimpleFmsynthAudioProcessor& p)
    : AudioProcessorEditor (&p), processor (p)
{
    // Make sure that before the constructor has finished, you've set the
    // editor's size to whatever you need it to be.
    setSize (300, 300);
    
    modulationIndex.setName("ModIndex");
    modulationIndex.addListener(this);
    modulationIndex.setSliderStyle (Slider::LinearBarVertical);
    modulationIndex.setRange(0.0, 1500.0, 0.0);
    modulationIndex.setTextBoxStyle(Slider::NoTextBox, false, 90, 0);
    modulationIndex.setPopupDisplayEnabled(true, this);
    modulationIndex.setTextValueSuffix(" % Gnarly");
    modulationIndex.setValue(0.0);
    
    addAndMakeVisible(modulationIndex);
    
    modulationFrequencyRatio.setName("ModFreqRatio");
    modulationFrequencyRatio.addListener(this);
    modulationFrequencyRatio.setSliderStyle (Slider::LinearBarVertical);
    modulationFrequencyRatio.setRange(-3.8, 3.8, 0);
    modulationFrequencyRatio.setTextBoxStyle(Slider::NoTextBox, false, 90, 0);
    modulationFrequencyRatio.setPopupDisplayEnabled(true, this);
    modulationFrequencyRatio.setTextValueSuffix(" Times The Carrier's Frequency");
    modulationFrequencyRatio.setValue(0.0);
    
    addAndMakeVisible(modulationFrequencyRatio);
    
    startTimer(100);
}

IncrediblySimpleFmsynthAudioProcessorEditor::~IncrediblySimpleFmsynthAudioProcessorEditor()
{
}

//==============================================================================
void IncrediblySimpleFmsynthAudioProcessorEditor::paint (Graphics& g)
{
    g.fillAll (Colours::white);

    g.setColour (Colours::black);
    g.setFont (15.0f);
    g.drawFittedText ("Simple FM Synth", 0, 0, getWidth(), 30, Justification::centred, 1);
    
    
}

void IncrediblySimpleFmsynthAudioProcessorEditor::resized()
{
    modulationIndex.setBounds(40, 30, 20, getHeight() - 60);
    
    modulationFrequencyRatio.setBounds(getWidth() - 40 - modulationFrequencyRatio.getWidth(), 30, 20, getHeight() - 60);
}

void IncrediblySimpleFmsynthAudioProcessorEditor::sliderValueChanged (Slider* slider)
{
    if (slider->getName() == "ModIndex")
    {
        setParameterValue("ModIndex", modulationIndex.getValue());
    }
    else if (slider->getName() == "ModFreqRatio")
    {
        setParameterValue("ModFreqRatio", modulationFrequencyRatio.getValue());
    }
}

void IncrediblySimpleFmsynthAudioProcessorEditor::timerCallback()
{
//    modulationFrequencyRatio.setValue(getParameterValue("ModFreqRatio"),
//                                      NotificationType::dontSendNotification);
//    modulationIndex.setValue(getParameterValue("ModIndex"),
//                             NotificationType::dontSendNotification);
}

AudioProcessorParameter* IncrediblySimpleFmsynthAudioProcessorEditor::getParameter (const String& paramId)
{
    if (AudioProcessor* processor = getAudioProcessor())
    {
        const OwnedArray<AudioProcessorParameter>& params = processor->getParameters();
        
        for (int i = 0; i < params.size(); ++i)
        {
            if (AudioProcessorParameterWithID* param = dynamic_cast<AudioProcessorParameterWithID*> (params[i]))
            {
                if (param->paramID == paramId)
                    return param;
            }
        }
    }
    
    return nullptr;
}

float IncrediblySimpleFmsynthAudioProcessorEditor::getParameterValue (const String& paramId)
{
    if (AudioProcessorParameter* param = getParameter (paramId))
        return param->getValue();
    
    return 0.0f;
}

void IncrediblySimpleFmsynthAudioProcessorEditor::setParameterValue (const String& paramId, float value)
{
    if (AudioProcessorParameter* param = getParameter (paramId))
        param->setValueNotifyingHost (value);
}

PluginProcessor.h

/*
  ==================================================================================

    This file was auto-generated!

    It contains the basic framework code for a JUCE plugin processor.

  ==================================================================================
*/


#ifndef PLUGINPROCESSOR_H_INCLUDED
#define PLUGINPROCESSOR_H_INCLUDED

#include <memory>
#include "maximilian.h"
#include "../JuceLibraryCode/JuceHeader.h"

//==================================================================================
/**
 */
struct SimpleSound : public SynthesiserSound
{
    bool appliesToNote(int) override { return true; }
    bool appliesToChannel(int) override { return true; }
};

//==================================================================================
/**
 */
class SimpleVoice : public SynthesiserVoice
{
public:
    SimpleVoice() : modulationFrequencyRatio(0),
                    modulationIndex(0),
                    frequency(1),
                    level(0)
    {}
    
    void setParameters(double modRatio,
                       double modIndex)
    {
        modulationFrequencyRatio = modRatio;
        modulationIndex          = modIndex;
    }
    
    bool canPlaySound(SynthesiserSound* sound) override
    {
        return dynamic_cast<SimpleSound*>(sound) != nullptr;
    }
    
    void startNote(int midiNoteNumber,
                   float velocity,
                   SynthesiserSound*,
                   int) override
    {
        frequency = MidiMessage::getMidiNoteInHertz(midiNoteNumber);
        level = velocity;
    }
    
    void stopNote(float, bool) override
    {
        level = 0;
        clearCurrentNote();
    }
    
    void renderNextBlock(AudioSampleBuffer& outputBuffer,
                         int                startSample,
                         int                numberSamples) override
    {
        while (--numberSamples >= 0)
        {
            const double modulatorFrequency = frequency + frequency * modulationFrequencyRatio;
            const double modulationLevel    = modulationIndex;
            const double modulationSample   = modulationLevel * modulator.sinewave(modulatorFrequency);
            
            const double carrierFrequency   = frequency + modulationSample;
            
            const float  currentSample      = level * float(carrier.sinewave(carrierFrequency));
            
            for (int i = outputBuffer.getNumChannels(); --i >= 0;)
                outputBuffer.addSample(i, startSample, currentSample);
                
            ++startSample;
        }
    }
    
    void pitchWheelMoved(int) override {}
    void controllerMoved(int, int) override {}
    
    
private:
    maxiOsc          carrier;
    maxiOsc          modulator;
    double           modulationFrequencyRatio;
    double           modulationIndex;
    double           frequency;
    double           level;
};


//==================================================================================
/**
 */
class IncrediblySimpleFmsynthAudioProcessor  : public AudioProcessor
{
public:
    //==============================================================================
    IncrediblySimpleFmsynthAudioProcessor();

    //==============================================================================
    void prepareToPlay (double sampleRate, int samplesPerBlock) override;
    void releaseResources() override;

   #ifndef JucePlugin_PreferredChannelConfigurations
    bool setPreferredBusArrangement (bool isInput, int bus, const AudioChannelSet& preferredSet) override;
   #endif

    void processBlock (AudioSampleBuffer&, MidiBuffer&) override;

    //==============================================================================
    AudioProcessorEditor* createEditor() override;
    bool hasEditor() const override;

    //==============================================================================
    const String getName() const override;

    bool acceptsMidi() const override;
    bool producesMidi() const override;
    double getTailLengthSeconds() const override;

    //==============================================================================
    int getNumPrograms() override;
    int getCurrentProgram() override;
    void setCurrentProgram (int index) override;
    const String getProgramName (int index) override;
    void changeProgramName (int index, const String& newName) override;

    //==============================================================================
    void getStateInformation (MemoryBlock& destData) override;
    void setStateInformation (const void* data, int sizeInBytes) override;

    static constexpr int maxNumVoices = 1;
    
private:
    SimpleVoice*         voice;
    Synthesiser          synth;
    AudioParameterInt*   modulationIndexParamter;
    AudioParameterFloat* modulationFrequencyRatioParameter;
    double               lastSampleRate;

    //==============================================================================
    JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (IncrediblySimpleFmsynthAudioProcessor)
};


#endif  // PLUGINPROCESSOR_H_INCLUDED

PluginProcesser.cpp

/*
  ==============================================================================

    This file was auto-generated!

    It contains the basic framework code for a JUCE plugin processor.

  ==============================================================================
*/

#include "PluginProcessor.h"
#include "PluginEditor.h"


//==============================================================================
IncrediblySimpleFmsynthAudioProcessor::IncrediblySimpleFmsynthAudioProcessor()
{
    addParameter(modulationIndexParamter = new AudioParameterInt("ModIndex",
                                                                 "Modulation Index",
                                                                 0,
                                                                 1500,
                                                                 0));
    addParameter(modulationFrequencyRatioParameter = new AudioParameterFloat("ModFreqRatio",
                                                                             "Modulation Frequency Ratio",
                                                                             -3.8,
                                                                             3.8,
                                                                             0));
    synth.clearVoices();
    for (int i = 0; i < maxNumVoices; ++i)
        synth.addVoice(new SimpleVoice());
    
    synth.clearSounds();
    synth.addSound(new SimpleSound());
}

//==============================================================================
const String IncrediblySimpleFmsynthAudioProcessor::getName() const
{
    return JucePlugin_Name;
}

bool IncrediblySimpleFmsynthAudioProcessor::acceptsMidi() const
{
   #if JucePlugin_WantsMidiInput
    return true;
   #else
    return false;
   #endif
}

bool IncrediblySimpleFmsynthAudioProcessor::producesMidi() const
{
   #if JucePlugin_ProducesMidiOutput
    return true;
   #else
    return false;
   #endif
}

double IncrediblySimpleFmsynthAudioProcessor::getTailLengthSeconds() const
{
    return 0.0;
}

int IncrediblySimpleFmsynthAudioProcessor::getNumPrograms()
{
    return 1;   // NB: some hosts don't cope very well if you tell them there are 0 programs,
                // so this should be at least 1, even if you're not really implementing programs.
}

int IncrediblySimpleFmsynthAudioProcessor::getCurrentProgram()
{
    return 0;
}

void IncrediblySimpleFmsynthAudioProcessor::setCurrentProgram (int index)
{
}

const String IncrediblySimpleFmsynthAudioProcessor::getProgramName (int index)
{
    return String();
}

void IncrediblySimpleFmsynthAudioProcessor::changeProgramName (int           index,
                                                               const String& newName)
{
}

//==============================================================================
void IncrediblySimpleFmsynthAudioProcessor::prepareToPlay (double sampleRate,
                                                           int    samplesPerBlock)
{
    ignoreUnused(samplesPerBlock);
    lastSampleRate = sampleRate;
    synth.setCurrentPlaybackSampleRate(lastSampleRate);
    maxiSettings::setup(lastSampleRate, getTotalNumOutputChannels(), samplesPerBlock);
}

void IncrediblySimpleFmsynthAudioProcessor::releaseResources()
{
    // When playback stops, you can use this as an opportunity to free up any
    // spare memory, etc.
}

#ifndef JucePlugin_PreferredChannelConfigurations
bool IncrediblySimpleFmsynthAudioProcessor::setPreferredBusArrangement (bool                   isInput,
                                                                        int                    bus,
                                                                        const AudioChannelSet& preferredSet)
{
    // Reject any bus arrangements that are not compatible with your plugin

    const int numChannels = preferredSet.size();

   #if JucePlugin_IsMidiEffect
    if (numChannels != 0)
        return false;
   #elif JucePlugin_IsSynth
    if (isInput || (numChannels != 1 && numChannels != 2))
        return false;
   #else
    if (numChannels != 1 && numChannels != 2)
        return false;

    if (! AudioProcessor::setPreferredBusArrangement (! isInput, bus, preferredSet))
        return false;
   #endif

    return AudioProcessor::setPreferredBusArrangement (isInput, bus, preferredSet);
}
#endif

void IncrediblySimpleFmsynthAudioProcessor::processBlock (AudioSampleBuffer& buffer,
                                                          MidiBuffer&        midiMessages)
{
    for (int i = synth.getNumVoices(); --i >= 0;)
    {
        if ((voice = dynamic_cast<SimpleVoice*>(synth.getVoice(i))))
        {
            voice->setParameters(double(modulationFrequencyRatioParameter->get()),
                                 double(modulationIndexParamter->get()));
        }
    }
    
    buffer.clear();
    
    synth.renderNextBlock(buffer,
                          midiMessages,
                          0,
                          buffer.getNumSamples());
    
    
}

//==============================================================================
bool IncrediblySimpleFmsynthAudioProcessor::hasEditor() const
{
    return true; // (change this to false if you choose to not supply an editor)
}

AudioProcessorEditor* IncrediblySimpleFmsynthAudioProcessor::createEditor()
{
    return new IncrediblySimpleFmsynthAudioProcessorEditor (*this);
}

//==============================================================================
void IncrediblySimpleFmsynthAudioProcessor::getStateInformation (MemoryBlock& destData)
{
    // You should use this method to store your parameters in the memory block.
    // You could do that either as raw data, or use the XML or ValueTree classes
    // as intermediaries to make it easy to save and load complex data.
    MemoryOutputStream stream(destData, true);
    
    stream.writeFloat(*modulationIndexParamter);
    stream.writeFloat(*modulationFrequencyRatioParameter);
}

void IncrediblySimpleFmsynthAudioProcessor::setStateInformation (const void* data, int sizeInBytes)
{
    MemoryInputStream stream (data, static_cast<size_t> (sizeInBytes), false);
    
    modulationIndexParamter->setValueNotifyingHost(stream.readFloat());
    modulationFrequencyRatioParameter->setValueNotifyingHost(stream.readFloat());
}

//==============================================================================
// This creates new instances of the plugin..
AudioProcessor* JUCE_CALLTYPE createPluginFilter()
{
    return new IncrediblySimpleFmsynthAudioProcessor();
}


#13

exciting project!
Do you already know how you will measure the sounds “similarity”?


#14

Very interesting project, I had the same idea but never had the impulse to realize it. And by the way genetic algorithms were what I supposed to be the best suited way of doing it at the time (when deep learning was not yet sort of mainstream).


#15

Thanks! Similarity of sounds can be measured in a variety of ways, namely taking the Euclidean distance from a given mathematical description of the sampled sound. So really it is down to choosing the correct description (or feature) or feature vector, which is a collection of features.

Deep Neural Networks can also build their own features during training. It might be that no features are needed to be input, we can just plug in the raw PCM data from the synth’s output, and that it will build it’s own features… But that is for another time. I’m starting simple and building up, so currently I am using FFT’s to create MFCC’s. It is likely I’ll need to more descriptions to analyse the sound, of which my ear will play a large role in by listening to the sounds outputted, so perhaps spectral flux and some sort of noise measurer.


#16

Yeah that’s what has been the traditional way of doing this. Infact, MLP networks were outclassed by GA’s in this thesis (it’s a really good one - see 3.3). I’m hoping a deeper and wider net, perhaps a recurrent one, not sure yet, should be able to outperform GA’s. We’ll see.


#17

Hi all, I’ve been working pretty hard on the project and have some updates. The links are towards the end of each bulletin, but for some reason aren’t formatted nicely - hover your mouse over the end of each sentence to get to the code!

  1. I have written a simple FM synth. You can find that here.
  2. I used an earlier iteration of RenderMan to generate loads of data for the simple synth, and learnt pretty well how to predict the correct parameters using TensorFlow. See the code and read the method here.
  3. I have rewritten RenderMan to be a class and have added Python bindings; RenderMan is now a Python library - if you are on Linux I have edited the makefile to produce a shared library loadable as a module. You can get the code here.
  4. I have demonstrated the new Python RenderMan library here.

Currently, the library is built with boost python bindings using a modified makefile. However, we all know the projucer overwrites the makefile each time you save changes to it. Is there a way of building a shared object using the projucer? I tried and from (my incredibly novice point of view) it isn’t configurable enough to tell the Projucer to produce .so files.

Another issue is sysex files. I know they are synth specific, but I would really like to be able to write an interpreter for the DX7 / Dexed synth that takes the array of uint8 values and outputs sets of parameter values at a given index. Anyone know of any projects I could take a look at as a starting point?


#18

Have you tried selecting the Dynamic Library project type from the starting page of the Projucer?


#19

Hi Tom thanks for dropping in. It produces .a binaries correct? I think I need to make .so files for this to work. However I have a feeling you are about to explain why I am really stupd…


#20

.a libraries are static, .so libraries are dynamic.

The Dynamic Library project type will produce .so library.