Need advice on a "sample library framework" project

GOAL: Make a cross-platform realtime gui and audio programming environment/framework to improve the ease of producing and distributing sample library products.

My project/plan is to create a framework for authoring digital audio instruments, more specifically sample libraries. The framework will allow you to visually and textually assemble a sample library product, and allow you to script the audio engine like Native Instruments Kontakt, except that the scripting platform will also be the same engine that runs the VST (or that seems to be the plan at the moment). In other words, the sample library developer could program the sample playback engine from scratch using guides that the framework offers, or use the framework to assist in accomplishing all of the most common needs of a sample library product, generate the code, and BOOM! A working VST sample library (btw since this is an audio engine scripting, unlike Kontakt/KSP, this could also be used to create full-fledged synthesizers rather than a sampler, or a hybrid sampler/synth).

Another VERY important addition is that the framework will also establish standards that all developers will be able to easily follow so that one developer's product can run in the same window nicely along side another developer's product, a la Kontakt rack view. This is possible with protoplug (see more info below). 

The framework will consist of 4 parts:

1. Assembling the audio files (samples) into groupings, defining "where" they sit in a grid of midi velocity/midi note (MAPPING ENGINE). The samples can even be left "floating" (so the sample/voice triggers only based on events).

2. Assembling the user interface with image files, defining types, positions, functions, etc. (GUI BUILDER)

3. Generating code from steps 1 and 2

4. Interpreting the code (protoplug? projcuer?) and adding additional scripting for specific usage of audio and gui. (NOTE: If using projucer, then obviously you'd have to actually assemble it into a VST whereas protoplug is already a VST, but the point is they are both real time coding languages, both visual, but projucer may have some advantages)

I'm putting together a multi-page PDF that I will be distributing to interested parties. It will also discuss sources of funding. So far I have a flow chart of how all of this will work.


 (click to enlarge)


My first idea was to generate code for Protoplug, which is a LuaJIT VST with JUCE bindings, check it out, it's very awesome:

Then I watched a video abou Projucer and that seemed really awesome. Now I'm wondering if I should remove protoplug from this project and somehow leverage Projucer instead. However, I am not an experienced programmer and I don't know if that's an idea that makes sense.

I explained this as best as I could, sorry if I sound very lost! Just kinda looking for advice, maybe the terms I'm using are incorrect, maybe my flow chart doesn't actually make sense with how things work. I'm half-confused in all of this, but I know my vision. I can answer questions to clarify things. Thanks for your time!

[edited for further clarification]

Well, good luck with it! It's very ambitious, and lots of other people have tried this kind of thing before.

What's the unique selling point of your idea, compared with other sample engines?

If Lua scripting or even C++ (of course abstracted to be as easy as possible to use) can be used, it would maybe allow more complicated sample selection and handling to be done than with NI Kontakt, which only has a very curious and simplistic scripting language. However, it seems to me that despite the flaws in the Kontakt scripting language, it has already been doing quite fine for many years with people even making some living writing scripts in it. Maybe nothing more elaborate is even needed? (Note, I am not an expert or really even a basic user in this area, I don't use large ready made sample libraries for doing my music.)

But perhaps Elan can open up the topic further...

Alright, I'm back after having spoken to a handful of developers and scripters.

[color=red]Jules[/color], here are the main selling points / goals of the project:

for the Consumer

  • The plugin component is free; unrestricted access to free sample libraries.
  • From start to finish, create an expressive sample library without ever having to manipulate code, great for small projects and power users.

for the Developer

  • Live textual and visual coding environment for faster conception, testing, and debugging.
  • Unrestricted customer base for your product; customers will not have to purchase third party software or own a specific OS or DAW.
  • Signal flow, control scheme, playback engine, all connected specifically by the developer's intentions via low-level control for an efficient and stable product.

for the Community

  • Open source tools and formats are conducive to 3rd party development.
  • Sell and share sample library components (FX, generators, DSP) to other developers and users.

[color=red]Xenakios[/color], now to answer your question. 

[quote=Xenakios]Maybe nothing more elaborate is even needed?[/quote]It’s not that developers need something more elaborate, devs need something more usable, something built for "today". Devs need something where you don’t have to create a legato engine every time you want to make a new instrument. Devs need a framework where commonly used features are a given. Kontakt [i]has[/i] round robin handling, but is simply unusable/incompatible with the requirements of the project. Devs end up having to use hacky cluttered code, and confusing control schemes to just to enable round robin handling. When devs have to make their own versions of things that is included in the software, something is wrong. And yes, Xenakios, Kontakt has been doing fine for many years.

Developers have gotten used to the hassle. Kontakt is the only choice and it is hindering innovation. Case in point: I have been developing phaselocking technology that will allow seamless crossfades between samples in a sampler engine, but Kontakt is not able to take full advantage of it. I believe this technology could revolutionize sample libraries, but that’s never going to happen until the sampler evolves.

There was a lot of excitement when MachFive's flexible development environment came out, but the excitement quickly wore off once people realized it was too inefficient for large modern sample library products. [u]Something has to be done.[/u] If I am not the one to usher in a new paradigm in sample library creation, someone else will... I want to do it, I am motivated. I know the problems and solutions because I've dealt with every aspect of sample libraries from cutting to scripting to selling, and have worked with  many companies and have heard all the horror stories, witnessed and experienced frustration first hand.

After thinking about it more, I now know the question I should really be asking is this: Is it possible to create a framework and a set of tools that leverage Projucer for a live coding environment for sample libraries?.. so that a VST is created from the ground up for utmost efficiency and flexibility.

...but another goal is that the VST needs to be able to host these sample libraries/engines in a rack view like kontakt, so it may not be a VST that JUCE spits out, maybe something else? Or how about creating a VST host that is specific to these sample library VSTs? I don't know, I guess that's why I titled this thread with "need advice". 


Well, yes, it's possible to use the projucer to work on a C++ project that compiles a sample library into a plugin.. But I'm not sure whether that's actually a good idea?

Are you perhaps imagining that the projucer is some sort of C++ interpreter that could be embedded in a plugin? Because although that would indeed be a handy tool for a sample library format to use, it's not the intention of the project!

Hey Jules, no haha, please bear with me with while I am figuring this out in my head. It's starting to become clear. 

Tools/Framework/Third Party Software/Template Code/Code Generator? --> Projucer --> VST

Hmm, let me make an example.

Joe the sample library developer has a set of audio files and a set of image files. Joe wants to create a VST to playback his samples and use his images in an expressive way. What is poor old Joe to do? He could use Projucer, but wait! He doesn't know anything about C++ or midi data or VST programming! He needs... dun dun dadunnn! A framework!

[my attempt at a definition follows]: A framework is a set of applications and ways of doing things that make it easy to create a specific kind of product. Maybe in my case I would create a piece of software called "The Mapper" and it would allow Joe to graphically place his audio files in 2 dimensions (note/velocity). That would then spit out classes/objects/variables for a template JUCE/projucer project for making a sample based VST. If Joe didn't want to mess with any code, he could immediate just hit the compile button in Projucer and get a final VST product. 

edit: BUT! If Joe wanted to, he could easily work in Projucer to edit a few things, audition the changes live [which would be amazing] before going to VST.

Hi Elan,

we skyped about the phaselocking stuff a year ago.

I spent the last two years writing such a framework (which covers every use case you mentioned so far).

Basically it is a collection of C++ modules  (sample players, effects, envelopes) and scripting processors for MIDI logic and interface design.

I would consider it in a pre beta state (it works pretty stable on my machine and I can build semi weight librarys with similar performance as KONTAKT), but for the last steps (beta testing, use case development, documentation, support, etc) I need serious manpower to make it competitive against KONTAKT.

It will be open source, and built on top of JUCE, so if you need a special synth to enhance your samples, you simply code a new module which can use other modules (so you don't need to write your own sine LFO)

BTW I would refrain from using Protoplug as foundation because it relies on LuaJIT which has a memory alignment issue with OSX 64bit - and as far as I know there is no or little intention from the head developer to fix this. And a sample library framework without 64bit support is not a sample library framework.

Also if you discard Mach Five because of its performance I would also be sceptical about LuaJIT's speed - I know there are benchmarks which are as fast as C, but heavy branching as well as exact control over memory allocation are reasons to hardcode the core modules in C++.

But I also played around with protoplug - I even embedded it in my framework as development tool and it seems like a nice little piece of software.

So I would suggest before you dive in an roll your own framework, lets have a chat about cooperating on this subject. I also strongly think that its time for a new player in the sample engine market.


Lua works fine for the script part, but for the DSP, I would'nt even think about it.

Better use some LLVM JIT


Note: I am one of the dev of MachFive :)

LuaJIT which has a memory alignment issue with OSX 64bit - and as far as I know there is no or little intention from the head developer to fix this. And a sample library framework without 64bit support is not a sample library framework.

Is that an issue on OS-X 64 bit that prevents anything from working at all or is it an issue that makes things very slow? Either way, that would be bad and I need to immediately reconsider whether Protoplug is something I should be spending any more time with... frown

[quote=pierrec @ protoplug forum]an OSX 64-bit version of protoplug will come when the corresponding issue in LuaJIT is fixed, or if someone comes up with a hack to make it work around that. Hopefully the issue has better chances of getting fixed now that LuaJIT is back on track for serious development.[/quote] link:


I haven't tried it out before - and if I understand the second link correctly it could only appear if multiple libraries are loaded and the bigger memory adresses are used, so it might not be easy to reproduce it.


oops wrong spot to reply

Just want to thank everyone for the replies. I now have a handful of ideas about how to approach this.