JUCE LV2 Plugin Wrapper

Yes, but not very much in the long term.
I understand that you may not want 2 new functions just for LV2, but afaik the ‘transition’ is very easy and fast. All the plugins code I saw so far used juce XML to save/load data, so it gets really easy to support the new calls.

I spoke to the lv2 developers about this, they will really appreciate if strings could be used instead of blobs. Main reason for this, quote: “Short answer, blobs are not portable”

Piffle! The reverse is true.

If you’re concerned about your stored data being passed safely between different platforms or hosts, then you’d be much wiser to pass it around as a binary blob. The “black box” nature of binary data means that there’s no danger of the host buggering things up by reformatting it, trimming whitespace off it, changing the newline characters, screwing up the character encoding, etc etc! As soon as you put the data into a format that the host might be able to tinker with, then you’re just asking for trouble!

LV2 developer here.

Um, that is obviously not true because you can put non-portable data in a blob. This isn’t much of a stretch, since even an int is non-portable.

To be able to save portable state, the plugin API must know whether or not data is portable. Without any other information attached to it (like a type), an opaque binary blob is not portable data. An XML string is portable data. You can’t assume a blob is portable when it might not be, since this will fail catastrophically.

If you state is already XML text, the obvious ideal solution is to simply save it as XML text. This is elegant, transparent, convenient, and portable.

Base64, while supported, doesn’t actually help, because it doesn’t make non-portable data portable, it just encodes it differently (and with overhead). The real world impact of this is that the plugins would not save portable state, which means hosts are actually pretty likely to just not save it at all in session files: trying to encode architecture is difficult and not really worth the effort, it is assumed plugins can save a portable version of their state. The option to save non-portable state (e.g. blobs) is mainly there for fast in-process snapshots and the like.

I am not at all familiar with JUCE and what guarantees are baked in to the API, but in general it all boils down to a simple requirement: the plugin must be able to tell the API with absolute certainty whether or not the state is portable. If you already have portable data in a nice standard textual format, converting it into binary to save on disk is insane.

Hello!

I think we might be using the word “portable” to mean different things. Perhaps you can define “portable”, and give an example of exactly what is being ported, and what the source source and destination are between which this porting is being performed…?

When I say “portable”, all I mean is that if I write a plugin that runs as a VST on OSX and also as an RTAS on Windows and a LV2 on linux, I want to write just one portable piece of code that stores my data.

And you must admit that it’s just silly to say “XML data is more portable than binary data” when you’re talking about an opaque piece of content! The plugin gives the host a lump of data that only the plugin itself can understand, so as far as the host is concerned, it doesn’t make the slightest bit of difference whether this data is encoded as binary, string, XML, or piles of sausages. The host’s job is not to manipulate the data’s content or magically make it “portable” (whatever that means). The host just is simply responsible for returning the same lump of data when the plugin asks for it.

In fact, most JUCE plugins actually do store their state using XML. The wrapper classes provide utility functions that convert the XML to the cross-platform binary format which gets passed to the host. But many plugins NEED to use binary because they’re storing enormous data-sets (sample banks, etc), so binary is the only practical solution for the base storage format.

Portable with respect to hardware architecture. The goal with saving state is to save the state in a way that a host can reload it later. The problems come from the fact that the host could be running on a different platform when it is restored.

For example, say I save state on an i386, throw that session on a USB key and restore it on a different x64 machine. If the state is just a binary blob, which is not portable (it may contain integers and pointers, for example) this is not only impossible to do correctly, it will fail catastrophically. It’s not even possible to know you can’t safely restore that state! The best you can do is just try and hope it doesn’t get you corrupt data, or crash the entire process.

It’s not a coincidence that most JUCE plugins (and VST in general) actually do save to a textual format like XML, because this is by far the easiest way of saving portable state. Doing endian conversion and making sure all your structs and whatnot are portable with respect to alignment and sizes and all the rest is a heck of a lot harder than just saving to portable formats (e.g. text-based stuff like XML). As you say, nobody wants to write a bunch of code to handle all those different situations, they just want to write one implementation that works everywhere. Text is the obvious solution, which is why pretty much everyone does it.

That said, sure, saving binary data is necessary some times - but it is absolutely crucial that the host knows which is which. It is not a matter of the host magically making anything portable, it’s a matter of the host even knowing the data is portable at all. To be able to restore data, the host and/or plugin needs to know what that data is.

It is not at all silly to say XML is more portable than an abitrary binary blob. What an odd thing to say! XML (unless you encode non-portable binary data in it, anyway) is a completely portable text-based format that has nothing to do with architecture, you can save it on i386 and restore it on ARM or PPC or x64 or whatever. This is not true of an arbitrary binary blob. It seems you aren’t actually considering the reality that multiple architectures exist - the integer 4 is not the same “lump of data” on different computers. If only life was so simple…

An API that saves state only to an opaque binary blob with no type/portability information associated with it is flawed because the saved data may be tied to the current architecture, which effectively means the entire session is tied to that architecture and can’t be safely distributed or loaded elsewhere. LV2 does not duplicate this flaw, but supports portable state properly, so hosts can at the very least by know if it is safe to save/restore state. To support this, the plugin (and thus JUCE) must tell the host what type the data it is saving is (or at least whether or not it is portable). This is not really LV2 specific, it is inherent to providing portable state (it is why Apple uses property lists, which have typed values, for many things including AU state. Apple actually had to care about portability. VST is just plain broken in this respect.).

So it’s literally IMPOSSIBLE to pass any binary data from one architecture to another? Wow! Good job you warned us.

Hopefully you’re viewing this website on a CPU that has the same endianness as the one that created its (binary) image files, otherwise your browser may fail catastrophically at any moment!

Of course it’s possible - sometimes. However, if you don’t know whether or not the data is indeed portable, then the only safe assumption is that it is not. Therefore, via such an interface, it is impossible. That is the problem.

My browser can read those files because it knows what format they are in, which is why the relevant protocols convey that information, exactly how LV2 state works, and exactly what the JUCE API needs. It is pretty telling that your own example illustrates this: the web could not possibly work if it was all binary blobs with no type information whatsoever. If not a type, then at least a flag to indicate whether or not the data is portable is necessary. Without that, yes, it is impossible.

P.S. Binary isn’t necessarily non-portable, those images work because their format is well-defined, like wav files, and any implementation will load them on any platform (one of the many advantages of using standard formats over meaningless binary gunk). Binary vs. text is not the issue here, the issue here is knowing the type of data, or at the very least knowing if that data is portable.

You’re right that it’s impossible to guarantee that a plugin will behave safely. But it’s not your job to try to rectify that situation.

A plugin shouldn’t crash. End of story. If a plugin emits some data and then crashes when it re-loads it, then the plugin is breaking its contract, and that’s not your fault. I’m not sure why you seem to think this is a problem that you need to solve?

I don’t intend to solve it. Plugins are free to save whatever, even if it’s completely non-portable. That is certainly a stupid idea, but it’s not my job to enforce it. It is simply a fact that the host needs to at least know whether or not the data is portable for things to work correctly. You can do this via rules (like “the data must always be portable”, but you don’t seem to agree with that either), or do it via allowing the plugin to specify which. You can’t allow arbitrary non-portable binary data and not even have a mechanism to say so. How is a host to know the state isn’t suitable for distribution (e.g. as part of a preset or example session) or loading on a different machine? It makes all plugins pay for the disadvantages of a few that make the dubious decision of storing non-portable state, and in this case it does so when the actual state is usually XML text, which instead of saving as plain readable text gets bloated out to 3/2 the size in base64 encoding (I pity anyone who has to debug such a thing). Plugins are using sane portable state (XML) with all the desired qualities, then it’s being crammed through a broken API so that as far as the host knows it’s a bunch of non-portable crap, and we’re all just supposed to trust that the plugin is actually saving the current architecture along with this data (highly doubtful and extremely difficult to do precisely in general) and dealing with all cross-platform issues itself (unlikely). In a perfect world that might be wise, but not in this one.

I am a bit shocked I have to argue this one, to be honest. Why would you choose something that requires such lofty unrealistic things of plugins (all that binary portability stuff), when a simple superior alternative provides all the benefits while requiring almost nothing of plugins (specify a type or a flag) and allows the host to avoid any potential problems whatsoever? That is not a sensible decision. What is the advantage? What the heck could the disadvantage be of allowing the host to avoid problems by not restoring non-portable state on the wrong platform? What could the disadvantage possibly be of specifying a type (or some flags) along with data? It doesn’t seem like I’m the one that needs to be doing the justifying here…

You can allow this. Because every VST/RTAS/AU host that has ever been created has allowed this. And that’s all been working just fine for over a decade without your help, thanks very much!

Remember PPC/intel? Remember all the agony when people tried to load a Cubase Mac project on an intel machine and all the plugins crashed? No? Me neither, because that was never a big problem.

There’s no reason for the host NOT to assume that state data is safe to load on another machine. Preferably the API would stipulate this requirement explicitly, but it’s a bit of a no-brainer when you’re writing a loadState() function that the damn thing shouldn’t crash, for f**k’s sake!

If you were writing an OS, would you stop users from transferring binary files between different machines, because one or two badly-written apps might have been written without taking endianness into consideration?

Lofty and unrealistic?? Seriously? That’s pretty patronising. People aren’t complete idiots, you know.

If you KNOW for sure that this state is non-portable, then that would be fine.

But the only way to KNOW whether a plugin can safely load something is to force it to make it implement a canStateBeSafelyLoaded() function. And if you’re going to make the plugin do that, why not just tell them to do this check internally in their loadState() function so that it doesn’t crash? Because that’s what 99.9% of programmers have already been doing for years!!

Look, I’ve no time to argue any more on this thread, but the bottom line is this: It doesn’t matter what you or I think. The reality is:

  • Some functionality is impossible without a binary storage format. End of discussion. I’ve seen many VSTs which store 10s of megabytes of compressed data. If LV2 can’t do binary, then some plugins simply won’t be able to run on the LV2 format.
  • Nobody in the real world gives a rat’s-ass what LV2 does anyway. Even if you come up with the world’s best, safest, mollycoddling API that gives seamless magical cross-architecture data, few people will care or even notice, because everyone has already written and tested all their storage functions to use binary blobs on VST or AU, or are using JUCE’s XML format wrappers. There might a dozen people who only write LV2 plugins, and who don’t port them to other platforms, so that’s your audience.
  • If LV2 (or any other plugin platform) provides a string storage interface, then of course JUCE’s XML wrapper should use it directly, rather than unnecessarily transcoding XML->binary->string. That’s trivial to do, and not in question. But I doubt if any of the users would notice.

We’ve got quite a discussion here… :twisted:

I think that is the key here. If we can access the XML strings directly, it seems a waste of time and resources converting to binary to store, just to have to restore it later again.
I’ll still be using direct XML usage for my plugins and all the ports I do, but for an official Juce LV2 wrapper, it’s up to you.
(In case you’re wondering - yes, LV2 supports saving/restoring data as binary)

Ah! Well in that case, what have I been ranting about!? We can use both. Problem solved!

After reading such a lengthy discussion, I felt adding a touch of silly humour was needed:

:lol:

(I am going to ignore the… shall we say less helpful… part of this discussion like I should have in the first place. Good grief.)

Indeed. Which is why the API makes it so you know…

This is what I have been trying to get across this entire time. I am amazed, but glad, that it is apparently not in question. Why, then, we have been arguing about it with increasing vitriol making us both look like total dickheads is well beyond me, but whatever :stuck_out_tongue:

Anyway, wonderful. If it’s not in question that the interface should be used correctly I can assume JUCE will be fixed and get on with less tedious things than this ridiculous thread.

Ok, things are advancing and only 1 feature is missing to complete the wrapper (TimePos information)

Let me point some changes needed to make in Juce source code.

1 - Accept LV2 as a build type. juce_CheckSettingMacros.h should have:

#if ! (JucePlugin_Build_VST || JucePlugin_Build_AU || JucePlugin_Build_RTAS || JucePlugin_Build_Standalone || JucePlugin_Build_LV2) #error "You need to enable at least one plugin format!" #endif

2 - To build LV2, JucePlugin_LV2URI macro must be defined. should be checked on juce_CheckSettingMacros.h too, like so:

#ifndef JucePlugin_LV2URI #error "You need to define the JucePlugin_LV2URI value!" #endif
LV2 uses URI as id system (kinda like VST uses UniqueId). URIs must be unique. By default the value can be “urn:juce:PluginName” (Plugin name with no spaces or special chars).
The auto-generated AppConfig.h should have something like:

(Note that a real URI, like http://www.rawmaterialsoftware.com/DemoPlugin, is of course good too)

3 - Defining LV2 category using JucePlugin_LV2Category macro.
This is not required, when not set the plugin will be a normal effect.
All Possible values currently are:

GeneratorPlugin InstrumentPlugin OscillatorPlugin UtilityPlugin ConverterPlugin AnalyserPlugin MixerPlugin SimulatorPlugin DelayPlugin ModulatorPlugin ReverbPlugin PhaserPlugin FlangerPlugin ChorusPlugin FilterPlugin LowpassPlugin BandpassPlugin HighpassPlugin CombPlugin AllpassPlugin EQPlugin ParaEQPlugin MultiEQPlugin SpatialPlugin SpectralPlugin PitchPlugin AmplifierPlugin DistortionPlugin WaveshaperPlugin DynamicsPlugin CompressorPlugin ExpanderPlugin LimiterPlugin GatePlugin
If a plugin is a synth, then it must have category “InstrumentPlugin”. example:

(If this gets into the IntroJucer, a combo-box will be nice for this)

4 - Feature Macros
I added some macros to the code so that non-needed features are disabled
They can be undefined or set to 0 to disable. Set to 1 to enable them,
The macros are:

JucePlugin_WantsLV2Presets - Wherever it makes sense to export presets or not
Note that on LV2 presets are stored on the host-side, not internally like in VST. So if a plugin has 1 single preset it won’t make sense to set this.

JucePlugin_WantsLV2State - Wherever the plugin saves data that are not parameters (files, nodes, etc)

JucePlugin_WantsLV2StateString - Save the state as strings (usually XML), using the new API I spoke about earlier in this thread
This macro requires JucePlugin_WantsLV2State

JucePlugin_WantsLV2TimePos - Wherever the plugin is interested on getting Time information or not
Note that some old hosts don’t support this, or may just export the BPM

JucePlugin_WantsLV2InstanceAccess - Wherever the GUI requires direct access to the plugin code
This is usually true for Juce plugins. It should be enabled if JucePlugin_WantsLV2State is enabled too.
(The only plugins that I saw that don’t need to enabled this are those who only store parameter values and have no “fancy” GUI stuff like a MIDI keyboard)

Wow, nice work ! I have a few questions:

How is the gui displayed by the host, is it still an “external ui” ?

Which hosts do support the lv2 extensions that your lv2 wrapper requires ?

[quote=“jpo”]Wow, nice work ! I have a few questions:

How is the gui displayed by the host, is it still an “external ui” ?

Which hosts do support the lv2 extensions that your lv2 wrapper requires ?[/quote]

Currently the wrapper contains an external-ui and a X11 UI.
The X11 UI is new, and it will require new lv2 libs (being prepared right now, shouldn’t take too long).
external-UI is widely supported, so most hosts will just use that.
(Note - although the wrapper is cross-platform, X11 UI is obviously linux only. So for now only Linux has UI embedding)

afaik, all lv2 hosts will load the effect plugins, but the same is not true for synths or to fully save/load state and presets.

so far in my testing I know that:

  • lv2rack will load synths but not properly save state
  • zynjacku will not work at all (doesn’t support URID)
  • qtractor works, but doesn’t handle presets
  • ardour2 works with FX, will not save state or presets
  • ardour3 (SVN) works as much as possible (presets is being worked on right now)
  • jalv (SVN) works fine, including presets

I’m currently working on my own host Carla and OpenOctave plugin support, so both should fully support juce-lv2 based plugins by the time they are released.
I haven’t tried FreeADSP or Traverso.

Special note - the code has been made to work with both new MIDI API (Atom ports) and “old” (Event ports).
For now the default will be using Event ports, but after some time (waiting for hosts to catch up), Atom will be used, as it is required to get time-pos.

Ok, I think I’m ready to share the code for this LV2 wrapper…
The big code is this cpp file:
http://distrho.git.sourceforge.net/git/gitweb.cgi?p=distrho/distrho;a=blob_plain;f=libs/juce-lv2/juce_LV2_Wrapper.cpp;hb=refs/heads/master

it has a few dependencies (lv2 headers, and an extra cpp file), so it’s probably better to download the whole repo and then check ./libs/juce-lv2

git clone git://distrho.git.sf.net/gitroot/distrho/distrho --depth 1

The LV2 turtle data is generated using an exported symbol ‘lv2_generate_ttl’, for which I created this very small app to do the job:
http://distrho.git.sourceforge.net/git/gitweb.cgi?p=distrho/distrho;a=blob_plain;f=libs/lv2-ttl-generator/lv2_ttl_generator.c;hb=HEAD

It builds fine on Linux, and probably on mac as well (haven’t tried in a while now though).
(Make sure the read the README file before building!)

Still a work in progress, but is already working pretty fine.
Waiting for opinions now.

1 Like

Nice ! I’ll have to try that.

Which host to you recommend, for testing ?