Oh does it? Ohhh… oh now then. I will experiment. :). Thanks for the insight. This would maybe explain why the main thread was doing the xml stuff.
EDIT: I concur. My bad. I had no idea that Reaper did that. I will tell you what is weird though. I swear I put a breakpoint in that method when testing further up there ^^^, and it never triggered. Which is bizarre. Thanks for the help though. I will update the title accordingly. :). Oh… it seems I can’t anymore. Anyway i am sure this counts as solved.
Oh and… No… it isn’t a Juce bug. hahahaha. The question is… did I really think it was? Or was I just getting the old desperations in my burnt out struggles to understand? Well… you live and learn. :D.
Wait so… see I was under the impression that the methods only got called when saving/loading. This changes things. So a couple of questions that maybe you could offer some advice about at some kind of macro level.
Should I only update attributes that have changed? Or should I just ditch xml entirely? Is there another way to write to the memory buffer that can just do it sequentially without any of the more expensive xml methods?
EDIT : I see that there is MemoryOutputStream for example… does anyone know of any resources for how to use this approach? Cheers.
Unless… it is just as simple as something like this:-
// In getStateInformation...
juce::MemoryOutputStream stream(destData, false);
// In setStateInformation...
juce::ValueTree tree = juce::ValueTree::readFromData(data, size_t(sizeInBytes));
apvts.state = tree;
And then all I have to do is create a value tree for custom data and do the same? Or is it more like… I have to create a parent tree, put the apvts into it, then my custom data block, and somehow separate them out again? Pretty much like I was doing with the XML stuff.
Okay all, to anyone reading this thread in the future who may also have similar issues, I have solved this problem. Here is what I did…
I tried stripping out the XML method calls and just used ValueTree’s, but it didn’t really improve the performance. So in the end I opted for sequentially reading / writing the data using the MemoryInputStream & MemoryOutputStream. This now reports only microseconds of time to call getStateInformation().
Average = 321 microsecs, minimum = 321 microsecs, maximum = 321 microsecs, total = 321 microsecs
Unfortunately… obviously I lose the robustness of using key / value pairs in a ValueTree, but in my case of having many parameters / banks, it really wasn’t a price that could be paid. In other plugins I have made I have not had nearly as many parameters and so this problem never presented itself to me.
So for example, this is a cutdown snippet of what I did:-
// in getStateInformation():-
juce::MemoryOutputStream stream(destData, false);
// etc for more parameters / settings
// in setStateInformation()
juce::MemoryInputStream stream(data, sizeInBytes, false);
advancedMode = stream.readBool();
selectedBankID = stream.readInt();
selectedPatchID = stream.readInt();
// etc for more parameters / settings
I was also able to chop out some parameters who’s values don’t need to be saved. Such as the toggling parameters that I would bind to midi controller buttons, to shave off a little bit more time.
I hope that this helps anyone else who should stumble upon it. Thanks for all the help to everyone who replied, and I have learnt quite a bit in the process about how some of the backend of Juce works which it seems I had some confusion over. So I’d say all is well that ends well :).
Well… what I did is write the plugins version number into the memory block, and then it can be retrieved out of there for comparison. If it is a different plugin version then it wont load the rest of the memory block in case of crashes / invalid data.
Then I have a way to export all the banks / patches into an XML file as a backup, which can be imported back in safely and of course… compared to getStateInformation() will only be ran once.
It seems to work well. :).
EDIT: But I will take a look at Protocol Buffers, thanks.
Sorry to “bump” this thread, but it seems I hacked around a problem here that can be easily solved within Reaper by selecting ‘save minimal undo states’ in the compatibility settings for the plugin. Which means… you can actually then still utilise XML and not have it lag whenever a parameter is moved. Just thought I’d drop an update on this in case anyone should stumble upon this thread. Lol. Smh. I wouldn’t want it to be misleading in making people think that Reaper has issues or anything which requires unsafe code for loading/saving plugin state. I cannot believe that it has taken me this long to realise that option exists.
I guess I now have to revert my plugin back to the safer method of doing things. Haha. Dear oh dear.
It was yes, but it isn’t now to my knowledge. How would you suggest improving it? We can either use XML or we can’t? By the way, that is how Reaper handles a lot of things because of an insanely large undo state system which is saved every time you move a parameter. What I wrote also applies to many other plugins too when running under Reaper.
For example… how would I save stuff as XML, if doing so will make it intrinsically inefficient? You say it is inefficient, but how so? If i cannot notify the host of a change, then obviously, it wont be notified. If I do notify the host then it calls the getStateInformation method. It was that method that was lagging it because of the XML, but to remove XML loses robustness. So what gives?
It isn’t a case of changing the default DAW settings though. It is a per plugin setting change that is remembered once set, so no other plugins would be affected. Still… I cannot say that I like the solution, but I cannot see any other way that I can write the code to save as an XML value tree. So the only other way to do it is with saving raw, but updating the plugin will break existing saved projects that way.
Being robust doesn’t mean using XML. Other formats, including custom binary ones, are completely valid.
And you are repeating loads of work for each change as well.
The trick to being efficient is to design a system which does as little work as possible. For example (I don’t know if this will work for you because it’s a long time since I looked at your code) you can cache the whole structure and only save it if there are any changes. Or only update the things that changed.
I’d definitely avoid lots of string processing if you have a lot of data to write though … just imagine that CPU doing all that work, it makes me squirm
This isn’t directly about your problem, but maybe sheds some light on how to think about performance:
I see that there is a lot of work being done for each change, and of course XML isn’t the only way to be robust. Am I right in believing that the XML approach is really only a particularly useful for plugins with a limited amount of parameters for the sake of convenience and is not designed for plugins that save large amount of parameters?
and what you are suggesting is a case of only replacing pieces of the saved memory block that relate to the parameter being changed? I will take a look at the video.
EDIT: I assume you are not talking about cache memory in particular when you say cache the whole structure? Or do you? I would have thought I could maybe store some kind of pointer to a piece of data in the memory block and then I could just modify that one piece directly? Something along those lines.
To be completely honest about it… I haven’t had the need to touch my plugin since I changed it to write raw binary data to the memory block so it is still rather efficient in how it works. I haven’t had the need to update it since because it works. I moved on and began working on other things in which this problem never arose again in that context. Eventually though, I may need to update it and so I need to figure out a robust solution. Still watching this video though.
FURTHER EDIT: Another solution could be to make an abstract base class that saves and loads the data, and then you derive child loaders from it which implement the virtual functions. This way it could call the correct loader for whatever version. But this would mean implementing different load methods. However, then when the plugin loads and the version is detected, it could then be used to call the correct loader method which work as it does now with raw binary data. It could then use the old loading code in another “loader” for projects that were saved with an older data set. I cannot say that I like the idea of writing multiple different “savers”/“loaders”, but I suppose it is an option - A sort of middle-ground compromising kind of… solution.
I mean it doesn’t have to be written that way. It could just be one class with different methods that get called. Either way, same principle.
A lot of parameters lead to different problems. You may also notice that some VST3 hosts block the UI for a while when you load a preset for a plugin with a lot of parameters (maybe > 10’000 params). Or opening the context menu that contains all parameters in logic X takes ages.
If you have a huge amount of parameters, then i would think about removing some of them and handling them in an old-fashioned way with your own state objects you share with the UI without using parameters and value trees. In this case, you also have the possibility to cache the state inside those objects and optimize things.
I would still use XML as a basis because we have a wonderful parser here. But you can also embed base64 encoded binary data inside the XML or comma-separated values to reduce the size of the XML.
but updating the plugin will break existing saved projects that way.
You can add a version tag inside your XML that contains the state version. This way you can migrate the parameters of old presets inside the new format when the plugin loads the preset.
The thing is the parameters control other parameters in other plugins in a chain of plugins. If I handle them the way you are proposing would that still work for that? I am not completely behind the door with coding, but I am still rather beginner-ish with Juce and the way that it works so sorry if I am making some silly mistakes here and there.
I currently write a version number into the binary stream so I could easily do that with XML if required. I just haven’t found a need to because if I am using XML then the loading / saving is safe, although now you have mentioned comma separated values and base64 encoded binary data that does change things… that is something I will take a look at. Yes so basically, I could have one XML identifier that refers to a whole block of values that can be pulled out in one go!? I see what you are saying.
It’s nothing wrong in your solution. It’s always good to start with a simple straightforward solution. IMHO simplicity is the most important thing when writing code.
Please remember that you don’t call setValueNotifyHost from inside the parameterChanged method. This can lead to deadlocks with the VST3 format, especially with a lot of parameters.
I also think dispatching into the main thread has deadlock risks inside the parameter tree. I would decouple this with a non-locking queue read with a timer-based consumer that finally calls setValueNotifyHost.
Good point! the good news is, this is a plugin for entirely live performance situations. You midi learn toggle switches to buttons on a foot controller basically. The onscreen buttons are just for testing / switching patches to edit too actually.
What if it was a non-realtime bounce operation though? It is going to come up eventually. Lolz.
Anything asynchronous, especially Timers occur with a multiple of the actual desired time period, because processed time proceeds much faster than processing time would in real time.
And operations that effect the played sounds need to block in non realtime operation, otherwise the rendering is done before your sound was even loaded
Those meta parameters come up a lot, and there seems to be no clean solution except from: don’t. Try to deduce the controlled parameter at runtime.
As example: Don’t overwrite a linked parameter. Instead let the linked parameter show the original parameter and ignore the one you would have overwritten.
I know it’s not always possible…
But I am sorry, I’m afraid I derailed the thread a bit from Reaper