AU example plugins crash on M1 in reaper/logic, when arming/unarming track

MacOSX and iOS
1

This happens with the unmodified Example AU plugins - not my own code:

I just pulled and built the latest JUCE develop branch to make sure - commit:
a4dfd8d6c6f499b9016157b9411653923eace215

I built (debug) with the following flags:
-G Xcode -DJUCE_BUILD_EXAMPLES=ON -DJUCE_BUILD_EXTRAS=ON -DJUCE_COPY_PLUGIN_AFTER_BUILD=ON -DCMAKE_OSX_ARCHITECTURES=arm64

I then start Reaper and attach to process.
I add a JUCE AU plugin to an audio track - any plugin, even the simplest example will do.

Then, I ā€œmashā€ the arm-record button on the track - be it Reaper or Logic, the crash is the same:

_os_object_retain 0x0000000190dcee24
RenderContextChangeGenerator::checkChange() 0x0000000192c04a4c
AudioUnitRender 0x0000000192deb720
AU_Plugin::ProcessSamplesImpl(float*, float*, int, int, int, int, double, MIDI_eventlist*, int) 0x0000000102a02b34
AU_Plugin::ProcessSamples(double*, int, int, int, int, double, MIDI_eventlist*, double, double, bool, bool) 0x0000000102a02288
FxDsp::processFxDsp(int, double*, int, int, int, int, double, midi_List*, double, bool, double, double, double, double, int) 0x000000010287abe8
FxChain::ProcessChainDsp(FxDsp*, int&, int, int, double*, int, int, midi_List*, double, double, double, double, double, int, double, int, bool, bool, int&, int&, int) 0x00000001028a218c
FxChain::ProcessChain(int, double*, int, int, int, int, double, midi_List*, double, bool, int) 0x00000001028a188c
MediaTrack::RenderSamples_nocache(double, long long, double*, int, int, double, MediaTrack* const*, int, int*, bool, int, int, bool*, SyncSMP_Context*) 0x0000000102623e48
MediaTrack::RenderSamples(double, long long, int, Track_RS_Output*, int, double, MediaTrack* const*, int, midi_List*, int, int, int, int, int, MediaTrack::Track_SendRec*, bool*, SyncSMP_Context*) 0x00000001026196dc
MediaTrack::RenderSamples_nocache(double, long long, double*, int, int, double, MediaTrack* const*, int, int*, bool, int, int, bool*, SyncSMP_Context*) 0x0000000102621614
ProcessProject(ReaProject*, int, int) 0x000000010268e584
audiostream_onsamples(double**, int, double**, int, int, int) 0x000000010268d498
audioStreamer_CoreAudio::onsamples(AudioBufferList const*, float*) 0x00000001026f4e74
caInproc(unsigned int, AudioTimeStamp const*, AudioBufferList const*, AudioTimeStamp const*, AudioBufferList*, AudioTimeStamp const*, void*) 0x00000001026f5d74
HALC_ProxyIOContext::IOWorkLoop() 0x000000019325793c
invocation function for block in HALC_ProxyIOContext::HALC_ProxyIOContext(unsigned int, unsigned int) 0x000000019325587c
HALB_IOThread::Entry(void*) 0x00000001933b8564
_pthread_start 0x0000000190f7ffa8

I know that stack trace isnā€™t too informative, but I canā€™t find what process I need to be attaching to to get the actual plugin one - is there another process for Reaper as for Logicā€™s ā€œAUHostingServiceXPC_arrowā€?
(I canā€™t test with Logic on my machine, I have no licenseā€¦)

What could this be? Is it a known issue? I could find anything on the forum after searching.

2

Debugging AU on macos is difficult. Iā€™m just reacting here to give more viewers for this, and maybe get some good advice how to do it. I get errors with AUā€™s too, which do not occur on iOS and not with vst3 on macos, with the same code.

3

I agree AU is trickier than vst3!

But this is not my code, but the stock, unmodified JUCE AU examples - so most likely a bug in JUCE itself.

It was only in the title of my original post, but I modified the post to convey this more clearly.

4

Thanks for reporting. At least for Logic I think this is the same bug as reported here. Toggling the record arm button was also how I got the error to occur. I think if you try the commit before the Audio Workgroup stuff was added, I expect it will run fine?

Iā€™ll add Reaper to my testing as I go. There is a fix in the pipeline but there are other edge cases we have to consider and I now expect that wonā€™t be ready until hopefully early sometime next week :crossed_fingers:.

1 Like
5

Great, thanks!
Iā€™m developing a multi-threaded audio plugin so the apple workgroup will be nice to have, great to know you are working on a fix, since itā€™s a known issue we can test the vst3 in the meanwhile!

1 Like
6

Thanks all for being so patient. This should now be fixed on the latest tip of develop.

7

Can we somehow completely opt out of that feature? I think the people who asked for it, assumed it would guarantee their plugin would run on a performance core and it has only caused pain for everybody else.

Ideally a define that leaves out all code related to this. I donā€™t want to ā€˜payā€™ (code size and bugs) for a feature I will never use.

2 Likes
8

Opting in/out of features like this makes for very difficult testing and support, itā€™s yet another flag we have to consider. This potentially means we have to multiply in this case our macOS and iOS builds by two to test with the flag on and off. If we did it for this feature what about other features where would/should it stop?

Iā€™m sorry that this bug took a couple of weeks to get a fix merged but it is on the develop branch where we trial new features which may change or have bugs. However we try our best to get these fixed as soon as possible.

I did a quick check on the code size, on a universal binary of a release build of an AUv2 plugin and found it added 998 bytes, this would normally equate to fractions of a percent of the final binary size. Iā€™ll give you this is more than I expected but I think there would be an awful lot of defines we would have to add if we were to have an opt in/out for every feature that added that amount of code.

1 Like
9

Iā€™d say this is a necessary feature, not an option. If you want to opt out, donā€™t pull the changes - but, at some point in the near future once this AudioWorkgroup feature has been tested by all of us and becomes a bit more stabilized, be prepared to deal with it - because its key to high performance audio on Apple platforms now, whether we like it or not ā€¦ and is clearly going to become more and more relevant with each new releaseā€¦

1 Like
10

This highlights a major concern I have had with Juceā€™s release model for a while and itā€™s really come to a head this year.

Of course we have unstable features in dev, itā€™s where they belong, but those being mixed in with so many minor but critical fixes with minimal probability of causing problems that people urgently need for so long is becoming an issue.

Will a new master release with lots of useful minor changes be delayed until this cutting edge new tech is stable? It feels like this is the model we are working with more often than not. I was picking bits and pieces out of dev since January for my local 7.0.5 right up until August and itā€™s now a month that weā€™re into this mode of working again even for those using 7.0.7.

We need a better way of dealing with this situation because it wastes so much time re-discovering issues while some major change is holding up the master branch, or causing people to run into avoidable problems because theyā€™ve ended up using dev for releases because they needed the fixes there.

3 Likes
11

I donā€™t see a reason to ever use this feature. For what exactly are you using it? Do you know what it does?

12

@reFX We never need Midi in our plugins since we create only pure audio plugins. I donā€™t see a reason to ever use this feature. Still I think it would not request to make Midi support optional in the AudioProcessor class ā€“ even if it would probably reduce the binary size a lot.

I personally wish for a JUCE with less macro based configuration options for the sake of an easier build of e.g. a static library version of it.

And I really appreciate the addition of this feature, since it can be a great performance to plug-ins that use their own extra threads for processing, which is something that can really make a difference for our plug-ins which spawn their own additional processing threads.

I totally understand people not needing some features of the library, but I think opting out of features so deeply integrated across multiple classes is probably the wrong solution to this. I also think itā€™s completely usual that big new features need a few fixes at first.

I also understand @Verbonaut 's point of view. I wonder if feature branches would be a good solution to this ā€“ if I remember correctly, this approach was chosen for the accessibility additions back then? With that approach people in need of the new feature can check out the feature branch, test it, report bugs and if there are no major problems reported after some time, the branch is finally rebased on the develop branch. Still it might be difficult for the JUCE team to decide which feature is big enough for a feature branch and which is notā€¦

3 Likes
13

I agree with the feature branch concept. I donā€™t think itā€™s hard to identify major features that would benefit from this. Good engineers are generally good at judging when things have the potential to cause problems. The VST3 build system change, the AAX track widths change, the big changes to timer behaviour, the big re-write of the files handling modal dialogs, this AudioWorkgroups change; only two of these five have gone smoothly to my knowledge and all were quite obvious candidates for a more cautious approach to the rollout. In a couple of areas we would be seeing things introduced more slowly, but in many other areas it would save people a lot of trouble.

14

I partly disagree, whatā€™s the meaning of feature branches when only few people actually testing them. In the end, issues will be revealed just later or when itā€™s too late at customers side.

I think there is no perfect solution for this, but having one official develop and one master I think is the optimum. For very huge changes separate feature branches might make sense.

And I am absolutely agree that new macro-switches should by avoided. With constexpr and template constructs we have a much cleaner solution for such optional features.

15

Feature branches with important new tech that only a few people use are the type of case where the team would do very well to engage directly with specific developers that are willing and capable of going into the depth required to test something specific and potentially can pass builds to their own beta team for extended long burn testing as I did with the track widths update. Throwing out big but special interest changes to dev hoping somebody with the right skillset will try them out at the right time is of course doomed to failure. Iā€™m sure there are similar developers looking at the Audio Workgroups feature right now that would be willing to do that (and it sounds as though some people are using it). In this situation itā€™s even more important, otherwise we end up holding up the master for months or pushing something out without sufficient real-world testing that can wreak havoc if something big goes wrong.

16

@PluginPenguin Unfortunately, you compare a fundamental, very essential feature for every single supported platform, used by virtually every JUCE developer, to some super-special targeted feature, only available for macOS/iOS, and it doesnā€™t even help that this specialty feature doesnā€™t solve any specific problem.

Is there now a guarantee your real-time threads will run on a performance core if they are part of the audio-workgroup? No.

Is there any guarantee that it will perform better than another thread of similar priority? No.

Plus, splitting the processing into multiple threads that all have to join before the process block is finished is a shaky concept at best. All you are doing is making the DAWs job harder. The DAW already splits processing over multiple threads and cores. If the plugin does it as well, it might throw off the scheduling of the DAW completely. You still need the same amount of time for your processing (plus the overhead you produce for synchronization, thread startup/shutdown, etc.), but instead of using one core to e.g. 20%, you now use four cores for 5% each. Itā€™s still 20%, just spread out. Youā€™re not saving anything. Youā€™re just making your own job and the DAWs job much harder without any practical gain.

17

I can see why feature branches are appealing but in practice Iā€™m not convinced they would be worth the effort. They could delay bug reports which only makes it harder to fix the bugs. The branches would need to be kept up to date, if there are several of them you have the issue for users who want to try more than one. Then who knows what will happen when they all come together!

I could see how short lived, temporary flags to opt out could be useful, but every solution brings with it a set of compromises. In that case we would have to make sure weā€™re adding a bunch of code to opt out of the feature with additional testing that it works to actually opt out, then we have additional commits to remove the flags before it makes it into master.

Thereā€™s rarely one obvious answer to this kind of thing. Weā€™ll keep an eye on how things progress, weā€™ll likely still do a new branch for very large changes such as a new major version for users to try, but right now I think this should be the exception rather than the rule.

In this particular instance as we were unable to deliver a fix quick enough maybe we could have benefitted in offering a better temporary solution for users. Iā€™ll give it some thought and chat with the team.

Regardless we appreciate the feedback, and Iā€™m sorry for any inconvenience caused.

2 Likes
18

If you use the trunk then you should accept stuff like that. simple as that.
I was annoyed by this bug as well but honestly that the joy of living on the edge :slight_smile:

Regarding the use of this, Iā€™m one of the requester. And just because you have no use, itā€™s stupid solution. We have a modal reverb for exemple with 20k mode and this required multithreaded rendering. This is useful as well for sampler where you need to crunch lots of voice in a single part (for example a piano with 3 mics)
As a side note obivously you donā€™t start thread each time so the only synchronization which if done right is minimal. The solution at the end of the day is maybe something like CLAP multithread stuff but in the meantime and for standalone version this is a quite useful feature.

1 Like
19

Obviously, this wasnā€™t very well tested, given that just compiling and running an empty plugin would make it crash on shutdown.

I find this reasoning very, very questionable. The whole concept of ā€œweā€™re not letting anyone outside help because they donā€™t test as well as we doā€ while every single release has some issue (major or minor) is a sham. Itā€™s an excuse with no basis in reality. 95% of the bugs you are only aware of because the unworthy have found them, usually within a few hours of release. We donā€™t even run the supplied JUCE unit tests but actually use those new features.

The JUCE team has to eat its own dog food. It seems youā€™re not using your own code, and thus most bugs just pass you by.

Here is a list of all the bugs we have found. This is just us using JUCE.

Each folder is a short code example showcasing a bug. All these bugs have been reported on these very forum. All with screenshots and descriptions (where appropriate). A lot of them are just ignored and we have to work around them or maintain our own fork with the appropriate fixes.

So yes, I would like you to put #if #endif around these instances, so we can opt out completely.

20

Wow! this is super useful, thatā€™ll be the next year of my life then :laughing:

I understand it must be frustrating. If it helps to understand I think the bug that introduced that issue was late breaking on our end and we should have done more testing at the end, but it also had a messy relationship with using workgroups in other contexts such as an application. Second to that in testing we found another extremely intermittent issue (I was finding it by loading and unloading 1ā€™000 instances at a time). Unfortunately adding tests for a feature like this is not easy but we might at least want to check that everything compiles correctly with a flag on/off.