Plug-ins using JUCE bring down Cubase 8


#1

Hi Jules,

Ulf here from Steinberg.

Cubase 8 got released in December and we receive lots of customer reports on Cubase crashing when opening the mixer window by pressing F3. Looking at the crash dumps we get, it always points into certain plug-ins that are causing the trouble.

Having further analyzed these crashes, it actually turns out that JUCE is causing them, because in dispatchMessageFromlParam() the reinterpret_cast on lParam succeeds, but the object behind it is invalid and so message->messageCallback() results in an access violation.

Cubase itself doesn't use JUCE, but we have one special plug-in that uses it and with this plug-in I can also reproduce that issue (that's also the reason why I can pin point to it in the debugger.) Our plug uses JUCE 3.0.8.

Have you heard of such issues before and will you look into it?

I can help in providing more details and also provide you with a temporary Cubase 8 license in case you need it.

As said, it's plug-ins from various vendors that expose this behaviour and from our perspective it is very serious, because it makes Cubase very unstable. and it doesn't help Cubase users to tell them not to use JUCE based plug-ins.

Also, if you have a fix it would be best to notify all JUCE developers of such fix, so that they can publish updated plug-ins in order that they run stable with Cubase 8 again.

Best regards,

UIf


#2

-  Another issue with Cubase 8 : popup-windows insde JUCE-Plugins do not appear when the "plugin window always stay in forground"-option is active (not sure how the text is, i use the german version) , it works in Cubase 7

 

 

 


#3

Another thing : in Cubase 7.5 Mac and probably in Cubase 8, when you load a plug-in, if you close the UI and open it again, Cubase crashes, and if not it will crash when you try to close the DAW


#4

Wolfen: can you reproduce the issue with the JUCE Demo-Plugin? Do you have stack-traces? I'm just wondering, because didn't get any error-reports from my costumers yet.

Ulf: I could reproduce the issue with one of my older Plugins. But it happens only once in a while, and i'm absolute no expert about windows messaging. I think you are in the perfect position (you have both sourcecodes)  to provide a fix, or a tip how to fix this.

Even if this issue can be fixed inside JUCE issue, this will be long-time issue with Cubase 8, there are so many JUCE-plugins outside.

Jules, any comment on this?

  


#5

 I think you are in the perfect position (you have both sourcecodes)  to provide a fix, or a tip how to fix this.

+1

Ulf, I would really like to try my plug with cubase 8, is there any demo version available for download?

ps : this thread should be moved to the plugin section


#6

Even if anyone else had the code of Cubase it wouldn't help, because Cubase just calls the WinAPI DispatchMessage() and then the JUCE plug-in that receives the message crashes. And it's not any Cubase messages that brings them down, they are JUCE defined messages, that's why I insist that it is a JUCE problem and needs to get fixed by Jules. He knows his own code best and also don't forget, we paid for it. But if Jules doesn't start working on it, I guess we have to do it ourselves.

It's a serious issue, because it also falls back on Steinberg, because end users don't care in the end if it is a plug-in or Cubase problem. Their system doesn't work as expected and hence will walk away if we can't fix it in time.


#7

btw: the popup issue seems to be fixed with 8.0.5

 


#8

Just spend 3 hours of my precious time to debug this, but i couldn't reconstruct the problem with the latest JUCE.

Ulf: Could you reproduce the problem with latest juce? Why do you think the problem occours, and why only with Cubase 8?

> don't forget, we paid for it. 

BTW: spend much more money on cubase updates, as i paid for the juce license ;)


#9

Guess how many hours of my precious time I already spent on this...

And yes, I've just downloaded the latest JUCE and still same crashing for me.

Normally Steinberg doesn't use JUCE, but we have one special plug-in that got externally developed for us and that uses JUCE. So with this I'm plunging now deeper into it. At the moment it seems to me that a CallTimersMessage object gets deleted while it is still referenced by a message under dispatching.


#10

So yes, the problem is with timer messages.

JUCE sends own timer messages around that get dispatched via the normal windows message handling system. Together with that timer message JUCE puts into the lParam a reference to a callback object of type CallTimersMessage. There is only one instance of this object and so the reference is always the same.

In dispatchMessageFromLParam() the object gets called and after that the reference count on it decremented. Then I see that the destructor gets called because the refCount drops to 0. But then comes in another timer message that still has a pointer to that just deleted object and hence the crash.

So an object gets deleted too early, though it is still referenced by some not yet dispatched window message.


#11

Hi Ulf

Ok, I've been staring at the code around dispatchMessageFromLParam and trying to think what could be going wrong, but am really struggling to find a bug..

You say it happens when the user opens the mixer panel, so am I right in assuming that it's not related to the plugin being deleted or unloaded? Normally when I see bugs involing the message-loop, the reason will involve the plugin DLL being unloaded from memory while there are still messages queued for delivery, and things get deleted in the wrong order. But in this case you say it's crashing while the plugin is open and running normally?


#12

Just a couple of other observations, but I'm really failing to see what could be the problem here..

I can't see any way that the ref-counted MessageBase object could be a dangling pointer when this is called, because every time that object is posted onto the message queue, it has its ref-count incremented. If it's the Timer class that's involved, that class dispatches new CallTimersMessage objects, and doesn't keep any other pointer to the objects, so they can only be deleted when their reference count drops to zero, there's no other mechanism.

So the only suggestions I can possibly think of are:

Does cubase do anything that might re-send a message? If a message that was posted to the queue was somehow made to be artificially delivered twice, then that could cause this behaviour, but I can't think of anything in my code that would do that.

Or, is it remotely possible that cubase is using the same number as the message ID (WM_APP + 0x4400), and is dispatching its own message to the plugin's HWND which is being mistakenly recognised by JUCE as one of these special ones, so that the LParam is actually junk?

 


#13

Probably completely unrelated, but I was encountering a very strange floatting point issue in the setParameter callback of my plugin in cubase 8 (32-bit only, windows) when opening the mixer window. Adding a call to _mm_empty() at the beginning of the plugin's setParameter method fixed that issue. So maybe there is some MMX code somewhere that lacks a call to _mm_empty to restore the fpu registers to a usable state.


#14

Hi Jules,

first, no, Cubase doesn't use that ID (WM_APP + 0x4400).

Then, as lParam for the timer message I always see the same object pointer. As already mentioned, then when the mixer window gets opened, I see the destructor of that object being called, because the refcount dropped to zero. Then I let the debugger run again and immediately comes the next message with the same object pointer, but then of course that object not being there anymore.

I will further investigate on your hypothesis on Cubase re-sending the same message twice.

And maybe a bit more background on this problem. It not only manifests itself with opening Cubase's mixer window, but can also be shown on other occasions. It's just most prominent there, because that is where our users are most frequently are running into it. E.g. also with creating lots of new tracks, one after the other, we can reproduce the same issue, but this is what people are not doing so often.

Furthermore, the problem doesn't happen immediately, from my experience it depends on the size of the project you are working on. It's much more likely to happen on projects with lots and lots of tracks (but not necessarily lots of plug-ins, just one JUCE plug-in is already enough). My nearly 100% repro is, create a new empty project, add 200 audio tracks and then instantiate our JUCE plug-in. Select the last track and press F3 to open the mixer, poof. Undisputably, opening the mixer window is not the fastest operation in Cubase that might take some hundred milliseconds. Maybe this is where the message dispatching gets slowed down or out of order, have to further investigate. On the other, this still shouldn't lead to a crash of the system.


#15

Thanks Ulf

It's correct that you'll see the same pointer appearing multiple times, because the internal Timer thread keeps a re-usable message object that it keeps re-sending. But it's reference-counted so it should be impossible for it to be deleted until after the Timer thread has exited (which only happens when the plugin DLL is unloaded). If it's becoming a dangling pointer before that time, then the only way I can imagine that happening would be if at least two extra duplicate messages were to be delivered somehow.


#16

Okay, as the dll definitely doesn't get unloaded, we have to find out why the object goes away or why maybe a message gets sent repeatedly.

I see that your refCounter is atomic, but could it be some other race condition?


#17

I think that if there was a race condition causing a leak of these objects then we'd certainly have heard about other problems with it.. The Timer thread and its message object get constantly hammered in all juce apps and plugins (and is the same on all OSes). It seems suspicious that there has never been any similar problems except in this particular version of Cubase.

I'll keep thinking about it but currently my best guess is still that something is re-posting messages..


#18

it's not something particularly found in Cubase 8, it's just that it now happens so much more often. With Cubase 7.5 we definitely have the same problem, but here it manifests itself more in crashes when repeatedly creating tracks.

And also with earlier versions we sometimes received crash dumps that pointed into plug-ins. It was always difficult to reproduce such issues and we always defered them to the plug-in vendors. Could be that, if not all, but at least some were already related to this.


#19

My research so far

changing the special id "specialId   = WM_APP + 0x4425;" does NOT help!

increasing message-resending time-out in the timer class " const uint32 messageDeliveryTimeout = now + 10000;" does NOT help!

I added some debug statements


MessageManager::MessageBase::MessageBase()
{
    DBG ("Create "+String::toHexString((int64)this)+"\n");
}
MessageManager::MessageBase::~MessageBase()
{
    DBG ("Destroy "+String::toHexString((int64)this)+"\n");
}
.....
bool MessageManager::postMessageToSystemQueue (MessageManager::MessageBase* const message)
{
    message->incReferenceCount();
    DBG ("Post "+String::toHexString((int64)message)+"\n");
    return PostMessage (juce_messageWindowHandle, WindowsMessageHelpers::specialId, 0, (LPARAM) message) != 0;
}

The Message is defenitly reposted by "MessageManager::postMessageToSystemQueue" after its deleted.


Post 288bfc98
Post 288bfc98
Post 288bfc98
Post 288bfc98
Destroy 288bfc98
Post 288bfc98
First-chance exception at 0x51750947 (myPlugin.dll) in Cubase8.exe: 0xC0000005: Access violation reading location 0x00000004.



 

 


#20

Sorry, maybe I didn't properly explain what I meant by "reposting".. Let me try again:

Yes: the same MessageBase object will definitely be sent many many times. The Timer class keeps one instance and re-uses it (see juce_Timer.cpp, line 52). That's intentional.

But when sending a MessageBase, it gets cast to an LParam and pushed onto the system queue as a win32 message, and its ref-count is incremented (see juce_win32_Messaging.cpp line 140 in MessageManager::postMessageToSystemQueue()). So that when delivered, its ref-count gets decremented and everything is good.

What I think is happening here is that once the win32 message has been sent via PostMessage, then a second (or third) copy of that raw win32 message is being re-posted to the plugin's HWND, so the same object will get its ref-count decremented multiple times.

As far as I know, nothing in my code will re-post a win32 message, but it's the kind of trick that DAW writers often use (e.g. in hooking events, forwarding key or mouse events etc), so that's why that's currently my main suspect.