Crash Reporting from within a DLL (Audio Plugin)?


#1

Hi all,

I set out this morning to add crashlyitics https://try.crashlytics.com/ to an audio plugin project in juce. I was confronted with "This Xcode project does not have any Mac or iOS targets". Then this got me thinking—is it even possible to dump a stack trace & send a crash report from within an audio plugin while its running in a host? Has anybody implemented a system where users can report crashes?

Many thanks!


#2

Hi Rusty Pine (<- nice name),

I think this may be difficult as a crash inside the plug-in will also bring down the host. It is possible to get a stack-trace by getting a callback when the app crashes (via signals) but hosts will typically override this as well to do there own crash reporting. 


#3

Thanks Fabian!

From conducting a narrowly scoped test it looks like SystemStats::setApplicationCrashHandler() seems to work in a plugin running in a host. Within the callback it can get the stack trace via SystemStats::getStackBacktrace() and write it to a file. However, from poking around online it seems that running code in a process that has "crashed" is quite risky. Some people have recomended using fork() to open up a new process to conduct the reporting. I see there's some support for accessing the current process in Process::, but is there a juce way to create a new process?


#4

I scout for crash logs the next time the plugin is opened ... that works on the Mac.  I think Microsoft have a service where they'll send you the crash logs from their crash reporting service ... never tried it. 

 

If you get something good working let me know:)


#5

Yes, there is - check out the ChildProcess class!


#6

Yes, there is - check out the ChildProcess class!

Dang, how did I miss that. Well in my meager defense, that doesn't come up when searching "Process" in http://learn.juce.com/doc/classes.php

I scout for crash logs the next time the plugin is opened ... that works on the Mac.  I think Microsoft have a service where they'll send you the crash logs from their crash reporting service ... never tried it. 

 Apparently Apple will, too, but you have to have your app registered through the App Store. Scanning upon opening is a good idea. I've gotten Crashlytics working for 64 bit OS X JUCE apps (working on 32 bit with their support). I'll keep you posted about the plugin situation.


#7

Interestingly enough, if you set a functionin SystemStats::setApplicationCrashHandler() within your juce plugin (say in the AudioProcessor constructor) & cause a crash within your plugin code, the OS X crash report won't be generated (but your crash handler code will). I've seen this kind of "hijacking" occur in Logic Pro & Reaper so far. I haven't tested if the same happens when another plugin, instantiated side by side with yours, causes the crash.

Now I know in the docs it says "The usefulness of the result will depend on the level of debug symbols that are available in the executable." If I could just get function names & line numbers I'd be a very happy camper. Does anybody have any experience with re-symbolcating before I dive in?

 


#8

This thread seems to be the most comprehensive and up-to-date about JUCE and Crash-reporting.

Currently our biggest issue is Windows arghh…

@RustyPine , were you able to get a decent report to you can parse under Windows and symbolicate?

What did you eventually do?

Thanks!


#9

Ah - yes - I’ve got this running in an app now. It’s pretty good for 80% of cases, but it’s lacking the level of detail provided by the OSX crash report (e.g. what were the other threads doing)…


#10

I was directed to other work shortly after this, so it didn’t get off the ground that far. Re Crashlytics: they don’t have OSX 32 bit support.


#11

Thanks. we need of-course cross-platform sophisticated solution.
Especially on the Windows side where even getting a basic crashlog from the user isn’t trivial as with macOS.

There’s breakpad that has some good documentation (still need to figure how it can be incorporated into JUCE. (would be nice to make a crashreport module :slight_smile: )
And there’s the latest and greatest with less documentation and examples [crashpad] (https://chromium.googlesource.com/crashpad/crashpad/+/master/README.md)


#12

To update on this, for Windows apps, we now have this super-smooth … crashes get uploaded to a server and when double clicked with visual studio resybolicate themselves and give up their secrets.

I don’t have anything better than Crash Reporter on the Mac yet!

Basics

Use symstore to archive all pdb files from your builds. https://msdn.microsoft.com/en-us/library/windows/desktop/ms681417(v=vs.85).aspx

Call the following code on a crash to generate a minidump file which you can then later load into VS, with the symbol path set to your symstore.

#ifdef JUCE_WINDOWS

#include <windows.h>
#include <DbgHelp.h>
#include <tchar.h>

/**
 * Crash handler for Windows.
 */
LONG WINAPI createMiniDump(LPEXCEPTION_POINTERS exceptionPointers)
{
	auto dumpFileName = _T(AnalyticsManager::windowsMiniDumpFile.toUTF8());

	HANDLE hFile = CreateFile(dumpFileName, GENERIC_READ | GENERIC_WRITE,
		0, nullptr, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, nullptr);

	if (hFile != nullptr && hFile != INVALID_HANDLE_VALUE)
	{
		MINIDUMP_EXCEPTION_INFORMATION exceptionInformation;

		exceptionInformation.ThreadId = GetCurrentThreadId();
		exceptionInformation.ExceptionPointers = exceptionPointers;
		exceptionInformation.ClientPointers = FALSE;

		MINIDUMP_TYPE dumpType = MiniDumpNormal;

		BOOL dumpWriteResult = MiniDumpWriteDump(GetCurrentProcess(),
			GetCurrentProcessId(),
			hFile,
			dumpType,
			exceptionPointers != nullptr ? &exceptionInformation : 0,
			nullptr,
			nullptr);

		if (!dumpWriteResult)
			_tprintf(_T("MiniDumpWriteDump failed. Error: %u \n"), GetLastError());
		else
			_tprintf(_T("Minidump created.\n"));

		CloseHandle(hFile);
	}
	else
	{
		_tprintf(_T("CreateFile failed. Error: %u \n"), GetLastError());
	}

	int selectedButtonId = MessageBox(nullptr,
		_T("Application crashed.  Send error report to vendor on application restart?"),
		_T("Crash"),
		MB_ICONWARNING | MB_YESNO | MB_DEFBUTTON1
	);

	_tprintf(_T("User pushed: %d \n"), selectedButtonId);

	if (selectedButtonId == IDNO)
		DeleteFile(dumpFileName);

	return EXCEPTION_EXECUTE_HANDLER;
}

#endif

Call the code on crash from an app using:

#ifdef JUCE_WINDOWS
		SetUnhandledExceptionFilter (createMiniDump);
#endif

In plugins (Windows)
SetUnhandledExceptionFilter and setApplicationCrashHandler is the wrong thing to do in a plugin. It overrides the whole process’s crash handler, which probably includes the DAW.

The safe thing on Windows is to wrap all entry points in __try {… } __except { … } on Windows. I haven’t tried this with a plugin yet, but it ought to be possible. Note these are not the normal C++ try/except things but some MS extension.


#13

I think it’s possible to modify JUCE such that all plugin crashes on Windows are captured and minidump files are created. Using roughly the process I’ve described above.

I’m not sure if a similar OSX solution is even possible. If anyone has a better idea than parsing Crash Reporter log files let me know :slight_smile:


#14

Sorry if I miss understand, but does this mean you current implement your crash reporter only on the Windows side?

Thanks.


#15

For an app, we’ve been using the SystemStats::getStackBacktrace - but the info is worse than that provided by the native Crash Reporter.

For Windows we’re doing the stuff described above.


#16

Thank you for providing this @jimc, deeply appreciated!

It works great for a standalone application.

If someone is interested in some more examples: http://www.debuginfo.com/examples/effmdmpexamples.html

I recommend to add

#define NOMINMAX

before including <windows.h>, because otherwise it will define min and max and will mess with std.

I’m struggling conceptually with these entry points. Are you talking about the wrappers, e.g. the VST wrapper? Or about all member functions of the audio processor that are callable from the host (including processBlock())? Is there an elegant way to catch a crash that was caused by a user clicking on a button in the GUI? Or do I have to add __try{} and __except{} to anything clickable? And probably also to every Value::Listener’s valueChanged() and to every Timer callback?


#17

I think all the UI code comes back to a common entry point in the message manager.


#18

Sorry, but I still do not get it. Is there a best practice available on how to implement a crash handler in a JUCE-based plugin (on Windows and other platforms)?

I think that this is a very general requirement that should be supported by JUCE out of the box as it is needed by nearly all applications.


#19

I think if you’re a plugin the host will intercept the signal handlers. You won’t know if the host crashes. You could argue this is the correct behaviour otherwise you’d get a crash handler callback if another plugin crashed, with their stack trace in…

If the host has been configured to create minidumps or macOS crash logs correctly, you can get users to send you these though. As long as you have the PDB files, you should be able to debug the minidump for your sections of the code.


#20

You can always acquire the crash handler from the host, but that’s mean :wink: