Captured audio becomes garbled after several hours


#1

On windows using DirectSound. The captured audio is fine for approximately 5 hours. After that, it suddenly becomes garbled. We experience this behaviour on several Windows XP desktops, on various sound cards.

Once the the audio becomes garbled, our Juce application needs to be restarted. Audio is then fine for another 5 hours approx.

Any idea what could be causing this ?


#2

Well, it sounds like a number is overflowing somewhere, though I really can’t imagine where that could be happening… There should be no problems in the audio i/o classes, which don’t care how many samples have gone by. And the stuff that writes to the wav file uses 64-bit numbers for all its positions. When you say it becomes garbled, have you any way of knowing whether it’s the incoming data that’s broken, or the act of writing it to the file?


#3

We write the audio to a .wav using libsndfile, not Juce. But I know for sure that the problem is not in libsndfile because of the way we noticed the problem : our application streams the audio to a speech recognizer through a socket and accuracy suddenly drops dramatically.

After we observed that accuracy drop, we noticed that the audio recorded to file was also bad. We then instrumented the application to record the audio upon entering audioDeviceIOCallback() : same thing, audio is garbled. We also code reviewed our implementation of audioDeviceIOCallback() to be sure that minimal work is done inside and that we don’t block.

Bottomline : audio is corrupted before being delivered to the application.


#4

I can’t see any counters in the DSound code that could be overflowing… Does it recover after it becomes garbled, or does it just stay mixed up? (There’s a chance with Dsound devices that the input and outputs could gradually drift apart and it’ll need to re-sync itself, but that should just be a temporary glitch).

Does sound very odd though… I guess it could also be the actual DSound driver itself messing up, you might want to try with a different soundcard to rule that out?


#5

When the audio becomes garbled, it never recovers. We’ve observed this behaviour on several desktops, with different sound cards.


#6

Really very odd. It does sound like a numeric overflow problem, but I really can’t see any places in the code that look like a candidate.

Do you know if it always happens at exactly the same time, or maybe it’s just something that’s random but which is likely to happen within roughly 5 hours?


#7

Is this only a DirectSound problem, or does this same issue occur with other APIs like WASAPI (Vista and above only) and ASIO?


#8

The garbled audio occurs at any time of the day, roughly 5 hours after application start().

So far we have observed the problem with DirectSound, on Windows XP and Windows 7.

We have not yet tried to replicate the issue with Windows Audio.


#9

Well, I’m normally very good at spotting problems with things like this, and there really isn’t anything dodgy about the DSound code - it simply doesn’t use any counters that could overrun. Assuming that the underlying OS drivers continue working, it should all just keep chugging along forever.

…but I’m assuming you’re using the latest juce code, right?

If I was debugging this in my own app, the first thing I’d try would be to remove all the other audio processing code, and do nothing in your audioCallback except to stream the audio to a file. That would rule out the possibility that an overflow somewhere else is overwriting memory and just screwing the program up more generally. (Or just run the juce demo app on the target machine, and leave its audio record demo running, to see if that has the same problem)

I’d also put a breakpoint or logging in DSoundAudioIODevice::resync, just to see if a loss of sync is happening.

And check that you’ve got juce assertions turned on, to catch anything strange in the Dsound code.


#10

We’ll try all these suggestions and let you know what we find out. Thanks.


#11

I’ve currently left the jucedemo audio page running, so will report back in about 5 hours whether it’s still ok on my machine!

Other ideas:

  • memory leaks? If your app is 32-bit it might be running out of memory.
  • are you also playing output? If so, does that also break?

#12

FYI, yesterday, we replicated the issue but it took 8 hours for the problem to show up.

The application plays output for a few seconds when it is started, after that playback is no longer used, just capture.

The application does not leak.


#13

Ah nice. So a random bug that takes at least 5 hours to appear. Every programmer’s nightmare!

Yeah, but do you leave the output channels running? It doesn’t matter whether they’re playing silence or not.

Another idea: could some other process on your machine be temporarily eating all the cpu/memory so that your app gets starved? E.g. virus checker, etc. If the sound device isn’t able to keep running in real-time, things can easily get messed up.


#14

Hello Jules,
I work with Dominc on this issue. We have made another discovery. We managed to reproduce the issue using only the Juce Demo in record mode but it took a very long time (the demo was running for at least a day and a half – it happens faster with our app, sometimes 5 hours, sometimes 8 hours)

Most interesting however was the action that I had done which immediately solved the problem. I went back in the demo Audio Device Setup Tab and changed the buffer size. Following this my recordings were OK again. I do not think it is linked with the actual buffer size; I did this as I was sure this would do a reset of the audio, and indeed it seems to have worked…

I attached a zip with a bad and OK file if you want to listen.

Thank you.


#15

I’m afraid I’m a bit stumped by this one… I really don’t know what else to suggest.

For a quick fix, certainly something worth trying would be to make 100% sure (i.e. get in there with your debugger and check) that you have no output channels active, so that the device code is definitely not trying to sync two devices. That’d cut down the code complexity involved quite a lot.

But for a “proper” fix, I just don’t know… The code is pretty simple and I really can’t see any problems in it. You’re running with assertions enabled, right? So if an assertion happened, you’d know about it? And did you check the log for any DS debug messages?


#16

This is some Heisenbug derivative…


#17

Hello Jules and thanks again for all your suggestions, we are trying them all and we’ll be hammering on this till we’ll find a solution.

Asserts are and have been always enabled, none happened during the bug.

I recently enabled the debug logs, about the same time when I added the log in DSoundAudioIODevice::resync, but was unable to reproduce the issue since. I did not see the resync log since and all juce logs I see as logs are at the beginning (attached).

The output channels ARE enabled, I will try with them disabled to see if I can reproduce it that way (takes me a while to reproduce).

I would like to ask your opinion on the way we implemented around the AudioDeviceManager: There is only one instance of an AudioDeviceManager that is initialized once(in has IN and OUT channels). Anytime we need audio functionality, IN or OUT, we add a callback with addAudioCallback (we might use it with a AudioSourcePlayer or to record the sound, or to play our own sound) then remove the callback. There are maybe 5 -5 callback, one of which is always active, used to display in sound like in the demo (while debugging this I disabled activity in all of them). We never close the device or reinitialize it (we change the buffer size on it once at the beginning). This works great, except for the current bug. I was curious on your opinion on this however.

The second opinion I would like to ask is about the version of Juce. We are unfortunately still at version 1.53. I know this is not ideal, but it works for us for now. I was however still wondering if you think this is a major issue, if you have done major work on the audio or DSound part since. (I will get the current version and look for myself, but would appreciate your opinion too, I might not be able to tell if the changes are relevant or not).

Again thank you very much for all your help,


#18

You could have mentioned that in your first post before I wasted time scouring my code looking for problems. :evil:

I can’t remember whether anything relavent has changed in DSound since then, but there’s a very good chance that it has.


#19

Reporting Bugs - Please read this before doing so!


#20

I was able to reproduce the issue with the Juce Demo of the latest version 2.0.

To reproduce it I modified the AudioRecorder of the AudioDemoRecordPage.cpp (I attached a copy) to automatically start recording every 15 minutes of so for 30 seconds. I then used an audio stereo Male - Male cable to feedback the output (headset) of the computer back in the intput (microphone). After wards I play any audio on repeat, and let it run. It will eventually happen.

Also to note I was systematically able to make it happen on every Windows XP desktop I tried it on IF I changed the audio buffer size in the Juce Demo Setup page to 44.6 ms (2048) instead of the default 58 ms. This could indicate it has to do with the buffer size, but also with the call to AudioDeviceManager::setAudioDeviceSetup, which seems to screw up the future audio, and immediately fix it, as I observed in the past.