Audio + Sensor Data (as audio) Recorder Question

General JUCE discussion
1

@Jules & friends - I have made quite a bit of progress over the last few days with the recorder application you have kindly helped me with in another thread (writing multitrack audio to disk). The audio portion (which although can use a few changes) is working well for now, and so I’ve spent the last couple days working on implementing and extending the JUCE style serial classes from http://www.anticore.org/juce/ for recording the sensor data as audio.

Basically, right now I am trying to get serial-data from sensors (via an Arduino) writing to disk in .wav format, synchronously with incoming audio, so that I can analyze the audio/sensor data for machine learning and other research projects I am working on as part of my PhD.

I just got a non-sexy, first effort try working pretty quickly, but wanted to ask if besides some of the larger structural changes I will make to fully implement this, if you guys for see any issues with the way I am currently doing things, in particular, either in synchronicity of the outputted audio/sensor recording, or with any problems that might arise from read/write access to a single array from two asynchronous sources. Heres how I have it set up.

My SerialRecorder class has a changeListenerCallback which gets called whenever my arduino has new data from the sensor (right now for testing, just a single potentiometer, connected at 9600baud, and sending data with a 10::ms sampling delay in between messages)-- it then fills an array (the same size as my audio buffer) with this one number (sort of a brute sample and hold method).

My SerialRecorder also registers an AudioIODeviceCallback with the same audio manager as my audio recorders. In here, just like my audio recorders, is where I add the samples to a circularbuffer (the same way as my AudioRecorders, and derived from the audio recording demo and Jules’ help). The difference is, instead of passing in one of the channels from my AudioIODevice / inputChannelData, I pass in the array with I’m continuous refilling with the newest sensor data in my sensor callback.

In this way, I was hoping to asynchronously have my sensor callback fill the array with the newest sensor sample data, as fast as it samples the sensor, and then have it actually pass to circularbuffer and write to disk synchronously with the incoming audio / audio recorders.

Do you see any issues with this approach? Particularly if the AudioIODeviceCallback is trying to access the sample array as the serial callback is trying to rewrite the array with the newest data?

Thanks for your help!

1 Like
2

I guess that if you need to sync your incoming data to the incoming audio, that’s not a bad approach. Presumably you’re handling the situation where your data buffer is getting too full or too empty to keep in sync with the audio rate by dropping or adding samples to it?

The alternative approach would be to just write your data directly using a thread, occasionally taking timestamps from the audio device to check that it’s in time, and correcting it if necessary.

3

yep, seems to be working pretty well right now, as one brute class, however, I am now trying to seperate out the SerialRecorder into two classes. Today I started structuring things as follows: a SerialDeviceManager class (to act kind of like AudioDeviceManager) which will handle connecting to a serial device, receive its data, and pack the different message streams into an array (kind of how AudioDeviceManager has an array of each inputChannelData, I’d like to have an array store difference messages, separated by start/stop serial messages). I have the basis of this created and working-- however, not a multidimensional yet, to deal with more than one sensor, but for now im just trying to get it working with one serial input. Right now, the SerialDeviceManager class allows you to select a serial port to connect to, it connects to my arduino, receives the serial data and packs/prepares it into an array to pass to my audio recorder.

Next, I’ve separated out the recording functions into a SensorRecorder class which should be sent the array being prepared by the SensorDeviceManager via a callback, and do all the recording functions triggered synchronously with my AudioRecorders via the AudioIODeviceCallback as previously described.

I had this working within a single class, however, now that I’ve separated responsibilities out into two separate classes, the SerialDeviceManager and SensorRecorder(which will be very necessary when I’m dealing with multiple inputs), I can’t figure out the best way to set up a message / callback to pass the Float* [] to the SensorRecorder class. I’ve been digging into the docs and forums, particularly looking at the Message and MessageListener classes, but cant seem to get it going. Any tips on setting those up to have the SerialDeviceManager send the Float* [] and have the SensorRecorder class be notified and passed the array? Should I be looking at other callback classes? From what I’ve gathered digging into the docs, the other callback classes won’t let me pass the Float* [] and are more of notification-only style callbacks…

So much to learn!

As always, much appreciation for all of your time, cheers!

4

After a few more hours, I managed to get it working 2 different ways-- but not sure if either are decent ways to do this. Here’s are my attempts, which I admit, are probably dirty and I’ll probably be ashamed for posting them in the morning! :slight_smile:

  1. I made my SerialDeviceManager class a ChangeListener and Changebroadcaster. In my MainComponent I create an instance of my SerialDeviceManager class (serialDeviceManager) and my SensorRecorder class (sensor1). I then register my MainComponent with my instance of SerialDeviceManager (serialDeviceManager->addChangeListener(this) ). In my SerialDeviceManager class’s callback that polls incoming serial data, each time there is a new message I simply sendChangeMessage(sensorData) (which is a float *) ,which notifies MainComponent. Back in MainComponent’s changeListenerCallback method, if the object that has changed is indeed sensorData, then I pass it to my AudioRecorder class through set and get methods like so sensor1->setSensorData(serialDeviceManager->getSensorData());

It seems to be working, but I’m not sure how safe this is, and it seems to me that there may be a more efficient and/or elegant way to set up a callback in MainComponent to be notified when new serial data is available in my SerialDeviceManager and to pass that on to my SensorRecorder class, or to cut out my MainComponent class from the equation, and simply register my SensorRecorder class as a listener to SerialDeviceManager. This led to:

Method 2:
In MainComponent, where I created my instance of SerialDeviceManager and SensorRecorder, I say: serialDeviceManager->addChangeListener(sensor1); --this is to register my SensorRecorder class (which I now made a ChangeListener) with the sendChangeMessage(sensorData) being broadcast by serialDeviceManager whenever it has new serial messages. Then in my my SensorRecorder class (sensor1), I just have a changeListenerCallback as such:

void SensorRecorder::changeListenerCallback(void* objectThatHasChanged){
sensorData = (float*)objectThatHasChanged;
}

Seems a little better than try 1, but as I am still pretty new to all of this (JUCE, C++, pointers…etc) I’m not sure if I’m taking the right approach, making inappropriate casting, being bad in general haha… Thoughts?

J

5

no… that’s not good. Don’t use the ChangeListener to pass data - multiple changelistener requests get coalesced into a single callback, so triggering multiple callbacks with different pointers will certainly lose some of them.

If you have chunks of data which you want to post to the message thread for processing asynchronously, put them in some kind of circular buffer and use an AsyncUpdater to wake up the message thread when stuff arrives.

6

Thanks for the heads up Jules. A couple quick questions as I try to synthesize your suggestion and the JUCE classes:

  1. If I have multiple instances of a class that’s a Changebroadcaster and other instances of classes which are ChangeListeners, the ChangeListener classes will ONLY receive the broadcasts being sent by the Changebroadcaster instances they are registered to, correct? Or am I missing something here?

  2. I’m trying to understand the problem you’re foreseeing as a result of multiple ChangeListener requests get coalesced into a single callback-- if my SerialDeviceManager has a callback which is constantly polling incoming serial data, and then a few registered ChangeListeners to be notified/sent the new data (all are sent/passed the same pointer), is this a problem? I tried testing this, putting a DBG print in my SerialDeviceManagers callback right after it receives the new serial data, but before it sends the data to the registered ChangeListeners, and then in the ChangeListener, I put another print DBG, and it seems to be working okay-- the DBG message in the serial callback prints and then immediately after, the ChangeListeners DBG prints confirming it received the message and data.

I was going to try and change things per your suggestion, but I didn’t understand what you meant by using “an AsyncUpdater to wake up the message thread when stuff arrives”… hmm sorry, a lot to bite off and chew!

7

You’re getting a little over-confused there about changelisteners. Yes, of course only those that you register will get a callback, but if you call “sendChangeMessage (1); sendChangeMessage (2)”, then only one (or possibly neither) of those numbers will arrive at your callback because only a single callback will happen. Read the docs, I’m sure it explains the coalescing idea. If you want both 1 and 2 to arrive, you’d need to send them as Message objects to a MessageListener.

An AsyncUpdater class just does a similar thing to a changelistener, with less overhead. Search for it and you’ll find many examples.

8

Thanks Jules. Got side-tracked with some other work but finally am back focusing on this project. The goal is to finish by the end of the week! I am trying to restructure the code from using ChangeBroadcasters/ChangeListeners to Messages, per your suggestion, but seem to be getting a little confused with using Messages for broadcasting / sending data (Floats or Strings) to registered listeners. Essentially, my SerialManager class connects to a serial device (e.g. Arduino), it receives data form the device, and then needs to broadcast the data to SensorRecorders which I have recording the incoming data as .wav files.

Are there any examples you can point me to which use Messages to broadcast data while I try to work through this? I’m getting a bit confused trying to pack the Messages as it looks like they are made to take int,int,int,pointers? I’ll keep plugging away meanwhile… thanks!

9

You can derive your own class from Message, and post it, so it can contain anything you want. The base class just provides a few ints and things as a convenient way of sending simple messages without writing a custom class.

1 Like
10

@Jules - thanks again for all your help, I have made a lot of progress in the last 2 days, and am feeling so much more comfortable working through the Juce docs, figuring things out myself :wink: and working in C++ in general. I have a question about how “kosher” something I am doing is-- I suspect it may not be very safe. Perhaps you (or another community member) can shed some light?

Most of the way the recording works is pretty identical to the AudioRecording demo you provide in the Juce examples. When looking at the output .wav files, I noticed that each recording (if I recorded from 2 or more inputs) was slightly longer in length. I loaded the files up in Ableton Live, zoomed in to the sample-level, and the actual data itself seems to be completely synchronous, which is great. The problem is, each subsequent recording was slightly longer (meaning, if I record 2 mono audio channels, and a sensor stream, the 2nd audio channel’s wav file is longer than the others, and the sensor stream’s wav file is slightly longer than the 2nd audio channels file…etc).

Trying to find the culprit, I played around and figured out that it has to do (or at least appears to) with the way the threads exit. Per the audio recording example, when stopping, I was calling stopThread(5000)

void AudioRecorder::stop(){ recording = false; stopThread (5000); }
Changing this to stopThread(0) fixes the issue, but as mentioned in the docs, this could cause problems were the thread still has locks or needs to complete other jobs. Setting the wait time to something short, even 1ms, still causes deviation in the outputted file lengths.

Is it super-dangerous for me to set the time to 0ms? Am I better off keeping it at something like 5000ms and then finding the difference between outputted file lengths and the shortest output file length, and then trimming the longer ones down?

My knowledge of Threading and whatnot is still in its infancy, so any help is much appreciated. Thanks!

11

Yes.

Forcing a thread to quit is a serious error. It could leak, it could leave corrupted memory on the heap, it could leave signals and locks in an indeterminate state, it could leave files open. Etc etc etc. It’s a very very bad thing to do. Don’t do it.

Presumably your file writing code is just exiting its thread in between writing channels 0 and 1, so just fix that instead.

12

Eek, that doesn’t sound like fun times :slight_smile: …from looking at your notes in the Thread class it seemed like that wasn’t safe. I am definitely going to try and resolve this…

As I try to synthesize your suggestion, are you saying that that each recorders writing code needs to wait for the other recorders to finish before finally exiting all of their threads? I’m trying to think of how to go about doing this (as my knowledge on threading is very basic), but don’t yet see the way. Any tips? My recorder classes are set up very closely to the AudioRecorderDemoPage example (just a few small changes to circularbuffer and whatnot to handle single channels + the data stream from the sensors, and other things, but nothing that would effect this)… thanks, I’ll keep taking a stab at it meanwhile…

Best,
J

13

@Jules & friends -

Still have to figure out your suggestion in regards to the threading/writing implementation, but thought I’d share a video I shot tonight of the application.

http://vimeo.com/13118225

I spent the last couple days working through a few other bugs, and implementing some other functionality, and most of the core functionality is now in there… I wanted to say thanks again for all your help, which without, this application (and not to mention my c++ learning) would not have taken shape nearly as fast or (relatively) as smoothly.

Cheers!

14

Cool, well done!

15

[quote=“jules”]
Forcing a thread to quit is a serious error. It could leak, it could leave corrupted memory on the heap, it could leave signals and locks in an indeterminate state, it could leave files open. Etc etc etc. It’s a very very bad thing to do. Don’t do it.

Presumably your file writing code is just exiting its thread in between writing channels 0 and 1, so just fix that instead.[/quote]

Been working on the app here and there the last couple months and have pretty much finished everything! I had gotten stuck here and spent the day today trying to solve this problem, and made the following observations.

As I am only writing mono channels / files, I don’t think it has to do with exiting the thread in between writing channels 0 and 1. That being said, upon further testing today, using stopThread(5000) in my recorders almost always results in multiple recorders .wav files being of slightly different lengths. However, if I use signalThreadShouldExit() instead of stopThread, they are only sometimes off-- or most of the recorders will be fine and only one or so will be off. The results were different each time (sometimes working fine) so I have no clue whats causing the inconsistency.

Does this clue you into any other possible scenarios? Could it have something to do with the fact that I also have some gui stuff ( paint() ) in the same class? The .wav writing is implemented as in the Juce demo, including the recorder classes thread starting when “record” is enabled" and ending when disabled…etc. Thanks for your help.

16

I had to do something similar recently, write multiple files at once.

Obviously there’s no chance whatsoever that you can do this reliably by just starting and stopping all the threads and hoping that they’re in sync, so what I did was to have a “shouldRecord” boolean flag that all the threads checked in their audio callback. Then by locking the AudioDeviceManger’s callback lock and changing that flag, you can make sure they all start and stop at the same time.

17

I had tried implemented using a flag like that earlier today, but was missing the callback lock. Do you have any other quick insight into locking the AudioDeviceManagers callback lock-- this is the first program I’ve started to use threading, and have only done a little bit of ScopedLock, etc (although, I spent a few hours today reading the forum and wiki on lock, thread safety, atomics, etc).

I guess im just a little unsure as to where to start with that suggestion as I don’t see anything about locks in the AudioDeviceManagers/AudioIODeviceCallback classes, and I’m assuming that means I need to insert a lock/unlock into my recorder classes AudioIODeviceCallback? Or perhaps it needs to happen on my AudioDeviceManager somehow (the device manager seems a little bit more like a black box to me since I basically create it, set it, and forget it after registering recorders/objects with it)…Really appreciate your help!

[code]void AudioRecorder::audioDeviceIOCallback (const float** inputChannelData, int numInputChannels,
float** outputChannelData, int numOutputChannels,
int numSamples)
{
numSamplesIn = numSamples;
if (recording)
{
const AudioSampleBuffer incomingData ((float**)inputChannelData, numInputChannels, numSamples);
circularBuffer->addSamplesToBuffer (incomingData, numSamples, channel);
}

// We need to clear the output buffers, in case they're full of junk..
for (int i = 0; i < numOutputChannels; ++i)
    if (outputChannelData[i] != 0)
        zeromem (outputChannelData[i], sizeof (float) * numSamples);

}[/code]

18

I added the lock accessor method quite recently, so you might need to get the latest version to use it. Then it’d just be

{ ScopedLock sl (myAudioDeviceManager.getAudioCallbackLock()); myStartStopFlag = false; }

19

Thanks Jules, I think I’m pretty close but I think my flag my not be setting correctly because sometimes they still record with different lengths-- perhaps you can spot what I’m doing incorrectly? Below is a condensed version of how I have it setup. Thanks!

In my main program/gui, I have my flag (bool finishWritingToDisk) which I pass into my AudioRecorder objects at time of their creation (through a set method). The AudioRecorder’s setter then points a local bool* (finishWriting) to finishWritingToDisk.

Then in my AudioRecorders callback:

void AudioRecorder::audioDeviceIOCallback (const float** inputChannelData, int numInputChannels, float** outputChannelData, int numOutputChannels, int numSamples)
{
	if (*finishWriting == false){
		numSamplesIn = numSamples;
		if (recording)
		{
			const AudioSampleBuffer incomingData ((float**)inputChannelData, numInputChannels, numSamples);
			circularBuffer->addSamplesToBuffer (incomingData, numSamples, channel);		
		}
	}
	//...I also do other things like clear output buffers, etc
}

Lastly, in MainComponent, when we should stop recording I called my device managers lock accessor and set the flag

void MainComponent::buttonClicked (Button* buttonThatWasClicked){
    //...some gui testing, etc, and then finally

		for(int i=0; i< audioRecorderManager.size(); i++)
		{
			//check to see if the recorder is currently recording, if so, call our lock on our audio callback, set our flag,  and call stop()  to tell the thread to exit and do some cleanup
			if(audioRecorderManager[i]->isRecording()){
				ScopedLock sl (audioDeviceManager.getAudioCallbackLock() );
				finishWritingToDisk = true;
				audioRecorderManager[i]->setShouldFinishWriting(finishWritingToDisk);
				DBG("isrecording");
				audioRecorderManager[i]->stop();
				counter++;
			}else{
				//...other stuff
			}
20

If it’s a shared flag, why are you setting it multiple times?