Any examples of MemoryMappedAudioFormatReader's usage?


#1

Hello,

I would like to preload the first second (most likely even less) of all samples of an instrument into memory.

I am looking into MemoryMappedAudioFormatReader and I've tried the mapSectionOfFile method.

I've replaced the default AudioFormatReader in loading the samples:

Range<int64> samplesToMap = Range<int64>( 0, 44100 );


reader->mapSectionOfFile( samplesToMap );

But I get an assertion error here:

//JUCE Assertion failure in juce_WavAudioFormat.cpp:1484

MemoryMappedWavReader::readSamples()

{

...

        if (map == nullptr || ! mappedSection.contains (Range<int64> (startSampleInFile, startSampleInFile + numSamples)))

        {

            jassertfalse; // you must make sure that the window contains all the samples you're going to attempt to read.

            return false;

        }

...

}

Can someone give an example, how can the memory-mapped reader be used? Thanks.


#2

http://www.juce.com/forum/topic/disk-streaming-sampler-classes


#3

I'm reading the code now.

Just a side-question: Do you happen to know if there is a project upgrade procedure I can follow?

Pretty much every example I find older than 6 months doesn't compile and is full of errors.

It would be awesome if the Projucer would upgrade projects the way Xcode and Visual Studio do.


#4

The introjucer/projucer file format hasn't changed for years, it should load old projects just fine.

If you're talking about upgrading the code itself to handle changes to the library, that's a harder question, and not something that Xcode or anyone else has ever been crazy enough to attempt!


#5

You could always travel back in time and get the JUCE version matching to the last commit date (in this case it would be something like 3.0)

IIRC there were some changes to the synthesiser base class some time ago that broke backward compatibility.


#6

Idea: can the projucer file contain info about the JUCE library version and pull the used version into the project?

It can be based on official releases (tar's of the released versions held somewhere), or commit number.

I apologize if I am not seeing something obviously-impossible/impractical here, in advance.


#7

Can I ask something else? Was there a SamplerSound class when you wrote this? It's totally OK if you don't remember and I can definitely check that on my own.

My guess is that either the class wasn't built then, so you inherited SynthesiserSound directly, or there is a problem implementing it with inheritor of SamplerSound - if you remember that this is, indeed, the case, I am interested why not inherit SamplerSound?


#8

Not really... 

What most people do is to have juce as a GIT submodule inside their own project, so GIT takes care of keeping things in sync, while giving you control over when you want to update the library.


#9

The sampler class did exist back then, but it is not intended to be subclassed so I could either hack around in the JUCE code to expose private members or copy some of the code into an own class so I went with the second option.


#10

I might be missing something conceptual, here.

If you map all the sample files (with the mapEntireFile method) doesn't this mean that the same space will be taken in-memory? They will not be copied and streaming will be faster, but it will take the same amount of memory, right?

Doesn't this, kind-of, beat the purpose of streaming? Isn't the speed of the default AudioFormatReader sufficient when buffering samples (and an appropriate buffer size is chosen)?

Yesterday I read a paper by someone on the Native Instruments team, explaining DFD. Maybe because it was a higher level document, it didn't say too much specifics from the implementation - memory-mapped files weren't mentioned.
And they quote very low numbers in memory consumption (tens of MB's) for loading instruments with GB's of samples and mention that through DFD they solve the issue of 4 GB of address space on the 32bit systems, too, which makes me think - they can't be using memory-mapped files.


#11

Nope its virtual memory. Basically the OS acts as if this file were in the memory but the access is still a file read and the actual RAM is left untouched (if the OS doesn't decide that caching it is faster).

RE 32bit I need to test this again but I remember that loading samples > 4GB was possible in a x86 build.


#12

Whoops, you're right, it doesn't work on x86 (it doesn't exceed the memory usage, but the address space is still 4GB only - weird that I didn't think of this).

I guess I'll need a standard fallback reader for the people still rolling on 32bit hardware.


#13

We had to do this in Tracktion - on 64-bit systems we just memory-map everything, but in 32-bit there's a horrible system where it maps chunks of files and moves these in and out of memory as you play.


Best way to read an audio file to memory
#14

Update: I apologize - I saw the 2 replies only after submitting this.

I see, so mapEntireFile will not take any of the addresses used for the physical memory. The confusion came from my own tests, in which I used SamplerSound with MemoryMappedAudioFormatReader and I observed no change in the memory taken, so naturally I asumed that the memory used will be the same reported by the OS, but there is still the benefit of higher speed of reading the file. And the reason was because SamplerSound will preload the whole sample, no matter the type of a reader. Now I understand why there was this assertion error when I used the mapSectionOfFile method - SamplerSound makes sure it has laoded the full sample file, because this is the intended behavior by design.

This still shouldn't be possible for files > 4GB on 32bit system - there aren't enough addresses. And coming to think of it... I apologize if this is a stupid question - if a memory-mapped file is accessible through the virtual address space allocated for the process doing the mapping, doesn't this mean that those addresses are practically "stolen"? Maybe on 64bit systems this is somehow implemented by getting address beyond the system available RAM, but on 32bit, where often the number of possible addresses equals the number of actually available addresses, how does that work? After a little search I found this which points in the same direction: http://blog.codinghorror.com/dude-wheres-my-4-gigabytes-of-ram/


#15

Is mapping each chunk + reading it still faster than just using a normal AudioFormatReader?


#16

Yes. They're big chunks.


#17

Then I'll have some chunk-size finding to do :)