MemoryMappedAudioFormatReader and File descriptors


#1

I’m running into an issue where I need to access thousands upon thousands of audio samples, and I was hoping to get some advice.

Managing memory is obviously a huge problem, but I was hoping to work around it by using memory-mapped files to keep the overhead down. However, I also am limited in the quantity of file handles I am allowed to have open at a time so I need to be able to map the file to memory and then close the descriptor, which I believe is at least supported on Linux (http://stackoverflow.com/a/17490185) and may also be supported elsewhere.

So my question is: Am I reading the code correctly if I see that the MemoryMappedFile retains an open handle to its file descriptor after mapping? If so, is it possible to request a feature that exposes some way to close the open file handle, particularly from a MemorymappedAudioFormatReader?


#2

Well that’s right - MemoryMappedFile is basically an RAII holder for the file handle. I guess that if you need to limit the number of handles then you need to have a pool of MemoryMappedFiles that you create/delete as needed.


#3

That makes sense. I guess I assumed the Input and Output StreamReaders were the only RAII wrappers around file handles which lead me to expect MemoryMappedFile to be a wrapper around the memory-mapping specifically.

This does open up new questions, if you wouldn’t mind: Since the file descriptor isn’t exposed by MemoryMappedFile, and the only way to re-map a file is to create a new instance of MemoryMappedFile, Are there places where the handle is being used that would require it to be kept around? Are there reasons to hold onto that resource beyond the scope of the constructor or could it be safely closed immediately after mapping is successful without any side effects?


#4

Not sure I understand your question… obviously the mapping object needs to be kept around for as long as you need to actually read from it.


#5

Yes, the void pointer that is mmapped does, but my question is about the descriptor saved as the member fileHandle, not the mapped memory pointer address. I ask because:

/** From the relevant part of juce_posix_SharedCode.h's implementation **/
MemoryMappedReader::openInternal()
{
    fileHandle = open (file.getFullPathName().toUTF8(),
                           mode == readWrite ? (O_CREAT + O_RDWR) : O_RDONLY, 00644);

    if (fileHandle != -1)
    {
        void* m = mmap (0, (size_t) range.getLength(),
                    mode == readWrite ? (PROT_READ | PROT_WRITE) : PROT_READ,
                    exclusive ? MAP_PRIVATE : MAP_SHARED, fileHandle,
                    (off_t) range.getStart());

        /*** All of the other mmap-related calls ***/
    }

    /*** 
     * My addition: Now that the mapping (successful or not) is done, close the handle. 
     * Continue to rely on ~MemoryMappedFile to call munmap as per RAII.
    ***/
    if (fileHandle != 0)
        close (fileHandle);
}

would be valid because, according to the documentation of mmap according to both it’s man page and the POSIX docs, closing the descriptor does not close or invalidate the map. Obviously MemoryMappedFile needs to keep address around for access to the data: are there reasons to keep fileHandle around as well?


#6

Oh, I see what you mean. Yes, that could indeed save on a file handle if it works like that.

Presumably it wouldn’t be a magic solution for your use-case though, as there’d probably still be an upper limit on the number of memory-mapped files the OS can handle?


#7

Ah, it looks like it actually wouldn’t make any difference - the docs say this:

The mmap() function shall add an extra reference to the file associated with the file descriptor fildes which is not removed by a subsequent close() on that file descriptor. This reference shall be removed when there are no more mappings to the file.


#8

Ah, fair point, it can’t possibly be a complete solution to the problem.

I don’t know enough about how the system manages such things, does “extra reference” mean that I now have two handles around for the same file counting against the imposed limits? It would seem, given the way that blurb from the docs is written, like one could open twice as many files by closing descriptors after mapping, but I am under no illusion that I’m reading that correctly.