Save std::map in ValueTree (e.g. plugin state with large data)

Hi all,

I’d like to load/save my loudness history in my plugin state. Depending on how long the captured timeline was (could possibly be one or more hours), the amount of data may be very large.

At the moment I’m writing the data in a std::map<int64, double> (Key = position in samples; Value = loudness at this position) where the loudness will be captured every 100ms, therefor quickly resulting in several thousands of records.

Now what would be the best approach on this?

Converting to/from a ValueTree would be easy, but seems to me a bit odd on that task.

any suggestions?

Thanks
Stefan

answering myself (maybe it is useful for somebody else):
I’ve built this with a ValueTree for testing. But wrapping every single measurement in a subtree with properties created too much overhead. A 60min measurement (2 x 36.000 subtrees with 2 properties each!!) eats up 6MB of data and blocked the message thread several seconds to store and recall those values.

The solution is to convert the std::map to a MemoryBlock and storing that into my ValueTree. You first have to convert the data into a std::vector because std::map cannot be stored to a MemoryBlock directly.
Thanks to @nonlethalapplications pointing me to the right direction!

template<class KeyType, class ValueType>
juce::MemoryBlock mapToMemoryBlock(const std::map<KeyType, ValueType>& map) const {
    
    std::vector<std::pair<KeyType, ValueType>> kvVector;
    
    for (const auto& element : map)
        kvVector.push_back(element);
    
    auto dataSize = kvVector.size() * sizeof(std::pair<KeyType, ValueType>);
    MemoryBlock memBlock;
    memBlock.insert(kvVector.data(), dataSize, 0);
    
    return memBlock;
    
}

template<class KeyType, class ValueType>
std::map<KeyType, ValueType> memoryBlockToMap(const juce::MemoryBlock* const mb) const {
    
    
    std::vector<std::pair<KeyType, ValueType>> kvVector;
    
    auto dataSize = mb->getSize();
    auto numElements = static_cast<int>(dataSize/sizeof(std::pair<KeyType, ValueType>));
    
    kvVector.resize(numElements);
    memcpy(kvVector.data(), mb->getData(), dataSize);
    
    std::map<KeyType, ValueType> map;
    
    for (auto& element : kvVector)
        map.insert(element);
    
    return map;
}


// ...

std::map<juce::int64, double> myMap;
// storing stuff in it ...
// ...

// saving std::map to ValueTree ...

const Identifier myIdentifier {"myIdentifier"};
ValueTree myTree;
auto memBlock = mapToMemoryBlock(myMap);
myTree.setProperty(myIdentifier, memBlock, nullptr);


// ...

// ... and back from ValueTree to std::map

auto memBlockPtr = myTree[myIdentifier].getBinaryData();
myMap = memoryBlockToMap<juce::int64, double>(memBlockPtr); 
// I got compiler errors when not specifying the template types here...

This works very well. The data size is now only about 1MB per hour and the conversions are instant.

cheers, Stefan

2 Likes