MemoryOutputStream::writeFromInputStream() keeps allocating memory on Windows, making it really slow

Windows
1

On Windows, downloading a file is very slow compared to the same code on Mac (10x). I noticed that even if I preallocate the full size, it just keeps allocating more memory here in modules\juce_core\streams\juce_MemoryOutputStream.cpp

int64 MemoryOutputStream::writeFromInputStream (InputStream& source, int64 maxNumBytesToWrite)
{
    // before writing from an input, see if we can preallocate to make it more efficient..
    int64 availableData = source.getTotalLength() - source.getPosition();

    if (availableData > 0)
    {
        if (maxNumBytesToWrite > availableData || maxNumBytesToWrite < 0)
            maxNumBytesToWrite = availableData;

        if (blockToUse != nullptr)
            preallocate (blockToUse->getSize() + (size_t) maxNumBytesToWrite);
    }

    return OutputStream::writeFromInputStream (source, maxNumBytesToWrite);
}

If I remove all the preallocation code from writeFromInputStream(), so it just becomes:

int64 MemoryOutputStream::writeFromInputStream (InputStream& source, int64 maxNumBytesToWrite)
{
    return OutputStream::writeFromInputStream (source, maxNumBytesToWrite);
}

Then it works fine and performance matches Mac version.

The relevant part of my code:

    auto content_length = response_headers.getValue("Content-Length", "");
    juce::int64 total_bytes = content_length.isNotEmpty() ? content_length.getLargeIntValue() : 50000000;
    auto total_size = juce::File::descriptionOfSizeInBytes(total_bytes);
    utils::log_info("Download size: " + total_size);

    double seconds {0.0};
    {
        juce::ScopedTimeMeasurement m(seconds);
        juce::MemoryOutputStream mo(dest, true);
        mo.preallocate(total_bytes);
        while (true) {
            if (threadShouldExit()) {
                return juce::Result::fail("Cancelled");
            }
            
            juce::int64 written = mo.writeFromInputStream(*inStream, 8192);
            if (written == 0) {
                break;
            }

I am on Windows 10 Pro 20H2 (19042.630) and Visual Studio Community 2019 Version 16.8.2

2

We saw the same issue. We referred to the Projucer’s AutoUpdater implementation, opening the url input stream, and then looping on MemoryOutputStream::writeFromInputStream. Very slow when downloading a large file in Windows!

We derived a class to preallocate and override writeFromInputStream to avoid the reallocation code:

class FastMemoryOutputStream : public MemoryOutputStream
{
public:
	FastMemoryOutputStream(MemoryBlock& memoryBlockToWriteTo, int64_t sizeBytes) : 
		MemoryOutputStream(memoryBlockToWriteTo, true)
	{
		preallocate(sizeBytes);
	};

	int64 writeFromInputStream(InputStream& source, int64 maxNumBytesToWrite) override final
	{
		return OutputStream::writeFromInputStream(source, maxNumBytesToWrite);
	}
};

Seems like this should be fixed in Juce!

-John

3

Thank you for reporting this. A fix is out on develop