Crash in ZipFile when trying to open a large ~6Gb file

Hey JUCE team. I’ve ran into a crash trying to open a ZipFile using a ~6Gb zipfile that was created with JUCE ZipBuilder.

auto selectedFile = fileChooser.getResult();
ZipFile packageZipFile(selectedFile);

In ZipFile::Init()

||centralDirectoryPos|1673525341|__int64|

this seems to return bad data:

 auto* buffer = static_cast<const char*> (headerData.getData()) + pos;
 auto fileNameLen = readUnalignedLittleEndianShort (buffer + 28);

||fileNameLen|37188|unsigned short|
|▶|buffer|0x000002adf1ffd070 “ˆ$\x1fÑJçn]šÈLxãžÊåÅ¥\vàË™BUçÒš¬D‘–6:¬S\x1dn¸úܶk\x1bwqÛ•¸«\x13×¢ôpZ9±ŸÊt¨\xe°µ¬qaÕTC\x16ýq\v,Riu–!\x1d\x1e®ÎŽ)«.ÃÖŏ\xe„Gi8¢$,´µVòV\x4K#¡‰”¼”|const char *|

Which causes this to assert on invalid string:

entries.add (new ZipEntryHolder (buffer, fileNameLen));

Any thoughts?

Thanks E

I’ve ran into issues with the zip classes on files over 2GB

Only suggestion i could think is to try smaller zips, as weak as that sounds : /

I really do need to be able to open large zip files. JUCE created the file and I can unzip it with a “normal” unzipper program but I need my app to be able to load large zips.

The original zip file format only supports files up to 4 GB and 65,535 files. It looks like JUCE creates corrupt zip files if you create them too big, as it casts the 64 bit values to 32 bit and writes them to the file. I suppose this is fairly common and most zip programs can work around it, if you can extract your files.

There is a newer zip64 format that juce does not support which does allow for files over 4 GB and total archive size over 4 GB.

If you need to support zip files over 4 GB, look at using libzip https://libzip.org/

1 Like

I would love a reply from the JUCE team about this. It does seem that the ZipBuilder creates the zip file correctly as I can use Total Commander to unzip the file without issues. I could look into libzip as a replacement for the JUCE ZipBuilder but would prefer not adding any more external dependencies.

E

I managed to fix the ZipFile class to support Zip64.

Just add the following code to the end of ZipEntryHolder constructor.

auto const extraFieldLength = ByteOrder::littleEndianShort (buffer + 30);
if (extraFieldLength != 0)
{
    auto* const extraHeaderPtr = buffer + 46 + fileNameLen;
    auto const extraHeaderTag = ByteOrder::littleEndianShort (extraHeaderPtr);
    if (extraHeaderTag == 0x0001) // Zip64 Tag
    {
        auto const extraFieldBLockLength = ByteOrder::littleEndianShort (extraHeaderPtr + 2);
        entry.uncompressedSize = (int64) ByteOrder::littleEndianInt64 (extraHeaderPtr + 4);
        if (extraFieldBLockLength > 8)
            compressedSize = (int64) ByteOrder::littleEndianInt64 (extraHeaderPtr + 12);
        if (extraFieldBLockLength > 16)
            streamOffset = (int64) ByteOrder::littleEndianInt64 (extraHeaderPtr + 20);
    }
}
1 Like

Hey there @flyingrub
Thanks for sharing your findings. I’ve tried adding the lines in juce_ZipFile.cpp on line 54, but I still hit a JUCE Assertion failure in juce_String.cpp:2143

{
    if (buffer != nullptr)
    {
        if (bufferSizeBytes < 0)
            return String (CharPointer_UTF8 (buffer));

        if (bufferSizeBytes > 0)
        {
            jassert (CharPointer_UTF8::isValidString ***DIES HERE

How are other people getting this to work?

Cheers
Jeff

EDIT: Silly me, I thought the modification by @flyingrub would allow for reading of large 64bit zips as well. It seems that this might not be the intention or the case. Sorry for the misunderstanding.

We ended up doing more change to the file to handle the ZIP64 Central Directory too.
Here is the full file. I hope this will fix it for your too. :slight_smile:

PS : this modifications are indeed for reading Zip64 files.

/*
  ==============================================================================

   This file is part of the JUCE library.
   Copyright (c) 2017 - ROLI Ltd.

   JUCE is an open source library subject to commercial or open-source
   licensing.

   The code included in this file is provided under the terms of the ISC license
   http://www.isc.org/downloads/software-support-policy/isc-license. Permission
   To use, copy, modify, and/or distribute this software for any purpose with or
   without fee is hereby granted provided that the above copyright notice and
   this permission notice appear in all copies.

   JUCE IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER
   EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE
   DISCLAIMED.

  ==============================================================================
*/

namespace juce
{

inline uint16 readUnalignedLittleEndianShort (const void* buffer)
{
    auto data = readUnaligned<uint16> (buffer);
    return ByteOrder::littleEndianShort (&data);
}

inline uint32 readUnalignedLittleEndianInt (const void* buffer)
{
    auto data = readUnaligned<uint32> (buffer);
    return ByteOrder::littleEndianInt (&data);
}

struct ZipFile::ZipEntryHolder
{
    ZipEntryHolder (const char* buffer, int fileNameLen)
    {
        isCompressed           = readUnalignedLittleEndianShort (buffer + 10) != 0;
        entry.fileTime         = parseFileTime (readUnalignedLittleEndianShort (buffer + 12),
                                                readUnalignedLittleEndianShort (buffer + 14));
        compressedSize         = (int64) readUnalignedLittleEndianInt (buffer + 20);
        entry.uncompressedSize = (int64) readUnalignedLittleEndianInt (buffer + 24);
        streamOffset           = (int64) readUnalignedLittleEndianInt (buffer + 42);

        entry.externalFileAttributes = readUnalignedLittleEndianInt (buffer + 38);
        auto fileType = (entry.externalFileAttributes >> 28) & 0xf;
        entry.isSymbolicLink = (fileType == 0xA);

        entry.filename = String::fromUTF8 (buffer + 46, fileNameLen);
        
		//BEGIN_MODIFIED_BY_RESOLUME
		auto const extraFieldLength = ByteOrder::littleEndianShort(buffer + 30);
		if (extraFieldLength != 0)
		{
            auto extraFieldOffset = 0;
            auto* extraHeaderPtr = buffer + 46 + fileNameLen;
            auto* extraSectionPtr = extraHeaderPtr;
			auto extraHeaderTag = ByteOrder::littleEndianShort(extraSectionPtr);
            auto extraFieldSectionLength = ByteOrder::littleEndianShort(extraSectionPtr + 2);
            while (extraFieldOffset < extraFieldLength) {
			    if (extraHeaderTag == 0x0001) // Zip64 Tag
			    {
				    entry.uncompressedSize = (int64)ByteOrder::littleEndianInt64(extraSectionPtr + 4);
                    if (extraFieldSectionLength > 8)
					    compressedSize = (int64)ByteOrder::littleEndianInt64(extraHeaderPtr + 12);
				    if (extraFieldSectionLength > 16)
					    streamOffset = (int64)ByteOrder::littleEndianInt64(extraHeaderPtr + 20);
			    }
                extraFieldOffset += extraFieldSectionLength + 4;
                extraSectionPtr = extraHeaderPtr + extraFieldOffset;
                extraHeaderTag = ByteOrder::littleEndianShort(extraSectionPtr);
                extraFieldSectionLength = ByteOrder::littleEndianShort(extraSectionPtr + 2);
            }
		}
		//END_MODIFIED_BY_RESOLUME
    }

    static Time parseFileTime (uint32 time, uint32 date) noexcept
    {
        auto year      = (int) (1980 + (date >> 9));
        auto month     = (int) (((date >> 5) & 15) - 1);
        auto day       = (int) (date & 31);
        auto hours     = (int) time >> 11;
        auto minutes   = (int) ((time >> 5) & 63);
        auto seconds   = (int) ((time & 31) << 1);

        return { year, month, day, hours, minutes, seconds };
    }

    ZipEntry entry;
    int64 streamOffset, compressedSize;
    bool isCompressed;
};

//BEGIN_MODIFIED_BY_RESOLUME
struct CentralDirectory
{
    int64 startOffset;
    size_t size;
    int numEntries;
    bool isValid = false;
};

//==============================================================================
static CentralDirectory findCentralDirectoryFileHeader (InputStream& input)
{
    CentralDirectory result;
    CentralDirectory resultZip64;
    BufferedInputStream in (input, 8192);

    in.setPosition (in.getTotalLength());
    auto pos = in.getPosition();
    auto lowestPos = jmax ((int64) 0, pos - 1048576);
    char buffer[64] = {};

    while (pos > lowestPos && (!result.isValid && !resultZip64.isValid))
    {
        in.setPosition (pos - 22);
        pos = in.getPosition();
        memcpy (buffer + 22, buffer, 4);

        if (in.read (buffer, 22) != 22)
            return {};

        for (int i = 0; i < 22; ++i)
        {
            if (readUnalignedLittleEndianInt(buffer + i) == 0x07064b50) // Zip64 end of central directory locator Magic Number
            {
                in.setPosition(pos + i);
                in.read(buffer, 22);
                auto enOfCentralDirectoryPos = ByteOrder::littleEndianInt64(buffer + 8);
                in.setPosition(enOfCentralDirectoryPos);
                in.read(buffer, 64);
                if (readUnalignedLittleEndianInt(buffer) == 0x06064b50)
                {
                    resultZip64.numEntries = static_cast<int>(ByteOrder::littleEndianInt64(buffer + 24));
                    resultZip64.size = (size_t)ByteOrder::littleEndianInt64(buffer + 40);
                    resultZip64.isValid = true;
                    
                    auto offset = static_cast<int64>(ByteOrder::littleEndianInt64(buffer + 48));
                    if (offset >= 4)
                    {
                        in.setPosition(offset);

                        // This is a workaround for some zip files which seem to contain the
                        // wrong offset for the central directory - instead of including the
                        // header, they point to the byte immediately after it.
                        if (in.readInt() != 0x02014b50)
                        {
                            in.setPosition(offset - 4);
                            result.isValid = false;

                            if (in.readInt() == 0x02014b50)
                            {
                                offset -= 4;
                                result.isValid = true;
                            }
                        }
                    }
                    resultZip64.startOffset = offset;
                }
            }
            else if (readUnalignedLittleEndianInt (buffer + i) == 0x06054b50)
            {
                in.setPosition (pos + i);
                in.read (buffer, 22);
                result.numEntries = readUnalignedLittleEndianShort (buffer + 10);
                result.size = (size_t) readUnalignedLittleEndianInt(buffer + 12);
                result.isValid = true;

                auto offset = (int64) readUnalignedLittleEndianInt (buffer + 16);
                if (offset >= 4)
                {
                    in.setPosition (offset);

                    // This is a workaround for some zip files which seem to contain the
                    // wrong offset for the central directory - instead of including the
                    // header, they point to the byte immediately after it.
                    if (in.readInt() != 0x02014b50)
                    {
                        in.setPosition (offset - 4);
                        result.isValid = false;

                        if (in.readInt() == 0x02014b50)
                        {
                            offset -= 4;
                            result.isValid = true;
                        }
                    }
                }
                result.startOffset = offset;
            }
        }
    }

    return resultZip64.isValid ? resultZip64 : result;
}
//END_MODIFIED_BY_RESOLUME

//==============================================================================
struct ZipFile::ZipInputStream  : public InputStream
{
    ZipInputStream (ZipFile& zf, const ZipFile::ZipEntryHolder& zei)
        : file (zf),
          zipEntryHolder (zei),
          inputStream (zf.inputStream)
    {
        if (zf.inputSource != nullptr)
        {
            streamToDelete.reset (file.inputSource->createInputStream());
            inputStream = streamToDelete.get();
        }
        else
        {
           #if JUCE_DEBUG
            zf.streamCounter.numOpenStreams++;
           #endif
        }

        char buffer[30];

        if (inputStream != nullptr
             && inputStream->setPosition (zei.streamOffset)
             && inputStream->read (buffer, 30) == 30
             && ByteOrder::littleEndianInt (buffer) == 0x04034b50)
        {
            headerSize = 30 + ByteOrder::littleEndianShort (buffer + 26)
                            + ByteOrder::littleEndianShort (buffer + 28);
        }
    }

    ~ZipInputStream() override
    {
       #if JUCE_DEBUG
        if (inputStream != nullptr && inputStream == file.inputStream)
            file.streamCounter.numOpenStreams--;
       #endif
    }

    int64 getTotalLength() override
    {
        return zipEntryHolder.compressedSize;
    }

    int read (void* buffer, int howMany) override
    {
        if (headerSize <= 0)
            return 0;

        howMany = (int) jmin ((int64) howMany, zipEntryHolder.compressedSize - pos);

        if (inputStream == nullptr)
            return 0;

        int num;

        if (inputStream == file.inputStream)
        {
            const ScopedLock sl (file.lock);
            inputStream->setPosition (pos + zipEntryHolder.streamOffset + headerSize);
            num = inputStream->read (buffer, howMany);
        }
        else
        {
            inputStream->setPosition (pos + zipEntryHolder.streamOffset + headerSize);
            num = inputStream->read (buffer, howMany);
        }

        pos += num;
        return num;
    }

    bool isExhausted() override
    {
        return headerSize <= 0 || pos >= zipEntryHolder.compressedSize;
    }

    int64 getPosition() override
    {
        return pos;
    }

    bool setPosition (int64 newPos) override
    {
        pos = jlimit ((int64) 0, zipEntryHolder.compressedSize, newPos);
        return true;
    }

private:
    ZipFile& file;
    ZipEntryHolder zipEntryHolder;
    int64 pos = 0;
    int headerSize = 0;
    InputStream* inputStream;
    std::unique_ptr<InputStream> streamToDelete;

    JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (ZipInputStream)
};


//==============================================================================
ZipFile::ZipFile (InputStream* stream, bool deleteStreamWhenDestroyed)
   : inputStream (stream)
{
    if (deleteStreamWhenDestroyed)
        streamToDelete.reset (inputStream);

    init();
}

ZipFile::ZipFile (InputStream& stream)  : inputStream (&stream)
{
    init();
}

ZipFile::ZipFile (const File& file)  : inputSource (new FileInputSource (file))
{
    init();
}

ZipFile::ZipFile (InputSource* source)  : inputSource (source)
{
    init();
}

ZipFile::~ZipFile()
{
    entries.clear();
}

#if JUCE_DEBUG
ZipFile::OpenStreamCounter::~OpenStreamCounter()
{
    /* If you hit this assertion, it means you've created a stream to read one of the items in the
       zipfile, but you've forgotten to delete that stream object before deleting the file..
       Streams can't be kept open after the file is deleted because they need to share the input
       stream that is managed by the ZipFile object.
    */
    jassert (numOpenStreams == 0);
}
#endif

//==============================================================================
int ZipFile::getNumEntries() const noexcept
{
    return entries.size();
}

const ZipFile::ZipEntry* ZipFile::getEntry (const int index) const noexcept
{
    if (auto* zei = entries[index])
        return &(zei->entry);

    return nullptr;
}

int ZipFile::getIndexOfFileName (const String& fileName, bool ignoreCase) const noexcept
{
    for (int i = 0; i < entries.size(); ++i)
    {
        auto& entryFilename = entries.getUnchecked (i)->entry.filename;

        if (ignoreCase ? entryFilename.equalsIgnoreCase (fileName)
                       : entryFilename == fileName)
            return i;
    }

    return -1;
}

const ZipFile::ZipEntry* ZipFile::getEntry (const String& fileName, bool ignoreCase) const noexcept
{
    return getEntry (getIndexOfFileName (fileName, ignoreCase));
}

InputStream* ZipFile::createStreamForEntry (const int index)
{
    InputStream* stream = nullptr;

    if (auto* zei = entries[index])
    {
        stream = new ZipInputStream (*this, *zei);

        if (zei->isCompressed)
        {
            stream = new GZIPDecompressorInputStream (stream, true,
                                                      GZIPDecompressorInputStream::deflateFormat,
                                                      zei->entry.uncompressedSize);

            // (much faster to unzip in big blocks using a buffer..)
            stream = new BufferedInputStream (stream, 32768, true);
        }
    }

    return stream;
}

InputStream* ZipFile::createStreamForEntry (const ZipEntry& entry)
{
    for (int i = 0; i < entries.size(); ++i)
        if (&entries.getUnchecked (i)->entry == &entry)
            return createStreamForEntry (i);

    return nullptr;
}

void ZipFile::sortEntriesByFilename()
{
    std::sort (entries.begin(), entries.end(),
               [] (const ZipEntryHolder* e1, const ZipEntryHolder* e2) { return e1->entry.filename < e2->entry.filename; });
}

//BEGIN_MODIFIED_BY_RESOLUME
//==============================================================================
void ZipFile::init()
{
    std::unique_ptr<InputStream> toDelete;
    InputStream* in = inputStream;

    if (inputSource != nullptr)
    {
        in = inputSource->createInputStream();
        toDelete.reset (in);
    }

    if (in != nullptr)
    {
        auto centralDirectory = findCentralDirectoryFileHeader (*in);
        if (!centralDirectory.isValid) return;

        auto centralDirectoryPos = centralDirectory.startOffset;
        auto size = centralDirectory.size;
        auto numEntries = centralDirectory.numEntries;

        auto totalLength = in->getTotalLength();
        if (centralDirectoryPos < 0 || in->getTotalLength() < centralDirectoryPos + size) return;

        in->setPosition (centralDirectoryPos);
        MemoryBlock headerData;

        if (in->readIntoMemoryBlock (headerData, (ssize_t) size) == size)
        {
            size_t pos = 0;

            for (int i = 0; i < numEntries; ++i)
            {
                if (pos + 46 > size)
                    break;

                auto* buffer = static_cast<const char*> (headerData.getData()) + pos;
                auto fileNameLen = readUnalignedLittleEndianShort (buffer + 28u);

                if (pos + 46 + fileNameLen > size)
                    break;

                entries.add (new ZipEntryHolder (buffer, fileNameLen));

                pos += 46u + fileNameLen
                        + readUnalignedLittleEndianShort (buffer + 30u)
                        + readUnalignedLittleEndianShort (buffer + 32u);
            }
        }
    }
}
//END_MODIFIED_BY_RESOLUME

Result ZipFile::uncompressTo (const File& targetDirectory,
                              const bool shouldOverwriteFiles)
{
    for (int i = 0; i < entries.size(); ++i)
    {
        auto result = uncompressEntry (i, targetDirectory, shouldOverwriteFiles);

        if (result.failed())
            return result;
    }

    return Result::ok();
}

Result ZipFile::uncompressEntry (int index, const File& targetDirectory, bool shouldOverwriteFiles)
{
    auto* zei = entries.getUnchecked (index);

   #if JUCE_WINDOWS
    auto entryPath = zei->entry.filename;
   #else
    auto entryPath = zei->entry.filename.replaceCharacter ('\\', '/');
   #endif

    if (entryPath.isEmpty())
        return Result::ok();

    auto targetFile = targetDirectory.getChildFile (entryPath);

    if (entryPath.endsWithChar ('/') || entryPath.endsWithChar ('\\'))
        return targetFile.createDirectory(); // (entry is a directory, not a file)

    std::unique_ptr<InputStream> in (createStreamForEntry (index));

    if (in == nullptr)
        return Result::fail ("Failed to open the zip file for reading");

    if (targetFile.exists())
    {
        if (! shouldOverwriteFiles)
            return Result::ok();

        if (! targetFile.deleteFile())
            return Result::fail ("Failed to write to target file: " + targetFile.getFullPathName());
    }

    if (! targetFile.getParentDirectory().createDirectory())
        return Result::fail ("Failed to create target folder: " + targetFile.getParentDirectory().getFullPathName());

    if (zei->entry.isSymbolicLink)
    {
        String originalFilePath (in->readEntireStreamAsString()
                                    .replaceCharacter (L'/', File::getSeparatorChar()));

        if (! File::createSymbolicLink (targetFile, originalFilePath, true))
            return Result::fail ("Failed to create symbolic link: " + originalFilePath);
    }
    else
    {
        FileOutputStream out (targetFile);

        if (out.failedToOpen())
            return Result::fail ("Failed to write to target file: " + targetFile.getFullPathName());

        out << *in;
    }

    targetFile.setCreationTime (zei->entry.fileTime);
    targetFile.setLastModificationTime (zei->entry.fileTime);
    targetFile.setLastAccessTime (zei->entry.fileTime);

    return Result::ok();
}


//==============================================================================
struct ZipFile::Builder::Item
{
    Item (const File& f, InputStream* s, int compression, const String& storedPath, Time time)
        : file (f), stream (s), storedPathname (storedPath), fileTime (time), compressionLevel (compression)
    {
        symbolicLink = (file.exists() && file.isSymbolicLink());
    }

    bool writeData (OutputStream& target, const int64 overallStartPosition)
    {
        MemoryOutputStream compressedData ((size_t) file.getSize());

        if (symbolicLink)
        {
            auto relativePath = file.getNativeLinkedTarget().replaceCharacter (File::getSeparatorChar(), L'/');

            uncompressedSize = relativePath.length();

            checksum = zlibNamespace::crc32 (0, (uint8_t*) relativePath.toRawUTF8(), (unsigned int) uncompressedSize);
            compressedData << relativePath;
        }
        else if (compressionLevel > 0)
        {
            GZIPCompressorOutputStream compressor (compressedData, compressionLevel,
                                                   GZIPCompressorOutputStream::windowBitsRaw);
            if (! writeSource (compressor))
                return false;
        }
        else
        {
            if (! writeSource (compressedData))
                return false;
        }

        compressedSize = (int64) compressedData.getDataSize();
        headerStart = target.getPosition() - overallStartPosition;

        target.writeInt (0x04034b50);
        writeFlagsAndSizes (target);
        target << storedPathname
               << compressedData;

        return true;
    }

    bool writeDirectoryEntry (OutputStream& target)
    {
        target.writeInt (0x02014b50);
        target.writeShort (symbolicLink ? 0x0314 : 0x0014);
        writeFlagsAndSizes (target);
        target.writeShort (0); // comment length
        target.writeShort (0); // start disk num
        target.writeShort (0); // internal attributes
        target.writeInt ((int) (symbolicLink ? 0xA1ED0000 : 0)); // external attributes
        target.writeInt ((int) (uint32) headerStart);
        target << storedPathname;

        return true;
    }

private:
    const File file;
    std::unique_ptr<InputStream> stream;
    String storedPathname;
    Time fileTime;
    int64 compressedSize = 0, uncompressedSize = 0, headerStart = 0;
    int compressionLevel = 0;
    unsigned long checksum = 0;
    bool symbolicLink = false;

    static void writeTimeAndDate (OutputStream& target, Time t)
    {
        target.writeShort ((short) (t.getSeconds() + (t.getMinutes() << 5) + (t.getHours() << 11)));
        target.writeShort ((short) (t.getDayOfMonth() + ((t.getMonth() + 1) << 5) + ((t.getYear() - 1980) << 9)));
    }

    bool writeSource (OutputStream& target)
    {
        if (stream == nullptr)
        {
            stream.reset (file.createInputStream());

            if (stream == nullptr)
                return false;
        }

        checksum = 0;
        uncompressedSize = 0;
        const int bufferSize = 4096;
        HeapBlock<unsigned char> buffer (bufferSize);

        while (! stream->isExhausted())
        {
            auto bytesRead = stream->read (buffer, bufferSize);

            if (bytesRead < 0)
                return false;

            checksum = zlibNamespace::crc32 (checksum, buffer, (unsigned int) bytesRead);
            target.write (buffer, (size_t) bytesRead);
            uncompressedSize += bytesRead;
        }

        stream.reset();
        return true;
    }

    void writeFlagsAndSizes (OutputStream& target) const
    {
        target.writeShort (10); // version needed
        target.writeShort ((short) (1 << 11)); // this flag indicates UTF-8 filename encoding
        target.writeShort ((! symbolicLink && compressionLevel > 0) ? (short) 8 : (short) 0); //symlink target path is not compressed
        writeTimeAndDate (target, fileTime);
        target.writeInt ((int) checksum);
        target.writeInt ((int) (uint32) compressedSize);
        target.writeInt ((int) (uint32) uncompressedSize);
        target.writeShort ((short) storedPathname.toUTF8().sizeInBytes() - 1);
        target.writeShort (0); // extra field length
    }

    JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (Item)
};

//==============================================================================
ZipFile::Builder::Builder() {}
ZipFile::Builder::~Builder() {}

void ZipFile::Builder::addFile (const File& file, int compression, const String& path)
{
    items.add (new Item (file, nullptr, compression,
                         path.isEmpty() ? file.getFileName() : path,
                         file.getLastModificationTime()));
}

void ZipFile::Builder::addEntry (InputStream* stream, int compression, const String& path, Time time)
{
    jassert (stream != nullptr); // must not be null!
    jassert (path.isNotEmpty());
    items.add (new Item ({}, stream, compression, path, time));
}

bool ZipFile::Builder::writeToStream (OutputStream& target, double* const progress) const
{
    auto fileStart = target.getPosition();

    for (int i = 0; i < items.size(); ++i)
    {
        if (progress != nullptr)
            *progress = (i + 0.5) / items.size();

        if (! items.getUnchecked (i)->writeData (target, fileStart))
            return false;
    }

    auto directoryStart = target.getPosition();

    for (auto* item : items)
        if (! item->writeDirectoryEntry (target))
            return false;

    auto directoryEnd = target.getPosition();

    target.writeInt (0x06054b50);
    target.writeShort (0);
    target.writeShort (0);
    target.writeShort ((short) items.size());
    target.writeShort ((short) items.size());
    target.writeInt ((int) (directoryEnd - directoryStart));
    target.writeInt ((int) (directoryStart - fileStart));
    target.writeShort (0);

    if (progress != nullptr)
        *progress = 1.0;

    return true;
}


//==============================================================================
//==============================================================================
#if JUCE_UNIT_TESTS

struct ZIPTests   : public UnitTest
{
    ZIPTests()
        : UnitTest ("ZIP", UnitTestCategories::compression)
    {}

    void runTest() override
    {
        beginTest ("ZIP");

        ZipFile::Builder builder;
        StringArray entryNames { "first", "second", "third" };
        HashMap<String, MemoryBlock> blocks;

        for (auto& entryName : entryNames)
        {
            auto& block = blocks.getReference (entryName);
            MemoryOutputStream mo (block, false);
            mo << entryName;
            mo.flush();
            builder.addEntry (new MemoryInputStream (block, false), 9, entryName, Time::getCurrentTime());
        }

        MemoryBlock data;
        MemoryOutputStream mo (data, false);
        builder.writeToStream (mo, nullptr);
        MemoryInputStream mi (data, false);

        ZipFile zip (mi);

        expectEquals (zip.getNumEntries(), entryNames.size());

        for (auto& entryName : entryNames)
        {
            auto* entry = zip.getEntry (entryName);
            std::unique_ptr<InputStream> input (zip.createStreamForEntry (*entry));
            expectEquals (input->readEntireStreamAsString(), entryName);
        }
    }
};

static ZIPTests zipTests;

#endif

} // namespace juce
1 Like

Hey there @flyingrub

Thanks so much for sharing this! After dropping into changes into a project-local copy of juce_ZipFile.cpp it compiled easily, which was very nice to see. Test unzipping seemed to go well too.

Things got weird when looking through after it threw no errors whilst decompressing a 6.03GB zipfile. A few png images (in the zip file) that were written out in the process had 0bytes. If I use 7z to do the same decompression I have no such issue.

Do you have any idea why this might be happening?

Cheers,
Jeff