I’m working on a 3D application and I have to read big (50 MB) text files containing 3D model data. As the file format is line oriented I tried to do it the clean way (the juce way) and use InputStream::readNextLine(). It turns out to be very slow.
The profiler revealed that the program spends its time in String::operator+=, in fact 10 times more than in any other relevant functions, including the actual I/O operations.
As the lines in my file type are generally longer than 32 characters, I tried to recompile juce with s.preallocateStorage(256) instead of the original s.preallocateStorage(32) in InputStream::readNextLine, but it only gave a marginal improvement.
I think that inside InputStream::readNextLine the code should not use a juce String to accumulate the characters. Reference counting, nice allocation growing and in general “juceness” are not important here, the chosen simple C data type could be converted to a juce String at the end.
Jules, what do you think? If it is not possible in juce I will have to fall back to file block reading and good old character pointers …