Stream read speed


#1

I am playing around creating an intermediate file format for a project idea but I am having problem with load speed. I have done some profiling and the problem seems to be iterative reads from stream->read{DataType}();

while(!stream->isExhausted())
{
    float amplitude = stream->readFloat();
    long begin = stream->readInt64();
    long end = stream->readInt64();
    short cIdx = stream->readShort();
    short dIdx = stream->readShort();
    long sIdx = stream->readInt64();
    // do some processing
 }

So I decided to look at reading into a temp buffer to improve speed which I think it does.

MemoryBlock bdata(256, true);

while(!stream->isExhausted())
{
    stream->readIntoMemoryBlock(bdata , 256 );
    for(int b=0; b < 256; b = b + 32)
    {
        float amplitude = 0;
        bdata.copyTo(&amplitude, b + 0, 4);
        long begin = 0;
        bdata.copyTo(&begin, b + 4, 8);
        long end = 0;
        bdata.copyTo(&end, b + 12, 8);
        short cIdx = 0;
        bdata.copyTo(&cIdx, b + 20, 2);
        short dIdx = 0;
        bdata.copyTo(&dIdx, b + 22, 2);
        long sIdx = 0;
        bdata.copyTo(&sIdx, b + 24, 8);
        // do some processing
    }

}

However the data is blank in second code although it is approx 10x faster.

How do I extract data from MemoryBlock and cast correctly.


#2

I haven’t tried, but it could work like that:

struct DataBlock
{
    float amplitude;
    long  begin;
    long  end;
    short cIdx;
    short dIdx;
    long  sIdx;
};
// =====
void load()
{
    File test = File::getSpecialLocation (File::userDesktopDirectory).getChildFile ("test.bin");
    FileInputStream stream(test);

    DataBlock block;
    stream.read (&block, sizeof(block));
    DBG ("amplitude: " << block.amplitude);
}

HTH


#3

You didn’t say what type of stream you’re using, but if it’s e.g. a FileInputStream then it won’t be very fast for small reads, because it simply calls the underlying OS functions each time you call a method on it.

The sensible way to add a buffer between your code and the OS calls is definitely not to mess around with your own memory blocks and bitwise casting - that’ll just end up causing bugs. Leave your original code unchanged, but just wrap your FileInputStream inside a BufferedInputStream before using it.


#4

Ah cool. Didn’t realise you could use like that. I should Imagine that would be an improvement…


#5

Yes it is fileInputStream sorry I should have clarified.
Yeah that’s exactly what I thought. Hence the memory block and increasing the block size significantly increases the speed. Forget bitwise operators and casting. I would just like to know how to read the data
a) from the stream into memory block - which I think I have done
b) get the read individual bytes or a block of bytes from a MemoryBlock


#6

Honestly, don’t attempt to buffer it yourself.

Your original code snippet where you used the read methods is the smart way to do this. Your second code snippet where you’re attempting to buffer it yourself is just awful. Don’t go down that route.

If you use a buffered stream, it’ll go at least as fast as if you wrote your own buffering system, but your code will be maintainable.


#7

A combination of reading directly to the structured datatype and BufferedInputStream seams to have done the trick. Getting the read speed I was expecting now.

Thanks for help