InterProcessConnection - Corrupt Value Tree over Network - Bug?

General JUCE discussion
1

Dear all,

I have a problem with the InterProcessConnection class and sending a large ValueTree (300kb) over the network, which results in corrupted data:

I have two applications, a server and a client and the client sends the project file (ValueTree) to the client:

void Server::sendFullProject()
{
    juce::MemoryOutputStream m;
    m.writeShort(XMCommand::requestFullSync);
    getProjectData().writeToStream(m);
    sendMessage(m.getMemoryBlock());
}

and the client receives the data and tries to rebuild the ValueTree:

void Client::messageReceived(const juce::MemoryBlock &message)
{  
    juce::MemoryInputStream data(message, false);
    ...
    juce::ValueTree projectData = juce::ValueTree::readFromStream(data);
}

When I run both applications on the same machine, everything works fine!
But when client and server run on different machines, the readFromStream function always hits asserts, that it is trying to read corrupt data - sometimes a name is empty or an compressed int suddenly has 30 bytes etc.

As far as I remember this used to work over network with no problem, but now in Juce 7.0.9 I encountered this problem.

Looking at the data sent, I found that the version over network adds a lot of zero bytes to it, maybe this shifts something and the reader gets confused?

(left side local version, right side remote version over network)

comparison_remote_local_IPC
comparison_remote_local_IPC2420×1170 410 KB

Hope someone has an idea or can point me in the right direction!

2

We did recently identify an issue in which sending large amounts of data may result in bytes being dropped.

As a first step could you pull the latest version of develop to see if these issues are resolved there? the relevant commit is…

If the above commit doesn’t resolve your issue, please let us know as there is also a way to increase the buffer size for a StreamingSocket instance which is what the InterprocessConnection and InterprocessConnectionServer classes use internally. We haven’t yet exposed those options on those classes but that’s maybe something we could look into if it turns out to be required.

3

Dear Anthony,

thanks for the quick reply!

I only had time for a quick check and the results are a bit weird, it worked once out of 10 tries.
So i started both apps 10x and on one connection it seemed to work and everything was fine, but in 9 other cases the problem remained.

Hopefully I will have more time to investigate later, but thanks again!

4

What sizes are we talking about here? Any insights or the possibility to add asserts?

5

Unfortunately I don’t think it’s as simple as saying size x. The report we got was using ~250 OSC messages, but only when the payload of the OSC messages was more than 32 bytes and this was using a UDP connection (InterProcessConnection uses TCP).

I didn’t work on this directly but as I understand it, it seems some internal OS buffers weren’t adequately sized so data was dropped (it was only reported and reproduced on macOS, Windows seemed to be fine). I strongly suspect it’s highly dependant on the connection too.

When investigating further we found that our implementation was setting the send and receive buffers in the sockets to a size of 65,536 bytes. Further testing revealed that the macOS default (on the versions we looked at) was actually closer to 700,000 bytes. As a result we now default to the maximum value between the OS default and 65,536. This ensures we’re not decreasing the default size for anyone accidentally but in most cases the default size of the buffers will increase.

We also added a SocketOptions struct for specifying the specific receive and send buffer sizes so that if anyone were to hit this in future one option would be to increase the buffer sizes, or at least specify a minimum buffer size (which might be appropriate here).

Unfortunately It seems we missed adding these options to the InterProcessConnection classes so right now it’s not possible. If it turns out that specifying these options is useful in this case we’ll certainly look to add those options to the interface ASAP.

As for assertions I’ll check in with the team but I suspect there isn’t an obvious place to assert.