Problem with POST requests over HTTPS with Keep-Alive (with workaround)

#1

It appears (other sources linked at the bottom of this post) that the networking code of Apple exhibits a bug in the following circumstances:

  • a client application on macOS or iOS makes a POST request over HTTPS
  • the server response includes the headers (shown below) to indicate its intention to keep the connection open, to serve future requests that may arrive in the near future.

The mentioned headers in the server response that trigger the bug are the following (the actual values for timeout and max parameters don’t matter):

Connection: Keep-Alive
Keep-Alive = timeout=5, max=100

They are perfectly legit and they cause no issues on Windows or with GET requests on macOS.

The timeout parameter in particular, informs the client that the current TCP connection can be reused for further requests for 5 more seconds.

After that time, the server will close the connection on its side, so the expected behavior for the client is to establish a new connection if it intends to make another request past that timeout.

In the case of Apple, and in the circumstances stated above (POST over HTTPS), what happens instead is that the client ignores that timeout indication, and even for a request that comes, for example, 10 seconds later, it attempts to reuse the connection well past the intended 5 seconds.

The result is that the buggy client gets disappointed when it finds that the connection has been closed by the server, and it produces the following error, printed by a DBG in didFailWithError(), in juce_mac_Network.mm

Error Domain=NSURLErrorDomain Code=-1005 "The network connection was lost."

Inspired by some of the workarounds found online for the same problem, I can confirm that immediately repeating the same request results in a success on the second attempt.

The attached patch for juce_mac_Network.mm does implement such workaround for this specific error condition.
EDIT: it is intended for the “master” branch, on which I am, not “develop”

patch.txt (2.6 KB)

It does so by adding a member to URLConnectionState to capture the resulting error code in case of a failure, and if that error code indicates the “lost connection” condition, then the connection attempt in createConnection() is repeated (only once).

For people interested in this problem, these are the descriptions given by other (non-JUCE) developers struggling with it:

#2

Interesting, do you have some code that reproduces this? I don’t know much about HTTP but the way that I understood it was that a POST request is a one-time request to send some data to a server so why would the connection need to be kept open?

Also, the patch that you have provided is in the deprecated part of the juce_mac_Network.mm code which will be removed at some point in the future so be wary of using it!

Ed

#3

Hi @ed95, I have sent you a private message with the code because reproduction requires interaction with a server whose URL we wish not to disclose.

I have not noticed that a part of juce_mac_Network.mm was marked as deprecated, perhaps it is because I have worked on master?

However, in favor of what has that part been marked as deprecated?

#4

Hi @yfede, thanks for sending the code! I’m afraid I can’t reproduce the problem though, I have tested on develop and master but both succeed on the second function call. I’m on macOS 10.12.2 if that helps.

This is the line explaining the deprecation - the newer version of this code uses the NSURLSession object and some extra methods but it shouldn’t be too difficult to apply the patch to the newer code as well if needed.

Ed

#5

Ah, I see. But if I understand those defines correctly, the new code is only enabled for products that targets OS X 10.10 or newer.

Unfortunately I need it in projects that target OS X versions much older than that, that’s why the deprecated code was used instead.

Did you try it with the new code or the deprecated one? Could you force the usage of the deprecated code and let me know if still it does not reproduce?

#6

Ah right, I was testing it using the newer code… Just tried it using an older deployment target and could reproduce the problem. There have been quite a few changes with the mac networking code lately so I’ll take a look at this and get back to you soon.

Ed

#7

@yfede This should be fixed on master and develop now.

Ed