Trying to get json data from a https url. This works fine on MacOS but returns 0 immediately on Windows 10. I have tried with two different Windows machines.
static juce::String get_url(const juce::URL& url) {
Utils::log_info(juce::String("Fetching url: ") + url.toString(false));
juce::StringPairArray responseHeaders;
int statusCode = 0;
auto stream = std::unique_ptr<juce::InputStream>(
url.createInputStream(false, nullptr, nullptr, "application/json", 5000, &responseHeaders, &statusCode));
if (stream && statusCode == 200) {
auto result = stream->readEntireStreamAsString();
return result;
}
Switching to the cpr library that uses libcurl everything works as expected
cpr::Response r = cpr::Get(cpr::Url(url.toString(false).toStdString()));
if (r.status_code == 200) {
return r.text;
}
What could be the issue with the JUCE version and how to fix it?
Because statusCode=0 before the call, it not clear if the createInputStream() sets it to 0 or if it just does not update it, which will seem to contradict the doc on statusCode:
EDIT: Do you have an Antivirus software running on Windows? Do you see any relevant exception in the console output if you can use one ?
In juce_win32_Network.cpp, the connect function fails: isError() below returns true and the createInputStream returns a nullptr in the end. I stepped through the createConnection but did not see any immediate reason for the failure…
@esgrove thanks!
I absolutely need the headers in my use cases.
I haven’t checked the new ParameterHandling thing, but I would assume it’s just a public interface thing? Would be great to get some response from the JUCE team on this issue.
I took a brief look into this but I don’t know yet if there is a bug here.
I ran the first snippet that @esgrove provided with the URL of some random HTTPS website. On Windows I am getting an ERROR_WINHTTP_HEADER_NOT_FOUND from HttpSendRequestEx() whereas MacOS seems to happily proceed with downloading the data from the server.
When I change the header to Content-Type: application/json as connorreviere suggested, the request successfully completes on Windows too.
So is it possible that your problem would be solved by using Content-Type: application/json instead of just application/json?
application/json isn’t a valid HTTP Content-Type: header when on its own, so don’t send that. The format you’re intending is outlined here and should definitely be Content-Type: application/json: https://datatracker.ietf.org/doc/html/rfc2616#section-14.17
In terms of JUCE, unfortunately there’s no easy way to check if this header string(s) provided is/are valid from a programmatic point of view… And if there is a method, I believe the bare-minimum checking would be really slow as you’d have to scroll through each Content-Type:'s media-type to know if that string matches anything known, only to then convert it to a Content-Type: existingString.
That being said, the cpr library mentioned above appears to be doing a lot of validation WRT to JSON: it might be patching up the headers accordingly.
You could debug this in your app probably, but when playing with HTTP connections it’s a good idea to stick a proxy, like Charles Proxy, to see what data is being sent by your app. HTH!
However, I’m getting the bug with my own custom headers and not Content-Type.
Those are absolutely valid headers and Postman/CURL/JUCE on Mac is happy to parse them, but it seems like somewhere in the JUCE implementation it triggers some extra checking, which I don’t think is needed in HTTP.
Are you positive that such a custom header is being sent out and received on the other end?
I’ve been testing this on an M1 Mac and haven’t seen what’s being sent out on the wire yet, but I couldn’t parse or receive that header with netcat or flask (which could be the same thing underneath).
Observing on the receiving end and seeing what request was ultimately sent out by createInputStream() I can see Content-Type: application/json but when I just use application/json that is just completely missing on the receiving end.
Is it possible that MacOS is just ignoring that header and making the request without it? Windows may be stricter in that it refuses to make a request altogether.
@attila I am am 100% certain, and I’m not sending application/json as an empty header, that was maybe a side-problem by the OP but not the bug here.
I’m sending custom headers like “header: content”. And they are 100% sent and received on my own server, and parsed correctly. This is tested on my Win/Mac application (that is now using CURL) and Postman along with my server that’s deployed on google servers and using cpp-httplib.
So far I have been unable to replicate the problem. The function below can be used to make a successful request on Windows, JUCE develop, using both http and https. I can see the weird header on the server side and read the content returned. Do you have any further pointers maybe?
juce::String getUrl (const juce::URL& url)
{
juce::Logger::writeToLog ("Fetching url: " + url.toString (false));
juce::StringPairArray responseHeaders;
int statusCode = 0;
auto options = juce::URL::InputStreamOptions (juce::URL::ParameterHandling::inAddress)
.withExtraHeaders ("Weird: header")
.withConnectionTimeoutMs (5000)
.withResponseHeaders (&responseHeaders)
.withStatusCode (&statusCode);
auto stream = url.createInputStream (options);
if (stream && statusCode == 200)
{
auto result = stream->readEntireStreamAsString();
return result;
}
return {};
}
Looks like I’m not able to reproduce this issue with a minimal example on 6.1 as well (My code is using 6.0.8). I’ll do further tests and check. Thanks again for spending time on the issue!
