URL::readEntireTextStream() inconsitent across platforms

Hi, My goal is to login successfully to a website. I have made a project isolating the issue, the code is basically the following.

first i isolate the authenticity token

URL lUrlGetTocken("https://thewebsite.com/login");
String lHtmlWithToken = lUrlGetTocken.readEntireTextStream().toUTF8();
lHtmlWithToken = lHtmlWithToken.fromLastOccurrenceOf("authenticity_token\" value=\"", false, true);
lHtmlWithToken = lHtmlWithToken.upToFirstOccurrenceOf("\"", false, false);

then I post the login datas.

URL lUrl = lUrlGetTocken.withParameter("password", aPasssword);
lUrl = lUrl.withParameter("email", aEmail);
lUrl = lUrl.withParameter("authenticity_token", lHtmlWithToken);

String lString = lUrl.readEntireTextStream(true);

finally i check if the response is the one matching a validated connection.

bool lLoggedIn = lString.contains("article class=\"registration\"") || lString.contains("/account\">redirected");
return lLoggedIn;

The response on Windows is the same as if the authenticity token was wrong. On OSX it logs me in. I am working with JUCE 5.4.7, and run the same code on XCode and Visual Studio 2015.
How can this be?
Thank you

You probably need to add a user-agent for your web server to be kosher.

With a WebInputStream, you would do this by calling withExtraHeaders ("User-Agent: something convincing here").

What HTTP response code are you getting back anyways?

Thank you, I will look in that direction.

Funny thing is that this is a code I compiled few years ago and which worked on Windows. The old compiled version still works today. But now that i am compiling the same code again, on a new machine, it doesn’t work anymore, and only on Windows. I find it confusing, there is something i must be missing.

I am getting back a 422 error.

Hm, I’m thinking there’s more to it than just a lack of user-agent in the case of a 422.

Hard to say what else it can be without knowing the server and service (eg WordPress, aws, etc…).

The datas sent to the server are different depending on which Windows computer I am using to compile it. Are we relying on some external library which lives on my computer? In this case is there a way to direct my code to an alternative library?

Do you have an example of that? Are you using Wireshark or Fiddler or something like that to analyse the network data being sent out and received?

Hi,

I am struggling with WireShark, sounds like there are things i could learn about networks. I will look more.

However I notice that the response header of the post request is almost empty with Windows, as opposed to the OSX version.

Windows:

HTTP/1.1 422 Unprocessable Entity = , Date = Mon, 19 Oct 2020 18:03:07 GMT, 
Server = Apache/2.4.18 (Ubuntu), X-Request-Id = 211dbab6-3ed3-41b1-9e76-f2049067f531, 
X-Runtime = 0.037202, X-Powered-By = Phusion Passenger 5.0.28, Content-Length = 1547, 
Status = 422 Unprocessable Entity, Content-Type = text/html; charset=utf-8,  = , 

OSX:

Keep-Alive = timeout=5, max=99, X-Runtime = 0.021651, Set-Cookie = locale=fr; path=/; expires=Tue, 19 Oct 2021 18:05:15 -0000, _orchplaymusic_session=U2xmWHVlQkVZcEp5R0s1MzJGSkpoUHA2QS9ac0Q5RnZjOTMxcEZ3N2xjY2JTdGcwMnMrcTZ5bVUrZzRONFVkUlpBK3N2SDByVXBMUUxnWDlGdGx0Q0VwR2tOWUtleVMrNTlBcHVJQnNRWEFxUmVPeXRveS80Yy9mSTRnRS95dVkyL0hpZldJZmRiTEV0d2VrZ0JVQ2ttMmpFZUFBVUF2U0hJNEZqNmdZRERiZThxWTVQNFVrTmMrR2FFOE5UQkgzZXAyV1lxMk9jUjRIRXNIWVRTU25ZaFBiY3d3Mk9tWUY0S083Y1VpU3Bzek5xVThBQjM0QVdQd3JDRzlHMWw1NGQzOERhY3AwaDBxUmpCWDVlMFlnR2c9PS0tRHhpY3ozTkowYTF6THZQTEZ5L2grQT09--1236df9e9977e6c419a15c180608190021b742a5; 
path=/; HttpOnly, request_method=; path=/; max-age=0; expires=Thu, 01 Jan 1970 00:00:00 -0000, P3P = CP="CAO PSA OUR", Server = Apache/2.4.18 (Ubuntu), Link = <https://www.orchplaymusic.com//account>; 
rel='alternate'; hreflang='en', Content-Encoding = gzip, Cache-Control = max-age=0, private, must-revalidate, Status = 200 OK, X-Powered-By = Phusion Passenger 5.0.28, 
Content-Length = 2726, X-XSS-Protection = 1; mode=block, Date = Mon, 19 Oct 2020 18:05:15 GMT, Connection = Keep-Alive, Vary = Accept-Encoding, 
X-Request-Id = 38e47ed4-99d8-4d70-a67f-ddd229b69d1c, X-Frame-Options = SAMEORIGIN, Etag = W/"7a89e45b724355901e05f02ca394732b-gzip", 
Content-Type = text/html; charset=utf-8, X-Content-Type-Options = nosniff, X-UA-Compatible = IE=edge,chrome=1, 

Could it be related to cookie management? I can see in juce_win32_Network.cpp that the INTERNET_FLAG_NO_COOKIES option is set.

this is a minimal code showing the inconsistence between Windows and OSX.

testNetwork.zip (3.6 MB)

That very well could be it. Initially it looked like you were trying to do OAuth, but that was a wrong assumption on my part.

What’s happening here is that you’re trying to use form validation which will very likely require cookies and redirect handling.

I think you’re in for a lot of work… there are too many specifics that you need to sort out to make that work correctly, let alone considering security…

What web back-end are you using - fully custom, WordPress, something else? (I’m hoping for the latter seeing that you can integrate super easily with a standard login/logout system called OAuth with WordPress, among other web services).

Also, I suggest getting set up with cURL and try that instead of the higher level functionality that JUCE provides. Your circumstance appears to warrant getting really up close and personal with the data…

An example with cookie handling: https://stackoverflow.com/a/12399176/1907103

The backend is made with Ruby on Rails.

Your solutions seem legit but I am still bumping my head against the fact that the same code project has been compiled by myself 2 years ago and successfully logs in, even today.

Before I go to so much work, is there a way to send back the cookies on windows using WinInet?

I’m assuming you’re not intending to have your app work strictly from within the bounds of Windows? I suggested an example with cURL because it’s cross-platform (that, and you’ll find more documentation and examples with it).

Either way, you can look at MSDN’s docs on getting and setting cookies here: Managing Cookies - Win32 apps | Microsoft Learn

You’re right, this app is for windows, osx, and in a near future iOS. Since it is working fine on OSX, I am looking for the shortest way since you guys already did a lot work developing other useful functionalities.

I will probably find my way from here, thank you. Before we close this topic, may i ask why Juce is not using Curl for every platform? Why only Linux? It would surely fix the inconsistencies, would’t it?

1 Like

@jules just want to bring the attention of the team on this. i believe it would be a great improvement to have a consistent stream reading and cookies management.