HTTP Requests on Linux

Linux
1

Hello!

I am trying to use the methods in the URL class to do HTTP requests, in particular I use “URL::readEntireXmlStream”. It is exactly the thing I need, and it works almost perfect, except that it does not work on non-Windows, when you are behind a proxy server.
In the windows version WinINet is used, that handles all the proxy stuff transparently. But in the non-Windows version, the network communication (connect, send, recv, …) is done on the socket level. In this case the handling of an http-proxy must also be done on that level. As I saw in the code, a socket-connect to the host specified in the url is done. This call hangs in my application, if the host I want to connect to is behind a proxy and is not directly accessible.
This is before the proxy information is added to the HTTP header, which is not sufficient to handle a proxy connection correctly. In case of a proxy the connect would have to go to the proxy host and the connection information of the original host has to be passed to the proxy.

Is there a chance to get that working? I am using Juce 1.45

Best Regards,
Andreas

2

Yes, that’s one of my to-do-list items, but not sure when I’ll have chance to do it. Any info anyone could throw my way about the best solution for this on linux would help speed things up!

3

In the file “juce_mac_HTTPStream.h” I changed the methods “open” and “createRequestHeader” of class “JUCE_HTTPSocketStream” to the following:

    bool open (const String& url,
               const String& optionalPostText,
               const bool isPost)
    {
        closeSocket();

        String hostName, hostPath;
        int hostPort;

        if (! decomposeURL (url, hostName, hostPath, hostPort))
            return false;

        String proxyURL (getenv ("http_proxy"));
        String proxyName, proxyPath;
        int proxyPort;

        const struct hostent* host = 0;
        int port = 0;

        if (proxyURL.startsWithIgnoreCase (T("http://")))
        {
            if (! decomposeURL (proxyURL, proxyName, proxyPath, proxyPort))
                return false;

            host = gethostbyname ((const char*) proxyName.toUTF8());
            port = proxyPort;
        }
        else
        {
            proxyURL = String::empty;
            host = gethostbyname ((const char*) hostName.toUTF8());
            port = hostPort;
        }

        if (host == 0)
            return false;

        struct sockaddr_in address;
        zerostruct (address);
        memcpy ((void*) &address.sin_addr, (const void*) host->h_addr, host->h_length);
        address.sin_family = host->h_addrtype;
        address.sin_port = htons (port);

        socketHandle = socket (host->h_addrtype, SOCK_STREAM, 0);

        if (socketHandle == -1)
            return false;

        int receiveBufferSize = 16384;
        setsockopt (socketHandle, SOL_SOCKET, SO_RCVBUF, (char*) &receiveBufferSize, sizeof (receiveBufferSize));
        setsockopt (socketHandle, SOL_SOCKET, SO_KEEPALIVE, 0, 0);

        if (connect (socketHandle, (struct sockaddr*) &address, sizeof (address)) == -1)
        {
            closeSocket();
            return false;
        }

        const String requestHeader (createRequestHeader (proxyURL.isEmpty() ? hostName:proxyName,
                                                         proxyURL.isEmpty() ? hostPort:proxyPort,
                                                         hostPath,
                                                         url,
                                                         proxyURL.isNotEmpty(),
                                                         optionalPostText,
                                                         isPost));

.....

and


    const String createRequestHeader (const String& hostName,
                                      const int hostPort,
                                      const String& hostPath,
                                      const String& originalURL,
                                      const bool isProxy,
                                      const String& optionalPostText,
                                      const bool isPost)
    {
        String header (isPost ? "POST " : "GET ");

        if (!isProxy)
        {
            header << hostPath << " HTTP/1.1\r\nHost: "
                   << hostName << ':' << hostPort;
        }
        else
        {
            header << originalURL << " HTTP/1.1\r\nHost: "
                   << hostName << ':' << hostPort;

            /* xxx needs finishing
            const char* proxyAuth = getenv ("http_proxy_auth");
            if (proxyAuth != 0)
                header << T("\r\nProxy-Authorization: ") << Base64Encode (proxyAuth);
            */
        }

.....

That works for my needs.

4

Cool - thanks, I’ll take a look through that when I get chance!

5

Just one additional wish/suggestion to that topic: It would be nice, if it were possible to explicitly specify the proxy to use for a Juce URL. If a proxy is explicitly specified, it should be used instead of checking the environment in Linux and by passing it to InternetOpen in Windows.
The default behaviour, if no proxy is explicitly specified, should be as it is now.
But as I said, that would be nice to have. At the moment I set the environment variable “http_proxy” in my code.

6

Thank you for version 1.46, but this fix didn’t get into it, so here are my changes again with the current version:

    bool open (const String& url,
               const String& headers,
               const MemoryBlock& postData,
               const bool isPost,
               URL::OpenStreamProgressCallback* callback,
               void* callbackContext)
    {
        closeSocket();

        String hostName, hostPath;
        int hostPort;

        if (! decomposeURL (url, hostName, hostPath, hostPort))
            return false;

        String proxyURL (getenv ("http_proxy"));
        String proxyName, proxyPath;
        int proxyPort;

        const struct hostent* host = 0;
        int port = 0;

        if (proxyURL.startsWithIgnoreCase (T("http://")))
        {
            if (! decomposeURL (proxyURL, proxyName, proxyPath, proxyPort))
                return false;

            host = gethostbyname ((const char*) proxyName.toUTF8());
            port = proxyPort;
        }
        else
        {
            proxyURL = String::empty;
            host = gethostbyname ((const char*) hostName.toUTF8());
            port = hostPort;
        }

        if (host == 0)
            return false;

        struct sockaddr_in address;
        zerostruct (address);
        memcpy ((void*) &address.sin_addr, (const void*) host->h_addr, host->h_length);
        address.sin_family = host->h_addrtype;
        address.sin_port = htons (port);

        socketHandle = socket (host->h_addrtype, SOCK_STREAM, 0);

        if (socketHandle == -1)
            return false;

        int receiveBufferSize = 16384;
        setsockopt (socketHandle, SOL_SOCKET, SO_RCVBUF, (char*) &receiveBufferSize, sizeof (receiveBufferSize));
        setsockopt (socketHandle, SOL_SOCKET, SO_KEEPALIVE, 0, 0);

#if JUCE_MAC
        setsockopt (socketHandle, SOL_SOCKET, SO_NOSIGPIPE, 0, 0);
#endif

        if (connect (socketHandle, (struct sockaddr*) &address, sizeof (address)) == -1)
        {
            closeSocket();
            return false;
        }

        const MemoryBlock requestHeader (createRequestHeader (proxyURL.isEmpty() ? hostName:proxyName,
                                                              proxyURL.isEmpty() ? hostPort:proxyPort,
                                                              hostPath,
                                                              url,
                                                              proxyURL.isNotEmpty(),
                                                              headers, postData,
                                                              isPost));

....

and


    const MemoryBlock createRequestHeader (const String& hostName,
                                           const int hostPort,
                                           const String& hostPath,
                                           const String& originalURL,
                                           const bool isProxy,
                                           const String& headers,
                                           const MemoryBlock& postData,
                                           const bool isPost)
    {
        String header (isPost ? "POST " : "GET ");

        if (!isProxy)
        {
            header << hostPath << " HTTP/1.0\r\nHost: "
                   << hostName << ':' << hostPort;
        }
        else
        {
            header << originalURL << " HTTP/1.0\r\nHost: "
                   << hostName << ':' << hostPort;

            /* xxx needs finishing
            const char* proxyAuth = getenv ("http_proxy_auth");
            if (proxyAuth != 0)
                header << T("\r\nProxy-Authorization: ") << Base64Encode (proxyAuth);
            */
        }
....

I hope you can add it to the next version of Juce. The proxy server support is not a big deal, and missing it makes the URL functionality rather useless, if you are behind a proxy server.

7

Ok, I’ve checked in an update with this - please have a go and check it, as I don’t have a proxy to test it on!

8

Works almost. There is one change missing to get it working. The line 98:

        address.sin_port = htons (hostPort);

must be changed to:

        address.sin_port = htons (port);
9

doh! Thankyou for that!