Http content length


#1

Are there any plans to implement the “content-length” value for http streams, to determine the size of a download before downloading it?

This would be extremely interesting for showing a progress.


#2

It’s 10 lines:

  #define CL "Content-Length"
  stream = URL("whatever").createInputStream(false);
  if(stream != 0)
  {
     String line = stream->readNextLine();
     int length = 0;
     while (line.length() && !length)
     { 
         int pos = line.indexOfIgnoreCase(T(CL)); 
         if (pos != -1)
                length = line.substring(pos+ strlen(CL) + 1, line.length()).getIntValue(); 
         line = stream->readNextLine();
    }
        
    // Create a substream for the data (can be empty if not content-length header
    SubregionStream outputStream(stream, stream->getPosition(), length, true);

#3

Thanks for the code, but it doesn’t work. It just seems to read the entire file I’m downloading and doesn’t find a “content-length” string.

Do you know where JUCE handles the header returned by the server?


#4

I’ve found a way, which is to implement a function like this (Windows in the example):

[code]int juce_getContentLength (voidhandle)
{
DWORD result = 0;
const ConnectionAndRequestStruct
const crs = (const ConnectionAndRequestStruct*) handle;

if (crs != 0)
{
    DWORD index = 0;
    DWORD size = sizeof (result);

    HttpQueryInfo (crs->request,
                   HTTP_QUERY_CONTENT_LENGTH | HTTP_QUERY_FLAG_NUMBER,
                   &result,
                   &size,
                   &index);
}

return (int) result;

}[/code]

… which is then called from WebInputStream::getTotalSize().

I hope it’ll also be possible for OSX (not there yet).

It would still be nice to have this included in JUCE btw.

In a related request, it would be nice to have an easy way to set the rangeStart and rangeEnd values of a http stream, to easily resume a download.


#5

The code I’ve given does work (I use it in one of my software to download JPEG streams).

I’ve renamed the variable (and “Content-Length:”, so I might have missed some, try to fix the compilation errors (mainly check with the debugger that “line.substring(pos+ strlen(CL) + 1, line.length())” starts with an integer).

It simply trash all HTTP headers except “Content-Length”.
BTW, you can even change that line :

length = line.substring(pos+ strlen(CL) + 1, line.length()).getIntValue();

by

length = line.getTrailingIntValue();

BTW, if you only need the content-length then it’s the way to go.
If you need to parse all HTTP headers, there is a HTTPClient in the UZI browser component I’ve posted in the “UT and Component” part of the forum (and there is also a minimalist HTTP client in MacOSX implementation of Juce)

Using HttpQueryInfo has multiple issues, the biggest one is that if the server doesn’t answer, you’ll have to deal with OS timeout which is like 30s or more.
It’s Windows specific, as you’ve said, and your current code doesn’t check for errors (Content-Length is not mandatory, so you should check for its presence).


#6

The code didn’t seem to find any http headers when I tried it, just the file I was downloading. At least as far as I could tell.

You’re right that the Windows code is far from perfect, but as I have to finish something today and it’s going to be OSX only for a while (I’ve got a far more evolved Delphi version for Windows), I’ll have to check this out again at a later time.

Thanks for your time.


#7

X-Ryl669, your code doesn’t work for me either. I don’t see how it could work, because the input stream you are using won’t contain the HTTP header, but the file itself !?


#8

Well this code was working 9 months ago.

The current code is:


    InputStream * imageFile = URL(...).createInputStream(false);
    if(imageFile != 0)
    {
        String line = imageFile->readNextLine();
        int length = 0;
        while (line.length()) 
        { 
            int pos = line.indexOfIgnoreCase(L"Content-Length:"); 
            if (pos != -1)
                length = line.substring(pos+15, line.length()).getIntValue(); 
            Logger::outputDebugString(String(L"Line length:") << line.length() << line); 
            line = imageFile->readNextLine(); 
        }
        
        // Create a subregion of the preview
        SubregionStream s(imageFile, imageFile->getPosition(), length, true);

        /* Decode Image*/
        JPEGImageFormat imageFormat;
        Image * image = imageFormat.decodeImage(s);
        return image;
    }

The server must answer with a content length for this to work. It must use “multipart” mixed too.
If you don’t own the server (or the server doesn’t fit the requirements above, then PM me I’ll send you my full HTTP client).


#9

Well, then my server doesn’t respond properly.

Can you, with your full HTTP client, retrieve the file size of this file (it’s just an example file i tried with): http://www.schaack-audio.com/downloads/quad/QuadSetup201.exe

… and does this work on OS X and Windows with your code?


#10

The connection is like this :

GET /downloads/quad/QuadSetup201.exe HTTP/1.1
Host: www.schaack-audio.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive


HTTP/1.1 200 OK
Date: Tue, 02 Sep 2008 09:31:55 GMT
Server: Apache
Last-Modified: Mon, 18 Aug 2008 13:06:58 GMT
ETag: "f84f7f3d-29069b-48a973f2"
Accept-Ranges: bytes
Content-Length: 2688667
Connection: close
Content-Type: application/x-msdos-program

...data ...

You have the content length in plain text.
URL use WebInputStream which use Win32 API InternetOpen (which could “eat” the headers).

If you want a true HTTP client, either PM me (I can’t publish my code in public, nor you can use it directly), or you can also use juce_mac_HTTPStream.cpp and uncomment the content length reading lines.


#11

would you be willing to share this hack with us?