OpenGL Weirdness


#1

First off thanks to everyone for the posts in the Android forum (especially hugh) as they have helped tremendously.

 

Currently I am trying to work with OpenGL as my drawing is too slow and laggy on newer devices (1920X1080 phones and Nexus 10). I do this by attaching the context to the main window and detaching it when I call a native Java intent (seems to lock up the app if you don't because the OGL texture is getting removed). I do not do a lot of custom code yet. I don't do an OGL Renderer and just use the standard attach method.

 

Most things work good, it is super fast and feels fluid. However I get these weird anomolies that show up and I am at a loss what is causing them. Can anyone recommend the next steps to complete the app when using OpenGL rather than just a plain attach context? Any idea what is causing the anomolies from the attached image? (you can see them around the bottom portion of the image photo that is inset)

 

I am currently scaling my main window using a transform. This way the DPI scales with the device and I keep the transform scale to an integer number (otherwise the renderer doesn't properly find the edges on a component in the repaint method).

 

Thanks for any help.

 

Michael

 

 


#2

You're welcome to any help.  Android is such a royal pain that the more shared fixes the better.

on your problem; i've always run into trouble detaching contexts. A while back i tried to have contexts not attached to the top-level component. Whenever the hierarchy or visibility changes, it blows the OGL context. I tried working around this but it got worse.

Launching intents locks up. This can be fixed by decoupling the juce thread. you're calling into juce from java, then back into java then launching an intent, then expect to return to juce. decouple your intents making them async.

here's an example of how i launch an android file selector (in this case to load an image).

in my activity:


 private static final int SELECT_PICTURE = 1;
 
    private long pickImageCallback = 0;
    private String pickedImage;

     // send the picked image filepath back to juce
    private native void notifyPickImage(long callback, String path);

    // initiate selection of an image or file selector.
    // when selected `notifyPickImage' is called with the initial callback
    // and the selected `path'
    public final void pickImage(long callback)
    {
        pickImageCallback = callback;
        post(new Runnable()
            {
                @Override
                public void run() {
                    Intent intent = new Intent();
                    intent.setType("image/*");
                    intent.setAction(Intent.ACTION_GET_CONTENT);
                    intent.putExtra(Intent.EXTRA_LOCAL_ONLY, true);
                    startActivityForResult(Intent.createChooser(intent,
                                                                "Select Picture"), SELECT_PICTURE);
                }
            });
    }

then..


 public void onActivityResult(int requestCode, int resultCode, Intent data) 
    {
        if (requestCode == SELECT_PICTURE && resultCode == RESULT_OK)
        {
            Uri uri = data.getData();
            // can be null
            String path = getPicturePath(uri);
            if (path != null && pickImageCallback != 0)
            {
                //Toast.makeText(this, path,Toast.LENGTH_LONG).show();
                // store the chosen file path and decouple the 
                // callback so that the intent can be cleared down
                pickedImage = path;
                post(new Runnable()
                    {
                        @Override
                            public void run() 
                        {
                            if (pickedImage != null
                                && pickImageCallback != 0)
                                notifyPickImage(pickImageCallback,
                                                pickedImage);
                        }
                    });
            }
        }

also for completeness (incase this code is useful for you).


 private final String getPicturePath(Uri uri) 
    {
        String path = null;
        String scheme = uri.getScheme();
        if (scheme.equals("content"))
        {
            // use content resolver
            String[] projection = { MediaStore.Images.Media.DATA };
            Cursor cursor = getContentResolver().query(uri, projection, null, null, null);
            if (cursor != null)
            {
                int column_index = 
                    cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATA);
                if (cursor.moveToFirst())
                    path = cursor.getString(column_index);
                cursor.close();
            }
        }
        else if (scheme.equals("file"))
        {
            // when using the file manager
            path = uri.getPath();
        }
        return path;
    }

 


#3


I have narrowed down my weirdness to stretched images. If I used a stretched images rather than tiling it across on the right and bottom side it comes up short when merging. The example is making a drop shadow around my image. The corners are all regular images but the image in-between is a stretch image stretched from the left to the right side. Anywhere I do this type of thing the Open GL compositing fails for the last few pixels and the further it stretches the more 'bad' pixels there are.
I agree with the trashing of the OGL context. That was a real pain. Right now I do a detach before calling my get photo java code and then re-attach once it comes back. Works decent but I will be tweaking that idea quite a bit using more of your concepts.
My getImage code in java is a bitt different as I added in Kit-Kat specific code as well so I will post that below. I am thinking we should start a specific thread with great 'fixes'.

 

private static int RESULT_LOAD_IMAGE = 1;
    private static int GALLERY_INTENT_CALLED = 1;
    private static int GALLERY_KITKAT_INTENT_CALLED = 3;
    
    public final void showGallery (long value)
    {
        if (Build.VERSION.SDK_INT <19){
            Intent intent = new Intent();
            intent.setType("image/jpeg");
            intent.setAction(Intent.ACTION_GET_CONTENT);
            startActivityForResult(Intent.createChooser(intent, "Choose Photo"),GALLERY_INTENT_CALLED);
        } else {
            Intent intent = new Intent(Intent.ACTION_OPEN_DOCUMENT);
            intent.addCategory(Intent.CATEGORY_OPENABLE);
            intent.setType("image/jpeg");
            startActivityForResult(intent, GALLERY_KITKAT_INTENT_CALLED);
        }
        
    }
    
    
    private native void galleryImageChosen (String url, String future);
/**
     * Get a file path from a Uri. This will get the the path for Storage Access
     * Framework Documents, as well as the _data field for the MediaStore and
     * other file-based ContentProviders.
     *
     * @param context The context.
     * @param uri The Uri to query.
     * @author paulburke
     */
    
    
    @SuppressLint("NewApi")
    
    public static String getPath(final Context context, final Uri uri) {
        
        
        
        final boolean isKitKat = Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT;
        
        // DocumentProvider
        if (isKitKat && DocumentsContract.isDocumentUri(context, uri)) {
            // ExternalStorageProvider
            if (isExternalStorageDocument(uri)) {
                final String docId = DocumentsContract.getDocumentId(uri);
                final String[] split = docId.split(":");
                final String type = split[0];
                
                if ("primary".equalsIgnoreCase(type)) {
                    return Environment.getExternalStorageDirectory() + "/" + split[1];
                }
                
                // TODO handle non-primary volumes
            }
            // DownloadsProvider
            else if (isDownloadsDocument(uri)) {
                
                final String id = DocumentsContract.getDocumentId(uri);
                final Uri contentUri = ContentUris.withAppendedId(
                                                                  Uri.parse("content://downloads/public_downloads"), Long.valueOf(id));
                
                return getDataColumn(context, contentUri, null, null);
            }
            // MediaProvider
            else if (isMediaDocument(uri)) {
                final String docId = DocumentsContract.getDocumentId(uri);
                final String[] split = docId.split(":");
                final String type = split[0];
                
                Uri contentUri = null;
                if ("image".equals(type)) {
                    contentUri = MediaStore.Images.Media.EXTERNAL_CONTENT_URI;
                } else if ("video".equals(type)) {
                    contentUri = MediaStore.Video.Media.EXTERNAL_CONTENT_URI;
                } else if ("audio".equals(type)) {
                    contentUri = MediaStore.Audio.Media.EXTERNAL_CONTENT_URI;
                }
                
                final String selection = "_id=?";
                final String[] selectionArgs = new String[] {
                    split[1]
                };
                
                return getDataColumn(context, contentUri, selection, selectionArgs);
            }
        }
        // MediaStore (and general)
        else if ("content".equalsIgnoreCase(uri.getScheme())) {
            
            // Return the remote address
            if (isGooglePhotosUri(uri))
                return uri.getLastPathSegment();
            
            return getDataColumn(context, uri, null, null);
        }
        // File
        else if ("file".equalsIgnoreCase(uri.getScheme())) {
            return uri.getPath();
        }
        
        return null;
    }
    
    /**
     * Get the value of the data column for this Uri. This is useful for
     * MediaStore Uris, and other file-based ContentProviders.
     *
     * @param context The context.
     * @param uri The Uri to query.
     * @param selection (Optional) Filter used in the query.
     * @param selectionArgs (Optional) Selection arguments used in the query.
     * @return The value of the _data column, which is typically a file path.
     */
    public static String getDataColumn(Context context, Uri uri, String selection,
                                       String[] selectionArgs) {
        
        Cursor cursor = null;
        final String column = "_data";
        final String[] projection = {
            column
        };
        
        try {
            cursor = context.getContentResolver().query(uri, projection, selection, selectionArgs,
                                                        null);
            if (cursor != null && cursor.moveToFirst()) {
                final int index = cursor.getColumnIndexOrThrow(column);
                return cursor.getString(index);
            }
        } finally {
            if (cursor != null)
                cursor.close();
        }
        return null;
    }
    
    
    /**
     * @param uri The Uri to check.
     * @return Whether the Uri authority is ExternalStorageProvider.
     */
    public static boolean isExternalStorageDocument(Uri uri) {
        return "com.android.externalstorage.documents".equals(uri.getAuthority());
    }
    
    /**
     * @param uri The Uri to check.
     * @return Whether the Uri authority is DownloadsProvider.
     */
    public static boolean isDownloadsDocument(Uri uri) {
        return "com.android.providers.downloads.documents".equals(uri.getAuthority());
    }
    
    /**
     * @param uri The Uri to check.
     * @return Whether the Uri authority is MediaProvider.
     */
    public static boolean isMediaDocument(Uri uri) {
        return "com.android.providers.media.documents".equals(uri.getAuthority());
    }
    
    /**
     * @param uri The Uri to check.
     * @return Whether the Uri authority is Google Photos.
     */
    public static boolean isGooglePhotosUri(Uri uri) {
        return "com.google.android.apps.photos.content".equals(uri.getAuthority());
    }
    
    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        super.onActivityResult(requestCode, resultCode, data);
        
        
        
        
        
        if (requestCode == RESULT_LOAD_IMAGE && resultCode == RESULT_OK && null != data) {
            Uri selectedImage = data.getData();
            String[] filePathColumn = { MediaStore.Images.Media.DATA };
            
            Cursor cursor = getContentResolver().query(selectedImage,
                                                       filePathColumn, null, null, null);
            cursor.moveToFirst();
            
            int columnIndex = cursor.getColumnIndex(filePathColumn[0]);
            String picturePath = cursor.getString(columnIndex);
            cursor.close();
            
            galleryImageChosen(picturePath, "");
            
        }
        else if (requestCode == GALLERY_KITKAT_INTENT_CALLED && resultCode == RESULT_OK && null != data)
        {
            
            Uri selectedImage = data.getData();
            
            String picturePath = getPath(this, selectedImage);
            
            galleryImageChosen(picturePath, "");
            
            
            
        }
        
        
        
    }

#4

And now it is completely narrowed down to a call like this

 

 

int leftSideWidth = leftImage.getWidth() / 2;

int rightSideWidth = rightImage.getWidth() / 2;


context.drawImage(centerImage, leftSideWidth, 0, this->getWidth() - (leftSideWidth + rightSideWidth), this->getHeight(), 0,0, centerImage.getWidth(), centerImage.getHeight());

The idea is that I create a left piece and a right piece and stretch an image in-between the two. Sometimes all of them may be stretched vertically and the center vertically and horizontally (in this case it would almost be a solid image). However in most cases height of the component is leftImage.getHeight()/2 (retina) and the center is just stretched across. It may be a bar with a gradient or such.

I rewrote most of the code to actual tile the image however it would seem that would be slower than stretching. 

 

Does anyone have any idea why the OpenGL code (across all platforms) would render a few pixels incorrectly at the end of the stretch? It seems the larger it is stretched the more bad pixels are in place. Over 2X the size of the image and its 1 pixel and about 10x is 5+ pixels.

 

 


#5

 

what happens if you switch off OGL (ie take out your attach). I assume that all works.

One point to mention, although this isn't what you're currently doing, is that (unless it's been fixed), Juce wont render anything but the entire source image in OGL, ie your last 4 args to `drawImage' must be 0, 0, srcW, srcH. I found this to my peril. 


#6

When I switch off OGL it works great. However I ran into another issue. I set a transform on my UI to scale it for high dpi devices. And now on the Nexus 5 little lines show up on the bottom right of most images that are drawn when OGL is active. This also happens if I do a transform on iPhone, or Mac. I don't think its quite calculating the bounds correctly in the OGL code. Learning Juce OGL now to see if I can narrow this one down and will report back.

 

 


#7

**moved


#8

I have found a very quick fix for the stretching issues as well as the little white boxes that show on the bottom right when the app is scaled.

 

Inside of juce_OpenGLTexture.cpp in the "create" function I have adjusted this code:

 

#if defined(JUCE_IOS) || defined(JUCE_ANDROID)

    width  = w;

    height = h;

#else

    width  = nextPowerOfTwo (w);

    height = nextPowerOfTwo (h);

#endif

 

With this change most of my issues are resolved. Now I am not entirely sure if there is an issue in the shader program that doesn't handle the bounds correctly or possibly the stretching correctly but at least I can move on.

 

Now my assumption is that on iPhone and Android the OpenGL ES implementations support non power of two textures. Any takers on the specifics of this one?


#9

That doesn't look like the latest code to me - I already added an option for non power of two textures a while back.


#10

Correct Jules. However the thread is still pertinent as people may not know why they are having issues when the requirement for NPOT textures is on or off. Right now it defaults to requiring POT textures and while this is on there are issues with the rendering for scaled textures and especially noticeable during component scaling.

 

Also, the new OGL rendering system still has the issue unless you turn on NPOT support with the #define you have setup.