Weird createInputStream behaviour (Android)

I’m getting weird behaviour when attempting url.createInputStream() on Android. Some phones work, some don’t:

Phones that DO work:

  • Samsung Note 20 Ultra
  • Xiaomi Mi A1

Phones that DON’T work:

  • Samsung Note 9
  • Samsung S9

The phones that don’t work return NULL when attempting createInputStream().

  • createInputStream() is called from a background thread, as instructed by the createInputStream docs (required for Android)
  • this happens both in a fresh JUCE project containing only the file loading code and also in my app
  • this happens on the latest develop tip of JUCE and also on a revision from 31st Jan 2021
  • read and write permissions are enabled in jucer file (by default)

For completeness, here’s the code from the fresh JUCE project:

When a button is clicked:

void TestAndroid2AudioProcessorEditor::buttonClicked (juce::Button* button)   
{
    if (button == &loadAudioFileButton)
    {
        RuntimePermissions::request (
                RuntimePermissions::readExternalStorage,
                [this] (bool wasGranted) {
                    this->importAudioFileAndroid();
                }
        );
    }
}

Where…

void TestAndroid2AudioProcessorEditor::importAudioFileAndroid()
{
    SafePointer<TestAndroid2AudioProcessorEditor> safeThis(this);
    
    fileChooser.reset (new FileChooser ("Choose an Audio File to Import...", File(),
                                        "*.wav;*.aif;*.aiff;*.flac;*.mp3;*.wma;*.ogg;*.aac;*.m4a", true));

    fileChooser->launchAsync (FileBrowserComponent::openMode | FileBrowserComponent::canSelectFiles,
                              [safeThis] (const FileChooser& fc) mutable
                              {
                                    if (safeThis != nullptr && fc.getURLResults().size() > 0)
                                    {
                                        auto results = fc.getURLResults();
                                        safeThis->urlAndroid = results.getReference(0);
                                        safeThis->loadAudioAndroidThread->startThread();
                                    }
                                    safeThis->fileChooser = nullptr;
                              }, nullptr);
}

And inside the thread loadAudioAndroidThread is the run()…

void run() override
{
    DBG("check not in message thread " + String((int)juce::MessageManager::getInstance()->isThisTheMessageThread()));
    DBG("getLocalFile: " + url.getLocalFile().getFullPathName());
    DBG("url.toString: " + url.toString(true));
    std::unique_ptr<InputStream> wi = url.createInputStream (true);// check for NULL
    DBG("debug breakpoint ");
}

urlAndroid is passed as a reference to the Thread object constructor.

I get the same result with

std::unique_ptr<InputStream> wi = url.createInputStream (false);

Here’s the files needed to run the fresh JUCE project, processor files are unchanged.

PluginEditor.cpp (2.9 KB)
PluginEditor.h (1.9 KB)

1 Like

You’ll have to use three backticks for that (ignore the quotes) “`:slight_smile:

Ah brilliant, that’s sorted that. Now that just leaves the createInputStream problem :sweat_smile:

I drilled down into createInputStream.

auto f = open (file.getFullPathName().toUTF8(), O_RDONLY);

in juce_posix_SharedCode.h (when called in phones that don’t work) is returning -1 which eventually results in a nullptr return.

I can’t figure out what’s going on inside the ‘open’ method above as apparently it takes me to code that’s not even inside my project code or the juce code!

I checked that the return string of file.getFullPathName() is identical for phones that do work vs phones that don’t work.

When I check status.getErrorMessage() from

void FileInputStream::openHandle()
{
    auto f = open (file.getFullPathName().toUTF8(), O_RDONLY);

    if (f != -1)
        fileHandle = fdToVoidPointer (f);
    else
        status = getResultForErrno();
}

in juce_posix_SharedCode.h, I get the string ‘Permission denied’. However, I have Read From External Storage enabled and as you can see from my code above I’m using RuntimePermissions::readExternalStorage.

What am I doing wrong?

To properly narrow this down, you’ll have to check the gritty details that dictate permissions.

Does your manifest reflect this permission?

What Android API, SDK, NDK are you using/targetting?

What Android version do your test devices use?

Did you try reading your file from the main thread, just to rule out inter-thread funk?

Maybe you need to use the legacy storage depending on the device? android:requestLegacyExternalStorage="true ? Storage updates in Android 11  |  Android Developers

Maybe your file is inaccessible, but some OS versions are playing it too cool, so you have to move it elsewhere (assuming it’s something you control?)?

@jrlanglois Not sure what you mean about manifest but all phones I’ve tested show ‘storage’ permission under the app in settings > apps.

API 26 and above, gradle 6.5, android plug-in 4.1.2, SDK 31, NDK 21.0.6113669

phones that work: android 9 and android 11
phones that don’t work: android 10

I did try reading from the message thread initially and hoped that switching to the background thread would help, alas no.

I’d be interested in trying requestLegacyExternalStorage. How do I go about doing this (presumably in jucer)?

I’ve tried loading files for many different locations on the device, the issue is at createInputStream-level so it’s not related to where the file will be saved.

This post covers the ‘what’ as there are really 2 components to add to the manifest: https://stackoverflow.com/a/63365276

As for what the manifest is, it’s a file that controls what goes into an Android app and its configuration. Usually it’s named AndroidManifest.xml and has a manifest tag in there. You can add stuff to it through the Projucer in the Android configuration, under “Custom Manifest XML Content”.

Custom Manifest XML Content, that’s what I needed to know, thanks.

Could you please tell me how to go about implementing requestLegacyExternalStorage?

We’ve found you also need to prompt the user to update their Permissions.

You can take a look to see how we tell the user to handle this in the new Wotja for Android, which is going live tomorrow (21.8.0).

When Wotja starts, it checks to see if has permission to write to the (shared) Storage area.
a) If it does, it will create a Wotja folder under Documents (or Music or Download, in descending order of preference…)
b) if it doesn’t, it shows an alert warning the user about this Permissions setting, and telling them how to fix it; if the user follows the instructions, and then restarts Wotja [and then, a) should apply!]; Wotja then attempts to move any files that might have had in the “old” private area, to this shared area; but Wotja can only do this of course once the user has changed permissions…

NB: if the user hasn’t set your Permissions as per the alert, Wotja will continue to save files in private storage area.

Not ideal, but at least it now works! Required some patching to Juce, of course.

Pete

1 Like

I have finally managed to fix this issue! Thanks to a tip off from @jrlanglois and @peteatjuce, all of the info I needed was on the thread @jrlanglois posted, so thank you very much.

I’m now going to break down the details of exactly what was done in case anybody else is has this problem in Android in future. Everything is done in Builds/Android/app/src/main/AndroidManifest.xml (this is on Mac, windows path may be different, just search for AndroidManifest.xml).

The following is placed in the <manifest element alongside WRITE_EXTERNAL_STORAGE and READ_EXTERNAL_STORAGE which should also be there if you have enabled these in Jucer (which you should).

<uses-permission android:name="android.permission.ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION"/>
<uses-permission android:name="android.permission.MANAGE_EXTERNAL_STORAGE"/>

The following is added to the end of the <application element as an attribute

... android:requestLegacyExternalStorage="true">

Note that I also use RuntimePermissions::request when calling the top-level file reading function (see code in the original post above) and am also executing the file reading code in a dedicated background thread (again, see code in the original post above). If anybody is having issues recreating my implementation shown in the original post, contact me at deltavaudio@gmail.com as there are other bits of code behind the scenes.

Some subset of all of the above might also have fixed the issue, I’m not entirely sure. I’m just happy that it’s working so I’ll leave everything as is for now.

Another thing to note is the method of actually implementing this via Projucer. There is a field within the Android exporter called Custom Manifest XML Content where you can add things to AndroidManifest.xml. However, I wasn’t sure how to update this such that it would add an attribute to an existing XML element. So in the end I didn’t use the Custom Manifest XML Content field and instead I integrated the changes via an existing python build script which now saves the Projucer then does string replace in AndroidManifest.xml to make the changes I listed above, then either opens android studio or builds the APK/AAB file. Unless you find a way to make the Custom Manifest XML Content field add an attribute to an existing XML element, you may need to take a similar approach to my build script.

2 Likes

Great news!

The thing I’ve found is that, in general, our users want to save their documents to public storage, rather than the private internal area. That allows them to share their documents with other platforms (e.g. via Google File Transfer app), easily back them up etc. That is why resolving this is so important :slight_smile:

We’re currently unable to upload apps to Google Play that are built using MANAGE_EXTERNAL_STORAGE - see Use of All files access (MANAGE_EXTERNAL_STORAGE) permission - Play Console Help for info on this oddity.

So, the only solution for us for now, is to prompt users to tell them to set their permissions themselves!

See Intermorphic Wotja 21 Help Center & FAQs for our user-facing documentation on this.

Pete

I actually was not actually able to get my app (Spacecraft Granular Synth) working when user and factory files were to be stored in the public documents/Spacecraft folder. So I had to force all users to store their data in the hidden internal folder. This is unfortunate for obvious reasons, namely that the users can’t easily access their own files, a la iOS AUv3.

However, now that I’ve made the change above, I will revisit this issue and see if it’s now magically fixed and that I can revert the location of the user files to the public folder going forward. If I do this, I’ll need to copy the existing user files from the hidden to the new public folder on startup for all users that have already installed my app. I believe you did something similar with your app, but perhaps in different circumstances.

I wasn’t able to figure out your MANAGE_EXTERNAL_STORAGE issue from the link you provided. It’s not clear how you are not able to publish with this in your build. Did google reject your build at the review stage? If so that’s a bit worrying as my app is currently in the review process having just made these very changes. Could you please explain a little more around the details of this issue you were having?

Android is such a nightmare for this stuff, but I also had my fair share if file access issues on iOS back in the day!

Hi again!

To be clear, the app (at start-up) checks to see if it can write to the public storage area. If it cannot then it prompts the user with an Alert, telling them how to fix permissions. If you were to download the free Wotja from Google Play, you can see how this looks. The user is then asked to restart the app. If they do this, then on next start, Wotja moves files from the private area, to the public area. The upshot is that, when they eventually get around to fixing their app permissions and restart the app, all their files are now in a public area with all the benefits!

The issue about the permission, is that (certainly last week!) the app was rejected when I tried uploading the aab. This was for both versions of Wotja that I tried uploading to Google Play. That led me to quickly create the above work-around.

Also, reading between the lines, I have a feeling that Google might eventually prevent apps from using MANAGE_EXTERNAL_STORAGE - as it is supposed to be used on by a few special cases of apps. I hope that the approach I’m using now will allow me to avoid using it; and it certainly seems to be working OK now.

Yes, iOS is much easier for your users to work with- provided you can persuade your users to use iCloud :slight_smile: Adding iCloud support to Wotja for iOS took me a lot of effort many years back - I’m certainly glad not to have to revisit it, and wouldn’t envy anybody who had to add that to an app from scratch.

The problem I have with Android, is that the File System access / permissions seem to change with each major version… maybe it isn’t actually the case, but it feels like the case :slight_smile:

Best wishes, Pete

The message I got (back on 5th May) when the aab files were rejected, was “Your APK or Android App Bundle requests the ‘android.permission.MANAGE_EXTERNAL_STORAGE’ permission, which Google Play doesn’t support yet.”

Hope that helps! Pete

Hi Pete,

I don’t understand why the user would have to manually change their permissions in the case of your app. When Spacecraft first starts up, a popup appears to ask to allow storage permissions, if the user accepts everything works (now at least) from that point onward. They don’t have to manually change permissions (in settings>apps>permissions for example).

Doesn’t your app also provide this popup on first start? I’m trying to figure out why the apps would behave differently. The only reasons I can think of would either be because I’m currently storing files in the hidden folder on my side or because in your case you were forced to remove MANAGE_EXTERNAL_STORAGE at the review stage!

My app seems to have passed the review with MANAGE_EXTERNAL_STORAGE included in AndroidManifest.xml. Not sure why they allowed my app and not yours, perhaps they’ve stopped blocking apps with this by now (?)

I contacted you via your website form submit to discuss something further

Hi @electropict!

IIRC - and my recollection isn’t certain, because as you know, there seem to be infinite possibilities in terms of permutations of permissions that might / might not make this work ! - things were OK until I had to remove the MANAGE_EXTERNAL_STORAGE permission (as Google Play is blocking uploading of apps that use that at the moment).

Hence the dance with the user to set permissions.

Anyhow: as you say, maybe Google have fixed the upload again!

But - bear in mind they might start rejecting apps that use MANAGE_EXTERNAL_STORAGE at some point.

Anyhow: what I’d suggest, is to double-check in Android settings to see whether or not your app has permissions set in:

    Privacy
    Permissions Manager
    Storage (or this might be called Files and Media, if e.g. using Emulator)
    Select your app ... is 'Allow Management of all files' set?

After the above check: if you uninstall your app, and re-install it - is that permission present, and set automatically?!

Best wishes,

Pete

What version is this user using of Android?

I have been going crazy trying to follow Google the last 3 years with this File stuff. They have changed things at least 5 times over 3 versions. 9, 10 and 11.

If your user is not on Android 11, I am pretty sure, I “may” be wrong but there is no way a user can get to a public space without custom vetting from Google specific to your app’s use case.

I think one cause of confusion is that as Android transits through these three version MAJOR things are happening leading to the ultimate demise of all File hacks into version 11.

If you are not testing on Android 11, you have no idea what the real end product of using these permissions are, unless you fully understand all this. Plus, since Google keeps moving goal posts, even people like me that are trying to keep up, are getting lost int he weeds. :slight_smile: