Weird createInputStream behaviour (Android)

When I check status.getErrorMessage() from

void FileInputStream::openHandle()
{
    auto f = open (file.getFullPathName().toUTF8(), O_RDONLY);

    if (f != -1)
        fileHandle = fdToVoidPointer (f);
    else
        status = getResultForErrno();
}

in juce_posix_SharedCode.h, I get the string ‘Permission denied’. However, I have Read From External Storage enabled and as you can see from my code above I’m using RuntimePermissions::readExternalStorage.

What am I doing wrong?

To properly narrow this down, you’ll have to check the gritty details that dictate permissions.

Does your manifest reflect this permission?

What Android API, SDK, NDK are you using/targetting?

What Android version do your test devices use?

Did you try reading your file from the main thread, just to rule out inter-thread funk?

Maybe you need to use the legacy storage depending on the device? android:requestLegacyExternalStorage="true ? Storage updates in Android 11  |  Android Developers

Maybe your file is inaccessible, but some OS versions are playing it too cool, so you have to move it elsewhere (assuming it’s something you control?)?

@jrlanglois Not sure what you mean about manifest but all phones I’ve tested show ‘storage’ permission under the app in settings > apps.

API 26 and above, gradle 6.5, android plug-in 4.1.2, SDK 31, NDK 21.0.6113669

phones that work: android 9 and android 11
phones that don’t work: android 10

I did try reading from the message thread initially and hoped that switching to the background thread would help, alas no.

I’d be interested in trying requestLegacyExternalStorage. How do I go about doing this (presumably in jucer)?

I’ve tried loading files for many different locations on the device, the issue is at createInputStream-level so it’s not related to where the file will be saved.

This post covers the ‘what’ as there are really 2 components to add to the manifest: https://stackoverflow.com/a/63365276

As for what the manifest is, it’s a file that controls what goes into an Android app and its configuration. Usually it’s named AndroidManifest.xml and has a manifest tag in there. You can add stuff to it through the Projucer in the Android configuration, under “Custom Manifest XML Content”.

Custom Manifest XML Content, that’s what I needed to know, thanks.

Could you please tell me how to go about implementing requestLegacyExternalStorage?

We’ve found you also need to prompt the user to update their Permissions.

You can take a look to see how we tell the user to handle this in the new Wotja for Android, which is going live tomorrow (21.8.0).

When Wotja starts, it checks to see if has permission to write to the (shared) Storage area.
a) If it does, it will create a Wotja folder under Documents (or Music or Download, in descending order of preference…)
b) if it doesn’t, it shows an alert warning the user about this Permissions setting, and telling them how to fix it; if the user follows the instructions, and then restarts Wotja [and then, a) should apply!]; Wotja then attempts to move any files that might have had in the “old” private area, to this shared area; but Wotja can only do this of course once the user has changed permissions…

NB: if the user hasn’t set your Permissions as per the alert, Wotja will continue to save files in private storage area.

Not ideal, but at least it now works! Required some patching to Juce, of course.

Pete

1 Like

I have finally managed to fix this issue! Thanks to a tip off from @jrlanglois and @peteatjuce, all of the info I needed was on the thread @jrlanglois posted, so thank you very much.

I’m now going to break down the details of exactly what was done in case anybody else is has this problem in Android in future. Everything is done in Builds/Android/app/src/main/AndroidManifest.xml (this is on Mac, windows path may be different, just search for AndroidManifest.xml).

The following is placed in the <manifest element alongside WRITE_EXTERNAL_STORAGE and READ_EXTERNAL_STORAGE which should also be there if you have enabled these in Jucer (which you should).

<uses-permission android:name="android.permission.ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION"/>
<uses-permission android:name="android.permission.MANAGE_EXTERNAL_STORAGE"/>

The following is added to the end of the <application element as an attribute

... android:requestLegacyExternalStorage="true">

Note that I also use RuntimePermissions::request when calling the top-level file reading function (see code in the original post above) and am also executing the file reading code in a dedicated background thread (again, see code in the original post above). If anybody is having issues recreating my implementation shown in the original post, contact me at deltavaudio@gmail.com as there are other bits of code behind the scenes.

Some subset of all of the above might also have fixed the issue, I’m not entirely sure. I’m just happy that it’s working so I’ll leave everything as is for now.

Another thing to note is the method of actually implementing this via Projucer. There is a field within the Android exporter called Custom Manifest XML Content where you can add things to AndroidManifest.xml. However, I wasn’t sure how to update this such that it would add an attribute to an existing XML element. So in the end I didn’t use the Custom Manifest XML Content field and instead I integrated the changes via an existing python build script which now saves the Projucer then does string replace in AndroidManifest.xml to make the changes I listed above, then either opens android studio or builds the APK/AAB file. Unless you find a way to make the Custom Manifest XML Content field add an attribute to an existing XML element, you may need to take a similar approach to my build script.

3 Likes

Great news!

The thing I’ve found is that, in general, our users want to save their documents to public storage, rather than the private internal area. That allows them to share their documents with other platforms (e.g. via Google File Transfer app), easily back them up etc. That is why resolving this is so important :slight_smile:

We’re currently unable to upload apps to Google Play that are built using MANAGE_EXTERNAL_STORAGE - see Use of All files access (MANAGE_EXTERNAL_STORAGE) permission - Play Console Help for info on this oddity.

So, the only solution for us for now, is to prompt users to tell them to set their permissions themselves!

See Intermorphic Wotja 21 Help Center & FAQs for our user-facing documentation on this.

Pete

I actually was not actually able to get my app (Spacecraft Granular Synth) working when user and factory files were to be stored in the public documents/Spacecraft folder. So I had to force all users to store their data in the hidden internal folder. This is unfortunate for obvious reasons, namely that the users can’t easily access their own files, a la iOS AUv3.

However, now that I’ve made the change above, I will revisit this issue and see if it’s now magically fixed and that I can revert the location of the user files to the public folder going forward. If I do this, I’ll need to copy the existing user files from the hidden to the new public folder on startup for all users that have already installed my app. I believe you did something similar with your app, but perhaps in different circumstances.

I wasn’t able to figure out your MANAGE_EXTERNAL_STORAGE issue from the link you provided. It’s not clear how you are not able to publish with this in your build. Did google reject your build at the review stage? If so that’s a bit worrying as my app is currently in the review process having just made these very changes. Could you please explain a little more around the details of this issue you were having?

Android is such a nightmare for this stuff, but I also had my fair share if file access issues on iOS back in the day!

Hi again!

To be clear, the app (at start-up) checks to see if it can write to the public storage area. If it cannot then it prompts the user with an Alert, telling them how to fix permissions. If you were to download the free Wotja from Google Play, you can see how this looks. The user is then asked to restart the app. If they do this, then on next start, Wotja moves files from the private area, to the public area. The upshot is that, when they eventually get around to fixing their app permissions and restart the app, all their files are now in a public area with all the benefits!

The issue about the permission, is that (certainly last week!) the app was rejected when I tried uploading the aab. This was for both versions of Wotja that I tried uploading to Google Play. That led me to quickly create the above work-around.

Also, reading between the lines, I have a feeling that Google might eventually prevent apps from using MANAGE_EXTERNAL_STORAGE - as it is supposed to be used on by a few special cases of apps. I hope that the approach I’m using now will allow me to avoid using it; and it certainly seems to be working OK now.

Yes, iOS is much easier for your users to work with- provided you can persuade your users to use iCloud :slight_smile: Adding iCloud support to Wotja for iOS took me a lot of effort many years back - I’m certainly glad not to have to revisit it, and wouldn’t envy anybody who had to add that to an app from scratch.

The problem I have with Android, is that the File System access / permissions seem to change with each major version… maybe it isn’t actually the case, but it feels like the case :slight_smile:

Best wishes, Pete

The message I got (back on 5th May) when the aab files were rejected, was “Your APK or Android App Bundle requests the ‘android.permission.MANAGE_EXTERNAL_STORAGE’ permission, which Google Play doesn’t support yet.”

Hope that helps! Pete

Hi Pete,

I don’t understand why the user would have to manually change their permissions in the case of your app. When Spacecraft first starts up, a popup appears to ask to allow storage permissions, if the user accepts everything works (now at least) from that point onward. They don’t have to manually change permissions (in settings>apps>permissions for example).

Doesn’t your app also provide this popup on first start? I’m trying to figure out why the apps would behave differently. The only reasons I can think of would either be because I’m currently storing files in the hidden folder on my side or because in your case you were forced to remove MANAGE_EXTERNAL_STORAGE at the review stage!

My app seems to have passed the review with MANAGE_EXTERNAL_STORAGE included in AndroidManifest.xml. Not sure why they allowed my app and not yours, perhaps they’ve stopped blocking apps with this by now (?)

I contacted you via your website form submit to discuss something further

Hi @electropict!

IIRC - and my recollection isn’t certain, because as you know, there seem to be infinite possibilities in terms of permutations of permissions that might / might not make this work ! - things were OK until I had to remove the MANAGE_EXTERNAL_STORAGE permission (as Google Play is blocking uploading of apps that use that at the moment).

Hence the dance with the user to set permissions.

Anyhow: as you say, maybe Google have fixed the upload again!

But - bear in mind they might start rejecting apps that use MANAGE_EXTERNAL_STORAGE at some point.

Anyhow: what I’d suggest, is to double-check in Android settings to see whether or not your app has permissions set in:

    Privacy
    Permissions Manager
    Storage (or this might be called Files and Media, if e.g. using Emulator)
    Select your app ... is 'Allow Management of all files' set?

After the above check: if you uninstall your app, and re-install it - is that permission present, and set automatically?!

Best wishes,

Pete

What version is this user using of Android?

I have been going crazy trying to follow Google the last 3 years with this File stuff. They have changed things at least 5 times over 3 versions. 9, 10 and 11.

If your user is not on Android 11, I am pretty sure, I “may” be wrong but there is no way a user can get to a public space without custom vetting from Google specific to your app’s use case.

I think one cause of confusion is that as Android transits through these three version MAJOR things are happening leading to the ultimate demise of all File hacks into version 11.

If you are not testing on Android 11, you have no idea what the real end product of using these permissions are, unless you fully understand all this. Plus, since Google keeps moving goal posts, even people like me that are trying to keep up, are getting lost int he weeds. :slight_smile:

As an aside I wrote these notes from a video of Google’s;

Android 11 September 8, 2020

  • Enabled File Paths APIs
  • Bulk media modifications APIs
  • All Files Access ← broadly read file system
  • Private app storage

Using File Paths

  • Allow devs to use file apis while keeping the new security features.
    • “FSE” File System - User Space; re-enabled File API with scoped storage
    • Under the hood file access is delegated to the MediaStore API
    • File path API is a convenience API to the MediaStore
  • Alternative to MediaStore APIs
  • Better compatibility
  • Java Files API
  • Native Libraries (C, C++)
  • Same scoped storage access
  • Immediately indexed by MediaStore (old way you had to scan the file)
  • MediaStore holds file path in ‘data’ column

Modifying Media

  • Can batch modify after confirmation from user
  • createWriteRquest()
  • createReadRequest()
  • createTrashRequest() (gives the user a chance to recover file later), deleted by OS after 30 days.
    • Can be untrashed by the file owner or user consent before 30 days
  • createFavoriteRequest()
    • works across apps

All Files Access

  • MANAGE_EXTERNAL_STORAGE
  • Write access to all shared files.
  • Special app access.
  • Google Play manual review.

Tips on Migrating

  • Legacy storage flag. (all files access/file paths)
    • To ensure correct behavior for users of Android 10
  • Storage access framework
    • Custom picking experience for custom files like documents. (does not require permissions)
  • Modifying media.
    • modifying media files you do not own, requires user consent (MediaStore consent APIs)
  • Top level directories (don’t save in custom top level directories)
    • requires the SAF which “is not the best experience for users”
    • App uninstalled and reinstalled, the app would loose access to the files
    • Use organized media collections or private storage instead
  • Content Providers
    • share data with other apps
    • most secure way to share files
1 Like

Oh my god this stuff is such a nightmare. That last post of yours @TeotiGraphix made my head spin!

With the latest updates I’ve made on my side (see all posts above) I now have file access working on android 11 phones (my Samsung note 20 ultra) and earlier android versions too (my app supports Oreo and above).

Note that the files my app stores are currently in a hidden internal folder at this point in time. However, I will again attempt to switch to a public folder after having made the recent changes I’ve made to file permission stuff. Who knows, it might magically work in a public folder now.

Hi Pete,

I sent a more detailed description to you via your website, but with my app the ‘allow management of all files’ is indeed set after opening my app for the first time and allowing storage permissions via the popup that appears the first time I load my app.

To answer your question. I just tried uninstalling then reinstalling my app. I found that if I don’t open the app it is denied permission, but after opening my app once and allowing media permissions via the popup, it changes to ‘allow management of all files’ thereon after.

Hope this is helpful

Yes, the safe private storage. It’s the public “root” that will be no longer accessible to normal apps.

The MANAGE_EXTERNAL_STORAGE permission is from my knowledge a verified by Google type thing and you have to really show you need it(they have said…). This in essence gives back old functionality of the File reading system down to the public root that users can see on other things like a PC.

FileExplorers etc would need this type of permission, audio apps seem to fit as well, after things move ahead a bit I will probably try Google with the MANAGE_EXTERNAL_STORAGE as well.

My apps currently don’t use JUCE. So I have implemented the Storage Access Framework and Content Providers for my use case. But storing public projects was what I did for years, and fast access to other audio directories.

If it turns out that I have to stick with the hidden internal folder in order to get all this stuff to work, I’ll just stick with that. It’s just a shame to have to limit the user experience like that. As I said, things are now finally working on android 11 phones and below so I’m pretty relieved.

2 Likes