The problem with big files

[szbk]

I want to reach the size of the video files on the device. Low size videos are okay (eg 50mb). But if the file size is too big, it takes too long (eg 4gb). Sometimes it doesn't work. How can I directly access the path of the file I selected? When the file size is large, cache loading is a problem. I am having this problem on android.

[miguelpruivo]

@sbilketay there isn't much to do here, unfortunately. Files are copy cached in chunks which is great to prevent memory issues, however, 4GB is too much for some devices to handle as you need always 4GB available + some space left.

If you have devices with 10/15 GB of left storage, it shouldn't be an issue (other than just taking the time to download it).

[szbk]

I'm not interested in downloading files. I want to get the size of a video that is already in the gallery. Is there a way to do this without waiting?

[miguelpruivo]

@sbilketay right now that isn’t possible. But it’s possible to add as a feature.

[tmthecoder]

I'm facing this same issue, caching the file then loading for larger files proves to be an issue when reading the files on devices, especially lower performance android devices. Is there a way to append an option to get a direct path?

[szbk]

☝️

[miguelpruivo]

@tmthecoder @sbilketay that's not possible since SDK 30. It is only possible on Android native (through URIs which allows you to manipulate files directly without relying on its original path, however Flutter doesn't support native Android URIs and needs absolute paths for files).

You can read more about it here in the first question.

[tmthecoder]

Oh, my bad, that's right I forgot about the Scoped Storage stuff on Android. I'm still wondering if there's a way to speed up the process as caching is one of the longest processes in my file work process.

[arianneorpilla]

I just want to write that I critically need a solution to this issue as well. My app is primarily a video player and some files I want to play are 4GB huge and crashes at file selection.

I can't seem to implement something like resuming video playback because that would require the video to be cached in the app's scoped storage.

My temporary solution is to offer separate versions for users below Android 11 and those on or above it, which is a bit of extra work. With legacy storage, videos play instantly. There isn't copying involved. No videos that have to be moved to cache and clearing after use. No videos that are too huge to play. It just worked.

Scoped storage has been incredibly high maintainance. Secure at the cost of mind numbingly slow limitations. I understand this must be hard to deal with but I would really appreciate any word on a solution to this problem.

[miguelpruivo]

@lrorpilla unfortunately, the only possible solution for this, is if Dart allows us to create File instances with plataform URIs, at least for Android. There’s no way around since even on native side you don’t use paths to access the file’s contents.

Other way I can think is if we could stream the data directly from each platform, which is doable with some effort I believe but might not fit your use case as well, since you need a File — that’s the bottleneck.

[isAlmogK]

Following this as we also ran into this issue

[tmthecoder]

Maybe we can add some sort of 'withDirect' parameter when opening the file which will stream the data directly from the platform (only applicable to android?). I'd be open to helping write this if needed

[yullg]

@lrorpilla unfortunately, the only possible solution for this, is if Dart allows us to create File instances with plataform URIs, at least for Android. There’s no way around since even on native side you don’t use paths to access the file’s contents.

Other way I can think is if we could stream the data directly from each platform, which is doable with some effort I believe but might not fit your use case as well, since you need a File — that’s the bottleneck.

If the APP has MANAGE_EXTERNAL_STORAGE permission, is it possible to use the original path directly?

[tony123S]

I get memory leakage issue when select a pdf file with size 40mb

[miguelpruivo]

Maybe we can add some sort of 'withDirect' parameter when opening the file which will stream the data directly from the platform (only applicable to android?). I'd be open to helping write this if needed

What would you return to Dart in this case?

[tmthecoder]

What would you return to Dart in this case?

I was thinking of a callback-style function that we can listen on in dart and add to a Stream with the bytes.

So my thoughts are: Start a BufferedReader in Java --> call Dart callback with the read bytes (of predetermined chunk size) --> add to a Stream<List<int>> that users can listen on

[miguelpruivo]

@tmthecoder well, that’s sort of the same that happens when you enable withReadStream while picking. Have you already tried?

[tmthecoder]

@miguelpruivo The withReadStream parameter helps in terms of convenience, but the stream is created from dart, so the file on Android has already been copied over to a locally accessible one, which is what I was aiming to stop. If we use the DocumentFile API to access the file on the Android end (if a specific parameter is set) we can bypass the copying and offer a direct stream of the file's data

[shilangyu]

Is there no way of disabling this caching and moving the responsibility to package users? Picking large files is unbearably slow. package:image_picker for example returns XFiles and explicitly states in the docs that using the path can cause errors and it is recommended to create a local copy. Is there then a similar alternative available in package:file_picker?

This also makes me wonder, how apps such as instagram are able to flawlessly handle huge local files (video) without any noticeable delay for caching.

[Dampfwalze]

I have this problem too, I am trying to pick files of an external USB device. The whole point of this is that I don't have enough memory on my device. This caching problem defeats the whole purpose of this! As a workaround, I am currently using getDirectoryPath() instead. This gives me the actual paths of the files and this works perfectly (not counting the hassle, that you can't really select the Videos you want). Then I am wondering why it isn't possible to get the path to the actual file with pickFiles() too.

Proposals:
Maybe you could use Symbolic Links to create a link in the cache that points to the actual file! The problem with this may be that FAT file systems (so most SD Cards) don't support them. But since the app cache should be in the internal storage, which, at least what I am aware of, is normally not a FAT file system, this should be no Problem!

Also, you could implement a switch to not cache files and just return an error, if the file exceeds a specified file size. Maybe even dependent on the storage left on the device. This wouldn't really solve the main Problem, but at least, the Application still works if the User tries to cache a file that is too large.

I really hope this can and will be fixed soon!

[miguelpruivo]

Android guidelines don’t allow you to have absolute path access since SDK 30 for files. It’s probably working for you when you use the directory path to access the files, but using the native API to launch the picker and then handling the intent result with the files, don’t allow me to access the absolute paths.

It was possible before, but then refactored due to this limitation. The recommended way to handle the original file is through its URI which is provided in the identifier property but isn’t much useful for a Dart File instance.

I’m not sure if what you’re asking is possible at all at this point, tbh.

[msarkrish]

@miguelpruivo Is there any way to restrict the file size while selecting file?

[miguelpruivo]

@msarkrish not that I'm aware of (at least, in all platforms). But you can always check the size after the user picks the file and then show an error to the user that the file is too big and discard it.

[arianneorpilla]

Thanks @abhi16180, I'll try to look into your fork sometime.

Just as an update because I still subscribe to this issue, I use filesystem_picker and supply root paths from external_path as an initial path and just list all the files in a directory and display them all in Flutter UI.

Really my use case just requires the absolute file path to be able to make a file for a video player, so I just use a non-native file picker made with Flutter that lists everything in a directory and allows navigating and changing directories. Really being able to list every file in a directory allows me to have the absolute path I need.

It's unfortunate I couldn't do this with the native picker plugins like this one, but to people who have a similar use case, you could consider what I did, it also allows you to have control over your UI.

[abhi16180]

Thanks @abhi16180, I'll try to look into your fork sometime.

Just as an update because I still subscribe to this issue, I use filesystem_picker and supply root paths from external_path as an initial path and just list all the files in a directory and display them all in Flutter UI.

Really my use case just requires the absolute file path to be able to make a file for a video player, so I just use a non-native file picker made with Flutter that lists everything in a directory and allows navigating and changing directories. Really being able to list every file in a directory allows me to have the absolute path I need.

It's unfortunate I couldn't do this with the native picker plugins like this one, but to people who have a similar use case, you could consider what I did, it also allows you to have control over your UI.

Yeah I will look into filesystem_picker package.
And there is a flutter file explorer application,it lists out all directories and file paths including downloads directory. I haven't checked how exactly it's been implemented. But it's really cool.
👇
FileX

[cnoons-cyq]

Hey @lrorpilla, I've been trying to do something similar to what you've explained here but I haven't dug too deeply to solve my issues yet.

Really my use case just requires the absolute file path to be able to make a file for a video player, so I just use a non-native file picker made with Flutter that lists everything in a directory and allows navigating and changing directories. Really being able to list every file in a directory allows me to have the absolute path I need.

I'll have to take a look into filesystem_picker myself, as I am aiming to display a list of videos too but from external storage. When you use filesystem_picker to list all of your video files in a UI to select for playing, do you then experience caching when you select a file to play or are you playing using the direct path?

[arianneorpilla]

@cnoons-cyq
It lets me pick the direct path. I then have access to relative files. My use case for using this is when I am picking a video file and I also want to get an SRT, or if I'm picking an HTML file and I also need access to relative images in a relative directory.

I've forked filesystem_picker for my purposes and it works great with no caching -- you get the absolute path you picked. I use it alongside external_paths to get roots to the internal storage and micro SD card if it exists on a device.

[miguelpruivo]

Is #1258 similar to this?

[abhi16180]

@miguelpruivo

Android 10+ still allows to access real path with "MANAGE_EXTERNAL_STORAGE" permission. I'm not familiar with native Android development, but I would suggest the below approach,

1. Ask permission to access the entire file system, and list out real paths.
2. If the user refuses to grant full permission, just fallback to current implementation.

[miguelpruivo]

@miguelpruivo

Android 10+ still allows to access real path with "MANAGE_EXTERNAL_STORAGE" permission. I'm not familiar with native Android development, but I would suggest the below approach,

1. Ask permission to access the entire file system, and list out real paths.
2. If the user refuses to grant full permission, just fallback to current implementation.

I got your point but I don't intend to let the users access/manipulate the original files directly with this to prevent, somehow, unexpected implementations and bad behavior when the user accidentally edits an original file in any given app. The original approach of the official image_picker was to always cache/copy the files, so I follow the same path as per good practices. The user can always delete the original file later if he wants to. Also, having to support this would make it a requisite for all platforms or else in Android it could pick the original file and in iOS the users would pick a copy. The dev would have to handle this cases.

Having a cache/copy behaves similar to mobile platforms. But I'm open to discuss alternatives.

[tomekit]

How about skipping expensive cache/copy and exposing content URI's on Android which then can be converted to Stream using this package: https://pub.dev/packages/uri_content ?

UPDATE:
Actually it works!
I've slightly modified the Android code to disable copy: 35f5c6d and get the size not from the file, but content URI, so no slow disk operation is required.
Disclaimer: This probably breaks: withData: true setting on Android, in which case empty bytes might be returned.

On the Flutter side user can get:

FilePickerResult? result = await FilePicker.platform.pickFiles(
  allowMultiple: true,
  allowCompression: false, 
  withReadStream: UniversalPlatform.isWeb
);

final uriContent = UriContent(); //  https://pub.dev/packages/uri_content 
for (final file in result.files) {
   final uri = Uri.parse(file.identifier!);
   final stream = uriContent.getContentStream(uri);
   // Then you can pass the stream to and enjoy your app not crashing if big file is used, alternatively if you need to process stream all at once, then use: "content = await stream.expand((element) => element).toList();"
}

The only drawback so far that I've experienced is that: getContentStream itself uses Platform code and won't work from within the Isolate / Compute.

[helomri]

flutter/flutter#147037 could solve this issue directly with Flutter's XFile (or a new Flutter package using XFiles). XFile support got added in file_picker so if this is done files won't need to be copied to be accessed. If a contributor could give feedback on if one of the option I proposed is feasible to implement effectively in file_picker. (the second one looks like it is the best one)

[J-avery32]

Are there any PRs for this issue or plans to fix it?

[tomekit]

It seems that there are two possible solutions on the radar.

a) flutter/flutter#147037 is feasible once implemented by Flutter team as mentioned by @theskyblockman comment above.

b) Solution mentioned here: #561 (comment) already works, except it doesn't work from a background isolate until Flutter team resolves this: flutter/flutter#119207 where they already make some progress.

[SupinePandora43]

@miguelpruivo
Android 10+ still allows to access real path with "MANAGE_EXTERNAL_STORAGE" permission. I'm not familiar with native Android development, but I would suggest the below approach,

1. Ask permission to access the entire file system, and list out real paths.
2. If the user refuses to grant full permission, just fallback to current implementation.

I got your point but I don't intend to let the users access/manipulate the original files directly with this to prevent, somehow, unexpected implementations and bad behavior when the user accidentally edits an original file in any given app. The original approach of the official image_picker was to always cache/copy the files, so I follow the same path as per good practices. The user can always delete the original file later if he wants to. Also, having to support this would make it a requisite for all platforms or else in Android it could pick the original file and in iOS the users would pick a copy. The dev would have to handle this cases.

Having a cache/copy behaves similar to mobile platforms. But I'm open to discuss alternatives.

Since android 11 there's READ_EXTERNAL_STORAGE permission

[devfemibadmus]

@miguelpruivo
Android 10+ still allows to access real path with "MANAGE_EXTERNAL_STORAGE" permission. I'm not familiar with native Android development, but I would suggest the below approach,

1. Ask permission to access the entire file system, and list out real paths.
2. If the user refuses to grant full permission, just fallback to current implementation.

I got your point but I don't intend to let the users access/manipulate the original files directly with this to prevent, somehow, unexpected implementations and bad behavior when the user accidentally edits an original file in any given app. The original approach of the official image_picker was to always cache/copy the files, so I follow the same path as per good practices. The user can always delete the original file later if he wants to. Also, having to support this would make it a requisite for all platforms or else in Android it could pick the original file and in iOS the users would pick a copy. The dev would have to handle this cases.
Having a cache/copy behaves similar to mobile platforms. But I'm open to discuss alternatives.

Since android 11 there's READ_EXTERNAL_STORAGE permission

thats permission is too big what if all we need is just access to specific folder

[iulian0512]

@miguelpruivo when are the cached files deleted ? i am worried that the device will run out of space, my use case handles multiple large (100-500mb) files.

[helomri]

@miguelpruivo when are the cached files deleted ? i am worried that the device will run out of space, my use case handles multiple large (100-500mb) files.

@iulian0512 You have to delete the files yourself, they are kept in your apps data directory.

[iulian0512]

@miguelpruivo when are the cached files deleted ? i am worried that the device will run out of space, my use case handles multiple large (100-500mb) files.

@iulian0512 You have to delete the files yourself, they are kept in your apps data directory.

the file_picker i noticed that it makes those files when picking files, so my initial assumption was that it might handle its cache.

[tomekit]

If any of guys have a snippet that can cleanup these cache files that would be welcomed and likely help everyone here to deal with the cache size issue.

In the meantime I believe this problem shall be tackled, so no cache files are needed at all. No cache files, no cleanup needed.

[iulian0512]

i think for android it's best to use the document api https://developer.android.com/training/data-storage/shared/documents-files

[helomri]

Sure, here's a snippet to delete all the files your user picked (not all of them).

for (PlatformFile file in pickedFiles.files) {
    File(file.path).deleteSync();
}

pickedFiles here being your instance of FilePickerResult.

[tomekit]

@iulian0512
Yes, however that's Kotlin and need to be implemented for Flutter, there are two solutions available so far: #561 (comment)

@helomri
That's great!

Is there anyway to trigger cleanup e.g. periodically or when app starts or when file picker is requested?
Solution that you've mentioned would cleanup most files, most of the time, but there maybe cases when app fails, e.g. memory crash etc. In such case it would leave left-overs that would never be deleted.
Is it possible to target these files somehow, are they using some prefix/suffix or dedicated location within the data directory?

[iulian0512]

@tomekit the getContentStream() from https://pub.dev/packages/uri_content is feasible only for read operations, but if you need to write to a file say we need to save a file to a user defined location no bueno.
the other option would be nice to be implemented out of the box by flutter team and provide a IOSink (File.openWrite())

[helomri]

@helomri That's great!

Is there anyway to trigger cleanup e.g. periodically or when app starts or when file picker is requested? Solution that you've mentioned would cleanup most files, most of the time, but there maybe cases when app fails, e.g. memory crash etc. In such case it would leave left-overs that would never be deleted. Is it possible to target these files somehow, are they using some prefix/suffix or dedicated location within the data directory?

You can simply use .clearTemporaryFiles() to clear all the temporary files and run it in your main for example (I recommend not awaiting it as it could slow down your app's startup time)

[helomri]

@tomekit the getContentStream() from https://pub.dev/packages/uri_content is feasible only for read operations, but if you need to write to a file say we need to save a file to a user defined location no bueno. the other option would be nice to be implemented out of the box by flutter team and provide a IOSink (File.openWrite())

I am supposed to write an implementation in cross_file for Android SAF/content:// URIs but to do that I first need the package to become federated and receive its breaking changes before proceeding (see flutter/flutter#147037)

[tomekit]

@iulian0512
As far I am concerned this issue is mostly about problem with reading big files and uri_content package as it is currently, satisfies that requirement.
It's also demonstrating the approach of Kotlin<>Flutter communication that could possibly be utilized for file writes: https://github.com/talesbarreto/uri_content/blob/main/android/src/main/kotlin/com/talesbarreto/uri_content/UriContentPlugin.kt#L87
I am not expert here though, so take it with a grain of salt.

Do you happen to know how this package: https://pub.dev/packages/file_picker saveFile() works on Android, does it still need to use cache or it writes files directly?

[helomri]

Do you happen to know how this package: https://pub.dev/packages/file_picker saveFile() works on Android, does it still need to use cache or it writes files directly?

Looking at

flutter_file_picker/android/src/main/java/com/mr/flutter/plugin/filepicker/FilePickerPlugin.java

Line 141 in 7726f3c

this.delegate.saveFile(fileName, type, initialDirectory, allowedExtensions, bytes,result);

I can assure you no file is ever created in cache but the whole file is read into memory and sent through platform channels to the native side, which is obviously a gigantic problem (do NOT use it if you are saving user-generated files with unlimited size).

[iulian0512]

Do you happen to know how this package: https://pub.dev/packages/file_picker saveFile() works on Android, does it still need to use cache or it writes files directly?

as @helomri said