Improve FileSystem detection #1663

macgills · 2019-12-19T09:47:14Z

Is your feature request related to a problem? Please describe.
Currently filesystem detection can take upwards of 1min and cause problems for users #1662 . This is because we write a file 4GB in size and see if it throws an exception or not.

Describe the solution you'd like
Use the mount command and do some string processing to extract the filesystem

Another ticket proposing a cache system #1664

kelson42 · 2019-12-19T09:48:35Z

See also kiwix/libkiwix#267

kelson42 · 2019-12-19T09:51:39Z

Describe the solution you'd like
Use the mount command and do some string processing to extract the filesystem

@macgills How would that work exactly?

macgills · 2019-12-19T09:55:46Z

 try{
        Process mount = Runtime.getRuntime().exec("mount");
        BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(mount.getInputStream()));
        mount.waitFor();

        String extPath = Environment.getExternalStorageDirectory().getAbsolutePath();
        String line;
        while ((line = bufferedReader.readLine()) != null)
        {
            String[] split = line.split("\\s+");
            for(int i = 0; i < split.length - 1; i++)
            {
                if(split[i].contentEquals(extPath) ||
                    split[i].contains("sdcard") ||
                    split[i].contains("_sd") ||
                    split[i].contains("extSd") ||
                    split[i].contains("_SD"))   // Add wildcards to match against here
                {
                    String strMount = split[i];
                    String strFileSystem = split[i+1];

                    // Add to a list/array of mount points and file systems here...
                    Log.i("SDCard", "mount point: "+ strMount + " file system: " + strFileSystem);
                }
            }
        }           
    }catch(IOException e){
        e.printStackTrace();            
    }catch(InterruptedException e){
        e.printStackTrace();            
    }

like this but a bit more complicated.
Execute mount command
Parse lines of text into models
iterate over models finding partial matches on the current preferred storage
select best matching model
check the filesystem string again a list of known FSs that can support 4GB
report that the FS can write 4GB files

kelson42 · 2019-12-19T10:06:58Z

@macgills So the mount command gives you the list of mounted devices and for each devices the type of the filesystem and you will the be able to know which one are fat32. Questions:

Are you really sure you have the always the right to execute that command?
Are you sure the mount output is always the same (depending the version Android)?
Why this has not been done before (this seems simple)?
On linux, you have a similar result with cat /proc/mounts, if it works might be better to just read this (there is a class here https://gist.github.com/ikonst/3394662)?

macgills · 2019-12-19T10:17:19Z

I think so
not positive
the rest of these answers
none of the SO answers I have seen have talked about that, they start talking about vold sometimes though

When I implemented an experiment it didn't give me an immediately easy answer say for example we have mount output like so

somedevice realdir exfat
realDir sdCardDir fake

when I iterated through the list the entry I matched on for the directory I was looking for sdcardDir/probably/some/other/path was the 2nd one but it didn't have a valid entry for a file system so I had to redo the search using the realDir to retrieve the result of exfat/CanWrite4GB

kelson42 · 2019-12-19T14:11:36Z

@macgills I'm not super confident on relying on a system command execution, but if we rely on a Linux Kernel /proc file then I'm more confident. If you can deal with one of these files /proc/mounts or /proc/self/mountinfo I would be quite supportive to try to implement that ticket in 3.2. See also:
https://stackoverflow.com/questions/11668284/is-there-any-limitations-in-using-proc-mounts-in-android

macgills · 2019-12-19T14:26:06Z

It'll take experimentation to figure out which is most reliable, mount probably just reads proc/mounts, if proc/self/mountinfos exists and can be read by us then it sounds like a good option

macgills · 2019-12-19T14:50:24Z

And to clarify I will be keeping the old implementation, just trying this one first and on failure falling back. Commonsware back in 2014 was warning of these apis when android introduced support for sd cards

kelson42 · 2019-12-19T17:24:15Z

@macgills OK, seems worth the effort.

macgills · 2020-01-08T15:26:15Z

Logs extracted on Nexus 8.
The storage we are trying to identify, note this is on the device

E/Storage: /storage/emulated/0/Android/data/org.kiwix.kiwixmobile/files

Output of procs/self/mountinfo

E/MOUNTINFO: 316 316 0:1 / / ro master:1 - rootfs rootfs ro,seclabel
E/MOUNTINFO: 317 316 0:12 / /dev rw,nosuid,relatime master:2 - tmpfs tmpfs rw,seclabel,size=923024k,nr_inodes=230756,mode=755
E/MOUNTINFO: 318 317 0:9 / /dev/pts rw,relatime master:3 - devpts devpts rw,seclabel,mode=600
E/MOUNTINFO: 319 317 0:16 / /dev/cpuctl rw,relatime master:4 - cgroup none rw,cpu
E/MOUNTINFO: 320 317 0:17 / /dev/cpuset rw,relatime master:5 - cgroup none rw,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent
E/MOUNTINFO: 321 317 0:20 / /dev/usb-ffs/adb rw,noatime master:20 - functionfs adb rw
E/MOUNTINFO: 322 316 0:3 / /proc rw,relatime master:6 - proc proc rw,gid=3009,hidepid=2
E/MOUNTINFO: 323 316 0:13 / /sys rw,relatime master:7 - sysfs sysfs rw,seclabel
E/MOUNTINFO: 324 323 0:11 / /sys/fs/selinux rw,relatime master:8 - selinuxfs selinuxfs rw
E/MOUNTINFO: 325 323 0:5 / /sys/kernel/debug rw,relatime master:9 - debugfs debugfs rw,seclabel
E/MOUNTINFO: 326 323 0:18 / /sys/fs/pstore rw,relatime master:10 - pstore pstore rw,seclabel
E/MOUNTINFO: 327 316 0:14 / /acct rw,relatime master:11 - cgroup none rw,cpuacct
E/MOUNTINFO: 328 316 0:15 / /mnt rw,relatime master:12 - tmpfs tmpfs rw,seclabel,size=923024k,nr_inodes=230756,mode=755,gid=1000
E/MOUNTINFO: 329 328 0:19 / /mnt/runtime/default/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media rw,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid
E/MOUNTINFO: 330 328 0:19 / /mnt/runtime/read/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media rw,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid
E/MOUNTINFO: 331 328 0:19 / /mnt/runtime/write/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media rw,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal
E/MOUNTINFO: 332 316 0:10 / /config rw,relatime master:13 - configfs none rw
E/MOUNTINFO: 333 316 253:0 / /system ro,relatime master:14 - ext4 /dev/block/dm-0 ro,seclabel,inode_readahead_blks=8
E/MOUNTINFO: 334 316 253:1 / /vendor ro,relatime master:15 - ext4 /dev/block/dm-1 ro,seclabel,inode_readahead_blks=8
E/MOUNTINFO: 335 316 259:8 / /cache rw,nosuid,nodev,noatime master:17 - ext4 /dev/block/platform/soc.0/f9824900.sdhci/by-name/cache rw,seclabel,noauto_da_alloc,errors=panic,data=ordered
E/MOUNTINFO: 336 316 179:24 / /persist rw,nosuid,nodev,noatime master:18 - ext4 /dev/block/platform/soc.0/f9824900.sdhci/by-name/persist rw,seclabel,nodelalloc,errors=panic,data=ordered
E/MOUNTINFO: 337 316 179:1 / /firmware ro,relatime master:19 - vfat /dev/block/platform/soc.0/f9824900.sdhci/by-name/modem ro,context=u:object_r:firmware_file:s0,uid=1000,gid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro
E/MOUNTINFO: 338 316 253:2 / /data rw,nosuid,nodev,noatime master:16 - ext4 /dev/block/dm-2 rw,seclabel,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8
E/MOUNTINFO: 339 316 0:15 /runtime/write /storage rw,relatime master:12 - tmpfs tmpfs rw,seclabel,size=923024k,nr_inodes=230756,mode=755,gid=1000
E/MOUNTINFO: 340 339 0:19 / /storage/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media rw,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal
E/MOUNTINFO: 341 339 0:15 /user/0 /storage/self rw,relatime master:12 - tmpfs tmpfs rw,seclabel,size=923024k,nr_inodes=230756,mode=755,gid=1000

Output of proc/mounts

E/MOUNTS: rootfs / rootfs ro,seclabel 0 0
E/MOUNTS: tmpfs /dev tmpfs rw,seclabel,nosuid,relatime,size=923024k,nr_inodes=230756,mode=755 0 0
E/MOUNTS: devpts /dev/pts devpts rw,seclabel,relatime,mode=600 0 0
E/MOUNTS: none /dev/cpuctl cgroup rw,relatime,cpu 0 0
E/MOUNTS: none /dev/cpuset cgroup rw,relatime,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent 0 0
E/MOUNTS: adb /dev/usb-ffs/adb functionfs rw,noatime 0 0
E/MOUNTS: proc /proc proc rw,relatime,gid=3009,hidepid=2 0 0
E/MOUNTS: sysfs /sys sysfs rw,seclabel,relatime 0 0
E/MOUNTS: selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
E/MOUNTS: debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0
E/MOUNTS: pstore /sys/fs/pstore pstore rw,seclabel,relatime 0 0
E/MOUNTS: none /acct cgroup rw,relatime,cpuacct 0 0
E/MOUNTS: tmpfs /mnt tmpfs rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000 0 0
E/MOUNTS: /data/media /mnt/runtime/default/emulated sdcardfs rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid 0 0
E/MOUNTS: /data/media /mnt/runtime/read/emulated sdcardfs rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid 0 0
E/MOUNTS: /data/media /mnt/runtime/write/emulated sdcardfs rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal 0 0
E/MOUNTS: none /config configfs rw,relatime 0 0
E/MOUNTS: /dev/block/dm-0 /system ext4 ro,seclabel,relatime,inode_readahead_blks=8 0 0
E/MOUNTS: /dev/block/dm-1 /vendor ext4 ro,seclabel,relatime,inode_readahead_blks=8 0 0
E/MOUNTS: /dev/block/platform/soc.0/f9824900.sdhci/by-name/cache /cache ext4 rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered 0 0
E/MOUNTS: /dev/block/platform/soc.0/f9824900.sdhci/by-name/persist /persist ext4 rw,seclabel,nosuid,nodev,noatime,nodelalloc,errors=panic,data=ordered 0 0
E/MOUNTS: /dev/block/platform/soc.0/f9824900.sdhci/by-name/modem /firmware vfat ro,context=u:object_r:firmware_file:s0,relatime,uid=1000,gid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro 0 0
E/MOUNTS: /dev/block/dm-2 /data ext4 rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8 0 0
E/MOUNTS: tmpfs /storage tmpfs rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000 0 0
E/MOUNTS: /data/media /storage/emulated sdcardfs rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal 0 0
E/MOUNTS: tmpfs /storage/self tmpfs rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000 0 0

output of mount command

E/COMMAND: rootfs on / type rootfs (ro,seclabel)
E/COMMAND: tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,size=923024k,nr_inodes=230756,mode=755)
E/COMMAND: devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600)
E/COMMAND: none on /dev/cpuctl type cgroup (rw,relatime,cpu)
E/COMMAND: none on /dev/cpuset type cgroup (rw,relatime,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent)
E/COMMAND: adb on /dev/usb-ffs/adb type functionfs (rw,noatime)
E/COMMAND: proc on /proc type proc (rw,relatime,gid=3009,hidepid=2)
E/COMMAND: sysfs on /sys type sysfs (rw,seclabel,relatime)
E/COMMAND: selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
E/COMMAND: debugfs on /sys/kernel/debug type debugfs (rw,seclabel,relatime)
E/COMMAND: pstore on /sys/fs/pstore type pstore (rw,seclabel,relatime)
E/COMMAND: none on /acct type cgroup (rw,relatime,cpuacct)
E/COMMAND: tmpfs on /mnt type tmpfs (rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000)
E/COMMAND: /data/media on /mnt/runtime/default/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid)
E/COMMAND: /data/media on /mnt/runtime/read/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid)
E/COMMAND: /data/media on /mnt/runtime/write/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal)
E/COMMAND: none on /config type configfs (rw,relatime)
E/COMMAND: /dev/block/dm-0 on /system type ext4 (ro,seclabel,relatime,inode_readahead_blks=8)
E/COMMAND: /dev/block/dm-1 on /vendor type ext4 (ro,seclabel,relatime,inode_readahead_blks=8)
E/COMMAND: /dev/block/platform/soc.0/f9824900.sdhci/by-name/cache on /cache type ext4 (rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered)
E/COMMAND: /dev/block/platform/soc.0/f9824900.sdhci/by-name/persist on /persist type ext4 (rw,seclabel,nosuid,nodev,noatime,nodelalloc,errors=panic,data=ordered)
E/COMMAND: /dev/block/platform/soc.0/f9824900.sdhci/by-name/modem on /firmware type vfat (ro,context=u:object_r:firmware_file:s0,relatime,uid=1000,gid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro)
E/COMMAND: /dev/block/dm-2 on /data type ext4 (rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8)
E/COMMAND: tmpfs on /storage type tmpfs (rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000)
E/COMMAND: /data/media on /storage/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal)
E/COMMAND: tmpfs on /storage/self type tmpfs (rw,seclabel,relatime,size=923024k,nr_inodes=230756,mode=755,gid=1000)

All 3 outputs are much the same
for mountinfo we would find this entry

E/MOUNTINFO: 340 339 0:19 / /storage/emulated rw,nosuid,nodev,noexec,noatime master:21 - sdcardfs /data/media

It gives us a useless value for the filesystem sdcardfs but gives us a directory /data/media which when we scan for again gives us

E/MOUNTINFO: 338 316 253:2 / /data rw,nosuid,nodev,noatime master:16 - ext4 /dev/block/dm-2 rw,seclabel,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8

which gives us the useful value of ext4 which is a real filesystem and does support 4gb files.

I'll probably read from proc/mounts as it has the least fluff in the way of the data I want eg device, mount point and filesystem

…file writing

…moving visited entries - add vfat to <4GB filesystems

…detection #1663 Improve FileSystem detection - read from mount and fallback to …

kelson42 added the enhancement label Dec 19, 2019

kelson42 mentioned this issue Dec 19, 2019

Add a cache to avoid checking filesystem too often (might be slow) #1664

Closed

kelson42 assigned macgills Dec 19, 2019

kelson42 added this to the 3.2 milestone Dec 19, 2019

macgills added a commit that referenced this issue Jan 9, 2020

#1663 Improve FileSystem detection - read from mount and fallback to …

0779080

…file writing

macgills mentioned this issue Jan 9, 2020

#1663 Improve FileSystem detection - read from mount and fallback to … #1692

Merged

macgills added a commit that referenced this issue Jan 9, 2020

#1663 Improve FileSystem detection - prevent infinite recursion by re…

cbee2b9

…moving visited entries - add vfat to <4GB filesystems

macgills closed this as completed in #1692 Jan 13, 2020

macgills added a commit that referenced this issue Jan 13, 2020

Merge pull request #1692 from kiwix/feature/macgills/1663-filesystem-…

e11f421

…detection #1663 Improve FileSystem detection - read from mount and fallback to …

MohitMaliFtechiz mentioned this issue Aug 28, 2023

This APP Performs 4GB Large File Write Test which Seriously Harms the Phone/External SDCard's Flash Memory Lifespan #2889

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve FileSystem detection #1663

Improve FileSystem detection #1663

macgills commented Dec 19, 2019 •

edited by kelson42

Loading

kelson42 commented Dec 19, 2019

kelson42 commented Dec 19, 2019 •

edited

Loading

macgills commented Dec 19, 2019

kelson42 commented Dec 19, 2019 •

edited

Loading

macgills commented Dec 19, 2019

kelson42 commented Dec 19, 2019

macgills commented Dec 19, 2019

macgills commented Dec 19, 2019

kelson42 commented Dec 19, 2019

macgills commented Jan 8, 2020

Improve FileSystem detection #1663

Improve FileSystem detection #1663

Comments

macgills commented Dec 19, 2019 • edited by kelson42 Loading

kelson42 commented Dec 19, 2019

kelson42 commented Dec 19, 2019 • edited Loading

macgills commented Dec 19, 2019

kelson42 commented Dec 19, 2019 • edited Loading

macgills commented Dec 19, 2019

kelson42 commented Dec 19, 2019

macgills commented Dec 19, 2019

macgills commented Dec 19, 2019

kelson42 commented Dec 19, 2019

macgills commented Jan 8, 2020

macgills commented Dec 19, 2019 •

edited by kelson42

Loading

kelson42 commented Dec 19, 2019 •

edited

Loading

kelson42 commented Dec 19, 2019 •

edited

Loading