Detect FileSystem from File

[macgills]

In the android app we currently detect the filesystem of the users chosen storage device by writing a 4GB file to it and if it throws an exception deciding that it is FAT32.

 private fun canCreate4GbFile(storage: String): Boolean {
    val path = "$storage/large_file_test.txt"
    File(path).delete()
    try {
      RandomAccessFile(path, "rw").use {
        it.setLength(FOUR_GIGABYTES_IN_BYTES)
        return@canCreate4GbFile true
      }
    } catch (e: Exception) {
      e.printStackTrace()
      Log.d("Fat32Checker", e.message)
      return false
    } finally {
      File(path).delete()
    }
  }

This is achieved using RandomAccessFile.setLength() which is currently giving terrible performance when used on an sd card ,1 minute vs the usual 1ms on internal storage.

setLength should be using the most optimised C calls based on the OS it is running on but it seems like it is not in this scenario.

An optimised call in kiwix-lib should hopefully alleviate this scenario.

[mgautierfr]

I don't think that this should be in kiwix-lib. This is not really related to zim file/reading. It is not because that a thing takes time in java word that we have to push it in kiwix-lib.

More over, as it is said on the link you gave, it.setLength should already call ftruncate function on linux. I'm not sure that calling it ourself in kiwix-lib will help here. The problem is maybe simply that if the fs doesn't support sparse file, ftruncate has to allocate (and zero) the whole 4GB (https://www.linuxquestions.org/questions/linux-kernel-70/the-question-about-ftruncate-function-4175550352/)

[kelson42]

@mgautierfr Considering that this is a function which is really necessary for Kiwix Android but not only (we should have the same check on Kiwix Desktop for example), I'm really in favour to have this primitive in the Kiwix lib. The Kiwix lib is not only about zim file/reading, this is about sharing code base between multiple ports. What kind of concrete alternative you propose here as the problem we face is quite serious and needs to be tacle rather soon than later?

[mgautierfr]

What kind of concrete alternative you propose here as the problem we face is quite serious and needs to be tacle rather soon than later?

I don't know for now :)
Probably a call to some OS specific functions to get the fs used.
But creating a 4Gib file to check if we can create it is globally a bad idea.

[macgills]

There was the option of using the mount command but on implementation I found it to be deeply unreliable

[mgautierfr]

We could directly read the /proc/mount pseudo file that mostly contains the same thing that what return the mount command.

Why do you found it unreliable ? (I haven't read the whole algorithm you link, but we don't have to blindly copy it)

[macgills]

I found what it was reporting inconsistent (file system info missing/in wrong place) if I remember correctly, it has been a good few months but I remember my distaste and abandoning it to pursue the 4GB files solution.

[kelson42]

@macgills @mgautierfr We should carry on this before 3.0 release. It is simpy not acceptable to have users waiting for one minute to know if a fs can support file over 4GB. I see a massive problem on 2.5.x from people simply not able to get the online catalog and I think this is part of the problem.

[macgills]

It is not a universal problem. Only some devices are affected but I do not know the prevalence.
I have also made an update that user do not need to wait to detect the filesystem, I emit Unknown state at the start of checking the file system and just don't present 4GB files until I know they can be downloaded

[kelson42]

@macgills Thank you for the update. This makes the whole topic a bit less urgent, but considering that @mgautierfr won't be available for a few months. I'm concerned if we can live with this until next year. In general, even if 2.5.3 has improved a lot the stability, the overall rating are not better and I try to understand why so many people complain about missing catalogue. Any other theory is welcome... but so far, this ticket is the only one I know about.

[macgills]

That is a potential reason for the library taking a long time to load and will be resolved for 3.0.
The library not loading on certain devices is still largely unknown because there is no repro case but I think could be improved with https and cert pinning, that is only a hunch though.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

Hmm.. not really, because the problem is not only with Android, it is everywhere.We have still not good solution for other ports.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.