How Does Firestore Cache Management Work and What Causes Data Loss in Offline Mode?

[vinhhoangbepos]

Hello @dconeybe and Firebase team,

We've recently encountered an issue with data loss in our app, specifically when operating in offline mode. To provide more context, here is our current Firestore cache configuration:
FIRFirestoreSettings *settings = [[FIRFirestoreSettings alloc] init];
settings.cacheSizeBytes = 50 * 1024 * 1024; // 50 MB
settings.persistenceEnabled = YES;
[FIRFirestore firestore].settings = settings;
My question is about the handling of newly added documents in Firestore while in offline mode, especially concerning the cache size limit. Specifically:

• Are documents added while the app is offline counted towards the cache size (cacheSizeBytes)?
• If the total size of newly added documents (while offline) exceeds the cache size limit, will these documents be subject to cleanup?
• Furthermore, could this behavior contribute to the data loss issues we are experiencing when the app is offline?

Thank you.

[dconeybe]

Hi @vinhhoangbepos. The cache size limit should not be causing any data loss. Data is only evicted from the cache if it is no longer needed. For example, documents that are added while offline and documents from a query that has an active snapshot listener are not candidate for eviction from the cache. In contrast, the result of a query that was run last week are candidate for eviction from the cache. So even if the cache size limit is exceeded the data will only be evicted once it is no longer needed.

Could you describe the "data loss" that you are observing in more detail? Unfortunately, the term "data loss" describes a large class of problems and specifics could be helpful in this case. For example, are you observing that you create documents and they "disappear"?

You could also try enabling Firestore debug logs by calling [FIRFirestore enableLogging:YES] which should reveal which, if any, documents are being evicted from the cache.

[dconeybe]

@vinhhoangbepos I don't have the answers to your questions off the top of my head. However, here are some thoughts based on your response.

orders that were successfully processed are later not found in the finished orders list at the end of the day.

At the point where the orders are noticed to be missing, has the device gone back online yet? Or would it still be offline? I'm curious if the device has had a chance to sync with the server.

Roughly how many documents appear to go "missing"? Is it just 1 or 2? Or is it a large chunk? Do you know if the missing orders are continuous or are they sporadic?

Note that FIRWriteBatch has a limit of 500 documents (https://firebase.google.com/docs/firestore/manage-data/transactions). Is it possible that this limit is being exceeded?

Although I've described how the cache should behave, it's not impossible that there is a bug. Could you see if setting the cache size to kFIRFirestoreCacheSizeUnlimited fixes the problem?

[vinhhoangbepos]

@dconeybe

At the point where the orders are noticed to be missing, has the device gone back online yet? Or would it still be offline? I'm curious if the device has had a chance to sync with the server.

We have recommended our users maintain a stable internet connection, so the scenario is most likely when the device is online. However, there is still a possibility that it could be offline at that time, though we are not certain about this.

Roughly how many documents appear to go "missing"? Is it just 1 or 2? Or is it a large chunk? Do you know if the missing orders are continuous or are they sporadic?

We have encountered both scenarios. Some users have reported sporadically missing orders, while others have reported continuous missing orders over a period of time. The number of missing documents varies, with some instances involving just a few orders and others involving more than 100 orders missing continuously over three days.

Note that FIRWriteBatch has a limit of 500 documents (https://firebase.google.com/docs/firestore/manage-data/transactions). Is it possible that this limit is being exceeded?

We have implemented logic to ensure that the 500-document limit per batch is never exceeded. In the cases of reported missing orders, the batch size tends to be relatively small due to the small number of items in the orders.

[dconeybe]

Hmm, there are a lot of variables in play here and nothing immediately comes to mind that would explain the data loss that you are experiencing. A FIRWriteBatch can fail to commit for any number of reasons, including quota exceeded or a security rules violation. Without knowing the error message and being unfamiliar with your app I can only guess at the root cause.

I think your best course of action here is to enable Firestore debug logging on the customer's device and extract the logs from their device as soon as possible after the data loss is observed. Those logs should indicate the success/failure of the write batches and any data that was evicted from the cache.

[vinhhoangbepos]

Hello, @dconeybe,

We have identified the cause of our data loss issue by monitoring FIRWriteBatch commitWithCompletion. The error returned from the server is: "Domain: FIRFirestoreErrorDomain Code: 7 NSLocalizedDescription: Missing or insufficient permissions."

From my understanding, this error typically occurs with unauthenticated users. However, the users in this case were logged in when this error occurred. The issue appears to be resolved after the users uninstall and then reinstall the app, followed by a fresh login. These errors seem to arise randomly, without the users changing their passwords or performing any other unexpected actions.

Could you offer any insights into what might cause this error to occur under these circumstances?

Thank you for your support.

[dconeybe]

Hi @vinhhoangbepos. Apologies for the delayed response. I just returned from vacation.

That error would definitely explain why you are observing missing data. Unfortunately, it's a very generic error that can have a multitude of causes. Off the top of my head, my immediate guess would be that the auth token that Firestore acquired before the device went offline expired while the device was offline, then was used when the device went back online to send the write batch, leading to the permissions error. Firestore is supposed to automatically refresh the auth token when it expires, but it is plausible that there is a bug where Firestore incorrectly uses an expired auth token. Note that this is just a guess and, if true, there isn't really anything you can do to work around it.

I'd recommend that you open a bug report at https://github.com/firebase/firebase-ios-sdk/issues/new?template=BUG_REPORT.yml. This problem definitely has enough information to warrant a bug report. If you're able to capture and attach debug logs from Firestore (call Firestore.enableLogging(true) to enable debug logs) that would also be helpful. The bug will then be picked up by the Firestore team to investigate further.