[Technical Questions] Tests passing locally but not on GitHub Workflow

[jennxu23]

In what area do you have a technical challenge?

Xcode, Simulator & Previews

Description

Currently, our tests pass locally. However, all of our PRs are failing the same tests (HealthKitUpload and Onboarding) in the GitHub workflow. This is a recent issue, as all of our previously merged PRs have passed these tests and we didn't modify these tests in our most recent PRs.

Reproduction

This issue began occurring recently, specifically after we started to push data to our firebase. Initially, we were also failing the tests locally. However, we solved this issue by uploading our user accounts and mock data to the actual firebase rather than the emulator.

Expected behavior

We expect the tests to pass both locally and on the GitHub workflow.

Additional context

Following the Simulator, I found that my tests can fail locally intermittently in two cases:

1. Data isn't uploaded or is too slow to upload to the MockUpload tab
2. The simulator has to go through the Apple Health App onboarding flow when manually adding in steps.

Code of Conduct

• I agree to follow this course's Code of Conduct and Contributing Guidelines

[PSchmiedmayer]

Hi @jennxu23 & @CS342/2023-utah,

The problem you are describing is due to your changes in the following PR: CS342/2023-Utah#44.
In this PR, you changed the Firebase Account settings to not use the emulator. While this is right and useful for releasing the application through, e.g., TestFlight or maybe testing it on your device where you would need to manually determine the IP of your computer to connect to the emulator there, it is essential for the UI tests, which must run in a controlled environment to establish a set of entry conditions for the system under test.

We achieve this using the Firebase emulator and also encourage you to use it during development. This follows the recommendations from Google about Firebase environments (https://firebase.google.com/docs/projects/dev-workflows/overview-environments) and general best practices when trying to verify the functionality of the software, which is especially critical for a medical context. While this might lead to a bit of a longer development time to write and maintain these tests, it is essential to ensure a high-quality system. As Oliver mentioned in the class: We don't want to lose any patient data or deploy a version of the software that doesn't work for our users.

To verify the behavior that the GitHub Actions use to test the code, you need to have the emulator up and running, and your application has to be tested against this controlled environment (e.g., no existing user account). This allows us to ensure that, e.g., the account sign-up function is tested every time to verify that it is working.

I have created a PR that addresses your issues by doing the following:

• Connect to the Firebase Emulator by default when running the application in the iOS simulator, and connect to the real Firebase instance when running the application on a device or when deploying it using TestFlight.
• Add a feature flag to test using the real firebase instance if you need to do that in the iOS simulator (--useFirebaseEmulator)

This should solve your problem; the tests are now passing on GitHub: CS342/2023-Utah#48