kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences

[ljmotta]

Closes apache/incubator-kie-issues#1114
Closes #2839

Description

This PR introduces containerized end-to-end (E2E) Playwright tests to ensure consistent and reliable test execution across environments.

Tests were executed in the following environments:

• macOS
  • arm64
  • Sonoma 14.6.1
  • Podman Desktop v1.15
  • Podman Engine 5.3.1
  • Docker CLI 27.2.1
• macOS
  • amd64
  • Sonoma 14.5
  • Podman Desktop v1.16
  • Podman Engine 5.3.2
  • Docker CLI 274.0
• Windows WSL
  • amd64
  • Ubuntu 22.04.3 LTS
  • Docker CE CLI 25.0.2
  • Docker CE Engine 25.0.2

Known Issues

• Running the online-editor containerized test-e2e script on macOS requires starting the services in a separate terminal.
• Performance: Tests may run slower on macOS due to architecture differences.
• Flakiness: Retries are enabled because tests can occasionally fail without valid reasons when running in the restricted containerized environment.
• Debugging tests is not currently supported but could be added in the future.
• The Docker CLI is required to run the containerized tests.

Trying it out

• Install the dependencies.
• Build the @kie-tools/playwright-base image with the following command:

KIE_TOOLS_BUILD__buildContainerImages=true pnpm -F @kie-tools/playwright-base image:docker:build

• In a package that contains Playwright e2e tests, run the following:

KIE_TOOLS_BUILD__runEndToEndTests=true KIE_TOOLS_BUILD__containerizedEndToEndTests=true pnpm test-e2e

This command will start the required services and run the tests inside the container.

Updating screenshots

To update screenshots, use the pnpm test-e2e:container:shell script to open a shell inside the container:

pnpm test-e2e:container:shell
# In the container shell, manually call the `test-e2e:run` passing the `-u` or `--update-snapshots` flag
pnpm test-e2e:run -u
# or
pnpm test-e2e:run --update-snapshots

To improve performance, you can filter which tests will run by adding the -g flag:

pnpm test-e2e:run -u -g "test name"

[jomarko]

Thank you @ljmotta . Please, let us know this is ready for a verification. I will try on non-MAC environment.

[ljmotta]

@jomarko This PR can be tested for macOS and Linux. I still need to make some adjustments for native Windows (non WSL).

[thiagoelg]

Awesome PR @ljmotta! This will make our e2e tests more stable and concise!

I've made some comments below about the package.json scripts; let me know what you think.

[ljmotta]

@tiagobento The symlinks are located in the node_modules directory, so switching to hard links might not be straightforward. After experimenting, I made some progress with volumes using Docker Desktop and HyperV. However, I encountered two additional issues:

1. It appears that playwright Windows browsers cannot be used inside the container. This means the browsers need to be reinstalled. Furthermore, the playwright binary is unavailable, requiring a full reinstallation of playwright, including erasing node_modules [1].
2. To address the first issue, all untracked files (e.g., node_modules) need to be removed. However, since pnpm creates recursive structures with symlinks, the git clean -fdx command cannot be used [2][3].

Even if the second issue were resolved, having to fix the first issue every time a container is started would be impractical. Considering these challenges, I am leaning toward not supporting Windows development without WSL for this pull request.

[1] microsoft/playwright#23098 (comment)
[2] git-for-windows/git#3358
[3] pnpm/pnpm#4401

[tiagobento]

@ljmotta I'm ok with not supporting Windows without WSL here, just please add a new statement to the README making that explicit. And thank you so much for the effort!

[tiagobento]

So I think the KIE_TOOLS_BUILD__runEndToEndTests env var should still be the one use to toggle the execution of the E2E tests, and this new flag should only toggle the MODE with which the tests run.

KIE_TOOLS_BUILD__runEndToEndTests=false
# E2E tests don't run and `KIE_TOOLS_BUILD__runContainerizedEndToEndTests` is ignored.

KIE_TOOLS_BUILD__runEndToEndTests=true
KIE_TOOLS_BUILD__runContainerizedEndToEndTests=true
# E2E tests run inside containers

KIE_TOOLS_BUILD__runEndToEndTests=true
KIE_TOOLS_BUILD__runContainerizedEndToEndTests=false (default)
# E2E tests run on host machine (without containers involved)

WDYT?

[ljmotta]

@tiagobento I've renamed KIE_TOOLS_BUILD__runContainerizedEndToEndTests to KIE_TOOLS_BUILD__containerizedEndToEndTests, and instead of using a new variable, I've made it a property of buildEnv.endToEndTests. I've also updated the scripts to follow the new behavior you described:

KIE_TOOLS_BUILD__runEndToEndTests=true ?
  KIE_TOOLS_BUILD__containerizedEndToEndTests=true ? 
    run_on_container : 
    run_on_host : 
  not_run

Currently, KIE Sandbox tests aren't working on macOS containers. I'm not sure why. I've tried debugging it, but I haven't found any obvious cause. The DMN and BPMN Editors are stuck in an infinite loading state and aren't fetching their respective envelopes. The shared worker is stuck in the git.init method, which calls the fs.exists method. From what I can see, the fs.exists method doesn't exist 😅.

[tiagobento]

Did you try serving online-editor with something like http-server, for instance, instead of the Webpack Dev Server (pnpm start)?

[ljmotta]

@tiagobento http-server doesn't work too. I've tested with macOS in a amd64 machine, and the online-editor Google Chrome and Chormium tests passed with Docker and Podman.

@thiagoelg gave me the idea of removing the amd64 compatibility that Docker and Podman provide and running the container natively on the arm macOS. After removing the platform specification from the image build script and docker-compose file, the online-editor Chromium tests passed🎉 (Unfortunelly doesn't exist Google Chrome for Linux arm64). Everything points to a bug in the platform compatibility.

EDIT:
I see three ways we can proceed with this PR:

1. Remove the platform flag, and disable all Google Chrome tests for Linux arm64.
2. Keep the platform (since it works for the other packages) and disable Chromium and Google Chrome tests in the online-editor package.
3. Make the arm64 macOS user generate an additional image for arm64 that will be used for the online-editor and without Google Chrome

[tiagobento]

Thank you @ljmotta. From what I understand I believe the option benefiting the larger surface area would be option 2.

Please confirm if the following statement is correct:

---

If we go with option 2:

• Developers on macOS arm64 trying to update online-editor screenshots won't be able to do so, and will need to find an alternative setup while we figure out what's the problem there.
• Windows users will need to rely on WSL to update screenshots and we won't be pursuing finding the solution for non-WSL.

---

If this is accurate, then you have my blessing to go with it 😛

[ljmotta]

Yeah, that's correct. And just to be more clear. With option 2, the macOS arm64 users will not be able to run any online-editor Chromium and Google Chrome test inside the container.