Add Pre-Orders E2E Shortcode/Blocks Tests #3932

lovo-h · 2025-02-22T16:50:49Z

Changes proposed in this Pull Request:

This PR includes the following changes:

Adds E2E tests for pre-order products
Extends the E2E setup script to install the WooCommerce Pre-Orders plugin.
Adds some stability to the plugin installation flow.
Code cleanup: Adds reusability and removes unused functions/imports.

Testing instructions

Test: Ensure npm run test:e2e-setup executes as intended

Warning

The npm run test:e2e-setup command should only be executed once per site.

Follow the Test Setup instructions in the E2E docs.
- Important: GITHUB_TOKEN is required in the local.env file for the plugins to be installed correctly.
- Important: SSH_* variables are required in the local.env file.
Ensure theWooCommerce Pre-Orders plugin was installed.
Ensure the installation script + E2E tests executed as intended.

Test: Ensure npm run test:e2e executes as intended

Follow the Test execution instructions in the E2E docs.
- Note: This will only work if the npm run test:e2e-setup has successfully run.
Ensure the installation script + E2E tests executed as intended.

Test: Ensure npm run test:e2e-legacy executes as intended

Use the Test execution instructions in the E2E docs as guidance.
- Note: This will only work if the npm run test:e2e-setup has successfully run.
Ensure the installation script + E2E tests executed as intended.

Covered with tests (or have a good reason not to test in description ☝️)
Tested on mobile (or does not apply)

Changelog entry

This Pull Request does not require a changelog entry. (Comment required below)

Changelog Entry Comment

Comment

Post merge

Added testing instructions to the Release Testing Instructions wiki page (or does not apply)
Added the needs docs label (or does not apply)
Included this PR in the Release Thread scope (or does not apply)

asumaran · 2025-02-25T16:23:19Z

@lovo-h your PR seems to be working very well except for a failing test:

  1) [default] › checkout/shortcode/pre-order-product.spec.js:20:5 › customer can purchase a pre-order product @pre-orders

    Test timeout of 90000ms exceeded.

    Error: locator.fill: Test timeout of 90000ms exceeded.
    Call log:
      - waiting for frameLocator('#stripe-card-element iframe[name^="__privateStripeFrame"]').locator('[name="cardnumber"]')


       at ../utils/payments.js:161

      159 | 		)
      160 | 		.locator( '[name="cardnumber"]' )
    > 161 | 		.fill( card.number );
          | 		 ^
      162 | 	await page
      163 | 		.frameLocator(
      164 | 			'#stripe-exp-element iframe[name^="__privateStripeFrame"]'

        at fillCreditCardDetailsShortcodeLegacy (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/utils/payments.js:161:4)
        at /Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/tests/checkout/shortcode/pre-order-product.spec.js:34:8

asumaran

@lovo-h I’ve tested your PR multiple times, and running npm run test:e2e-setup -- --base_url=https://my.jurassic.ninja/ never wprked for me without issues on fresh sites. The Link payment method was never enabled using this command, and more recently, more tests started failing when using it.

However, running npm run test:e2e -- --base_url=https://my.jurassic.ninja/ almost worked, except for one or two tests failing. After setting retries to 1, everything passed after some retries.

So, I’m approving this because all indications point to test flakiness rather than an actual issue in the PR itself. That said, the tests are flaky, so we should keep an eye on that.

lovo-h · 2025-02-26T00:41:48Z

@asumaran, thanks for the review. I also noticed a significant amount of flakiness in some of the tests. I changed how CC input fields are retrieved in the legacy E2E test suite (3d55a3a). I am hoping this added the needed stability. Could you please review the latest changes once more?

Also, the E2E tests are passing on GH now. The Woo Pre-Orders plugin needed to be installed through the setup.sh script (see commit 00ca1fe). Thanks to @diegocurbelo for the assist.

asumaran · 2025-02-26T23:05:32Z

tests/e2e/tests/checkout/blocks/pre-order-product.spec.js

+test( 'customer can purchase a pre-order product @blocks @pre-orders', async ( {
+	page,
+} ) => {
+	await page.goto( `?p=${ productId }` );


I guess we are using permalinks but this is easier for redirecting the browser to the product URL without knowing the post slug.

asumaran · 2025-02-26T23:17:41Z

tests/e2e/utils/payments.js

+	if ( ! cardNumberLocator ) {
+		throw new Error(
+			'Could not find the credit card number form element.'
+		);
+	} else if ( ! cardExpiryLocator ) {
+		throw new Error(
+			'Could not find the credit card expiry form element.'
+		);
+	} else if ( ! cardCvcLocator ) {
+		throw new Error( 'Could not find the credit card cvc form element.' );
+	}


Printing the selector here would help quickly identify which one is failing. It may not be immediately clear which payment element we are dealing with.

I thought about adding the selectors but because they may vary depending on the iFrame, I figured I would use regular language to identify the elements (e.g. 'cc number', 'cc expiry', 'cc cvc').

Otherwise, we would need to be verbose and list out all of the potential selectors (e.g. 'name=cardnumber' OR 'name=number', 'name=exp-date' OR 'name='expiry', 'name=cvc').

What are your thoughts?

I see—since the selectors for inputs are the same, we could just print the iframe selector.

My idea was to print the full selector used to retrieve the input, like:

Could not find the credit card number form element using #wc-stripe-upe-form iframe[name^='__privateStripeFrame'] [name='number']

This way, we can clearly see that we’re trying to obtain the card number input using the UPE selector. So if the test fail because of this, we might suspect it’s due to UPE not being activated.

Good call. It would be nice to see the selector in the error. Added in 1e90f06.

asumaran · 2025-02-26T23:24:53Z

tests/e2e/bin/setup.sh

+	-H "Authorization: token $GITHUB_TOKEN" \
+	-H "Accept: application/octet-stream" \
+	--output $E2E_ROOT/woocommerce-pre-orders.zip \
+	https://api.github.com/repos/woocommerce/woocommerce-pre-orders/releases/assets/"$LATEST_RELEASE_ASSET_ID"


Since this is an open-source plugin that anyone can contribute to, only people with access to this repository would be able to run the E2E tests. Wondering if we should not run pre-order tests by default.

Or perhaps skip pre-order tests if the plugin isn’t installed successfully?

Good call. Addressed in d838091.

asumaran

@lovo-h I’ve left some comments for you. Just like in my last review, Link payment method is never activated when I run npm run test:e2e-setup -- --base_url=https://my-site.jurassic.ninja/.

However, everything works fine when I run npm run test:e2e -- --base_url=https://smy-site.jurassic.ninja/.

asumaran

For some reason, the E2E setup never works without issues for me.

Here's the output

~/Developer/woocommerce-gateway-stripe ‹add/pre-orders-e2e-tests› » npm run test:e2e-setup -- --base_url=https://artificial-cattle.jurassic.ninja/

> [email protected] test:e2e-setup
> ./tests/e2e/bin/setup.sh --base_url=https://artificial-cattle.jurassic.ninja/

Base URL: https://artificial-cattle.jurassic.ninja

======

Admin state file deleted successfully.
wp config set WP_DEBUG false --raw
Success: Updated the constant 'WP_DEBUG' in the 'wp-config.php' file with the raw value 'false'.
wp plugin install woocommerce --force --activate
Installing WooCommerce (9.7.0)
Downloading installation package from https://downloads.wordpress.org...
wp plugin install woocommerce-gateway-stripe --activate
Installing WooCommerce Stripe Payment Gateway (9.2.0)
Downloading installation package from https://...
wp plugin install disable-emails --activate
Installing Disable Emails (1.8.2)
Downloading installation package from https://downloads.wordpress....
wp theme install storefront --activate
Installing Storefront (4.6.1)
Downloading installation package from https://downloads.wordpress.org/...
wp option set woocommerce_store_address "60 29th Street"
Success: Updated 'woocommerce_store_address' option.
wp option set woocommerce_store_address_2 "#343"
Success: Updated 'woocommerce_store_address_2' option.
wp option set woocommerce_store_city "San Francisco"
Success: Updated 'woocommerce_store_city' option.
wp option set woocommerce_default_country "US:CA"
Success: Value passed for 'woocommerce_default_country' option is unchanged.
wp option set woocommerce_store_postcode "94110"
Success: Updated 'woocommerce_store_postcode' option.
wp option set woocommerce_currency "USD"
Success: Value passed for 'woocommerce_currency' option is unchanged.
wp option set woocommerce_product_type "both"
Success: Updated 'woocommerce_product_type' option.
wp option set woocommerce_allow_tracking "no"
Success: Value passed for 'woocommerce_allow_tracking' option is unchanged.
wp option set woocommerce_coming_soon "no"
Success: Updated 'woocommerce_coming_soon' option.
wp wc --user=demo tool run install_pages
Error: Parameter errors:
 unknown --user parameter
wp plugin install wordpress-importer --activate
Installing WordPress Importer (0.8.3)
Downloading installation package from https://downloads.wordpr...
wp import wp-content/plugins/woocommerce/sample-data/sample_products.xml --authors=skip
Starting the import process...


Processing post #6 ("V-Neck T-Shirt") (post_type: product)
-- 1 of ...
wp wc shipping_zone create --name="Everywhere" --order=1 --user=demo
Error: Parameter errors:
 unknown --user parameter
wp wc shipping_zone_method create 1 --method_id="flat_rate" --user=demo
Error: Parameter errors:
 unknown --user parameter
wp wc shipping_zone_method create 1 --method_id="free_shipping" --user=demo
Error: Parameter errors:
 unknown --user parameter
wp option update --format=json woocommerce_flat_rate_1_settings '{"title":"Flat rate","tax_status":"...
Success: Updated 'woocommerce_flat_rate_1_settings' option.
wp post create --post_type=page --post_title='Cart Shortcode' --post_name='cart-shortcode' --post_st...
Success: Created post 59.
wp post create --post_type=page --post_title='Checkout Shortcode' --post_name='checkout-shortcode' -...
Success: Created post 60.
- Trying to log-in as admin...
✔ Logged-in as admin successfully.
- Trying to add consumer token...
Skipping plugin update. The version already installed on the test site will be used.
- Trying to install woocommerce-subscriptions@latest from woocommerce/woocommerce-subscriptions repository...
wp option delete woocommerce_stripe_settings
✔ Added consumer token successfully.
Warning: Could not delete 'woocommerce_stripe_settings' option. Does it exist?
wp option set woocommerce_stripe_settings --format=json '{"enabled":"yes","title":"Credit Card (Stri...
Success: Updated 'woocommerce_stripe_settings' option.
✔ Updated Stripe plugin settings successfully.
✔ woocommerce-subscriptions plugin installed successfully.
- Trying to install woocommerce-pre-orders@latest from woocommerce/woocommerce-pre-orders repository...
✔ woocommerce-pre-orders plugin installed successfully.
Total Setup Time: 1:55.712 (m:ss.mmm)

======



Running 54 tests using 5 workers

  ✓  1 [legacy-setup] › _legacy-experience/legacy.setup.js:7:6 › Enable legacy checkout experience (6.9s)
  ✓  2 [default-setup] › default.setup.js:7:6 › Disable legacy checkout experience (7.8s)
  ✘  3 [default-setup] › default.setup.js:25:6 › enable Link (1.5m)
  ✓  4 [legacy] › _legacy-experience/checkout/card-failures.spec.js:53:6 › customer cannot checkout with invalid cards › a card with insufficient funds shows the correct error message (16.4s)
  ✘  5 [legacy] › _legacy-experience/checkout/card-failures.spec.js:57:6 › customer cannot checkout with invalid cards › a card with invalid number shows the correct error message (12.3s)
  ✓  6 [legacy] › _legacy-experience/checkout/card-failures.spec.js:49:6 › customer cannot checkout with invalid cards › a declined card shows the correct error message @smoke (16.8s)
  ✘  7 [legacy] › _legacy-experience/checkout/card-failures.spec.js:61:6 › customer cannot checkout with invalid cards › an expired card shows the correct error message (12.2s)
  ✘  8 [legacy] › _legacy-experience/checkout/card-failures.spec.js:65:6 › customer cannot checkout with invalid cards › a card with incorrect CVC shows the correct error message @smoke (11.3s)
  ✓  9 [legacy] › _legacy-experience/checkout/card-failures.spec.js:69:6 › customer cannot checkout with invalid cards › an error processing the card shows the correct error message (10.4s)
  ✓  10 [legacy] › _legacy-experience/checkout/normal-card.spec.js:12:5 › customer can checkout with a normal credit card @smoke (13.2s)
  ✓  11 [legacy] › _legacy-experience/checkout/saved-card.spec.js:30:5 › customer can checkout with a saved card @smoke (34.4s)
  ✓  12 [legacy] › _legacy-experience/checkout/sca-card.spec.js:12:5 › customer can checkout with a SCA card @smoke (20.9s)
  ✓  13 [legacy] › _legacy-experience/checkout/subscription-product.spec.js:20:5 › customer can purchase a subscription product @smoke @subscriptions (14.6s)
  ✓  14 [legacy] › _legacy-experience/order/full-refund.spec.js:13:5 › merchant can issue a full refund @smoke (27.1s)
  ✓  15 [legacy] › _legacy-experience/subscriptions/subscription-renewal.spec.js:35:5 › customer can renew a subscription @smoke @subscriptions (38.5s)
  ✓  16 [legacy] › _legacy-experience/woocommerce-blocks/card-failures.spec.js:43:6 › customer cannot checkout with invalid cards @blocks › a declined card shows the correct error message @smoke (16.0s)
  ✓  17 [legacy] › _legacy-experience/woocommerce-blocks/card-failures.spec.js:47:6 › customer cannot checkout with invalid cards @blocks › a card with insufficient funds shows the correct error message (17.1s)
  ✓  18 [legacy] › _legacy-experience/woocommerce-blocks/card-failures.spec.js:51:6 › customer cannot checkout with invalid cards @blocks › a card with invalid number shows the correct error message (8.2s)
  ✓  19 [legacy] › _legacy-experience/woocommerce-blocks/card-failures.spec.js:55:6 › customer cannot checkout with invalid cards @blocks › an expired card shows the correct error message (15.2s)
  ✓  20 [legacy] › _legacy-experience/woocommerce-blocks/card-failures.spec.js:59:6 › customer cannot checkout with invalid cards @blocks › a card with incorrect CVC shows the correct error message @smoke (15.3s)
  ✓  21 [legacy] › _legacy-experience/woocommerce-blocks/card-failures.spec.js:63:6 › customer cannot checkout with invalid cards @blocks › an error processing the card shows the correct error message (13.1s)
  ✓  22 [legacy] › _legacy-experience/woocommerce-blocks/normal-card.spec.js:12:5 › customer can checkout with a normal credit card @smoke @blocks (14.7s)
  ✓  23 [legacy] › _legacy-experience/woocommerce-blocks/saved-card.spec.js:30:5 › customer can checkout with a saved card @smoke @blocks (34.4s)
  ✓  24 [legacy] › _legacy-experience/woocommerce-blocks/sca-card.spec.js:12:5 › customer can checkout with a SCA card @smoke @blocks (25.3s)
  ✓  25 [legacy] › _legacy-experience/woocommerce-blocks/subscription-product.spec.js:17:5 › customer can purchase a subscription product @smoke @blocks @subscriptions (19.1s)

======

- Trying to clear consumer token... Try:1
✔ Cleared up consumer token successfully.

======



  1) [default-setup] › default.setup.js:25:6 › enable Link ─────────────────────────────────────────

    Test timeout of 90000ms exceeded.

    Error: locator.check: Test ended.
    Call log:
      - waiting for getByLabel('Link by Stripe Input')


      32 | 		'/wp-admin/admin.php?page=wc-settings&tab=checkout&section=stripe&panel=methods'
      33 | 	);
    > 34 | 	await page.getByLabel( 'Link by Stripe Input' ).check();
         | 	                                                ^
      35 | 	await page.click( 'text=Save changes' );
      36 |
      37 | 	await expect( page.getByText( 'Settings saved.' ) ).toBeDefined();

        at /Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/tests/default.setup.js:34:50

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    tests/e2e/test-results/output/default.setup.js-enable-Link-default-setup/test-failed-2.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: screenshot (image/png) ──────────────────────────────────────────────────────────
    tests/e2e/test-results/output/default.setup.js-enable-Link-default-setup/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #3: trace (application/zip) ─────────────────────────────────────────────────────────
    tests/e2e/test-results/output/default.setup.js-enable-Link-default-setup/trace.zip
    Usage:

        npx playwright show-trace tests/e2e/test-results/output/default.setup.js-enable-Link-default-setup/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

  2) [legacy] › _legacy-experience/checkout/card-failures.spec.js:57:6 › customer cannot checkout with invalid cards › a card with invalid number shows the correct error message

    Error: Could not find the credit card number form element using selector: #stripe-card-element iframe[name^="__privateStripeFrame"] [name="cardnumber"]

       at ../utils/payments.js:201

      199 |
      200 | 		if ( ! ( await isVisible( frameLocator, inputSelector ) ) ) {
    > 201 | 			throw new Error(
          | 			      ^
      202 | 				`Could not find the credit card ${ description } form element using selector: ${ frameSelector } ${ inputSelector }`
      203 | 			);
      204 | 		}

        at getLocator (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/utils/payments.js:201:10)
        at fillCreditCardDetailsShortcodeLegacy (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/utils/payments.js:239:23)
        at testCard (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/tests/_legacy-experience/checkout/card-failures.spec.js:24:2)

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    tests/e2e/test-results/output/_legacy-experience-checkou-9da5a-s-the-correct-error-message-legacy/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: trace (application/zip) ─────────────────────────────────────────────────────────
    tests/e2e/test-results/output/_legacy-experience-checkou-9da5a-s-the-correct-error-message-legacy/trace.zip
    Usage:

        npx playwright show-trace tests/e2e/test-results/output/_legacy-experience-checkou-9da5a-s-the-correct-error-message-legacy/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

  3) [legacy] › _legacy-experience/checkout/card-failures.spec.js:61:6 › customer cannot checkout with invalid cards › an expired card shows the correct error message

    Error: Could not find the credit card number form element using selector: #stripe-card-element iframe[name^="__privateStripeFrame"] [name="cardnumber"]

       at ../utils/payments.js:201

      199 |
      200 | 		if ( ! ( await isVisible( frameLocator, inputSelector ) ) ) {
    > 201 | 			throw new Error(
          | 			      ^
      202 | 				`Could not find the credit card ${ description } form element using selector: ${ frameSelector } ${ inputSelector }`
      203 | 			);
      204 | 		}

        at getLocator (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/utils/payments.js:201:10)
        at fillCreditCardDetailsShortcodeLegacy (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/utils/payments.js:239:23)
        at testCard (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/tests/_legacy-experience/checkout/card-failures.spec.js:24:2)

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    tests/e2e/test-results/output/_legacy-experience-checkou-db8d5-s-the-correct-error-message-legacy/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: trace (application/zip) ─────────────────────────────────────────────────────────
    tests/e2e/test-results/output/_legacy-experience-checkou-db8d5-s-the-correct-error-message-legacy/trace.zip
    Usage:

        npx playwright show-trace tests/e2e/test-results/output/_legacy-experience-checkou-db8d5-s-the-correct-error-message-legacy/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

  4) [legacy] › _legacy-experience/checkout/card-failures.spec.js:65:6 › customer cannot checkout with invalid cards › a card with incorrect CVC shows the correct error message @smoke

    Error: Could not find the credit card number form element using selector: #stripe-card-element iframe[name^="__privateStripeFrame"] [name="cardnumber"]

       at ../utils/payments.js:201

      199 |
      200 | 		if ( ! ( await isVisible( frameLocator, inputSelector ) ) ) {
    > 201 | 			throw new Error(
          | 			      ^
      202 | 				`Could not find the credit card ${ description } form element using selector: ${ frameSelector } ${ inputSelector }`
      203 | 			);
      204 | 		}

        at getLocator (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/utils/payments.js:201:10)
        at fillCreditCardDetailsShortcodeLegacy (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/utils/payments.js:239:23)
        at testCard (/Users/asumaran/Developer/woocommerce-gateway-stripe/tests/e2e/tests/_legacy-experience/checkout/card-failures.spec.js:24:2)

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    tests/e2e/test-results/output/_legacy-experience-checkou-ce5af-correct-error-message-smoke-legacy/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: trace (application/zip) ─────────────────────────────────────────────────────────
    tests/e2e/test-results/output/_legacy-experience-checkou-ce5af-correct-error-message-smoke-legacy/trace.zip
    Usage:

        npx playwright show-trace tests/e2e/test-results/output/_legacy-experience-checkou-ce5af-correct-error-message-smoke-legacy/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

  Slow test file: [default-setup] › default.setup.js (1.6m)
  Slow test file: [legacy] › _legacy-experience/subscriptions/subscription-renewal.spec.js (38.5s)
  Slow test file: [legacy] › _legacy-experience/checkout/saved-card.spec.js (34.4s)
  Slow test file: [legacy] › _legacy-experience/woocommerce-blocks/saved-card.spec.js (34.4s)
  Slow test file: [legacy] › _legacy-experience/order/full-refund.spec.js (27.1s)
  Consider splitting slow test files to speed up parallel execution
  4 failed
    [default-setup] › default.setup.js:25:6 › enable Link ──────────────────────────────────────────
    [legacy] › _legacy-experience/checkout/card-failures.spec.js:57:6 › customer cannot checkout with invalid cards › a card with invalid number shows the correct error message
    [legacy] › _legacy-experience/checkout/card-failures.spec.js:61:6 › customer cannot checkout with invalid cards › an expired card shows the correct error message
    [legacy] › _legacy-experience/checkout/card-failures.spec.js:65:6 › customer cannot checkout with invalid cards › a card with incorrect CVC shows the correct error message @smoke
  29 did not run
  21 passed (5.4m)

but running the tests command works fine without issues. If for you works fine I'm assuming this is an issue on my end.

~/Developer/woocommerce-gateway-stripe ‹add/pre-orders-e2e-tests› » npm run test:e2e -- --base_url=https://artificial-cattle.jurassic.ninja/                                                                                                                                                                                                                                                              130 ↵

> [email protected] test:e2e
> ./tests/e2e/bin/run-tests.sh --project=default --base_url=https://artificial-cattle.jurassic.ninja/

Base URL: https://artificial-cattle.jurassic.ninja

======

Admin state file deleted successfully.
Skipping Woo Setup.
- Trying to log-in as admin...
✔ Logged-in as admin successfully.
- Trying to add consumer token...
Skipping plugin update. The version already installed on the test site will be used.
Skipping plugins installation. The plugin versions pre-installed on the site will be used, if needed.
Skipping Stripe setup. Ensure Stripe webhook and keys are already setup in this environment.
✔ Added consumer token successfully.
Total Setup Time: 14.334s

======



Running 31 tests using 5 workers

  ✓  1 [default-setup] › default.setup.js:7:6 › Disable legacy checkout experience (5.6s)
  ✓  2 [default-setup] › default.setup.js:25:6 › enable Link (6.6s)
  ✓  3 [default] › checkout/blocks/card-failures.spec.js:55:6 › customer cannot checkout with invalid cards @blocks › a card with insufficient funds shows the correct error message (21.5s)
  ✓  4 [default] › checkout/blocks/card-failures.spec.js:51:6 › customer cannot checkout with invalid cards @blocks › a declined card shows the correct error message @smoke (20.4s)
  ✓  5 [default] › checkout/blocks/card-failures.spec.js:59:6 › customer cannot checkout with invalid cards @blocks › a card with invalid number shows the correct error message (16.2s)
  ✓  6 [default] › checkout/blocks/card-failures.spec.js:63:6 › customer cannot checkout with invalid cards @blocks › an expired card shows the correct error message (20.8s)
  ✓  7 [default] › checkout/blocks/card-failures.spec.js:67:6 › customer cannot checkout with invalid cards @blocks › a card with incorrect CVC shows the correct error message @smoke (21.2s)
  ✓  8 [default] › checkout/blocks/card-failures.spec.js:71:6 › customer cannot checkout with invalid cards @blocks › an error processing the card shows the correct error message (13.8s)
  ✓  9 [default] › checkout/blocks/normal-card.spec.js:12:5 › customer can checkout with a normal credit card @smoke @blocks (16.2s)
  ✓  10 [default] › checkout/blocks/pre-order-product.spec.js:26:5 › customer can purchase a pre-order product @blocks @pre-orders (12.8s)
  ✓  11 [default] › checkout/blocks/saved-card.spec.js:30:5 › customer can checkout with a saved card @smoke @blocks (34.2s)
  ✓  12 [default] › checkout/blocks/sca-card.spec.js:12:5 › customer can checkout with a SCA card @smoke @blocks (23.3s)
  ✓  13 [default] › checkout/blocks/subscription-product.spec.js:17:5 › customer can purchase a subscription product @smoke @blocks @subscriptions (14.7s)
  ✓  14 [default] › checkout/shortcode/card-failures.spec.js:34:6 › customer cannot checkout with invalid cards › a declined card shows the correct error message @smoke (12.4s)
  ✓  15 [default] › checkout/shortcode/card-failures.spec.js:38:6 › customer cannot checkout with invalid cards › a card with insufficient funds shows the correct error message (12.3s)
  ✓  16 [default] › checkout/shortcode/card-failures.spec.js:42:6 › customer cannot checkout with invalid cards › a card with invalid number shows the correct error message (9.5s)
  ✓  17 [default] › checkout/shortcode/card-failures.spec.js:46:6 › customer cannot checkout with invalid cards › an expired card shows the correct error message (14.8s)
  ✓  18 [default] › checkout/shortcode/card-failures.spec.js:50:6 › customer cannot checkout with invalid cards › a card with incorrect CVC shows the correct error message @smoke (13.0s)
  ✓  19 [default] › checkout/shortcode/card-failures.spec.js:54:6 › customer cannot checkout with invalid cards › an error processing the card shows the correct error message (13.6s)
  ✓  20 [default] › checkout/shortcode/normal-card.spec.js:12:5 › customer can checkout with a normal credit card @smoke (14.9s)
  ✓  21 [default] › checkout/shortcode/pre-order-product.spec.js:29:5 › customer can purchase a pre-order product @pre-orders (14.9s)
  ✓  22 [default] › checkout/shortcode/saved-card.spec.js:30:5 › customer can checkout with a saved card @smoke (33.9s)
  ✓  23 [default] › checkout/shortcode/sca-card.spec.js:12:5 › customer can checkout with a SCA card @smoke (23.7s)
  ✓  24 [default] › checkout/shortcode/subscription-product.spec.js:17:5 › customer can purchase a subscription product @smoke @subscriptions (16.0s)
  ✓  25 [default] › express-checkout/express-checkout.spec.js:54:2 › customer can use Link express checkout › inside the product page (7.5s)
  ✓  26 [default] › express-checkout/express-checkout.spec.js:57:2 › customer can use Link express checkout › inside the cart page (classic) (9.4s)
  ✓  27 [default] › orders/full-refund.spec.js:13:5 › merchant can issue a full refund @smoke (25.3s)
  ✓  28 [default] › subscriptions/subscription-renewal.spec.js:32:5 › customer can renew a subscription @smoke @subscriptions (36.1s)
  ✓  29 [default] › express-checkout/express-checkout.spec.js:62:2 › customer can use Link express checkout › inside the checkout page (classic) (9.7s)
  ✓  30 [default] › express-checkout/express-checkout.spec.js:67:2 › customer can use Link express checkout › inside the cart page (block) (10.2s)
  ✓  31 [default] › express-checkout/express-checkout.spec.js:72:2 › customer can use Link express checkout › inside the checkout page (block) (10.0s)

======

- Trying to clear consumer token... Try:1
✔ Cleared up consumer token successfully.

======


  Slow test file: [default] › express-checkout/express-checkout.spec.js (46.8s)
  Slow test file: [default] › subscriptions/subscription-renewal.spec.js (36.1s)
  Slow test file: [default] › checkout/blocks/saved-card.spec.js (34.2s)
  Slow test file: [default] › checkout/shortcode/saved-card.spec.js (33.9s)
  Slow test file: [default] › orders/full-refund.spec.js (25.3s)
  Consider splitting slow test files to speed up parallel execution
  31 passed (2.7m)

To open last HTML report run:

  npx playwright show-report tests/e2e/test-results/report-html

lovo-h · 2025-02-28T16:16:40Z

For some reason, the E2E setup never works without issues for me.

@asumaran, we discussed this over Slack and confirmed that it's not an issue introduced in this PR since you can also replicate it in the develop branch. Because it is unrelated, I'll go ahead and merge this PR.

Thanks again for the code review 🙏

lovo-h added 7 commits February 22, 2025 10:53

Clean-up: Remove unused 'loginCustomerAndSaveState' function

f29ffe3

Refactor: Re-use 'installPluginFromRepository()' function

89c9382

Clean-up: Remove unused 'checkWooGutenbergProductsBlockVersion' import

b66a2c2

Add 'woocommerce-pre-orders' plugin as part of e2e global-setup

a95eb33

Rephrase plugin installation errors' verbiage

a3625fb

Add 'products' utility to re-use product data

a4b1fbf

Add Pre-Order e2e tests

36055eb

lovo-h requested review from a team and asumaran and removed request for a team February 22, 2025 16:50

lovo-h added 4 commits February 25, 2025 13:14

Update Stripe iFrame selectors

1d58d42

Fix legacy tests: Differentiate between UPE and legacy iFrame

fde397f

Fix setup script to include the Pre-Orders plugin

00ca1fe

Add additional logging for Pre-Orders installation steps

bd2e091

asumaran approved these changes Feb 25, 2025

View reviewed changes

diegocurbelo and others added 7 commits February 25, 2025 17:16

Merge branch 'develop' into add/pre-orders-e2e-tests

9249ba3

Enable DEBUG mode for docker setup

97e1788

Disable DEBUG mode

3d21f42

Code cleanup: Re-use E2E_ROOT variable

7ba1bec

Fix installation path for Woo Pre-Orders

fb40165

Code clean-up: Add options variable for readability

1fdfaf3

Fix legacy E2E tests: Retrieve independent CC input iFrames

3d55a3a

lovo-h requested a review from asumaran February 26, 2025 00:41

Merge branch 'develop' into add/pre-orders-e2e-tests

218b89f

asumaran reviewed Feb 26, 2025

View reviewed changes

lovo-h added 3 commits February 27, 2025 12:42

Only run Pre-Order E2E tests if the plugin is installed

d838091

Add quotation marks around text selectors

f327b8b

Log selector used when error thrown because CC input field not found

1e90f06

lovo-h requested a review from asumaran February 27, 2025 20:42

asumaran approved these changes Feb 28, 2025

View reviewed changes

Merge branch 'develop' into add/pre-orders-e2e-tests

6e8ab30

lovo-h enabled auto-merge (squash) February 28, 2025 16:17

lovo-h merged commit 79eaa8a into develop Feb 28, 2025
41 checks passed

lovo-h deleted the add/pre-orders-e2e-tests branch February 28, 2025 16:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Pre-Orders E2E Shortcode/Blocks Tests #3932

Add Pre-Orders E2E Shortcode/Blocks Tests #3932

lovo-h commented Feb 22, 2025 •

edited

Loading

asumaran commented Feb 25, 2025

asumaran left a comment

lovo-h commented Feb 26, 2025

asumaran Feb 26, 2025

asumaran Feb 26, 2025 •

edited

Loading

lovo-h Feb 27, 2025

asumaran Feb 27, 2025

lovo-h Feb 27, 2025

asumaran Feb 26, 2025

asumaran Feb 26, 2025

lovo-h Feb 27, 2025

asumaran left a comment •

edited

Loading

asumaran left a comment

lovo-h commented Feb 28, 2025

Add Pre-Orders E2E Shortcode/Blocks Tests #3932

Add Pre-Orders E2E Shortcode/Blocks Tests #3932

Conversation

lovo-h commented Feb 22, 2025 • edited Loading

Changes proposed in this Pull Request:

Testing instructions

Changelog entry

Comment

asumaran commented Feb 25, 2025

asumaran left a comment

Choose a reason for hiding this comment

lovo-h commented Feb 26, 2025

asumaran Feb 26, 2025

Choose a reason for hiding this comment

asumaran Feb 26, 2025 • edited Loading

Choose a reason for hiding this comment

lovo-h Feb 27, 2025

Choose a reason for hiding this comment

asumaran Feb 27, 2025

Choose a reason for hiding this comment

lovo-h Feb 27, 2025

Choose a reason for hiding this comment

asumaran Feb 26, 2025

Choose a reason for hiding this comment

asumaran Feb 26, 2025

Choose a reason for hiding this comment

lovo-h Feb 27, 2025

Choose a reason for hiding this comment

asumaran left a comment • edited Loading

Choose a reason for hiding this comment

asumaran left a comment

Choose a reason for hiding this comment

lovo-h commented Feb 28, 2025

lovo-h commented Feb 22, 2025 •

edited

Loading

asumaran Feb 26, 2025 •

edited

Loading

asumaran left a comment •

edited

Loading