Add ability to run block tests against specific Gutenberg versions

[glendaviesnz]

Changes proposed in this Pull Request:

• Adds a script to allow running of block unit and fixture tests against specific versions of gutenberg

Does this pull request change what data or activity we track or use?

No

Testing instructions:

• Checkout PR
• Run pnpm fixtures:test:gbv trunk 3774b1489f17de1da53a96a2066ce9a8f996293c
• This will check out the commit that caused the recent 10.9.0 regressions with the subscriptions block, so once this task has completed you should see 2 block fixture failures
• Run pnpm fixtures:test:gbv test-validation to just run the tests again
• Run pnpm fixtures:test:gbv reset and check that ./packages and ./tests/temp-gutenberg-checkout dirs have been removed and the ./package.json changes have been reverted and pnpm test-extensions completes successfully
• Run pnpm fixtures:test:gbv trunk and make sure all the tests complete successfully

Todo:

• Get block unit tests running

[matticbot]

Caution: This PR has changes that must be merged to WordPress.com
Hello glendaviesnz! These changes need to be synced to WordPress.com - If you 're an a11n, please commandeer and confirm D63561-code works as expected before merging this PR. Once this PR is merged, please commit the changes to WP.com. Thank you!
This revision will be updated with each commit to this PR

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ All commits were linted before commit.
  
• ✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.

---

Once your PR is ready for review, check one last time that all required checks (other than "Required review") appearing at the bottom of this PR are passing or skipped.
Then, add the "[Status] Needs Team review" label and ask someone from your team review the code.
Once you’ve done so, switch to the "[Status] Needs Review" label; someone from Jetpack Crew will then review this PR and merge it to be included in the next Jetpack release.

---

Jetpack plugin:

• Next scheduled release: July 20, 2021.
• Scheduled code freeze: July 15, 2021.

[brbrr]

@glendaviesnz Is there any info where I can learn more about the motivation behind this PR?

[glendaviesnz]

@glendaviesnz Is there any info where I can learn more about the motivation behind this PR?

p1HpG7-cuI-p2 Very open to ideas about other ways of approaching this.

[anomiex]

I think the idea of being able to run against a different version of Gutenberg is a good idea, but I don't like the amount of hackery going on in the script you're adding here.

I have to wonder how this even works given the failures in #19328 where renovate is trying to update the same pacakges.

Unfortunately I don't know enough about how all this stuff works to be able to suggest a better way to do things.

[anomiex]

We should probably add .mjs to the set of files that are linted, like we just did for .cjs as part of PR 20085 (see here)

[glendaviesnz]

Good idea, will do that if we decide to go ahead with this PR based on outcome of discussions at p1HpG7-cuI-p2

[anomiex]

What's going on here? Are all of these going to have to be added someday when we get around to making #19328 happen?

[glendaviesnz]

This is just temporarily adding some additional dependencies that the latest Gutenberg needs to successfully build - I assume that these could be removed when #19328 happens, although there is always the potential that the cutting edge Gutenberg is going to have some dependencies that the Jetpack repo doesn't.

[glendaviesnz]

I think the idea of being able to run against a different version of Gutenberg is a good idea, but I don't like the amount of hackery going on in the script you're adding here.

Not the ideal setup in my view either, but I couldn't find any other way of making this happen without a temporary adjustment of the current dependencies - and the only way to automate that is a temporary hack of the package.json files as far as I can see.

If the issues with #19328 are resolved would the plan be to try and keep the Jetpack repo more in-sync with the latest @WordPress package releases? If that was the case maybe this approach isn't necessary?

I have to wonder how this even works given the failures in #19328 where renovate is trying to update the same pacakges.

It probably works because we aren't trying to build the whole project with the new dependencies, it is only the Jetpack block code that is being compiled against them on the fly via the jasmine test runner

[anomiex]

If the issues with #19328 are resolved would the plan be to try and keep the Jetpack repo more in-sync with the latest @WordPress package releases? If that was the case maybe this approach isn't necessary?

It'd be good to keep better on top of the renovate backlog, yes. Part of the problem is that we don't have anyone really familiar with all this React code on Crew, so if something breaks in an update it's hard for us to fix on our own.

It probably works because we aren't trying to build the whole project with the new dependencies, it is only the Jetpack block code that is being compiled against them on the fly via the jasmine test runner

But even test-extensions is failing there.

[glendaviesnz]

But even test-extensions is failing there.

Yeh, because of changes in the Gutenberg packages a few of the tests were failing, I had to update quite a few of the tests to get them to run against the different dependency trees.

In theory, if I get this PR finalised and merged, and #19328 is rebased, the test-extensions issues should be resolved there.

[anomiex]

If the issues with #19328 are resolved would the plan be to try and keep the Jetpack repo more in-sync with the latest @WordPress package releases? If that was the case maybe this approach isn't necessary?

Yeh, because of changes in the Gutenberg packages a few of the tests were failing, I had to update quite a few of the tests to get them to run against the different dependency trees.

In theory, if I get this PR finalised and merged, and #19328 is rebased, the test-extensions issues should be resolved there.

Maybe it'd make more sense to fix #19328 so it can be merged, then see if the controversial parts of this approach are still necessary?

That might also simplify the necessary changes, as when doing it in #19328 you wouldn't need to continue to support the old versions of the packages too.

[glendaviesnz]

Maybe it'd make more sense to fix #19328 so it can be merged, then see if the controversial parts of this approach are still necessary?

Yeh, good idea, I will take another look at it from that angle.

[glendaviesnz]

Closed in favour of #20327 which removes the test updates which are moved to #20326