[INFRA] Streamline the way we notify other repositories when a new version is available

[tbouffard]

Current status and problems

Historically, when a new tag was created, we built the navigation demo and then notified the repository of examples that the demo was available.Then, when we introduced the R package, we notify it of a new release when the npm publish was done (it requires to download the IIFE bundle from the npm package).

At a first glance, we see that we notify the repositories at a different pace, whereas we should notify them at the same time. And this should be done when we are sure that the npm package is published. If this fails, we generally fix the problem and create a new release.

But there are more important issues with the current way of doing things.
In the repository of examples, the projects' package.json files are updated when a new version is available. This triggers a build of these projects, which requires the new package version to be available on npmjs.
By notifying the repository of examples after a tag (and building the navigation demo), we have no guarantee that the npm package is available for building the projects. The "npm publish" workflow can take time to run, or can be queued because all GitHub runners are already running other jobs.
In the past, errors occurred several times because of that. This requires running the GitHub workflow again, which is boring and slows down the release.

This has been even more critical since we have merged process-analytics/bpmn-visualization-examples#541: the first step of the GitHub workflow updating the version downloads the npm package (to compute SubResource Integrity). There are more chances than ever that it is not available at that time.It occurs during the first update done after this PR was merged.
See errors in https://github.com/process-analytics/bpmn-visualization-examples/actions/runs/6222658937/job/16887034563:

Mon, 18 Sep 2023 12:42:40 GMT Run curl https://registry.npmjs.org/bpmn-visualization/-/bpmn-visualization-0.40.0.tgz -o bpmn-visualization.tgz
Mon, 18 Sep 2023 12:42:40 GMT   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Mon, 18 Sep 2023 12:42:40 GMT                                  Dload  Upload   Total   Spent    Left  Speed
Mon, 18 Sep 2023 12:42:40 GMT
Mon, 18 Sep 2023 12:42:41 GMT   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Mon, 18 Sep 2023 12:42:41 GMT 100    21  100    21    0     0     49      0 --:--:-- --:--:-- --:--:--    49
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! code E404
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 No match found for version 0.40.0
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404  '[email protected]' is not in this registry.
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 Note that you can also install from a
Mon, 18 Sep 2023 12:42:42 GMT npm ERR! 404 tarball, folder, http url, or git url.

Here are the logs of the npm publish process in the bpmn-visualization repository : they show that the package is published more than one minute later, so the update fails for sure (see https://github.com/process-analytics/bpmn-visualization-js/actions/runs/6222648900/job/16887005367):

Mon, 18 Sep 2023 12:43:59 GMT npm notice
Mon, 18 Sep 2023 12:43:59 GMT npm notice  [email protected]
Mon, 18 Sep 2023 12:43:59 GMT npm notice === Tarball Contents ===
Mon, 18 Sep 2023 12:43:59 GMT npm notice 11.4kB  LICENSE
Mon, 18 Sep 2023 12:43:59 GMT npm notice 552B    NOTICE
Mon, 18 Sep 2023 12:43:59 GMT npm notice 10.6kB  README.md
Mon, 18 Sep 2023 12:43:59 GMT npm notice 59.9kB  dist/bpmn-visualization.d.ts
Mon, 18 Sep 2023 12:43:59 GMT npm notice 202.8kB dist/bpmn-visualization.esm.js
Mon, 18 Sep 2023 12:43:59 GMT npm notice 1.7MB   dist/bpmn-visualization.js   
Mon, 18 Sep 2023 12:43:59 GMT npm notice 987.0kB dist/bpmn-visualization.min.js
Mon, 18 Sep 2023 12:43:59 GMT npm notice 932B    dist/not-supported-ts-versions.d.ts
Mon, 18 Sep 2023 12:43:59 GMT npm notice 6.8kB   package.json
Mon, 18 Sep 2023 12:43:59 GMT npm notice === Tarball Details ===
Mon, 18 Sep 2023 12:43:59 GMT npm notice name:          bpmn-visualization
Mon, 18 Sep 2023 12:43:59 GMT npm notice version:       0.40.0
Mon, 18 Sep 2023 12:43:59 GMT npm notice filename:      bpmn-visualization-0.40.0.tgz
Mon, 18 Sep 2023 12:43:59 GMT npm notice package size:  570.3 kB
Mon, 18 Sep 2023 12:43:59 GMT npm notice unpacked size: 3.0 MB
Mon, 18 Sep 2023 12:43:59 GMT npm notice shasum:   bfc14d6d01647c5c8982ca0d20f750d233ae77df
Mon, 18 Sep 2023 12:43:59 GMT npm notice integrity:     sha512-74oZO1M2J3TL6[...]s4QqkWWUL8QcA==
Mon, 18 Sep 2023 12:43:59 GMT npm notice total files:   9
Mon, 18 Sep 2023 12:43:59 GMT npm notice
Mon, 18 Sep 2023 12:43:59 GMT npm notice Publishing to https://registry.npmjs.org/ with tag latest and public access
Mon, 18 Sep 2023 12:43:59 GMT npm notice publish Signed provenance statement with source and build information from GitHub Actions
Mon, 18 Sep 2023 12:43:59 GMT npm notice publish Provenance statement published to transparency log: https://search.sigstore.dev/?logIndex=37060827

Proposals

The examples repository must be notified of new version availability only when the npm package is published.
It also requires to retrieve the "navigation demo".

flowchart LR
    newTag((new tag)) --> bng(build navigation demo) --> notifEx((notify repository\n  of examples))
    newTag --> npm(npm publish) --> notifR((notify\n R package))

Loading

Current implementation

Proposal 1

IMPORTANT: this will not be implemented, see "Proposal 2"

A first solution could be like the following:

• keep the existing "npm publish" workflow which is triggered when a new tag is created
• at the end of this workflow, it triggers a "new workflow" in charge of preparing assets and notifying the other repositories
• the triggering can be done by sending an repository_dispatch at the end of the "npm publish" workflow including the tag/version.
• A workflow_run could also trigger the "new workflow", but passing the version info is more tricky. See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#using-data-from-the-triggering-workflow that show the work to do to retrieve a value from the initial workflow.
• the "new workflow" can also be triggered using a workflow_dispatch for manual replay
• the "new workflow" includes the job that currently build the navigation demo and notify the repository of examples
• it also includes a job that send a repository_dispatch to all repositories (using a matrix configuration - array) requiring to be notified that a new version is available on npm (for now, bpmn-visualization-examples, for others, see below)

Note: In the description above, the "new workflow" could be the existing workflow building the navigation demo that we would update with the proposal. It would probably have to be renamed to match what it achieves. Note that this involves modifying the event sent to the example repository, as it contains the name of the workflow that stores the navigation demo artifact.

flowchart LR
    newTag((new tag)) --> npm(npm publish) --> sendEvent((notify\n wf2))

    wf2((wf2:\n received\n event)) --> gateway{+} --> bng(build navigation demo) --> notifEx((notify repository\n  of examples))
    gateway --> notifR((notify\n R package))
    gateway --> notifRepo1((notify\n repo1))
    gateway --> notifRepo2((notify\n repo2))

Loading

Proposal 1

Proposal 2

IMPORTANT: this is the proposal that will be implemented

Overview

• The workflow that manages the "npm publish" only performs the publishing to better separate the responsibilities
• Once the publishing is done, an event is sent to notify that the operation is completed
• The existing workflow that build the "demo" is now triggered when it receives the event, not when the git tag is created
• A new workflow is created to notify other repositories that the new version of the npm package is available

flowchart LR
    newTag((new tag)) --> npm(npm publish) --> sendEvent((send event\n new npm package\n version available))

    wf1((wf1:\n received\n event)) --> bng(build navigation demo) --> notifEx((notify repository\n  of examples))

    wf2((wf2:\n received\n event)) --> gateway{+}
    gateway --> notifR((notify\n R package))
    gateway --> notifRepo1((notify\n repo1))
    gateway --> notifRepo2((notify\n repo2))

Loading

Proposal 2

Repositories to notify when a new version is available

While we currently notify the repository of examples and bpmnVisualizationR, other repositories may also require to be notified when a new version is available.

Managed with dependabot: if there is an urgent need for an update, it can be run manually:

• bv-addons: chore(deps): bump bpmn-visualization from 0.39.0 to 0.40.0 bpmn-visualization-addons#125
• demo template: chore(deps): bump bpmn-visualization from 0.39.0 to 0.40.0 bpmn-visualization-demo-template#31
• bonita-day 2023 demo: chore(deps): bump the process-analytics group with 2 updates bonita-day-demo-2023#131
• icpm 2022 demo: chore(deps): bump bpmn-visualization from 0.37.0 to 0.39.0 icpm-demo-2022#99

Not possible to managed with dependabot

• bpmnVisualizationR: require to download the IIFE bundle
• repository of examples: url to update (with SubResource Integrity computation), projects update not manageable by dependabot as not a "workspace" aware project (may be supported if we used npm workspaces, pnpm, yarn, ... or alterative to dependabot like renovate?)

To check

• bpmn-visualization-tools: not possible today, but should be possible if we switch to a "workspaces" aware build
• pm4py?

Tasks

Preview Give feedback

1. Validate the solution in the playground repository. See feat: simulate "trigger workflows when new version available on npm" github-actions-playground#373 / fix: clean syntax when sending repository_dispatch github-actions-playground#374
2. Adapt existing workflows to implement the proposal 2 - ci: redesign notifications for new releases #3156
3. Update the documentation of the release process (BPMN diagram + screenshot) - ci: redesign notifications for new releases #3156
4. Create issues in new targeted repositories that will receive the event for an automatic update: [INFRA] Automate the version bump of bpmn-visualization bpmn-visualization-tools#35 / [INFRA] Automate the update of bpmn-visualization bpmn-visualization-pm4py#46
5. Send version update event to the new targeted repositories --> not now, this will be done when the targeted repositories will be updated automatically