ReleaseAutomation

Release data collection

There are a number of steps in the release process (https://github.com/open-mpi/ompi/issues/4240 is a good outline of the checklist process) that could be completely automated or generally accelerated by adding some metadata to commits / pull requests and then building tooling to pull down that information prior to a release. Examples include shared library versions, README, and NEWS.

We propose the following metadata in commit messages / pull request descriptions to automate collecting the required data:

• NEWS: <blob> automatically adds a NEWS item for all releases in which this commit appears for the first time of <blob>. <blob> should not include the leading dash and the NEWS system will re-paginate the text to fit in 75 columns. <blob> can include multiple lines, and will include all text until the next blank line. NEWS files will be automatically generated based on this data.
• README: <blob> informs the Release Managers to add a README item for all releases in which this commit appears for the first time of <blob>. <blob> can include multiple lines (i.e., the committer should paginate according to git rules for the git commit), and will include all data until the next blank line. Unlike NEWS: items, README: items will be manually reformatted / inserted into the README file by the release manager prior to release. This is because while NEWS is linear in time, README is grouped by topics. If the commit should remove text from the README file, simply note that -- perhaps something like README: remove note about gcc 4.8 not being supported. The important thing is to let the release manager know what to do.
• ABI_CHANGE: <add,remove,change>. If the commit changes the interface for the shared library you're modifying (examples: the MPI interface for libmpi, the ORTE interface for liborte, the interface for the common library in the case of common libraries), add an ABI_CHANGE: flag with the type of change. Remember that remove or change breaks ABI, which may be ok in certain circumstances, but release managers need to know this in order to properly calculate shared library version numbers at each release.

These three items would get us the history on a given branch, back to the first commit, which won't give the history we want. For the second release and onwards in a series (ie the Z in X.Y.Z), it's pretty straightforward: run the history from the HEAD back to the next newest tag in the branch. We haven't always (and likely won't always) be consistent between branch names and release series, but close. However, the first release in a series (X.Y.0) is harder; you don't want the history back to the start of the branch off master, but back to where the previous branch started from master, minus everything that's in a tagged release on that branch. For example,

  A - B - C - D - E ------ H             master
        \                  \
          F - D - E - G     \            v3.0.x
                             \
                              I - G      v3.1.x

In this case, assume v3.0.0 is a tag pointing to D and and v3.0.1 is a tag pointing to G. When we build history v3.1.x, we want any data for commits C, H, and I (because while A, B, D, E, and G appear in the path from G to A, any information for them has been covered in the README, HISTORY, and shared library updates for the previous releases.

To do this search, we need two pieces of information: the current branch name and the previous branch name. The current branch name should, in theory, be obvious, but in practice can be hard to pin down for a given hash in Git (particularly if we've just created the branch and so HEAD is the same on master and the release branch). There is no reasonable convention for finding the next previous branch. So, to avoid any surprises, we propose to add two new fields to the VERSION file: current_branch and previous_branch. current_branch is the current git branch, which may be set to master. previous_branch is the previous branch from master, which tooling should use when the release version on the current_branch is still 0, to trace history back since last release. When a release branch is created from master, a commit on master should increase previous_branch to the recently made release branch and a commit on the release branch should change current_branch from master to the new branch name.

LICENSE file updates

To date, RMs have been making sure that the LICENSE file is up to date. Does this make sense and what are they doing (the v3.1.x RMs have only been bugging people about updating the LICENSE for their companies)?

Release Building

A whole lot of work goes into sanity checking a tarball (see https://github.com/open-mpi/ompi/issues/4240). The proposed (and half implemented for v3.0.x/v3.1.x) process which automates much of the sanity checking work away is:

1. Nightly tarballs and dist tarballs are built using the same make_dist_tarball script (with different --greek-only / --no-greek options as appropriate) from the same Jenkins build instances.
2. MTT is used to do complete checkout of a particular git hash to make sure a tarball is useful based on the last nightly tarball.
3. Release manager runs ompi-scripts/dist/add-release-tag <tag name> <hash>, which will do a bunch of sanity checks: ensuring you're on a branch that looks like tag name falls on it, that the tag name version (major, minor, and release) matches the VERSION file, README and NEWS have been updated, shared library versions have been updated, and making sure the tag doesn't already exist. Some of these may be skipped for rc tags, of course. Then it will tag the release and push the tag to github.
4. Release manager runs the build dist job on Jenkins, which runs a bunch of sanity checks and pushes the resulting build artifacts to S3.
5. Release manager updates the web site, adding the release to the versions[] array in the right index.php and building / checking in the man pages.
6. Release manager sends announce email
7. Release manager runs ompi-scripts/dist/prep-next-release --branch <branch> --next-release <next release version> --next-release-date <date>, which will update the VERSION file, push to the user's fork, post a pull request for the version change, and create the milestone for the next release.