23.05: Snap Package and Linux and Windows Hotfix Release

[vincent6767]

Tasks

Preview Give feedback

1. @Ulrick28 Release the snap package.
2. Identify the PR's for the hotfix (Fix for corrupt binaries in a snap package o3de#15906, Remove PyYAML o3de#16395, Fix the first-time Editor crash bugs o3de#16197, Update dependancies for snap and disable patchelf for drivers o3de#16372, and Update the deb and snap package node label o3de#16094)
3. @amzn-changml Create a branch from main 23.05. This branch should be hotfix/23.05.1
4. @amzn-changml Merge cherry picked PR's to branch
5. @amzn-changml Build the branch in AR
6. @amzn-changml Create installer builds for Windows, Debian, and Snap.
7. @Ulrick28 Test branch. We can perform testing internally on my team overnight, however would be good to some external testing as well particularly from RedHat engineers who found the primary issue.
8. @brianherrera If all goes well, integrate hotfix branch to mainline, with appropriate tags. Push new LFS objects to the release tag
9. @amzn-changml Update the o3debinaries.org website with the new version
10. @amzn-changml Push new snap package to the snap store
11. @brianherrera Delete hotfix branch
12. Document reason for hotfix and changes
13. @Ulrick28 @tonybalandiuk @vincent6767 Writes the hotfix release notes.
14. @tonybalandiuk @vincent6767 Inform marketing team about the hotfix.
15. @vincent6767 to create another issue for documenting the hotfix release process.

Notes

1. Discussion thread: https://discord.com/channels/805939474655346758/1126207780475379832/1126518697939451914

[vincent6767]

@tonybalandiuk Are there any docs that explain the point release process?

@Ulrick28 I assume you have got anything you need to release the snap package on Monday?

BTW, @Ulrick28 Is this snap package related to this #178 (comment)?

[Ulrick28]

Yes, the snap package is related to the comment, and we have decided to release the snap package once it is approved, instead of waiting for the hotfix. The primary PR for the hotfix will be o3de/o3de#16197. We will need to decide if there are other crash fixes we will want to include.

[vincent6767]

Snap package was released: https://discord.com/channels/805939474655346758/816043864858951710/1130230226882928732

[tonybalandiuk]

@vincent6767 we do not have a minor release (aka point release) process documented. I, personally, have not done a minor release. The last minor release was here https://github.com/o3de/o3de/projects/8 - the major release project tasks were actually taken from this project. In almost all ways a minor release should follow the same steps as a major release.

A few comments about potentially doing a minor release

1. We don't have the stabilization branch anymore from the 23.05. We need to sync with sig-build to see if they can stand up a branch based on the 23.05 release tag.
2. My point of view is that we would do everything we do for a major release. The stabilization branch would have less code to stabilize - that's the only difference I expect.
3. We still need a release manager and need to sort out testing before we can move forward on the minor release.

[Ulrick28]

Hi! This is more of a hotfix than a point release. The intent of a hotfix is to get the fix into mainline as quickly as possible, since mainline is currently broken. Vincent, do you want me to create a separate issue detailing the steps for the hotfix, or just add a comment here and you can then modify the steps in this issue?

I found the instructions for hotfixes - "Hotfixes are emergency updates to fix a specific set of bugs deemed as either critical security issues or blocking basic utilization of the project. These also result in an incremental version number update (ie 22.11.0 would become 22.11.1) If there are hotfixes to be applied at a later date, an engineer can branch a tagged Main branch release to a Hotfix branch. This developer would then apply fixes to the HotFix branch. Once completed, fixes will be merged to the Main branch as well as the Dev branch. Merged Main branch would then be tagged with a new release "hotfix" number. When merging is complete, the Hotfix branch will be deleted as at that point no further changes will be done to that branch and it is identical to newly tagged Main branch."

The steps are:

1. Identify the PR's for the hotfix (Remove PyYAML o3de#16395 and Fix the first-time Editor crash bugs o3de#16197)
2. Create a branch from main 23.05. This branch should be 23.05.1
3. Merge cherry picked PR's to branch
4. Create installer builds for Windows, Debian, and Snap.
5. Test branch. We can perform testing internally on my team overnight, however would be good to some external testing as well. Particularly from RedHat engineers who found the primary issue.
6. Document reason for hotfix and changes
7. If all goes well, integrate hotfix branch to mainline, with appropriate tags
8. Delete hotfix branch

[vincent6767]

Hi @Ulrick28
Sounds good to me. Let's use this ticket and I'll update the steps in the issue

[vincent6767]

@Ulrick28 I have updated the issue with the steps you mentioned. Can you help assigns the person who is responsible for each step? Feel free to edit the above checklist as you see fit. Thanks!

[amzn-changml]

I would also add this PR once it's merged, as it incorporates all the non-code changes needed to make the 23.05 snap :o3de/o3de#16372

[vincent6767]

Sure thanks @amzn-changml I added this PR to the checklist.

[tonybalandiuk]

@Ulrick28 please make sure to get names next to each task so we have a strong artifact

[amzn-changml]

Hotfix branch created and PRs cherry-picked: o3de/o3de@main...hotfix/23.05.1

[amzn-changml]

For o3de/o3de#16197 (editor crash fixes), there is a loose dependency to the model asset tags feature but it was not included in the 2305 release.

I've added this commit to the hotfix that removes that dependency: o3de/o3de@b7b9e1e

[amzn-changml]

The hotfix branch is now being built in AR for validation: https://jenkins.build.o3de.org/view/O3DE/job/O3DE/job/hotfix%252F23.05.1/2/

[amzn-changml]

There's another commit we need to cherry-pick, which defines the build node for the snap and deb package on Ubuntu 20 (it was set to Ubuntu 22 for 2305): o3de/o3de#16094.

This does not affect AR, but it does when we create the installers. I'll add that to the list.

[amzn-changml]

AR for the hotfix branch is green. I'm building the installer packages as we speak. Once done, they will be located here:

Windows - https://o3debinaries.org/hotfix-23.05.1/Latest/Windows/o3de_installer.exe
Deb - https://o3debinaries.org/hotfix-23.05.1/Latest/Linux/O3DE_latest.deb
Snap - https://o3debinaries.org/23.05.1/Linux/o3de_23.05.1_amd64.snap

[Ulrick28]

I have put in the request for QA.

[brianherrera]

After testing is done, I'll start the process to merge the hotfix branch to main and create the release tag.

[Ulrick28]

Testing is complete, no new issues found. We should start the merge and release tag.

[brianherrera]

PR to merge hotfix branch to main: o3de/o3de#16423

[lemonade-dm]

For the hotfix release, we should also update the "engine_version" field in the main branch engine.json file.

The version listed is 1.2.0 and should be updated to 1.2.1 per the RFC for the Engine, Project and Gem Versions

[amzn-changml]

For the hotfix release, we should also update the "engine_version" field in the main branch engine.json file.

The version listed is 1.2.0 and should be updated to 1.2.1 per the RFC for the Engine, Project and Gem Versions

I've updated the engine.json version to 1.2.1. Will also start the re-building process for the install packages

[amzn-changml]

We need another change related to the snap packaging: o3de/o3de#15906. This fixes the snap for Ubuntu20. This also doesn't affect AR, but the snap will be rebuilt (which is already starting)

[amzn-changml]

PR for changes to o3debinaries.org queued: o3de/o3debinaries.org#8

[amzn-changml]

Installer binaries updated. Snap store update is going through Canonical AR

[brianherrera]

PR merged to main. LFS objects pushed to release tag.

[amzn-changml]

o3debinaries.org and snap store page updated

[amzn-changml]

Preliminary changelog:

-- Removed PyYAML from the python dependencies: o3de/o3de#16395

• Reason: This failed python initialization during build and installation, even though it was unused. This is due to a unpinned downstream Cython dependancy, which builds PyYAML at install even though its version is pinned

-- Fixes for first time editor crashes: o3de/o3de#16197

• Reason: Resolves race conditions when the DiffuseProbeGem is enabled, which crashes both AssetProcessor and Editor

-- Updates to Snap package dependancies: o3de/o3de#15906, o3de/o3de#16372, o3de/o3de#16094

• Reason: Retargets the base OS for the snap package for Ubuntu 20.04 to better support the minimum spec. Optimizes package dependancies to reduce overall package size

-- Bump Engine version to 1.2.1

[vincent6767]

Thanks, everyone! I believe the next step is to inform marketing about the hotfix release. Can you confirm? @tonybalandiuk @Ulrick28

[tonybalandiuk]

done @vincent6767
https://discord.com/channels/805939474655346758/826862580618100786/1133884778060456067

[vincent6767]

@tonybalandiuk @Ulrick28 For the release notes, it seems we can just use what Michael has provided and share it with the marketing team. Thoughts? #205 (comment)

Updates

1. There is no response from the marketing team. I wonder whether they still need hotfix release notes or not.

[vincent6767]

O3DE Hotfix Release Notes 23.05.1

This hotfix addresses the following issues:

• Removed PyYAML from the Python dependencies to prevent Python initialization failures during build and installation. PyYAML was an unused dependency that conflicted with a downstream Cython dependency. (Remove PyYAML o3de#16395)
• Fixed first-time editor crashes due to race conditions in the DiffuseProbeGem that caused crashes in the AssetProcessor and the Editor when the gem was enabled. (Fix the first-time Editor crash bugs o3de#16197)
• Updated the snap package dependencies and base OS to Ubuntu 20.04 for better compatibility and performance. Reduced the snap package size by optimizing the dependencies. (Fix for corrupt binaries in a snap package o3de#15906, Update dependancies for snap and disable patchelf for drivers o3de#16372, Update the deb and snap package node label o3de#16094)

---

Let me know whether this is accurate @amzn-changml @tonybalandiuk @Ulrick28 If no concerns, I'll share this with the marketing team next week.

[amzn-changml]

Just a suggestion for clarity:

Removed PyYAML from the Python dependencies to prevent Python initialization failures during build and installation. PyYAML was an unused dependency that caused conflicts conflicted with a downstream Cython dependency. (o3de/o3de#16395)

Let me know whether this is accurate @amzn-changml @tonybalandiuk @Ulrick28 If no concerns, I'll share this with the marketing team next week.

Other than that, this looks reasonable and accurate

[tonybalandiuk]

@vincent6767 see above ^

[vincent6767]

@amzn-changml Thanks! I updated the hotfix release notes.

[vincent6767]

All tasks are done. Thanks to everyone who is helping release the hotfix! I'll close this GHI in two days. Let me know if you have any concerns!