Enable checks to make sure SDK release requests meet release requirement.

[lirenhe]

Tracking this as requirement for the SDK Release Readiness epic - #5659

Data Plane: New Beta Requirements

• SDK Breaking changes are reviewed and approved. This is done via openapi Hub in the PR for management plane API reviews.

• GAP: What is the process for data plane breaking changes?

• For new services, package name/namespace are approved by archboard via APIView. During the introduction meeting namespace review and approval should be on the agenda. We need to know which APIView is in review for approval and need status to show user in the release planner.

• API readiness must be met first. (first milestone)

• For new services, we should release beta first and then release stable SDK after one month.

• Pull Request for SDK Release must be from public repo.

• CLARIFICATION: Need to provide timeline on when the PR should be in the public repo to release SDKs (i.e. 3 days before code freeze for monthly release, etc).

• Testing and samples: At least one test per language and one sample for the most common scenario of usage.

• Onboarded to docs repo for updates to existing docs or new docs.

Data Plane: New Stable Requirements

• For new services, we should release beta first and then release stable SDK after one month. (Need to check to ensure previous beta packages were released).
• API readiness must be met first. (first milestone)
• Pull Request for SDK Release must be from public repo.
• CLARIFICATION: Need to provide timeline on when the PR should be in the public repo to release SDKs (i.e. 3 days before code freeze for monthly release, etc).
• Testing and samples: at least one test per shipped method and 4 samples for the most common scenarios of usage.
• APIViews approved for all languages. We need to know which APIView is in review for approval and need status to show user in the release planner.
• Onboarded to docs repo for updates to existing docs or new docs.

Data Plane: New Beta Version Requirements TBD

Data Plane: New Stable Version Requirements TBD

Management Plane: New Beta

• For new services, package name/namespace are approved by archboard via APIView. There is a GitHub issue used to track management plane namespace approval. We should discuss if we want to keep the process as-is or if we want the arch board use APIView as they are doing for data plane namespace approvals.
• API readiness must be met first. (first milestone)
• non-RPaaS services: We must validate that the Swagger KPI quality or Early Phase Reports are green for the API version(s) that the product is enabled.
• GAP: How do we automate Swagger KPIs are green for non-RPaaS Services? Is there a way to query and get results and if there are errors, provide link to the Swagger KPI dashboard?
• For new services, we should release beta first and then release stable SDK after one month.
• GAP: How do we enforce this via the Release Planner apps?
• Pull Request for SDK Release must be from public repo.
• CLARIFICATION: Need to provide timeline on when the PR should be in the public repo to release SDKs (i.e. 3 days before code freeze for monthly release, etc).
• Onboarded to docs repo for updates to existing docs or new docs.
• Testing and samples: At least one test and one sample for .NET

Management Plane: New Stable

• API readiness must be met first. (first milestone)
• non-RPaaS services: We must validate that the Swagger KPI quality or Early Phase Reports are green for the API version(s) that the product is enabled.
• GAP: How do we automate Swagger KPIs are green for non-RPaaS Services? Is there a way to query and get results and if there are errors, provide link to the Swagger KPI dashboard?
• For new services, we should release beta first and then release stable SDK after one month.
• GAP: How do we enforce this via the Release Planner apps?
• Pull Request for SDK Release must be from public repo.
• CLARIFICATION: Need to provide timeline on when the PR should be in the public repo to release SDKs (i.e. 3 days before code freeze for monthly release, etc).
• Onboarded to docs repo for updates to existing docs or new docs.
• Testing and samples: at least one test per shipped method and 4 samples for the most common scenarios of usage.

Management Plane New Beta Version TBD
Management Plane New Stable Version TBD

Outstanding discussions in the comments:

SDK Versions: How to identify which SDK version is mapped to a release plan and better enforce consistent versioning

• [Mariana] As part of the Creation of the Release Plan, ask the user to input the version of the SDK.
• [Mariana]If we are generating the SDKs, it means that there are no previous ones, so def this will be a beta release. Right?
• [Mariana]If we are not generating the SDKs we will need to ask the user for input on where the code for the SDKs currently is and with that information look at the version?
• [Mariana]Other idea, could we look into every language repo and see if there is an SDK already there, and if there is, what version it has? We should already have this logic in our release pipelines to determine the version so there might be a script or something we could leverage here.
• [Wes] Currently SDK version validation is done manually in the SDK PR where people need to ensure the version is set correctly and merged into the repo before we can do the release.
• [Wes] We could do these status checks, like ApiView approval, by looking at the PR pipeline. If we need to pull this status after the PR for some reason we could consider pulling from the scheduled build or even the release build we need to trigger for doing the final release.
• [Wes] Another option is to use the prepare-release script which we currently require today before doing a release and it actually checks the ApiView status as part of that check among other release checks. We need to figure out how to include that into our process in the release planner, even if we don't call it directly we need to do all the work it currently does (both the checks as well as updates to the release DevOps items).
• [Wes] No matter what we end up doing here we need to use our source code in github as the source of truth and get it from there. Our pipelines use the source code as the truth as well which is why they are good option to do these release checks. Using our pipelines also helps us eliminate duplication of all the checks we are already doing there.

**Does our requirements change based on the product type and maturity of the service and/or SDKs? **

• [La Donna] What about new features? Is it okay for a service to decide that they will only maintain the stable API version and stable SDKs (management and/or data plane)? This has been an ask from team's with management plane APIs/SDKs.

[dw511214992]

For the 1st requirement, we can add it to sign off page.

For the second requirement, there is a archboard review to review the namespace of sdk package in reviewing swagger PR. It's tracked by a review ticket, and it will be assiciated with the powerapps.

For the 3rd requirement, is it required? I remember service team can release stable sdk in the first release.

[ladonnaq]

@dw511214992 We need to keep bullet 3. I just had an email discussion on this topic with Elendil/Josephine/Laurent, as it came up with Azure Load Test APEX signoff. This is part of the APEX launch readiness requirements. It is also stated on the SDK policies page - https://azure.github.io/azure-sdk/policies_releases.html (see screenshot below)

[josefree]

Per discussion, there is an agreement the 3 requirements are valid. We should consider updating or adding new feature in next iteration. BTW, what's the 'urgent' criteria? what's the expectation for 'urgent' issues? This definitely requires a design spec first.

[live1206]

For new services, package name/namespace are approved by archboard.

Add namespace review in data-plane API Readiness App: #5320
Need to get the criteria from SDK issue when the namespace is approved

[live1206]

SDK Breaking changes are reviewed and approved.

Need to investigate how to detect the SDK breaking changes

[josefree]

For new services, package name/namespace are approved by archboard.

1. The requirement is communicated after data plane apps deployment. Need to add the same feature implemented in the Management plane apps.
2. the previous design only setting up the link between the issue for naming review and the release plan. There is no final check if the review is done with approval. Need to add this logic in SDK release app. However, @ladonnaq
3. Could you help to clarify how system could know the review is completed with approval? For example this issue is closed https://github.com/Azure/azure-sdk-pr/issues/847 with approval on each language in the comments. However, system may not be smart enough to tell if it is all approved based on unstructured comments. Could we define some labels that represent approval?

[live1206]

Could you help to clarify how system could know the review is completed with approval? For example this issue is closed https://github.com/Azure/azure-sdk-pr/issues/847 with approval on each language in the comments. However, system may not be smart enough to tell if it is all approved based on unstructured comments. Could we define some labels that represent approval?

@maririos Could you help to confirm that ApiView can provide the approval info for each language? @ladonnaq mentioned we could leverage ApiView for this.

[maririos]

@maririos Could you help to confirm that ApiView can provide the approval info for each language? @ladonnaq mentioned we could leverage ApiView for this.

Correct. APIView currently has the information if a Data plane package name has been approved. You can use the API for this. @praveenkuttappan can help with the details.

For management plane, the goal is to have the same logic which means use APIVIew for the approval. This issue is tracking that work. For the purposes of our tool, we should build the logic to query APIView and have it disable until it is implemented. No point on adding logic to understand GH commetns

[zzvswxy]

Add a conclusion.

• For new services, package name/namespace are approved by archboard.

For naming review, only one review is required in the life cycle of a product.

[maririos]

Add a conclusion.

• For new services, package name/namespace are approved by archboard.

For naming review, only one review is required in the life cycle of a product.

@zzvswxy how will this affect the logic? are you thinking on saving the language approval in Dataverse so the tool is not constantly asking for it?

[zzvswxy]

how will this affect the logic?

We need to detect the released SDK is first beta/GA or not. Can you give any suggestions? Thanks

are you thinking on saving the language approval in Dataverse so the tool is not constantly asking for it?

Yes, if we solve the question above, we can save this info in dataverse.

[zzvswxy]

Update:

• SDK Breaking changes are reviewed and approved.

For DPG: We don't have a way to detect SDK had breaking. For management, we will add it in future version.

• For new services, package name/namespace are approved by archboard.

Move it to API readiness App to detect. But need more discuss with apiview team.

• For new services, we should release beta first and then release stable SDK after one month.

Need more disguss, will consider it when we can detect whether the SDK is first beta or not.

[maririos]

We need to detect the released SDK is first beta/GA or not. Can you give any suggestions? Thanks

Couple ideas:

• As part of the Creation of the Release Plan, ask the user to input the version of the SDK.
• If we are generating the SDKs, it means that there are no previous ones, so def this will be a beta release. Right?
• If we are not generating the SDKs we will need to ask the user for input on where the code for the SDKs currently is and with that information look at the version?
• Other idea, could we look into every language repo and see if there is an SDK already there, and if there is, what version it has? We should already have this logic in our release pipelines to determine the version so there might be a script or something we could leverage here.

@weshaggard for suggestions here.

Yes, if we solve the question above, we can save this info in dataverse.

I would prefer not to save the information in Dataverse and always reach out for the answer. If one day, something changes in the process, and there is a discrepancy between Dataverse and the real version of the library, we will have problems.

[weshaggard]

I would prefer not to save the information in Dataverse and always reach out for the answer. If one day, something changes in the process, and there is a discrepancy between Dataverse and the real version of the library, we will have problems.

It would be OK to store this in Dataverse as a cache but we shouldn't rely on it solely. We should be able to query the information for the real source of truth (i.e. github, package managers, etc) to always be able to rebuild the cache and display the current state in the release planner app.

[maririos]

@weshaggard Do you have suggestions for how to know the version of the SDKs?

[weshaggard]

SDK versions for a release plan is an entire topic itself because we will need to figure out how we set the version for the SDK we plan to release. Currently it is manual in the SDK PR where people need to ensure the version is set correctly and merged into the repo before we can do the release.

So, we could do these status checks, like ApiView approval, by looking at the PR pipeline. If we need to pull this status after the PR for some reason we could consider pulling from the scheduled build or even the release build we need to trigger for doing the final release.

Another option is to use the prepare-release script which we currently require today before doing a release and it actually checks the ApiView status as part of that check among other release checks. We need to figure out how to include that into our process in the release planner, even if we don't call it directly we need to do all the work it currently does (both the checks as well as updates to the release DevOps items).

No matter what we end up doing here we need to use our source code in github as the source of truth and get it from there. Our pipelines use the source code as the truth as well which is why they are good option to do these release checks. Using our pipelines also helps us eliminate duplication of all the checks we are already doing there.

[ladonnaq]

I will review this GitHub issue and create new issues for anything that is relevant.

[ladonnaq]

@praveenkuttappan @maririos Can you take a look at the GitHub issue and let me know if we should move any specific task to new issues. I think that some of the work that Praveen is doing right now will help with this but not sure that it will cover everything. I think that once a "release request" is raised with the DPG team, all of the pre-requisites should have been met. The only outstanding should be whatever is in the release pipelines, which I would expect that the DPG team checks and they either resolve or pull in the service partner to resolve.

[maririos]

We can close this issue. Most of the items here are process related and have either been addressed, been documented, or are open things as a team we need to figure out.
For tooling specifics, the tooling has changed and no longer touches this points