TypeSpec data related to interval between start and completion of tasks, milestones, and overall release plan

[MrJustinB]

Related GitHub issues:

• Investigate the current time for E2E process for management plane from TypeSpec ready to SDK released (Committed for dilithium-1H)  #7878
• Investigate the current time for E2E process for data plane from TypeSpec ready to SDK released (Committed for dilithium-1H)  #7890

Most of metrics requested in related GitHub issues is not accessible via current engagement data model. We can revisit when we begin planning for Selenium to determine feasibility of obtaining the data.

Meeting notes:
• TypeSpec data dashboard: Justin and Mariana will work on creating a dashboard that shows the average and outlier times for service teams to go through the TypeSpec pipeline, as requested by Mario, the TypeSpec PM. We discussed a visual timeline with major milestones along the timeline and average time to get to each milestone and then show the outliers for each (i.e. which service teams were slower and which were faster) so that Mario can follow up to understand challenges faced.
• Data plane vs management plane: Mario wants to see the distinction between data plane and management plane services in the dashboard, to understand if there is a significant difference in their TypeSpec experience.
• TypeSpec adoption: Mario is less interested in tracking the adoption of TypeSpec, since it is now mandated for all service teams, but he would like to see how many services have moved to TypeSpec as a secondary metric.
Follow-up tasks:

• Create a dashboard for TypeSpec metrics: Figure out how to collect the data for the average and outlier times for service teams to go through the TypeSpec pipeline.
  Jonathon has provided PowerBI Report with data to Mario. Link - https://msit.powerbi.com/groups/3e17dcb0-4257-4a30-b843-77f47f1d4121/reports/a98bc8dc-dceb-4c04-a288-932e5ab5fed5?ctid=72f988bf-86f1-41af-91ab-2d7cd011db47&bookmarkGuid=4bac8027-4970-46f6-a150-e818878d8dd7&experience=power-bi

• Extend the PowerBI Report provided by Jonathon to include visuals. (Angelos and La Donna)
  Angelos is working on options for visuals. @mario-guerra - Angelos has one example created so far. Let us know if you find this visual useful.
  

[MrJustinB]

Mario: "Thanks Justin, one other thing that just occurred to me – it would be good to differentiate between greenfield and brownfield services in the averages."

[maririos]

• Questions to answer:
  • How long does it take to finish a release plan
  • How long does each milestone and each task in the milestone
• Be able to distinguish:
  • Management plane vs Data plane
  • TypeSpec or not TypeSpec

Ideas on what we should do:

• PowerBi dashboard from DataVerse diectly as the DB has the date the specific task or milestone was completed
• It shouldn't take more than 1 to 1.5 days building
• This is a temporary view as there are way too many variables to take this data as accurate

[ladonnaq]

@MrJustinB @maririos I believe that you are using this issue to track the work that you are doing for Mario for TypeSpec dashboard. I have added a table to track the metrics and information needed. I wanted to sync on the approach and todos so we capture for now and next semester.

• I think it would be best to create a dashboard view within the current release planner PowerBI report. We could start by showing the data we do have like release plans data for TypeSpec (data vs mgmt plane), upcoming releases for TypeSpec (based on release plan target date), etc. PowerBI is more user friendly and better for getting a program level view for reporting and understanding insights vs ADO. ADO is the tool we use for APEX because we are actively updating and maintaining the ADO artifacts throughout the engagement. (i.e. copying and pasting information that was in email but important like extension details, ....). It is also the tool most frequently used by service partners for tracking their upcoming new product/service releases and also the tool we use to track SDK releases.
• I consider this part of the work to continue to ensure that the engagement experience model is aligned across the systems of record so added it to this epic - New features needed to ensure alignment and increase productivity across the Azure SDK Service Partner Engagement Model  #7822.

.

[maririos]

I wanted to sync on the approach and todos so we capture for now and next semester.

This work is done from eng side and we have sent it to Mario who can now modify it and own it.
I don't expect Eng Team to do any other work in this dashboard

[maririos]

@JonathanCrd what do you think?

[MrJustinB]

I wanted to sync on the approach and todos so we capture for now and next semester.

This work is done from eng side and we have sent it to Mario who can now modify it and own it. I don't expect Eng Team to do any other work in this dashboard

That was not the expectation. My expectation is that we would own the dashboard and enhance over time. Mario's recent enhancement request is to be able to show a visual trend i.e. how are the time going up/down over time.

[maririos]

Eng exp team can def own it. I am saying Eng team gave all the data needed and now for visualization, we can have PM team work on that.
We can talk more about this tomorrow in our sync.

[JonathanCrd]

I would like to keep the reports separate and only include usage-related metrics in the current report. This will help to keep the data schema simple for the purpose of each report, especially considering the complexity of the relational database schema we have in Dataverse and all the information coming from ADO.

It is also the tool most frequently used by service partners for tracking their upcoming new product/service releases and also the tool we use to track SDK releases.

@ladonnaq are you talking about ADO or PowerBI? I wouldn't recommend using a PowerBI report for partner teams to trach their progress, since that is the goal of the Release Planner app.

Eng exp team can def own it. I am saying Eng team gave all the data needed

@maririos totally, we can create and update the data model in PowerBI as needed.

However, not all data is already calculated in the release planner tool. For example, in the TypeSpec report I created as part of this issue, I connected the Dataverse and ADO tables together to get stuff like service name, release plan info, if it is typespec or not, etc. But we didn't have the time each team spent on each task. I had to create a new column and write a formulate to calculate this. For some insights, new columns or quick calculations will need to be created from the data that we, the eng team, will provide to the dashboard.

[mario-guerra]

@maririos, @AngelosP, @MrJustinB, @ladonnaq would it be helpful to have a sync meeting to discuss the current state and clarify expectations one what the TypeSpec team wants to see?
CC @allenjzhang for awareness.

[ladonnaq]

Hello @mario-guerra, I sent you an email but also adding a comment to this GitHub issue. Can you look at the data in the report and the example visual that Angelos created? We can continue to improve this dashboard but I want to make sure that you (and the TypeSpec) find the data valuable. Also, we should only be looking at the 2024 data. The MVP Release Planner and data model was not complete for non-APEX scenarios until beginning of Feb 2024.