Add the Flyte agent to provision and manage K8s (data) service for deep learning (GNN) use cases

[shuyingliang]

Why are the changes needed?

Graph Neural Networks are critical for understanding complex relationships across LinkedIn's professional networks. However, training these models at scale involves intricate data loading, sampling, and processing across multiple nodes and GPUs. The missing piece is the infrastructure to support how and where to run these Kubernetes data services, making them scalable and reliable along with the training or inference processes.

To simplify the complex orchestration pipeline, we decided to leverage flyte agent framework to provision and manage the data services for GNN use case.

What changes were proposed in this pull request?

This PR adds the flyte agent to create/update/delete the K8s statefulset and service.

How was this patch tested?

• The same code (with removed company related internal environments and set up) has been running in production along with the training job MPIJobs (for deep learning GNN training) or TFJob (for offline inference)
• This is also tested in local sandbox

Setup process

pip install flytekitplugins-k8sdataservice

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Docs link

Blog from Flyte community sync

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.46%. Comparing base (f99d50e) to head (944a500).
Report is 4 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3004       +/-   ##
===========================================
+ Coverage   51.08%   90.46%   +39.38%     
===========================================
  Files         201      100      -101     
  Lines       21231     4920    -16311     
  Branches     2731        0     -2731     
===========================================
- Hits        10846     4451     -6395     
+ Misses       9787      469     -9318     
+ Partials      598        0      -598     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pingsutw]

This is amazing!!! leave some minor comments