You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After 24 hours of creation, all logs belonging to a pipeline run disappear from the Charmed Kubeflow UI, despite the logs still being present in MinIO/mlpipeline (AWS S3). This leads to difficulty in troubleshooting and tracking the progress or failures of pipeline runs after the 24-hour period.
!aws --endpoint-url $MINIO_ENDPOINT_URL s3 ls s3://mlpipeline
[...]
PRE addition-pipeline-4g94d/
PRE addition-pipeline-4qwt4/
PRE download-preprocess-train-deploy-pipeline-8wjv9/
PRE mnist-pipeline-fcmgr/
[...]
!aws --endpoint-url $MINIO_ENDPOINT_URL s3 ls s3://mlpipeline/download-preprocess-train-deploy-pipeline-8wjv9/download-preprocess-train-deploy-pipeline-8wjv9-system-container-impl-1190848556/
2024-10-15 15:27:49 10796 main.log
To Reproduce
Deploy Charmed Kubeflow 1.9 using Juju.
Create a pipeline and run it.
After the run completes, observe that logs are available in the Kubeflow UI.
Wait for 24 hours after the pipeline run completes.
Attempt to view the pipeline logs in the UI again. Expected: Logs should still be accessible. Actual: Logs are no longer visible in the UI, but are still present in the underlying MinIO/mlpipeline (AWS S3).
Bug Description
After 24 hours of creation, all logs belonging to a pipeline run disappear from the Charmed Kubeflow UI, despite the logs still being present in MinIO/mlpipeline (AWS S3). This leads to difficulty in troubleshooting and tracking the progress or failures of pipeline runs after the 24-hour period.
To Reproduce
Expected: Logs should still be accessible.
Actual: Logs are no longer visible in the UI, but are still present in the underlying MinIO/mlpipeline (AWS S3).
Environment
CKF: 1.9/stable
minio: ckf-1.9/stable
argo-controller: 3.4/stable
Juju: 3.5.4
See the full bundle on: https://paste.ubuntu.com/p/NXXFhDqmVn/
Relevant Log Output
Additional Context
Notebook that is used to create a pipeline, which was ran on a notebook server with a GPU:
Could be related to upstream: kubeflow/pipelines#7617
The text was updated successfully, but these errors were encountered: