apiserver cannot find minio object store if minio application is named anything other than minio

[ca-scribner]

Bug Description

Note: This bug is against 1.8-updates-dev-branch, not against main

If kfp-api is deployed without a minio application called minio, it fails to find the object store. I think this is because there are issues with passing config to the running application? This is discussed more in #367. It appears that deploying minio as minio works because it happens to be the default name expected in the application(?).

This was introduced in #354, which refactored the configuration passing from file to environment variable but appears to have missed some variables like object storage host/port

To Reproduce

Deploy with bundle:

bundle.yaml

bundle: kubernetes
applications:
  argo-controller:
    charm: argo-controller
    channel: latest/edge
    revision: 393
    series: focal
    resources:
      oci-image: 343
    scale: 1
    constraints: arch=amd64
    trust: true
  envoy:
    charm: envoy
    channel: latest/edge
    revision: 91
    series: focal
    resources:
      oci-image: 91
    scale: 1
    constraints: arch=amd64
  istio-ingressgateway:
    charm: istio-gateway
    channel: latest/edge
    revision: 715
    series: focal
    scale: 1
    options:
      kind: ingress
    constraints: arch=amd64
    trust: true
  istio-pilot:
    charm: istio-pilot
    channel: latest/edge
    revision: 704
    series: focal
    scale: 1
    options:
      default-gateway: kubeflow-gateway
    constraints: arch=amd64
    trust: true
  kfp-api:
    charm: kfp-api
    channel: latest/edge/pr-361
    revision: 941
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  kfp-db:
    charm: mysql-k8s
    channel: 8.0/edge
    revision: 110
    resources:
      mysql-image: 106
    scale: 1
    constraints: arch=amd64 mem=2048
    storage:
      database: kubernetes,1,1024M
    trust: true
  kfp-metadata-writer:
    charm: kfp-metadata-writer
    channel: latest/edge/pr-361
    revision: 29
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  kfp-persistence:
    charm: kfp-persistence
    channel: latest/edge/pr-361
    revision: 945
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  kfp-profile-controller:
    charm: kfp-profile-controller
    channel: latest/edge/pr-361
    revision: 905
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  kfp-schedwf:
    charm: kfp-schedwf
    channel: latest/edge/pr-361
    revision: 958
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  kfp-ui:
    charm: kfp-ui
    channel: latest/edge/pr-361
    revision: 940
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  kfp-viewer:
    charm: kfp-viewer
    channel: latest/edge/pr-361
    revision: 970
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  kfp-viz:
    charm: kfp-viz
    channel: latest/edge/pr-361
    revision: 894
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  kubeflow-dashboard:
    charm: kubeflow-dashboard
    channel: latest/edge
    revision: 448
    series: focal
    resources:
      oci-image: 672
    scale: 1
    constraints: arch=amd64
    trust: true
  kubeflow-profiles:
    charm: kubeflow-profiles
    channel: latest/edge
    revision: 350
    series: focal
    resources:
      kfam-image: 565
      profile-image: 563
    scale: 1
    constraints: arch=amd64
    trust: true
  kubeflow-roles:
    charm: kubeflow-roles
    channel: latest/edge
    revision: 183
    series: focal
    scale: 1
    constraints: arch=amd64
    trust: true
  minio2:
    charm: minio
    channel: latest/edge
    revision: 251
    series: focal
    resources:
      oci-image: 533
    scale: 1
    constraints: arch=amd64
    storage:
      minio-data: kubernetes,1,10240M
  mlmd:
    charm: mlmd
    channel: latest/edge
    revision: 124
    series: focal
    resources:
      oci-image: 127
    scale: 1
    constraints: arch=amd64
    storage:
      mlmd-data: kubernetes,1,10240M
relations:
- - argo-controller:object-storage
  - minio:object-storage
- - kfp-api:relational-db
  - kfp-db:database
- - kfp-api:kfp-api
  - kfp-persistence:kfp-api
- - kfp-api:kfp-api
  - kfp-ui:kfp-api
- - kfp-api:kfp-viz
  - kfp-viz:kfp-viz
- - kfp-api:object-storage
  - minio:object-storage
- - kfp-profile-controller:object-storage
  - minio:object-storage
- - kfp-ui:object-storage
  - minio:object-storage
- - kubeflow-profiles:kubeflow-profiles
  - kubeflow-dashboard:kubeflow-profiles
- - kubeflow-dashboard:links
  - kfp-ui:dashboard-links
- - mlmd:grpc
  - envoy:grpc
- - mlmd:grpc
  - kfp-metadata-writer:grpc
- - istio-pilot:istio-pilot
  - istio-ingressgateway:istio-pilot
- - istio-pilot:ingress
  - kfp-ui:ingress
- - envoy:ingress
  - istio-pilot:ingress
- - istio-pilot:ingress
  - kubeflow-dashboard:ingress

then watch logs for apiserver using kubectl logs kfp-api-0 -c ml-pipeline-api-server

Environment

Tested with juju 3.1, microk8s 1.25-strict, and above bundle

Relevant log output

example failures:

kubectl logs kfp-api-0 -c ml-pipeline-api-server -f

2023-11-01T17:20:17.487Z [pebble] HTTP API server listening on ":38813".
2023-11-01T17:20:17.487Z [pebble] Started daemon.
2023-11-01T17:22:31.396Z [pebble] GET /v1/services?names= 2.177115ms 200
2023-11-01T17:22:31.406Z [pebble] GET /v1/plan?format=yaml 1.493758ms 200
2023-11-01T17:22:48.929Z [pebble] GET /v1/plan?format=yaml 347.46µs 200
2023-11-01T17:22:48.939Z [pebble] POST /v1/layers 1.426821ms 200
2023-11-01T17:22:48.989Z [pebble] POST /v1/services 21.296452ms 202
2023-11-01T17:22:48.998Z [pebble] Service "apiserver" starting: bash -c 'sleep 1.1 && /bin/apiserver --config=/config --sampleconfig=/config/sample_config.json -logtostderr=true '
2023-11-01T17:22:50.009Z [pebble] GET /v1/changes/1/wait?timeout=4.000s 1.014947211s 200
2023-11-01T17:22:50.515Z [apiserver] I1101 17:22:50.515110      23 client_manager.go:160] Initializing client manager
2023-11-01T17:22:50.518Z [apiserver] I1101 17:22:50.516518      23 config.go:57] Config DBConfig.ExtraParams not specified, skipping
2023-11-01T17:22:58.881Z [pebble] GET /v1/plan?format=yaml 1.721058ms 200
2023-11-01T17:22:58.890Z [pebble] POST /v1/layers 1.198154ms 200
2023-11-01T17:22:58.931Z [pebble] POST /v1/services 10.78643ms 202
2023-11-01T17:22:58.948Z [pebble] GET /v1/changes/2/wait?timeout=4.000s 14.352408ms 200
2023-11-01T17:24:45.092Z [pebble] GET /v1/plan?format=yaml 2.233712ms 200
2023-11-01T17:24:45.100Z [pebble] POST /v1/layers 917.223µs 200
2023-11-01T17:24:45.169Z [pebble] POST /v1/services 11.829742ms 202
2023-11-01T17:24:45.183Z [pebble] GET /v1/changes/3/wait?timeout=4.000s 10.954627ms 200
2023-11-01T17:24:45.462Z [pebble] GET /v1/checks?names=kfp-api-up 612.191µs 200
2023-11-01T17:25:12.022Z [pebble] GET /v1/plan?format=yaml 751.033µs 200
2023-11-01T17:29:06.638Z [pebble] GET /v1/plan?format=yaml 489.109µs 200
2023-11-01T17:29:06.642Z [pebble] POST /v1/layers 677.329µs 200
2023-11-01T17:29:06.660Z [pebble] POST /v1/services 4.159969ms 202
2023-11-01T17:29:06.671Z [pebble] GET /v1/changes/4/wait?timeout=4.000s 10.279515ms 200
2023-11-01T17:29:06.895Z [pebble] GET /v1/checks?names=kfp-api-up 85.649µs 200
2023-11-01T17:29:39.791Z [apiserver] F1101 17:29:39.791389      23 minio.go:76] Failed to create Minio client. Error: Error while creating minio client: Endpoint:  does not follow ip address or domain name standards.: Endpoint:  does not follow ip address or domain name standards.
2023-11-01T17:29:39.815Z [pebble] Service "apiserver" stopped unexpectedly with code 255
2023-11-01T17:29:39.815Z [pebble] Service "apiserver" on-failure action is "restart", waiting ~500ms before restart (backoff 1)
2023-11-01T17:29:40.343Z [pebble] Service "apiserver" starting: bash -c 'sleep 1.1 && /bin/apiserver --config=/config --sampleconfig=/config/sample_config.json -logtostderr=true '
2023-11-01T17:29:41.544Z [apiserver] I1101 17:29:41.544909      63 client_manager.go:160] Initializing client manager
2023-11-01T17:29:41.545Z [apiserver] I1101 17:29:41.545109      63 config.go:57] Config DBConfig.ExtraParams not specified, skipping
2023-11-01T17:32:38.608Z [pebble] GET /v1/services?names= 109.461µs 200
2023-11-01T17:32:44.230Z [pebble] GET /v1/files?action=list&path=%2F 1.108685ms 200
2023-11-01T17:32:47.054Z [pebble] GET /v1/files?action=list&path=%2Fconfig 300.03µs 200
2023-11-01T17:33:01.778Z [pebble] POST /v1/exec 647.601µs 400
2023-11-01T17:33:13.603Z [pebble] POST /v1/exec 201.846µs 400
2023-11-01T17:33:19.414Z [pebble] POST /v1/exec 6.278075ms 202
2023-11-01T17:33:19.423Z [pebble] GET /v1/tasks/5/websocket/control 6.298648ms 200
2023-11-01T17:33:19.424Z [pebble] GET /v1/tasks/5/websocket/stdio 145.64µs 200
2023-11-01T17:33:28.385Z [pebble] GET /v1/changes/5/wait 8.960705919s 200
2023-11-01T17:34:06.644Z [pebble] Check "kfp-api-up" failure 1 (threshold 3): Get "http://localhost:8888/apis/v1beta1/healthz": dial tcp [::1]:8888: connect: connection refused
2023-11-01T17:34:36.692Z [pebble] GET /v1/plan?format=yaml 452.927µs 200
2023-11-01T17:34:36.697Z [pebble] POST /v1/layers 1.576055ms 200
2023-11-01T17:34:36.715Z [pebble] POST /v1/services 4.787856ms 202
2023-11-01T17:34:36.730Z [pebble] GET /v1/changes/6/wait?timeout=4.000s 13.826906ms 200
2023-11-01T17:34:36.971Z [pebble] GET /v1/checks?names=kfp-api-up 82.805µs 200
2023-11-01T17:36:22.532Z [apiserver] F1101 17:36:22.532228      63 minio.go:76] Failed to create Minio client. Error: Error while creating minio client: Endpoint:  does not follow ip address or domain name standards.: Endpoint:  does not follow ip address or domain name standards.
2023-11-01T17:36:22.536Z [pebble] Service "apiserver" stopped unexpectedly with code 255
2023-11-01T17:36:22.536Z [pebble] Service "apiserver" on-failure action is "restart", waiting ~500ms before restart (backoff 1)
2023-11-01T17:36:23.052Z [pebble] Service "apiserver" starting: bash -c 'sleep 1.1 && /bin/apiserver --config=/config --sampleconfig=/config/sample_config.json -logtostderr=true '
2023-11-01T17:36:24.227Z [apiserver] I1101 17:36:24.227802      80 client_manager.go:160] Initializing client manager
2023-11-01T17:36:24.228Z [apiserver] I1101 17:36:24.228125      80 config.go:57] Config DBConfig.ExtraParams not specified, skipping
2023-11-01T17:37:37.011Z [pebble] POST /v1/exec 6.93502ms 202
2023-11-01T17:37:37.018Z [pebble] GET /v1/tasks/7/websocket/control 5.607235ms 200
2023-11-01T17:37:37.019Z [pebble] GET /v1/tasks/7/websocket/stdio 304.864µs 200
2023-11-01T17:37:37.034Z [pebble] GET /v1/changes/7/wait 13.254476ms 200
2023-11-01T17:37:39.932Z [pebble] POST /v1/exec 8.235057ms 202
2023-11-01T17:37:39.939Z [pebble] GET /v1/tasks/8/websocket/control 6.073866ms 200
2023-11-01T17:37:39.940Z [pebble] GET /v1/tasks/8/websocket/stdio 118.281µs 200
2023-11-01T17:37:39.958Z [pebble] GET /v1/changes/8/wait 16.469671ms 200
2023-11-01T17:37:44.222Z [pebble] POST /v1/exec 8.204988ms 202
2023-11-01T17:37:44.230Z [pebble] GET /v1/tasks/9/websocket/control 6.569159ms 200
2023-11-01T17:37:44.230Z [pebble] GET /v1/tasks/9/websocket/stdio 189.88µs 200
2023-11-01T17:37:44.250Z [pebble] GET /v1/changes/9/wait 17.820131ms 200
2023-11-01T17:37:49.390Z [pebble] POST /v1/exec 10.865234ms 202
2023-11-01T17:37:49.399Z [pebble] GET /v1/tasks/10/websocket/control 4.813009ms 200
2023-11-01T17:37:49.399Z [pebble] GET /v1/tasks/10/websocket/stdio 144.091µs 200
2023-11-01T17:37:49.416Z [pebble] GET /v1/changes/10/wait 16.009475ms 200
2023-11-01T17:37:53.373Z [pebble] POST /v1/exec 305.187µs 400
2023-11-01T17:37:55.617Z [pebble] POST /v1/exec 11.324035ms 202
2023-11-01T17:37:55.625Z [pebble] GET /v1/tasks/11/websocket/control 6.191422ms 200
2023-11-01T17:37:55.626Z [pebble] GET /v1/tasks/11/websocket/stdio 99.221µs 200
2023-11-01T17:37:55.643Z [pebble] GET /v1/changes/11/wait 16.937856ms 200
2023-11-01T17:37:59.223Z [pebble] POST /v1/exec 10.113631ms 202
2023-11-01T17:37:59.233Z [pebble] GET /v1/tasks/12/websocket/control 8.554426ms 200
2023-11-01T17:37:59.234Z [pebble] GET /v1/tasks/12/websocket/stdio 121.123µs 200
2023-11-01T17:37:59.254Z [pebble] GET /v1/changes/12/wait 19.098103ms 200
2023-11-01T17:39:08.324Z [pebble] GET /v1/plan?format=yaml 494.677µs 200
2023-11-01T17:39:08.329Z [pebble] POST /v1/layers 992.744µs 200
2023-11-01T17:39:08.363Z [pebble] POST /v1/services 18.185372ms 202
2023-11-01T17:39:08.395Z [pebble] GET /v1/changes/13/wait?timeout=4.000s 29.491044ms 200
2023-11-01T17:39:08.611Z [pebble] GET /v1/checks?names=kfp-api-up 94.444µs 200
2023-11-01T17:42:39.007Z [apiserver] F1101 17:42:39.006993      80 minio.go:76] Failed to create Minio client. Error: Error while creating minio client: Endpoint:  does not follow ip address or domain name standards.: Endpoint:  does not follow ip address or domain name standards.
2023-11-01T17:42:39.023Z [pebble] Service "apiserver" stopped unexpectedly with code 255
2023-11-01T17:42:39.023Z [pebble] Service "apiserver" on-failure action is "restart", waiting ~500ms before restart (backoff 1)
2023-11-01T17:42:39.563Z [pebble] Service "apiserver" starting: bash -c 'sleep 1.1 && /bin/apiserver --config=/config --sampleconfig=/config/sample_config.json -logtostderr=true '
2023-11-01T17:42:40.747Z [apiserver] I1101 17:42:40.747558     102 client_manager.go:160] Initializing client manager
2023-11-01T17:42:40.751Z [apiserver] I1101 17:42:40.750801     102 config.go:57] Config DBConfig.ExtraParams not specified, skipping
2023-11-01T17:44:08.330Z [pebble] Check "kfp-api-up" failure 1 (threshold 3): Get "http://localhost:8888/apis/v1beta1/healthz": dial tcp [::1]:8888: connect: connection refused
2023-11-01T17:44:58.245Z [pebble] GET /v1/plan?format=yaml 447.912µs 200
2023-11-01T17:44:58.249Z [pebble] POST /v1/layers 574.189µs 200
2023-11-01T17:44:58.271Z [pebble] POST /v1/services 10.046961ms 202
2023-11-01T17:44:58.300Z [pebble] GET /v1/changes/14/wait?timeout=4.000s 27.561082ms 200
2023-11-01T17:44:58.527Z [pebble] GET /v1/checks?names=kfp-api-up 89.478µs 200
2023-11-01T17:48:45.915Z [pebble] GET /v1/changes?select=all 451.838µs 200
2023-11-01T17:49:18.498Z [pebble] Cannot get service-name for change 8
2023-11-01T17:49:18.498Z [pebble] Cannot get service-name for change 9
2023-11-01T17:49:18.498Z [pebble] Cannot get service-name for change 10
2023-11-01T17:49:18.498Z [pebble] Cannot get service-name for change 11
2023-11-01T17:49:18.498Z [pebble] Cannot get service-name for change 12
2023-11-01T17:49:18.498Z [pebble] Cannot get service-name for change 5
2023-11-01T17:49:18.498Z [pebble] Cannot get service-name for change 7
2023-11-01T17:49:18.498Z [pebble] GET /v1/changes?for=apiserver&select=all 302.607µs 200
2023-11-01T17:49:34.824Z [pebble] GET /v1/warnings 99.611µs 200
2023-11-01T17:49:40.323Z [apiserver] F1101 17:49:40.322972     102 minio.go:76] Failed to create Minio client. Error: Error while creating minio client: Endpoint:  does not follow ip address or domain name standards.: Endpoint:  does not follow ip address or domain name standards.
2023-11-01T17:49:40.328Z [pebble] Service "apiserver" stopped unexpectedly with code 255
2023-11-01T17:49:40.328Z [pebble] Service "apiserver" on-failure action is "restart", waiting ~500ms before restart (backoff 1)
2023-11-01T17:49:40.868Z [pebble] Service "apiserver" starting: bash -c 'sleep 1.1 && /bin/apiserver --config=/config --sampleconfig=/config/sample_config.json -logtostderr=true '
2023-11-01T17:49:42.033Z [apiserver] I1101 17:49:42.033853     118 client_manager.go:160] Initializing client manager
2023-11-01T17:49:42.033Z [apiserver] I1101 17:49:42.033956     118 config.go:57] Config DBConfig.ExtraParams not specified, skipping
2023-11-01T17:49:45.167Z [pebble] GET /v1/warnings 84.51µs 200
2023-11-01T17:49:51.241Z [pebble] GET /v1/logs?n=30 520.153µs 200
2023-11-01T17:49:58.251Z [pebble] Check "kfp-api-up" failure 1 (threshold 3): Get "http://localhost:8888/apis/v1beta1/healthz": dial tcp [::1]:8888: connect: connection refused
2023-11-01T17:50:55.980Z [pebble] GET /v1/plan?format=yaml 761.581µs 200
2023-11-01T17:50:55.986Z [pebble] POST /v1/layers 858.636µs 200
2023-11-01T17:50:56.014Z [pebble] POST /v1/services 10.523863ms 202
2023-11-01T17:50:56.046Z [pebble] GET /v1/changes/15/wait?timeout=4.000s 30.09712ms 200
2023-11-01T17:50:56.295Z [pebble] GET /v1/checks?names=kfp-api-up 76.77µs 200
2023-11-01T17:55:55.987Z [pebble] Check "kfp-api-up" failure 1 (threshold 3): Get "http://localhost:8888/apis/v1beta1/healthz": dial tcp [::1]:8888: connect: connection refused
2023-11-01T17:56:11.867Z [apiserver] F1101 17:56:11.867047     118 minio.go:76] Failed to create Minio client. Error: Error while creating minio client: Endpoint:  does not follow ip address or domain name standards.: Endpoint:  does not follow ip address or domain name standards.
2023-11-01T17:56:11.891Z [pebble] Service "apiserver" stopped unexpectedly with code 255
2023-11-01T17:56:11.891Z [pebble] Service "apiserver" on-failure action is "restart", waiting ~500ms before restart (backoff 1)
2023-11-01T17:56:12.413Z [pebble] Service "apiserver" starting: bash -c 'sleep 1.1 && /bin/apiserver --config=/config --sampleconfig=/config/sample_config.json -logtostderr=true '
2023-11-01T17:56:13.572Z [apiserver] I1101 17:56:13.572589     130 client_manager.go:160] Initializing client manager
2023-11-01T17:56:13.572Z [apiserver] I1101 17:56:13.572738     130 config.go:57] Config DBConfig.ExtraParams not specified, skipping
2023-11-01T17:56:33.465Z [pebble] GET /v1/plan?format=yaml 800.204µs 200
2023-11-01T17:56:33.471Z [pebble] POST /v1/layers 1.099992ms 200
2023-11-01T17:56:33.525Z [pebble] POST /v1/services 34.762332ms 202
2023-11-01T17:56:33.536Z [pebble] GET /v1/changes/16/wait?timeout=4.000s 9.024238ms 200
2023-11-01T17:56:33.790Z [pebble] GET /v1/checks?names=kfp-api-up 121.566µs 200
2023-11-01T18:00:41.319Z [pebble] GET /v1/plan?format=yaml 628.728µs 200
2023-11-01T18:00:41.323Z [pebble] POST /v1/layers 897.92µs 200
2023-11-01T18:00:41.345Z [pebble] POST /v1/services 9.736479ms 202
2023-11-01T18:00:41.374Z [pebble] GET /v1/changes/17/wait?timeout=4.000s 25.488785ms 200
2023-11-01T18:00:41.625Z [pebble] GET /v1/checks?names=kfp-api-up 114.593µs 200
2023-11-01T18:02:35.315Z [apiserver] F1101 18:02:35.315226     130 minio.go:76] Failed to create Minio client. Error: Error while creating minio client: Endpoint:  does not follow ip address or domain name standards.: Endpoint:  does not follow ip address or domain name standards.

Additional context

No response

[ca-scribner]

to summarise further, after #354 kfp-api's apiserver does not use the provided object store hostname/port to find the object store. Instead, because we do not pass any value, apiserver uses its built-in default of "minio" (or maybe "minio.kubeflow"?) to look for the object store. This means it accidentally works if charmed kubeflow includes a minio application specifically named minio, but it doesn't work if you do say juju deploy minio minio2; juju relate minio2 kfp-api

[DnPlas]

The pipelines apiserver will not use the application name, but the object store host and port, which is built from minio's Service, Namespace and Port. These are taken from env variables to form something like minio-service.kubeflow:9090. The apiserver will use a default minio service (see here for more details) in the absence of those:

// If the env vars do not exist, we guess that we are running in KFP multi user mode, so default minio service should be minio-service.kubeflow:9000.

The above is guaranteed by the minio charm as we have explicitly set the default Service name to minio-service, and since minio gets deployed in the kubeflow model, the namespace is also guaranteed. The application name does not really affect this behaviour. What is concerning, though, is the fact that the apiserver is trying to be restarted over and over again until it can actually see the minio service, but as described #367 that does not seem to happen correctly.

[ca-scribner]

After talking with @DnPlas we think that, with #375 merged, we can close this.

The original issue here and dnplas's comment are sort of both correct. afaict they each refer to a different version of the minio charm (original aligns to a pre-KF 1.7 minio charm, and @DnPlas comment is correct with a minio charm including this change. Either way though, with #375 merged both discussions should be settled.