Monitoring stack - Adding custom metrics

[furedal]

Hi
I am new to k8 and struggling to get custom metrics passed on to Grafana.

I have a rails application that I want to monitor requests and workload using this yabeda-rails gem.
Im pretty sure that i have setup rails correctly as I can access https://MYAPP/metrics and it gives me the correct information (I added a sample in the bottom).

In the bottom i also included my Service and Deployment config (I removed all the clutter).
Now, because /metrics is available in my rails app on port 3000, I was expecting
that I would only need to apply the annotations for prometheus in the Services or Deployments metadata.

  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '3000'

I should now be able to see the graphs using the Grafana dashboard id 11668.
I only get No data in the graphs which is telling me that the scraping doesn't work as I assumed.

I was expecting that the current scrape_configs scraped all pods if prometheus.io/scrape: 'true', am i wrong?
Do I need to add a custom scrape_config for this, how do I add new scrape_configs in a convenient way?

Examples are telling me that I need to add something like this somehow:

scrape_configs:
- job_name: 'rails'
  scheme: http
  metrics_path: '/metrics'
  static_configs:
    - targets:
      - 'rails:3000'

Service and Deployment

---
apiVersion: v1
kind: Service
metadata:
  name: rails
  namespace: dev
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '3000'
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 3000
  selector:
    app: budid
    component: api
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rails
  namespace: dev
  labels:
    app: budid
    component: api
spec:
  selector:
    matchLabels:
      app: budid
      component: api
  template:
    metadata:
      labels:
        app: budid
        component: api
    spec:
      containers:
        - name: rails
      ...

Metrics

# TYPE sidekiq_jobs_enqueued_total counter
# HELP sidekiq_jobs_enqueued_total A counter of the total number of jobs sidekiq enqueued.
# TYPE http_request_total counter
# HELP http_request_total A counter of the total number of external HTTP \\\n                         requests.
# TYPE http_response_total counter
# HELP http_response_total A counter of the total number of external HTTP \\\n                         responses.
# TYPE http_response_duration_milliseconds histogram
# HELP http_response_duration_milliseconds A histogram of the response                                                duration (milliseconds).
# TYPE rails_requests_total counter
# HELP rails_requests_total A counter of the total number of HTTP requests rails processed.
rails_requests_total{controller="api/v1/base",action="status",status="200",format="json",method="get"} 2.0
rails_requests_total{controller="api/v1/brokers",action="theme",status="200",format="json",method="get"} 2.0
rails_requests_total{controller="api/v1/biddings",action="get_by_bidder",status="200",format="json",method="get"} 2.0
rails_requests_total{controller="api/v1/bidders",action="show",status="200",format="json",method="get"} 2.0
rails_requests_total{controller="api/v1/banks",action="index",status="200",format="json",method="get"} 2.0
# TYPE rails_request_duration_seconds histogram
# HELP rails_request_duration_seconds A histogram of the response latency.
rails_request_duration_seconds_bucket{controller="api/v1/base",action="status",status="200",format="json",method="get",le="0.005"} 2.0
rails_request_duration_seconds_bucket{controller="api/v1/base",action="status",status="200",format="json",method="get",le="0.01"} 2.0
...