chore: decouple feature gates configuration from manager creation and improve

[programmer04]

What this PR does / why we need it:

• pkg/telemetry/reports.go - simplification of the usage - following a convention that when an error is returned caller doesn't need to care about any outstanding actions (in this case shutting down reporting), it has been moved to pkg for further adjustments - prerequisite for Telemetry data allows verifying that KIC instance is run as a library #7048 and decoupling telemetry from manager creation, the same as it's been done for health checks
• the rest is dedicated to not having any logic that alters passed configuration during the creation of the manager (see removal) since the manager will be a reusable block, the reaction should be as lean as possible, and particular features gate has to be configured per instance bases for KO

Which issue this PR fixes:

Part of the effort to achieve

• ☔ [PoC] KIC as a library to support multiple instances in a single process #7038

[codecov]

Codecov Report

Attention: Patch coverage is 74.35897% with 10 lines in your changes missing coverage. Please review.

Please upload report for BASE (feat/kic-as-library@e61ca89). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
internal/manager/flags/feature_gates_shim.go	72.2%	4 Missing and 1 partial ⚠️
internal/manager/run.go	75.0%	2 Missing and 1 partial ⚠️
pkg/telemetry/reports.go	0.0%	2 Missing ⚠️

Additional details and impacted files

@@                  Coverage Diff                  @@
##             feat/kic-as-library   #7085   +/-   ##
=====================================================
  Coverage                       ?   76.8%           
=====================================================
  Files                          ?     214           
  Lines                          ?   24870           
  Branches                       ?       0           
=====================================================
  Hits                           ?   19117           
  Misses                         ?    4767           
  Partials                       ?     986           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[czeslavo]

it has been moved to pkg for further adjustments - prerequisite for #7048 and decoupling telemetry from manager creation, the same as it's been done for health checks

Do we need to extract telemetry outside of the manager? I assumed that every instance would report its data, and we'd only make it distinguishable via an optional additional report key, e.g. embedded-kic-instance=true or something similar when it's run in KO.

[programmer04]

This is a good question - #7085 (review)
because now the report looks kinda like that

id=57a7a76c-25d0-4394-ab9a-954f7190e39a;
signal=kic-ping
db=off
feature-combinedservicesfromdifferenthttproutes=false
feature-fallbackconfiguration=false
feature-fillids=true
feature-gateway-service-discovery=false
feature-gatewayalpha=false
feature-kongcustomentity=true
feature-kongservicefacade=false
feature-konnect-sync=false
feature-rewriteuris=false
feature-sanitizekonnectconfigdumps=true
...
k8s_gatewayclasses_count=2
k8s_gateways_count=4
k8s_grpcroutes_count=4
k8s_httproutes_count=4
k8s_nodes_count=2
k8s_pods_count=4
k8s_referencegrants_count=4
k8s_services_count=5
k8s_tcproutes_count=4
k8s_tlsroutes_count=4
k8s_udproutes_count=4

where it reports from the whole cluster, so reporting the same from many instances of KIC that are a part of one binary leads to duplication and multiple necessary connections.

My idea was that only one place executes queries for reports and submits them because all metrics besides feature gates (described later) are cluster-wide. Such aan pproach reduces the overhead and ensures that id is the same. Of course, the additional fields will be introduced e.g. if it's single or multiple KICs. Probably for KO, a mix of metrics will be collected here, as well as what KGO collects. The one thing that bothers me is feature flags because, with the current configuration, we leave the ability to adjust them on a level of KIC instance. But maybe we can tackle it later

[czeslavo]

This is a good question - #7085 (review) because now the report looks kinda like that

id=57a7a76c-25d0-4394-ab9a-954f7190e39a;
signal=kic-ping
db=off
feature-combinedservicesfromdifferenthttproutes=false
feature-fallbackconfiguration=false
feature-fillids=true
feature-gateway-service-discovery=false
feature-gatewayalpha=false
feature-kongcustomentity=true
feature-kongservicefacade=false
feature-konnect-sync=false
feature-rewriteuris=false
feature-sanitizekonnectconfigdumps=true
...
k8s_gatewayclasses_count=2
k8s_gateways_count=4
k8s_grpcroutes_count=4
k8s_httproutes_count=4
k8s_nodes_count=2
k8s_pods_count=4
k8s_referencegrants_count=4
k8s_services_count=5
k8s_tcproutes_count=4
k8s_tlsroutes_count=4
k8s_udproutes_count=4

where it reports from the whole cluster, so reporting the same from many instances of KIC that are a part of one binary leads to duplication and multiple necessary connections.

My idea was that only one place executes queries for reports and submits them because all metrics besides feature gates (described later) are cluster-wide. Such aan pproach reduces the overhead and ensures that id is the same. Of course, the additional fields will be introduced e.g. if it's single or multiple KICs. Probably for KO, a mix of metrics will be collected here, as well as what KGO collects. The one thing that bothers me is feature flags because, with the current configuration, we leave the ability to adjust them on a level of KIC instance. But maybe we can tackle it later

Ah, yes, I missed the point that telemetry not only reports KIC-specific data, but also aggregates info about cluster-wide stuff like number of routes, etc. It makes sense then to extract it. 👍