SLO1 performance issues

[majamassarini]

We are having SLO1 error budget issues.

These are mostly caused by db performance issues and we are working on them:

• Speed-up queries after introduction of groups #1908.

We are also planning further improvements:

• Periodically delete old data from the db #1923
• Load the package config once per pipeline run #1940
• Check if the repo is private only when the pipeline is started #1942
• Don't update test flags while builds are still in progress #1943

[lachmanfrantisek]

Related to the research currently being worked on.

[lbarcziova]

Recently several improvements regarding this were implemented: storing the package configuration in DB, not running private repo check for result events and optimizing getting the PR info via API; so let's see how the situation will improve. If needed, we can implement also #1923 .

[majamassarini]

After this fix the number of events taking more than 15s before giving a feedback has lowered, still we have some of them in a day.

An issue is already being tracked here

There is another recurring issue which I am not sure we could resolve.

"2024-10-28T10:02:49.215000529+00:00 stderr F [2024-10-28 10:02:49,214: DEBUG/MainProcess] task.steve_jobs.process_message[38fdf0c8-5dcd-49fc-91b5-1a80c32f8dd9] Package config before loading:","2024-10-28T10:02:49.215+0000"
"2024-10-28T10:02:48.314737040+00:00 stderr F [2024-10-28 10:02:48,314: DEBUG/MainProcess] task.steve_jobs.process_message[38fdf0c8-5dcd-49fc-91b5-1a80c32f8dd9] Found a config file '.packit.yaml' on ref '2daaaa1588eaa6b3a7d65c3153982d1cd1358624' of the 'containers/podman' repository.","2024-10-28T10:02:48.314+0000"
"2024-10-28T10:02:47.735683002+00:00 stderr F [2024-10-28 10:02:47,735: DEBUG/MainProcess] task.steve_jobs.process_message[38fdf0c8-5dcd-49fc-91b5-1a80c32f8dd9] Getting packages_config:","2024-10-28T10:02:47.735+0000"
"2024-10-28T10:02:36.028291591+00:00 stderr F [2024-10-28 10:02:36,028: INFO/MainProcess] task.steve_jobs.process_message[38fdf0c8-5dcd-49fc-91b5-1a80c32f8dd9] Pushing the metrics to pushgateway.","2024-10-28T10:02:36.028+0000"
"2024-10-28T10:02:36.028074010+00:00 stderr F [2024-10-28 10:02:36,027: INFO/MainProcess] task.steve_jobs.process_message[38fdf0c8-5dcd-49fc-91b5-1a80c32f8dd9] Target repo: containers/podman.","2024-10-28T10:02:36.028+0000"

After Pushing the metrics before Getting packages_config there are 10s (and I have seen even more -> 16s).

I think this time is needed for cloning the project.
There are 4 seconds left, in this case, for doing all the other stuff.

Probably we should really avoid logging events taking less than 25/30s if half of this time is needed to clone (?) the project locally?

The latests exceptions I examined have taken the listed time for the listed repos...
16s - podman
14s - ramalama
12s - tmt
10s - systemd
7s - bootc