Labels: let's make them easier to work with

[lukaszachy]

The discussion has started on the weekly call and this is the follow-up. We find out there are too many labels, noone from attendies was 100% sure with the meaning of each label so it seems there is a need to rethink how we work with them.
Bellow list of all currently used labels as they are defined plus the their historical usage.

Seems we have several categories (please free to extend the list if something is missing or should be split):

• tmt commands as 'init'
• tmt run commands as 'login'
• plan steps as 'prepare' (which are also run commands)
• chore activities as 'packaging', 'documentation'
• ci control as 'full test'
• status of review as 'ready for merge', 'blocked'
• code related as 'type annotations', 'base'
• priority as 'must'
• type as 'bug', 'enhancement'
• action item as 'discuss', 'help wanted'

Questions are: What do we want to use labels for and which labels will satisfy that?
IMO it shouldn't be more that one page (30) of labels and it should be clear (with just little glance to the note) what the label is used for.

name	description	ever	open/close	decision
provision	Provision step	310	37/273	step/provision
documentation	Improvements or additions to documentation	268	29/239	keep
enhancement	New feature or request	257	49/208	drop
tests		255	11/244	command/tests
bug	Something isn't working	215	13/202	keep
full test	Pull request is ready for the full test execution	213	16/197	ci/full test
execute	Execute step	200	19/181	step/execute
code style		166	22/144	code/style
prepare	Prepare step	148	25/123	step/prepare
ready for merge	The only missing piece is to do the rebase the current 'main' and let the CI finish.	144	5/139	status/ready for merge
no functional change	"No Functional Change" intended. Patch should not change tmt's behavior in any way.	134	13/121	code/no functional change
packaging		134	4/130	keep
discover	Discover step	130	18/112	step/discover
specification		130	19/111	keep
type annotations	Related to type annotations and type cleanup	128	11/117	code/type annotations
import		122	6/116	command/import
hardware		114	28/86	area/hardware
export		108	9/99	command/export
tmt tests	Improvements or additions to test coverage of tmt itself	108	5/103	test coverage
must	high priority	102	4/98	priority/must
cli		98	10/88	code/cli
testcloud		88	10/78	plugins/testcloud
trivial	A simple patch - a couple of lines, an easy-to-understand change, a typo fix.	83	8/75	code/trivial
report		78	8/70	step/report
base		66	8/58	code/base
plans		64	9/55	command/plans
good first issue	Good for newcomers	62	19/43	keep
lint		61	13/48	command/lint
mrack		54	9/45	plugins/mrack
multihost		53	6/47	area/multihost
logging		51	2/49	code/logging
polarion		51	5/46	plugins/polarion
utils		50	1/49	code/utils
results	Related to how tmt stores and shares results	47	9/38	area/results
libraries		41	8/33	area/libraries
run		39	10/29	command/run
discuss	Needs more discussion before closing	38	22/16	status/discuss
plugins		35	6/29	drop
stories	User story	35	0/35	command/stories
blocking other work	An important PR, blocking other PRs or issues	31	0/31	status/blocking other work
finish	Finish step	28	6/22	step/finish
ansible		27	6/21	plugins/ansible
schema		26	4/22	code/schema
context		26	5/21	area/context
environment		26	3/23	area/environment
check		25	3/22	area/check
containers		24	5/19	plugins/containers
steps		24	4/20	drop
reboot	reboot of guests during tmt run	24	3/21	command/reboot
artemis	Related to Artemis provision plugin	23	4/19	plugins/artemis
should	medium priority	21	7/14	priority/should
nitrate		21	4/17	plugins/nitrate
blocked	The merging of PR is blocked on some other issue	21	7/14	status/blocked
core		21	2/19	code/core
try		19	8/11	command/try
login		17	5/12	command/login
usability		16	6/10	keep
breakage	Change will change current behaviour	15	0/15	keep
clean		14	2/12	command/clean
restraint		13	2/11	area/restraint
pre-commit		13	3/10	drop
reportportal		13	1/12	plugins/reportportal
minute		13	0/13	plugins/minute
html		11	1/10	plugins/html
package managers	Changes related to implementations of package managers	9	2/7	area/package managers
init		9	0/9	command/init
process		9	1/8	drop
junit		7	5/2	plugins/junit
status		7	2/5	command/status
release note	Include a short release note, possibly with an example or demo	7	0/7
needs tests	Test coverage to be added for the affected code	7	1/6	status/need tests
filter		5	2/3	drop
could	low priority	4	2/2	priority/could
config		4	0/4	area/config
upgrade		4	1/3	area/upgrade
invalid	This doesn't seem right	3	0/3	drop
duplicate	This issue or pull request already exists	3	0/3	drop
freeze	Do not merge into main yet but it is ready	2	0/2	drop
need docs	Documentation to be added for the affected code	1	0/1	status/need docs
docs test	Run the documentation test for all pull request changes	1	0/1	drop
known issues		1	0/1	known issue
help wanted	Extra attention is needed	1	0/1	status/need help
debug		1	0/1	drop
link		1	1/0	command/link
security		1	0/1	keep
question	Further information is requested	1	0/1	drop
wontfix	This will not be worked on	0	0/0	drop

[lukaszachy]

IMO we have some leftovers from dates before the 'checklist' ("need docs", "needs tests") we can drop right away.

What I am not sure are the 'logging', 'utils' etc. which seem to indicate that some part of tmt library was changed.. But does it need extra label? If we want to group them isn't tracking issue better?
Similar for 'tmt' commands as 'lint' or run phases... Too many of them.

As for usage (all IMO):

Make some PR/issue to get more attention

• "good first issue" marking issues we thing are easy enough for new contributors
• "discuss" for issues / PR that need more discussion (decide from several possibilities, WDY for design, ...).
• "must" for something that has to be fixed (IMO we do not need other "severity" labels as picking work to do is otherwise 'scratch your itch' or 'I want that part to be better')

Type of issue/PR

• "bug" when something doesn't work but it should
• "enhancement" for adding new feature

Scope

• "packaging" if it touches spec, packit, hatch ... without changing the tmt code itself
• "specification" for changes in the tmt metadata specification
• "documentation" when only documentation is changed (PR checklist reminds that docs should be up-to-date with code changes)
• "code style" just style improvements, no functional change intended

Control labels

• "full test" for packit to run the full testsuite
• "ready for merge" in the feature, when we have bot to do the merging

[LecrisUT]

How about merging "type of issue" and "scope". That could be used to make automated release notes easier with .github/release.yaml

[lukaszachy]

What means 'merging' here? As in mutually exclusive labels (which bug x enhancement are)?

[lukaszachy]

Or in the sence of "pretending namespace"?

[LecrisUT]

As in pretend namespacing and conceptually. These should not have mutual exclusivity

[happz]

There are a few "helper" flag-like labels I'd like to keep in play: trivial and no functional change. These would be orthogonal to the rest, and I believe these are improving the review & merge process by reducing the set of steps needed for the wrap-up. Plus tmt tests in the "scope" category - from time to time, we improve tests without touching docs or functionality.

[LecrisUT]

Even though Github does not support namespacing, there is no reason why we can't pretend it does

[martinhoyer]

I see, so it would be steps/provisioning steps/execute ...

[martinhoyer]

While it wouldn't solve the issue of having too many available labels, it might be worth considering imho.
Description can be more or less copy-pasted and with single color for 'fake' namespace it could be easy to navigate.

[lukaszachy]

Nice, as well as 'impact/high' or 'kind/refactor'

[happz]

Yes, please, namespaces could be very helpful. impact/high or impact/must...

[psss]

I also like the idea of introducing prefixes. I'd suggest to:

• keep them all in singular form (e.g. step/provision, command/run)
• reuse any terminology already used (e.g. use priority/must, priority/should similar to the story priority)

[martinhoyer]

Just want to mention, because it wasn't mentioned before afaik. We should group the (groups of)labels by colours.

[psss]

Here I'm not completely sure. That works nicely for some namespaces (like step/provision, step/prepare, step/execute) but for example for priority I'd rather choose a different color (priority/must versus priority/could).

[lukaszachy]

Do we need (want) to have labels for 'steps', 'plugins' and 'commands'? It will add bunch of them

[LecrisUT]

If it's done automatically by a github action, than there wouldn't be a downside for having them other than clutter. The labeler can be setup quite easily for these by checking which files are being changed.

[psss]

I personally would keep them. It's not always completely clear from the issue or pull request summary which areas are affected. On the other hand, I'd say we don't have to cover all command levels, what abouting having a label for each top level subcommand only?

• command/run
• command/tests
• command/plans
• command/stories
• command/init
• command/try
• command/status
• command/clean
• command/lint
• command/setup

[martinhoyer]

I'd also like to bring attention to changelog generation. Regardless of how it will be generated, I'd imagine it would be good to have a way to figute out:

• What to put in "Highlights"
  • either list of labels to include/exclude ([impact/high, impact/must]?) or an explicit label for this (hightlight?)
• What PRs to omit from changelog
  • trivial label can tell that the change is minimal, easy to review, but from my PoV it doesn't necessarily say that it should be dropped from changelog
  • intoducing new label probably not worth it(?), but I'd suggest properly describeing what trivial means - for example, can I have trivial which is also impact/high?

[martinhoyer]

What to put in "Highlights"

These I think should be written manually, like a Main Changes section at the top, after which the generated changelog is added below it.

Hmm, only if it's the same as is being written to rtd Release section. -1 to manual work during release process.

[LecrisUT]

Hmm, only if it's the same as is being written to rtd Release section

Yes. Either extract from the sphinx source, or have sphinx import from the generated changelog. I am inclined to go with the former until some of the issues in the tooling is resolved, or we have references for other projects using other tooling.

[martinhoyer]

Not super sure about this to be honest. Maybe I'm wrong, but What's in the rtd Releases section is verbose description of some of the changes. I don't think it should be 1:1 with gh release note highlings. For example. Highlight could be "python 3.13 compatibilty PR merged" while it doesn't need an entry in rtd. Let alone dealing with processing.
Also, #3125 for discussing towncrier-like things :)

[psss]

I don't see much benefit in auto-generated release notes. I'd say that hand-crafted content in the release section of the docs brings the most value to the user. However, I'd like the impact/high or highlight label which would allow to easily identify the changes which are the most significant for users.

[LecrisUT]

In the workflow of towncrier, you would add the snippets to accumulate in a sub-folder like .changelog.d/(fix|feat|bug) and these are concatenated according to the file ordering. Highlights could work the same but without the bullet points, which I think already works for rst backend.

This would be used to automatically add labels depending on the changes of those folders, and it would help prevent merge conflicts. On the other hand it would add a chore for the release to order/reword the highlights section to be more concise. There is an approach I have experimented with creating a perpetual PR titled Release which gets rebased after each push to main, and it runs towncrier build as a commit to give a preview of the generated changelog.

[martinhoyer]

Seems we have several categories (please free to extend the list if something is missing or should be split):

Perhaps a category for "optional extras"? i.e. plugins depending on external dependencies.
extras/test-convert, extras/polarion, extras/testcloud, extras/mrack, ..

tmt commands as 'init'
tmt run commands as 'login'

Perhaps core/run, core/cli.. or should for example run be a top-level namespace?
..'core' might be the best word, admitedly.

plan steps as 'prepare' (which are also run commands)

steps namespace for all steps: steps/discover, steps/execute, .... ?

chore activities as 'packaging', 'documentation'  

chore/packaging, chore/documentation ? I'd like a label for janitorial chores, linting, formatting, version bumps, etc.
Then there is type annotation, code style. Let's think about what to keep/consolidate under what namespace.

ci control as 'full test'
status of review as 'ready for merge', 'blocked'

Does blocked mean blocked by other tmt issue/PR, or also blocked by external factors?

code related as 'type annotations', 'base'
priority as 'must'

impact/ or priority/ namespace?
-1 to 'severity', as that's more appropriate only for bugs/issues.

type as 'bug', 'enhancement'

I'd keep bug or issue, but enhancement feels redundant.
Perhaps we could also do bug/regression

action item as 'discuss', 'help wanted'

Would be good to properly describe what each of these means. help wanted to me says that "I'm stuck, please someone take over or help me to figure it out", while discuss being "There are multiple ways to proceed, we need consensus on what is the "best" way".
I'm thinking, need help could/should be used more often, allowing experienced maintainers to easily filter PRs requiring attention.

Should these be under blocked namespace? blocked/need-help, blocked/discuss, or some other one?

[LecrisUT]

Namespace wise, I think there are some "standards":

• impact/(high|medium|low)
• priority/(high|medium|low): maybe this one could be calculated based on the other ones, not sure if there is a tool for that though
• kind/(bug|feat|doc|...)
• complexity/...
• area/(steps|plugin|...)

Others I think are more up for debate, maybe:

• do/(discuss|blocked|merge|full-test)
• status/(blocking|need-help|discuss)

chore is the one I am not sure about, it could be just a kind/chore or it could be its own namespace, or both. One thing -1 for me is chore/documentation, instead it should be kind/doc + trivial if we don't want it generated.

Should also keep some of the defaults that others would expect/search: good first issue, help wanted

[psss]

I'd keep bug or issue, but enhancement feels redundant.

I never understood much the point for bug and enhancement. I could live without both of them (and just use priority/must for important bugs). Basically everything is an enhancement (includig a bug fix) and for me bug can cover new features as well (it's a bug that the feature is not already there) 😁

I'd also vote against chore.

[psss]

Cleanup step one

Seems we already have quite a consensus about some core labels. I propose to rename the following as the first step:

priority/must
priority/should
priority/could

step/discover
step/provision
step/prepare
step/execute
step/report
step/finish

status/blocked
status/blocking other work
status/discuss
status/full test
status/need docs
status/need tests
status/ready for merge

code/style
code/trivial
code/logging
code/pre-commit

command/run
command/tests
command/plans
command/stories
command/init
command/try
command/status
command/clean
command/lint
command/setup

These doesn't seem to be used or desired, let's just drop them?

debug
invalid
freeze
docs test
question
wontfix

[martinhoyer]

• Not sure about pre-commit for "checking tmt code" and pre-commit as "tmt pre-commit hook".
• Does full test need to be within a namespace?
• How about docs namespace?
• Do we need logging?
• A label declaring that the issue "needs bugfix release"? Something like priority/critical?

[happz]

code/logging also does not look that useful. And I miss tmt tests, into which the "checking tmt code with pre-commit" case would fit easily.

[psss]

Cleanup step two

The first round of labels renamed. I tried to categorize the rest. Please, have a look and comment the proposed changes.

drop

Let's drop these?

• core
• duplicate
• enhancement
• filter
• plugins
• process
• release note
• steps

code

Would those fit into the code namespace?

• base
• cli
• logging
• schema
• utils

plugin

Do we want to track individual plugins as well? Or drop all these?

• ansible
• artemis
• containers
• html
• junit
• minute
• mrack
• nitrate
• polarion
• reportportal
• testcloud

command

Not sure, they do not fit anywhere else. Perhaps command?

• reboot
• login
• export
• import

area

What about an area namespace for these?

• hardware
• multihost
• results
• libraries
• package managers
• restraint
• context
• environment
• check
• upgrade
• config

keep

Keep without change?

• breakage
• documentation
• good first issue
• packaging
• security
• specification
• usability

Rename

• tmt tests --> test coverage

[psss]

Updated with the feedback from the hacking session.

[psss]

Cleanup step three

Agreed changes applied. The last remaining label seems to be release note. Thinking about it a bit: Sounds related to documentation. What having about a docs namespace for these?

• docs | specification
• docs | release note

And perhaps adding one for the guide?

• docs | guide

Plus convert documentation to docs | general or something similar?

[LecrisUT]

Could be useful. Ideally would you want to have a labeler to label these based on the files that are changed, or add this label manually and have a check that fails the CI if those files are not changed, or both? The former is fairly straightforward to do, the latter I haven't investigated yet.

[psss]

Any automation is welcome. I think we're just fine with manually applying the labels but if there's an easy way to add labels auto-magically for known files, why not :) The check shouting in red when files are not changed is not necessary, imho.

[martinhoyer]

+1, I guess we can implement any future changes to the labels as part of implementing automation in this context. Like #3125 & Autolabeler

[psss]

Just found a clear use case for the release note label: Quite often it happens that there is a pull request with an interesting feature which would deserve a release note. What about renaming the label to status | need release note? It would nicely work with status | need tests, status | need help and status | need docs.

[martinhoyer]

A few notes about the current labels

area | libraries Issues related to beakerlib libraries support

I understand that currently only beakerlib libraries are supported, but this description makes it explicitly just beakerlib (nitpick :) )

area | restraint Backward compatibility support for restraint

Do we want this? i.e. what is the future of tmt's interoperability with beaker, restraint? Wouldn't a more generalized label cover similar issues/prs?

breakage Change will change current behaviour

I usually see this described as breaking change. Just saying.

code | logging Changes related to debug logging

Is the description accurate? i.e. only for debug logging?

code | plugins Changes to the plugin implementation

Do I understand it correctly, that since we have plugin namespace, this label would be used for code that is being used by a plugin, but not necessarily changes to a plugin?

known issue

Not sure about this one, is it intended to be used for automatically include the labelled issue on an external page / list of known issues? If yes, couldn't it be filtered by bug and severity labels?

That said, I think we do miss a label that would mark the issue needs to be fixed in bugfix release/asap. I'm thinking severity/critical, severity/bugfix, or something like this. Basically one above "must - needs to be included in next release".

---

Other than that, I think some labels could do with a more verbose description so people with 0 experience with tmt would understand what it means. I'm not very good at this, but first example: area | config is described as "User configuration" - how about "tmt user configuration, stored by default in ~/.config/tmt/ or defined by TMT_CONFIG_DIR which includes things like custom templates" Or even link to a documentation.
Anyway, just brainstorming. I remember @seberm mentioned label descriptions, wdyt?

Lastly, do we want namespace | foo or namespace/foo? No strong opinion, but would go with / personaly.

[LecrisUT]

Note that only Collaborators and above may set labels

[martinhoyer]

Good point. I assume it's not possible to have a group of labels to be available for anyone?
Regardless, Collaborators are also starting with little knowledge about the labels usage :)

[psss]

area | libraries Issues related to beakerlib libraries support

I understand that currently only beakerlib libraries are supported, but this description makes it explicitly just beakerlib (nitpick :) )

Yes, and yes. Only beakerlib libraries are there. I don't see the point.

area | restraint Backward compatibility support for restraint

Do we want this? i.e. what is the future of tmt's interoperability with beaker, restraint? Wouldn't a more generalized label cover similar issues/prs?

I don't know about any plans to further extend the integration with Beaker.

breakage Change will change current behaviour

I usually see this described as breaking change. Just saying.

The breaking change alternative sounds good to me as well.

code | logging Changes related to debug logging

Is the description accurate? i.e. only for debug logging?

That was my impression from the existing issues. However, we can extend to more. Any suggestion for a better wording? There was also suggestion to completely drop it. But I was not sure as there are already ~50 issues labeled.

known issue

Not sure about this one, is it intended to be used for automatically include the labelled issue on an external page / list of known issues? If yes, couldn't it be filtered by bug and severity labels?

This one was added to highlight the important issues affecting users before the tmt is deployed into production:

• Document the known issue label on the contribute page #3220

Link added to the Introduction section of Contribute page.

That said, I think we do miss a label that would mark the issue needs to be fixed in bugfix release/asap. I'm thinking severity/critical, severity/bugfix, or something like this. Basically one above "must - needs to be included in next release".

Why not using priority | must for these?

Lastly, do we want namespace | foo or namespace/foo? No strong opinion, but would go with / personaly.

This was discussed on the chat. I tried area/topic-with-dashes first but it was a bit hardly readable and there was a concensus that area | topic with spaces looks better.

[martinhoyer]

Why not using priority | must for these?

must's description is "must be included in the next release". So it's "needs to be fixed in the next release" vs "it needs to be fixed asap in a bugfix release"

[psss]

So it's "needs to be fixed in the next release" vs "it needs to be fixed asap in a bugfix release"

I see, thanks for clarification. Don't know if it's worth it. Maybe in the future when we have tens of critical bugs for a hotfix release? 🤓 But hopefully we don't get there 😁