[SWT-NNNN] Introduce API allowing traits to customize test execution

[stmontgomery]

This includes an API proposal and code changes to introduce new API for custom traits to customize test execution.

View the API proposal for more details.

Motivation:

One of the primary motivations for the trait system in Swift Testing, as described in the vision document, is to provide a way to customize the behavior of tests which have things in common. If all the tests in a given suite type need the same custom behavior, init and/or deinit (if applicable) can be used today. But if only some of the tests in a suite need custom behavior, or tests across different levels of the suite hierarchy need it, traits would be a good place to encapsulate common logic since they can be applied granularly per-test or per-suite. This aspect of the vision for traits hasn't been realized yet, though: the Trait protocol does not offer a way for a trait to customize the execution of the tests or suites it's applied to.

Customizing a test's behavior typically means running code either before or after it runs, or both. Consolidating common set-up and tear-down logic allows each test function to be more succinct with less repetitive boilerplate so it can focus on what makes it unique.

Checklist:

• Code and documentation should follow the style of the Style Guide.
• If public symbols are renamed or modified, DocC references should be updated.

[stmontgomery]

@swift-ci please test Windows

[grynspan]

Gerunds for protocol names are common in Objective-C, but in Swift we usually prefer an adjective form, e.g. (bikeshedding here!) CustomTestExecutable or CustomExecutable or CustomTestSetuppableAndTeardownable.

[stmontgomery]

Definitely open to these alternate suggestions, thanks. But I'll note: as I was considering names myself, I found this in the Swift API Design Guidelines:

• Protocols that describe a capability should be named using the suffixes able, ible, or ing (e.g. Equatable, ProgressReporting).

Based on that, I think an ing suffix would be acceptable.

[grynspan]

Oh it's not wrong, just uncommon. So it's worth thinking about something more idiomatic if we can come up with something.

[stmontgomery]

One thing I slightly dislike about the able suffixes is that it makes it seem as if the work is optional. "This type is able to execute the test… but will it? Who knows!?". But if you define a trait this way, it will execute tests it's applied to, and to me the ing conveys that a little more clearly. "This type is responsible for executing the test."

[grynspan]

We have plenty of counter-examples too where a noun is used instead: Collection, Sequence, Actor… maybe it really just is CustomTestExecutor.

[stmontgomery]

We have plenty of counter-examples too where a noun is used instead: Collection, Sequence, Actor… maybe it really just is CustomTestExecutor.

The API Design Guidelines address that other common naming convention for protocols in the bullet above the one I quoted:

• Protocols that describe what something is should read as nouns (e.g. Collection).

So I think this adheres to the recommendations well: the Trait protocol best describes what the type is, and the CustomTestExecuting protocol describes one optional capability a type (usually, a trait) can have.

[grynspan]

We think task locals are going to be a particularly common use case, right? Perhaps we should build out a trait type that sets a task local:

@Test(.withValue(123, for: $foo)) func f() { ... }

Or even if we don't make it a trait, can we make it easier to write a trait that sets a task local with minimal boilerplate?

[stmontgomery]

Agree that would be a very useful, built-in trait to have. Let's consider that separately, since it would ultimately leverage the API proposed here

[grynspan]

Doesn't need to be part of this proposal, but I think it does affect this proposal. Might be worth coming up with an example of when you'd use this stuff other than to set a task local.

[stmontgomery]

Sure. Briefly: not all work before/after a test is simply mutating state in that process. It may involve writing to the filesystem, or interacting with other system-wide resources. (Clearly, parallelism would need to be considered in such scenarios.)

[stmontgomery]

Also: Users can still use this facility to mutate global or static state, as long as it complies with Swift concurrency rules

[grynspan]

Can we make sure to add this example to the discussion?

[stmontgomery]

@swift-ci please test Windows

[grynspan]

I think you want this to be TestTrait?

[stmontgomery]

The compiler doesn't allow that, because then it's ambiguous for types which conform to both TestTrait and SuiteTrait

[stmontgomery]

The way I have it now, the extension on Trait acts as the "default" implementation, and the SuiteTrait refines it conditionally.

[grynspan]

Perhaps it should be ambiguous for types that conform to both, or we should provide an additional default for them that executes once per suite or test?

[stmontgomery]

I don't think we want to break the current ability to have a trait type conform to both SuiteTrait and TestTrait. Do you think something is problematic about the current approach?

[grynspan]

I think this paragraph is now out of date? At a minimum, it should probably refer to the other function.

[jakepetroules]

@stmontgomery Per our conversation offline you mentioned that order of execution of custom execution traits is well-defined to be outer-to-inner, left-to-right -- it would be great if we could explicitly add a note about that in the documentation so that folks are aware of the expected behavior!