diff --git a/Documentation/Proposals/NNNN-custom-test-execution-traits.md b/Documentation/Proposals/NNNN-custom-test-execution-traits.md new file mode 100644 index 00000000..a936e5f0 --- /dev/null +++ b/Documentation/Proposals/NNNN-custom-test-execution-traits.md @@ -0,0 +1,484 @@ +# Custom Test Execution Traits + +* Proposal: [SWT-NNNN](NNNN-filename.md) +* Authors: [Stuart Montgomery](https://github.com/stmontgomery) +* Status: **Awaiting review** +* Implementation: [swiftlang/swift-testing#733](https://github.com/swiftlang/swift-testing/pull/733), [swiftlang/swift-testing#86](https://github.com/swiftlang/swift-testing/pull/86) +* Review: ([pitch](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055)) + +### Revision history + +* **v1**: Initial pitch. +* **v2**: Dropped 'Custom' prefix from the proposed API names (although kept the + word in certain documentation passages where it clarified behavior). +* **v3**: Changed the `Trait` requirement from a property to a method which + accepts the test and/or test case, and modify its default implementations such + that custom behavior is either performed per-suite or per-test case by default. + +## Introduction + +This introduces API which enables a custom `Trait`-conforming type to customize +the execution of test functions and suites, including running code before or +after them. + +## Motivation + +One of the primary motivations for the trait system in Swift Testing, as +[described in the vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#trait-extensibility), +is to provide a way to customize the behavior of tests which have things in +common. If all the tests in a given suite type need the same custom behavior, +`init` and/or `deinit` (if applicable) can be used today. But if only _some_ of +the tests in a suite need custom behavior, or tests across different levels of +the suite hierarchy need it, traits would be a good place to encapsulate common +logic since they can be applied granularly per-test or per-suite. This aspect of +the vision for traits hasn't been realized yet, though: the `Trait` protocol +does not offer a way for a trait to customize the execution of the tests or +suites it's applied to. + +Customizing a test's behavior typically means running code either before or +after it runs, or both. Consolidating common set-up and tear-down logic allows +each test function to be more succinct with less repetitive boilerplate so it +can focus on what makes it unique. + +## Proposed solution + +At a high level, this proposal entails adding API to the `Trait` protocol +allowing a conforming type to opt-in to customizing the execution of test +behavior. We discuss how that capability should be exposed to trait types below. + +### Supporting scoped access + +There are different approaches one could take to expose hooks for a trait to +customize test behavior. To illustrate one of them, consider the following +example of a `@Test` function with a custom trait whose purpose is to set mock +API credentials for the duration of each test it's applied to: + +```swift +@Test(.mockAPICredentials) +func example() { + // ... +} + +struct MockAPICredentialsTrait: TestTrait { ... } + +extension Trait where Self == MockAPICredentialsTrait { + static var mockAPICredentials: Self { ... } +} +``` + +In this hypothetical example, the current API credentials are stored via a +static property on an `APICredentials` type which is part of the module being +tested: + +```swift +struct APICredentials { + var apiKey: String + + static var shared: Self? +} +``` + +One way that this custom trait could customize the API credentials during each +test is if the `Trait` protocol were to expose a pair of method requirements +which were then called before and after the test, respectively: + +```swift +public protocol Trait: Sendable { + // ... + func setUp() async throws + func tearDown() async throws +} + +extension Trait { + // ... + public func setUp() async throws { /* No-op */ } + public func tearDown() async throws { /* No-op */ } +} +``` + +The custom trait type could adopt these using code such as the following: + +```swift +extension MockAPICredentialsTrait { + func setUp() { + APICredentials.shared = .init(apiKey: "...") + } + + func tearDown() { + APICredentials.shared = nil + } +} +``` + +Many testing systems use this pattern, including XCTest. However, this approach +encourages the use of global mutable state such as the `APICredentials.shared` +variable, and this limits the testing library's ability to parallelize test +execution, which is +[another part of the Swift Testing vision](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#parallelization-and-concurrency). + +The use of nonisolated static variables is generally discouraged now, and in +Swift 6 the above `APICredentials.shared` property produces an error. One way +to resolve that is to change it to a `@TaskLocal` variable, as this would be +concurrency-safe and still allow tests accessing this state to run in parallel: + +```swift +extension APICredentials { + @TaskLocal static var current: Self? +} +``` + +Binding task local values requires using the scoped access +[`TaskLocal.withValue()`](https://developer.apple.com/documentation/swift/tasklocal/withvalue(_:operation:isolation:file:line:)) +API though, and that would not be possible if `Trait` exposed separate methods +like `setUp()` and `tearDown()`. + +For these reasons, I believe it's important to expose this trait capability +using a single, scoped access-style API which accepts a closure. A simplified +version of that idea might look like this: + +```swift +public protocol Trait: Sendable { + // ... + + // Simplified example, not the actual proposal + func executeTest(_ body: @Sendable () async throws -> Void) async throws +} + +extension MockAPICredentialsTrait { + func executeTest(_ body: @Sendable () async throws -> Void) async throws { + let mockCredentials = APICredentials(apiKey: "...") + try await APICredentials.$current.withValue(mockCredentials) { + try await body() + } + } +} +``` + +### Avoiding unnecessarily lengthy backtraces + +A scoped access-style API has some potential downsides. To apply this approach +to a test function, the scoped call of a trait must wrap the invocation of that +test function, and every _other_ trait applied to that same test which offers +custom behavior _also_ must wrap the other traits' calls in a nesting fashion. +To visualize this, imagine a test function with multiple traits: + +```swift +@Test(.traitA, .traitB, .traitC) +func exampleTest() { + // ... +} +``` + +If all three of those traits customize test execution behavior, then each of +them needs to wrap the call to the next one, and the last trait needs to wrap +the invocation of the test, illustrated by the following: + +``` +TraitA.executeTest { + TraitB.executeTest { + TraitC.executeTest { + exampleTest() + } + } +} +``` + +Tests may have an arbitrary number of traits applied to them, including those +inherited from containing suite types. A naïve implementation in which _every_ +trait is given the opportunity to customize test behavior by calling its scoped +access API might cause unnecessarily lengthy backtraces that make debugging the +body of tests more difficult. Or worse: if the number of traits is great enough, +it could cause a stack overflow. + +In practice, most traits probably will _not_ need to customize test behavior, so +to mitigate these downsides it's important that there be some way to distinguish +traits which customize test behavior. That way, the testing library can limit +these scoped access calls to only the traits which require it. + +### Avoiding unnecessary (re-)execution + +Traits can be applied to either test functions or suites, and traits applied to +suites can optionally support inheritance by implementing the `isRecursive` +property of the `SuiteTrait` protocol. When a trait is directly applied to a +test function, if the trait customizes the behavior of tests it's applied to, it +should be given the opportunity to perform its custom behavior once for every +invocation of that test function. In particular, if the test function is +parameterized and runs multiple times, then the trait applied to it should +perform its custom behavior once for every invocation. This should not be +surprising to users, since it's consistent with the behavior of `init` and +`deinit` for an instance `@Test` method. + +It may be useful for certain kinds of traits to perform custom logic once for +_all_ the invocations of a parameterized test. Although this should be possible, +we believe it shouldn't be the default since it could lead to work being +repeated multiple times needlessly, or unintentional state sharing across tests, +unless the trait is implemented carefully to avoid those problems. + +When a trait conforms to `SuiteTrait` and is applied to a suite, the question of +when its custom behavior (if any) should be performed is less obvious. Some +suite traits support inheritance and are recursively applied to all the test +functions they contain (including transitively, via sub-suites). Other suite +traits don't support inheritance, and only affect the specific suite they're +applied to. (It's also worth noting that a sub-suite _can_ have the same +non-recursive suite trait one of its ancestors has, as long as it's applied +explicitly.) + +As a general rule of thumb, we believe most traits will either want to perform +custom logic once for _all_ children or once for _each_ child, not both. +Therefore, when it comes to suite traits, the default behavior should depend on +whether it supports inheritance: a recursive suite trait should by default +perform custom logic before each test, and a non-recursive one per-suite. But +the APIs should be flexible enough to support both, for advanced traits which +need it. + +## Detailed design + +I propose the following new APIs: + +- A new protocol `TestExecuting` with a single required `execute(...)` method. + This will be called to run a test, and allows the conforming type to perform + custom logic before or after. +- A new property `testExecutor` on the `Trait` protocol whose type is an + `Optional` value of a type conforming to `TestExecuting`. A `nil` value for + this property will skip calling the `execute(...)` method. +- A default implementation of `Trait.testExecutor` whose value is `nil`. +- A conditional implementation of `Trait.testExecutor` whose value is `self` + in the common case where the trait type conforms to `TestExecuting` itself. + +Since the `testExecutor` property is optional and `nil` by default, the testing +library cannot invoke the `execute(...)` method unless a trait customizes test +behavior. This avoids the "unnecessarily lengthy backtraces" problem above. + +Below are the proposed interfaces: + +```swift +/// A protocol that allows customizing the execution of a test function (and +/// each of its cases) or a test suite by performing custom code before or after +/// it runs. +/// +/// Types conforming to this protocol may be used in conjunction with a +/// ``Trait``-conforming type by implementing the +/// ``Trait/executor(for:testCase:)-26qgm`` method, allowing custom traits to +/// customize the execution of tests. Consolidating common set-up and tear-down +/// logic for tests which have similar needs allows each test function to be +/// more succinct with less repetitive boilerplate so it can focus on what makes +/// it unique. +public protocol TestExecuting: Sendable { + /// Execute a function for the specified test and/or test case. + /// + /// - Parameters: + /// - function: The function to perform. If `test` represents a test suite, + /// this function encapsulates running all the tests in that suite. If + /// `test` represents a test function, this function is the body of that + /// test function (including all cases if it is parameterized.) + /// - test: The test under which `function` is being performed. + /// - testCase: The test case, if any, under which `function` is being + /// performed. When invoked on a suite, the value of this argument is + /// `nil`. + /// + /// - Throws: Whatever is thrown by `function`, or an error preventing + /// execution from running correctly. An error thrown from this method is + /// recorded as an issue associated with `test`. If an error is thrown + /// before `function` is called, the corresponding test will not run. + /// + /// When the testing library is preparing to run a test, it finds all traits + /// applied to that test (including those inherited from containing suites) + /// and asks each for its test executor (if any) by calling + /// ``Trait/executor(for:testCase:)-26qgm``. It then calls this method + /// on all non-`nil` instances, giving each an opportunity to perform + /// arbitrary work before or after invoking `function`. + /// + /// This method should either invoke `function` once before returning or throw + /// an error if it is unable to perform its custom logic successfully. + /// + /// This method is invoked once for the test its associated trait is applied + /// to, and then once for each test case in that test, if applicable. If a + /// test is skipped, this method is not invoked for that test or its cases. + /// + /// Issues recorded by this method are associated with `test`. + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws +} + +public protocol Trait: Sendable { + // ... + + /// The type of the test executor for this trait. + /// + /// The default type is `Never`, which cannot be instantiated. The + /// ``executor(for:testCase:)-26qgm`` method for any trait with this default + /// test executor type must return `nil`, meaning that trait will not perform + /// any custom behavior for the tests it's applied to. + associatedtype TestExecutor: TestExecuting = Never + + /// Get this trait's executor for the specified test and/or test case, if any. + /// + /// - Parameters: + /// - test: The test for which an executor is being requested. + /// - testCase: The test case for which an executor is being requested, if + /// any. When `test` represents a suite, the value of this argument is + /// `nil`. + /// + /// - Returns: An value of ``Trait/TestExecutor`` which should be used to + /// customize the behavior of `test` and/or `testCase`, or `nil` if custom + /// behavior should not be performed. + /// + /// If this trait's type conforms to ``TestExecuting``, the default value + /// returned by this method depends on `test` and/or `testCase`: + /// + /// - If `test` represents a suite, this trait must conform to ``SuiteTrait``. + /// If the value of this suite trait's ``SuiteTrait/isRecursive`` property + /// is `true`, then this method returns `nil`; otherwise, it returns `self`. + /// This means that by default, a suite trait will _either_ perform its + /// custom behavior once for the entire suite, or once per-test function it + /// contains. + /// - Otherwise `test` represents a test function. If `testCase` is `nil`, + /// this method returns `nil`; otherwise, it returns `self`. This means that + /// by default, a trait which is applied to or inherited by a test function + /// will perform its custom behavior once for each of that function's cases. + /// + /// A trait may explicitly implement this method to further customize the + /// default behaviors above. For example, if a trait should perform custom + /// test behavior both once per-suite and once per-test function in that suite, + /// it may implement the method and return a non-`nil` executor under those + /// conditions. + /// + /// A trait may also implement this method and return `nil` if it determines + /// that it does not need to perform any custom behavior for a particular test + /// at runtime, even if the test has the trait applied. This can improve + /// performance and make diagnostics clearer by avoiding an unnecessary call + /// to ``TestExecuting/execute(_:for:testCase:)``. + /// + /// If this trait's type does not conform to ``TestExecuting`` and its + /// associated ``Trait/TestExecutor`` type is the default `Never`, then this + /// method returns `nil` by default. This means that instances of this type + /// will not perform any custom test execution for tests they are applied to. + func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? +} + +extension Trait where Self: TestExecuting { + // Returns `nil` if `testCase` is `nil`, else `self`. + public func executor(for test: Test, testCase: Test.Case?) -> Self? +} + +extension SuiteTrait where Self: TestExecuting { + // If `test` is a suite, returns `nil` if `isRecursive` is `true`, else `self`. + // Otherwise, `test` is a function and this returns `nil` if `testCase` is + // `nil`, else `self`. + public func executor(for test: Test, testCase: Test.Case?) -> Self? +} + +extension Trait where TestExecutor == Never { + // Returns `nil`. + public func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? +} + +extension Never: TestExecuting {} +``` + +Here is a complete example of the usage scenario described earlier, showcasing +the proposed APIs: + +```swift +@Test(.mockAPICredentials) +func example() { + // ...validate API usage, referencing `APICredentials.current`... +} + +struct MockAPICredentialsTrait: TestTrait, TestExecuting { + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + let mockCredentials = APICredentials(apiKey: "...") + try await APICredentials.$current.withValue(mockCredentials) { + try await function() + } + } +} + +extension Trait where Self == MockAPICredentialsTrait { + static var mockAPICredentials: Self { + Self() + } +} +``` + +## Source compatibility + +The proposed APIs are purely additive. + +This proposal will replace the existing `CustomExecutionTrait` SPI, and after +further refactoring we anticipate it will obsolete the need for the +`SPIAwareTrait` SPI as well. + +## Integration with supporting tools + +Although some built-in traits are relevant to supporting tools (such as +SourceKit-LSP statically discovering `.tags` traits), custom test behaviors are +only relevant within the test executable process while tests are running. We +don't anticipate any particular need for this feature to integrate with +supporting tools. + +## Future directions + +Some test authors have expressed interest in allowing custom traits to access +the instance of a suite type for `@Test` instance methods, so the trait could +inspect or mutate the instance. Currently, only instance-level members of a +suite type (including `init`, `deinit`, and the test function itself) can access +`self`, so this would grant traits applied to an instance test method access to +the instance as well. This is certainly interesting, but poses several technical +challenges that puts it out of scope of this proposal. + +## Alternatives considered + +### Separate set up & tear down methods on `Trait` + +This idea was discussed in [Supporting scoped access](#supporting-scoped-access) +above, and as mentioned there, the primary problem with this approach is that it +cannot be used with scoped access-style APIs, including (importantly) +`TaskLocal.withValue()`. For that reason, it prevents using that common Swift +concurrency technique and reduces the potential for test parallelization. + +### Add `execute(...)` directly to the `Trait` protocol + +The proposed `execute(...)` method could be added as a requirement of the +`Trait` protocol instead of being part of a separate `TestExecuting` protocol, +and it could have a default implementation which directly invokes the passed-in +closure. But this approach would suffer from the lengthy backtrace problem +described above. + +### Extend the `Trait` protocol + +The original, experimental implementation of this feature included a protocol +named`CustomExecutionTrait` which extended `Trait` and had roughly the same +method requirement as the `TestExecuting` protocol proposed above. This design +worked, provided scoped access, and avoided the lengthy backtrace problem. + +After evaluating the design and usage of this SPI though, it seemed unfortunate +to structure it as a sub-protocol of `Trait` because it means that the full +capabilities of the trait system are spread across multiple protocols. In the +proposed design, the ability to provide a test executor value is exposed via the +main `Trait` protocol, and it relies on an associated type to conditionally +opt-in to custom test behavior. In other words, the proposed design expresses +custom test behavior as just a _capability_ that a trait may have, rather than a +distinct sub-type of trait. + +Also, the implementation of this approach within the testing library was not +ideal as it required a conditional `trait as? CustomExecutionTrait` downcast at +runtime, in contrast to the simpler and more performant Optional property of the +proposed API. + +### API names + +We considered "run" as the base verb for the proposed new concept instead of +"execute", which would imply the names `TestRunning`, `TestRunner`, +`runner(for:testCase)`, and `run(_:for:testCase:)`. The word "run" is used in +many other contexts related to testing though, such as the `Runner` SPI type and +more casually to refer to a run which occurred of a test, in the past tense, so +overloading this term again may cause confusion. + +## Acknowledgments + +Thanks to [Dennis Weissmann](https://github.com/dennisweissmann) for originally +implementing this as SPI, and for helping promote its usefulness. + +Thanks to [Jonathan Grynspan](https://github.com/grynspan) for exploring ideas +to refine the API, and considering alternatives to avoid unnecessarily long +backtraces. diff --git a/Sources/Testing/Running/Runner.swift b/Sources/Testing/Running/Runner.swift index 95448533..108f60e8 100644 --- a/Sources/Testing/Running/Runner.swift +++ b/Sources/Testing/Running/Runner.swift @@ -56,19 +56,19 @@ public struct Runner: Sendable { // MARK: - Running tests extension Runner { - /// Execute the ``CustomExecutionTrait/execute(_:for:testCase:)`` functions - /// associated with the test in a plan step. + /// Execute the ``TestExecuting/execute(_:for:testCase:)`` functions of any + /// test executors for traits associated with the test in a plan step. /// /// - Parameters: /// - step: The step being performed. /// - testCase: The test case, if applicable, for which to execute the - /// custom trait. + /// function. /// - body: A function to execute from within the - /// ``CustomExecutionTrait/execute(_:for:testCase:)`` functions of each - /// trait applied to `step.test`. + /// ``TestExecuting/execute(_:for:testCase:)`` function of each non-`nil` + /// test executor of the traits applied to `step.test`. /// /// - Throws: Whatever is thrown by `body` or by any of the - /// ``CustomExecutionTrait/execute(_:for:testCase:)`` functions. + /// ``TestExecuting/execute(_:for:testCase:)`` functions. private func _executeTraits( for step: Plan.Step, testCase: Test.Case?, @@ -90,11 +90,11 @@ extension Runner { // and ultimately the first trait is the first one to be invoked. let executeAllTraits = step.test.traits.lazy .reversed() - .compactMap { $0 as? any CustomExecutionTrait } - .compactMap { $0.execute(_:for:testCase:) } - .reduce(body) { executeAllTraits, traitExecutor in + .compactMap { $0.executor(for: step.test, testCase: testCase) } + .map { $0.execute(_:for:testCase:) } + .reduce(body) { executeAllTraits, testExecutor in { - try await traitExecutor(executeAllTraits, step.test, testCase) + try await testExecutor(executeAllTraits, step.test, testCase) } } diff --git a/Sources/Testing/Testing.docc/Traits/Trait.md b/Sources/Testing/Testing.docc/Traits/Trait.md index 1528ec1b..8a24bd3f 100644 --- a/Sources/Testing/Testing.docc/Traits/Trait.md +++ b/Sources/Testing/Testing.docc/Traits/Trait.md @@ -39,8 +39,15 @@ See https://swift.org/CONTRIBUTORS.txt for Swift project authors - ``Trait/bug(_:id:_:)-3vtpl`` ### Adding information to tests + - ``Trait/comments`` ### Preparing internal state - ``Trait/prepare(for:)-3s3zo`` + +### Customizing the execution of tests + +- ``TestExecuting`` +- ``Trait/executor(for:testCase:)-26qgm`` +- ``Trait/TestExecutor`` diff --git a/Sources/Testing/Traits/Trait.swift b/Sources/Testing/Traits/Trait.swift index e6a42b4d..fde71a74 100644 --- a/Sources/Testing/Traits/Trait.swift +++ b/Sources/Testing/Traits/Trait.swift @@ -41,6 +41,125 @@ public protocol Trait: Sendable { /// /// By default, the value of this property is an empty array. var comments: [Comment] { get } + + /// The type of the test executor for this trait. + /// + /// The default type is `Never`, which cannot be instantiated. The + /// ``executor(for:testCase:)-26qgm`` method for any trait with this default + /// test executor type must return `nil`, meaning that trait will not perform + /// any custom behavior for the tests it's applied to. + associatedtype TestExecutor: TestExecuting = Never + + /// Get this trait's executor for the specified test and/or test case, if any. + /// + /// - Parameters: + /// - test: The test for which an executor is being requested. + /// - testCase: The test case for which an executor is being requested, if + /// any. When `test` represents a suite, the value of this argument is + /// `nil`. + /// + /// - Returns: An value of ``Trait/TestExecutor`` which should be used to + /// customize the behavior of `test` and/or `testCase`, or `nil` if custom + /// behavior should not be performed. + /// + /// If this trait's type conforms to ``TestExecuting``, the default value + /// returned by this method depends on `test` and/or `testCase`: + /// + /// - If `test` represents a suite, this trait must conform to ``SuiteTrait``. + /// If the value of this suite trait's ``SuiteTrait/isRecursive`` property + /// is `true`, then this method returns `nil`; otherwise, it returns `self`. + /// This means that by default, a suite trait will _either_ perform its + /// custom behavior once for the entire suite, or once per-test function it + /// contains. + /// - Otherwise `test` represents a test function. If `testCase` is `nil`, + /// this method returns `nil`; otherwise, it returns `self`. This means that + /// by default, a trait which is applied to or inherited by a test function + /// will perform its custom behavior once for each of that function's cases. + /// + /// A trait may explicitly implement this method to further customize the + /// default behaviors above. For example, if a trait should perform custom + /// test behavior both once per-suite and once per-test function in that suite, + /// it may implement the method and return a non-`nil` executor under those + /// conditions. + /// + /// A trait may also implement this method and return `nil` if it determines + /// that it does not need to perform any custom behavior for a particular test + /// at runtime, even if the test has the trait applied. This can improve + /// performance and make diagnostics clearer by avoiding an unnecessary call + /// to ``TestExecuting/execute(_:for:testCase:)``. + /// + /// If this trait's type does not conform to ``TestExecuting`` and its + /// associated ``Trait/TestExecutor`` type is the default `Never`, then this + /// method returns `nil` by default. This means that instances of this type + /// will not perform any custom test execution for tests they are applied to. + func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? +} + +/// A protocol that allows customizing the execution of a test function (and +/// each of its cases) or a test suite by performing custom code before or after +/// it runs. +/// +/// Types conforming to this protocol may be used in conjunction with a +/// ``Trait``-conforming type by implementing the +/// ``Trait/executor(for:testCase:)-26qgm`` method, allowing custom traits to +/// customize the execution of tests. Consolidating common set-up and tear-down +/// logic for tests which have similar needs allows each test function to be +/// more succinct with less repetitive boilerplate so it can focus on what makes +/// it unique. +public protocol TestExecuting: Sendable { + /// Execute a function for the specified test and/or test case. + /// + /// - Parameters: + /// - function: The function to perform. If `test` represents a test suite, + /// this function encapsulates running all the tests in that suite. If + /// `test` represents a test function, this function is the body of that + /// test function (including all cases if it is parameterized.) + /// - test: The test under which `function` is being performed. + /// - testCase: The test case, if any, under which `function` is being + /// performed. When invoked on a suite, the value of this argument is + /// `nil`. + /// + /// - Throws: Whatever is thrown by `function`, or an error preventing + /// execution from running correctly. An error thrown from this method is + /// recorded as an issue associated with `test`. If an error is thrown + /// before `function` is called, the corresponding test will not run. + /// + /// When the testing library is preparing to run a test, it finds all traits + /// applied to that test (including those inherited from containing suites) + /// and asks each for its test executor (if any) by calling + /// ``Trait/executor(for:testCase:)-26qgm``. It then calls this method + /// on all non-`nil` instances, giving each an opportunity to perform + /// arbitrary work before or after invoking `function`. + /// + /// This method should either invoke `function` once before returning or throw + /// an error if it is unable to perform its custom logic successfully. + /// + /// This method is invoked once for the test its associated trait is applied + /// to, and then once for each test case in that test, if applicable. If a + /// test is skipped, this method is not invoked for that test or its cases. + /// + /// Issues recorded by this method are associated with `test`. + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws +} + +extension Trait where Self: TestExecuting { + public func executor(for test: Test, testCase: Test.Case?) -> Self? { + testCase == nil ? nil : self + } +} + +extension SuiteTrait where Self: TestExecuting { + public func executor(for test: Test, testCase: Test.Case?) -> Self? { + if test.isSuite { + isRecursive ? nil : self + } else { + testCase == nil ? nil : self + } + } +} + +extension Never: TestExecuting { + public func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws {} } /// A protocol describing traits that can be added to a test function. @@ -72,43 +191,14 @@ extension Trait { } } +extension Trait where TestExecutor == Never { + public func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? { + nil + } +} + extension SuiteTrait { public var isRecursive: Bool { false } } - -/// A protocol extending ``Trait`` that offers an additional customization point -/// for trait authors to execute code before and after each test function (if -/// added to the traits of a test function), or before and after each test suite -/// (if added to the traits of a test suite). -@_spi(Experimental) -public protocol CustomExecutionTrait: Trait { - - /// Execute a function with the effects of this trait applied. - /// - /// - Parameters: - /// - function: The function to perform. If `test` represents a test suite, - /// this function encapsulates running all the tests in that suite. If - /// `test` represents a test function, this function is the body of that - /// test function (including all cases if it is parameterized.) - /// - test: The test under which `function` is being performed. - /// - testCase: The test case, if any, under which `function` is being - /// performed. This is `nil` when invoked on a suite. - /// - /// - Throws: Whatever is thrown by `function`, or an error preventing the - /// trait from running correctly. - /// - /// This function is called for each ``CustomExecutionTrait`` on a test suite - /// or test function and allows additional work to be performed before and - /// after the test runs. - /// - /// This function is invoked once for the test it is applied to, and then once - /// for each test case in that test, if applicable. - /// - /// Issues recorded by this function are recorded against `test`. - /// - /// - Note: If a test function or test suite is skipped, this function does - /// not get invoked by the runner. - func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws -} diff --git a/Tests/TestingTests/Traits/CustomExecutionTraitTests.swift b/Tests/TestingTests/Traits/CustomExecutionTraitTests.swift deleted file mode 100644 index aedc06de..00000000 --- a/Tests/TestingTests/Traits/CustomExecutionTraitTests.swift +++ /dev/null @@ -1,104 +0,0 @@ -// -// This source file is part of the Swift.org open source project -// -// Copyright (c) 2024 Apple Inc. and the Swift project authors -// Licensed under Apache License v2.0 with Runtime Library Exception -// -// See https://swift.org/LICENSE.txt for license information -// See https://swift.org/CONTRIBUTORS.txt for Swift project authors -// - -@testable @_spi(Experimental) @_spi(ForToolsIntegrationOnly) import Testing - -@Suite("CustomExecutionTrait Tests") -struct CustomExecutionTraitTests { - @Test("Execute code before and after a non-parameterized test.") - func executeCodeBeforeAndAfterNonParameterizedTest() async { - // `expectedCount` is 2 because we run it both for the test and the test case - await confirmation("Code was run before the test", expectedCount: 2) { before in - await confirmation("Code was run after the test", expectedCount: 2) { after in - await Test(CustomTrait(before: before, after: after)) { - // do nothing - }.run() - } - } - } - - @Test("Execute code before and after a parameterized test.") - func executeCodeBeforeAndAfterParameterizedTest() async { - // `expectedCount` is 3 because we run it both for the test and each test case - await confirmation("Code was run before the test", expectedCount: 3) { before in - await confirmation("Code was run after the test", expectedCount: 3) { after in - await Test(CustomTrait(before: before, after: after), arguments: ["Hello", "World"]) { _ in - // do nothing - }.run() - } - } - } - - @Test("Custom execution trait throws an error") - func customExecutionTraitThrowsAnError() async throws { - var configuration = Configuration() - await confirmation("Error thrown", expectedCount: 1) { errorThrownConfirmation in - configuration.eventHandler = { event, _ in - guard case let .issueRecorded(issue) = event.kind, - case let .errorCaught(error) = issue.kind else { - return - } - - #expect(error is CustomThrowingErrorTrait.CustomTraitError) - errorThrownConfirmation() - } - - await Test(CustomThrowingErrorTrait()) { - // Make sure this does not get reached - Issue.record("Expected trait to fail the test. Should not have reached test body.") - }.run(configuration: configuration) - } - } - - @Test("Teardown occurs after child tests run") - func teardownOccursAtEnd() async throws { - await runTest(for: TestsWithCustomTraitWithStrongOrdering.self, configuration: .init()) - } -} - -// MARK: - Fixtures - -private struct CustomTrait: CustomExecutionTrait, TestTrait { - var before: Confirmation - var after: Confirmation - func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { - before() - defer { - after() - } - try await function() - } -} - -private struct CustomThrowingErrorTrait: CustomExecutionTrait, TestTrait { - fileprivate struct CustomTraitError: Error {} - - func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { - throw CustomTraitError() - } -} - -struct DoSomethingBeforeAndAfterTrait: CustomExecutionTrait, SuiteTrait, TestTrait { - static let state = Locked(rawValue: 0) - - func execute(_ function: @Sendable () async throws -> Void, for test: Testing.Test, testCase: Testing.Test.Case?) async throws { - #expect(Self.state.increment() == 1) - - try await function() - #expect(Self.state.increment() == 3) - } -} - -@Suite(.hidden, DoSomethingBeforeAndAfterTrait()) -struct TestsWithCustomTraitWithStrongOrdering { - @Test(.hidden) func f() async { - #expect(DoSomethingBeforeAndAfterTrait.state.increment() == 2) - } -} diff --git a/Tests/TestingTests/Traits/TestExecutingTraitTests.swift b/Tests/TestingTests/Traits/TestExecutingTraitTests.swift new file mode 100644 index 00000000..7a6f72f6 --- /dev/null +++ b/Tests/TestingTests/Traits/TestExecutingTraitTests.swift @@ -0,0 +1,184 @@ +// +// This source file is part of the Swift.org open source project +// +// Copyright (c) 2024 Apple Inc. and the Swift project authors +// Licensed under Apache License v2.0 with Runtime Library Exception +// +// See https://swift.org/LICENSE.txt for license information +// See https://swift.org/CONTRIBUTORS.txt for Swift project authors +// + +@testable @_spi(Experimental) @_spi(ForToolsIntegrationOnly) import Testing + +@Suite("TestExecuting-conforming Trait Tests") +struct TestExecutingTraitTests { + @Test("Execute code before and after a non-parameterized test.") + func executeCodeBeforeAndAfterNonParameterizedTest() async { + await confirmation("Code was run before the test") { before in + await confirmation("Code was run after the test") { after in + await Test(CustomTrait(before: before, after: after)) { + // do nothing + }.run() + } + } + } + + @Test("Execute code before and after a parameterized test.") + func executeCodeBeforeAndAfterParameterizedTest() async { + // `expectedCount` is 2 because we run it for each test case + await confirmation("Code was run before the test", expectedCount: 2) { before in + await confirmation("Code was run after the test", expectedCount: 2) { after in + await Test(CustomTrait(before: before, after: after), arguments: ["Hello", "World"]) { _ in + // do nothing + }.run() + } + } + } + + @Test("Custom execution trait throws an error") + func customExecutionTraitThrowsAnError() async throws { + var configuration = Configuration() + await confirmation("Error thrown", expectedCount: 1) { errorThrownConfirmation in + configuration.eventHandler = { event, _ in + guard case let .issueRecorded(issue) = event.kind, + case let .errorCaught(error) = issue.kind else { + return + } + + #expect(error is CustomThrowingErrorTrait.CustomTraitError) + errorThrownConfirmation() + } + + await Test(CustomThrowingErrorTrait()) { + // Make sure this does not get reached + Issue.record("Expected trait to fail the test. Should not have reached test body.") + }.run(configuration: configuration) + } + } + + @Test("Teardown occurs after child tests run") + func teardownOccursAtEnd() async throws { + await runTest(for: TestsWithCustomTraitWithStrongOrdering.self, configuration: .init()) + } + + struct ExecutionControl { + @Test("Trait applied directly to function is executed once") + func traitAppliedToFunction() async { + let counter = Locked(rawValue: 0) + await DefaultExecutionTrait.$counter.withValue(counter) { + await Test(DefaultExecutionTrait()) {}.run() + } + #expect(counter.rawValue == 1) + } + + @Test("Non-recursive suite trait with default custom test executor implementation") + func nonRecursiveSuiteTrait() async { + let counter = Locked(rawValue: 0) + await DefaultExecutionTrait.$counter.withValue(counter) { + await runTest(for: SuiteWithNonRecursiveDefaultExecutionTrait.self) + } + #expect(counter.rawValue == 1) + } + + @Test("Recursive suite trait with default custom test executor implementation") + func recursiveSuiteTrait() async { + let counter = Locked(rawValue: 0) + await DefaultExecutionTrait.$counter.withValue(counter) { + await runTest(for: SuiteWithRecursiveDefaultExecutionTrait.self) + } + #expect(counter.rawValue == 1) + } + + @Test("Recursive, all-inclusive suite trait") + func recursiveAllInclusiveSuiteTrait() async { + let counter = Locked(rawValue: 0) + await AllInclusiveExecutionTrait.$counter.withValue(counter) { + await runTest(for: SuiteWithAllInclusiveExecutionTrait.self) + } + #expect(counter.rawValue == 3) + } + } +} + +// MARK: - Fixtures + +private struct CustomTrait: TestTrait, TestExecuting { + var before: Confirmation + var after: Confirmation + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + before() + defer { + after() + } + try await function() + } +} + +private struct CustomThrowingErrorTrait: TestTrait, TestExecuting { + fileprivate struct CustomTraitError: Error {} + + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + throw CustomTraitError() + } +} + +struct DoSomethingBeforeAndAfterTrait: SuiteTrait, TestTrait, TestExecuting { + static let state = Locked(rawValue: 0) + + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + #expect(Self.state.increment() == 1) + + try await function() + #expect(Self.state.increment() == 3) + } +} + +@Suite(.hidden, DoSomethingBeforeAndAfterTrait()) +struct TestsWithCustomTraitWithStrongOrdering { + @Test(.hidden) func f() async { + #expect(DoSomethingBeforeAndAfterTrait.state.increment() == 2) + } +} + +private struct DefaultExecutionTrait: SuiteTrait, TestTrait, TestExecuting { + @TaskLocal static var counter: Locked? + var isRecursive: Bool = false + + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + Self.counter!.increment() + try await function() + } +} + +@Suite(.hidden, DefaultExecutionTrait()) +private struct SuiteWithNonRecursiveDefaultExecutionTrait { + @Test func f() {} +} + +@Suite(.hidden, DefaultExecutionTrait(isRecursive: true)) +private struct SuiteWithRecursiveDefaultExecutionTrait { + @Test func f() {} +} + +private struct AllInclusiveExecutionTrait: SuiteTrait, TestTrait, TestExecuting { + @TaskLocal static var counter: Locked? + + var isRecursive: Bool { + true + } + + func executor(for test: Test, testCase: Test.Case?) -> AllInclusiveExecutionTrait? { + // Unconditionally returning self makes this trait "all inclusive". + self + } + + func execute(_ function: () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + Self.counter!.increment() + try await function() + } +} + +@Suite(.hidden, AllInclusiveExecutionTrait()) +private struct SuiteWithAllInclusiveExecutionTrait { + @Test func f() {} +}