From 75dd9b6159a07fde74b73d10549e44cbb1a60fbd Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Tue, 24 Sep 2024 12:54:37 -0500 Subject: [PATCH 1/9] Introduce custom test execution trait API --- .../NNNN-custom-test-execution-traits.md | 376 ++++++++++++++++++ Sources/Testing/Running/Runner.swift | 21 +- Sources/Testing/Testing.docc/Traits/Trait.md | 7 + Sources/Testing/Traits/Trait.swift | 101 +++-- ...ft => CustomTestExecutingTraitTests.swift} | 12 +- 5 files changed, 466 insertions(+), 51 deletions(-) create mode 100644 Documentation/Proposals/NNNN-custom-test-execution-traits.md rename Tests/TestingTests/Traits/{CustomExecutionTraitTests.swift => CustomTestExecutingTraitTests.swift} (90%) diff --git a/Documentation/Proposals/NNNN-custom-test-execution-traits.md b/Documentation/Proposals/NNNN-custom-test-execution-traits.md new file mode 100644 index 00000000..c5a593cc --- /dev/null +++ b/Documentation/Proposals/NNNN-custom-test-execution-traits.md @@ -0,0 +1,376 @@ +# Custom Test Execution Traits + +* Proposal: [SWT-NNNN](NNNN-filename.md) +* Authors: [Stuart Montgomery](https://github.com/stmontgomery) +* Status: **Awaiting review** +* Implementation: [swiftlang/swift-testing#733](https://github.com/swiftlang/swift-testing/pull/733), [swiftlang/swift-testing#86](https://github.com/swiftlang/swift-testing/pull/86) +* Review: ([pitch](https://forums.swift.org/...)) + +## Introduction + +This introduces API which enables a custom `Trait`-conforming type to customize +the execution of test functions and suites, including running code before or +after them. + +## Motivation + +One of the primary motivations for the trait system in Swift Testing, as +[described in the vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#trait-extensibility), +is to provide a way to customize the behavior of tests which have things in +common. If all the tests in a given suite type need the same custom behavior, +`init` and/or `deinit` (if applicable) can be used today. But if only _some_ of +the tests in a suite need custom behavior, or tests across different levels of +the suite hierarchy need it, traits would be a good place to encapsulate common +logic since they can be applied granularly per-test or per-suite. This aspect of +the vision for traits hasn't been realized yet, though: the `Trait` protocol +does not offer a way for a trait to customize the execution of the tests or +suites it's applied to. + +Customizing a test's behavior typically means running code either before or +after it runs, or both. Consolidating common set-up and tear-down logic allows +each test function to be more succinct with less repetitive boilerplate so it +can focus on what makes it unique. + +## Proposed solution + +At a high level, this proposal entails adding API to the `Trait` protocol +allowing a conforming type to opt-in to customizing the execution of test +behavior. We discuss how that capability should be exposed to trait types below. + +### Supporting scoped access + +There are different approaches one could take to expose hooks for a trait to +customize test behavior. To illustrate one of them, consider the following +example of a `@Test` function with a custom trait whose purpose is to set mock +API credentials for the duration of each test it's applied to: + +```swift +@Test(.mockAPICredentials) +func example() { + // ... +} + +struct MockAPICredentialsTrait: TestTrait { ... } + +extension Trait where Self == MockAPICredentialsTrait { + static var mockAPICredentials: Self { ... } +} +``` + +In this hypothetical example, the current API credentials are stored via a +static property on an `APICredentials` type which is part of the module being +tested: + +```swift +struct APICredentials { + var apiKey: String + + static var shared: Self? +} +``` + +One way that this custom trait could customize the API credentials during each +test is if the `Trait` protocol were to expose a pair of method requirements +which were then called before and after the test, respectively: + +```swift +public protocol Trait: Sendable { + // ... + func setUp() async throws + func tearDown() async throws +} + +extension Trait { + // ... + public func setUp() async throws { /* No-op */ } + public func tearDown() async throws { /* No-op */ } +} +``` + +The custom trait type could adopt these using code such as the following: + +```swift +extension MockAPICredentialsTrait { + func setUp() { + APICredentials.shared = .init(apiKey: "...") + } + + func tearDown() { + APICredentials.shared = nil + } +} +``` + +Many testing systems use this pattern, including XCTest. However, this approach +encourages the use of global mutable state such as the `APICredentials.shared` +variable, and this limits the testing library's ability to parallelize test +execution, which is +[another part of the Swift Testing vision](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#parallelization-and-concurrency). + +The use of nonisolated static variables is generally discouraged now, and in +Swift 6 the above `APICredentials.shared` property produces an error. One way +to resolve that is to change it to a `@TaskLocal` variable, as this would be +concurrency-safe and still allow tests accessing this state to run in parallel: + +```swift +extension APICredentials { + @TaskLocal static var current: Self? +} +``` + +Binding task local values requires using the scoped access +[`TaskLocal.withValue()`](https://developer.apple.com/documentation/swift/tasklocal/withvalue(_:operation:isolation:file:line:) +API though, and that would not be possible if `Trait` exposed separate methods +like `setUp()` and `tearDown()`. + +For these reasons, I believe it's important to expose this trait capability +using a single, scoped access-style API which accepts a closure. A simplified +version of that idea might look like this: + +```swift +public protocol Trait: Sendable { + // ... + + // Simplified example, not the actual proposal + func executeTest(_ body: @Sendable () async throws -> Void) async throws +} + +extension MockAPICredentialsTrait { + func executeTest(_ body: @Sendable () async throws -> Void) async throws { + let mockCredentials = APICredentials(apiKey: "...") + try await APICredentials.$current.withValue(mockCredentials) { + try await body() + } + } +} +``` + +### Avoiding unnecessarily lengthy backtraces + +A scoped access-style API has some potential downsides. To apply this approach +to a test function, the scoped call of a trait must wrap the invocation of that +test function, and every _other_ trait applied to that same test which offers +custom behavior _also_ must wrap the other traits' calls in a nesting fashion. +To visualize this, imagine a test function with multiple traits: + +```swift +@Test(.traitA, .traitB, .traitC) +func exampleTest() { + // ... +} +``` + +If all three of those traits customize test execution behavior, then each of +them needs to wrap the call to the next one, and the last trait needs to wrap +the invocation of the test, illustrated by the following: + +``` +TraitA.executeTest { + TraitB.executeTest { + TraitC.executeTest { + exampleTest() + } + } +} +``` + +Tests may have an arbitrary number of traits applied to them, including those +inherited from containing suite types. A naïve implementation in which _every_ +trait is given the opportunity to customize test behavior by calling its scoped +access API might cause unnecessarily lengthy backtraces that make debugging the +body of tests more difficult. Or worse: if the number of traits is great enough, +it could cause a stack overflow. + +In practice, most traits probably will _not_ need to customize test behavior, so +to mitigate these downsides it's important that there be some way to distinguish +traits which customize test behavior. That way, the testing library can limit +these scoped access calls to only the traits which require it. + +## Detailed design + +I propose the following new APIs: + +- A new protocol `CustomTestExecuting` with a single required `execute(...)` + method. This will be called to run a test, and allows the conforming type to + perform custom logic before or after. +- A new property `customTestExecutor` on the `Trait` protocol whose type is an + `Optional` value of a type conforming to `CustomTestExecuting`. A `nil` value + from this property will skip calling the `execute(...)` method. +- A default implementation of `Trait.customTestExecutor` whose value is `nil`. +- A conditional implementation of `Trait.customTestExecutor` whose value is + `self` in the common case where the trait type conforms to + `CustomTestExecuting` itself. + +Since the `customTestExecutor` property is optional and `nil` by default, the +testing library cannot invoke the `execute(...)` method unless a trait +customizes test behavior. This avoids the "unnecessarily lengthy backtraces" +problem above. + +Below are the proposed interfaces: + +```swift +/// A protocol that allows customizing the execution of a test function (and +/// each of its cases) or a test suite by performing custom code before or after +/// it runs. +public protocol CustomTestExecuting: Sendable { + /// Execute a function for the specified test and/or test case. + /// + /// - Parameters: + /// - function: The function to perform. If `test` represents a test suite, + /// this function encapsulates running all the tests in that suite. If + /// `test` represents a test function, this function is the body of that + /// test function (including all cases if it is parameterized.) + /// - test: The test under which `function` is being performed. + /// - testCase: The test case, if any, under which `function` is being + /// performed. This is `nil` when invoked on a suite. + /// + /// - Throws: Whatever is thrown by `function`, or an error preventing + /// execution from running correctly. + /// + /// This function is called for each ``Trait`` on a test suite or test + /// function which has a non-`nil` value for ``Trait/customTestExecutor-1dwpt``. + /// It allows additional work to be performed before or after the test runs. + /// + /// This function is invoked once for the test its associated trait is applied + /// to, and then once for each test case in that test, if applicable. If a + /// test is skipped, this function is not invoked for that test or its cases. + /// + /// Issues recorded by this function are associated with `test`. + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws +} + +public protocol Trait: Sendable { + // ... + + /// The type of the custom test executor for this trait. + /// + /// The default type is `Never`. + associatedtype CustomTestExecutor: CustomTestExecuting = Never + + /// The custom test executor for this trait, if any. + /// + /// If this trait's type conforms to ``CustomTestExecuting``, the default + /// value of this property is `self` and this trait will be used to customize + /// test execution. This is the most straightforward way to implement a trait + /// which customizes the execution of tests. + /// + /// However, if the value of this property is an instance of another type + /// conforming to ``CustomTestExecuting``, that instance will be used to + /// perform custom test execution instead. Otherwise, the default value of + /// this property is `nil` (with the default type `Never?`), meaning that + /// custom test execution will not be performed for tests this trait is + /// applied to. + var customTestExecutor: CustomTestExecutor? { get } +} + +extension Trait { + // ... + + // The default implementation, which returns `nil`. + public var customTestExecutor: CustomTestExecutor? { get } +} + +extension Trait where CustomTestExecutor == Self { + // Returns `self`. + public var customTestExecutor: CustomTestExecutor? { get } +} + +extension Never: CustomTestExecuting { + public func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws +} +``` + +Here is a complete example of the usage scenario described earlier, showcasing +the proposed APIs: + +```swift +@Test(.mockAPICredentials) +func example() { + // ...validate API usage, referencing `APICredentials.current`... +} + +struct MockAPICredentialsTrait: TestTrait, CustomTestExecuting { + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + let mockCredentials = APICredentials(apiKey: "...") + try await APICredentials.$current.withValue(mockCredentials) { + try await function() + } + } +} + +extension Trait where Self == MockAPICredentialsTrait { + static var mockAPICredentials: Self { + Self() + } +} +``` + +## Source compatibility + +The proposed APIs are purely additive. + +## Integration with supporting tools + +Although some built-in traits are relevant to supporting tools (such as +SourceKit-LSP statically discovering `.tags` traits), custom test behaviors are +only relevant within the test executable process while tests are running. We +don't anticipate any particular need for this feature to integrate with +supporting tools. + +## Future directions + +Some test authors have expressed interest in allowing custom traits to access +the instance of a suite type for `@Test` instance methods, so the trait could +inspect or mutate the instance. Currently, only instance-level members of a +suite type (including `init`, `deinit`, and the test function itself) can access +`self`, so this would grant traits applied to an instance test method access to +the instance as well. This is certainly interesting, but poses several technical +challenges that puts it out of scope of this proposal. + +## Alternatives considered + +### Separate set up & tear down methods on `Trait` + +This idea was discussed in [Supporting scoped access](#supporting-scoped-access) +above, and as mentioned there, the primary problem with this approach is that it +cannot be used with scoped access-style APIs, including (importantly) +`TaskLocal.withValue()`. For that reason, it prevents using that common Swift +concurrency technique and reduces the potential for test parallelization. + +### Add `execute(...)` directly to the `Trait` protocol + +The proposed `execute(...)` method could be added as a requirement of the +`Trait` protocol instead of being part of a separate `CustomTestExecuting` +protocol, and it could have a default implementation which directly invokes the +passed-in closure. But this approach would suffer from the lengthy backtrace +problem described above. + +### Extend the `Trait` protocol + +The original SPI implementation of this feature included a protocol named +`CustomExecutionTrait` which extended `Trait` and had roughly the same method +requirement as the `CustomTestExecuting` protocol proposed above. This design +worked, provided scoped access, and avoided the lengthy backtrace problem. + +After evaluating the design and usage of this SPI though, it seemed unfortunate +to structure it as a sub-protocol of `Trait` because it means that the full +capabilities of the trait system are spread across multiple protocols. In the +proposed design, the ability to provide a custom test executor value is exposed +via the main `Trait` protocol, and it relies on an associated type to +conditionally opt-in to custom test behavior. In other words, the proposed +design expresses custom test behavior as just a _capability_ that a trait may +have, rather than a distinct sub-type of trait. + +Also, the implementation of this approach within the testing library was not +ideal as it required a conditional `trait as? CustomExecutionTrait` downcast at +runtime, in contrast to the simpler and more performant Optional property of the +proposed API. + +## Acknowledgments + +Thanks to [Dennis Weissmann](https://github.com/dennisweissmann) for originally +implementing this as SPI, and for helping promote its usefulness. + +Thanks to [Jonathan Grynspan](https://github.com/grynspan) for exploring ideas +to refine the API, and considering alternatives to avoid unnecessarily long +backtraces. diff --git a/Sources/Testing/Running/Runner.swift b/Sources/Testing/Running/Runner.swift index 95448533..e327a465 100644 --- a/Sources/Testing/Running/Runner.swift +++ b/Sources/Testing/Running/Runner.swift @@ -56,19 +56,20 @@ public struct Runner: Sendable { // MARK: - Running tests extension Runner { - /// Execute the ``CustomExecutionTrait/execute(_:for:testCase:)`` functions - /// associated with the test in a plan step. + /// Execute the ``CustomTestExecuting/execute(_:for:testCase:)`` functions of + /// any custom test executors for traits associated with the test in a plan + /// step. /// /// - Parameters: /// - step: The step being performed. /// - testCase: The test case, if applicable, for which to execute the - /// custom trait. + /// function. /// - body: A function to execute from within the - /// ``CustomExecutionTrait/execute(_:for:testCase:)`` functions of each - /// trait applied to `step.test`. + /// ``CustomTestExecuting/execute(_:for:testCase:)`` function of each + /// non-`nil` custom test executor of the traits applied to `step.test`. /// /// - Throws: Whatever is thrown by `body` or by any of the - /// ``CustomExecutionTrait/execute(_:for:testCase:)`` functions. + /// ``CustomTestExecuting/execute(_:for:testCase:)`` functions. private func _executeTraits( for step: Plan.Step, testCase: Test.Case?, @@ -90,11 +91,11 @@ extension Runner { // and ultimately the first trait is the first one to be invoked. let executeAllTraits = step.test.traits.lazy .reversed() - .compactMap { $0 as? any CustomExecutionTrait } - .compactMap { $0.execute(_:for:testCase:) } - .reduce(body) { executeAllTraits, traitExecutor in + .compactMap { $0.customTestExecutor } + .map { $0.execute(_:for:testCase:) } + .reduce(body) { executeAllTraits, testExecutor in { - try await traitExecutor(executeAllTraits, step.test, testCase) + try await testExecutor(executeAllTraits, step.test, testCase) } } diff --git a/Sources/Testing/Testing.docc/Traits/Trait.md b/Sources/Testing/Testing.docc/Traits/Trait.md index 1528ec1b..f7c1f08f 100644 --- a/Sources/Testing/Testing.docc/Traits/Trait.md +++ b/Sources/Testing/Testing.docc/Traits/Trait.md @@ -39,8 +39,15 @@ See https://swift.org/CONTRIBUTORS.txt for Swift project authors - ``Trait/bug(_:id:_:)-3vtpl`` ### Adding information to tests + - ``Trait/comments`` ### Preparing internal state - ``Trait/prepare(for:)-3s3zo`` + +### Customizing the execution of tests + +- ``CustomTestExecuting`` +- ``Trait/customTestExecutor-1dwpt`` +- ``Trait/CustomTestExecutor`` diff --git a/Sources/Testing/Traits/Trait.swift b/Sources/Testing/Traits/Trait.swift index e6a42b4d..5da94b7c 100644 --- a/Sources/Testing/Traits/Trait.swift +++ b/Sources/Testing/Traits/Trait.swift @@ -41,6 +41,68 @@ public protocol Trait: Sendable { /// /// By default, the value of this property is an empty array. var comments: [Comment] { get } + + /// The type of the custom test executor for this trait. + /// + /// The default type is `Never`. + associatedtype CustomTestExecutor: CustomTestExecuting = Never + + /// The custom test executor for this trait, if any. + /// + /// If this trait's type conforms to ``CustomTestExecuting``, the default + /// value of this property is `self` and this trait will be used to customize + /// test execution. This is the most straightforward way to implement a trait + /// which customizes the execution of tests. + /// + /// However, if the value of this property is an instance of another type + /// conforming to ``CustomTestExecuting``, that instance will be used to + /// perform custom test execution instead. Otherwise, the default value of + /// this property is `nil` (with the default type `Never?`), meaning that + /// custom test execution will not be performed for tests this trait is + /// applied to. + var customTestExecutor: CustomTestExecutor? { get } +} + +/// A protocol that allows customizing the execution of a test function (and +/// each of its cases) or a test suite by performing custom code before or after +/// it runs. +public protocol CustomTestExecuting: Sendable { + /// Execute a function for the specified test and/or test case. + /// + /// - Parameters: + /// - function: The function to perform. If `test` represents a test suite, + /// this function encapsulates running all the tests in that suite. If + /// `test` represents a test function, this function is the body of that + /// test function (including all cases if it is parameterized.) + /// - test: The test under which `function` is being performed. + /// - testCase: The test case, if any, under which `function` is being + /// performed. This is `nil` when invoked on a suite. + /// + /// - Throws: Whatever is thrown by `function`, or an error preventing + /// execution from running correctly. + /// + /// This function is called for each ``Trait`` on a test suite or test + /// function which has a non-`nil` value for ``Trait/customTestExecutor-1dwpt``. + /// It allows additional work to be performed before or after the test runs. + /// + /// This function is invoked once for the test its associated trait is applied + /// to, and then once for each test case in that test, if applicable. If a + /// test is skipped, this function is not invoked for that test or its cases. + /// + /// Issues recorded by this function are associated with `test`. + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws +} + +extension Trait where CustomTestExecutor == Self { + public var customTestExecutor: CustomTestExecutor? { + self + } +} + +extension Never: CustomTestExecuting { + public func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + fatalError("Unreachable codepath: Never cannot be instantiated.") + } } /// A protocol describing traits that can be added to a test function. @@ -70,6 +132,10 @@ extension Trait { public var comments: [Comment] { [] } + + public var customTestExecutor: CustomTestExecutor? { + nil + } } extension SuiteTrait { @@ -77,38 +143,3 @@ extension SuiteTrait { false } } - -/// A protocol extending ``Trait`` that offers an additional customization point -/// for trait authors to execute code before and after each test function (if -/// added to the traits of a test function), or before and after each test suite -/// (if added to the traits of a test suite). -@_spi(Experimental) -public protocol CustomExecutionTrait: Trait { - - /// Execute a function with the effects of this trait applied. - /// - /// - Parameters: - /// - function: The function to perform. If `test` represents a test suite, - /// this function encapsulates running all the tests in that suite. If - /// `test` represents a test function, this function is the body of that - /// test function (including all cases if it is parameterized.) - /// - test: The test under which `function` is being performed. - /// - testCase: The test case, if any, under which `function` is being - /// performed. This is `nil` when invoked on a suite. - /// - /// - Throws: Whatever is thrown by `function`, or an error preventing the - /// trait from running correctly. - /// - /// This function is called for each ``CustomExecutionTrait`` on a test suite - /// or test function and allows additional work to be performed before and - /// after the test runs. - /// - /// This function is invoked once for the test it is applied to, and then once - /// for each test case in that test, if applicable. - /// - /// Issues recorded by this function are recorded against `test`. - /// - /// - Note: If a test function or test suite is skipped, this function does - /// not get invoked by the runner. - func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws -} diff --git a/Tests/TestingTests/Traits/CustomExecutionTraitTests.swift b/Tests/TestingTests/Traits/CustomTestExecutingTraitTests.swift similarity index 90% rename from Tests/TestingTests/Traits/CustomExecutionTraitTests.swift rename to Tests/TestingTests/Traits/CustomTestExecutingTraitTests.swift index aedc06de..fae72e8b 100644 --- a/Tests/TestingTests/Traits/CustomExecutionTraitTests.swift +++ b/Tests/TestingTests/Traits/CustomTestExecutingTraitTests.swift @@ -10,8 +10,8 @@ @testable @_spi(Experimental) @_spi(ForToolsIntegrationOnly) import Testing -@Suite("CustomExecutionTrait Tests") -struct CustomExecutionTraitTests { +@Suite("CustomTestExecuting-conforming Trait Tests") +struct CustomTestExecutingTraitTests { @Test("Execute code before and after a non-parameterized test.") func executeCodeBeforeAndAfterNonParameterizedTest() async { // `expectedCount` is 2 because we run it both for the test and the test case @@ -65,7 +65,7 @@ struct CustomExecutionTraitTests { // MARK: - Fixtures -private struct CustomTrait: CustomExecutionTrait, TestTrait { +private struct CustomTrait: TestTrait, CustomTestExecuting { var before: Confirmation var after: Confirmation func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { @@ -77,7 +77,7 @@ private struct CustomTrait: CustomExecutionTrait, TestTrait { } } -private struct CustomThrowingErrorTrait: CustomExecutionTrait, TestTrait { +private struct CustomThrowingErrorTrait: TestTrait, CustomTestExecuting { fileprivate struct CustomTraitError: Error {} func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { @@ -85,10 +85,10 @@ private struct CustomThrowingErrorTrait: CustomExecutionTrait, TestTrait { } } -struct DoSomethingBeforeAndAfterTrait: CustomExecutionTrait, SuiteTrait, TestTrait { +struct DoSomethingBeforeAndAfterTrait: SuiteTrait, TestTrait, CustomTestExecuting { static let state = Locked(rawValue: 0) - func execute(_ function: @Sendable () async throws -> Void, for test: Testing.Test, testCase: Testing.Test.Case?) async throws { + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { #expect(Self.state.increment() == 1) try await function() From fae5f7867bc7745e8ba7867880a3b1d32ff63916 Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Wed, 25 Sep 2024 16:32:55 -0500 Subject: [PATCH 2/9] Proposal edits --- .../Proposals/NNNN-custom-test-execution-traits.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Documentation/Proposals/NNNN-custom-test-execution-traits.md b/Documentation/Proposals/NNNN-custom-test-execution-traits.md index c5a593cc..98d5f22d 100644 --- a/Documentation/Proposals/NNNN-custom-test-execution-traits.md +++ b/Documentation/Proposals/NNNN-custom-test-execution-traits.md @@ -119,7 +119,7 @@ extension APICredentials { ``` Binding task local values requires using the scoped access -[`TaskLocal.withValue()`](https://developer.apple.com/documentation/swift/tasklocal/withvalue(_:operation:isolation:file:line:) +[`TaskLocal.withValue()`](https://developer.apple.com/documentation/swift/tasklocal/withvalue(_:operation:isolation:file:line:)) API though, and that would not be possible if `Trait` exposed separate methods like `setUp()` and `tearDown()`. @@ -347,10 +347,10 @@ problem described above. ### Extend the `Trait` protocol -The original SPI implementation of this feature included a protocol named -`CustomExecutionTrait` which extended `Trait` and had roughly the same method -requirement as the `CustomTestExecuting` protocol proposed above. This design -worked, provided scoped access, and avoided the lengthy backtrace problem. +The original, experimental implementation of this feature included a protocol +named`CustomExecutionTrait` which extended `Trait` and had roughly the same +method requirement as the `CustomTestExecuting` protocol proposed above. This +design worked, provided scoped access, and avoided the lengthy backtrace problem. After evaluating the design and usage of this SPI though, it seemed unfortunate to structure it as a sub-protocol of `Trait` because it means that the full From 30bb7008f67e34531766f3fc8d76bcfe3fb58654 Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Thu, 26 Sep 2024 15:54:24 -0500 Subject: [PATCH 3/9] PR feedback --- Sources/Testing/Traits/Trait.swift | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/Sources/Testing/Traits/Trait.swift b/Sources/Testing/Traits/Trait.swift index 5da94b7c..44c0f6be 100644 --- a/Sources/Testing/Traits/Trait.swift +++ b/Sources/Testing/Traits/Trait.swift @@ -54,12 +54,13 @@ public protocol Trait: Sendable { /// test execution. This is the most straightforward way to implement a trait /// which customizes the execution of tests. /// - /// However, if the value of this property is an instance of another type + /// If the value of this property is an instance of a different type /// conforming to ``CustomTestExecuting``, that instance will be used to - /// perform custom test execution instead. Otherwise, the default value of - /// this property is `nil` (with the default type `Never?`), meaning that - /// custom test execution will not be performed for tests this trait is - /// applied to. + /// perform custom test execution instead. + /// + /// The default value of this property is `nil` (with the default type + /// `Never?`), meaning that instances of this type will not perform any custom + /// test execution for tests they are applied to. var customTestExecutor: CustomTestExecutor? { get } } @@ -76,10 +77,13 @@ public protocol CustomTestExecuting: Sendable { /// test function (including all cases if it is parameterized.) /// - test: The test under which `function` is being performed. /// - testCase: The test case, if any, under which `function` is being - /// performed. This is `nil` when invoked on a suite. + /// performed. When invoked on a suite, the value of this argument is + /// `nil`. /// /// - Throws: Whatever is thrown by `function`, or an error preventing - /// execution from running correctly. + /// execution from running correctly. An error thrown from this function is + /// recorded as an issue associated with `test`. If an error is thrown + /// before `function` is called, the corresponding test will not run. /// /// This function is called for each ``Trait`` on a test suite or test /// function which has a non-`nil` value for ``Trait/customTestExecutor-1dwpt``. @@ -100,9 +104,7 @@ extension Trait where CustomTestExecutor == Self { } extension Never: CustomTestExecuting { - public func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { - fatalError("Unreachable codepath: Never cannot be instantiated.") - } + public func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws {} } /// A protocol describing traits that can be added to a test function. @@ -132,7 +134,9 @@ extension Trait { public var comments: [Comment] { [] } +} +extension Trait where CustomTestExecutor == Never { public var customTestExecutor: CustomTestExecutor? { nil } From 147f9cef7f735b6d116783d6d9a3f897c6c0fe4b Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Fri, 27 Sep 2024 11:57:51 -0500 Subject: [PATCH 4/9] Refine the documentation and proposal further --- Sources/Testing/Traits/Trait.swift | 33 ++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-) diff --git a/Sources/Testing/Traits/Trait.swift b/Sources/Testing/Traits/Trait.swift index 44c0f6be..726bb5d5 100644 --- a/Sources/Testing/Traits/Trait.swift +++ b/Sources/Testing/Traits/Trait.swift @@ -44,7 +44,10 @@ public protocol Trait: Sendable { /// The type of the custom test executor for this trait. /// - /// The default type is `Never`. + /// The default type is `Never`, which cannot be instantiated. This means the + /// value of the ``customTestExecutor-1dwpt`` property for all traits with the + /// default custom executor type is `nil`, meaning such traits will not + /// perform any custom execution for the tests they're applied to. associatedtype CustomTestExecutor: CustomTestExecuting = Never /// The custom test executor for this trait, if any. @@ -67,6 +70,14 @@ public protocol Trait: Sendable { /// A protocol that allows customizing the execution of a test function (and /// each of its cases) or a test suite by performing custom code before or after /// it runs. +/// +/// Types conforming to this protocol may be used in conjunction with a +/// ``Trait``-conforming type by implementing the +/// ``Trait/customTestExecutor-1dwpt`` property, allowing custom traits to +/// customize the execution of tests. Consolidating common set-up and tear-down +/// logic for tests which have similar needs allows each test function to be +/// more succinct with less repetitive boilerplate so it can focus on what makes +/// it unique. public protocol CustomTestExecuting: Sendable { /// Execute a function for the specified test and/or test case. /// @@ -81,19 +92,25 @@ public protocol CustomTestExecuting: Sendable { /// `nil`. /// /// - Throws: Whatever is thrown by `function`, or an error preventing - /// execution from running correctly. An error thrown from this function is + /// execution from running correctly. An error thrown from this method is /// recorded as an issue associated with `test`. If an error is thrown /// before `function` is called, the corresponding test will not run. /// - /// This function is called for each ``Trait`` on a test suite or test - /// function which has a non-`nil` value for ``Trait/customTestExecutor-1dwpt``. - /// It allows additional work to be performed before or after the test runs. + /// When the testing library is preparing to run a test, it finds all traits + /// applied to that test (including those inherited from containing suites) + /// and asks each for the value of its + /// ``Trait/customTestExecutor-1dwpt`` property. It then calls this method on + /// all non-`nil` instances, giving each an opportunity to perform + /// arbitrary work before or after invoking `function`. + /// + /// This method should either invoke `function` once before returning or throw + /// an error if it is unable to perform its custom logic successfully. /// - /// This function is invoked once for the test its associated trait is applied + /// This method is invoked once for the test its associated trait is applied /// to, and then once for each test case in that test, if applicable. If a - /// test is skipped, this function is not invoked for that test or its cases. + /// test is skipped, this method is not invoked for that test or its cases. /// - /// Issues recorded by this function are associated with `test`. + /// Issues recorded by this method are associated with `test`. func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws } From 636da1af70bb29c3400b30caf24e3294760a5a9f Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Fri, 27 Sep 2024 12:36:40 -0500 Subject: [PATCH 5/9] Update proposal to reflect latest API changes --- .../NNNN-custom-test-execution-traits.md | 63 ++++++++++++------- 1 file changed, 40 insertions(+), 23 deletions(-) diff --git a/Documentation/Proposals/NNNN-custom-test-execution-traits.md b/Documentation/Proposals/NNNN-custom-test-execution-traits.md index 98d5f22d..ba3b7275 100644 --- a/Documentation/Proposals/NNNN-custom-test-execution-traits.md +++ b/Documentation/Proposals/NNNN-custom-test-execution-traits.md @@ -212,6 +212,14 @@ Below are the proposed interfaces: /// A protocol that allows customizing the execution of a test function (and /// each of its cases) or a test suite by performing custom code before or after /// it runs. +/// +/// Types conforming to this protocol may be used in conjunction with a +/// ``Trait``-conforming type by implementing the +/// ``Trait/customTestExecutor-1dwpt`` property, allowing custom traits to +/// customize the execution of tests. Consolidating common set-up and tear-down +/// logic for tests which have similar needs allows each test function to be +/// more succinct with less repetitive boilerplate so it can focus on what makes +/// it unique. public protocol CustomTestExecuting: Sendable { /// Execute a function for the specified test and/or test case. /// @@ -222,20 +230,29 @@ public protocol CustomTestExecuting: Sendable { /// test function (including all cases if it is parameterized.) /// - test: The test under which `function` is being performed. /// - testCase: The test case, if any, under which `function` is being - /// performed. This is `nil` when invoked on a suite. + /// performed. When invoked on a suite, the value of this argument is + /// `nil`. /// /// - Throws: Whatever is thrown by `function`, or an error preventing - /// execution from running correctly. + /// execution from running correctly. An error thrown from this method is + /// recorded as an issue associated with `test`. If an error is thrown + /// before `function` is called, the corresponding test will not run. /// - /// This function is called for each ``Trait`` on a test suite or test - /// function which has a non-`nil` value for ``Trait/customTestExecutor-1dwpt``. - /// It allows additional work to be performed before or after the test runs. + /// When the testing library is preparing to run a test, it finds all traits + /// applied to that test (including those inherited from containing suites) + /// and asks each for the value of its + /// ``Trait/customTestExecutor-1dwpt`` property. It then calls this method on + /// all non-`nil` instances, giving each an opportunity to perform + /// arbitrary work before or after invoking `function`. /// - /// This function is invoked once for the test its associated trait is applied + /// This method should either invoke `function` once before returning or throw + /// an error if it is unable to perform its custom logic successfully. + /// + /// This method is invoked once for the test its associated trait is applied /// to, and then once for each test case in that test, if applicable. If a - /// test is skipped, this function is not invoked for that test or its cases. + /// test is skipped, this method is not invoked for that test or its cases. /// - /// Issues recorded by this function are associated with `test`. + /// Issues recorded by this method are associated with `test`. func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws } @@ -244,7 +261,10 @@ public protocol Trait: Sendable { /// The type of the custom test executor for this trait. /// - /// The default type is `Never`. + /// The default type is `Never`, which cannot be instantiated. This means the + /// value of the ``customTestExecutor-1dwpt`` property for all traits with the + /// default custom executor type is `nil`, meaning such traits will not + /// perform any custom execution for the tests they're applied to. associatedtype CustomTestExecutor: CustomTestExecuting = Never /// The custom test executor for this trait, if any. @@ -254,30 +274,27 @@ public protocol Trait: Sendable { /// test execution. This is the most straightforward way to implement a trait /// which customizes the execution of tests. /// - /// However, if the value of this property is an instance of another type + /// If the value of this property is an instance of a different type /// conforming to ``CustomTestExecuting``, that instance will be used to - /// perform custom test execution instead. Otherwise, the default value of - /// this property is `nil` (with the default type `Never?`), meaning that - /// custom test execution will not be performed for tests this trait is - /// applied to. + /// perform custom test execution instead. + /// + /// The default value of this property is `nil` (with the default type + /// `Never?`), meaning that instances of this type will not perform any custom + /// test execution for tests they are applied to. var customTestExecutor: CustomTestExecutor? { get } } -extension Trait { - // ... - - // The default implementation, which returns `nil`. - public var customTestExecutor: CustomTestExecutor? { get } -} - extension Trait where CustomTestExecutor == Self { // Returns `self`. public var customTestExecutor: CustomTestExecutor? { get } } -extension Never: CustomTestExecuting { - public func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws +extension Trait where CustomTestExecutor == Never { + // Returns `nil`. + public var customTestExecutor: CustomTestExecutor? { get } } + +extension Never: CustomTestExecuting {} ``` Here is a complete example of the usage scenario described earlier, showcasing From 4d88f1cf16776423dddbdf2c9b356f016f62e260 Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Fri, 27 Sep 2024 15:42:10 -0500 Subject: [PATCH 6/9] Correct the conditional Trait extensions --- Documentation/Proposals/NNNN-custom-test-execution-traits.md | 4 ++-- Sources/Testing/Traits/Trait.swift | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Documentation/Proposals/NNNN-custom-test-execution-traits.md b/Documentation/Proposals/NNNN-custom-test-execution-traits.md index ba3b7275..914df86d 100644 --- a/Documentation/Proposals/NNNN-custom-test-execution-traits.md +++ b/Documentation/Proposals/NNNN-custom-test-execution-traits.md @@ -284,9 +284,9 @@ public protocol Trait: Sendable { var customTestExecutor: CustomTestExecutor? { get } } -extension Trait where CustomTestExecutor == Self { +extension Trait where Self: CustomTestExecuting { // Returns `self`. - public var customTestExecutor: CustomTestExecutor? { get } + public var customTestExecutor: Self? { get } } extension Trait where CustomTestExecutor == Never { diff --git a/Sources/Testing/Traits/Trait.swift b/Sources/Testing/Traits/Trait.swift index 726bb5d5..d1c43457 100644 --- a/Sources/Testing/Traits/Trait.swift +++ b/Sources/Testing/Traits/Trait.swift @@ -114,8 +114,8 @@ public protocol CustomTestExecuting: Sendable { func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws } -extension Trait where CustomTestExecutor == Self { - public var customTestExecutor: CustomTestExecutor? { +extension Trait where Self: CustomTestExecuting { + public var customTestExecutor: Self? { self } } From a120b3ece40960fd9a7a11a84bcbf15e1b4da16e Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Thu, 3 Oct 2024 11:09:43 -0500 Subject: [PATCH 7/9] Drop 'Custom' prefix from proposed API names --- .../NNNN-custom-test-execution-traits.md | 100 +++++++++--------- Sources/Testing/Running/Runner.swift | 13 ++- Sources/Testing/Testing.docc/Traits/Trait.md | 6 +- Sources/Testing/Traits/Trait.swift | 42 ++++---- ...ts.swift => TestExecutingTraitTests.swift} | 10 +- 5 files changed, 87 insertions(+), 84 deletions(-) rename Tests/TestingTests/Traits/{CustomTestExecutingTraitTests.swift => TestExecutingTraitTests.swift} (91%) diff --git a/Documentation/Proposals/NNNN-custom-test-execution-traits.md b/Documentation/Proposals/NNNN-custom-test-execution-traits.md index 914df86d..510c8ce2 100644 --- a/Documentation/Proposals/NNNN-custom-test-execution-traits.md +++ b/Documentation/Proposals/NNNN-custom-test-execution-traits.md @@ -6,6 +6,12 @@ * Implementation: [swiftlang/swift-testing#733](https://github.com/swiftlang/swift-testing/pull/733), [swiftlang/swift-testing#86](https://github.com/swiftlang/swift-testing/pull/86) * Review: ([pitch](https://forums.swift.org/...)) +### Revision history + +* **v1**: Initial pitch. +* **v2**: Dropped 'Custom' prefix from the proposed API names (although kept the + word in certain documentation passages where it clarified behavior). + ## Introduction This introduces API which enables a custom `Trait`-conforming type to customize @@ -190,21 +196,19 @@ these scoped access calls to only the traits which require it. I propose the following new APIs: -- A new protocol `CustomTestExecuting` with a single required `execute(...)` - method. This will be called to run a test, and allows the conforming type to - perform custom logic before or after. -- A new property `customTestExecutor` on the `Trait` protocol whose type is an - `Optional` value of a type conforming to `CustomTestExecuting`. A `nil` value - from this property will skip calling the `execute(...)` method. -- A default implementation of `Trait.customTestExecutor` whose value is `nil`. -- A conditional implementation of `Trait.customTestExecutor` whose value is - `self` in the common case where the trait type conforms to - `CustomTestExecuting` itself. - -Since the `customTestExecutor` property is optional and `nil` by default, the -testing library cannot invoke the `execute(...)` method unless a trait -customizes test behavior. This avoids the "unnecessarily lengthy backtraces" -problem above. +- A new protocol `TestExecuting` with a single required `execute(...)` method. + This will be called to run a test, and allows the conforming type to perform + custom logic before or after. +- A new property `testExecutor` on the `Trait` protocol whose type is an + `Optional` value of a type conforming to `TestExecuting`. A `nil` value for + this property will skip calling the `execute(...)` method. +- A default implementation of `Trait.testExecutor` whose value is `nil`. +- A conditional implementation of `Trait.testExecutor` whose value is `self` + in the common case where the trait type conforms to `TestExecuting` itself. + +Since the `testExecutor` property is optional and `nil` by default, the testing +library cannot invoke the `execute(...)` method unless a trait customizes test +behavior. This avoids the "unnecessarily lengthy backtraces" problem above. Below are the proposed interfaces: @@ -215,12 +219,12 @@ Below are the proposed interfaces: /// /// Types conforming to this protocol may be used in conjunction with a /// ``Trait``-conforming type by implementing the -/// ``Trait/customTestExecutor-1dwpt`` property, allowing custom traits to +/// ``Trait/testExecutor-714gp`` property, allowing custom traits to /// customize the execution of tests. Consolidating common set-up and tear-down /// logic for tests which have similar needs allows each test function to be /// more succinct with less repetitive boilerplate so it can focus on what makes /// it unique. -public protocol CustomTestExecuting: Sendable { +public protocol TestExecuting: Sendable { /// Execute a function for the specified test and/or test case. /// /// - Parameters: @@ -241,8 +245,8 @@ public protocol CustomTestExecuting: Sendable { /// When the testing library is preparing to run a test, it finds all traits /// applied to that test (including those inherited from containing suites) /// and asks each for the value of its - /// ``Trait/customTestExecutor-1dwpt`` property. It then calls this method on - /// all non-`nil` instances, giving each an opportunity to perform + /// ``Trait/testExecutor-714gp`` property. It then calls this method + /// on all non-`nil` instances, giving each an opportunity to perform /// arbitrary work before or after invoking `function`. /// /// This method should either invoke `function` once before returning or throw @@ -259,42 +263,42 @@ public protocol CustomTestExecuting: Sendable { public protocol Trait: Sendable { // ... - /// The type of the custom test executor for this trait. + /// The type of the test executor for this trait. /// /// The default type is `Never`, which cannot be instantiated. This means the - /// value of the ``customTestExecutor-1dwpt`` property for all traits with the - /// default custom executor type is `nil`, meaning such traits will not + /// value of the ``testExecutor-714gp`` property for all traits with + /// the default custom executor type is `nil`, meaning such traits will not /// perform any custom execution for the tests they're applied to. - associatedtype CustomTestExecutor: CustomTestExecuting = Never + associatedtype TestExecutor: TestExecuting = Never - /// The custom test executor for this trait, if any. + /// The test executor for this trait, if any. /// - /// If this trait's type conforms to ``CustomTestExecuting``, the default - /// value of this property is `self` and this trait will be used to customize - /// test execution. This is the most straightforward way to implement a trait - /// which customizes the execution of tests. + /// If this trait's type conforms to ``TestExecuting``, the default value of + /// this property is `self` and this trait will be used to customize test + /// execution. This is the most straightforward way to implement a trait which + /// customizes the execution of tests. /// /// If the value of this property is an instance of a different type - /// conforming to ``CustomTestExecuting``, that instance will be used to - /// perform custom test execution instead. + /// conforming to ``TestExecuting``, that instance will be used to perform + /// test execution instead. /// /// The default value of this property is `nil` (with the default type /// `Never?`), meaning that instances of this type will not perform any custom /// test execution for tests they are applied to. - var customTestExecutor: CustomTestExecutor? { get } + var testExecutor: TestExecutor? { get } } -extension Trait where Self: CustomTestExecuting { +extension Trait where Self: TestExecuting { // Returns `self`. - public var customTestExecutor: Self? { get } + public var testExecutor: Self? { } -extension Trait where CustomTestExecutor == Never { +extension Trait where TestExecutor == Never { // Returns `nil`. - public var customTestExecutor: CustomTestExecutor? { get } + public var testExecutor: TestExecutor? { } -extension Never: CustomTestExecuting {} +extension Never: TestExecuting {} ``` Here is a complete example of the usage scenario described earlier, showcasing @@ -306,7 +310,7 @@ func example() { // ...validate API usage, referencing `APICredentials.current`... } -struct MockAPICredentialsTrait: TestTrait, CustomTestExecuting { +struct MockAPICredentialsTrait: TestTrait, TestExecuting { func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { let mockCredentials = APICredentials(apiKey: "...") try await APICredentials.$current.withValue(mockCredentials) { @@ -357,26 +361,26 @@ concurrency technique and reduces the potential for test parallelization. ### Add `execute(...)` directly to the `Trait` protocol The proposed `execute(...)` method could be added as a requirement of the -`Trait` protocol instead of being part of a separate `CustomTestExecuting` -protocol, and it could have a default implementation which directly invokes the -passed-in closure. But this approach would suffer from the lengthy backtrace -problem described above. +`Trait` protocol instead of being part of a separate `TestExecuting` protocol, +and it could have a default implementation which directly invokes the passed-in +closure. But this approach would suffer from the lengthy backtrace problem +described above. ### Extend the `Trait` protocol The original, experimental implementation of this feature included a protocol named`CustomExecutionTrait` which extended `Trait` and had roughly the same -method requirement as the `CustomTestExecuting` protocol proposed above. This -design worked, provided scoped access, and avoided the lengthy backtrace problem. +method requirement as the `TestExecuting` protocol proposed above. This design +worked, provided scoped access, and avoided the lengthy backtrace problem. After evaluating the design and usage of this SPI though, it seemed unfortunate to structure it as a sub-protocol of `Trait` because it means that the full capabilities of the trait system are spread across multiple protocols. In the -proposed design, the ability to provide a custom test executor value is exposed -via the main `Trait` protocol, and it relies on an associated type to -conditionally opt-in to custom test behavior. In other words, the proposed -design expresses custom test behavior as just a _capability_ that a trait may -have, rather than a distinct sub-type of trait. +proposed design, the ability to provide a test executor value is exposed via the +main `Trait` protocol, and it relies on an associated type to conditionally +opt-in to custom test behavior. In other words, the proposed design expresses +custom test behavior as just a _capability_ that a trait may have, rather than a +distinct sub-type of trait. Also, the implementation of this approach within the testing library was not ideal as it required a conditional `trait as? CustomExecutionTrait` downcast at diff --git a/Sources/Testing/Running/Runner.swift b/Sources/Testing/Running/Runner.swift index e327a465..530fba24 100644 --- a/Sources/Testing/Running/Runner.swift +++ b/Sources/Testing/Running/Runner.swift @@ -56,20 +56,19 @@ public struct Runner: Sendable { // MARK: - Running tests extension Runner { - /// Execute the ``CustomTestExecuting/execute(_:for:testCase:)`` functions of - /// any custom test executors for traits associated with the test in a plan - /// step. + /// Execute the ``TestExecuting/execute(_:for:testCase:)`` functions of any + /// test executors for traits associated with the test in a plan step. /// /// - Parameters: /// - step: The step being performed. /// - testCase: The test case, if applicable, for which to execute the /// function. /// - body: A function to execute from within the - /// ``CustomTestExecuting/execute(_:for:testCase:)`` function of each - /// non-`nil` custom test executor of the traits applied to `step.test`. + /// ``TestExecuting/execute(_:for:testCase:)`` function of each non-`nil` + /// test executor of the traits applied to `step.test`. /// /// - Throws: Whatever is thrown by `body` or by any of the - /// ``CustomTestExecuting/execute(_:for:testCase:)`` functions. + /// ``TestExecuting/execute(_:for:testCase:)`` functions. private func _executeTraits( for step: Plan.Step, testCase: Test.Case?, @@ -91,7 +90,7 @@ extension Runner { // and ultimately the first trait is the first one to be invoked. let executeAllTraits = step.test.traits.lazy .reversed() - .compactMap { $0.customTestExecutor } + .compactMap { $0.testExecutor } .map { $0.execute(_:for:testCase:) } .reduce(body) { executeAllTraits, testExecutor in { diff --git a/Sources/Testing/Testing.docc/Traits/Trait.md b/Sources/Testing/Testing.docc/Traits/Trait.md index f7c1f08f..632c323d 100644 --- a/Sources/Testing/Testing.docc/Traits/Trait.md +++ b/Sources/Testing/Testing.docc/Traits/Trait.md @@ -48,6 +48,6 @@ See https://swift.org/CONTRIBUTORS.txt for Swift project authors ### Customizing the execution of tests -- ``CustomTestExecuting`` -- ``Trait/customTestExecutor-1dwpt`` -- ``Trait/CustomTestExecutor`` +- ``TestExecuting`` +- ``Trait/testExecutor-714gp`` +- ``Trait/TestExecutor`` diff --git a/Sources/Testing/Traits/Trait.swift b/Sources/Testing/Traits/Trait.swift index d1c43457..3682c03a 100644 --- a/Sources/Testing/Traits/Trait.swift +++ b/Sources/Testing/Traits/Trait.swift @@ -42,29 +42,29 @@ public protocol Trait: Sendable { /// By default, the value of this property is an empty array. var comments: [Comment] { get } - /// The type of the custom test executor for this trait. + /// The type of the test executor for this trait. /// /// The default type is `Never`, which cannot be instantiated. This means the - /// value of the ``customTestExecutor-1dwpt`` property for all traits with the - /// default custom executor type is `nil`, meaning such traits will not + /// value of the ``testExecutor-714gp`` property for all traits with + /// the default custom executor type is `nil`, meaning such traits will not /// perform any custom execution for the tests they're applied to. - associatedtype CustomTestExecutor: CustomTestExecuting = Never + associatedtype TestExecutor: TestExecuting = Never - /// The custom test executor for this trait, if any. + /// The test executor for this trait, if any. /// - /// If this trait's type conforms to ``CustomTestExecuting``, the default - /// value of this property is `self` and this trait will be used to customize - /// test execution. This is the most straightforward way to implement a trait - /// which customizes the execution of tests. + /// If this trait's type conforms to ``TestExecuting``, the default value of + /// this property is `self` and this trait will be used to customize test + /// execution. This is the most straightforward way to implement a trait which + /// customizes the execution of tests. /// /// If the value of this property is an instance of a different type - /// conforming to ``CustomTestExecuting``, that instance will be used to - /// perform custom test execution instead. + /// conforming to ``TestExecuting``, that instance will be used to perform + /// test execution instead. /// /// The default value of this property is `nil` (with the default type /// `Never?`), meaning that instances of this type will not perform any custom /// test execution for tests they are applied to. - var customTestExecutor: CustomTestExecutor? { get } + var testExecutor: TestExecutor? { get } } /// A protocol that allows customizing the execution of a test function (and @@ -73,12 +73,12 @@ public protocol Trait: Sendable { /// /// Types conforming to this protocol may be used in conjunction with a /// ``Trait``-conforming type by implementing the -/// ``Trait/customTestExecutor-1dwpt`` property, allowing custom traits to +/// ``Trait/testExecutor-714gp`` property, allowing custom traits to /// customize the execution of tests. Consolidating common set-up and tear-down /// logic for tests which have similar needs allows each test function to be /// more succinct with less repetitive boilerplate so it can focus on what makes /// it unique. -public protocol CustomTestExecuting: Sendable { +public protocol TestExecuting: Sendable { /// Execute a function for the specified test and/or test case. /// /// - Parameters: @@ -99,8 +99,8 @@ public protocol CustomTestExecuting: Sendable { /// When the testing library is preparing to run a test, it finds all traits /// applied to that test (including those inherited from containing suites) /// and asks each for the value of its - /// ``Trait/customTestExecutor-1dwpt`` property. It then calls this method on - /// all non-`nil` instances, giving each an opportunity to perform + /// ``Trait/testExecutor-714gp`` property. It then calls this method + /// on all non-`nil` instances, giving each an opportunity to perform /// arbitrary work before or after invoking `function`. /// /// This method should either invoke `function` once before returning or throw @@ -114,13 +114,13 @@ public protocol CustomTestExecuting: Sendable { func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws } -extension Trait where Self: CustomTestExecuting { - public var customTestExecutor: Self? { +extension Trait where Self: TestExecuting { + public var testExecutor: Self? { self } } -extension Never: CustomTestExecuting { +extension Never: TestExecuting { public func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws {} } @@ -153,8 +153,8 @@ extension Trait { } } -extension Trait where CustomTestExecutor == Never { - public var customTestExecutor: CustomTestExecutor? { +extension Trait where TestExecutor == Never { + public var testExecutor: TestExecutor? { nil } } diff --git a/Tests/TestingTests/Traits/CustomTestExecutingTraitTests.swift b/Tests/TestingTests/Traits/TestExecutingTraitTests.swift similarity index 91% rename from Tests/TestingTests/Traits/CustomTestExecutingTraitTests.swift rename to Tests/TestingTests/Traits/TestExecutingTraitTests.swift index fae72e8b..14b15afc 100644 --- a/Tests/TestingTests/Traits/CustomTestExecutingTraitTests.swift +++ b/Tests/TestingTests/Traits/TestExecutingTraitTests.swift @@ -10,8 +10,8 @@ @testable @_spi(Experimental) @_spi(ForToolsIntegrationOnly) import Testing -@Suite("CustomTestExecuting-conforming Trait Tests") -struct CustomTestExecutingTraitTests { +@Suite("TestExecuting-conforming Trait Tests") +struct TestExecutingTraitTests { @Test("Execute code before and after a non-parameterized test.") func executeCodeBeforeAndAfterNonParameterizedTest() async { // `expectedCount` is 2 because we run it both for the test and the test case @@ -65,7 +65,7 @@ struct CustomTestExecutingTraitTests { // MARK: - Fixtures -private struct CustomTrait: TestTrait, CustomTestExecuting { +private struct CustomTrait: TestTrait, TestExecuting { var before: Confirmation var after: Confirmation func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { @@ -77,7 +77,7 @@ private struct CustomTrait: TestTrait, CustomTestExecuting { } } -private struct CustomThrowingErrorTrait: TestTrait, CustomTestExecuting { +private struct CustomThrowingErrorTrait: TestTrait, TestExecuting { fileprivate struct CustomTraitError: Error {} func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { @@ -85,7 +85,7 @@ private struct CustomThrowingErrorTrait: TestTrait, CustomTestExecuting { } } -struct DoSomethingBeforeAndAfterTrait: SuiteTrait, TestTrait, CustomTestExecuting { +struct DoSomethingBeforeAndAfterTrait: SuiteTrait, TestTrait, TestExecuting { static let state = Locked(rawValue: 0) func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { From dae13984154fec283e043311bebdfcd9104f25a8 Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Thu, 3 Oct 2024 10:35:16 -0500 Subject: [PATCH 8/9] Revise proposed API to ensure custom behaviors are only performed once per scope, by default --- .../NNNN-custom-test-execution-traits.md | 127 +++++++++++++++--- Sources/Testing/Running/Runner.swift | 2 +- Sources/Testing/Testing.docc/Traits/Trait.md | 2 +- Sources/Testing/Traits/Trait.swift | 84 ++++++++---- .../Traits/TestExecutingTraitTests.swift | 92 ++++++++++++- 5 files changed, 254 insertions(+), 53 deletions(-) diff --git a/Documentation/Proposals/NNNN-custom-test-execution-traits.md b/Documentation/Proposals/NNNN-custom-test-execution-traits.md index 510c8ce2..a4b2a581 100644 --- a/Documentation/Proposals/NNNN-custom-test-execution-traits.md +++ b/Documentation/Proposals/NNNN-custom-test-execution-traits.md @@ -11,6 +11,9 @@ * **v1**: Initial pitch. * **v2**: Dropped 'Custom' prefix from the proposed API names (although kept the word in certain documentation passages where it clarified behavior). +* **v3**: Changed the `Trait` requirement from a property to a method which + accepts the test and/or test case, and modify its default implementations such + that custom behavior is either performed per-suite or per-test case by default. ## Introduction @@ -192,6 +195,42 @@ to mitigate these downsides it's important that there be some way to distinguish traits which customize test behavior. That way, the testing library can limit these scoped access calls to only the traits which require it. +### Avoiding unnecessary (re-)execution + +Traits can be applied to either test functions or suites, and traits applied to +suites can optionally support inheritance by implementing the `isRecursive` +property of the `SuiteTrait` protocol. When a trait is directly applied to a +test function, if the trait customizes the behavior of tests it's applied to, it +should be given the opportunity to perform its custom behavior once for every +invocation of that test function. In particular, if the test function is +parameterized and runs multiple times, then the trait applied to it should +perform its custom behavior once for every invocation. This should not be +surprising to users, since it's consistent with the behavior of `init` and +`deinit` for an instance `@Test` method. + +It may be useful for certain kinds of traits to perform custom logic once for +_all_ the invocations of a parameterized test. Although this should be possible, +we believe it shouldn't be the default since it could lead to work being +repeated multiple times needlessly, or unintentional state sharing across tests, +unless the trait is implemented carefully to avoid those problems. + +When a trait conforms to `SuiteTrait` and is applied to a suite, the question of +when its custom behavior (if any) should be performed is less obvious. Some +suite traits support inheritance and are recursively applied to all the test +functions they contain (including transitively, via sub-suites). Other suite +traits don't support inheritance, and only affect the specific suite they're +applied to. (It's also worth noting that a sub-suite _can_ have the same +non-recursive suite trait one of its ancestors has, as long as it's applied +explicitly.) + +As a general rule of thumb, we believe most traits will either want to perform +custom logic once for _all_ children or once for _each_ child, not both. +Therefore, when it comes to suite traits, the default behavior should depend on +whether it supports inheritance: a recursive suite trait should by default +perform custom logic before each test, and a non-recursive one per-suite. But +the APIs should be flexible enough to support both, for advanced traits which +need it. + ## Detailed design I propose the following new APIs: @@ -219,7 +258,7 @@ Below are the proposed interfaces: /// /// Types conforming to this protocol may be used in conjunction with a /// ``Trait``-conforming type by implementing the -/// ``Trait/testExecutor-714gp`` property, allowing custom traits to +/// ``Trait/executor(for:testCase:)-26qgm`` method, allowing custom traits to /// customize the execution of tests. Consolidating common set-up and tear-down /// logic for tests which have similar needs allows each test function to be /// more succinct with less repetitive boilerplate so it can focus on what makes @@ -244,8 +283,8 @@ public protocol TestExecuting: Sendable { /// /// When the testing library is preparing to run a test, it finds all traits /// applied to that test (including those inherited from containing suites) - /// and asks each for the value of its - /// ``Trait/testExecutor-714gp`` property. It then calls this method + /// and asks each for its test executor (if any) by calling + /// ``Trait/executor(for:testCase:)-26qgm``. It then calls this method /// on all non-`nil` instances, giving each an opportunity to perform /// arbitrary work before or after invoking `function`. /// @@ -265,37 +304,72 @@ public protocol Trait: Sendable { /// The type of the test executor for this trait. /// - /// The default type is `Never`, which cannot be instantiated. This means the - /// value of the ``testExecutor-714gp`` property for all traits with - /// the default custom executor type is `nil`, meaning such traits will not - /// perform any custom execution for the tests they're applied to. + /// The default type is `Never`, which cannot be instantiated. The + /// ``executor(for:testCase:)-26qgm`` method for any trait with this default + /// test executor type must return `nil`, meaning that trait will not perform + /// any custom behavior for the tests it's applied to. associatedtype TestExecutor: TestExecuting = Never - /// The test executor for this trait, if any. + /// Get this trait's executor for the specified test and/or test case, if any. + /// + /// - Parameters: + /// - test: The test for which an executor is being requested. + /// - testCase: The test case for which an executor is being requested, if + /// any. When `test` represents a suite, the value of this argument is + /// `nil`. + /// + /// - Returns: An value of ``Trait/TestExecutor`` which should be used to + /// customize the behavior of `test` and/or `testCase`, or `nil` if custom + /// behavior should not be performed. + /// + /// If this trait's type conforms to ``TestExecuting``, the default value + /// returned by this method depends on `test` and/or `testCase`: /// - /// If this trait's type conforms to ``TestExecuting``, the default value of - /// this property is `self` and this trait will be used to customize test - /// execution. This is the most straightforward way to implement a trait which - /// customizes the execution of tests. + /// - If `test` represents a suite, this trait must conform to ``SuiteTrait``. + /// If the value of this suite trait's ``SuiteTrait/isRecursive`` property + /// is `true`, then this method returns `nil`; otherwise, it returns `self`. + /// This means that by default, a suite trait will _either_ perform its + /// custom behavior once for the entire suite, or once per-test function it + /// contains. + /// - Otherwise `test` represents a test function. If `testCase` is `nil`, + /// this method returns `nil`; otherwise, it returns `self`. This means that + /// by default, a trait which is applied to or inherited by a test function + /// will perform its custom behavior once for each of that function's cases. /// - /// If the value of this property is an instance of a different type - /// conforming to ``TestExecuting``, that instance will be used to perform - /// test execution instead. + /// A trait may explicitly implement this method to further customize the + /// default behaviors above. For example, if a trait should perform custom + /// test behavior both once per-suite and once per-test function in that suite, + /// it may implement the method and return a non-`nil` executor under those + /// conditions. /// - /// The default value of this property is `nil` (with the default type - /// `Never?`), meaning that instances of this type will not perform any custom - /// test execution for tests they are applied to. - var testExecutor: TestExecutor? { get } + /// A trait may also implement this method and return `nil` if it determines + /// that it does not need to perform any custom behavior for a particular test + /// at runtime, even if the test has the trait applied. This can improve + /// performance and make diagnostics clearer by avoiding an unnecessary call + /// to ``TestExecuting/execute(_:for:testCase:)``. + /// + /// If this trait's type does not conform to ``TestExecuting`` and its + /// associated ``Trait/TestExecutor`` type is the default `Never`, then this + /// method returns `nil` by default. This means that instances of this type + /// will not perform any custom test execution for tests they are applied to. + func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? } extension Trait where Self: TestExecuting { - // Returns `self`. - public var testExecutor: Self? { + // Returns `nil` if `testCase` is `nil`, else `self`. + public func executor(for test: Test, testCase: Test.Case?) -> Self? +} + +extension SuiteTrait where Self: TestExecuting { + // If `test` is a suite, returns `nil` if `isRecursive` is `true`, else `self`. + // Otherwise, `test` is a function and this returns `nil` if `testCase` is + // `nil`, else `self`. + public func executor(for test: Test, testCase: Test.Case?) -> Self? } extension Trait where TestExecutor == Never { // Returns `nil`. - public var testExecutor: TestExecutor? { + public func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? } extension Never: TestExecuting {} @@ -387,6 +461,15 @@ ideal as it required a conditional `trait as? CustomExecutionTrait` downcast at runtime, in contrast to the simpler and more performant Optional property of the proposed API. +### API names + +We considered "run" as the base verb for the proposed new concept instead of +"execute", which would imply the names `TestRunning`, `TestRunner`, +`runner(for:testCase)`, and `run(_:for:testCase:)`. The word "run" is used in +many other contexts related to testing though, such as the `Runner` SPI type and +more casually to refer to a run which occurred of a test, in the past tense, so +overloading this term again may cause confusion. + ## Acknowledgments Thanks to [Dennis Weissmann](https://github.com/dennisweissmann) for originally diff --git a/Sources/Testing/Running/Runner.swift b/Sources/Testing/Running/Runner.swift index 530fba24..108f60e8 100644 --- a/Sources/Testing/Running/Runner.swift +++ b/Sources/Testing/Running/Runner.swift @@ -90,7 +90,7 @@ extension Runner { // and ultimately the first trait is the first one to be invoked. let executeAllTraits = step.test.traits.lazy .reversed() - .compactMap { $0.testExecutor } + .compactMap { $0.executor(for: step.test, testCase: testCase) } .map { $0.execute(_:for:testCase:) } .reduce(body) { executeAllTraits, testExecutor in { diff --git a/Sources/Testing/Testing.docc/Traits/Trait.md b/Sources/Testing/Testing.docc/Traits/Trait.md index 632c323d..8a24bd3f 100644 --- a/Sources/Testing/Testing.docc/Traits/Trait.md +++ b/Sources/Testing/Testing.docc/Traits/Trait.md @@ -49,5 +49,5 @@ See https://swift.org/CONTRIBUTORS.txt for Swift project authors ### Customizing the execution of tests - ``TestExecuting`` -- ``Trait/testExecutor-714gp`` +- ``Trait/executor(for:testCase:)-26qgm`` - ``Trait/TestExecutor`` diff --git a/Sources/Testing/Traits/Trait.swift b/Sources/Testing/Traits/Trait.swift index 3682c03a..fde71a74 100644 --- a/Sources/Testing/Traits/Trait.swift +++ b/Sources/Testing/Traits/Trait.swift @@ -44,27 +44,55 @@ public protocol Trait: Sendable { /// The type of the test executor for this trait. /// - /// The default type is `Never`, which cannot be instantiated. This means the - /// value of the ``testExecutor-714gp`` property for all traits with - /// the default custom executor type is `nil`, meaning such traits will not - /// perform any custom execution for the tests they're applied to. + /// The default type is `Never`, which cannot be instantiated. The + /// ``executor(for:testCase:)-26qgm`` method for any trait with this default + /// test executor type must return `nil`, meaning that trait will not perform + /// any custom behavior for the tests it's applied to. associatedtype TestExecutor: TestExecuting = Never - /// The test executor for this trait, if any. + /// Get this trait's executor for the specified test and/or test case, if any. /// - /// If this trait's type conforms to ``TestExecuting``, the default value of - /// this property is `self` and this trait will be used to customize test - /// execution. This is the most straightforward way to implement a trait which - /// customizes the execution of tests. - /// - /// If the value of this property is an instance of a different type - /// conforming to ``TestExecuting``, that instance will be used to perform - /// test execution instead. + /// - Parameters: + /// - test: The test for which an executor is being requested. + /// - testCase: The test case for which an executor is being requested, if + /// any. When `test` represents a suite, the value of this argument is + /// `nil`. /// - /// The default value of this property is `nil` (with the default type - /// `Never?`), meaning that instances of this type will not perform any custom - /// test execution for tests they are applied to. - var testExecutor: TestExecutor? { get } + /// - Returns: An value of ``Trait/TestExecutor`` which should be used to + /// customize the behavior of `test` and/or `testCase`, or `nil` if custom + /// behavior should not be performed. + /// + /// If this trait's type conforms to ``TestExecuting``, the default value + /// returned by this method depends on `test` and/or `testCase`: + /// + /// - If `test` represents a suite, this trait must conform to ``SuiteTrait``. + /// If the value of this suite trait's ``SuiteTrait/isRecursive`` property + /// is `true`, then this method returns `nil`; otherwise, it returns `self`. + /// This means that by default, a suite trait will _either_ perform its + /// custom behavior once for the entire suite, or once per-test function it + /// contains. + /// - Otherwise `test` represents a test function. If `testCase` is `nil`, + /// this method returns `nil`; otherwise, it returns `self`. This means that + /// by default, a trait which is applied to or inherited by a test function + /// will perform its custom behavior once for each of that function's cases. + /// + /// A trait may explicitly implement this method to further customize the + /// default behaviors above. For example, if a trait should perform custom + /// test behavior both once per-suite and once per-test function in that suite, + /// it may implement the method and return a non-`nil` executor under those + /// conditions. + /// + /// A trait may also implement this method and return `nil` if it determines + /// that it does not need to perform any custom behavior for a particular test + /// at runtime, even if the test has the trait applied. This can improve + /// performance and make diagnostics clearer by avoiding an unnecessary call + /// to ``TestExecuting/execute(_:for:testCase:)``. + /// + /// If this trait's type does not conform to ``TestExecuting`` and its + /// associated ``Trait/TestExecutor`` type is the default `Never`, then this + /// method returns `nil` by default. This means that instances of this type + /// will not perform any custom test execution for tests they are applied to. + func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? } /// A protocol that allows customizing the execution of a test function (and @@ -73,7 +101,7 @@ public protocol Trait: Sendable { /// /// Types conforming to this protocol may be used in conjunction with a /// ``Trait``-conforming type by implementing the -/// ``Trait/testExecutor-714gp`` property, allowing custom traits to +/// ``Trait/executor(for:testCase:)-26qgm`` method, allowing custom traits to /// customize the execution of tests. Consolidating common set-up and tear-down /// logic for tests which have similar needs allows each test function to be /// more succinct with less repetitive boilerplate so it can focus on what makes @@ -98,8 +126,8 @@ public protocol TestExecuting: Sendable { /// /// When the testing library is preparing to run a test, it finds all traits /// applied to that test (including those inherited from containing suites) - /// and asks each for the value of its - /// ``Trait/testExecutor-714gp`` property. It then calls this method + /// and asks each for its test executor (if any) by calling + /// ``Trait/executor(for:testCase:)-26qgm``. It then calls this method /// on all non-`nil` instances, giving each an opportunity to perform /// arbitrary work before or after invoking `function`. /// @@ -115,8 +143,18 @@ public protocol TestExecuting: Sendable { } extension Trait where Self: TestExecuting { - public var testExecutor: Self? { - self + public func executor(for test: Test, testCase: Test.Case?) -> Self? { + testCase == nil ? nil : self + } +} + +extension SuiteTrait where Self: TestExecuting { + public func executor(for test: Test, testCase: Test.Case?) -> Self? { + if test.isSuite { + isRecursive ? nil : self + } else { + testCase == nil ? nil : self + } } } @@ -154,7 +192,7 @@ extension Trait { } extension Trait where TestExecutor == Never { - public var testExecutor: TestExecutor? { + public func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? { nil } } diff --git a/Tests/TestingTests/Traits/TestExecutingTraitTests.swift b/Tests/TestingTests/Traits/TestExecutingTraitTests.swift index 14b15afc..7a6f72f6 100644 --- a/Tests/TestingTests/Traits/TestExecutingTraitTests.swift +++ b/Tests/TestingTests/Traits/TestExecutingTraitTests.swift @@ -14,9 +14,8 @@ struct TestExecutingTraitTests { @Test("Execute code before and after a non-parameterized test.") func executeCodeBeforeAndAfterNonParameterizedTest() async { - // `expectedCount` is 2 because we run it both for the test and the test case - await confirmation("Code was run before the test", expectedCount: 2) { before in - await confirmation("Code was run after the test", expectedCount: 2) { after in + await confirmation("Code was run before the test") { before in + await confirmation("Code was run after the test") { after in await Test(CustomTrait(before: before, after: after)) { // do nothing }.run() @@ -26,9 +25,9 @@ struct TestExecutingTraitTests { @Test("Execute code before and after a parameterized test.") func executeCodeBeforeAndAfterParameterizedTest() async { - // `expectedCount` is 3 because we run it both for the test and each test case - await confirmation("Code was run before the test", expectedCount: 3) { before in - await confirmation("Code was run after the test", expectedCount: 3) { after in + // `expectedCount` is 2 because we run it for each test case + await confirmation("Code was run before the test", expectedCount: 2) { before in + await confirmation("Code was run after the test", expectedCount: 2) { after in await Test(CustomTrait(before: before, after: after), arguments: ["Hello", "World"]) { _ in // do nothing }.run() @@ -61,6 +60,44 @@ struct TestExecutingTraitTests { func teardownOccursAtEnd() async throws { await runTest(for: TestsWithCustomTraitWithStrongOrdering.self, configuration: .init()) } + + struct ExecutionControl { + @Test("Trait applied directly to function is executed once") + func traitAppliedToFunction() async { + let counter = Locked(rawValue: 0) + await DefaultExecutionTrait.$counter.withValue(counter) { + await Test(DefaultExecutionTrait()) {}.run() + } + #expect(counter.rawValue == 1) + } + + @Test("Non-recursive suite trait with default custom test executor implementation") + func nonRecursiveSuiteTrait() async { + let counter = Locked(rawValue: 0) + await DefaultExecutionTrait.$counter.withValue(counter) { + await runTest(for: SuiteWithNonRecursiveDefaultExecutionTrait.self) + } + #expect(counter.rawValue == 1) + } + + @Test("Recursive suite trait with default custom test executor implementation") + func recursiveSuiteTrait() async { + let counter = Locked(rawValue: 0) + await DefaultExecutionTrait.$counter.withValue(counter) { + await runTest(for: SuiteWithRecursiveDefaultExecutionTrait.self) + } + #expect(counter.rawValue == 1) + } + + @Test("Recursive, all-inclusive suite trait") + func recursiveAllInclusiveSuiteTrait() async { + let counter = Locked(rawValue: 0) + await AllInclusiveExecutionTrait.$counter.withValue(counter) { + await runTest(for: SuiteWithAllInclusiveExecutionTrait.self) + } + #expect(counter.rawValue == 3) + } + } } // MARK: - Fixtures @@ -102,3 +139,46 @@ struct TestsWithCustomTraitWithStrongOrdering { #expect(DoSomethingBeforeAndAfterTrait.state.increment() == 2) } } + +private struct DefaultExecutionTrait: SuiteTrait, TestTrait, TestExecuting { + @TaskLocal static var counter: Locked? + var isRecursive: Bool = false + + func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + Self.counter!.increment() + try await function() + } +} + +@Suite(.hidden, DefaultExecutionTrait()) +private struct SuiteWithNonRecursiveDefaultExecutionTrait { + @Test func f() {} +} + +@Suite(.hidden, DefaultExecutionTrait(isRecursive: true)) +private struct SuiteWithRecursiveDefaultExecutionTrait { + @Test func f() {} +} + +private struct AllInclusiveExecutionTrait: SuiteTrait, TestTrait, TestExecuting { + @TaskLocal static var counter: Locked? + + var isRecursive: Bool { + true + } + + func executor(for test: Test, testCase: Test.Case?) -> AllInclusiveExecutionTrait? { + // Unconditionally returning self makes this trait "all inclusive". + self + } + + func execute(_ function: () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { + Self.counter!.increment() + try await function() + } +} + +@Suite(.hidden, AllInclusiveExecutionTrait()) +private struct SuiteWithAllInclusiveExecutionTrait { + @Test func f() {} +} From fb4f00e8c10d586e367946a7e169ef673a7d4ddb Mon Sep 17 00:00:00 2001 From: Stuart Montgomery Date: Tue, 8 Oct 2024 14:53:31 -0500 Subject: [PATCH 9/9] Add a few more stray details to the proposal --- .../Proposals/NNNN-custom-test-execution-traits.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Documentation/Proposals/NNNN-custom-test-execution-traits.md b/Documentation/Proposals/NNNN-custom-test-execution-traits.md index a4b2a581..a936e5f0 100644 --- a/Documentation/Proposals/NNNN-custom-test-execution-traits.md +++ b/Documentation/Proposals/NNNN-custom-test-execution-traits.md @@ -4,7 +4,7 @@ * Authors: [Stuart Montgomery](https://github.com/stmontgomery) * Status: **Awaiting review** * Implementation: [swiftlang/swift-testing#733](https://github.com/swiftlang/swift-testing/pull/733), [swiftlang/swift-testing#86](https://github.com/swiftlang/swift-testing/pull/86) -* Review: ([pitch](https://forums.swift.org/...)) +* Review: ([pitch](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055)) ### Revision history @@ -404,6 +404,10 @@ extension Trait where Self == MockAPICredentialsTrait { The proposed APIs are purely additive. +This proposal will replace the existing `CustomExecutionTrait` SPI, and after +further refactoring we anticipate it will obsolete the need for the +`SPIAwareTrait` SPI as well. + ## Integration with supporting tools Although some built-in traits are relevant to supporting tools (such as