Catastrophic failures when upgraded to v3

[bbascarevic]

I'm on xUnit v2.9.2 and I upgraded xunit.runner.visualstudio from 2.8.2 to 3.0.0. I'm now getting these messages when running dotnet test from the console:

[xUnit.net 00:00:00.11]     [FATAL ERROR] System.InvalidOperationException
[xUnit.net 00:00:00.11]       System.InvalidOperationException : Key '586d9f8cc375d7e88d0d36659e3d796fc5a0dcd0416faab7d08903c0ce5372d6' already exists in the message metadata cache.
[xUnit.net 00:00:00.11]       Old item: TestCollectionStarting("586d9f8cc375d7e88d0d36659e3d796fc5a0dcd0416faab7d08903c0ce5372d6") name="" class=null
[xUnit.net 00:00:00.11]       New item: TestCollectionStarting("586d9f8cc375d7e88d0d36659e3d796fc5a0dcd0416faab7d08903c0ce5372d6") name="" class=null
[xUnit.net 00:00:00.11] Catastrophic failure: System.InvalidOperationException : Key '586d9f8cc375d7e88d0d36659e3d796fc5a0dcd0416faab7d08903c0ce5372d6' already exists in the message metadata cache.
Old item: TestCollectionStarting("586d9f8cc375d7e88d0d36659e3d796fc5a0dcd0416faab7d08903c0ce5372d6") name="" class=null
New item: TestCollectionStarting("586d9f8cc375d7e88d0d36659e3d796fc5a0dcd0416faab7d08903c0ce5372d6") name="" class=null

The same errors are appearing for TestCollectionStarting and TestClassStarting. All tests complete normally and report no failures despite these messages.

Is the v3 runner supposed to be used only with xUnit v3?
What does the error mean? I'm using my own discoverer, executor and runner but I don't know what is this "message metadata cache". Where do I start looking?

[bbascarevic]

One more thing, my discoverer really is reporting multiple IXunitTestCases for the same test class, however each reported instance has different TestMethodArguments, like theories do, and each has a different UniqueID.

[bbascarevic]

It seems that the issue is in Xunit2MessageAdapter.UniqueIDForTestCollection:

private static string UniqueIDForTestCollection(string assemblyUniqueID, Xunit.Abstractions.ITestCollection testCollection)
{
  return UniqueIDGenerator.ForTestCollection(assemblyUniqueID, testCollection.DisplayName, testCollection.CollectionDefinition?.Name);
}

The implementation uses the collection's DisplayName for deriving a unique ID, not the ITestCollection.UniqueID despite its definition:

/// <summary>
/// Gets the test collection ID. Test collection equality is determined by comparing IDs.
/// </summary>
Guid UniqueID { get; }

My collections had unique UniqueIds but not their display names (null).

So in short, collections need to have unique DisplayName values and their UniqueID seems to be ignored. Making collection display names unique made all the other catastrophic failures go away (TestClassStarting, etc.).

[bradwilson]

Yes, there is an unstated part of the contract for test collection display names that they are unique, due to the way [Collection] works. Simulating it with unexpected data isn't something that we support. This is true whether the project is v2 or v3 (as the v2 adapter is using the same logic that v3 uses).

[bradwilson]

Closing as "by design".