Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Look into and reconsider CTS tests with high fail rate across adapters. #2514

Open
aarongreig opened this issue Jan 2, 2025 · 1 comment
Assignees
Labels
conformance Conformance test suite issues.
Milestone

Comments

@aarongreig
Copy link
Contributor

aarongreig commented Jan 2, 2025

We've got a few CTS tests that only pass on one or two adapters. The behaviour these tests are enforcing isn't being missed upstream in the backends that are failing, which I think is a sign that either: these tests should have some skipping mechanism, we need a spec adjustment, or in the cases where only one adapter passes we could consider making them adapter specific tests.

These are the ones I noticed while working on the device parameterization PR:

  • urEventSetCallbackTest.ValidateParameters has a match entry for every adapter (although the opencl one may be config specific) (Wrap urEventSetCallback when ran through loader #2527 and maybe look into other adapters)
  • urEventGetProfilingInfo.Success - UR_PROFILING_INFO_COMMAND_COMPLETE fails on all adapters apart from CL, maybe just needs to be optional (Make profiling info optional and update tests #2533 )
  • urProgramSetSpecializationConstants InvalidValueSize, InvalidValueId and InvalidValuePtr have match entries for all adapters apart form CL - likely there's a missing skip for these one because cuda and hip don't support the entry point at all (Added DEVICE_INFO_PROGRAM_SET_SPECIALIZATION_CONSTANTS #2539 )
  • urUSM*Alloc InvalidUSMSize tests all fail for cuda, hip and native cpu - might simply not be possible to check this on these backends
  • urDeviceGetGlobalTimestampTest SuccessSynchronizedTime fails on every adapter, only passes on CL CPU ( Be less strict with timer test #2547 )
  • urEnqueueKernelLaunchKernelSubGroupTest fails on every adapter but CL
  • urEnqueueKernelLaunchTest.InvalidKernelArgs fails on every adapter but CL
  • urMemBufferCreateWithNativeHandleTest has a lot of match entries, all native handle entry points should be optional so almost certainly just missing a skip
@aarongreig aarongreig added the conformance Conformance test suite issues. label Jan 2, 2025
@aarongreig aarongreig added this to the Backlog milestone Jan 2, 2025
@RossBrunton RossBrunton self-assigned this Jan 7, 2025
@aarongreig
Copy link
Contributor Author

add to the list urKernelGetSubGroupInfoFixedSubGroupSizeTest.CompileNumSubGroups which seems to fail on every adapter (maybe this is something that only works with a specific platform? some of the other tests like this one only work on opencl igpu + fpga emulator)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
conformance Conformance test suite issues.
Projects
None yet
Development

No branches or pull requests

2 participants