Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show more queries that are too long. #4009

Merged
merged 1 commit into from
Oct 21, 2024

Conversation

greggman
Copy link
Contributor

@greggman greggman commented Oct 21, 2024

2 issues this solves

  1. Show all the failures instead of only the first failure

    This was an issue because I'd fix the one failure only to later find there were worse cases. Just the worst case is also not enough.

  2. Show a representation of each test

    The issue where is whatever is printing the Error message truncates the list so when I printed them all and there were more than about 20 because of so many permutations it would cut off the list, defeating the point of (1) above.

    So, opted to show one query per test of each length.

Example:

Before

Error: Generated test variant would produce too-long -actual.txt filename. Possible solutions:
 - Reduce the length of the parts of the test query
 - Reduce the parameterization of the test
 - Make the test function faster and regenerate the listing_meta entry
 - Reduce the specificity of test expectations (if you're using them) 
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-8x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*

After

Error: Generated test variant would produce too-long -actual.txt filename. Possible solutions:
 - Reduce the length of the parts of the test query
 - Reduce the parameterization of the test
 - Make the test function faster and regenerate the listing_meta entry
 - Reduce the specificity of test expectations (if you're using them)
|<------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>|
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-8x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";* 
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-12x12-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";* 
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-10x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";* 
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="fragment";format="astc-12x12-unorm-srgb";dim="cube";filt="nearest";modeU="m";modeV="m";modeW="m";* 
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-12x12-unorm-srgb";dim="cube";filt="nearest";modeU="m";modeV="m";modeW="m";* 
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="linear";modeU="m";modeV="m";offset=false;* 
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="fragment";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false;* 
webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false;* 
webgpu:shader,execution,expression,call,builtin,textureSampleLevel:sampled_array_2d_coords:stage="fragment";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false 
webgpu:shader,execution,expression,call,builtin,textureSampleLevel:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false

2 issues this solves

1. Show all the failures instead of only the first failure

   This was an issue because I'd fix the one failure only to later
   find there were worse cases. Just the worst case is also not
   enough.

2. Show a representation of each test

   The issue where is whatever is printing the `Error` message
   truncates the list so when I printed them all and there were
   more than about 20 because of so many permutations it would
   cut off the list, defeating the point of (1) above.

   So, opted to show one query per test of each length.

Example:

            Error: Generated test variant would produce too-long -actual.txt filename. Possible solutions:
              - Reduce the length of the parts of the test query
              - Reduce the parameterization of the test
              - Make the test function faster and regenerate the listing_meta entry
              - Reduce the specificity of test expectations (if you're using them)
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-8x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*

            Error: Generated test variant would produce too-long -actual.txt filename. Possible solutions:
              - Reduce the length of the parts of the test query
              - Reduce the parameterization of the test
              - Make the test function faster and regenerate the listing_meta entry
              - Reduce the specificity of test expectations (if you're using them)
            |<------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>|
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-8x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-12x12-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-10x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="fragment";format="astc-12x12-unorm-srgb";dim="cube";filt="nearest";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-12x12-unorm-srgb";dim="cube";filt="nearest";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="linear";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="fragment";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleLevel:sampled_array_2d_coords:stage="fragment";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false
            webgpu:shader,execution,expression,call,builtin,textureSampleLevel:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false
Copy link
Collaborator

@kainino0x kainino0x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much nicer!

@kainino0x kainino0x merged commit f45a9e0 into gpuweb:main Oct 21, 2024
1 check passed
teoxoy pushed a commit to mozilla/gpuweb-cts that referenced this pull request Oct 25, 2024
2 issues this solves

1. Show all the failures instead of only the first failure

   This was an issue because I'd fix the one failure only to later
   find there were worse cases. Just the worst case is also not
   enough.

2. Show a representation of each test

   The issue where is whatever is printing the `Error` message
   truncates the list so when I printed them all and there were
   more than about 20 because of so many permutations it would
   cut off the list, defeating the point of (1) above.

   So, opted to show one query per test of each length.

Example:

            Error: Generated test variant would produce too-long -actual.txt filename. Possible solutions:
              - Reduce the length of the parts of the test query
              - Reduce the parameterization of the test
              - Make the test function faster and regenerate the listing_meta entry
              - Reduce the specificity of test expectations (if you're using them)
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-8x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*

            Error: Generated test variant would produce too-long -actual.txt filename. Possible solutions:
              - Reduce the length of the parts of the test query
              - Reduce the parameterization of the test
              - Make the test function faster and regenerate the listing_meta entry
              - Reduce the specificity of test expectations (if you're using them)
            |<------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>|
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-8x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-12x12-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-10x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="fragment";format="astc-12x12-unorm-srgb";dim="cube";filt="nearest";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-12x12-unorm-srgb";dim="cube";filt="nearest";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="linear";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="fragment";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleLevel:sampled_array_2d_coords:stage="fragment";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false
            webgpu:shader,execution,expression,call,builtin,textureSampleLevel:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false
teoxoy pushed a commit to mozilla/gpuweb-cts that referenced this pull request Oct 25, 2024
2 issues this solves

1. Show all the failures instead of only the first failure

   This was an issue because I'd fix the one failure only to later
   find there were worse cases. Just the worst case is also not
   enough.

2. Show a representation of each test

   The issue where is whatever is printing the `Error` message
   truncates the list so when I printed them all and there were
   more than about 20 because of so many permutations it would
   cut off the list, defeating the point of (1) above.

   So, opted to show one query per test of each length.

Example:

            Error: Generated test variant would produce too-long -actual.txt filename. Possible solutions:
              - Reduce the length of the parts of the test query
              - Reduce the parameterization of the test
              - Make the test function faster and regenerate the listing_meta entry
              - Reduce the specificity of test expectations (if you're using them)
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-8x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*

            Error: Generated test variant would produce too-long -actual.txt filename. Possible solutions:
              - Reduce the length of the parts of the test query
              - Reduce the parameterization of the test
              - Make the test function faster and regenerate the listing_meta entry
              - Reduce the specificity of test expectations (if you're using them)
            |<------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>|
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-8x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-12x12-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-10x8-unorm-srgb";dim="cube";filt="linear";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="fragment";format="astc-12x12-unorm-srgb";dim="cube";filt="nearest";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_3d_coords:stage="compute";format="astc-12x12-unorm-srgb";dim="cube";filt="nearest";modeU="m";modeV="m";modeW="m";*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="linear";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="fragment";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleGrad:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false;*
            webgpu:shader,execution,expression,call,builtin,textureSampleLevel:sampled_array_2d_coords:stage="fragment";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false
            webgpu:shader,execution,expression,call,builtin,textureSampleLevel:sampled_array_2d_coords:stage="compute";format="astc-12x12-unorm-srgb";filt="nearest";modeU="m";modeV="m";offset=false
@greggman greggman deleted the show-more-too-long-queries branch November 1, 2024 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants