analyzers/checks to generate metrics based on certain dimensions/group by columns

[obaidcuet]

Hello,

Does deequ has a feature to run analyzers or checks to generate metrics based on certain dimensions/group by columns?

Like how we do in SQL like below:

select count(id), count(distinct id) 
from mytable
group by country, state;

In deequ, I am expecting something like below:

val analysisResult: AnalyzerContext = { AnalysisRunner
  // data to run the analysis on
  .onData(df)
  // domensions/columns to group to generate metrics for each group insted of full table
  .groupingColumns(Seq("country", "state")) 
  // define analyzers that compute metrics
  .addAnalyzer(Size())
  .addAnalyzer(Completeness("id"))
  .addAnalyzer(Distinctness("id")) 
  .addAnalyzer(Compliance("employee_name : proper size", "length(trim(employee_name)) > 1")) 
  // compute metrics
  .run()
}

Sorry if I missed something. I checked all the documentation, couldn't able to find an example.

-Obaid

[obaidcuet]

Hi,

I could see a work in progress #384

Another similar issue #381. From here I can see not possible using current API.

There is some code that can be used as a workaround, but will be hard to manage for high number of grouped values:

deequ/src/test/scala/com/amazon/deequ/analyzers/StateAggregationIntegrationTest.scala

Line 62 in 4e68020

val completenessStateNA = completeness.computeStateFrom(partitionNA)

Are there any other references?

-Obaid