AresIndexer::augmentConceptFiles fails with "! object 'CONCEPT_ID' not found"

[lav-patel]

Code which is inpired from ares docs

# run achilles
Achilles::achilles(cdmVersion = cdmVersion,
    connectionDetails = connectionDetails,
    cdmDatabaseSchema = cdmDatabaseSchema,
    resultsDatabaseSchema = resultsDatabaseSchema,
    #numThreads=numThreads,
    #sqlOnly = sqlOnly,
    #createIndices = createIndices
)
# obtain the data source release key (naming convention for folder structures)
releaseKey <- AresIndexer::getSourceReleaseKey(connectionDetails, cdmDatabaseSchema)
datasourceReleaseOutputFolder <- file.path(aresDataRoot, releaseKey)

# run data quality dashboard and output results to data source release folder in ares data folder
dqResults <- DataQualityDashboard::executeDqChecks(
    connectionDetails = connectionDetails,
    cdmDatabaseSchema = cdmDatabaseSchema,
    resultsDatabaseSchema = resultsDatabaseSchema,
    vocabDatabaseSchema = cdmDatabaseSchema,
    cdmVersion = cdmVersion,
    cdmSourceName = cdmSourceName,
    outputFile = "dq-result.json",
    outputFolder = datasourceReleaseOutputFolder
    #numThreads = numThreads,
    #sqlOnly = sqlOnly,
    # verboseMode = verboseMode,
    # writeToTable = writeToTable,
    # checkLevels = checkLevels,
    # checkNames = checkNames
)

# inspect logs
#ParallelLogger::launchLogViewer(logFileName = file.path(outputFolder, 
#                                                      sprintf("log_DqDashboard_%s.txt", cdmSourceName)))

# export the achilles results to the ares folder
Achilles::exportAO(
    connectionDetails = connectionDetails,
    cdmDatabaseSchema = cdmDatabaseSchema,
    resultsDatabaseSchema = resultsDatabaseSchema,
    vocabDatabaseSchema = cdmDatabaseSchema,
    outputPath = aresDataRoot
)

# perform temporal characterization
outputFile <- file.path(datasourceReleaseOutputFolder, "temporal-characterization.csv")
Achilles::performTemporalCharacterization(
    connectionDetails = connectionDetails,
    cdmDatabaseSchema = cdmDatabaseSchema,
    resultsDatabaseSchema = resultsDatabaseSchema,
    outputFile = outputFile
)

# augment concept files with temporal characterization data
AresIndexer::augmentConceptFiles(releaseFolder = file.path(aresDataRoot, releaseKey))

The error does not mention which cdm_table_name it is referring to:

Error in `count()`:
ℹ In argument: `CONCEPT_ID`.
Caused by error:
! object 'CONCEPT_ID' not found
Run `rlang::last_error()` to see where the error occurred.
> rlang::last_error()
<error/dplyr:::mutate_error>
Error in `count()`:
ℹ In argument: `CONCEPT_ID`.
Caused by error:
! object 'CONCEPT_ID' not found
---
Backtrace:
  1. AresIndexer::augmentConceptFiles(...)
  4. dplyr:::count.data.frame(., CONCEPT_ID, tolower(CDM_TABLE_NAME))
  6. dplyr:::group_by.data.frame(x, ..., .add = TRUE, .drop = .drop)
  7. dplyr::group_by_prepare(.data, ..., .add = .add, error_call = current_env())
  8. dplyr:::add_computed_columns(.data, new_groups, error_call = error_call)
  9. dplyr:::mutate_cols(...)
 11. dplyr:::mutate_col(dots[[i]], data, mask, new_columns)
 12. mask$eval_all_mutate(quo)
 13. dplyr (local) eval()
Run `rlang::last_trace()` to see the full context.

can Someone help? What is the reason for it failure?

[alondhe]

Looks like the DQD JSON file uses camelcase for the column names, and AresIndexer is set up to rely on upper case snake case.

[alondhe]

DataQualityDashboard's new convertJsonResultsFileCase function can help convert results files from camel to snake (and vice versa):

DataQualityDashboard::convertJsonResultsFileCase(jsonFilePath = "~/Downloads/dq-result.json",
                                                 writeToFile = TRUE,
                                                 outputFolder = "~/Downloads",
                                                 targetCase = "snake")

[fdefalco]

I'll be pushing an update to AresIndexer that switches to the new DQD naming conventions before symposium. I'll be pushing an update to ARES that accounts for this update at the same time.