[Feature]: [FRSM-14] Does the software come with test cases to demonstrate it is working?

[karacolada]

D5.2 p20+p30

Detailed Description

The provision of test cases improves confidence in the software.

Domain-agnostic comments

Most programming languages have commonly associated test frameworks. The specific definition of what constitutes adequate testing is often defined by community norms. It is hard to check the relevance / correctness of this information, but it is possible to automatically check for existence.

CESSDA comments

See Software Maturity Levels (SML) for: CMA9 - Verification and Testing and CMA7 Standards Compliance. CESSDA periodically runs the SQAaaS tool against its publicly accessible repositories and displays the results via a badge in the README file.

Context

R1: Software is described with a plurality of accurate and relevant attributes.

Possible Implementation

domain-agnostic

requirements	Software source code
method	Check for the presence of automated tests
essential	Tests and data are provided to check that the software is operating as expected
important	Automated unit and system tests are provided
useful	Code coverage / test coverage is reported

CESSDA

requirements	Software source code
method	Check the README file.
essential	The README file includes badges that link to a comprehensive code quality assessment tool (SonarQube) and automated build tool (Jenkins).
important	CMA9-SML5 - Demonstrable usability: A production system has been tested and validated through successful use of the application. CMA7-SML5 - Demonstrable usability: Compliance with open or internationally recognised standards for the software and software development process, is evident and documented, and verified through testing of all components. Ideally independent verification is documented through regular testing and certification from an independent group.
useful	The README file badges indicate the status of the tests and other code quality metrics. The repository contains a subdirectory containing code for the test cases that are run automatically.

[karacolada]

Skeleton evaluator:

fuji/fuji_server/evaluators/fair_evaluator_test_cases.py

Line 10 in 1778e73

class FAIREvaluatorTestCases(FAIREvaluator):