-
Notifications
You must be signed in to change notification settings - Fork 0
Testing System
Application Programming Interface (API) -- A set of functions that allow creation of applications that access the library components features.
Adaptive Software Quality Management -- An iterative process where software quality is observed, deficiencies are identified and corrected, and software quality is improved over time.
Continuous Integration (CI) -- A software development practice where code is automatically built and tested continuously throughout the development process.
Component Testing -- Testing performed on individual software components prior to being integrated with other components.
Exploratory Testing -- Studying code and simultaneously reasoning about its design and operation while developing test cases to illustrate deficiencies.
Regression Testing -- Testing designed to confirm that the most recent code changes do not adversely impact existing software functionality.
Software Lifecycle -- A phased repeating cycle where software is designed, developed, released, in production, improved, and eventually retired.
Software Quality -- The degree to which the software meets requirements and is fit for its intended purpose.
Unit Testing -- Fine grained testing performed at the software unit level. A unit can be thought of as a small testable piece of source code, usually an individual function.
EPANET is a tool for the simulation of hydraulics and water quality in drinking water distribution systems. EPANET is widely used to plan, design, analyze, and regulate WDNs. It is also an important component of water security tools and has been adopted by third party commercial vendors as the computational core of their derivative products. It is of the utmost importance that EPANET is well engineered software that is fully capable of supporting these important stakeholder activities.
ORD developed EPANET in the 1990’s out of a need to improve understanding of water quality processes that occur in water distribution networks. Major work on EPANET ceased in the early 2000’s. It last received a minor update in 2008. Over this period of time EPANET has become an industry standard for hydraulic and water quality simulation with significant commercial value. Major water infrastructure planning and management decisions are made using EPANET on a daily basis in countries around the world. It must operate dependably and supply the correct numerical results in production. Therefore, software quality assurance is critically important to EPANET development activities.
At nearly 30 years old, EPANET is a mature application. Historically, EPANET has been used by engineers to perform fire flow calculations for new subdivisions, design master plans for asset management, and gain insights into their systems operation. Today, EPANET is evolving from an offline planning tool to a continuous online operational tool. This evolution demands a higher degree or reliability going forward. Despite all the modern advances in computing we enjoy, however, software development is still an expensive and error prone activity. Guaranteeing 100% software reliability is simply not possible. Quality assurance is an important aspect of software project management and can be used to target an appropriate level of reliability considering its intended use.
The Agency has invested heavily in the development of EPANET software and makes large investments in various research software prototypes in an effort to deliver value to ORD's customers. Unfortunately, these efforts have met with mixed success. The quality of EPANET software, however, has always been adequate for its intended purpose. Historically, EPANET quality control has been ad hoc and occurred in private. Because EPANET development was closed source and performed by one individual, quality assurance SOPs were informal, undocumented, and not communicated. The team developing EPANET is growing to encompass multiple organizations and individuals both inside and outside the Agency. Software development techniques and the development team itself are evolving. It is therefore necessary for us to consider how EPANET's software quality assurance processes should evolve to best meet the demands of the project and stakeholder's needs.
The EPANET Project Team practices Adaptive Software Quality Management. The central idea is that project management actions are adaptive and based in part on the current state of software quality and the desired project objectives. Testing is used as a metric for software quality and thus feeds into project management decisions. Testing is integral to preliminary algorithmic development, software development and maintenance, and release management activities within our quality management plan.
The highest level of efficacy can be achieved when testing is integrated early within the software lifecycle. For example, by emphasizing testing in preliminary development, deficiencies to be identified and corrected early when they are easiest to identify and the least expensive to fix. As a software application matures, new features are developed by adding to or modifying the existing source code base. During this phase of the lifecycle testing can be used to detect and prevent regressions in existing application functionality during feature development and software maintenance activities. Finally, when an application is ready for release to external users, testing can be used to certify that it meets requirements and is indeed ready for use.
The overarching goals of our quality management plan can be adapted and applied specifically to testing activities. Stated precisely, our testing activities should be objective, thorough, consistent, and transparent.
-
Objective: Testing assumptions should be explicitly stated and adhered to
-
Thorough: Testing should be both intensive and extensive in scope to provide a meaningful indicator of software quality
-
Consistent: Testing methods should be consistently applied and repeatable
-
Transparent: Testing methods and results should be accessible by all interested stakeholders
Testing is an essential element of our quality assurance and project management plans.
As stated previously, the main objectives of our current testing activities are to insure correct operation of algorithms under development, flag regressions during feature development and maintenance activities, and when delivering software to certify that it is ready for release.
The EPANET project contains several modular components. The most significant of which is the hydraulic and water quality solver itself. Going forward we will refer to this component as the epanet-solver or "solver" for short. The solver has been written as a software library to facilitate use by other applications. For example, the solver library is used by the US EPA's EPANET UI application. It is also used by other ORD research projects such as WNTR, WIST, and TEVA SPOT. And by third party software vendors in their derivative software products.
The solver library's API has been expanded significantly in the latest release -- going from 50 and 120 functions respectively and consists of 5400 lines of code. The API allows developers to access the solver data model, build models programmatically, control hydraulic and water quality solver execution, modify model element parameters, access simulation results, and much more. With complex functionality spread over hundreds of functions and thousands of lines of code testing is critical. We employ component testing to insure that each API function call operates correctly in isolation and under realistic usage scenarios. We have developed a testing framework to write, build, and execute our component tests. Scripts have been written to make it easy to build and execute component tests. The resulting framework, however, is only as useful as the tests that are written. Developer participation is therefore critically important.
The main purpose of the solver is to perform hydraulic and water quality simulations. At a basic level is uses numerical methods to solve systems of partial differential equations that describe the physics of water movement and basic water chemistry in pipe networks. In this respect, it is a prototypical example of a scientific and engineering software application. From a software quality assurance perspective, these types of applications are particularly challenging to develop and maintain. These applications by their nature are abstract and highly complex making the source code difficult to understand. The applications must work for each user's unique Water Distribution Network (WDN) but by necessity is written to solve the general case. The source code for numerical methods can be highly sensitive to seemingly insignificant changes.
To address some of these challenges we employ regression testing to run a suite of WDN examples and scrutinize the results. The suite currently contains approximately 70 examples. These examples taken together they exercise a broad set of application features. The simulation output for each example is compared to a known "good" result called a benchmark. If any change in results gets detected, the developer is alerted and should determine if it is nominal or indicative of a potential defect in the software. We have written custom components for an Open Source regression testing framework, command scripts, and SOPs to make running the tests, and maintaining the examples and benchmarks as straight forward as possible. The regression testing framework performs many of these tasks automatically, however, developer participation and judgement remain essential.
Testing is essential to insuring software quality. How frequently tests ought to be run, however, remains an open question. The EPANET project has adopted a software development practice called "Continuous Integration" (CI). The idea behind it is to automate the build and testing workflows so they can be triggered anytime a change to the project occurs. In the course of routine development this means the software is built and tested several times a day. This may seem extreme, however, because the workflows have been fully automated the majority of the work is performed by the computer locally or on a remote server out on the cloud. What CI does is allow us to maintain the project's source code in a known state of quality at all times. This is simple yet powerful idea that is extremely useful to the overall management of the project.
Preparing a software release involves finalizing changes made to the source code, fixing defects, testing, documentation, and installation packaging. The purpose of release testing is to certify that the software is ready for production. We use a release test suite of approximately 270 EPANET examples that have been contributed by users over the history of the project. The regression test framework described previously is used to run the release suite and monitor results for failures, crashes, and other anomalous behavior. Because the release suite is separate from the regression suite and has been contributed by users, running it indicates how the software will perform in production. From our release testing results we estimate that the EPANET v2.2.0 solver can successfully run 99% of user's networks. Our release testing procedures are new and evolving. We will continue to apply adaptive management techniques and make adjustments in order to achieve our software quality objectives.
In the sections that follow each major component associated with EPANET testing framework will be described.
Component and unit tests are source files that get compiled into individual executables that automatically perform many rigorous checks on the EPANET Toolkit and Output libraries. They require Boost libraries to build and run. The component tests check the functionality of the EPANET Toolkit and Output APIs. There are also a small number of unit tests that target individual software objects that will be expanded going forward. The project is configured to register tests with a test runner as they are built to make executing them more convenient.
Both the component and unit tests have been written using the Boost Unit Testing Framework and other Boost libraries -- including Boost System, Thread, and File System -- to perform various checks. Boost libraries are high quality, freely available, open source, and fully documented.
Boost Download: https://www.boost.org/users/download/
Boost Documentation: https://www.boost.org/doc/libs/
Component and unit tests can be found in the epanet/tests folder along with any data they require to run. As is standard practice, the tests folder is arranged to mimic the hierarchy used to organize the source code to ease navigation. The test files are written in C++11 and prefixed with the word "test" followed by the name of the source file or other designation of the organizational unit under test.
The tests/outfile folder contains the component tests for the epanet-output library. The tests/shared folder contains the unit tests for individual software objects that will eventually be shared between the epanet-output and epanet-solver libraries. The tests/solver folder contains the component tests for the Toolkit API. There is one test file for each category of Toolkit API functions. Each file is organized as a test suite containing individual test cases. The build system is configured to combine Toolkit test suites into one test executable. Linking tests with the Boost Unit Test Framework is expensive. This strategy was adopted to minimize build time. The tests folder also contains test_net_builder.cpp and test_reent.cpp for testing the Toolkit API model element creation functions and thread reentrancy feature respectively.
The CMake build system for the EPANET project has been configured to be modular. Each individual test executable is a target with its own build configuration file with the exception of the net builder and reentrancy tests whos build configuration is found in tests/CMakeLists.txt. The build system has been configured with a build option for building tests. When it is selected (-DBUILD_TESTS=ON) the required Boost libraries are found (Boost v1.67.0), and the test executables are built and registered with the CTest test runner. The default value for the test build option is off (-DBUILD_TESTS=OFF).
The make.cmd helper script has been written to perform these build steps and execute the tests locally using the test runner. To run it requires that CMake, Visual Studio Build Tools, and Boost are installed. Once the development environment has been configured the following simple command builds the project's component and unit tests in Debug configuration and executes them.
\> cd epanet
\epanet>scripts\make.cmd /t
Sometimes when new features are developed or bugs are fixed, new bugs get inadvertently introduced. When this occurs it is referred to as a "regression." Regression testing is used to help alert developers when their current activities have affected the existing functionality of an application. Our current regression testing approach is to run a suite of EPANET input files with known "good" results and make comparisons to detect differences.
The EPANET project's regression testing framework requires a Python environment and several custom Python packages; the most important of which is nrtest. nrtest is an open source package for performing regression testing on scientific software and was designed for flexibility. It uses a simple plugin interface for comparison operations making it easy to modify for testing different scientific software packages. It is well designed, written, and documented; however, it is no longer being actively maintained by its developer. Our project has been relying on it for several years and found it to be dependable. In that time we have had contact with the developer and he has added minor features and fixed bugs at our request.
nrtest download: https://pypi.org/project/nrtest/
nrtest documentation: https://nrtest.readthedocs.io/en/latest/?badge=latest
Regression testing requires two additional Python packages -- nrtest-epanet and epanet.output. nrtest-epanet is a nrtest comparison operator extension written for EPANET. It depends on epanet.output a thin Python wrapper for the epanet-output library to read values from EPANET's custom binary output file format. nrtest-epanet is essentially an iterator that reads the results section of the output file in the exact order in which they are written during an EPANET simulation. The epanet-output library was written in C to reduce time associated with file IO operations. This design makes nrtest-epanet efficient for reading large binary files.
Regression testing currently requires Python 3.6 64 bit. Once Python has been installed it is easy to configure for regression testing using Python packaging system utility pip and the requirements file found in the epanet/scripts folder.
\>cd epanet
\epanet>pip install -r scripts\requirements-appveyor.txt
The suite of EPANET input files are stored in a repository separate from the main EPANET project named epanet-nrtests to reduce clutter while maintaining configuration management on test files. Within this repository "releases" are used to hold benchmark archives containing the known "good" results. Versioning and build metadata are used to keep everything organized.
EPANET builds can potentially occur on multiple platforms -- Windows 32 and 64 bit, Linux, and MacOS. Benchmarks are currently platform specific. Software under active development is constantly changing. We use git commit hash to uniquely identify the version of the software being built. Lastly, any particular version of the software can be built many times with different build settings, so a build identifier is also useful. Therefore, three pieces of metadata are needed to uniquely identify a build; 1) the build platform, 2) the commit hash, and 3) a build identifier. The regression testing framework uses these three pieces of build metadata to uniquely identify a benchmark. This data is stored in the manifest file found in each benchmark archive. Release tags are used to access benchmarks. Each time a benchmark gets changed or updated a new release should be created. This way the latest release version tag can be used to easily retrieve the latest benchmark.
Running regression tests locally is a three step process. The first step builds runepanet.exe -- the software under test (SUT). The second step prepares the shell environment and stages the test and benchmark files. And the third step runs nrtest execute and compare commands. Helper scripts have been written to perform each of these steps to make running regression tests locally on Windows easy for developers.
\> cd epanet
\epanet>scripts\make.cmd
\epanet>scripts\before-nrtests.cmd
\epanet>scripts\run-nrtests.cmd
Regression testing is even more useful when running under Continuous Integration (CI) linked with the code repository. This way code can be checked for regressions during pull request review prior to being merged. Under CI the build and tests execute automatically on a remote build worker. Appveyor is a third party CI provider that is easily integrated with GitHub. For the EPANET project Appveyor has been configured to save a receipt with test results when regression tests pass or the SUT benchmark archives when they fail. This is useful for maintaining QA/QC records, inspection of results, debugging, and for keeping rolling development benchmarks updated.
See Wiki/Regression Testing SOPs
All to often testing is neglected and delayed or not performed at all. The reasoning often being that the project can't afford the time or money testing involves. Software testing is an integral part of the software lifecycle not just the last or an unnecessary step. It can be used throughout the development process to insure the "correctness" of each unit of functionality, larger more complex components, and the program as a whole. Testing helps developers and stakeholders have confidence in the quality of the software being created. Therefore, testing is a critical aspect of software quality assurance and essential to overall project management. By focusing attention on software quality we can insure that EPANET continues to meet the needs of our customers and stakeholders.