Skip to content

Latest commit

 

History

History
292 lines (214 loc) · 14.6 KB

testem-component.md

File metadata and controls

292 lines (214 loc) · 14.6 KB

fluid.testem

The fluid.testem component is designed to launch browser-based tests in one or more browsers. By default, this component:

  1. Instruments the code under test.
  2. Starts test fixtures.
  3. Runs Testem tests.
  4. Prepares a coverage report.
  5. Cleans up temporary files (instrumented source, browser data, coverage data).

Environment Variables

The default grade fluid.testem supports a handful of environment variables that modify its behaviour. If you need more control than these switches, you should consider extending one of the underlying grades instead (see below).

DISABLE_COVERAGE

If this option is set, code will not be instrumented. This intended to support things like launching a configuration file and debugging test problems locally, as in this example command on a system that supports a unix-like shell:

DISABLE_COVERAGE=true node node_modules/testem/testem.js --file tests/testem.js

If you want to permanently disable instrumentation, you should look at the fluid.testem.coverage grade described below.

TESTEM_ENVIRONMENT

To help streamline local testing, two predefined "environments" are supported. These override the default testem options to launch a minimal subset of browsers (only Firefox and Chrome). This is intended for cases in which you want to be able to test for maximum compatibility by default, and to more quickly test incremental changes by only running a subset of the available browsers. On a system that supports a unix-like shell, you might use this functionality with commands like the following:

TESTEM_ENVIRONMENT=headless npm test

TESTEM_ENVIRONMENT=headed node node_modules/testem/testem.js --file tests/testem.js

The supported values are "headed" and "headless", each of which is described below.

"headed"

The "minimal headed" environment launches only Firefox and Chrome. Note that if you do not have either of these installed, runs launched with this environment will fail. Note also that because of a bug in testem, Firefox will not close itself when it is launched "headed" on Windows.

"headless"

The "minimal headless" environment launches only Headless Firefox and Headless Chrome. Note that if you do not have either of these installed, runs launched with this environment will fail.

Component Options

These are the options supported by the default fluid.testem component when it is run with no environmental variables:

Option Type Description
cwd {String} Defaults to process.cwd(), i.e. the directory from which the script was called.
testemDir {String} The directory in which testem's browser settings and temporary files should be stored.
wrappedEventTimeout {Number} How long to wait (in milliseconds) before triggering a timeout when waiting for key startup/shutdown events.
coveragePort (required) {Number} The port fluid-express should listen on to record coverage data. Defaults to 7003.
coverageUrl {String} The URL on which the coverage listener should be run. Set based on coveragePort by default.
coverageDir (required) {String} The full or package-relative path where coverage data should be saved. By default, a unique subdirectory is created in os.tmpdir().
reportsDir (required) {String} The full or package-relative path where coverage reports and test results should be saved. By default, a unique subdirectory is created in os.tmpdir().
instrumentationOptions {Object} The options to use when instrumenting source. See the instrumenter docs for details.
instrumentedSourceDir {String} The location in which to store instrumented code. By default, a unique subdirectory is created in os.tmpdir().
sourceDirs {Array} One or more source directories to load, relative to the directory in which your configuration file is stored (see "Content and Source Directories" below). Note that although Testem itself supports globbing and file patterns, you are expected to supply only directory paths here.
testPages {Array} One or more test pages to load in the browser, relative to the directory in which your configuration file is stored (see "paths" below).
contentDirs {Array} One or more directories to host and make available in tests. See "Content and Source Directories" below for the supported format.
testemOptions {Object} The raw configuration options to pass to Testem. See the Testem docs for supported options.
browserArgs {Object} The browser arguments that will be passed to Testem by default.
headlessBrowserArgs {Object} The browser arguments that will be passed to Testem if the HEADLESS environment variable is set to a non-empty value. If there is no "headless" option for a given browser, the options from browserArgs will be used.
cleanup.initial {Array} An array of cleanup definitions (see below) to be cleaned up before the tests are run.
cleanup.final {Array} An array of cleanup definitions (see below) to be cleaned up after the tests are run and all reporting is complete.

Please note, although you can change options.testemOptions.framework, fluid.testem is only tested with QUnit and is unlikely to work with other frameworks supported by Testem.

Cleanup Definitions

The initial and final cleanup options expect to be passed an array of "cleanup definitions", which support the following attributes:

Option Type Description
name (required) {String} A "nickname" to use for this directory in log messages.
path (required) {String} The path to remove.
isTestemContent {Boolean} If this directory contains testem-generated content, we use a different cleanup method.

Component Invokers

{fluid.testem}.handleTestemStart(config, data, callback)

  • config: The configuration information Testem exposes as part of its lifecycle.
  • data: The data Testem exposes as part of its lifecycle.
  • callback: A function to be called when it is safe for Testem to run tests. If you do not call this callback, Testem will hang indefinitely before running tests.
  • Returns: Nothing.

An invoker which is called before testem begins its test run. Starts a chain of events which cleanup before the tests, instrument code, start test fixtures, and perform other preparatory work. For details on how this works, see "The Testem Event Lifecycle".

{fluid.testem}.handleTestemExit(config, data, callback)

  • config: The configuration information Testem exposes as part of its lifecycle.
  • data: The data Testem exposes as part of its lifecycle.
  • callback: A function to be called when it is safe for Testem to exit. If you do not call this callback, you will not be able to quit Testem without killing the process.
  • Returns: Nothing.

An invoker which is called when Testem has completed all tests. Used to stop test fixtures, prepare reports, and remove temporary content. For details on how this works, see "The Testem Event Lifecycle".

{fluid.testem}.getTestemOptions()

An invoker which retrieves the Testem options, including all "generated" options, such as routes to replace the original source with instrumented source. This invoker is intended to be used with module.exports to expose the component options in the way the Testem expects when working with javascript Testem configuration files.

See the README file for an example of using this invoker.

Content and Source Directories

Both "source" and "content" (non-source) directories can be referred to using a "short" or "long" notation. Here is an example of hosting our source and node_modules directories using "short" notation:

sourceDirs: {
  src: "src"
},
contentDirs: {
  nm: "node_modules"
}

Here's the same in "long" format.

sourceDirs: {
  src: {
   filePath: "src"
  }
},
contentDirs: {
  nm: {
    filePath: "node_modules"
  }
}

The following sub-options are supported:

Option Type Description
filePath {String} A full path, relative path, or package-relative path to the content to be hosted.
routePath {String} The path at which this content should be mounted within our express instance. Defaults to the last segment of filePath.
proxyPath {String} The path associated with this content that should be redirected from Testem to our express instance. Defaults to the last segment of filePath.

Cleanup

By default, Testem generates browser content in the directory os.tmpdir(), which it does not clean up when the test run is complete. The fluid.testem component automatically cleans this up by default. The component also cleans up instrumented code and raw coverage data.

Collecting Browser Coverage Data

See the README for an example of using the components in this package to collect only browser coverage data.

Combining Browser Coverage with non-Browser coverage

If your work involves a mixture of node and browser tests, you may want to collect coverage data across a range of test runs and then collate it yourself. This package makes use of the same libraries as Istanbul, so that combined reports can be prepared.

Additional Components

fluid.testem.base

The underlying base grade, without any instrumentation, coverage collection, or coverage reporting.

fluid.testem.coverage

This grade collects coverage data, but does not instrument source. It is intended for use when you have already instrumented the code by other means. Please note, you must set instrumentedSourceDir (see above) to the location of your instrumented code to use this grade. This grade only removes the temporary content generated by Testem itself, and not any pre-instrumented code or coverage data. This grade is mainly provided for legacy setups, in almost all cases you should use thefluid.testem.instrumentation grade instead.

fluid.testem.instrumentation

This grade instruments source itself and collects coverage data, but does not prepare a report at the end or remove the coverage data during its cleanup phase. It is provided for use in combining reports from different test runs, such as when running node tests and browser tests in the same package. See below for a detailed example.

fluid.testem.coverageReporting

This grade instruments code, collects coverage data, and generates a coverage report. If you are testing non-browser code such as node scripts, if you save the coverage reporting data to the same directory, you can run this grade after your node tests to produce a combined report.

Example: Combining Node Coverage with Browser Coverage.

The key to preparing a combined report is to ensure that:

  1. Each stage saves its coverage data to the same location.
  2. Each stage avoids creating a misleading interim report.
  3. Each stage avoids removing its coverage data at the end of its run.
  4. A combined report is prepared using nyc report once all stages are completed.
Registering Your Package

In your package's main (node) entry point, you will need code like the following:

var fluid = require("infusion");
fluid.module.register("my-package", __dirname, require);
Setting up your Testem Component
var fluid = require("infusion");
fluid.require("%my-package");
var my = registerNamespace("my");
fluid.defaults("my.testem.grade", {
    gradeNames: ["fluid.testem.coverage"],
    reportsDir: "%my-package/reports",
    coverageDir: "%my-package/coverage",
    instrumentedSourceDir: "%my-package/instrumented",
    sourceDirs: {
        src: "%my-package/src"
    },
    contentDirs: {
        nm: "%my-package/node_modules"
    }
});

module.exports = my.testem.grade.getTestemOptions();

Save this to tests/testem.js.

Setting up your package.json file.

For this example, you need to install a few modules, using a command like:

npm install --save-dev nyc rimraf fluid-testem

Once you've done this, you'll need to update the scripts section of your package.json:

  "scripts": {
    "pretest": "node node_modules/.bin/rimraf reports/* coverage/*",
    "test": "npm run test:node && npm run test:browser",
    "test:browser": "node node_modules/.bin/testem ci --file tests/testem.js",
    "test:node": "node node_modules/.bin/nyc --temp-directory coverage --reporter none tests/node-tests.js",
    "posttest": "node node_modules/.bin/nyc report --temp-directory coverage --reporter text-summary --reporter html"
  }
Running the Tests and Combined Report

To use this configuration, you would simply call npm test from the root of your repository. Let's go through what happens step by step.

  1. The pretest step cleans out any previous coverage data and reports.
  2. The test step calls the browser and node tests in sequence.
  3. The test:browser step uses a Testem Component to instrument our source and save coverage data to ./coverage.
  4. The test:node step uses nyc to instrument our code and save coverage data to the same ./coverage subdirectory.
  5. The posttest step uses nyc to create a combined report based on all of the coverage data found in ./coverage. A text summary is also displayed.

In this example, I'm assuming that we are only exercising browser-side code in our browser tests. If you are also exercising server-side fixtures (for example, by making AJAX requests), you would change test:browser to something like:

    "test:browser": "node node_modules/.bin/nyc --temp-directory coverage node_modules/testem/testem.js ci --file tests/testem.js",

In this scenario, our Testem component collect coverage data from browsers, and nyc itself collects coverage data from any server-side fixtures used in our tests.