Add optional Implementations section

act-rules-format/act-rules-format.bs

@@ -109,6 +109,7 @@ An ACT Rule MAY also contain:
 
 * [Issues List](#issues-list) (optional)
 * [Background](#background) (optional)
+* [Implementations](#implementations) (optional)
 * [Acknowledgments](#acknowledgments) (optional)
 
 The ACT Rules format does not prescribe what format ACT Rules can be written in (e.g. HTML, DOCX, PDF, etc.). However, ACT Rules <em class="rfc2119">must</em> be written in a document that conforms to the Web Content Accessibility Guidelines [[WCAG]] or a comparable accessibility standard. This ensures that ACT Rules are accessible to people with disabilities. ACT Rule [test cases](#test-cases) are allowed to contain inaccessible content. If any test case contains accessibility issues listed in [WCAG 2.1 Section 5.2.5 Non-Interference](https://www.w3.org/TR/WCAG21/#cc5), users <em class="rfc2119">must</em> be warned of this in advance. In addition to supporting people with disabilities, using an accessible format also makes internationalization of ACT Rules easier.
@@ -476,8 +477,8 @@ All expectations of an [=atomic rule=] <em class="rfc2119">must</em> describe th
 <aside class=example>
   <header>Example expectation of a rule to test if a `role` attribute is valid:</header>
   <blockquote><ol>
-    <li>Each `role` attribute has a value that corresponds to a non-abstract [WAI-ARIA 1.1](https://www.w3.org/TR/wai-aria/) role.</li></blockquote>
-  </ol>
+    <li>Each `role` attribute has a value that corresponds to a non-abstract [WAI-ARIA 1.1](https://www.w3.org/TR/wai-aria/) role.</li>
+  </ol></blockquote>
 </aside>
 
 <div class="note">
@@ -544,6 +545,8 @@ Test Cases {#test-cases}
 
 Test cases are (snippets of) content that can be used to validate the implementation of ACT Rules. Each rule <em class="rfc2119">must</em> have one or more test cases for `passed`, `failed`, and `inapplicable` [=outcomes=]. A test case consists of two pieces of data, a snippet of each [input aspect](#input-aspects) for a rule, and the expected outcome for that rule. Test cases serve two functions, firstly as example scenarios for readers to understand when the outcome of a rule is `passed`, `failed`, or `inapplicable`. It also serves developers and users of accessibility testing tools and methodologies to validate that a rule is correctly implemented.
 
+Every `passed` and `inapplicable` test cases of an ACT Rule <em class="rfc2119">must</em> satisfy all the rule's [=conformance requirements=]. For every `failed` test cases, all conformance requirements <em class="rfc2119">must</em> be *not satisfied*.
+
 <aside class="example">
   <header>Example of HTML test cases for a rule that checks if `img` elements have a text alternative:</header>
   <blockquote>
@@ -589,6 +592,65 @@ Background (optional) {#background}
 An ACT Rule <em class="rfc2119">may</em> contain information about the background for the development of the rule, or references to relevant reading. The background information is optional, but whenever it is included in the rule, the relationship to the relevant reading can be specified. Examples of relevant background references for a rule for a [Web Content Accessibility Guidelines](https://www.w3.org/WAI/standards-guidelines/wcag/) [[WCAG]] success criterion could be [WCAG 2.1 Understanding documents](https://www.w3.org/WAI/WCAG21/Understanding/), [WCAG 2.1 Techniques](https://www.w3.org/WAI/WCAG21/Techniques/), or [WAI-ARIA 1.1](https://www.w3.org/TR/wai-aria/), CSS [[css-2018]], or HTML [[HTML]] specifications.
 
 
+Implementations (optional) {#implementation}
+--------------------------------
+
+An ACT Rule <em class="rfc2119">may</em> contain a list of [=implementations=] with [=checks=] that are consistent or partially consistent with that rule. An implementation is an accessibility test methodology or test tool that uses [=checks=] that give [=outcomes=] indicating whether [=accessibility requirement=] could be satisfied. These checks can be fully automated, completely manual, or some combination of the two.
+
+[=Checks=] are not required to have a one-to-one mapping to an ACT Rule. A single check can be consistent with multiple ACT Rules. A [combination of checks](#impl-partial-consistency) can together be consistent with a single ACT Rule.
+
+If implementations are included in an ACT Rule, consistency <em class="rfc2119">must</em> be determined as defined in this section. Other information can also be included for an implementation, such as the following:
+
+- Name, title or identifier of any consistent or partially consistent checks
+- [test modes](https://www.w3.org/TR/EARL10-Schema/#TestMode) of the checks (e.g. automatic, manual, semiAuto)
+- Name of implementation
+- Version used in testing consistency and coverage
+- Name of the vendor
+
+### Consistency ### {#impl-consistency}
+
+Consistency says how well a [=check=] is aligned with the intent of an ACT Rule. A consistent check of an ACT Rule is one that can be used to identify some or all non-conformance issues as described by the rule. A check is <dfn>consistent</dfn> when all the following are true:
+
+1. <dfn>True positives</dfn>: There are no inconsistencies with the test cases, meaning:
+    1. The `passed` and `inapplicable` test cases are **not** reported as `failed`; and
+    2. The `failed` test cases are **not** reported as `passed` or `inapplicable`; and
+
+2. <dfn>Completeness</dfn>: There are no gaps in coverage, meaning: 
+    1. None of the [=check=]'s outcomes are `untested`; and
+    2. At least one of the `failed` test cases is reported as `failed`; and
+
+3. <dfn>Rule Mapping</dfn>: The accessibility requirements are reported consistently, meaning:
+    1. The [=check=] is reported as testing for all the rule's [=conformance requirements=], except requirements of a level or standard not supported by the implementation; and
+    2. All accessibility requirements the rule reports to be a part of are either [=conformance requirements=] or [=secondary requirements=], except for requirements of [=accessibility requirement documents=] not mentioned in the rule.
+
+When a [=check=] reports more than one outcomes for a test case, the first outcome that appears on the following ordered list is considered for determining consistency:
+
+1. `failed`
+2. `untested`
+3. `passed`
+4. `inapplicable`
+5. `cantTell`
+
+### Partial Consistency ### {#impl-partial-consistency}
+
+A [=check=] that is not [=consistent=] is considered <dfn>partially consistent</dfn> if the [=true positive=] condition is true and not all outcomes it reports for the rule's test cases are `incomplete` or `untested`.
+
+### Consistency with multiple checks ### {#impl-multi-check}
+
+[=Implementation=] can include [=checks=] that test only parts of a single ACT Rule. The combination of those checks can be consistent if all parts of the ACT Rule are covered. The consistency of a <dfn>set of checks</dfn> can be determined by treating all [=partially consistent=] checks as though they were a single check.
+
+A [=set of checks=] is [=consistent=] with an ACT Rule if it meets all requirements for a [=consistent|consistent check=]. The [=outcomes=] for this set of checks is a list of all outcomes of checks that are partially consistent with the ACT Rule. The accessibility requirements of this set of checks is a list of all accessibility requirements of checks that are partially consistent with the ACT rule.
+
+<aside class=example>
+  <header>Example of a consistent set of checks:</header>
+  <blockquote>
+    <p>An ACT Rule checks that all elements with a native button elements, and element with a role="button" have an accessible name. The ACT Rule has test cases including both native buttons and role="button" elements.</p>
+    <p>The ACME Conformance Tool has a check for native button elements. This check passes all native buttons with an accessible name, it fails all native buttons without an accessible name, and gives inapplicable for pages  with only elements with role="button". Because it reports inapplicable for some of the failed test cases of the ACT Rule, this check is partially consistent.
+    <p>ACME Conformance Tool has a second check for elements with role="button". This works similarly to the previous check, except it passes and fails role="button" elements, and is inapplicable for pages with only native buttons. This check is also partially consistent. The combination of these two checks is consistent though, because all failed test cases of the ACT Rule are failed by either one or the other check.</p>
+  </blockquote>
+</aside>
+
+
 Acknowledgments (optional) {#acknowledgments}
 -----------------------------------------------
 
@@ -656,6 +718,16 @@ Definitions {#definitions}
     <p>A document, such as a standard, contract, policy or regulation, that includes [=accessibility requirements=]. For example, WCAG 2.1, WAI-ARIA 1.1, HTML 5.2, EPUB Accessibility 1.0, BBC HTML Accessibility Standards v2.0</p>
   </dd>
 
+  <dt><dfn>Implementation</dfn><dt>
+  <dd>
+    <p>An accessibility test methodology or test tool, containing a set of [=checks=] which can be used to determine (non-)conformance of [=accessibility requirements=].
+  </dd>
+
+  <dt><dfn>Check</dfn><dt>
+  <dd>
+    <p>A procedure resulting in an [=outcome=]. The procedure can be a step by step description of how to manually perform a test, a fully automated test, or some combination of manual and automated testing.</p>
+  </dd>
+
   <dt><dfn>Outcome</dfn></dt>
   <dd>
     <p>A conclusion that comes from evaluating an ACT Rule on a [=test subject=] or one of its constituent [=test target=]. An outcome can be one of the three following types:</p>