Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add optional Implementations section #541

Merged
merged 15 commits into from
Nov 21, 2023
Merged

Add optional Implementations section #541

Changes from 2 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
41eeb6c
Add optional Implementations section
WilcoFiers Sep 8, 2023
c5d8e71
Rewrite implementation section
WilcoFiers Sep 8, 2023
aa567f7
Fix typos
WilcoFiers Sep 8, 2023
6a27aa6
Apply suggestions from code review
WilcoFiers Oct 11, 2023
85635a6
Update to implementation section based on feedback
WilcoFiers Oct 11, 2023
9b1341e
Merge branch 'consistency' of github.com:w3c/wcag-act into consistency
WilcoFiers Oct 11, 2023
6c50185
Apply suggestions from code review
WilcoFiers Oct 12, 2023
5894297
Update act-rules-format/act-rules-format.bs
WilcoFiers Oct 12, 2023
7066828
Update act-rules-format/act-rules-format.bs
WilcoFiers Oct 12, 2023
5581a5d
Update act-rules-format/act-rules-format.bs
WilcoFiers Oct 12, 2023
b1c6bdc
Apply suggestions from code review
WilcoFiers Nov 9, 2023
0a4cdd9
Work through feedback
WilcoFiers Nov 9, 2023
243710c
Add leading slashes for closing dt tags
daniel-montalvo Nov 17, 2023
d78ea2f
Fix wrong id fragment
daniel-montalvo Nov 17, 2023
fe4ff6e
Apply suggestions from code review
WilcoFiers Nov 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions act-rules-format/act-rules-format.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ An ACT Rule MAY also contain:

* [Issues List](#issues-list) (optional)
* [Background](#background) (optional)
* [Implementations](#implementations) (optional)
* [Acknowledgments](#acknowledgments) (optional)

The ACT Rules format does not prescribe what format ACT Rules can be written in (e.g. HTML, DOCX, PDF, etc.). However, ACT Rules <em class="rfc2119">must</em> be written in a document that conforms to the Web Content Accessibility Guidelines [[WCAG]] or a comparable accessibility standard. This ensures that ACT Rules are accessible to people with disabilities. ACT Rule [test cases](#test-cases) are allowed to contain inaccessible content. If any test case contains accessibility issues listed in [WCAG 2.1 Section 5.2.5 Non-Interference](https://www.w3.org/TR/WCAG21/#cc5), users <em class="rfc2119">must</em> be warned of this in advance. In addition to supporting people with disabilities, using an accessible format also makes internationalization of ACT Rules easier.
Expand Down Expand Up @@ -544,6 +545,8 @@ Test Cases {#test-cases}

Test cases are (snippets of) content that can be used to validate the implementation of ACT Rules. Each rule <em class="rfc2119">must</em> have one or more test cases for `passed`, `failed`, and `inapplicable` [=outcomes=]. A test case consists of two pieces of data, a snippet of each [input aspect](#input-aspects) for a rule, and the expected outcome for that rule. Test cases serve two functions, firstly as example scenarios for readers to understand when the outcome of a rule is `passed`, `failed`, or `inapplicable`. It also serves developers and users of accessibility testing tools and methodologies to validate that a rule is correctly implemented.

All `passed` and `inapplicable` test cases of an ACT Rule <em class="rfc2119">must</em> satisfy at least one of that rule's [=accessibility requirements=]. For all `failed` test cases, all accessibility requirements <em class="rfc2119">must</em> be *not satisfied*.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a backward compatible change. We need to think about how we want to handle that. Maybe only require this if the rule has metadata indicating it was created for ACT-RF 1.1?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention this is not about the secondary requirements?
(i.e. Failed case do not need to fail all secondary requirements)

WilcoFiers marked this conversation as resolved.
Show resolved Hide resolved

<aside class="example">
<header>Example of HTML test cases for a rule that checks if `img` elements have a text alternative:</header>
<blockquote>
Expand Down Expand Up @@ -589,6 +592,52 @@ Background (optional) {#background}
An ACT Rule <em class="rfc2119">may</em> contain information about the background for the development of the rule, or references to relevant reading. The background information is optional, but whenever it is included in the rule, the relationship to the relevant reading can be specified. Examples of relevant background references for a rule for a [Web Content Accessibility Guidelines](https://www.w3.org/WAI/standards-guidelines/wcag/) [[WCAG]] success criterion could be [WCAG 2.1 Understanding documents](https://www.w3.org/WAI/WCAG21/Understanding/), [WCAG 2.1 Techniques](https://www.w3.org/WAI/WCAG21/Techniques/), or [WAI-ARIA 1.1](https://www.w3.org/TR/wai-aria/), CSS [[css-2018]], or HTML [[HTML]] specifications.


Implementations (optional) {#implementation}
--------------------------------

An ACT Rule <em class="rfc2119">may</em> contain a list of implementations that are consistent or partially consistent with that rule. An [=implementation=] is an accessibility test methodology or test tool that can be used to produce [=outcomes=] of an ACT Rule.

An [=implementation=] consists of one or more [=implementation procedures=]. The [=Implementation procedures=] are not required to have a one-to-one mapping to an ACT Rule. A combination of implementation procedures can together be consistent with a single ACT Rule. A single implementation procedure can also be consistent with multiple ACT Rules.

For each implementation the following data can be provided. If included, consistency <em class="rfc2119">must</em> be determined as defined in this section.

- Name of implementation
- Name of the vendor
- [Consistency](#impl-consistency)
- Version used in testing consistency and coverage
- [test modes](https://www.w3.org/TR/EARL10-Schema/#TestMode) of the set implementation procedures (e.g. automatic, manual, semiAuto)

### Consistency ### {#impl-consistency}

Consistency says how well one or more [=implementation procedure=] are aligned with the intent of an ACT Rule. A **consistent** implementation of an ACT Rule is one that can be used to identify some or all non-conformance issues as described by the rule. An implementation procedure is <dfn>consistent</dfn> when all the following are true:

1. <dfn>True positives<dfn>: There are no inconsistencies with the test cases, meaning:
1. The `passed` and `inapplicable` test cases are **not** reported as `failed`; and
2. The `failed` test cases are **not** reported as `passed` or `inapplicable`; and
WilcoFiers marked this conversation as resolved.
Show resolved Hide resolved

2. <dfn>Completeness</dfn>: There are no gaps in coverage, meaning:
1. None of the [=implementation procedure=] outcomes are `untested`; and
2. At least one of the `failed` test cases is reported as `failed`; and

3. <dfn>Rule Mapping<dfn>: The accessibility requirements are reported consistently, meaning:
1. The [=implementation procedure=] is reported as testing for all the rule's[=conformance requirements=], except requirements of a level or standard not supported by the implementation; and
WilcoFiers marked this conversation as resolved.
Show resolved Hide resolved
2. All accessibility requirements the rule reports to be a part of are either [=conformance requirements=] or [=secondary requirements=], except for requirements of [=accessibility requirement documents=] not mentioned in the rule.

When an [=implementation procedure=] reports more than one outcomes for a test case, the first outcome that appears on the following ordered list is considered for determining consistency:

1. `untested`
2. `failed`
3. `passed`
4. `incomplete`
WilcoFiers marked this conversation as resolved.
Show resolved Hide resolved
5. `cantTell`
WilcoFiers marked this conversation as resolved.
Show resolved Hide resolved

### Partial Consistency ### {#impl-partial-consistency}

An [=implementation procedure=] that is not **consistent** is considered **partially consistent** if the [=true positive=] conditions are true, and not all outcomes it reports for the rule's test cases are `incomplete` or `untested`.

An implementation can have more than one partially consistent [=implementation procedures=]. The combination of [=implementation procedures=] can be **consistent** if the union of the outcomes and a union of the accessibility requirements meet the requirements for [=consistency=].


Acknowledgments (optional) {#acknowledgments}
-----------------------------------------------

Expand Down Expand Up @@ -656,6 +705,16 @@ Definitions {#definitions}
<p>A document, such as a standard, contract, policy or regulation, that includes [=accessibility requirements=]. For example, WCAG 2.1, WAI-ARIA 1.1, HTML 5.2, EPUB Accessibility 1.0, BBC HTML Accessibility Standards v2.0</p>
</dd>

<dt><dfn>Implementation</dfn><dt>
<dd>
<p>An accessibility test methodology or test tool, containing a set of [=implementation procedures=] that describes how to get the expected [=outcomes=] of an ACT Rule.</p>
</dd>

<dt><dfn>Implementation Procedure</dfn><dt>
WilcoFiers marked this conversation as resolved.
Show resolved Hide resolved
<dd>
<p>A procedure resulting in an [=outcome=]. The procedure can be a step by step description of how to manually perform the test, a fully automated test, or some combination of manual and automated testing.</p>
</dd>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(⚠️ set theory / computability vocabulary included)
So, a (mathematical or computer program) function can either be described as an intention, i.e. a formula or code to compute it (e.g. x => x*x is an intention for the squaring function); or it can be described as an extension, i.e. the set of (input, output) pairs (e.g. { (0, 0), (1, 1), (2, 4), ... }for the squaring function.
Intentions are usually easier to manipulate and we are used to them. Extensions are sometimes useful to consider.

Anyway, here the procedure/check is described as an intention ("a step by step procedure"). But ultimately, the only thing that matters for consistency is the outcomes it gives, i.e. its extension.
So, I'm wondering whether it could make sense to instead define the procedure/check as "a set of outcomes" rather than as the piece of code that produces this set.
It may be a bit harder to grasp at first, but it might allow to simplify the other definitions down the line. It is also a bit closer to what we do in practice since we work with the implementation reports (an extension) rather than the code itself (an intension).


<dt><dfn>Outcome</dfn></dt>
<dd>
<p>A conclusion that comes from evaluating an ACT Rule on a [=test subject=] or one of its constituent [=test target=]. An outcome can be one of the three following types:</p>
Expand Down
Loading