MeetingNotes Feb 02 2018

_Disclaimers/Limits_

Because compliance cannot be verified – it does not mean that an LMS (or AU) is not cmi5 conformant.

Failure to import course structures ( with over 1000 AU's ) due to size/capacity of the system does not indicate failure of compliance with cmi5. (

Test suite AU failure to execute due to software/environment conflicts…

LMS

• (TestSuite) Create Test Course Structure(s)
  • Multiple course structures
    • ZIP Package vs XML only
    • Mix of Relative and Fully Qualified URLS

    • One for size test 1000 + AU's

    • Smaller one for Completion (Satisfied Test)
      • Large enough to test all MoveOn Criteria
        • Blocks, AU's
• (Human) Import that structure(s) into the LMS (admin user)
• (Human) Register the learner (admin User)
• (Human) Access the course as a learner
  • Selected AU's (not all)
• (Machine) AU's (Testsuite) inspect LMS/LRS data
• (Human) Procedure for learner to follow thru the course(s).
  • Initial entry
  • Re-entry of AU's
• (Human) Modify course structure (admin user) and verify effect as a learner
  • Eg, MasteryScore, MoveOn, etc

AU (Content) - Individually

• (Human) add a reference (URL) to AU to inspect
• (Human) Launch the AU and Interact with AU
• (Human) Re-entry to AU's
  • Launch in different modes
    • Review, Normal, Browse
• (Machine) Testsuite will inspect AU xapi data

AU (Content) – Course Structure (on or more)

• (Machine) Testsuite (LMS) – import/validate a course structure
• (Human) Launch each AU in the course and Interact with AU
• (Human) Re-entry to AU's
  • Launch in different modes
    • Review, Normal, Browse
• (Machine) Testsuite will inspect AU xapi data

Course Structure/Package

• (Machine) Inspect XML
  • XSD validation
  • Relative pathing & Fully Qualified URL
• (Machine) Inspect Package
  • Inspect ZIP
  • File existence validation (ZIP)
    • Content
    • XML file
  • ZIP format (32/64)

Testing Procedures

Small course testing

•  (Human) Import that structure(s) into the LMS (admin user)
•  (Human) Register multiple learners (admin User) – User 1, 2, 3….
•  (Human) Access the course as a learner 1
  • oo Launch AU's in order
    •  Do Not Launch (AU5 – Not applicable)
  • oo Complete all AU's with minimum MoveOn Criterial
  • oo Single Entry
  • oo Final Check AU (Assess results)
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  AU x is a test suite specific tool (not Normal AU scenario)
    •  Alternate Credentials to Access to LRS for LMS issued statements.
      •  AUx may not have sufficient access to query the LRS for statements issued by other AU's – So alternate credentials would need to be provided (e.g. AUx will need to have a login form for alternate access- for "deep queries")
•  (Human) Access the course as a learner 2
  • oo Launch AU's in order
  • oo Complete all AU's with MoveOn Criteria
  • oo Multiple Entry
  • oo Final Check AU (Assess results
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  Alternate Credentials to Access to LRS for LMS issued statements.
•  (Human) Access the course as a learner 3
  • oo Launch AU's in any order
  • oo Multiple Entry (Re-launching - Exiting and Re-entering AU)
  • oo Complete all AU's except AU6
    •  Partially Complete AU6 and re-enter
    •  Complete AU6
  • oo Final Check AU(Assess results
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  Alternate Credentials to Access to LRS for LMS issued statements.

Large course testing

•  (Machine) AU's (Testsuite) inspect LMS/LRS data
•  (Human) Procedure for learner to follow thru the course(s).
  • oo Initial entry
  • oo Re-entry of AU's
•  (Human) Modify course structure (admin user) and verify effect as a learner
  • oo Eg, MasteryScore, MoveOn, etc
•  Final Check AU(Assess results)

<?xml version="1.0" encoding="utf-8"?>

<courseStructure xmlns="https://w3id.org/xapi/profiles/cmi5/v1/CourseStructure.xsd">

<course id="http://course-repository.example.edu/identifiers/courses/02baafcf">

<title>

  <langstring lang="en-US">Intro to Web Based Content</langstring>

</title>

<description>

  <langstring lang="en-US">

    This course will introduce you into the basics integrating xAPI with web

    based content.

  </langstring>

</description>

</course>

<block id=" http://www.example.com/identifiers/aublock/005430bf-b3ba-45e6-b47b-d629603d83d8" >

<au id= http://course-repository.example.edu/identifiers/courses/02baafcf/aus/4c07

 launchMethod="OwnWindow"

 masteryScore="0.8501"

 moveOn="Passed"

 activityType="http://adlnet.gov/expapi/activities/media"

>

<title>

  <langstring lang="en-US">Guess a Number</langstring>

</title>

<description>

  <langstring lang="en-US">

    This game uses xAPI to track the start and stop as well as the player's

    guesses.

  </langstring>

</description>

<url>./game.html</url>

<launchParameters>xyz123</launchParameters>

<entitlementKey>xyz-123-9999</entitlementKey>

</au>

</block>

</courseStructure>

• Rollup discussion – Blocks with Blocks only as children

Small Course Structure test (LMS)

• Created by testsuite (Static or generated)
• Zip File with content
• Mix of relative and fully qualified URL's
• Nested Blocks
• Mixed Blocks - Blocks with AU's and Block Children
• Blocks with Block children only

Block1

AU0

• launchMethod="OwnWindow"
• moveOn=" Completed "
• <Fully Qualified url>
• <launchParameters>
• <entitlementKey>

Block1.1

AU1

• launchMethod="OwnWindow"
• masteryScore="0.8501"
• moveOn="Passed"
• <Relative url>
• <launchParameters>
• <entitlementKey>

AU2

• launchMethod="OwnWindow"
• moveOn=" Completed "
• <Relative url>
• <launchParameters>
• <entitlementKey>

AU3

• launchMethod="OwnWindow"
• masteryScore="0.8501"
• moveOn="CompletedAndPassed "
• <Relative url>
• <launchParameters>
• <entitlementKey>

AU4

• launchMethod="OwnWindow"
• masteryScore="0.8501"
• moveOn="CompletedOrPassed "
• <Relative url>
• <launchParameters>
• <entitlementKey>

AU5

• launchMethod="OwnWindow"
• moveOn="NotApplicable"
• <Relative url>
• <launchParameters>
• <entitlementKey>

Block1.1.1

Block1.1.1.1

AU6

• launchMethod="OwnWindow"
• moveOn=" Completed"
• <Fully Qualified url>
• <launchParameters>
• <entitlementKey>

AUx

• launchMethod="OwnWindow"
• moveOn="NotApplicable"
• <Relative url>
• <launchParameters>
• <entitlementKey>

Large Course Structure test (LMS) Test

• Created by testsuite (Static or generated)
• Has 1001 AU's
• Fully Qualified URL
• Zipped and XML only

Block1

AU1

• launchMethod="OwnWindow"
• moveOn=" Completed "
• <url>
• <launchParameters>
• <entitlementKey>

…

AU1 1001

• launchMethod="OwnWindow"
• moveOn=" Completed "
• <url>
• <launchParameters>
• <entitlementKey>

Testing Procedures

Small course testing

•  TestSuite Generates a test course structure (XML)
  • oo (Initially) Static one to start
  • oo (Later) a parameter driven generation of course structure.
•  (Human) Import that structure(s) into the LMS (admin user)
•  (Human) Register multiple learners (admin User) – User 1, 2, 3….
•  (Human) Access the course as a learner 1
  • oo Launch AU's in order
    •  Do Not Launch (AU5 – Not applicable)
  • oo Complete all AU's with minimum MoveOn Criterial
  • oo Single Entry
  • oo Final Check AU
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  Alternate Credentials to Access to LRS for LMS issued statements.
•  (Human) Access the course as a learner 2
  • oo Launch AU's in order
  • oo Complete all AU's with MoveOn Criterial
  • oo Multiple Entry
  • oo Final Check AU
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  Alternate Credentials to Access to LRS for LMS issued statements.
•  (Human) Access the course as a learner 3
  • oo Launch AU's in any order
  • oo Multiple Entry (Re-launching - Exiting and Re-entering AU)
  • oo Complete all AU's except AU6
    •  Partially Complete AU6 and re-enter
    •  Complete AU6
  • oo Final Check AU
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  Alternate Credentials to Access to LRS for LMS issued statements.

Large course testing

•  TestSuite Generates a test course structure (XML)
  • oo (Initially) Static one to start
  • oo (Later) a parameter driven generation of course structure.
•  (Machine) AU's (Testsuite) inspect LMS/LRS data
•  (Human) Procedure for learner to follow thru the course(s).
  • oo Initial entry
  • oo Re-entry of AU's
•  (Human) Modify course structure (admin user) and verify effect as a learner
  • oo Eg, MasteryScore, MoveOn, etc

Detail Procedure

Small course testing

•  TestSuite Generates a test course structure (XML)
  • oo (Initially) Static one to start
  • oo (Later) a parameter driven generation of course structure.
•  (Human) Import that structure(s) into the LMS (admin user)
•  (Human) Register multiple learners (admin User) – User 1, 2, 3….
•  (Human) Access the course as a learner 1
  • oo Launch AU0
    •  Inspect URL (for all N/V pairs)
    •  Get Auth Token (fetch)
    •  Inspect State API
    •  Issue Initialized Statement
    •  Send Complete Statement
    • 
    •  Do Not Launch (AU5 – Not applicable)
  • oo Complete all AU's with minimum MoveOn Criterial
  • oo Single Entry
  • oo Final Check AU
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  Alternate Credentials to Access to LRS for LMS issued statements.
•  (Human) Access the course as a learner 2
  • oo Launch AU's in order
  • oo Complete all AU's with MoveOn Criterial
  • oo Multiple Entry
  • oo Final Check AU
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  Alternate Credentials to Access to LRS for LMS issued statements.
•  (Human) Access the course as a learner 3
  • oo Launch AU's in any order
  • oo Multiple Entry (Re-launching - Exiting and Re-entering AU)
  • oo Complete all AU's except AU6
    •  Partially Complete AU6 and re-enter
    •  Complete AU6
  • oo Final Check AU
    •  Run AU x after course complete (check for all satisfied)
    •  A wait period should be established
    •  AU x is Not Applicable
    •  Alternate Credentials to Access to LRS for LMS issued statements.

Individual AU Test (Simple)

Scope:

• Simple Launch (Initial and re-entrant)

Setup:

• Course Structure
• Fully qualified URL to content
• LaunchData= AU specific parameters
• Moveon=Completed

1. 1. Add setup data – import course Structure
2. 2. Register Learner in course, etc.
3. 3.** Action: Launch AU**

• oo Result: LaunchConsole (LMS)
  •  Launched Statement Issued
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
    •  Call Fetch URL 2 nd Time to verify Error Issued
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list

4. 4.** Action: Exit AU (Button)**

• •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
  •  Write Terminated Statement (Success)
  •  Return To LMS (Use Return URL if provided or Close The AU window)

5. 5.** Action: **** (re) Launch AU**

• oo Result: LaunchConsole (LMS)
  •  Launched Statement Issued
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Read AU-Specific State API Document. (Verify contents same)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list
    •  Previous Statements

6. 6.** Action: Send "Completed" (Button)**

• oo Result: AU
  •  Write Completed Statement (Success)

7. 7.** Action: Exit AU (Button)**

• oo Result: AU
  •  Write Terminated Statement (Success)
  •  Return To LMS (Use Return URL if provided or Close The AU window)
• oo Result: LaunchConsole (LMS)
  •  Satisfied Statement Issued – At Course Level
  •  Verify Abandon Statement not issued for each session
    •  After a reasonable period of time

Individual AU Test (One Entrance - Passed)

Scope:

• Simple Launch (ne launch pass)

Setup:

• Course Structure
• Fully qualified URL to content
• LaunchData= AU specific parameters
• Moveon=Passed

1. 1. Add setup data – import course Structure
2. 2. Register Learner in course, etc.
3. 3.** Action: Launch AU**

• oo Result: LaunchConsole (LMS)
  •  Launched Statement Issued
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
    •  Call Fetch URL 2 nd Time to verify Error Issued
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list

4. 4.** Action: Send "Pass" (Button)**

• oo Result: AU
  •  Write Passed Statement (Success)

5. 5.** Action: Exit AU (Button)**

• •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
  •  Write Terminated Statement (Success)
  •  Return To LMS (Use Return URL if provided or Close The AU window)
• oo Result: LaunchConsole (LMS)
  •  Satisfied Statement Issued – At Course Level
  •  Verify Abandon Statement not issued for each session
    •  After a reasonable period of time

Individual AU Test (Abandoned – AU crash After Launch)

Scope:

• Simple Launch (Abandoned – AU crash After Launch)

Setup:

• Course Structure
• Fully qualified URL to content
• LaunchData= AU specific parameters
• Moveon=Completed

1. 6. Add setup data – import course Structure
2. 7. Register Learner in course, etc.
3. 8.** Action: Launch AU**

• oo Result: LaunchConsole (LMS)
  •  Launched Statement Issued
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
    •  Call Fetch URL 2 nd Time to verify Error Issued
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list

4. 9.** Action: Close AU (Abnormally Exit)**
5. 10.** Action: Re-Launch AU (new session)**

• oo Result: LaunchConsole (LMS)
  •  Abandoned Statement Issued
  •  Launched Statement Issued
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
    •  Call Fetch URL 2 nd Time to verify Error Issued
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list
  • 

6. 11.** Action: Exit AU (Button)**

• •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
  •  Write Terminated Statement (Success)
  •  Return To LMS (Use Return URL if provided or Close The AU window)
• oo Result: LaunchConsole (LMS)
  •  Verify No Satisfied Statement
  •  Verify Abandon Statement not issued for second session
    •  After a reasonable period of time

Individual AU Test (2 AU Course - Abandoned)

Scope:

• Simple Launch (abandoned)

Setup:

• Course Structure – 2 AUs
• Fully qualified URL to content
• LaunchData= AU specific parameters
• Moveon=Completed, Completed

1. 1. Add setup data – import course Structure
2. 2. Register Learner in course, etc.
3. 3.** Action: Launch AU 1**

• oo Result: LaunchConsole (LMS)
  •  Launched Statement Issued
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
    •  Call Fetch URL 2 nd Time to verify Error Issued
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list

4. 4.** Action: Close AU 1 (Abnormally Exit)**
5. 5.** Action: Launch AU2 (new session)**

• oo Result: LaunchConsole (LMS)
  •  Abandoned Statement Issued (for AU1 session)
  •  Launched Statement Issued (AU2 Session)
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
    •  Call Fetch URL 2 nd Time to verify Error Issued
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list
  • 

6. 6.** Action: Exit AU (Button)**

• •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
  •  Write Terminated Statement (Success)
  •  Return To LMS (Use Return URL if provided or Close The AU window)
• oo Result: LaunchConsole (LMS)
  •  Verify No Satisfied Statement
  •  Verify Abandon Statement not issued for second session
    •  After a reasonable period of time

Individual AU Test (Terminate – Then write more)

Scope:

• )

Setup:

• Course Structure (1 AU)
• Fully qualified URL to content
• LaunchData= AU specific parameters
• Moveon=Completed

1. 1. Add setup data – import course Structure
2. 2. Register Learner in course, etc.
3. 3.** Action: Launch AU**

• oo Result: LaunchConsole (LMS)
  •  Launched Statement Issued
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
    •  Call Fetch URL 2 nd Time to verify Error Issued
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list

4. 4.** Action: AU Terminate Button (Issue Terminate)**

• oo Result: AU
  •  Issue Terminate Statement
  •  Then – Issue Completed Statement (after reasonable time –as defined in se)

5. 5.** Action: Re-Launch AU (new session)**

• oo Result: LaunchConsole (LMS)
  •  Abandoned Statement Issued
  •  Launched Statement Issued
  •  Write State API Document LMS.launchdata
  •  Redirect Browser to AU URL +cmi5 Query string
• oo Result: AU
  •  Parse Query String
  •  Call Fetch URL – Retrieve Authentication Token
    •  Call Fetch URL 2 nd Time to verify Error Issued
  •  Read State API Document LMS.launchdata (verify structure)
    •  Verify Launch Data Parameters from course structure are the same
  •  Write Initialized Statement (success)
  •  Present AU UI to learner (with results of checks)
    •  OK/Failed list
  • 

6. 6.** Action: Exit AU (Button)**

• •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
  •  Write Terminated Statement (Success)
  •  Return To LMS (Use Return URL if provided or Close The AU window)
• oo Result: LaunchConsole (LMS)
  •  Verify No Satisfied Statement
  •  Verify Abandon Statement not issued for second session
    •  After a reasonable period of time

_Disclaimers/Limits_

Because compliance cannot be verified – it does not mean that an LMS (or AU) is not cmi5 conformant.

Test suite AU failure to execute due to software/environment conflicts…

"** Reasonable Time" set by TestSuite parameters****. (Procedure will specify)**

Rationale: How long a learner user would expect to wait for LMS corrective action due to AU failure - up to 3 seconds