Web Context

[AdamSobieski]

1. Introduction

Which tasks are users performing (and why)? Where are they in their tasks?

With such context data, developers could more readily develop next-generation context-aware and adaptive solutions, these including: adaptive hypermedia, context-sensitive user interfaces, context-sensitive help and documentation, search engines, question-answering systems, recommender systems, and AI assistants.

Having users' task context data available on their client devices would also enable or benefit a number of attention management scenarios such as: measuring cognitive load and fatigue, reducing interruptions and distractions, scheduling messages, alerts, and notifications (e.g., to task boundaries), and supporting better multitasking.

2. Client-side

2.1. Task Context Awareness

Let us consider a thought experiment where users have context management applications which can search for, load, and display diagrams for tasks. As envisioned, users would be able to click upon, or otherwise select, components of these diagrams as they progressed through their tasks.

Next, we can consider that, instead of users having to manually select nodes in potentially intricate business process model diagrams, a simpler interaction layer of content (e.g., HTML) might accompany these diagrams. Interaction layers could, for instance, include input forms for users to enter data into.

Content in interaction layers might interact with more intricate models and diagrams in one or more backing layers of content (e.g., BPMN, BPEL, XPDL) and these would, instead or additionally, be responsible for updating users' task contexts.

Context management applications would be able to receive messages from other applications. Such messaging might be mediated by operating systems. These received messages would be relayed to those business process model diagrams being executed and could result in task context changes.

Context management applications would provide multiple tabs to support multitasking scenarios. As envisioned, eventually, other software applications (these potentially including webpages in Web browsers) could open tabs in, load content into opened tabs, send messages to opened tabs, and close their previously opened tabs.

Context management applications would send users' dynamic task contexts to operating systems so that other software applications, including Web browsers, could receive these data and/or subscribe to notifications raised when such data had changed.

2.2. Goal Awareness

Users' context management applications could contain goal management components. Users might inherit overarching goals from their organizations, departments, teams, managers, assignments, and generate these on an individual basis by manually entering them into such application components. Goals could be selected and/or provided to these components by referencing them, using URLs, or by using natural-language or structured-knowledge data entry. Users' tasks could be connected to their overarching, project-specific, and task-specific goals, explained or justified as supporting them.

If a user's task-related context data pertained to what they were doing, their goal-related context data would pertain to why they were doing it.

3. Accessing and Transmitting Users' Task Context and Goal Data

3.1. A JavaScript API

A new JavaScript API could allow webpages in Web browsers to request permission to access to users' task context data.

As envisioned, requesting users' task context data would be a powerful feature, resembling other standardized permissions (geolocation, notifications, push, web share) and provisional permissions (accelerometer, window management, local fonts).

3.2. HTTP POST

Users' task context data could be transmitted to servers using the HTTP POST method, e.g., alongside text queries to context-aware search engines, Q&A systems, recommender systems, and AI assistants.

3.3. HTTP Request Header

In theory, a new HTTP request header, e.g., Context, could also be of use for users to efficiently share their task contexts with trusted services. Such a request header could make use of structured fields or JSON.

3.4. WebSocket

WebSocket is well-suited for scenarios involving real-time streaming and could enable transmitting or streaming users' task contexts to servers.

3.5. WebRTC

Similarly, WebRTC could be of use for transmitting or streaming users' task contexts to servers or peers. Scenarios to consider, in these regards, include providing users with context-sensitive help or assistance from either human personnel or multimodal conversational AI systems.

4. Server-side

Servers could utilize users' task context data to retrieve referenced business process models, deserialize their states, and query these deserialized stateful models in a number of ways, perhaps using knowledge-graph representations (see also: BBO) or, perhaps, using foundation models or generative AI.

Embedding vectors for business process models are being explored.

5. Examples

5.1. Multi-layered Content

Multi-layered content, discussed above, loaded by users' context management applications could resemble either:

<content version="0.1" xmlns="...">
  <layer id="layer-1" kind="interaction" type="application/xhtml+xml">
    ...
  </layer>
  <layer id="layer-2" kind="backing" type="application/xpdl+xml">
    ...
  </layer>
</content>

or:

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <script id="layer-2" type="application/xpdl+xml" src="..." />
    ...
  </head>
  <body>
    ...
  </body>
</html>

5.2. Representing Task Context

Task context data for a simple task might resemble:

{
  "model": "https://en.wikiprocess.org/wiki/simple-sequence-1.xpdl",
  "at": [ { "label": "step-1" } ]
}

Task context data for the example scenario, a book chapter report, might resemble:

{
  "model": "https://en.wikiprocess.org/wiki/book-report-3.xpdl",
  "inputs": { "book": "The Odyssey", "chapter": "Book 8" },
  "at": [ { "label": "step-2" } ]
}

Task context data for a more complex task might resemble:

{
  "model": "https://en.wikiprocess.org/wiki/complex-task-1.xpdl",
  "inputs": { "input-1": "value-1" },
  "variables": { "x": 1, "y": 2, "z": 3 },
  "at": [ { "label": "step-7" }, { "label": "step-22", "state": [ {
    "model": "https://en.wikiprocess.org/wiki/called-subprocess-1.xpdl",
    "inputs": { "input-2": "value-2" },
    "variables": { "w": 4 },
    "at": [ { "label": "step-2" } ]
  } ] } ]
}

5.3. Event and Experience Modeling

By considering a user's task context to be what activity that they are doing, more intricate event or experience models could be considered which also include who is performing the activity and when the activity is taking place.

{
  "actor": "[email protected]",
  "timestamp": "2024-12-04T12:34Z",
  "state": [ {
    "model": "https://en.wikiprocess.org/wiki/complex-task-1.xpdl",
    "inputs": { "input-1": "value-1" },
    "variables": { "x": 1, "y": 2, "z": 3 },
    "at": [ { "label": "step-7" }, { "label": "step-22", "state": [ {
      "model": "https://en.wikiprocess.org/wiki/called-subprocess-1.xpdl",
      "inputs": { "input-2": "value-2" },
      "variables": { "w": 4 },
      "at": [ { "label": "step-2" } ]
    } ] } ]
  } ]
}

Other examples of models for representing users' experiences include the Experience API, an e-learning software specification which records and tracks various types of learning experiences for learning systems. Standards involving representing events, broadly, for logs and streams include XES.

5.4. Goals, Reasons, and Justifications

In addition to who, what, when, and where, there is a matter of adding, as optional, why to models of task contexts as well as justifications connecting each what to its why.

5.4.1. Annotating Reified Connections Between Tasks and Goals

A first approach involves three components:

1. Collections of recursive data structures for tasks and subtasks.
2. Collections of recursive data structures for goals and subgoals.
3. JSON-LD content which interrelates nodes from each set of data structures.
   1. With reification (e.g.: JSON-LD-Star), can more readily express justifications.

So, for example:

{
  "actor": "[email protected]",
  "timestamp": "2024-12-04T12:34Z",
  "state": [ {
    "@id": "urn:uuid:651a8749-6fdd-4751-a263-b2ee1aec3d18",
    "model": "https://en.wikiprocess.org/wiki/complex-task-1.xpdl",
    "inputs": { "input-1": "value-1" },
    "variables": { "x": 1, "y": 2, "z": 3 },
    "at": [ { "label": "step-7" }, { "label": "step-22", "state": [ {
      "@id": "urn:uuid:fe9656a8-3b08-4b8f-87d8-baab9124bfec",
      "model": "https://en.wikiprocess.org/wiki/called-subprocess-1.xpdl",
      "inputs": { "input-2": "value-2" },
      "variables": { "w": 4 },
      "at": [ { "label": "step-2" } ]
    } ] } ]
  } ],
  "goals": [ {
    "@id": "urn:uuid:88bc8fd1-9466-4b10-8fe7-55d83d085ea4",
    "model": "https://domain.com/goals/goal-123.xml"
  },
  {
    "@id": "urn:uuid:d066e92c-0464-49c2-a965-4e69ea85ce79",
    "model": "https://domain.com/goals/goal-456.xml"
  } ],
  "explanation": {
    "@context": {
      "@base": "http://example.org/",
      "@vocab": "http://example.org/"
    },
    "@graph": [
      {
        "@id": "urn:uuid:651a8749-6fdd-4751-a263-b2ee1aec3d18",
        "supports": {
          "@id": "urn:uuid:88bc8fd1-9466-4b10-8fe7-55d83d085ea4",
          "@annotation": {
            "justification": {
              "model": "https://domain.com/justifications/justification-123.xml"
            }
          }
        }
      }
    ]
  }
}

5.5. Walkthrough: Book Chapter Report

The following example scenario involves a college student writing a book report about a chapter of a book.

1. A student opens their context management application and, in a tab in it, searches for: "writing a book chapter report".
   1. This application is configured to use a well-known third-party service, e.g., https://en.wikiprocess.org.
2. A number of search results are presented to the student. There is more than one way to do that task.
3. The student selects a search result item which best describes the specific task that they want to perform.
4. The corresponding business process model content (both backing layer and interaction layer) is loaded into their context management application.
5. Their context management application presents an input form asking: "Enter the title of the book that you are writing a book report about".
6. The student enters "The Odyssey" into this input form.
7. This value becomes the value of a context parameter.
8. Their context management application transitions to another interaction layer page with an input form asking: "Enter which chapter".
9. The student enter "Book 8" into this input form.
10. This value becomes the value of another context parameter.
11. Next, their context management application displays, for this example, a simple sequence of steps for the student to perform.
    1. These steps, using templates or generative AI, contain the book title and chapter number provided by the student.
    2. The context management application has educational value as the student can refer to the displayed diagram or flowchart, in this case a simple sequence of steps, at their convenience.
12. The first step in the displayed sequence might be: "Open a word-processing application."
13. The student opens a word-processing application by clicking on a desktop icon.
    1. Alternatively, the student could click upon a hyperlink or a button on that step displayed in the interaction layer and their word-processing application could automatically be opened.
14. Their word-processing application, upon opening, sends a message to their context management application in a manner mediated by the operating system.
15. Their context management application receives this message and changes the active and highlighted step to the next one in the sequence.
    1. As it does, it notifies the operating system of a task context change event.
16. The student then opens a Web browser.
17. The student navigates the Web browser to their preferred service for performing searches or asking questions, enters a search query or asks a question, and, next, presses enter or a button.
    1. The student has previously allowed this service to access their task context data.
    2. The Web browser has previously subscribed to or at this point retrieves the user's task context from the operating system.
18. Their selected service utilizes a JavaScript API to access the student's task context data and utilizes HTTP POST to send it. Alternatively, their selected service could receive the student's task context data via a HTTP request header.
    1. Their selected service gleans that the student is: writing a book report about book 8 of The Odyssey.
19. The search results or answers provided are contextual, task-specific, and immediately what the student needs at that current step of their current task.
20. The student uses the search results or answers and starts or continues writing their book report in their word-processing application.

6. Discussion

6.1. Security

Communicating users' task contexts involves referencing business process model resources. Task context data from users' client devices could refer to such resources on:

1. well-known third-party services, e.g., https://en.wikiprocess.org,
2. other third-party locations, e.g., cloud-computing services which might be orchestrated by users' AI assistants,
3. users' client devices.

A security-related concern is that a malicious actor could wield a botnet to initiate or escalate a DDOS attack using a large-scale service, e.g., a search engine, which would retrieve content using URLs in specified context data.

6.2. Privacy

As envisioned, users would be able to open, pause, resume, and close their context management applications.

Sharing users’ task contexts with trusted services would require users’ permissions. Users could configure this permission, per domain, alongside their other permissions, using lock-icon menus provided in many Web browsers’ address bars.

6.3. Artificial Intelligence

6.3.1. Towards Automated Task Context and Goal Management

Users might, into the future, want their artificial-intelligence assistants to be able to handle interacting with their context management applications on their behalves so that the users could focus their attentions entirely on performing and completing their tasks utilizing their other software tools. Context management applications, then, could be mostly minimized on system trays or could be or involve background services.

6.3.2. Agents' Task Contexts and Goals

Context management applications and/or background services could additionally manage task contexts for artificial-intelligence agents. Artificial-intelligence agents could coordinate with these to be better able to answer users' (or their own) questions about which tasks that they (or invoked or otherwise interoperating tools or services) were performing, where they were in these tasks, and why they were performing these tasks.

Advanced users, e.g., awaiting responses or results from artificial-intelligence agents, could make use of interoperating context management applications to obtain and explore pertinent real-time data and analytics.

6.3.3. Multi-agent Systems and Teamwork

Artificial-intelligence agents, acting on the behalves of users, could utilize enhanced awarenesses of their and their users' task contexts and goals to better provide features and services and to coordinate in man-machine teams.

7. Conclusion

I wanted to share these ideas, here, first, with the W3C WICG for discussion and brainstorming. What do you think about these ideas to enable next-generation context-aware and adaptive solutions, these solutions including: adaptive hypermedia, context-sensitive user interfaces, context-sensitive help and documentation systems, search engines, question-answering systems, recommender systems, and AI assistants?

[reschke]

Formal comments:

1. RFC9110 does not preclude "#" in field values.
2. New header fields in general should use Structured Field syntax.

[reschke]

I'd like to see a discussion why this needs a new header field. It seems you are just suggesting another dimension for content negotiation.

• Did you consider modeling your use case into different HTTP resources?
• How do you envision browsers to implement this?
• Did you consider existing header fields such as "Link"?

[AdamSobieski]

Thank you for the information about structured field syntax (RFC 9651) and about the potential applicability of the existing Link header (RFC 8288).

Users' task context data enable context-aware computing and context can be considered to be another dimension of content negotiation.

The use cases of context-aware computing are fairly broad and, so, mapping different client states to different HTTP resources isn't feasible across all of them. Considered use cases include information retrieval technologies: search engines, recommender systems, Q&A systems, and AI assistants.

Considered use cases also include users' AI assistants' tasks. AI assistants could retrieve content using HTTP while providing users with assistance.

New "context management" applications could ship with operating systems or be otherwise available in app stores, perhaps in both productivity and education categories. Users could use these applications to search for local and remote BPM diagrams, to load these, to display these, and to enable interactivity with respect to these, allowing users to select paths through them or nodes in them.

As envisioned, these "context management" applications would write users' task contexts to operating systems. Other software applications, including Web browsers, would read users' task contexts from operating systems and, perhaps, subscribe to receive notifications raised when such data had changed.

Eventually, other software applications, these potentially including webpages in Web browsers, could open tabs in, load diagrams into new tabs, send messages to tabs, and close their opened tabs in interoperable "context management" applications. Messages from other software applications could, depending on loaded BPM diagrams, modify users' task contexts.

[reschke]

I find this really really vague.

Maybe you could supply a concrete example (use case, message flow etc) to clarify why this is needed?

I also do not buy the argument "can't make seperate URIs" - if you can add a new header field and use it for conneg then there's really no reason whatsover that it could not be modeled as separate URIs.

[AdamSobieski]

Ok. I will brainstorm towards a more concrete example. Perhaps I should add to the original post a scenario involving a user utilizing one or more office software apps?

Also, what do you mean by separate URLs or URIs? Do you mean creating standard query-string parameters, like fbclid, and then passing a formatted URL as a value to one such parameter?

[reschke]

a) yes, any example would be useful

b) no; what I meant is really separate resources, and a message flow that uses these separate URLs based on content or metadata (see https://en.wikipedia.org/wiki/HATEOAS) - but maybe it's better to brainstorm on that once there is one concrete example to use

[AdamSobieski]

Towards selecting and revising a concrete example to add to the original post, let's consider a task from the education domain: a college student writing a book report about a chapter of a book.

1. A student opens their "context management" application and, in a tab in it, searches for: "writing a book chapter report".
   1. This application is configured to use a well-known third-party service, e.g., https://en.wikiprocess.org.
2. A number of search results are presented to the student. There is more than one way to do that task.
3. The student selects a search result item which best describes the specific task that they want to perform.
4. The corresponding business process model content (both "backing layer" and "interaction layer") is loaded into their "context management" application.
5. Their "context management" application presents an input form asking: "Enter the title of the book that you are writing a book report about".
6. The student enters "The Odyssey" into this input form.
7. This value becomes the value of a context parameter.
8. Their "context management" application transitions to another "interaction layer" page with an input form asking: "Enter which chapter".
9. The student enter "Book 8" into this input form.
10. This value becomes the value of another context parameter.
11. Next, their "context management" application displays, for this example, a simple sequence of steps for the student to perform.
    1. These steps, using templates or generative AI, contain the book title and chapter number provided by the student.
    2. The "context management" application has educational value as the student can refer to the displayed diagram or flowchart, in this case a simple sequence of steps, at their convenience.
12. The first step in the displayed sequence might be: "Open a word-processing application."
13. The student opens a word-processing application by clicking on a desktop icon.
    1. Alternatively, the student could click upon a hyperlink or a button on that “interaction layer” step and their word-processing application could automatically be opened.
14. Their word-processing application, upon opening, sends a message to their "context management" application in a manner mediated by the operating system.
15. Their "context management" application receives this message and changes the active and highlighted step to the next one in the sequence.
    1. As it does, it notifies the operating system of a task context change event.
16. The student then opens a Web browser.
17. The student navigates the Web browser to their preferred service for performing searches or asking questions, enters a search query or asks a question, and, next, presses enter or a button.
    1. The student has previously allowed this service to access their task context data.
    2. The Web browser has previously subscribed to or at this point retrieves the user's task context from the operating system.
18. Their selected service, importantly any, receives the student's task context data using a HTTP request header.
    1. Their selected service can then glean that the student is: writing a book report about book 8 of The Odyssey.
19. The search results or answers provided are contextual, task-specific, and immediately what the student needs at that current step of their current task.
20. The student uses the search results or answers and starts or continues writing their book report in their word-processing application.

What do you think of this example?

[reschke]

a) You are proposing a new header field named "Context" and start your example with a "Context Management Application"???

b) Do you envision that new header field to be supported by browsers?

Ask yourself: could this be done with a Web Site and a regular browser today? Of course it could. How would it work? Either the site stores the state on the server (HTTP resources), or pushes it into the client (local storage or cookies). Why do we need a new header field to do this?

[AdamSobieski]

As considered, "context management" applications would produce users' task context data while Web browser applications would consume and transmit these data to trusted services in accordance with users' preferences and permissions.

In addition to users' AI assistants, I am hoping that Web browsers would make use of a new HTTP request header for transmitting users' task context data to users' trusted services.

One reason for the considered approach, using a new HTTP request header, is that users' task context data, consumed by any one of multiple Web browser applications, could be transmitted to any one of multiple trusted services, e.g., to any one of multiple search engines, Q&A systems, recommender systems, or AI assistants.

Also, interestingly, under the hood, "context management" applications could reuse a Web browser's components or be built using (embedded) Web browsers.

Accordingly, with respect to multi-layer business process model diagrams discussed, above, "interaction layer" content could be HTML-based and "backing layer" content could be BPMN-based and surfaced to any HTML documents through a JavaScript API to a BPM diagram execution and interaction environment.

As envisioned, this BPM diagram execution and interaction environment component would also have access to a special JavaScript API for interacting with the host application, this API including means of setting or modifying the task context data, the host application then interacting with the operating system, as described above.

The host "context management" application would also be responsible for exchanging messages with other applications, in particular receiving these. Such messaging might be mediated by operating systems. The BPM execution and interaction environment component would, then, also include a means of relaying received messages to loaded BPM diagrams.

In a very preliminary way, the content loaded by a "context management" application could resemble:

<content version="0.1" xmlns="...">
  <layer id="layer-1" kind="interaction" type="application/xhtml+xml">
    ...
  </layer>
  <layer id="layer-2" kind="backing" type="application/bpmn+xml">
    ...
  </layer>
</content>

or:

<content version="0.1" xmlns="...">
  <interaction id="layer-1" type="application/xhtml+xml">
    ...
  </interaction>
  <backing id="layer-2" type="application/bpmn+xml">
    ...
  </backing>
</content>

or:

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <script id="layer-2" type="application/bpmn+xml" src="..." />
    ...
  </head>
  <body>
    ...
  </body>
</html>

[AdamSobieski]

For a simple task involving a sequence of steps, a task context might resemble:

{
  "model": "https://en.wikiprocess.org/wiki/simple-sequence-1.bpmn",
  "at": "step-1"
}

For our example scenario of a college student doing a book report on book 8 of The Odyssey, we can consider that the student's task context (after the student entered data into input textboxes and opened their word-processing application) might resemble:

{
  "model": "https://en.wikiprocess.org/wiki/book-report-3.bpmn",
  "inputs": { "book": "The Odyssey", "chapter": "Book 8" },
  "at": "step-2"
}

This initial conceptualization of the at property is a simplification. Firstly, a business process model diagram could contain one or more portions with parallel flows. Secondly, the content loaded by a context management application could have multiple backing layers.

Brainstorming, the value of the at property could be an array of the unique identifiers or labels of a business process model diagram's currently active nodes.

{
  "model": "https://en.wikiprocess.org/wiki/book-report-3.bpmn",
  "inputs": { "book": "The Odyssey", "chapter": "Book 8" },
  "at": [ "step-2" ]
}

There are also variables to consider. The serialization of context data could involve a property, variables, which, like inputs, could be omitted when a diagram doesn't utilize any.

{
  "model": "https://en.wikiprocess.org/wiki/book-report-3.bpmn",
  "inputs": { "book": "The Odyssey", "chapter": "Book 8" },
  "variables": { "...": "..." },
  "at": [ "step-2" ]
}

Next, we can consider Call Activities (BPMN 2.0.2, p. 182), combining these with hyperlinks, and that values for the at property could be recursive, capable of containing nested task contexts as their array elements, perhaps resembling a hierarchical collection (tree) of call stack frames.

A more complex task context, then, might resemble:

{
  "model": "https://en.wikiprocess.org/wiki/complex-task-1.bpmn",
  "inputs": { "input-1": "value-1" },
  "variables": { "x": 1, "y": 2, "z": 3 },
  "at": [ { "id": "step-7" }, { "id": "step-22", "state": {
    "model": "https://en.wikiprocess.org/wiki/called-subprocess-1.bpmn",
    "inputs": { "input-2": "value-2" },
    "variables": { "w": 4 },
    "at": [ { "id": "step-2" } ]
  } } ]
}