Skip to content

Commit

Permalink
fix doc
Browse files Browse the repository at this point in the history
  • Loading branch information
vyokky committed Jul 5, 2024
1 parent 6686587 commit ba4c7c3
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 32 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,44 +8,18 @@ For complex tasks, users can demonstrate the task using [Step Recorder](https://

## Mechanism

### Step 1: Record the Task
- **Tool**: [Step Recorder](https://support.microsoft.com/en-us/windows/record-steps-to-reproduce-a-problem-46582a9b-620f-2e36-00c9-04e25d784e47)
- **Output**: Zip file containing the task description and action trajectories

### Step 2: Save the Demonstration
- **Action**: Save the recorded demonstration as a zip file

### Step 3: Extract and Summarize the Demonstration
- **Tool**: `DemonstrationSummarizer`
- **Process**:
1. Extract the zip file
2. Summarize the demonstration
- **Configuration**: Save the summarized demonstration in the `DEMONSTRATION_SAVED_PATH` as specified in the `config_dev.yaml` file

### Step 4: Retrieve and Utilize the Demonstration
- **When**: AppAgent encounters a similar task
- **Action**: Retrieve the saved demonstration from the demonstration database
- **Tool**: `DemonstrationRetriever`
- **Outcome**: Generate a plan based on the retrieved demonstration

### Demonstration Workflow Diagram
```mermaid
graph TD;
A[User Records Task] --> B[Save as Zip File]
B --> C[Extract Zip File]
C --> D[Summarize with DemonstrationSummarizer]
D --> E[Save in DEMONSTRATION_SAVED_PATH]
F[AppAgent Encounters Similar Task] --> G[Retrieve Demonstration from Database]
G --> H[Generate Plan]
```
UFO use the [Step Recorder](https://support.microsoft.com/en-us/windows/record-steps-to-reproduce-a-problem-46582a9b-620f-2e36-00c9-04e25d784e47) tool to record the task and action trajectories. The recorded demonstration is saved as a zip file. The `DemonstrationSummarizer` class extracts and summarizes the demonstration. The summarized demonstration is saved in the `DEMONSTRATION_SAVED_PATH` as specified in the `config_dev.yaml` file. When the AppAgent encounters a similar task, the `DemonstrationRetriever` class retrieves the saved demonstration from the demonstration database and generates a plan based on the retrieved demonstration.

!!! info
You can find how to record the task and action trajectories using the Step Recorder tool in the [User Demonstration Provision](../../creating_app_agent/demonstration_provision.md) document.


You can find a demo video of learning from user demonstrations:

<iframe width="560" height="315" src="https://github.com/yunhao0204/UFO/assets/59384816/0146f83e-1b5e-4933-8985-fe3f24ec4777" frameborder="0" allowfullscreen></iframe>

<br>

<br>

## Activating Learning from User Demonstrations

Expand Down
1 change: 0 additions & 1 deletion ufo/automator/ui_control/controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@

from ufo.automator.basic import CommandBasic, ReceiverBasic, ReceiverFactory
from ufo.config.config import Config
from ufo.prompter.agent_prompter import APIPromptLoader
from ufo.utils import print_with_color

configs = Config.get_instance().config_data
Expand Down

0 comments on commit ba4c7c3

Please sign in to comment.