Here is the polished document for your Python code project:
Learning from User Demonstration
For complex tasks, users can demonstrate the task using Step Recorder to record the action trajectories. UFO can learn from these user demonstrations to improve the AppAgent's performance.
Mechanism
UFO use the Step Recorder tool to record the task and action trajectories. The recorded demonstration is saved as a zip file. The DemonstrationSummarizer
class extracts and summarizes the demonstration. The summarized demonstration is saved in the DEMONSTRATION_SAVED_PATH
as specified in the config_dev.yaml
file. When the AppAgent encounters a similar task, the DemonstrationRetriever
class retrieves the saved demonstration from the demonstration database and generates a plan based on the retrieved demonstration.
Info
You can find how to record the task and action trajectories using the Step Recorder tool in the User Demonstration Provision document.
You can find a demo video of learning from user demonstrations:
Activating Learning from User Demonstrations
Step 1: User Demonstration
Please follow the steps in the User Demonstration Provision document to provide user demonstrations.
Step 2: Configure the AppAgent
Configure the following parameters to allow UFO to use RAG from user demonstrations:
Configuration Option | Description | Type | Default Value |
---|---|---|---|
RAG_DEMONSTRATION |
Whether to use RAG from user demonstrations | Boolean | False |
RAG_DEMONSTRATION_RETRIEVED_TOPK |
The top K documents to retrieve offline | Integer | 5 |
RAG_DEMONSTRATION_COMPLETION_N |
The number of completion choices for the demonstration result | Integer | 3 |
Reference
Demonstration Summarizer
The DemonstrationSummarizer
class is located in the record_processor/summarizer/summarizer.py
file. The DemonstrationSummarizer
class provides methods to summarize the demonstration:
The DemonstrationSummarizer class is the summarizer for the demonstration learning. It summarizes the demonstration record to a list of summaries, and save the summaries to the YAML file and the vector database. A sample of the summary is as follows: { "example": { "Observation": "Word.exe is opened.", "Thought": "The user is trying to create a new file.", "ControlLabel": "1", "ControlText": "Sample Control Text", "Function": "CreateFile", "Args": "filename='new_file.txt'", "Status": "Success", "Plan": "Create a new file named 'new_file.txt'.", "Comment": "The user successfully created a new file." }, "Tips": "You can use the 'CreateFile' function to create a new file." }
Initialize the DemonstrationSummarizer.
Parameters: |
|
---|
Source code in summarizer/summarizer.py
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
|
__build_prompt(demo_record)
Build the prompt by the user demonstration record.
Parameters: |
|
---|
Source code in summarizer/summarizer.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
|
__parse_response(response_string)
Parse the response string to a dict of summary.
Parameters: |
|
---|
Source code in summarizer/summarizer.py
105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 |
|
create_or_update_vector_db(summaries, db_path)
staticmethod
Create or update the vector database.
Parameters: |
|
---|
Source code in summarizer/summarizer.py
171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 |
|
create_or_update_yaml(summaries, yaml_path)
staticmethod
Create or update the YAML file.
Parameters: |
|
---|
Source code in summarizer/summarizer.py
136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 |
|
get_summary_list(record)
Get the summary list for a record
Parameters: |
|
---|
Source code in summarizer/summarizer.py
60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 |
|
Demonstration Retriever
The DemonstrationRetriever
class is located in the rag/retriever.py
file. The DemonstrationRetriever
class provides methods to retrieve the demonstration:
Bases: Retriever
Class to create demonstration retrievers.
Create a new DemonstrationRetriever. :db_path: The path to the database.
Source code in rag/retriever.py
197 198 199 200 201 202 |
|
get_indexer(db_path)
Create a demonstration indexer. :db_path: The path to the database.
Source code in rag/retriever.py
204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 |
|