Application Automator
The Automator application is a tool that allows UFO to automate and take actions on applications. Currently, UFO supports two types of actions: UI Automation
and API
.
Note
UFO can also call in-app AI tools, such as Copilot
, to assist with the automation process. This is achieved by using either UI Automation
or API
to interact with the in-app AI tool.
- UI Automator - This action type is used to interact with the application's UI controls, such as buttons, text boxes, and menus. UFO uses the UIA or Win32 APIs to interact with the application's UI controls.
- API - This action type is used to interact with the application's native API. Users and app developers can create their own API actions to interact with specific applications.
- Web - This action type is used to interact with web applications. UFO uses the crawl4ai library to extract information from web pages.
- Bash - This action type is used to interact with the command line interface (CLI) of an application.
- AI Tool - This action type is used to interact with the LLM-based AI tools.
Action Design Patterns
Actions in UFO are implemented using the command design pattern, which encapsulates a receiver, a command, and an invoker. The receiver is the object that performs the action, the command is the object that encapsulates the action, and the invoker is the object that triggers the action.
The basic classes for implementing actions in UFO are as follows:
Role | Class | Description |
---|---|---|
Receiver | ufo.automator.basic.ReceiverBasic |
The base class for all receivers in UFO. Receivers are objects that perform actions on applications. |
Command | ufo.automator.basic.CommandBasic |
The base class for all commands in UFO. Commands are objects that encapsulate actions to be performed by receivers. |
Invoker | ufo.automator.puppeteer.AppPuppeteer |
The base class for the invoker in UFO. Invokers are objects that trigger commands to be executed by receivers. |
The advantage of using the command design pattern in the agent framework is that it allows for the decoupling of the sender and receiver of the action. This decoupling enables the agent to execute actions on different objects without knowing the details of the object or the action being performed, making the agent more flexible and extensible for new actions.
Receiver
The Receiver
is a central component in the Automator application that performs actions on the application. It provides functionalities to interact with the application and execute the action. All available actions are registered in the with the ReceiverManager
class.
You can find the reference for a basic Receiver
class below:
Bases: ABC
The abstract receiver interface.
command_registry: Dict[str, Type[CommandBasic]]
property
Get the command registry.
supported_command_names: List[str]
property
Get the command name list.
register(command_class)
classmethod
Decorator to register the state class to the state manager.
Parameters: |
|
---|
Returns: |
|
---|
Source code in automator/basic.py
46 47 48 49 50 51 52 53 54 |
|
register_command(command_name, command)
Add to the command registry.
Parameters: |
|
---|
Source code in automator/basic.py
24 25 26 27 28 29 30 31 |
|
self_command_mapping()
Get the command-receiver mapping.
Source code in automator/basic.py
40 41 42 43 44 |
|
Command
The Command
is a specific action that the Receiver
can perform on the application. It encapsulates the function and parameters required to execute the action. The Command
class is a base class for all commands in the Automator application.
You can find the reference for a basic Command
class below:
Bases: ABC
The abstract command interface.
Initialize the command.
Parameters: |
|
---|
Source code in automator/basic.py
67 68 69 70 71 72 73 |
|
execute()
abstractmethod
Execute the command.
Source code in automator/basic.py
75 76 77 78 79 80 |
|
redo()
Redo the command.
Source code in automator/basic.py
88 89 90 91 92 |
|
undo()
Undo the command.
Source code in automator/basic.py
82 83 84 85 86 |
|
Note
Each command must register with a specific Receiver
to be executed using the register_command
decorator. For example:
@ReceiverExample.register
class CommandExample(CommandBasic):
...
Invoker (AppPuppeteer)
The AppPuppeteer
plays the role of the invoker in the Automator application. It triggers the commands to be executed by the receivers. The AppPuppeteer
equips the AppAgent
with the capability to interact with the application's UI controls. It provides functionalities to translate action strings into specific actions and execute them. All available actions are registered in the Puppeteer
with the ReceiverManager
class.
You can find the implementation of the AppPuppeteer
class in the ufo/automator/puppeteer.py
file, and its reference is shown below.
The class for the app puppeteer to automate the app in the Windows environment.
Initialize the app puppeteer.
Parameters: |
|
---|
Source code in automator/puppeteer.py
22 23 24 25 26 27 28 29 30 31 32 |
|
full_path: str
property
Get the full path of the process. Only works for COM receiver.
Returns: |
|
---|
add_command(command_name, params, *args, **kwargs)
Add the command to the command queue.
Parameters: |
|
---|
Source code in automator/puppeteer.py
94 95 96 97 98 99 100 101 102 103 |
|
close()
Close the app. Only works for COM receiver.
Source code in automator/puppeteer.py
145 146 147 148 149 150 151 |
|
create_command(command_name, params, *args, **kwargs)
Create the command.
Parameters: |
|
---|
Source code in automator/puppeteer.py
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
|
execute_all_commands()
Execute all the commands in the command queue.
Returns: |
|
---|
Source code in automator/puppeteer.py
82 83 84 85 86 87 88 89 90 91 92 |
|
execute_command(command_name, params, *args, **kwargs)
Execute the command.
Parameters: |
|
---|
Returns: |
|
---|
Source code in automator/puppeteer.py
68 69 70 71 72 73 74 75 76 77 78 79 80 |
|
get_command_queue_length()
Get the length of the command queue.
Returns: |
|
---|
Source code in automator/puppeteer.py
105 106 107 108 109 110 |
|
get_command_string(command_name, params)
staticmethod
Generate a function call string.
Parameters: |
|
---|
Returns: |
|
---|
Source code in automator/puppeteer.py
153 154 155 156 157 158 159 160 161 162 163 164 165 |
|
get_command_types(command_name)
Get the command types.
Parameters: |
|
---|
Returns: |
|
---|
Source code in automator/puppeteer.py
53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
|
save()
Save the current state of the app. Only works for COM receiver.
Source code in automator/puppeteer.py
124 125 126 127 128 129 130 |
|
save_to_xml(file_path)
Save the current state of the app to XML. Only works for COM receiver.
Parameters: |
|
---|
Source code in automator/puppeteer.py
132 133 134 135 136 137 138 139 140 141 142 143 |
|
Receiver Manager
The ReceiverManager
manages all the receivers and commands in the Automator application. It provides functionalities to register and retrieve receivers and commands. It is a complementary component to the AppPuppeteer
.
The class for the receiver manager.
Initialize the receiver manager.
Source code in automator/puppeteer.py
175 176 177 178 179 180 181 182 183 |
|
com_receiver: WinCOMReceiverBasic
property
Get the COM receiver.
Returns: |
|
---|
receiver_factory_registry: Dict[str, Dict[str, Union[str, ReceiverFactory]]]
property
Get the receiver factory registry.
Returns: |
|
---|
receiver_list: List[ReceiverBasic]
property
Get the receiver list.
Returns: |
|
---|
create_api_receiver(app_root_name, process_name)
Get the API receiver.
Parameters: |
|
---|
Source code in automator/puppeteer.py
208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 |
|
create_ui_control_receiver(control, application)
Build the UI controller.
Parameters: |
|
---|
Returns: |
|
---|
Source code in automator/puppeteer.py
185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 |
|
get_receiver_from_command_name(command_name)
Get the receiver from the command name.
Parameters: |
|
---|
Returns: |
|
---|
Source code in automator/puppeteer.py
235 236 237 238 239 240 241 242 243 244 |
|
register(receiver_factory_class)
classmethod
Decorator to register the receiver factory class to the receiver manager.
Parameters: |
|
---|
Returns: |
|
---|
Source code in automator/puppeteer.py
276 277 278 279 280 281 282 283 284 285 286 287 288 289 |
|
For further details, refer to the specific documentation for each component and class in the Automator module.