Application Automator

The Automator application is a tool that allows UFO to automate and take actions on applications. Currently, UFO supports two types of actions: UI Automation and API.

Note

UFO can also call in-app AI tools, such as Copilot, to assist with the automation process. This is achieved by using either UI Automation or API to interact with the in-app AI tool.

  • UI Automator - This action type is used to interact with the application's UI controls, such as buttons, text boxes, and menus. UFO uses the UIA or Win32 APIs to interact with the application's UI controls.
  • API - This action type is used to interact with the application's native API. Users and app developers can create their own API actions to interact with specific applications.
  • Web - This action type is used to interact with web applications. UFO uses the crawl4ai library to extract information from web pages.
  • AI Tool - This action type is used to interact with the LLM-based AI tools.

Action Design Patterns

Actions in UFO are implemented using the command design pattern, which encapsulates a receiver, a command, and an invoker. The receiver is the object that performs the action, the command is the object that encapsulates the action, and the invoker is the object that triggers the action.

The basic classes for implementing actions in UFO are as follows:

Role Class Description
Receiver ufo.automator.basic.ReceiverBasic The base class for all receivers in UFO. Receivers are objects that perform actions on applications.
Command ufo.automator.basic.CommandBasic The base class for all commands in UFO. Commands are objects that encapsulate actions to be performed by receivers.
Invoker ufo.automator.puppeteer.AppPuppeteer The base class for the invoker in UFO. Invokers are objects that trigger commands to be executed by receivers.

The advantage of using the command design pattern in the agent framework is that it allows for the decoupling of the sender and receiver of the action. This decoupling enables the agent to execute actions on different objects without knowing the details of the object or the action being performed, making the agent more flexible and extensible for new actions.

Receiver

The Receiver is a central component in the Automator application that performs actions on the application. It provides functionalities to interact with the application and execute the action. All available actions are registered in the with the ReceiverManager class.

You can find the reference for a basic Receiver class below:

Bases: ABC

The abstract receiver interface.

command_registry: Dict[str, Type[CommandBasic]] property

Get the command registry.

supported_command_names: List[str] property

Get the command name list.

register(command_class) classmethod

Decorator to register the state class to the state manager.

Parameters:
  • command_class (Type[CommandBasic]) –

    The state class to be registered.

Returns:
Source code in automator/basic.py
46
47
48
49
50
51
52
53
54
@classmethod
def register(cls, command_class: Type[CommandBasic]) -> Type[CommandBasic]:
    """
    Decorator to register the state class to the state manager.
    :param command_class: The state class to be registered.
    :return: The state class.
    """
    cls._command_registry[command_class.name()] = command_class
    return command_class

register_command(command_name, command)

Add to the command registry.

Parameters:
  • command_name (str) –

    The command name.

  • command (CommandBasic) –

    The command.

Source code in automator/basic.py
24
25
26
27
28
29
30
31
def register_command(self, command_name: str, command: CommandBasic) -> None:
    """
    Add to the command registry.
    :param command_name: The command name.
    :param command: The command.
    """

    self.command_registry[command_name] = command

self_command_mapping()

Get the command-receiver mapping.

Source code in automator/basic.py
40
41
42
43
44
def self_command_mapping(self) -> Dict[str, CommandBasic]:
    """
    Get the command-receiver mapping.
    """
    return {command_name: self for command_name in self.supported_command_names}


Command

The Command is a specific action that the Receiver can perform on the application. It encapsulates the function and parameters required to execute the action. The Command class is a base class for all commands in the Automator application.

You can find the reference for a basic Command class below:

Bases: ABC

The abstract command interface.

Initialize the command.

Parameters:
Source code in automator/basic.py
67
68
69
70
71
72
73
def __init__(self, receiver: ReceiverBasic, params: Dict = None) -> None:
    """
    Initialize the command.
    :param receiver: The receiver of the command.
    """
    self.receiver = receiver
    self.params = params if params is not None else {}

execute() abstractmethod

Execute the command.

Source code in automator/basic.py
75
76
77
78
79
80
@abstractmethod
def execute(self):
    """
    Execute the command.
    """
    pass

redo()

Redo the command.

Source code in automator/basic.py
88
89
90
91
92
def redo(self):
    """
    Redo the command.
    """
    self.execute()

undo()

Undo the command.

Source code in automator/basic.py
82
83
84
85
86
def undo(self):
    """
    Undo the command.
    """
    pass


Note

Each command must register with a specific Receiver to be executed using the register_command decorator. For example: @ReceiverExample.register class CommandExample(CommandBasic): ...

Invoker (AppPuppeteer)

The AppPuppeteer plays the role of the invoker in the Automator application. It triggers the commands to be executed by the receivers. The AppPuppeteer equips the AppAgent with the capability to interact with the application's UI controls. It provides functionalities to translate action strings into specific actions and execute them. All available actions are registered in the Puppeteer with the ReceiverManager class.

You can find the implementation of the AppPuppeteer class in the ufo/automator/puppeteer.py file, and its reference is shown below.

The class for the app puppeteer to automate the app in the Windows environment.

Initialize the app puppeteer.

Parameters:
  • process_name (str) –

    The process name of the app.

  • app_root_name (str) –

    The app root name, e.g., WINWORD.EXE.

Source code in automator/puppeteer.py
22
23
24
25
26
27
28
29
30
31
32
def __init__(self, process_name: str, app_root_name: str) -> None:
    """
    Initialize the app puppeteer.
    :param process_name: The process name of the app.
    :param app_root_name: The app root name, e.g., WINWORD.EXE.
    """

    self._process_name = process_name
    self._app_root_name = app_root_name
    self.command_queue: Deque[CommandBasic] = deque()
    self.receiver_manager = ReceiverManager()

full_path: str property

Get the full path of the process. Only works for COM receiver.

Returns:
  • str

    The full path of the process.

add_command(command_name, params, *args, **kwargs)

Add the command to the command queue.

Parameters:
  • command_name (str) –

    The command name.

  • params (Dict[str, Any]) –

    The arguments.

Source code in automator/puppeteer.py
 93
 94
 95
 96
 97
 98
 99
100
101
102
def add_command(
    self, command_name: str, params: Dict[str, Any], *args, **kwargs
) -> None:
    """
    Add the command to the command queue.
    :param command_name: The command name.
    :param params: The arguments.
    """
    command = self.create_command(command_name, params, *args, **kwargs)
    self.command_queue.append(command)

close()

Close the app. Only works for COM receiver.

Source code in automator/puppeteer.py
144
145
146
147
148
149
150
def close(self) -> None:
    """
    Close the app. Only works for COM receiver.
    """
    com_receiver = self.receiver_manager.com_receiver
    if com_receiver is not None:
        com_receiver.close()

create_command(command_name, params, *args, **kwargs)

Create the command.

Parameters:
  • command_name (str) –

    The command name.

  • params (Dict[str, Any]) –

    The arguments for the command.

Source code in automator/puppeteer.py
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
def create_command(
    self, command_name: str, params: Dict[str, Any], *args, **kwargs
) -> Optional[CommandBasic]:
    """
    Create the command.
    :param command_name: The command name.
    :param params: The arguments for the command.
    """
    receiver = self.receiver_manager.get_receiver_from_command_name(command_name)
    command = receiver.command_registry.get(command_name.lower(), None)

    if receiver is None:
        raise ValueError(f"Receiver for command {command_name} is not found.")

    if command is None:
        raise ValueError(f"Command {command_name} is not supported.")

    return command(receiver, params, *args, **kwargs)

execute_all_commands()

Execute all the commands in the command queue.

Returns:
  • List[Any]

    The execution results.

Source code in automator/puppeteer.py
81
82
83
84
85
86
87
88
89
90
91
def execute_all_commands(self) -> List[Any]:
    """
    Execute all the commands in the command queue.
    :return: The execution results.
    """
    results = []
    while self.command_queue:
        command = self.command_queue.popleft()
        results.append(command.execute())

    return results

execute_command(command_name, params, *args, **kwargs)

Execute the command.

Parameters:
  • command_name (str) –

    The command name.

  • params (Dict[str, Any]) –

    The arguments.

Returns:
  • str

    The execution result.

Source code in automator/puppeteer.py
68
69
70
71
72
73
74
75
76
77
78
79
def execute_command(
    self, command_name: str, params: Dict[str, Any], *args, **kwargs
) -> str:
    """
    Execute the command.
    :param command_name: The command name.
    :param params: The arguments.
    :return: The execution result.
    """

    command = self.create_command(command_name, params, *args, **kwargs)
    return command.execute()

get_command_queue_length()

Get the length of the command queue.

Returns:
  • int

    The length of the command queue.

Source code in automator/puppeteer.py
104
105
106
107
108
109
def get_command_queue_length(self) -> int:
    """
    Get the length of the command queue.
    :return: The length of the command queue.
    """
    return len(self.command_queue)

get_command_string(command_name, params) staticmethod

Generate a function call string.

Parameters:
  • command_name (str) –

    The function name.

  • params (Dict[str, str]) –

    The arguments as a dictionary.

Returns:
  • str

    The function call string.

Source code in automator/puppeteer.py
152
153
154
155
156
157
158
159
160
161
162
163
164
@staticmethod
def get_command_string(command_name: str, params: Dict[str, str]) -> str:
    """
    Generate a function call string.
    :param command_name: The function name.
    :param params: The arguments as a dictionary.
    :return: The function call string.
    """
    # Format the arguments
    args_str = ", ".join(f"{k}={v!r}" for k, v in params.items())

    # Return the function call string
    return f"{command_name}({args_str})"

get_command_types(command_name)

Get the command types.

Parameters:
  • command_name (str) –

    The command name.

Returns:
  • str

    The command types.

Source code in automator/puppeteer.py
53
54
55
56
57
58
59
60
61
62
63
64
65
66
def get_command_types(self, command_name: str) -> str:
    """
    Get the command types.
    :param command_name: The command name.
    :return: The command types.
    """

    try:
        receiver = self.receiver_manager.get_receiver_from_command_name(
            command_name
        )
        return receiver.type_name
    except:
        return ""

save()

Save the current state of the app. Only works for COM receiver.

Source code in automator/puppeteer.py
123
124
125
126
127
128
129
def save(self) -> None:
    """
    Save the current state of the app. Only works for COM receiver.
    """
    com_receiver = self.receiver_manager.com_receiver
    if com_receiver is not None:
        com_receiver.save()

save_to_xml(file_path)

Save the current state of the app to XML. Only works for COM receiver.

Parameters:
  • file_path (str) –

    The file path to save the XML.

Source code in automator/puppeteer.py
131
132
133
134
135
136
137
138
139
140
141
142
def save_to_xml(self, file_path: str) -> None:
    """
    Save the current state of the app to XML. Only works for COM receiver.
    :param file_path: The file path to save the XML.
    """
    com_receiver = self.receiver_manager.com_receiver
    dir_path = os.path.dirname(file_path)
    if not os.path.exists(dir_path):
        os.makedirs(dir_path)

    if com_receiver is not None:
        com_receiver.save_to_xml(file_path)


Receiver Manager

The ReceiverManager manages all the receivers and commands in the Automator application. It provides functionalities to register and retrieve receivers and commands. It is a complementary component to the AppPuppeteer.

The class for the receiver manager.

Initialize the receiver manager.

Source code in automator/puppeteer.py
174
175
176
177
178
179
180
181
182
def __init__(self):
    """
    Initialize the receiver manager.
    """

    self.receiver_registry = {}
    self.ui_control_receiver: Optional[ControlReceiver] = None

    self._receiver_list: List[ReceiverBasic] = []

com_receiver: WinCOMReceiverBasic property

Get the COM receiver.

Returns:
  • WinCOMReceiverBasic

    The COM receiver.

receiver_factory_registry: Dict[str, Dict[str, Union[str, ReceiverFactory]]] property

Get the receiver factory registry.

Returns:
  • Dict[str, Dict[str, Union[str, ReceiverFactory]]]

    The receiver factory registry.

receiver_list: List[ReceiverBasic] property

Get the receiver list.

Returns:
  • List[ReceiverBasic]

    The receiver list.

create_api_receiver(app_root_name, process_name)

Get the API receiver.

Parameters:
  • app_root_name (str) –

    The app root name.

  • process_name (str) –

    The process name.

Source code in automator/puppeteer.py
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
def create_api_receiver(self, app_root_name: str, process_name: str) -> None:
    """
    Get the API receiver.
    :param app_root_name: The app root name.
    :param process_name: The process name.
    """
    for receiver_factory_dict in self.receiver_factory_registry.values():

        # Check if the receiver is API
        if receiver_factory_dict.get("is_api"):
            receiver = receiver_factory_dict.get("factory").create_receiver(
                app_root_name, process_name
            )
            if receiver is not None:
                self.receiver_list.append(receiver)

    self._update_receiver_registry()

create_ui_control_receiver(control, application)

Build the UI controller.

Parameters:
  • control (UIAWrapper) –

    The control element.

Returns:
  • ControlReceiver

    The UI controller receiver.

Source code in automator/puppeteer.py
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
def create_ui_control_receiver(
    self, control: UIAWrapper, application: UIAWrapper
) -> "ControlReceiver":
    """
    Build the UI controller.
    :param control: The control element.
    :return: The UI controller receiver.
    """
    factory: ReceiverFactory = self.receiver_factory_registry.get("UIControl").get(
        "factory"
    )
    self.ui_control_receiver = factory.create_receiver(control, application)
    self.receiver_list.append(self.ui_control_receiver)
    self._update_receiver_registry()

    return self.ui_control_receiver

get_receiver_from_command_name(command_name)

Get the receiver from the command name.

Parameters:
  • command_name (str) –

    The command name.

Returns:
  • ReceiverBasic

    The mapped receiver.

Source code in automator/puppeteer.py
228
229
230
231
232
233
234
235
236
237
def get_receiver_from_command_name(self, command_name: str) -> ReceiverBasic:
    """
    Get the receiver from the command name.
    :param command_name: The command name.
    :return: The mapped receiver.
    """
    receiver = self.receiver_registry.get(command_name, None)
    if receiver is None:
        raise ValueError(f"Receiver for command {command_name} is not found.")
    return receiver

register(receiver_factory_class) classmethod

Decorator to register the receiver factory class to the receiver manager.

Parameters:
  • receiver_factory_class (Type[ReceiverFactory]) –

    The receiver factory class to be registered.

Returns:
  • ReceiverFactory

    The receiver factory class instance.

Source code in automator/puppeteer.py
269
270
271
272
273
274
275
276
277
278
279
280
281
282
@classmethod
def register(cls, receiver_factory_class: Type[ReceiverFactory]) -> ReceiverFactory:
    """
    Decorator to register the receiver factory class to the receiver manager.
    :param receiver_factory_class: The receiver factory class to be registered.
    :return: The receiver factory class instance.
    """

    cls._receiver_factory_registry[receiver_factory_class.name()] = {
        "factory": receiver_factory_class(),
        "is_api": receiver_factory_class.is_api(),
    }

    return receiver_factory_class()


For further details, refer to the specific documentation for each component and class in the Automator module.