API Automator

UFO currently support the use of Win32 API API automator to interact with the application's native API. We implement them in python using the pywin32 library. The API automator now supports Word and Excel applications, and we are working on extending the support to other applications.

Configuration

There are several configurations that need to be set up before using the API Automator in the config_dev.yaml file. Below is the list of configurations related to the API Automator:

Configuration Option Description Type Default Value
USE_APIS Whether to allow the use of application APIs. Boolean True
APP_API_PROMPT_ADDRESS The prompt address for the application API. Dict {"WINWORD.EXE": "ufo/prompts/apps/word/api.yaml", "EXCEL.EXE": "ufo/prompts/apps/excel/api.yaml", "msedge.exe": "ufo/prompts/apps/web/api.yaml", "chrome.exe": "ufo/prompts/apps/web/api.yaml"}

Note

Only WINWORD.EXE and EXCEL.EXE are currently supported by the API Automator.

Receiver

The base class for the receiver of the API Automator is the WinCOMReceiverBasic class defined in the ufo/automator/app_apis/basic module. It is initialized with the application's win32 com object and provides functionalities to interact with the application's native API. Below is the reference for the WinCOMReceiverBasic class:

Bases: ReceiverBasic

The base class for Windows COM client.

Initialize the Windows COM client.

Parameters:
  • app_root_name (str) –

    The app root name.

  • process_name (str) –

    The process name.

  • clsid (str) –

    The CLSID of the COM object.
Source code in automator/app_apis/basic.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
def __init__(self, app_root_name: str, process_name: str, clsid: str) -> None:
    """
    Initialize the Windows COM client.
    :param app_root_name: The app root name.
    :param process_name: The process name.
    :param clsid: The CLSID of the COM object.
    """

    self.app_root_name = app_root_name
    self.process_name = process_name

    self.clsid = clsid

    self.client = win32com.client.Dispatch(self.clsid)
    self.com_object = self.get_object_from_process_name()

full_path: str property

Get the full path of the process.

Returns:
  • str –

    The full path of the process.

app_match(object_name_list)

Check if the process name matches the app root.

Parameters:
  • object_name_list (List[str]) –

    The list of object name.
Returns:
  • str –

    The matched object name.
Source code in automator/app_apis/basic.py 
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
def app_match(self, object_name_list: List[str]) -> str:
    """
    Check if the process name matches the app root.
    :param object_name_list: The list of object name.
    :return: The matched object name.
    """

    suffix = self.get_suffix_mapping()

    if self.process_name.endswith(suffix):
        clean_process_name = self.process_name[: -len(suffix)]
    else:
        clean_process_name = self.process_name

    if not object_name_list:
        return ""

    return max(
        object_name_list,
        key=lambda x: self.longest_common_substring_length(clean_process_name, x),
    )

close()

Close the app.

Source code in automator/app_apis/basic.py 
110
111
112
113
114
115
116
117
def close(self) -> None:
    """
    Close the app.
    """
    try:
        self.com_object.Close()
    except:
        pass

get_object_from_process_name() abstractmethod

Get the object from the process name.

Source code in automator/app_apis/basic.py 
36
37
38
39
40
41
@abstractmethod
def get_object_from_process_name(self) -> win32com.client.CDispatch:
    """
    Get the object from the process name.
    """
    pass

get_suffix_mapping()

Get the suffix mapping.

Returns:
  • Dict[str, str] –

    The suffix mapping.
Source code in automator/app_apis/basic.py 
43
44
45
46
47
48
49
50
51
52
53
54
55
def get_suffix_mapping(self) -> Dict[str, str]:
    """
    Get the suffix mapping.
    :return: The suffix mapping.
    """
    suffix_mapping = {
        "WINWORD.EXE": "docx",
        "EXCEL.EXE": "xlsx",
        "POWERPNT.EXE": "pptx",
        "olk.exe": "msg",
    }

    return suffix_mapping.get(self.app_root_name, None)

longest_common_substring_length(str1, str2) staticmethod

Get the longest common substring of two strings.

Parameters:
  • str1 (str) –

    The first string.

  • str2 (str) –

    The second string.
Returns:
  • int –

    The length of the longest common substring.
Source code in automator/app_apis/basic.py 
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
@staticmethod
def longest_common_substring_length(str1: str, str2: str) -> int:
    """
    Get the longest common substring of two strings.
    :param str1: The first string.
    :param str2: The second string.
    :return: The length of the longest common substring.
    """

    m = len(str1)
    n = len(str2)

    dp = [[0] * (n + 1) for _ in range(m + 1)]

    max_length = 0

    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if str1[i - 1] == str2[j - 1]:
                dp[i][j] = dp[i - 1][j - 1] + 1
                if dp[i][j] > max_length:
                    max_length = dp[i][j]
            else:
                dp[i][j] = 0

    return max_length

save()

Save the current state of the app.

Source code in automator/app_apis/basic.py 
91
92
93
94
95
96
97
98
def save(self) -> None:
    """
    Save the current state of the app.
    """
    try:
        self.com_object.Save()
    except:
        pass

save_to_xml(file_path)

Save the current state of the app to XML.

Parameters:
  • file_path (str) –

    The file path to save the XML.
Source code in automator/app_apis/basic.py 
100
101
102
103
104
105
106
107
108
def save_to_xml(self, file_path: str) -> None:
    """
    Save the current state of the app to XML.
    :param file_path: The file path to save the XML.
    """
    try:
        self.com_object.SaveAs(file_path, self.xml_format_code)
    except:
        pass

The receiver of Word and Excel applications inherit from the WinCOMReceiverBasic class. The WordReceiver and ExcelReceiver classes are defined in the ufo/automator/app_apis/word and ufo/automator/app_apis/excel modules, respectively:

Command

The command of the API Automator for the Word and Excel applications in located in the client module in the ufo/automator/app_apis/{app_name} folder inheriting from the WinCOMCommand class. It encapsulates the function and parameters required to execute the action. Below is an example of a WordCommand class that inherits from the SelectTextCommand class:

@WordWinCOMReceiver.register
class SelectTextCommand(WinCOMCommand):
    """
    The command to select text.
    """

    def execute(self):
        """
        Execute the command to select text.
        :return: The selected text.
        """
        return self.receiver.select_text(self.params.get("text"))

    @classmethod
    def name(cls) -> str:
        """
        The name of the command.
        """
        return "select_text"

Note

The concrete command classes must implement the execute method to execute the action and the name method to return the name of the atomic command.

Note

Each command must register with a concrete WinCOMReceiver to be executed using the register decorator.

Below is the list of available commands in the API Automator that are currently supported by UFO:

Word API Commands

Command Name Function Name Description
InsertTableCommand insert_table Insert a table to a Word document.
SelectTextCommand select_text Select the text in a Word document.
SelectTableCommand select_table Select a table in a Word document.

Excel API Commands

Command Name Function Name Description
GetSheetContentCommand get_sheet_content Get the content of a sheet in the Excel app.
Table2MarkdownCommand table2markdown Convert the table content in a sheet of the Excel app to markdown format.
InsertExcelTableCommand insert_excel_table Insert a table to the Excel sheet.

Tip

Please refer to the ufo/prompts/apps/{app_name}/api.yaml file for the prompt details for the commands.

Tip

You can customize the commands by adding new command classes to the ufo/automator/app_apis/{app_name}/ module.