WindowsAppEnv

The usage scenarios for the WindowsAppEnv class are as follows:

• Opening a specified document.
• Matching document windows using different strategies (contains, fuzzy, and regex).
• Matching the controls required for each step in the instantiated plan using various strategies (contains, fuzzy, and regex).
• Closing a specified document.

The following sections provide a detailed explanation of the matching strategies for windows and controls, as well as their usage methods.

Matching Strategies

In the WindowsAppEnv class, matching strategies are rules that determine how to match window or control names with a given document name or target text. Based on the configuration file, three different matching strategies can be selected: contains, fuzzy, and regex.

• Contains matching is the simplest strategy, suitable when the window and document names match exactly.
• Fuzzy matching is more flexible and can match even when there are spelling errors or partial matches between the window title and document name.
• Regex matching offers the most flexibility, ideal for complex matching patterns in window titles.

1. Window Matching Example

The method find_matching_window is responsible for matching windows based on the configured matching strategy. Here's how you can use it to find a window by providing a document name:

Example:

# Initialize your application object (assuming app_object is already defined)
app_env = WindowsAppEnv(app_object)

# Define the document name you're looking for
doc_name = "example_document_name"

# Call find_matching_window to find the window that matches the document name
matching_window = app_env.find_matching_window(doc_name)

if matching_window:
    print(f"Found matching window: {matching_window.element_info.name}")
else:
    print("No matching window found.")

Explanation:

• app_env.find_matching_window(doc_name) will search through all open windows and match the window title using the strategy defined in the configuration (contains, fuzzy, or regex).
• If a match is found, the matching_window object will contain the matched window, and you can print the window's name.
• If no match is found, it will return None.

2. Control Matching Example

To find a matching control within a window, you can use the find_matching_controller method. This method requires a dictionary of filtered controls and a control text to match against.

Example:

# Initialize your application object (assuming app_object is already defined)
app_env = WindowsAppEnv(app_object)

# Define a filtered annotation dictionary of controls (control_key, control_object)
# Here, we assume you have a dictionary of UIAWrapper controls from a window.
filtered_annotation_dict = {
    1: some_control_1,  # Example control objects
    2: some_control_2,  # Example control objects
}

# Define the control text you're searching for
control_text = "submit_button"

# Call find_matching_controller to find the best match
controller_key, control_selected = app_env.find_matching_controller(filtered_annotation_dict, control_text)

if control_selected:
    print(f"Found matching control with key {controller_key}: {control_selected.window_text()}")
else:
    print("No matching control found.")

Explanation:

• filtered_annotation_dict is a dictionary where the key represents the control's ID and the value is the control object (UIAWrapper).
• control_text is the text you're searching for within those controls.
• app_env.find_matching_controller(filtered_annotation_dict, control_text) will calculate the matching score for each control based on the defined strategy and return the control with the highest match score.
• If a match is found, it will return the control object (control_selected) and its key (controller_key), which can be used for further interaction.

Reference

29
30
31
32
33
34
35
36
37
38

def __init__(self, app_object: object) -> None:
    """
    Initializes the Windows Application Environment.
    :param app_object: The app object containing information about the application.
    """

    self.app_window = None
    self.app_root_name = app_object.app_root_name
    self.app_name = app_object.description.lower()
    self.win_app = app_object.win_app

57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

def close(self) -> None:
    """
    Tries to gracefully close the application; if it fails or is not closed, forcefully terminates the process.
    """

    try:
        # Gracefully close the application window
        if self.app_window and self.app_window.process_id():
            self.app_window.close()
        sleep(1)
        # Forcefully close the application window  
        if self.app_window.element_info.name.lower() != '':            
            self._check_and_kill_process()
    except Exception as e:
        logging.warning(f"Graceful close failed: {e}. Attempting to forcefully terminate the process.")
        self._check_and_kill_process()
        raise e

155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177

def find_matching_controller(self, filtered_annotation_dict: Dict[int, UIAWrapper], control_text: str) -> Tuple[str, UIAWrapper]:
    """"
    Select the best matched controller.
    :param filtered_annotation_dict: The filtered annotation dictionary.
    :param control_text: The text content of the control for additional context.
    :return: Tuple containing the key of the selected controller and the control object.s
    """
    control_selected = None
    controller_key = None
    highest_score = 0

    # Iterate through the filtered annotation dictionary to find the best match
    for key, control in filtered_annotation_dict.items():
        # Calculate the matching score using the match function
        score = self._calculate_match_score(control, control_text)

        # Update the selected control if the score is higher
        if score > highest_score:
            highest_score = score
            controller_key = key
            control_selected = control

    return controller_key, control_selected

 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103

def find_matching_window(self, doc_name: str) -> Optional[UIAWrapper]:
    """
    Finds a matching window based on the process name and the configured matching strategy.
    :param doc_name: The document name associated with the application.
    :return: The matched window or None if no match is found.
    """

    desktop = Desktop(backend=_BACKEND)
    windows_list = desktop.windows()
    for window in windows_list:
        window_title = window.element_info.name.lower()
        if self._match_window_name(window_title, doc_name):
            self.app_window = window
            return window
    return None

40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

def start(self, copied_template_path: str) -> None:
    """
    Starts the Windows environment.
    :param copied_template_path: The file path to the copied template to start the environment.
    """

    from ufo.automator.ui_control import openfile

    file_controller = openfile.FileController(_BACKEND)
    try:
        file_controller.execute_code(
            {"APP": self.win_app, "file_path": copied_template_path}
        )
    except Exception as e:
        logging.exception(f"Failed to start the application: {e}")
        raise