Control Filtering

There may be many controls items in the application, which may not be relevant to the task. UFO can filter out the irrelevant controls and only focus on the relevant ones. This filtering process can reduce the complexity of the task.

Execept for configuring the control types for selection on CONTROL_LIST in config_dev.yaml, UFO also supports filtering the controls based on semantic similarity or keyword matching between the agent's plan and the control's information. We currerntly support the following filtering methods:

Filtering Method Description
Text Filter the controls based on the control text.
Semantic Filter the controls based on the semantic similarity.
Icon Filter the controls based on the control icon image.

Configuration

You can activate the control filtering by setting the CONTROL_FILTER in the config_dev.yaml file. The CONTROL_FILTER is a list of filtering methods that you want to apply to the controls, which can be TEXT, SEMANTIC, or ICON.

You can configure multiple filtering methods in the CONTROL_FILTER list.

Reference

The implementation of the control filtering is base on the BasicControlFilter class located in the ufo/automator/ui_control/control_filter.py file. Concrete filtering class inherit from the BasicControlFilter class and implement the control_filter method to filter the controls based on the specific filtering method.

BasicControlFilter represents a model for filtering control items.

__new__(model_path)

Creates a new instance of BasicControlFilter.

Parameters:
  • model_path –

    The path to the model.
Returns:
  • –

    The BasicControlFilter instance.
Source code in automator/ui_control/control_filter.py 
72
73
74
75
76
77
78
79
80
81
82
def __new__(cls, model_path):
    """
    Creates a new instance of BasicControlFilter.
    :param model_path: The path to the model.
    :return: The BasicControlFilter instance.
    """
    if model_path not in cls._instances:
        instance = super(BasicControlFilter, cls).__new__(cls)
        instance.model = cls.load_model(model_path)
        cls._instances[model_path] = instance
    return cls._instances[model_path]

control_filter(control_dicts, plans, **kwargs) abstractmethod

Calculates the cosine similarity between the embeddings of the given keywords and the control item.

Parameters:
  • control_dicts –

    The control item to be compared with the plans.

  • plans –

    The plans to be used for calculating the similarity.
Returns:
  • –

    The filtered control items.
Source code in automator/ui_control/control_filter.py 
104
105
106
107
108
109
110
111
112
@abstractmethod
def control_filter(self, control_dicts, plans, **kwargs):
    """
    Calculates the cosine similarity between the embeddings of the given keywords and the control item.
    :param control_dicts: The control item to be compared with the plans.
    :param plans: The plans to be used for calculating the similarity.
    :return: The filtered control items.
    """
    pass

cos_sim(embedding1, embedding2) staticmethod

Computes the cosine similarity between two embeddings.

Parameters:
  • embedding1 –

    The first embedding.

  • embedding2 –

    The second embedding.
Returns:
  • float –

    The cosine similarity between the two embeddings.
Source code in automator/ui_control/control_filter.py 
153
154
155
156
157
158
159
160
161
162
163
@staticmethod
def cos_sim(embedding1, embedding2) -> float:
    """
    Computes the cosine similarity between two embeddings.
    :param embedding1: The first embedding.
    :param embedding2: The second embedding.
    :return: The cosine similarity between the two embeddings.
    """
    import sentence_transformers

    return sentence_transformers.util.cos_sim(embedding1, embedding2)

get_embedding(content)

Encodes the given object into an embedding.

Parameters:
  • content –

    The content to encode.
Returns:
  • –

    The embedding of the object.
Source code in automator/ui_control/control_filter.py 
 95
 96
 97
 98
 99
100
101
102
def get_embedding(self, content):
    """
    Encodes the given object into an embedding.
    :param content: The content to encode.
    :return: The embedding of the object.
    """

    return self.model.encode(content)

load_model(model_path) staticmethod

Loads the model from the given model path.

Parameters:
  • model_path –

    The path to the model.
Returns:
  • –

    The loaded model.
Source code in automator/ui_control/control_filter.py 
84
85
86
87
88
89
90
91
92
93
@staticmethod
def load_model(model_path):
    """
    Loads the model from the given model path.
    :param model_path: The path to the model.
    :return: The loaded model.
    """
    import sentence_transformers

    return sentence_transformers.SentenceTransformer(model_path)

plans_to_keywords(plans) staticmethod

Gets keywords from the plan. We only consider the words in the plan that are alphabetic or Chinese characters.

Parameters:
  • plans (List[str]) –

    The plan to be parsed.
Returns:
  • List[str] –

    A list of keywords extracted from the plan.
Source code in automator/ui_control/control_filter.py 
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
@staticmethod
def plans_to_keywords(plans: List[str]) -> List[str]:
    """
    Gets keywords from the plan. We only consider the words in the plan that are alphabetic or Chinese characters.
    :param plans: The plan to be parsed.
    :return: A list of keywords extracted from the plan.
    """

    keywords = []
    for plan in plans:
        words = plan.replace("'", "").strip(".").split()
        words = [
            word
            for word in words
            if word.isalpha() or bool(re.fullmatch(r"[\u4e00-\u9fa5]+", word))
        ]
        keywords.extend(words)
    return keywords

remove_stopwords(keywords) staticmethod

Removes stopwords from the given list of keywords. If you are using stopwords for the first time, you need to download them using nltk.download('stopwords').

Parameters:
  • keywords –

    The list of keywords to be filtered.
Returns:
  • –

    The list of keywords with the stopwords removed.
Source code in automator/ui_control/control_filter.py 
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
@staticmethod
def remove_stopwords(keywords):
    """
    Removes stopwords from the given list of keywords. If you are using stopwords for the first time, you need to download them using nltk.download('stopwords').
    :param keywords: The list of keywords to be filtered.
    :return: The list of keywords with the stopwords removed.
    """

    try:
        from nltk.corpus import stopwords

        stopwords_list = stopwords.words("english")
    except LookupError as e:
        import nltk

        nltk.download("stopwords")
        stopwords_list = nltk.corpus.stopwords.words("english")

    return [keyword for keyword in keywords if keyword in stopwords_list]