Icon Filter

The icon control filter is a method to filter the controls based on the similarity between the control icon image and the agent's plan using the image/text embeddings.

Configuration

To activate the icon control filtering, you need to add ICON to the CONTROL_FILTER list in the config_dev.yaml file. Below is the detailed icon control filter configuration in the config_dev.yaml file:

  • CONTROL_FILTER: A list of filtering methods that you want to apply to the controls. To activate the icon control filtering, add ICON to the list.
  • CONTROL_FILTER_TOP_K_ICON: The number of controls to keep after filtering.
  • CONTROL_FILTER_MODEL_ICON_NAME: The control filter model name for icon similarity. By default, it is set to "clip-ViT-B-32".

Reference

Bases: BasicControlFilter

A class that represents a icon model for control filtering.

control_filter(control_dicts, cropped_icons_dict, plans, top_k)

Filters control items based on their scores and returns the top-k items.

Parameters:
  • control_dicts –

    The dictionary of all control items.

  • cropped_icons_dict –

    The dictionary of the cropped icons.

  • plans –

    The plans to compare the control icons against.

  • top_k –

    The number of top items to return.
Returns:
  • –

    The list of top-k control items based on their scores.
Source code in automator/ui_control/control_filter.py 
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
def control_filter(self, control_dicts, cropped_icons_dict, plans, top_k):
    """
    Filters control items based on their scores and returns the top-k items.
    :param control_dicts: The dictionary of all control items.
    :param cropped_icons_dict: The dictionary of the cropped icons.
    :param plans: The plans to compare the control icons against.
    :param top_k: The number of top items to return.
    :return: The list of top-k control items based on their scores.
    """

    scores_items = []
    filtered_control_dict = {}

    for label, cropped_icon in cropped_icons_dict.items():
        score = self.control_filter_score(cropped_icon, plans)
        scores_items.append((score, label))
    topk_scores_items = heapq.nlargest(top_k, scores_items, key=lambda x: x[0])
    topk_labels = [scores_items[1] for scores_items in topk_scores_items]

    for label, control_item in control_dicts.items():
        if label in topk_labels:
            filtered_control_dict[label] = control_item
    return filtered_control_dict

control_filter_score(control_icon, plans)

Calculates the score of a control icon based on its similarity to the given keywords.

Parameters:
  • control_icon –

    The control icon image.

  • plans –

    The plan to compare the control icon against.
Returns:
  • –

    The maximum similarity score between the control icon and the keywords.
Source code in automator/ui_control/control_filter.py 
240
241
242
243
244
245
246
247
248
249
250
def control_filter_score(self, control_icon, plans):
    """
    Calculates the score of a control icon based on its similarity to the given keywords.
    :param control_icon: The control icon image.
    :param plans: The plan to compare the control icon against.
    :return: The maximum similarity score between the control icon and the keywords.
    """

    plans_embedding = self.get_embedding(plans)
    control_icon_embedding = self.get_embedding(control_icon)
    return max(self.cos_sim(control_icon_embedding, plans_embedding).tolist()[0])