UFO² as UFO³ Galaxy Device

Integrate UFO² (Windows Desktop Automation Agent) into the UFO³ Galaxy framework as a managed sub-agent device. This enables Galaxy to orchestrate complex cross-platform workflows combining Windows desktop automation with Linux server operations and other heterogeneous devices.

Overview

UFO² can function as a device agent within the UFO³ Galaxy multi-tier orchestration framework. When configured as a Galaxy device, UFO² operates in server-client mode, allowing the Galaxy ConstellationAgent to:

• Dispatch Windows automation subtasks to UFO² devices
• Coordinate cross-platform workflows (Windows desktop + Linux servers)
• Leverage UFO²'s HostAgent and AppAgent capabilities at scale
• Manage multiple Windows devices from a unified control plane
• Dynamically select devices based on capabilities and installed applications

UFO² integration follows the server-client architecture pattern where the UFO² Server manages task orchestration and state machines, the UFO² Client executes Windows automation commands via MCP tools, and the Galaxy ConstellationAgent acts as the top-level orchestrator. Communication is enabled through the Agent Interaction Protocol (AIP). For detailed architecture information, see Server-Client Architecture.

Galaxy Integration Architecture

Example Multi-Device Workflow:

User Request: "Generate a sales report from the database, create an Excel dashboard, and email it to the team"

Galaxy orchestrates:

1. Linux DB Server: Extract sales data from PostgreSQL → CSV export
2. UFO² Desktop: Open Excel, import CSV, create visualizations and pivot tables
3. UFO² Desktop: Open Outlook, compose email with Excel attachment
4. UFO² Desktop: Send email to distribution list

Prerequisites

Before configuring UFO² as a Galaxy device, ensure you have:

Configure Agent Configuration

Before proceeding with Galaxy integration, you must configure your agent settings in config/ufo/agents.yaml:

1. Copy the template file:

   Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
   

2. Configure your LLM provider (OpenAI, Azure OpenAI, etc.) and add API keys

Without proper agent configuration, UFO² cannot function as a Galaxy device. See Agents Configuration Guide for detailed setup instructions.

Server-Client Mode Setup

UFO² must operate in server-client mode when integrated into Galaxy. This architecture separates orchestration (server) from execution (client), enabling Galaxy to manage multiple UFO² devices efficiently. Unlike standalone UFO² usage (local mode), Galaxy integration requires running UFO² in distributed server-client mode to ensure Galaxy can communicate with UFO² via Agent Interaction Protocol (AIP), multiple UFO² clients can be managed by a single server, task state is managed server-side for reliability, and clients remain stateless execution endpoints.

Step 1: Start UFO² Server

The UFO² Server handles task orchestration, state management, and LLM-driven decision-making. It communicates with Galaxy and dispatches commands to UFO² clients.

Basic Server Startup

Launch UFO² Server on the machine that will host the server (can be any Windows/Linux machine):

python -m ufo.server.app --port 5000

Expected Output:

2025-11-06 10:30:22 - ufo.server.app - INFO - Starting UFO Server on 0.0.0.0:5000
INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)

Once you see "Uvicorn running", the server is ready at ws://0.0.0.0:5000/ws.

Server Configuration Options

Examples:

Specific port:

python -m ufo.server.app --port 5000

Specific IP binding:

python -m ufo.server.app --host 192.168.1.100 --port 5000

Debug mode:

python -m ufo.server.app --port 5000 --log-level DEBUG

Verify Server Health

# Test server health endpoint
curl http://localhost:5000/api/health

Expected Response:

{
  "status": "healthy",
  "online_clients": []
}

Step 2: Start UFO² Client (Windows Machine)

The UFO² Client runs on the Windows machine where you want to perform desktop automation. It connects to the UFO² server via WebSocket and executes automation commands through MCP tools.

Basic Client Startup

Connect UFO² Client to Server on the Windows machine where you want to run desktop automation:

python -m ufo.client.client `
  --ws `
  --ws-server ws://192.168.1.100:5000/ws `
  --client-id ufo2_desktop_1 `
  --platform windows

Note: In PowerShell, use backtick ` for line continuation. In Command Prompt, use ^.

Client Parameters Explained

Important: - --client-id must be globally unique - No two devices can share the same ID - --platform windows is mandatory - Without this flag, UFO² won't work correctly - Server address must be correct - Replace 192.168.1.100:5000 with your actual server IP and port

Understanding the WebSocket URL

The --ws-server parameter format is:

ws://<server-ip>:<server-port>/ws

Examples:

Connection Success Indicators

Client Logs:

INFO - Platform detected/specified: windows
INFO - UFO Client initialized for platform: windows
INFO - [WS] Connecting to ws://192.168.1.100:5000/ws (attempt 1/5)
INFO - [WS] [AIP] Successfully registered as ufo2_desktop_1
INFO - [WS] Heartbeat loop started (interval: 30s)

Server Logs:

INFO - [WS] ✅ Registered device client: ufo2_desktop_1
INFO - [WS] Device ufo2_desktop_1 platform: windows

When you see "Successfully registered", the UFO² client is connected and ready to receive tasks.

Verify Connection

# Check connected clients on server
curl http://192.168.1.100:5000/api/clients

Expected Response:

{
  "clients": [
    {
      "client_id": "ufo2_desktop_1",
      "type": "device",
      "platform": "windows",
      "connected_at": 1730899822.0,
      "uptime_seconds": 45
    }
  ]
}

Step 3: Configure MCP Services

UFO² relies on MCP (Model Context Protocol) servers to provide Windows automation capabilities. Unlike Linux agents that may require separate HTTP MCP servers, UFO² MCP servers are primarily local and start automatically with the client.

UFO² uses local MCP servers that run in-process with the client:

• UI Automation MCP: Click, type, screenshot, control detection
• File Operations MCP: Read, write, copy, delete files
• Application Control MCP: Launch apps, switch windows, close processes

These are automatically initialized when the UFO² client starts.

Default MCP Configuration

By default, UFO² client automatically starts all necessary local MCP servers. No additional configuration is required for standard Windows automation.

When you start the UFO² client, it automatically initializes UI automation tools, registers file operation handlers, configures application control interfaces, and sets up screenshot and OCR capabilities.

Optional: HTTP MCP Server (Advanced)

For specialized scenarios requiring remote MCP access (e.g., hardware automation via external tools), you can optionally start HTTP-based MCP servers. However, note that there is no windows_mcp_server.py in the codebase. Available HTTP MCP servers are:

• hardware_mcp_server.py - For hardware-level operations
• linux_mcp_server.py - For Linux-specific operations

Start an HTTP MCP server if needed:

python -m ufo.client.mcp.http_servers.hardware_mcp_server

Note: For standard Galaxy integration with UFO², local MCP servers are sufficient and HTTP MCP servers are not required.

Step 4: Configure as Galaxy Device

To integrate UFO² into the Galaxy framework, register it in the Galaxy device configuration file.

Device Configuration File

The Galaxy device pool is configured in config/galaxy/devices.yaml.

Add UFO² Device Configuration

Edit config/galaxy/devices.yaml and add your UFO² device(s) under the devices section:

devices:
  - device_id: "ufo2_desktop_1"
    server_url: "ws://192.168.1.100:5000/ws"
    os: "windows"
    capabilities:
      - "desktop_automation"
      - "office_applications"
      - "web_browsing"
      - "email"
      - "file_management"
    metadata:
      os: "windows"
      version: "11"
      performance: "high"
      installed_apps:
        - "Microsoft Excel"
        - "Microsoft Word"
        - "Microsoft PowerPoint"
        - "Microsoft Outlook"
        - "Google Chrome"
        - "Adobe Acrobat"
      description: "Primary office workstation for document automation"
    auto_connect: true
    max_retries: 5

Configuration Fields Explained

Capabilities-Based Task Routing

Galaxy uses the capabilities field to intelligently route subtasks to appropriate UFO² devices. Define capabilities based on application categories (e.g., "office_applications", "web_browsing"), task types (e.g., "desktop_automation", "data_entry"), specific software (e.g., "excel", "outlook"), and user workflows (e.g., "email", "reporting").

Example capability configurations:

Office Workstation:

capabilities:
  - "desktop_automation"
  - "office_applications"
  - "excel"
  - "word"
  - "powerpoint"
  - "outlook"
  - "email"
  - "reporting"

Web Development Machine:

capabilities:
  - "desktop_automation"
  - "web_browsing"
  - "chrome"
  - "visual_studio_code"
  - "git"
  - "development"

Testing Workstation:

capabilities:
  - "desktop_automation"
  - "ui_testing"
  - "web_browsing"
  - "screenshot_comparison"
  - "quality_assurance"

Media Production:

capabilities:
  - "desktop_automation"
  - "media_editing"
  - "photoshop"
  - "premiere"
  - "video_processing"
  - "image_manipulation"

The metadata field provides contextual information that the LLM can use when generating automation commands.

Metadata Examples:

Office Workstation Metadata:

metadata:
  os: "windows"
  version: "11"
  performance: "high"
  installed_apps:
    - "Microsoft Excel"
    - "Microsoft Word"
    - "Microsoft Outlook"
    - "Adobe Acrobat Reader"
  default_paths:
    documents: "C:\\Users\\user\\Documents"
    downloads: "C:\\Users\\user\\Downloads"
    desktop: "C:\\Users\\user\\Desktop"
  email_account: "user@company.com"
  description: "Primary office workstation"

Development Workstation Metadata:

metadata:
  os: "windows"
  version: "11"
  performance: "high"
  installed_apps:
    - "Visual Studio Code"
    - "Google Chrome"
    - "Git"
    - "Node.js"
    - "Python"
  default_paths:
    projects: "C:\\Users\\dev\\Projects"
    repos: "C:\\Users\\dev\\Repos"
  git_username: "developer"
  description: "Development environment"

Testing Workstation Metadata:

metadata:
  os: "windows"
  version: "10"
  performance: "medium"
  installed_apps:
    - "Google Chrome"
    - "Microsoft Edge"
    - "Firefox"
    - "Selenium"
  test_data_path: "C:\\TestData"
  screenshot_path: "C:\\Screenshots"
  description: "Automated testing environment"

How Metadata is Used:

The LLM receives metadata in the system prompt, enabling context-aware automation:

System Context:
- Device: ufo2_desktop_1
- OS: Windows 11
- Installed Apps: Microsoft Excel, Microsoft Word, Microsoft Outlook
- Documents Path: C:\Users\user\Documents

User Request: "Create a new Excel spreadsheet and save it as Q4_Report.xlsx"

UFO² Output: 
1. Launch Microsoft Excel
2. Create new workbook
3. Save as C:\Users\user\Documents\Q4_Report.xlsx

Step 5: Multiple UFO² Devices Configuration

Galaxy can manage multiple UFO² devices simultaneously, enabling parallel Windows automation across different machines.

Multi-Device Galaxy Configuration Example:

devices:
  # UFO² Office Desktop 1
  - device_id: "ufo2_office_1"
    server_url: "ws://192.168.1.100:5000/ws"
    os: "windows"
    capabilities:
      - "desktop_automation"
      - "office_applications"
      - "excel"
      - "word"
      - "outlook"
      - "email"
    metadata:
      os: "windows"
      version: "11"
      installed_apps: ["Microsoft Excel", "Microsoft Word", "Microsoft Outlook"]
      description: "Primary office desktop"
    auto_connect: true
    max_retries: 5

  # UFO² Office Desktop 2
  - device_id: "ufo2_office_2"
    server_url: "ws://192.168.1.101:5001/ws"
    os: "windows"
    capabilities:
      - "desktop_automation"
      - "office_applications"
      - "excel"
      - "powerpoint"
      - "web_browsing"
    metadata:
      os: "windows"
      version: "11"
      installed_apps: ["Microsoft Excel", "Microsoft PowerPoint", "Google Chrome"]
      description: "Secondary office desktop"
    auto_connect: true
    max_retries: 5

  # UFO² Development Workstation
  - device_id: "ufo2_dev_1"
    server_url: "ws://192.168.1.102:5002/ws"
    os: "windows"
    capabilities:
      - "desktop_automation"
      - "development"
      - "web_browsing"
      - "code_editing"
    metadata:
      os: "windows"
      version: "11"
      installed_apps: ["Visual Studio Code", "Google Chrome", "Git"]
      description: "Development workstation"
    auto_connect: true
    max_retries: 5

  # Linux Database Server (for cross-platform workflows)
  - device_id: "linux_db_server"
    server_url: "ws://192.168.1.200:5010/ws"
    os: "linux"
    capabilities:
      - "database_server"
      - "postgresql"
      - "data_export"
    metadata:
      os: "linux"
      logs_file_path: "/var/log/postgresql/postgresql.log"
      description: "Production database server"
    auto_connect: true
    max_retries: 5

Step 6: Launch Galaxy with UFO² Devices

Once all components are configured, launch Galaxy to begin orchestrating multi-device workflows.

Prerequisites Checklist

Ensure all components are running before starting Galaxy:

1. ✅ UFO² Server(s) running on configured ports
2. ✅ UFO² Client(s) connected to their respective servers
3. ✅ MCP Services initialized (automatic with UFO² client)
4. ✅ LLM configured in config/ufo/agents.yaml
5. ✅ Network connectivity between all components

Launch Sequence

Step 1: Start all UFO² Servers

# On first Windows machine (192.168.1.100)
python -m ufo.server.app --port 5000

# On second Windows machine (192.168.1.101)
python -m ufo.server.app --port 5001

# On third Windows machine (192.168.1.102)
python -m ufo.server.app --port 5002

Step 2: Start all UFO² Clients

# On first Windows desktop
python -m ufo.client.client `
  --ws `
  --ws-server ws://192.168.1.100:5000/ws `
  --client-id ufo2_office_1 `
  --platform windows

# On second Windows desktop
python -m ufo.client.client `
  --ws `
  --ws-server ws://192.168.1.101:5001/ws `
  --client-id ufo2_office_2 `
  --platform windows

# On development workstation
python -m ufo.client.client `
  --ws `
  --ws-server ws://192.168.1.102:5002/ws `
  --client-id ufo2_dev_1 `
  --platform windows

Step 3: Launch Galaxy

# On your control machine (interactive mode)
python -m galaxy --interactive

Or launch with a specific request:

python -m galaxy "Your task description here"

Galaxy will automatically connect to all configured UFO² devices (based on config/galaxy/devices.yaml) and display the orchestration interface.

Example Multi-Device Workflows

Workflow 1: Cross-Platform Report Generation

User Request:

"Generate a weekly sales report: extract data from PostgreSQL, create Excel dashboard, and email to management"

Galaxy Orchestration:

Workflow 2: Parallel Document Processing

User Request:

"Process all invoices in the shared folder: convert PDFs to Excel, categorize by vendor, and summarize totals"

Galaxy Orchestration:

1. UFO² Desktop 1: Process invoices A-M (parallel batch 1)
2. UFO² Desktop 2: Process invoices N-Z (parallel batch 2)
3. UFO² Desktop 1: Consolidate results into master Excel file
4. UFO² Desktop 1: Generate summary report
5. UFO² Desktop 1: Send notification email

Workflow 3: Development Workflow Automation

User Request:

"Pull latest code, run tests, and create deployment package"

Galaxy Orchestration:

1. UFO² Dev Workstation: Open VS Code, pull from Git repository
2. UFO² Dev Workstation: Run automated tests, capture results
3. Linux Build Server: Build deployment package
4. UFO² Dev Workstation: Open browser, upload to staging server
5. UFO² Desktop: Send deployment notification email

Task Assignment Behavior

How Galaxy Routes Tasks to UFO² Devices

Galaxy's ConstellationAgent uses several factors to select the appropriate UFO² device for each subtask:

Example Task Decomposition

User Request:

"Prepare quarterly financial reports and distribute to stakeholders"

Galaxy Decomposition:

Task 1:
  Description: "Extract financial data from database"
  Target: linux_db_server
  Reason: Has "database_server" capability

Task 2:
  Description: "Create Excel financial dashboard"
  Target: ufo2_office_1
  Reason: Has "excel" capability, device is idle

Task 3:
  Description: "Generate PowerPoint presentation"
  Target: ufo2_office_2
  Reason: Has "powerpoint" capability

Task 4:
  Description: "Email reports to stakeholders"
  Target: ufo2_office_1
  Reason: Has "outlook" and "email" capabilities

Critical Configuration Requirements

Monitoring & Debugging

Verify Device Registration

Check if clients are connected to UFO² server:

curl http://192.168.1.100:5000/api/clients

Expected response:

{
  "online_clients": [
    {
      "client_id": "ufo2_office_1",
      "type": "device",
      "platform": "windows",
      "connected_at": 1730899822.0,
      "uptime_seconds": 45
    },
    {
      "client_id": "ufo2_office_2",
      "type": "device",
      "platform": "windows",
      "connected_at": 1730899850.0,
      "uptime_seconds": 17
    }
  ]
}

View Task Assignments

Galaxy logs show task routing decisions:

INFO - [Galaxy] Task decomposition: 3 subtasks created
INFO - [Galaxy] Subtask 1 → linux_db_server (capability match: database_server)
INFO - [Galaxy] Subtask 2 → ufo2_office_1 (capability match: excel)
INFO - [Galaxy] Subtask 3 → ufo2_office_1 (capability match: email)

Troubleshooting Device Connection

Issue: UFO² device not appearing in Galaxy device pool

Diagnosis:

1. Check if client is connected to server:

   curl http://192.168.1.100:5000/api/clients
   

2. Verify devices.yaml configuration matches client parameters

3. Check Galaxy logs for connection errors

4. Ensure auto_connect: true in devices.yaml

5. Verify UFO² server is running and accessible

Common Issues & Troubleshooting

Issue 1: UFO² Client Cannot Connect to Server

ERROR - [WS] Failed to connect to ws://192.168.1.100:5000/ws
Connection refused

# On server machine
curl http://localhost:5000/api/health

# From client machine
curl http://192.168.1.100:5000/api/health

# Test connectivity
ping 192.168.1.100

# Test port accessibility (requires telnet client)
Test-NetConnection -ComputerName 192.168.1.100 -Port 5000

# Allow port through firewall
New-NetFirewallRule -DisplayName "UFO Server" `
  -Direction Inbound `
  -LocalPort 5000 `
  -Protocol TCP `
  -Action Allow

Issue 2: Missing --platform windows Flag

# Wrong (missing platform)
python -m ufo.client.client --ws --client-id ufo2_desktop_1

# Correct
python -m ufo.client.client `
  --ws `
  --client-id ufo2_desktop_1 `
  --platform windows

Issue 3: Duplicate Client ID

ERROR - [WS] Registration failed: client_id already exists
ERROR - Another device is using ID 'ufo2_desktop_1'

Issue 4: Galaxy Cannot Find UFO² Device

ERROR - Device 'ufo2_desktop_1' not found in configuration
WARNING - Cannot dispatch task to unknown device

Issue 5: MCP Tools Not Available

ERROR - MCP tool 'click' not found
ERROR - Cannot execute Windows automation command

python -m ufo.client.client `
  --ws `
  --ws-server ws://192.168.1.100:5000/ws `
  --client-id ufo2_desktop_1 `
  --platform windows

INFO - MCP servers initialized: ui_automation, file_operations, app_control
INFO - UFO Client ready with 15 available tools

Comparison with Standalone UFO²

When to use Standalone UFO²:

• Simple, single-device Windows automation
• Development and testing
• Personal productivity tasks
• No need for cross-platform workflows

When to use UFO² as Galaxy Device:

• Enterprise-scale automation
• Multi-device orchestration
• Cross-platform workflows (Windows + Linux)
• Centralized management and monitoring
• Parallel task execution across multiple machines

Related Documentation

• UFO² Overview - Architecture and core concepts
• HostAgent - Desktop-level automation
• AppAgent - Application-specific automation
• Galaxy Overview - Multi-tier orchestration framework
• Server-Client Architecture - Distributed agent design
• Linux as Galaxy Device - Linux agent integration (similar pattern)
• Quick Start Linux - Similar server-client setup for Linux

Summary

Integrating UFO² into UFO³ Galaxy enables:

• Multi-tier orchestration - Galaxy coordinates UFO² + Linux + other devices
• Cross-platform workflows - Seamlessly combine Windows desktop + Linux servers
• Capability-based routing - Intelligent task assignment to appropriate devices
• Scalable automation - Manage multiple UFO² devices from unified control plane
• Enterprise-ready - Centralized monitoring, fault isolation, load balancing
• Server-client architecture - Separation of orchestration and execution
• Local MCP services - Automatic initialization, no manual setup required

Next Steps:

1. Start with a single UFO² device to verify the setup
2. Add more UFO² devices as needed for parallel execution
3. Integrate Linux agents for cross-platform workflows
4. Define custom capabilities for your specific use cases
5. Monitor Galaxy logs to understand task routing decisions