UFO³ — Weaving the Digital Agent Galaxy

From isolated device agents to interconnected constellations — Building the Digital Agent Galaxy

🚀 What is UFO³ Galaxy?

UFO³ Galaxy is a revolutionary cross-device orchestration framework that transforms isolated device agents into a unified digital ecosystem. It models complex user requests as Task Constellations (星座) — dynamic distributed DAGs where nodes represent executable subtasks and edges capture dependencies across heterogeneous devices.

🎯 The Vision

Building truly ubiquitous intelligent agents requires moving beyond single-device automation. UFO³ Galaxy addresses four fundamental challenges in cross-device agent orchestration:

🔄 Asynchronous Parallelism
Enabling concurrent task execution across multiple devices while maintaining correctness through event-driven coordination and safe concurrency control

⚡ Dynamic Adaptation
Real-time workflow evolution in response to intermediate results, transient failures, and runtime observations without workflow abortion

🌐 Distributed Coordination
Reliable, low-latency communication across heterogeneous devices via WebSocket-based Agent Interaction Protocol with fault tolerance

🛡️ Safety Guarantees
Formal invariants ensuring DAG consistency during concurrent modifications and parallel execution, verified through rigorous proofs

🏗️ Architecture

UFO³ Galaxy Layered Architecture — From natural language to distributed execution

Layered Design

UFO³ Galaxy follows a hierarchical orchestration model that separates global coordination from local execution. This architecture enables scalable cross-device orchestration while maintaining consistent control and responsiveness across diverse operating systems and network environments.

🎛️ Hierarchical Control Plane

ConstellationClient serves as the global control plane, maintaining a live registry of all connected device agents with their: - Capability profiles and system specifications - Runtime health metrics and availability status - Current load and resource utilization

This registry enables intelligent task placement based on device capabilities, avoiding mismatches between task requirements and device capacity.

Each device hosts a device agent server that manages local orchestration through persistent WebSocket sessions with ConstellationClient. The server: - Maintains execution contexts on the host - Provides unified interface to underlying tools via MCP servers - Handles task execution, telemetry streaming, and resource monitoring

Clean separation: Global orchestration policies are decoupled from device-specific heterogeneity, providing consistent abstraction across endpoints with different OS, hardware, or network conditions.

🔄 Orchestration Flow

1. DAG Synthesis: ConstellationClient invokes the Constellation Agent to construct a TaskConstellation—a dynamic DAG encoding task decomposition, dependencies, and device mappings
2. Device Assignment: Each TaskStar (DAG node) is assigned to suitable device agents based on capability profiles and system load
3. Asynchronous Execution: The Constellation Orchestrator executes the DAG in an event-driven manner:
4. Task completions trigger dependent nodes
5. Failures prompt retry, migration, or partial DAG rewrites
6. Workflows adapt to real-time system dynamics (device churn, network variability)

Result: Highly parallel and resilient execution that sustains workflow completion even as subsets of devices fail or reconnect.

🔌 Cross-Agent Communication

The Agent Interaction Protocol (AIP) handles all cross-agent interactions: - Agent registration and capability synchronization - Task dispatch and progress reporting
- Result aggregation and telemetry streaming

Built on persistent WebSocket channels, AIP provides: - Lightweight: Minimal overhead for control messages - Bidirectional: Full-duplex communication between client and agents - Multiplexed: Concurrent message streams over single connection - Low-latency: Fast propagation of control signals and state updates - Resilient: Maintains global consistency despite intermittent connectivity

Together, these design elements form a cohesive foundation for orchestrating large-scale, heterogeneous, and adaptive workflows across a resilient multi-device execution fabric.

✨ Core Design Principles

UFO³ Galaxy realizes cross-device orchestration through five tightly integrated design principles:

1. 🌟 Declarative Decomposition into Dynamic DAG (Task Constellation)

Natural-language or programmatic requests are decomposed by the Constellation Agent into a structured DAG of TaskStars (nodes) and TaskStarLines (edges) that encode workflow logic, dependencies, and device assignments. This declarative structure is amenable to automated scheduling, introspection, and dynamic modification throughout execution.

Key Benefits: - 📋 Declarative structure for automated scheduling - 🔍 Runtime introspection for workflow visibility - ✏️ Dynamic rewriting throughout execution - 🔄 Automated orchestration across heterogeneous devices

Learn more →

2. 🔄 Continuous, Result-Driven Graph Evolution

The Task Constellation is a living data structure that evolves in response to execution feedback. Intermediate outputs, transient failures, and new observations trigger controlled rewrites—adding diagnostic TaskStars, creating fallbacks, rewiring dependencies, or pruning completed nodes—so the system adapts dynamically instead of aborting on errors.

Adaptation Mechanisms: - 🩺 Diagnostic TaskStars added for debugging - 🛡️ Fallback creation for error recovery - 🔗 Dependency rewiring for workflow optimization - ✂️ Node pruning after completion

The Constellation Agent operates in two modes: - Creation Mode: Synthesizes initial DAG from user request with device-aware task decomposition - Editing Mode: Incrementally refines constellation based on task completion events and runtime feedback

Learn more →

3. 🎯 Heterogeneous, Asynchronous, and Safe Orchestration

Each Task Star is matched to the most suitable device agent via rich Agent Profiles reflecting OS, hardware capabilities, and installed tools. The Constellation Orchestrator executes tasks asynchronously, allowing multiple TaskStars to progress in parallel.

Safety Guarantees: - 🔒 Safe assignment locking prevents race conditions - 📅 Event-driven scheduling monitors DAG readiness - ✅ DAG consistency checks maintain structural integrity - 🔄 Batched edits ensure atomicity - 📐 Formal verification reinforces correctness - ⏱️ Timeout protection prevents deadlocks

These mechanisms collectively ensure high efficiency without compromising reliability.

Learn more →

4. 🔌 Unified Agent Interaction Protocol (AIP)

Built atop persistent WebSocket channels, AIP provides a unified, secure, and fault-tolerant layer for the entire agent ecosystem.

Core Capabilities: - 📝 Agent registry with capability profiles - 🔐 Session management for secure communication - 📤 Task dispatch with intelligent routing - 🎯 Coordination primitives for distributed workflows - 💓 Heartbeat monitoring for health tracking - 🔌 Automatic reconnection under network fluctuations - 🔄 Retry mechanisms for reliability

Architecture Benefits: - 🪶 Lightweight interface for easy integration - 🧩 Extensible design supports new agent types - 🛡️ Fault tolerance ensures continuous operation

This protocol abstracts OS and network heterogeneity, enabling seamless collaboration among agents across desktops, servers, and edge devices, while allowing new agents to integrate seamlessly into the UFO³ ecosystem.

Learn more →

5. 🛠️ Template-Driven Framework for Device Agents

To democratize agent creation, UFO³ provides a lightweight development template and toolkit for rapidly building new device agents.

Development Framework: - 📄 Capability declaration defines agent profiles - 🔗 Environment binding connects to local systems - 🧩 MCP server integration for tool augmentation - 🔧 Modular design accelerates development

Model Context Protocol (MCP) Integration: - 🎁 Tool packages via MCP servers - 🔌 Plug-and-play capability extension - 🌐 Cross-platform tool standardization - 🚀 Rapid prototyping of new agents

This modular architecture maintains consistency across the constellation while enabling developers to extend UFO³ to new platforms (mobile, web, IoT, embedded systems, etc.) with minimal effort.

🔌 Extensibility: UFO³ is designed as a universal framework that supports developing new device agents for different platforms (mobile, web, IoT, embedded systems, etc.) and applications. Through the Agent Interaction Protocol (AIP), custom device agents can seamlessly integrate into UFO³ Galaxy for coordinated multi-device automation. Want to build your own device agent? See our Creating Custom Device Agents tutorial to learn how to extend UFO³ to new platforms.

Learn more → | MCP Integration →

🎯 Key Capabilities

🌐 Cross-Device Collaboration

Execute workflows that span Windows desktops, Linux servers, GPU clusters, mobile devices, and edge nodes—all from a single natural language request.

⚡ Asynchronous Parallelism

Automatically identify parallelizable subtasks and execute them concurrently across devices through: - Event-driven scheduling that continuously monitors DAG topology for ready tasks - Non-blocking execution with Python asyncio for maximum concurrency - Dynamic adaptation that integrates new tasks without interrupting running execution

Result: Dramatically reduced end-to-end latency compared to sequential execution.

🛡️ Safety & Consistency

• Three formal invariants (I1-I3) enforced at runtime for DAG correctness
• Safe assignment locking prevents race conditions during concurrent modifications
• Acyclicity validation ensures no circular dependencies
• State merging algorithm preserves execution progress during dynamic edits
• Timeout protection prevents deadlocks from agent failures

🔄 Dynamic Workflow Evolution

• Dual-mode operation: Separate creation and editing phases with controlled transitions
• Feedback-driven adaptation: Task completion events trigger intelligent constellation refinement
• LLM-powered reasoning: ReAct architecture for context-aware DAG modifications
• Undo/redo support: ConstellationEditor with command pattern for safe interactive editing

👁️ Rich Observability

• Real-time constellation visualization with DAG topology updates
• Event bus with publish-subscribe pattern for monitoring task progress
• Detailed execution logs with markdown trajectory support
• Task status tracking (pending, running, completed, failed, cancelled)
• Dependency graph inspection and validation tools

🎨 Use Cases

🖥️ Software Development & Deployment

"Clone the repo on my laptop, build the Docker image on the GPU server, deploy to staging, and run the test suite on the CI cluster."

Workflow DAG:

📊 Data Science Workflows

"Fetch the dataset from cloud storage, preprocess on the Linux workstation, train the model on the A100 node, and generate a visualization dashboard on my Windows machine."

Workflow DAG:

📝 Cross-Platform Document Processing

"Extract data from Excel on Windows, process with Python scripts on Linux, generate PDF reports, and send summary emails."

Workflow DAG:

🔬 Distributed System Monitoring

"Collect server logs from all Linux machines, analyze for errors, generate alerts, and create a consolidated report."

Workflow DAG:

🏢 Enterprise Automation

"Query the database on the server, process the results, update Excel spreadsheets on Windows, and generate PowerPoint presentations."

Workflow DAG:

🗺️ Documentation Structure

🚀 Quick Start

Get UFO³ Galaxy up and running in minutes with our step-by-step guide

👥 Galaxy Client

Device coordination, connection management, and ConstellationClient API

🧠 Constellation Agent

LLM-driven task decomposition, DAG creation, and dynamic workflow evolution

⚙️ Constellation Orchestrator

Asynchronous execution engine, event-driven coordination, and safety guarantees

📊 Task Constellation

DAG structure, TaskStar nodes, TaskStarLine edges, and constellation editor

🆔 Agent Registration

Device registry, agent profiles, and registration flow

🌐 Agent Interaction Protocol

WebSocket messaging, protocol specification, and communication patterns

⚙️ Configuration

Device pools, capabilities, and orchestration policies

🚦 Getting Started

Ready to build your Digital Agent Galaxy? Follow these steps:

1. Install UFO³

# Clone the repository
git clone https://github.com/microsoft/UFO.git
cd UFO

# Install dependencies
pip install -r requirements.txt

2. Configure Device Pool

Create configuration files in config/galaxy/:

config/galaxy/devices.yaml - Define your devices:

devices:
  - device_id: "windowsagent"
    server_url: "ws://localhost:5005/ws"
    os: "windows"
    capabilities:
      - "web_browsing"
      - "office_applications"
      - "file_management"
    metadata:
      location: "home_office"
      os: "windows"
      performance: "medium"
    max_retries: 5

  - device_id: "linux_agent_1"
    server_url: "ws://localhost:5001/ws"
    os: "linux"
    capabilities:
      - "server"
      - "python"
      - "docker"
    metadata:
      os: "linux"
      performance: "high"
      logs_file_path: "/root/log/log1.txt"
    auto_connect: true
    max_retries: 5

config/galaxy/constellation.yaml - Configure runtime settings:

# Constellation Runtime Settings
CONSTELLATION_ID: "my_constellation"
HEARTBEAT_INTERVAL: 30.0  # Heartbeat interval in seconds
RECONNECT_DELAY: 5.0  # Delay before reconnecting in seconds
MAX_CONCURRENT_TASKS: 6  # Maximum concurrent tasks
MAX_STEP: 15  # Maximum steps per session

# Device Configuration
DEVICE_INFO: "config/galaxy/devices.yaml"

# Logging Configuration
LOG_TO_MARKDOWN: true

See Galaxy Configuration for complete documentation.

3. Start Device Agents

On each device, launch the Agent Server:

On Windows:

# Start Agent Server on port 5005
python -m ufo --mode agent-server --port 5005

On Linux:

# Start Agent Server on port 5001
python -m ufo --mode agent-server --port 5001

4. Launch Galaxy Client

Interactive Mode:

python -m galaxy --interactive

Direct Request:

python -m galaxy "Your cross-device task here"

Programmatic API:

from galaxy.galaxy_client import GalaxyClient

async def main():
    client = GalaxyClient(session_name="my_session")
    await client.initialize()
    result = await client.process_request("Your task request")
    await client.shutdown()

For detailed instructions, see the Quick Start Guide.

🔧 System Components

UFO³ Galaxy consists of several integrated components working together:

Core Components

Supporting Infrastructure

For detailed component documentation, see the respective sections in Documentation Structure.

Technology Stack

🌟 From Devices to Constellations to Galaxy

UFO³ represents a paradigm shift in intelligent automation:

• Single Device → Isolated agents operating within one OS
• Task Constellation → Coordinated multi-device workflows for one task
• Digital Agent Galaxy → Interconnected constellations spanning your entire digital estate

Over time, multiple constellations can interconnect, weaving together agents, devices, and capabilities into a self-organizing Digital Agent Galaxy. This design elevates cross-device automation from a brittle engineering challenge to a unified orchestration paradigm, where multi-device workflows become naturally expressive, paving the way for large-scale, adaptive, and resilient intelligent ubiquitous computing systems.

📊 Performance Monitoring & Evaluation

UFO³ Galaxy provides comprehensive performance monitoring and evaluation tools to analyze multi-device workflow execution:

Automated Metrics Collection

Galaxy automatically collects detailed performance metrics during execution through an event-driven observer pattern:

• Task Metrics: Execution times, success rates, bottleneck identification
• Constellation Metrics: DAG statistics, parallelism analysis, critical path computation
• Modification Metrics: Dynamic editing patterns and adaptation frequency
• Device Metrics: Per-device performance and resource utilization

All metrics are captured in real-time without impacting execution performance and saved to structured JSON files for programmatic analysis.

Trajectory Report

Galaxy automatically generates a comprehensive Markdown trajectory report (output.md) documenting the complete execution lifecycle:

logs/galaxy/<task_name>/output.md

This human-readable report includes: - Step-by-step execution timeline with agent actions - Interactive DAG topology visualizations showing constellation evolution - Detailed task execution logs with results and errors - Device connection status and coordination events - Complete before/after constellation states at each step

The trajectory report provides visual debugging and workflow understanding, complementing the quantitative result.json metrics.

Result JSON Format

After each session, Galaxy also generates a comprehensive result.json file containing:

logs/galaxy/<task_name>/result.json

This file includes: - Complete session metadata and execution timeline - Task-by-task performance breakdown - Constellation statistics (parallelism ratio, critical path, max concurrency) - Modification history showing DAG evolution - Final results and outcomes

Example Key Metrics:

Performance Analysis Tools

import json

# Load session results
with open("logs/galaxy/task_32/result.json", 'r') as f:
    result = json.load(f)

# Extract key metrics
metrics = result["session_results"]["metrics"]
task_stats = metrics["task_statistics"]
const_stats = result["session_results"]["final_constellation_stats"]

print(f"✅ Success Rate: {task_stats['success_rate'] * 100:.1f}%")
print(f"⏱️  Avg Task Duration: {task_stats['average_task_duration']:.2f}s")
print(f"🔀 Parallelism Ratio: {const_stats['parallelism_ratio']:.2f}")

Documentation:

• Trajectory Report Guide - Complete guide to the human-readable execution log with DAG visualizations
• Performance Metrics Guide - Comprehensive metrics documentation with analysis examples
• Result JSON Reference - Complete schema reference and programmatic access guide

�📚 Learn More

• Research Paper: UFO³: Weaving the Digital Agent Galaxy (Coming Soon)
• UFO² (Desktop AgentOS): Documentation
• UFO (Original): GitHub Repository

🤝 Contributing

We welcome contributions! Whether you're building new device agents, improving orchestration algorithms, or enhancing the protocol, check out our Contributing Guide on GitHub.

📄 License

UFO³ Galaxy is released under the MIT License.