Fabric Control Tower — Reference Component

Part of the Frontier Fabric AgentOps RVAS. This is the control tower itself — the Fabric workspace you stand up across Challenge 3 (OneLake foundation), Challenge 4 (medallion pipeline), and Challenge 5 (Direct Lake dashboards).

Overview#

This component implements the Microsoft Fabric analytics layer of the AgentOps Control Tower. It processes data from Azure Monitor, Cost Management, and Cosmos DB through a medallion architecture (Bronze → Silver → Gold) in Microsoft Fabric, producing Power BI reports for operational insights.

Raw telemetry — cost exports, platform metrics, diagnostic logs, and AI-agent conversations — flows into a Fabric Lakehouse via ADLS Gen2 shortcuts and Cosmos DB Mirroring. PySpark notebooks transform the data through progressively refined layers, and a Direct Lake semantic model powers interactive Power BI dashboards without any data duplication.

Architecture#

Copy┌─────────────────────────────────────────────────────────────────────────────┐
│                           Data Sources                                      │
│                                                                             │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────────────┐  │
│  │  ADLS Gen2        │  │  ADLS Gen2        │  │  Cosmos DB               │  │
│  │  Cost Exports     │  │  Metrics / Logs   │  │  Agent Conversations     │  │
│  │  Resource Metadata│  │  (Azure Monitor)  │  │  (Agent Workload)        │  │
│  └────────┬─────────┘  └────────┬─────────┘  └────────────┬─────────────┘  │
│           │                      │                          │                │
└───────────┼──────────────────────┼──────────────────────────┼────────────────┘
            │                      │                          │
            ▼                      ▼                          ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         Microsoft Fabric                                    │
│                                                                             │
│  ┌─────────────────────────────────────┐   ┌────────────────────────────┐  │
│  │  Lakehouse (OneLake)                │   │  Mirrored Database         │  │
│  │  ├─ Files/                          │   │  Cosmos DB → Delta Tables  │  │
│  │  │   ├─ costs/          (shortcut)  │   │  (near-real-time sync)     │  │
│  │  │   ├─ metadata/       (shortcut)  │   └────────────┬───────────────┘  │
│  │  │   ├─ telemetry/      (shortcuts) │                │                  │
│  │  │   └─ diagnostics/    (shortcuts) │                │                  │
│  │  └─ Tables/dbo/                     │                │                  │
│  │      ├─ bronze_*  (raw Delta)       │◄───────────────┘                  │
│  │      ├─ silver_*  (cleansed Delta)  │                                   │
│  │      └─ gold_*    (aggregated Delta)│                                   │
│  └─────────────────┬───────────────────┘                                   │
│                    │                                                        │
│  ┌─────────────────▼───────────────────┐                                   │
│  │  Notebooks (PySpark)                │                                   │
│  │  01_bronze_ingestion                │                                   │
│  │  02_silver_transformation           │                                   │
│  │  03_gold_aggregation                │                                   │
│  │  04_cosmos_mirroring_transform      │                                   │
│  └─────────────────┬───────────────────┘                                   │
│                    │                                                        │
│  ┌─────────────────▼───────────────────┐                                   │
│  │  Pipelines                          │                                   │
│  │  ├─ Load E2E Pipeline               │                                   │
│  │  └─ Daily Refresh Pipeline          │                                   │
│  └─────────────────┬───────────────────┘                                   │
│                    │                                                        │
│  ┌─────────────────▼───────────────────┐   ┌────────────────────────────┐  │
│  │  Semantic Model (Direct Lake)       │──▶│  Power BI Reports          │  │
│  │  CostSummary, OperationalMetrics    │   │  Cost Overview             │  │
│  │  AgentAnalytics + dimensions        │   │  Operational Health        │  │
│  │  Measures: TotalCost, ErrorRate ... │   │  Performance               │  │
│  └─────────────────────────────────────┘   │  Agent Performance         │  │
│                                            │  Resource Inventory        │  │
│                                            └────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────┘

Prerequisites#

Deployment#

1. Deploy Azure Infrastructure#

Provision the storage account, key vault, and managed identity used by Fabric:

Copyazd auth login
azd provision

This creates:

• Storage Account (ADLS Gen2) — landing zone for cost exports, metrics, and logs
• Key Vault — stores connection strings and Fabric API credentials
• User-Assigned Managed Identity — grants Fabric access to ADLS Gen2

2. Configure Fabric Workspace#

Create the Fabric workspace, lakehouse, ADLS shortcuts, and import notebooks:

Copypip install -r src/setup/requirements.txt

python src/setup/setup_fabric_workspace.py \
  --workspace-name "Observability-Analytics" \
  --storage-account-url "https://<account>.dfs.core.windows.net" \
  --connection-id "<fabric-connection-id>" \
  --capacity-id "<fabric-capacity-id>"

The script performs the following:

1. Creates (or reuses) a Fabric workspace assigned to the specified capacity.
2. Creates an Observability lakehouse inside the workspace.
3. Adds ADLS Gen2 shortcuts under Files/ for costs, metadata, the three required am-* application telemetry containers, and the two required insights-* diagnostic containers.
4. Uploads and imports all PySpark notebooks from fabric/notebooks/.
5. Creates the Load E2E Pipeline and Daily Refresh Pipeline.

3. Configure Cosmos DB Mirroring#

Enable Fabric Mirroring for the Cosmos DB database containing agent conversations:

Copypython src/setup/setup_cosmos_mirroring.py \
  --workspace-id "<workspace-id>" \
  --cosmos-account "<cosmos-account-name>" \
  --database "agentsdb" \
  --connection-id "<fabric-cosmos-connection-id>"

The script:

1. Resolves the participant-owned Azure Cosmos DB v2 cloud connection by ID or account endpoint.
2. Creates a mirrored database definition for the Cosmos DB database in the Fabric workspace.
3. Starts replication and confirms that Mirroring reaches a running state. Verify that the conversations and interactions containers from Challenge 1 appear as mirrored tables.

4. Run Notebooks#

Run the notebooks in order — either manually in Fabric or via the E2E pipeline:

To trigger all four in sequence, open the Load E2E Pipeline in Fabric and click Run.

5. Configure GitHub Actions (optional)#

.github/workflows/deploy.yml runs steps 1–3 automatically on every push to main that touches infra/, src/, or fabric/. It reads every value from repository configuration, so populate all of the following before the first run:

Both *_CONNECTION_ID values are Fabric cloud connection IDs, not Azure resource IDs. Create them under Settings → Manage connections and gateways: an ADLS Gen2 connection for the storage account, and an Azure Cosmos DB v2 connection for the Cosmos account.

A name that is missing or misspelled expands to an empty string rather than failing, so the run reaches the setup step and errors there instead. If a step fails on an argument you believe you configured, check the spelling of the corresponding name above first.

The job targets the dev, staging, or prod environment. Define these at repository level to share them, or per environment to vary them.

Notebooks#

01_bronze_ingestion.ipynb#

Reads raw Parquet files from ADLS Gen2 shortcuts and writes Delta tables into the schema-enabled Lakehouse Tables/dbo/ section.

Input: Files/costs/, Files/metadata/, Files/telemetry/apprequests/, Files/telemetry/appdependencies/, Files/telemetry/appmetrics/, Files/diagnostics/audit/, Files/diagnostics/platformmetrics/ Output: bronze_costs, bronze_metrics, bronze_logs, bronze_resource_metadata

Key behaviors:

• Schema inference with explicit type overrides for known columns.
• Append mode with deduplication using _source_file and _ingestion_timestamp watermarks.
• Data quality checks: null key detection, row-count validation, schema-drift alerts.

02_silver_transformation.ipynb#

Cleanses and transforms Bronze tables into an analysis-ready Silver layer.

Transformations:

• Costs: Normalizes raw billing data to the FOCUS cost schema — standardized column names, currency conversion, amortization of reservations and savings plans.
• Metrics: Pivots time-series metric records from long to wide format; interpolates missing intervals; aligns to 5-minute grain.
• Logs: Parses semi-structured log messages; extracts severity, category, operation, and correlation ID; filters noise.
• Resource Metadata: Flattens nested resource properties and tag maps; adds computed columns for resource age, region normalization, and service categorization.

Output: silver_costs, silver_metrics, silver_logs, silver_resource_metadata

03_gold_aggregation.ipynb#

Creates analytical aggregates consumed by the semantic model.

Output tables:

05_semantic_model_dimensions.ipynb#

Creates the physical Direct Lake dimensions after notebooks 03 and 04 have populated the Gold layer.

04_cosmos_mirroring_transform.ipynb#

Transforms mirrored Cosmos DB conversation data into agent analytics.

Input: Mirrored tables conversations, messages, feedback Output: gold_agent_analytics, gold_conversation_details

Key behaviors:

• Sessionizes messages into conversation threads.
• Calculates per-conversation metrics: message count, total tokens, elapsed time, resolution status.
• Aggregates conversations, interactions, token usage, and response time by model and topic.
• Handles late-arriving mirrored records with merge-on-read reconciliation.

Pipeline Schedule#

Both pipelines include:

• Dependency ordering: Bronze → Silver → Gold plus mirrored-agent transform → semantic-model dimensions.
• Retry policy: 2 retries with 5-minute backoff.
• Failure notifications via Fabric alerts (email and Teams webhook).

Semantic Model#

The Direct Lake semantic model connects Power BI directly to Delta tables in OneLake — no import or DirectQuery overhead.

Tables and Relationships#

CopyCalendar (dim_date) ──1:*──▶ CostSummary
                    ├─1:*──▶ OperationalMetrics
                    └─1:*──▶ AgentAnalytics

ResourceInventory (dim_resource) ──1:*──▶ CostSummary

Key Measures#

Time Intelligence#

The model includes YTD and prior-month cost measures using the physical dim_date table.

Power BI Report#

Connecting to the Semantic Model#

1. Open Power BI Desktop or the Power BI service.
2. Select OneLake data hub → choose the semantic model published from this workspace.
3. The connection uses Direct Lake mode — no data copy is created.

Suggested Report Pages#

Design Guidelines#

• Use the organization's brand palette for consistent theming.
• Apply row-level security (RLS) roles mapped to subscription or resource-group ownership.
• Enable paginated export for the Resource Inventory page.

Integration with Other Components#

Project Structure#

Copyfabric-control-tower/
├── .github/
│   └── workflows/
│       └── deploy.yml              # CI/CD pipeline
├── infra/
│   ├── main.bicep                  # Bicep entry point (storage, identity)
│   └── main.parameters.json        # Default parameters
├── fabric/
│   ├── notebooks/
│   │   ├── 01_bronze_ingestion.ipynb
│   │   ├── 02_silver_transformation.ipynb
│   │   ├── 03_gold_aggregation.ipynb
│   │   ├── 04_cosmos_mirroring_transform.ipynb
│   │   └── 05_semantic_model_dimensions.ipynb
│   └── pipelines/
│       ├── pipeline_load_e2e.json
│       └── pipeline_daily_refresh.json
├── src/
│   └── setup/
│       ├── requirements.txt
│       ├── semantic_model.json     # Direct Lake semantic model definition
│       ├── setup_fabric_workspace.py
│       └── setup_cosmos_mirroring.py
├── .gitignore
├── azure.yaml                      # azd manifest
└── README.md

Contributing#

1. Fork the repository and create a feature branch from main.
2. Follow existing code style — run ruff check and ruff format before committing.
3. Add or update tests for any new setup scripts.
4. Open a pull request with a clear description of your changes.
5. Ensure the CI pipeline passes before requesting review.

License#

This project is licensed under the MIT License.