Fabric Control Tower — Reference Component

Part of the Frontier Fabric AgentOps RVAS. This is the control tower itself — the Fabric workspace you stand up across Challenge 3 (OneLake foundation), Challenge 4 (medallion pipeline), and Challenge 5 (Direct Lake dashboards).

Overview#

This component implements the Microsoft Fabric analytics layer of the AgentOps Control Tower. It processes data from Azure Monitor, Cost Management, and Cosmos DB through a medallion architecture (Bronze → Silver → Gold) in Microsoft Fabric, producing Power BI reports for operational insights.

Raw telemetry — cost exports, platform metrics, diagnostic logs, and AI-agent conversations — flows into a Fabric Lakehouse via ADLS Gen2 shortcuts and Cosmos DB Mirroring. PySpark notebooks transform the data through progressively refined layers, and a Direct Lake semantic model powers interactive Power BI dashboards without any data duplication.

Architecture#

Copy┌─────────────────────────────────────────────────────────────────────────────┐
│                           Data Sources                                      │
│                                                                             │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────────────┐  │
│  │  ADLS Gen2        │  │  ADLS Gen2        │  │  Cosmos DB               │  │
│  │  Cost Exports     │  │  Metrics / Logs   │  │  Agent Conversations     │  │
│  │  Resource Metadata│  │  (Azure Monitor)  │  │  (Agent Workload)        │  │
│  └────────┬─────────┘  └────────┬─────────┘  └────────────┬─────────────┘  │
│           │                      │                          │                │
└───────────┼──────────────────────┼──────────────────────────┼────────────────┘
            │                      │                          │
            ▼                      ▼                          ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         Microsoft Fabric                                    │
│                                                                             │
│  ┌─────────────────────────────────────┐   ┌────────────────────────────┐  │
│  │  Lakehouse (OneLake)                │   │  Mirrored Database         │  │
│  │  ├─ Files/                          │   │  Cosmos DB → Delta Tables  │  │
│  │  │   ├─ costs/        (shortcut)    │   │  (near-real-time sync)     │  │
│  │  │   ├─ metrics/      (shortcut)    │   └────────────┬───────────────┘  │
│  │  │   ├─ logs/         (shortcut)    │                │                  │
│  │  │   └─ resource-metadata/ (shortcut│                │                  │
│  │  └─ Tables/                         │                │                  │
│  │      ├─ bronze_*  (raw Delta)       │◄───────────────┘                  │
│  │      ├─ silver_*  (cleansed Delta)  │                                   │
│  │      └─ gold_*    (aggregated Delta)│                                   │
│  └─────────────────┬───────────────────┘                                   │
│                    │                                                        │
│  ┌─────────────────▼───────────────────┐                                   │
│  │  Notebooks (PySpark)                │                                   │
│  │  01_bronze_ingestion                │                                   │
│  │  02_silver_transformation           │                                   │
│  │  03_gold_aggregation                │                                   │
│  │  04_cosmos_mirroring_transform      │                                   │
│  └─────────────────┬───────────────────┘                                   │
│                    │                                                        │
│  ┌─────────────────▼───────────────────┐                                   │
│  │  Pipelines                          │                                   │
│  │  ├─ Load E2E Pipeline               │                                   │
│  │  └─ Daily Refresh Pipeline          │                                   │
│  └─────────────────┬───────────────────┘                                   │
│                    │                                                        │
│  ┌─────────────────▼───────────────────┐   ┌────────────────────────────┐  │
│  │  Semantic Model (Direct Lake)       │──▶│  Power BI Reports          │  │
│  │  fact_costs, fact_metrics, ...      │   │  Cost Overview             │  │
│  │  dim_resource, dim_date, ...        │   │  Capacity Utilization      │  │
│  │  Measures: TotalCost, ErrorRate ... │   │  Operational Health        │  │
│  └─────────────────────────────────────┘   │  Agent Performance         │  │
│                                            │  Resource Inventory        │  │
│                                            └────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────┘

Prerequisites#

Deployment#

1. Deploy Azure Infrastructure#

Provision the storage account, key vault, and managed identity used by Fabric:

Copyazd auth login
azd provision

This creates:

• Storage Account (ADLS Gen2) — landing zone for cost exports, metrics, and logs
• Key Vault — stores connection strings and Fabric API credentials
• User-Assigned Managed Identity — grants Fabric access to ADLS Gen2

2. Configure Fabric Workspace#

Create the Fabric workspace, lakehouse, ADLS shortcuts, and import notebooks:

Copypip install -r src/setup/requirements.txt

python src/setup/setup_fabric_workspace.py \
  --workspace-name "Observability-Analytics" \
  --storage-account-url "https://<account>.dfs.core.windows.net" \
  --capacity-id "<fabric-capacity-id>"

The script performs the following:

1. Creates (or reuses) a Fabric workspace assigned to the specified capacity.
2. Creates an Observability lakehouse inside the workspace.
3. Adds ADLS Gen2 shortcuts under Files/ for each data container (costs, metrics, logs, resource-metadata).
4. Uploads and imports all PySpark notebooks from fabric/notebooks/.
5. Creates the Load E2E Pipeline and Daily Refresh Pipeline.

3. Configure Cosmos DB Mirroring#

Enable Fabric Mirroring for the Cosmos DB database containing agent conversations:

Copypython src/setup/setup_cosmos_mirroring.py \
  --workspace-id "<workspace-id>" \
  --cosmos-account "<cosmos-account-name>" \
  --database "observability"

The script:

1. Enables the Cosmos DB account for Fabric Mirroring (continuous backup, analytical store).
2. Creates a mirrored database item in the Fabric workspace.
3. Configures table selection and replication for the conversations, messages, and feedback containers.

4. Run Notebooks#

Run the notebooks in order — either manually in Fabric or via the E2E pipeline:

To trigger all four in sequence, open the Load E2E Pipeline in Fabric and click Run.

Notebooks#

01_bronze_ingestion.ipynb#

Reads raw Parquet files from ADLS Gen2 shortcuts and writes Delta tables into the Lakehouse Tables/ section.

Input: Files/costs/, Files/metrics/, Files/logs/, Files/resource-metadata/ Output: bronze_costs, bronze_metrics, bronze_logs, bronze_resource_metadata

Key behaviors:

• Schema inference with explicit type overrides for known columns.
• Append mode with deduplication using _source_file and _ingestion_timestamp watermarks.
• Data quality checks: null key detection, row-count validation, schema-drift alerts.

02_silver_transformation.ipynb#

Cleanses and transforms Bronze tables into an analysis-ready Silver layer.

Transformations:

• Costs: Normalizes raw billing data to the FOCUS cost schema — standardized column names, currency conversion, amortization of reservations and savings plans.
• Metrics: Pivots time-series metric records from long to wide format; interpolates missing intervals; aligns to 5-minute grain.
• Logs: Parses semi-structured log messages; extracts severity, category, operation, and correlation ID; filters noise.
• Resource Metadata: Flattens nested resource properties and tag maps; adds computed columns for resource age, region normalization, and service categorization.

Output: silver_costs, silver_metrics, silver_logs, silver_resource_metadata

03_gold_aggregation.ipynb#

Creates analytical aggregates consumed by the semantic model.

Output tables:

04_cosmos_mirroring_transform.ipynb#

Transforms mirrored Cosmos DB conversation data into agent analytics.

Input: Mirrored tables conversations, messages, feedback Output: gold_agent_analytics, gold_conversation_details

Key behaviors:

• Sessionizes messages into conversation threads.
• Calculates per-conversation metrics: message count, total tokens, elapsed time, resolution status.
• Joins feedback scores and computes rolling satisfaction averages.
• Handles late-arriving mirrored records with merge-on-read reconciliation.

Pipeline Schedule#

Both pipelines include:

• Dependency ordering: Bronze → Silver → Gold → Semantic Model refresh.
• Retry policy: 2 retries with 5-minute backoff.
• Failure notifications via Fabric alerts (email and Teams webhook).

Semantic Model#

The Direct Lake semantic model connects Power BI directly to Delta tables in OneLake — no import or DirectQuery overhead.

Tables and Relationships#

Copydim_date ──────────┐
                   │ 1:*
fact_costs ◄───────┤
                   │ 1:*
dim_resource ──────┤
                   │ 1:*
fact_metrics ◄─────┤
                   │ 1:*
fact_operations ◄──┘
                   
fact_agent_analytics ──▶ dim_date

Key Measures#

Time Intelligence#

All cost and metric measures include time-intelligence variants: YTD, MTD, QTD, prior period, and rolling 30-day averages. These are generated via a calculation group applied to dim_date.

Power BI Report#

Connecting to the Semantic Model#

1. Open Power BI Desktop or the Power BI service.
2. Select OneLake data hub → choose the semantic model published from this workspace.
3. The connection uses Direct Lake mode — no data copy is created.

Suggested Report Pages#

Design Guidelines#

• Use the organization's brand palette for consistent theming.
• Apply row-level security (RLS) roles mapped to subscription or resource-group ownership.
• Enable paginated export for the Resource Inventory page.

Integration with Other Components#

Project Structure#

Copyfabric-control-tower/
├── .github/
│   └── workflows/
│       └── deploy.yml              # CI/CD pipeline
├── infra/
│   ├── main.bicep                  # Bicep entry point (storage, identity)
│   └── main.parameters.json        # Default parameters
├── fabric/
│   ├── notebooks/
│   │   ├── 01_bronze_ingestion.ipynb
│   │   ├── 02_silver_transformation.ipynb
│   │   ├── 03_gold_aggregation.ipynb
│   │   └── 04_cosmos_mirroring_transform.ipynb
│   └── pipelines/
│       ├── pipeline_load_e2e.json
│       └── pipeline_daily_refresh.json
├── src/
│   └── setup/
│       ├── requirements.txt
│       ├── semantic_model.json     # Direct Lake semantic model definition
│       ├── setup_fabric_workspace.py
│       └── setup_cosmos_mirroring.py
├── .gitignore
├── azure.yaml                      # azd manifest
└── README.md

Contributing#

1. Fork the repository and create a feature branch from main.
2. Follow existing code style — run ruff check and ruff format before committing.
3. Add or update tests for any new setup scripts.
4. Open a pull request with a clear description of your changes.
5. Ensure the CI pipeline passes before requesting review.

License#

This project is licensed under the MIT License.