Observability Ingestion — Reference Component

Part of the Frontier Fabric AgentOps RVAS. This component is the telemetry landing zone you build in Challenge 2 — Azure infrastructure and export scripts that land cost, metrics, logs, diagnostics, and resource metadata into ADLS Gen2 so Fabric can shortcut to it in later challenges.

Azure infrastructure and scripts that ingest observability data from the Azure Control Plane, Azure Monitor, and Cost Management into an ADLS Gen2 landing zone.

Architecture Overview#

This component provisions the landing zone of the AgentOps Control Tower:

Copy┌──────────────────────────────────────────────────────────────────────┐
│                       Azure Control Plane                           │
│                                                                     │
│  ┌─────────────────┐  ┌──────────────────┐  ┌───────────────────┐  │
│  │ Cost Management  │  │ Resource Graph   │  │ Diagnostic        │  │
│  │ (FOCUS Export)   │  │ (Metadata/Tags)  │  │ Settings          │  │
│  └────────┬────────┘  └────────┬─────────┘  └────────┬──────────┘  │
│           │                    │                      │             │
│  ┌────────┴────────────────────┴──────────────────────┴──────────┐  │
│  │                   ADLS Gen2 Storage Account                   │  │
│  │  ┌───────┐ ┌──────────┐ ┌──────────┐ ┌───────────────────┐ │  │
│  │  │ costs │ │ metadata │ │   am-*   │ │    insights-*     │ │  │
│  │  └───────┘ └──────────┘ └──────────┘ └───────────────────┘ │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                                                     │
│  ┌─────────────────────┐                                           │
│  │ Log Analytics        │──── Data Export Rules ──────────┐        │
│  │ Workspace            │                                 │        │
│  └──────────┬───────────┘                                 ▼        │
│             │                                     Storage Account  │
│             ▼                                                      │
│    Application Insights                                            │
│    (AppRequests, AppTraces,                                        │
│     AppDependencies, etc.)                                         │
└──────────────────────────────────────────────────────────────────────┘

Data Flows#

Only costs and metadata are created by this component. Log Analytics data export creates physical containers such as am-apprequests, am-appdependencies, and am-appmetrics when those tables emit new records after the export rule is enabled. Diagnostic settings similarly create containers such as insights-logs-audit and insights-metrics-pt1m; Azure does not write to a single diagnostics container.

Prerequisites#

• Azure CLI v2.60+
• Azure Developer CLI (azd) v1.9+
• An Azure subscription with Contributor and Cost Management Reader permissions
• A Fabric workspace assigned to a Fabric capacity
• Workspace administrator access to create the Fabric workspace identity
• Python 3.11+

Deployment#

Create the Fabric workspace identity#

In the Fabric portal:

1. Open the workspace created for the workshop, or create one and assign it to your Fabric capacity.
2. Open Workspace settings > Workspace identity.
3. Select + Workspace identity.
4. Copy the identity Object ID. Do not use the Fabric workspace ID or client ID.
5. Store the Object ID in the azd environment used for the workshop:

Copyazd env select ctl-tower
azd env set FABRIC_WORKSPACE_IDENTITY_PRINCIPAL_ID <WORKSPACE_IDENTITY_OBJECT_ID>

The Bicep deployment uses this principal ID to grant the workspace identity Storage Blob Data Contributor on the landing-zone storage account.

With Azure Developer CLI#

Copy# Authenticate
azd auth login

# Provision infrastructure
azd provision

You will be prompted for:

• Environment name — a short identifier (e.g., dev, demo)
• Azure location — region for all resources (e.g., eastus2)
• Azure subscription — the target subscription

Manual Bicep Deployment#

Copyaz deployment group create \
  --resource-group rg-observability-demo \
  --template-file infra/main.bicep \
  --parameters infra/main.parameters.json \
  --parameters environmentName=demo location=eastus2

Seed and Validate the Landing Zone#

The scripts are a one-time workshop bootstrap. They make data available immediately without adding a production scheduler before the Fabric data path is built. Challenge 6 can move the same exporter to unattended cloud execution without changing its ADLS data contract.

Authenticate with Azure, then create an isolated Python environment:

Copyaz login
cd src/scripts
python -m venv .venv

Activate it on Windows PowerShell:

Copy.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Or activate it on macOS/Linux:

Copysource .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Resource Graph Export#

Queries Azure Resource Graph for resource metadata and exports to ADLS Gen2:

Copypython resource_graph_export.py \
  --subscription-id <SUBSCRIPTION_ID> \
  --storage-account <STORAGE_ACCOUNT_NAME> \
  --container metadata

Diagnostic settings for the reused Log Analytics workspace are deployed by Bicep. No setup script is required.

Validate Exports#

Discovers costs, metadata, am-*, and insights-* containers and reports on ingested data:

Copypython validate_exports.py \
  --storage-account <STORAGE_ACCOUNT_NAME>

Data Format Reference#

FOCUS Cost Format#

The Cost Management export uses the FinOps Open Cost and Usage Specification (FOCUS) format. Key columns:

Resource Graph Parquet Schema#

Exported by resource_graph_export.py:

Storage Layout#

Copy<storage-account>/
├── costs/
│   └── focus/
│       └── <yyyyMMdd-yyyyMMdd>/
│           └── *.parquet          # FOCUS cost data
├── metadata/
│   └── resource-graph/
│       └── year=*/month=*/day=*/
│           ├── all_resources_with_tags_*.parquet
│           ├── resource_counts_by_type_*.parquet
│           └── resources_by_location_*.parquet
├── am-apprequests/
│   └── .../PT5M.json             # Log Analytics AppRequests export
├── am-appdependencies/
│   └── .../PT5M.json             # Log Analytics AppDependencies export
├── am-appmetrics/
│   └── .../PT5M.json             # Log Analytics AppMetrics export
├── am-apptraces/ and am-appexceptions/
│   └── .../PT5M.json             # Created when those tables emit records
├── insights-logs-audit/
│   └── resourceId=*/...          # Diagnostic settings log output
└── insights-metrics-pt1m/
  └── resourceId=*/...          # Diagnostic settings metric output

Integration with the Fabric Control Tower#

This landing zone is the data source for the Fabric Control Tower you connect in Challenge 3. The ADLS Gen2 storage account serves as a OneLake shortcut target:

1. Fabric Lakehouse creates shortcuts to each container in this storage account
2. Spark notebooks read Parquet cost data and resource metadata
3. Power BI datasets are built on top of the lakehouse tables
4. Cost dashboards combine FOCUS cost data with resource tags for showback/chargeback

The storage account uses hierarchical namespace (ADLS Gen2) and date-partitioned directories for optimal Spark performance.

Infrastructure Resources#

CI/CD#

Two GitHub Actions workflows automate deployment:

• deploy.yml — provisions infrastructure on push to main and runs the resource graph export
• scheduled-export.yml — runs the resource graph export daily at 2:00 AM UTC via cron

Both workflows use OIDC (federated credentials) for passwordless Azure authentication.