Observability Ingestion — Reference Component

Part of the Frontier Fabric AgentOps RVAS. This component is the telemetry landing zone you build in Challenge 2 — Azure infrastructure and export scripts that land cost, metrics, logs, diagnostics, and resource metadata into ADLS Gen2 so Fabric can shortcut to it in later challenges.

Azure infrastructure and scripts that ingest observability data from the Azure Control Plane, Azure Monitor, and Cost Management into an ADLS Gen2 landing zone.

Architecture Overview#

This component provisions the landing zone of the AgentOps Control Tower:

Copy┌──────────────────────────────────────────────────────────────────────┐
│                       Azure Control Plane                           │
│                                                                     │
│  ┌─────────────────┐  ┌──────────────────┐  ┌───────────────────┐  │
│  │ Cost Management  │  │ Resource Graph   │  │ Diagnostic        │  │
│  │ (FOCUS Export)   │  │ (Metadata/Tags)  │  │ Settings          │  │
│  └────────┬────────┘  └────────┬─────────┘  └────────┬──────────┘  │
│           │                    │                      │             │
│  ┌────────┴────────────────────┴──────────────────────┴──────────┐  │
│  │                   ADLS Gen2 Storage Account                   │  │
│  │  ┌───────┐ ┌────────┐ ┌──────┐ ┌──────────┐ ┌─────────────┐ │  │
│  │  │ costs │ │metrics │ │ logs │ │ metadata │ │ diagnostics │ │  │
│  │  └───────┘ └────────┘ └──────┘ └──────────┘ └─────────────┘ │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                                                     │
│  ┌─────────────────────┐                                           │
│  │ Log Analytics        │──── Data Export Rules ──────────┐        │
│  │ Workspace            │                                 │        │
│  └──────────┬───────────┘                                 ▼        │
│             │                                     Storage Account  │
│             ▼                                                      │
│    Application Insights                                            │
│    (AppRequests, AppTraces,                                        │
│     AppDependencies, etc.)                                         │
└──────────────────────────────────────────────────────────────────────┘

Data Flows#

Prerequisites#

• Azure CLI v2.60+
• Azure Developer CLI (azd) v1.9+
• An Azure subscription with Contributor and Cost Management Reader permissions
• Python 3.11+

Deployment#

With Azure Developer CLI#

Copy# Authenticate
azd auth login

# Provision infrastructure
azd up

You will be prompted for:

• Environment name — a short identifier (e.g., dev, demo)
• Azure location — region for all resources (e.g., eastus2)
• Azure subscription — the target subscription

Manual Bicep Deployment#

Copyaz deployment group create \
  --resource-group rg-observability-demo \
  --template-file infra/main.bicep \
  --parameters infra/main.parameters.json \
  --parameters environmentName=demo location=eastus2

Running Scripts#

Resource Graph Export#

Queries Azure Resource Graph for resource metadata and exports to ADLS Gen2:

Copycd src/scripts
pip install -r requirements.txt

python resource_graph_export.py \
  --subscription-id <SUBSCRIPTION_ID> \
  --storage-account <STORAGE_ACCOUNT_NAME> \
  --container metadata

Setup Diagnostic Settings#

Discovers resources and creates diagnostic settings (supports dry-run):

Copypython setup_diagnostic_settings.py \
  --subscription-id <SUBSCRIPTION_ID> \
  --workspace-id <WORKSPACE_RESOURCE_ID> \
  --storage-account-id <STORAGE_ACCOUNT_RESOURCE_ID> \
  --dry-run

Validate Exports#

Inspects storage containers and reports on ingested data:

Copypython validate_exports.py \
  --storage-account <STORAGE_ACCOUNT_NAME>

Data Format Reference#

FOCUS Cost Format#

The Cost Management export uses the FinOps Open Cost and Usage Specification (FOCUS) format. Key columns:

Resource Graph Parquet Schema#

Exported by resource_graph_export.py:

Storage Layout#

Copy<storage-account>/
├── costs/
│   └── focus/
│       └── <yyyyMMdd-yyyyMMdd>/
│           └── *.parquet          # FOCUS cost data
├── metrics/
│   └── am-<workspace>/
│       └── AppMetrics/
│           └── y=*/m=*/d=*/h=*/  # Log Analytics metrics export
├── logs/
│   └── am-<workspace>/
│       └── App*/
│           └── y=*/m=*/d=*/h=*/  # Log Analytics log export
├── metadata/
│   └── resource-graph/
│       └── year=*/month=*/day=*/
│           ├── all_resources_with_tags_*.parquet
│           ├── resource_counts_by_type_*.parquet
│           └── resources_by_location_*.parquet
└── diagnostics/
    └── insights-*/
        └── resourceId=*/
            └── y=*/m=*/d=*/h=*/  # Diagnostic settings output

Integration with the Fabric Control Tower#

This landing zone is the data source for the Fabric Control Tower you connect in Challenge 3. The ADLS Gen2 storage account serves as a OneLake shortcut target:

1. Fabric Lakehouse creates shortcuts to each container in this storage account
2. Spark notebooks read Parquet cost data and resource metadata
3. Power BI datasets are built on top of the lakehouse tables
4. Cost dashboards combine FOCUS cost data with resource tags for showback/chargeback

The storage account uses hierarchical namespace (ADLS Gen2) and date-partitioned directories for optimal Spark performance.

Infrastructure Resources#

CI/CD#

Two GitHub Actions workflows automate deployment:

• deploy.yml — provisions infrastructure on push to main and runs the resource graph export
• scheduled-export.yml — runs the resource graph export daily at 2:00 AM UTC via cron

Both workflows use OIDC (federated credentials) for passwordless Azure authentication.