Challenge 3 — Connect Fabric to the Enterprise

Mission log. The Control Tower has signals now — agent conversations in Cosmos DB, telemetry and cost files in ADLS Gen2 — but the customer's enterprise data is still sitting outside the tower walls. Your job is to connect it to Microsoft Fabric without copying piles of data around. This is where OneLake becomes the customer's unifying data estate.

In this challenge you build the Fabric foundation for the rest of the RVAS: a Lakehouse that sees the Challenge 2 landing-zone files through OneLake shortcuts, and a mirrored Cosmos DB source that brings Challenge 1 agent conversations into OneLake Delta tables for near-real-time analytics.

The important distinction: shortcuts are zero-copy pointers to existing files; Mirroring is managed replication into Fabric. You need both.

Objectives#

By the end of this challenge you will have:

• A team Fabric workspace assigned to your Fabric capacity.
• An Observability Lakehouse in that workspace.
• Seven working OneLake shortcuts to the ADLS Gen2 landing-zone containers from Challenge 2: costs, metadata, three application telemetry sources, and two platform diagnostic sources.
• Cosmos DB Mirroring configured for the agent conversation data from Challenge 1.
• Proof that Fabric can query both a shortcut-backed file and a mirrored Cosmos table.
• A crisp explanation of zero-copy shortcuts vs Mirroring replication.

Prerequisites#

• ✅ Challenge 1 complete — the agent workload has written conversation data to Cosmos DB.
• ✅ Challenge 2 complete — the ADLS Gen2 landing zone contains files in costs, metadata, am-apprequests, am-appdependencies, am-appmetrics, insights-logs-audit, and insights-metrics-pt1m.
• Fabric capacity ID from Challenge 0.
• ADLS Gen2 DFS endpoint from Challenge 2, like https://<account>.dfs.core.windows.net.
• Fabric cloud connection ID for that ADLS Gen2 endpoint, created in Step 1.
• Cosmos DB account name and database name from Challenge 1.
• Fabric tenant settings allow service principal / managed identity access and Mirroring.
• The reference setup assets in resources/fabric-control-tower/.

The foundation#

Your architecture now gets its Fabric entry point:

CopyChallenge 2 ADLS Gen2 landing zone       Challenge 1 Cosmos DB
costs · metadata · am-* · insights-*     conversations · interactions
          │                                         │
          │ OneLake shortcuts (zero copy)           │ Fabric Mirroring
          ▼                                         ▼
  Fabric Lakehouse Files/                    Mirrored Delta tables
          └───────────────► Challenge 4 Bronze layer ◄───────────────┘

Shortcuts keep the source files where they are. Mirroring continuously lands operational records into OneLake-managed Delta tables. Challenge 4 will use both as Bronze inputs.

Your mission#

1. Create the ADLS Gen2 cloud connection#

• Grant your organizational account Storage Blob Data Reader on the ADLS Gen2 storage account from Challenge 2.
• Confirm the storage account DFS endpoint and the seven required containers are available: costs, metadata, am-apprequests, am-appdependencies, am-appmetrics, insights-logs-audit, and insights-metrics-pt1m.
• In Fabric, open Settings > Manage connections and gateways, select New, and create a Cloud connection of type Azure Data Lake Storage Gen2.
• Use the storage account DFS endpoint as the server, select Organizational account authentication, and set the privacy level to Organizational. If prompted for a path, use the storage account root so the connection can reach all seven containers.
• Open the new connection's settings and copy its Connection ID. You will pass this GUID to the setup script.

This is a one-time setup. One cloud connection can authenticate all seven shortcuts because they use the same storage account.

2. Create the Lakehouse and OneLake shortcuts#

• Create or reuse your team Fabric workspace on the Fabric capacity.

• Create a Lakehouse for the Control Tower foundation.

• Add the seven OneLake shortcuts under the Lakehouse Files/ area using this contract:

  Lakehouse shortcut	ADLS Gen2 container
  Files/costs	costs
  Files/metadata	metadata
  Files/telemetry/apprequests	am-apprequests
  Files/telemetry/appdependencies	am-appdependencies
  Files/telemetry/appmetrics	am-appmetrics
  Files/diagnostics/audit	insights-logs-audit
  Files/diagnostics/platformmetrics	insights-metrics-pt1m

• Browse each shortcut and confirm you are seeing the same files that landed in Challenge 2 — not a copied export.

The provided setup script can automate the workspace, Lakehouse, shortcuts, notebook imports, and pipeline imports:

Copycd resources/fabric-control-tower
pip install -r src/setup/requirements.txt

python src/setup/setup_fabric_workspace.py \
  --workspace-name "Observability-Analytics" \
  --storage-account-url "https://<account>.dfs.core.windows.net" \
  --connection-id "<fabric-cloud-connection-id>" \
  --capacity-id "<fabric-capacity-id>"

3. Mirror the agent conversation database#

• Enable the Cosmos DB prerequisites for Fabric Mirroring, including continuous backup / analytical store as required by your Cosmos DB configuration.
• In Fabric, create an Azure Cosmos DB v2 cloud connection for the Challenge 1 account using Organizational account authentication. Copy its Connection ID from the connection settings.
• Create a Fabric mirrored database item for the Cosmos DB database used by the agent workload.
• Select the conversation containers from Challenge 1. The reference assets describe conversations, messages, and feedback; the included setup script currently configures conversations and interactions, so verify the actual container names created by your workload.
• Start Mirroring and wait for the status to become healthy/running.

Reference command:

Copycd resources/fabric-control-tower
python src/setup/setup_cosmos_mirroring.py \
  --workspace-id "<fabric-workspace-id>" \
  --cosmos-account "<cosmos-account-name>" \
  --database "observability" \
  --connection-id "<fabric-cosmos-connection-id>"

4. Expose the mirrored tables in the Lakehouse#

A mirrored database is a separate Fabric item, so Spark notebooks cannot read its Delta tables by path. Add one OneLake table shortcut per mirrored container into the Lakehouse Tables/dbo area so the Challenge 4 notebooks can read them:

In the Fabric UI:

1. Open the Lakehouse, hover Tables, then choose New shortcut → Microsoft OneLake.
2. Pick the CosmosDB-agentsdb mirrored database as the data source.
3. Expand Tables → agentsdb and select both conversations and interactions.
4. Create the shortcuts and confirm they appear under Tables/dbo with the same names.

This step is not automated by the setup scripts — create the two shortcuts yourself after Mirroring reports a running status, otherwise the source tables will not exist yet.

5. Prove both paths work#

Show your team and coach that Fabric can read from both enterprise connection patterns:

• Shortcut path: browse all seven paths in the shortcut contract and open or query at least one Parquet file and one JSON file.
• Mirroring path: open the mirrored database, confirm the expected tables are present, and verify the sync status shows recent activity.
• Latency target: after adding one new agent conversation, look for it in the mirrored table within roughly a minute.
• Query proof: run one simple query against a shortcut-backed file and one simple query against a mirrored table.

Success criteria#

• Fabric workspace is assigned to the correct capacity and the team can open it.
• Lakehouse exists for the Control Tower foundation.
• All seven shortcuts in the documented contract exist and browse successfully.
• Browsed shortcut files match the ADLS Gen2 files from Challenge 2 — no duplicate copy was made.
• Cosmos DB mirrored database exists with the expected conversation tables.
• Mirroring is running/healthy and recent records are syncing.
• Tables/dbo/conversations and Tables/dbo/interactions OneLake shortcuts resolve in the Lakehouse.
• Your team can query one shortcut file and one mirrored table.
• Your team can explain zero-copy shortcut vs Mirroring replication to your coach.

🧭 Checkpoint: show your coach the Lakehouse shortcuts, the mirrored Cosmos tables, and one query from each path. Then explain which path copies data and which path does not.

Hints#

Resources#

• resources/fabric-control-tower/README.md — Fabric setup, shortcuts, Mirroring, notebooks, and pipelines
• resources/fabric-control-tower/src/setup/setup_fabric_workspace.py — workspace, Lakehouse, shortcuts, imports
• resources/fabric-control-tower/src/setup/setup_cosmos_mirroring.py — Cosmos DB Mirroring setup
• docs/architecture.md — OneLake shortcuts and Mirroring in the end-to-end architecture

⬅️ Previous: Challenge 2 — Build the Telemetry Landing Zone
➡️ Next: Challenge 4 — Refine the Signal