Home / Challenges / Light Up the Agents
CH 01 Est. 1.5–2 h Level 200 Cloud/Platform engineerAI engineer

Challenge 1 — Light Up the Agents

Mission log. You can't monitor what doesn't exist yet. The Control Tower needs something to watch — a real, running agent that talks to a model, calls tools, stores conversations, and (crucially) emits telemetry at every hop. Today you bring the customer's agent workload online and prove the signal is flowing.

This is the source of everything the Control Tower will eventually correlate: traces, custom token/cost metrics, and conversation records. Get it emitting cleanly and the rest of the platform has something real to chew on.

Objectives

By the end of this challenge you will have:

  • Deployed a full-stack Azure AI Foundry agent workload to Azure.
  • Generated real agent traffic through the gateway → backend → agent → model path.
  • Confirmed distributed traces connect every hop in Application Insights.
  • Found the custom token/cost metrics the agent emits.
  • Seen conversations land in Cosmos DB (the data the Control Tower will later mirror).

Prerequisites

  • ✅ Challenge 0 complete — Azure access, region, and Foundry quota confirmed.
  • The reference workload in resources/agent-workload/.

The workload

The provided app has three Container Apps behind API Management, backed by Cosmos DB and a Foundry model deployment, all wired to Application Insights:

User → API Management → Frontend (Next.js) → Backend (FastAPI) → Agent (FastAPI) → Azure AI Foundry
                                                   │
                                                   └→ Cosmos DB (conversations & interactions)

Everything is instrumented with the Azure Monitor OpenTelemetry SDK, and the agent records custom metrics for token usage and latency.

Your mission

1. Deploy the workload

  • Provision and deploy the agent stack to your chosen region using the provided infrastructure.
  • Capture the output service URLs (frontend, backend, agent) and the names of the Application Insights and Cosmos DB resources.

2. Make the agent work for its telemetry

  • Open the frontend and have a conversation with the agent — send several different prompts so there's a variety of traffic. (Bonus: script a handful of requests to the agent API to generate volume.)
  • Confirm the agent is actually calling the model and returning completions (not erroring).

3. Prove the signal is flowing

Using Application Insights for this workload, demonstrate all of the following:

  • A distributed trace (end-to-end transaction) that shows the request flowing across the services and out to the model and Cosmos DB.
  • The Application Map, showing the services and their dependencies (Cosmos DB, Azure OpenAI).
  • The custom metrics the agent emits for token consumption (and/or latency) — and explain to a teammate what they'll be worth to the Control Tower later.

4. Confirm the data exhaust

  • In Cosmos DB, find the conversations and interactions your traffic created. Note the database/container names and the partition key — Challenge 3 will mirror this into Fabric.

Success criteria

  • The agent workload is deployed and the frontend responds to prompts end-to-end
  • You can show one end-to-end trace spanning gateway → backend → agent → model in App Insights
  • The Application Map shows the services plus Cosmos DB and Azure OpenAI as dependencies
  • You can point to a custom token/cost metric emitted by the agent
  • Conversation/interaction documents are visible in Cosmos DB
  • You've recorded the App Insights + Cosmos DB resource names for later challenges

🧭 Checkpoint: walk your coach through a single user message and trace its journey across the telemetry — from the click to the model and back.

Hints

Deploying with azd 
cd resources/agent-workload
azd auth login
azd up      # pick your env name, region, and subscription when prompted

Prefer gpt-4o-mini to conserve quota — check the model/deployment parameters before deploying. After deploy, azd prints the service URLs.

Generating a burst of traffic 
AGENT=https://<agent-url>
for i in $(seq 1 20); do
  curl -s -X POST "$AGENT/api/agent/invoke" \
    -H 'Content-Type: application/json' \
    -d '{"messages":[{"role":"user","content":"Give me one fact about observability."}]}' \
    >/dev/null && echo "req $i ok"
done
Finding telemetry in Application Insights
  • Transaction search → pick a recent POST /api/... → open end-to-end transaction details to see the waterfall across services, Cosmos DB, and Azure OpenAI.
  • Application map → hover the edges for latency/volume.
  • Metrics → choose the custom metric namespace and plot agent.tokens.completion (or similar).
  • Logs (KQL) to confirm token flow:
    customMetrics
| where name startswith "agent."
| summarize sum(value) by name, bin(timestamp, 5m)
Where's the telemetry contract?

The shared resources/observability-sdk/ defines the event and metric schema (AgentStart/Step/ExternalCall/End/Error, token & cost metrics). Skim its README to understand what the agent should be emitting and why it matters for cost attribution later.

Resources

➡️ Next: Challenge 2 — Build the Telemetry Landing Zone

← Back to all content View source on GitHub ↗