Chapter 2 of 6
Architecture at a glance#
The approved request carries W3C trace context through API Management, the agent, the model, and the tool. One non-sensitive correlation ID joins the supported spans, while each span keeps its own result. A tool failure must not become a model failure.
The workbook and alert queries read Application Insights. APIM emits bounded token metrics for a faster usage estimate. Cost Management reports the bill later. The customer-owned APIM repository remains authoritative for gateway policy, and the controlled promotion workflow consumes the temporary smoke result.
Design choices and tradeoffs#
| Decision | Chosen approach | Benefit | Limit |
|---|---|---|---|
| Runtime content | Exclude prompts, responses, and tool payloads by default | Trace service behavior without creating a content archive | A diagnostic exception needs separate, time-limited approval |
| Trace volume | Sample at the source, preserve selected traces end to end, and do not sample metrics | Bound ingestion while keeping joined traces | Sampling can miss rare failures |
| Cost signal | Use low-cardinality APIM token metrics for estimates and Cost Management for billing | Give operators a fast signal without treating it as an invoice | Counts can be incomplete; billing normally lags 8-24 hours |