Task 02 - Trace operations in Microsoft Foundry

Introduction

In the prior task, you looked at some of the default monitoring capabilities available in Microsoft Foundry. However, the information that is captured by default is at an aggregate level and may not tell the entire story. To gain deeper insights into model behavior and application performance, you can instrument your client application with the OpenTelemetry SDK. This will allow you to capture detailed trace information, including the actual messages that were sent to the model and the responses that were received. This information can be invaluable for diagnosing issues and optimizing model performance.

Description

In this task, you will trace operations in Microsoft Foundry by instrumenting your client application with the OpenTelemetry SDK. You will then generate some traffic to populate the Application analytics dashboard with data. Finally, you will review the default application metrics available in the dashboard.

Success Criteria

  • You have added instrumentation logic to your client application.
  • You have generated traffic to populate the Application analytics dashboard.

Learning Resources

View trace results for AI applications using OpenAI SDK View trace results for AI agents in Microsoft Foundry

Key Tasks

01: Review existing trace details

Before you add any additional logic to the client application, first review the information that is currently available within Microsoft Foundry’s Tracing menu.

Expand this section to view the solution

First, navigate to Microsoft Foundry and select the AI project associated with this training. Then, select Tracing from the Observe and optimize menu on the left-hand side.

View trace details in Microsoft Foundry.

Inside of the main grid, you will see a list of trace operations that have been captured. Some trace operations have drill-through indicators, which you can select to view additional operations that occurred as part of the parent operation. For example, when calculating a discount, you can see a process_thread_run operation that includes multiple steps, including start_thread_run, get_thread_run, and more. Each of these steps contains some relevant metadata, such as duration, status, and timestamps. But what you generally will not find is the actual content of the messages that were sent to the model or the responses that were received. This is because the client application has not been instrumented to capture this information.

02: Instrument the client application

As you saw in the first task, the Tracing menu in Microsoft Foundry contains some useful information about the operations that have occurred. However, it does not include any details about the actual messages that were sent to the model or the responses that were received. To capture this information, you will need to add some instrumentation logic to your client application.

Expand this section to view the solution

First, open the src/chat_app.py file in your code editor. Then, uncomment the following import statement on line 14.

from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

On line 67, you will see a call to configure_azure_monitor(). This call enables telemetry collection and is why we see the information that we do in the Tracing menu. Immediately after this line (that is, on line 68), uncomment the following line of code to instrument OpenAI operations.

OpenAIInstrumentor().instrument()

Repeat this process in two more files: src/app/agents/agent_processor.py and src/app/tools/discountLogic.py. In both of these files, add the import statement for OpenAIInstrumentor, the call to OpenAIInstrumentor().instrument() immediately after the call to configure_azure_monitor().

Once you have made your changes, save the files and restart the application by stopping the Uvicorn server (Ctrl+C) and running the following command again.

uvicorn chat_app:app --host 0.0.0.0 --port 8000

03: Populate with additional data

Once you have updated the client application with the necessary instrumentation logic, you will need to generate some traffic to populate the Application analytics dashboard with data. Navigate back to your deployed application and interact with it to generate requests. Sample prompts may include the following:

  • “What are the latest trends in home decor?”
  • “Can you help me find a sofa that fits my style?”
  • “Do you have any blue paint in stock?”
  • “What is the availability of the ‘Modern Chair’?”
  • “I am ready to check out.”

Enter a variety of prompts to generate different types of requests.

04: Review new trace details

As you interact with the application, new trace operations will be captured and sent to Microsoft Foundry. Navigate back to the Tracing menu in Microsoft Foundry to review the new trace details. If you do not see any new trace operations, select the Refresh button in the top menu to reload the data.

Trace details now include input and output information.

New trace operations that come in may now populate the Input and Output columns in the grid. Furthermore, you can select a trace operation to view additional details, including the actual messages that were sent to the model and the responses that were received. This includes the system prompt and any user messages that were part of the conversation.

Selecting a trace detail link displays more information on the trace operation, including times, payloads, and operations.

05: Deploy to production

To conclude this task, you will deploy the latest version of the application to production. This will ensure that the instrumentation logic is included in the production environment and that you can continue to capture trace information for future requests.

Expand this section to view the solution

In order to deploy the latest version of the application to production, first ensure that all files are saved. Then, in the terminal, navigate to the src directory of your project if you are not already there. Stop the Uvicorn server if it is still running by pressing Ctrl+C. Then, build the Docker image by running the following command:

docker build -t {YOUR_REGISTRY_NAME}.azurecr.io/chat-app:latest .

Next, log in to your Azure Container Registry service by running the following command:

az acr login -n {YOUR_REGISTRY_NAME}.azurecr.io

Finally, push the Docker image to your Azure Container Registry by running the following command:

docker push {YOUR_REGISTRY_NAME}.azurecr.io/chat-app:latest

Because you have already configured the Azure App Service to use this image and Continuous Deployment is set to On, the App Service will automatically pull the latest version of the image and restart the application. This will take several minutes and you may need to stop and start the application manually if it does not restart automatically. Once the application is running again, you can navigate to the URL of your App Service to access the chat application.

This particular way of deploying an application is not ideal. It requires you to have direct access to the production environment, which is not always feasible or secure. A better approach would be to use a CI/CD pipeline that automatically deploys the latest version of the application whenever changes are made to the codebase. This would ensure that the production environment is always up-to-date with the latest code and would reduce the risk of human error during the deployment process. This will be the main focus of Exercise 05.