Tracing from local to cloud

Experimental feature

This is an experimental feature, and may change at any time. Learn more.

Prompt flow tracing feature enables users to trace LLM calls, functions and even LLM frameworks. Besides, with promptflow[azure] installed, prompt flow can also log traces to an Azure ML workspace or Azure AI project, which makes it possible to share traces with your team members.

Installing the package

pip install "promptflow[azure]>=1.11.0"

Set cloud destination

To log traces to cloud, first of all, you need an Azure ML workspace or Azure AI project. Then, you can set the destination. Make sure you have logged in Azure CLI (az login, refer to Azure CLI doc for more informations) before execute below CLI command:

pf config set trace.destination=azureml://subscriptions/<subscription-id>/resourcegroups/<resource-group-name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace-or-project-name>

Fill in with your own subscription ID, resource group name, workspace or project name, and all is ready now. You can make LLM calls, run LLM application or execute your flow with pf flow test or pf run create, you will see an Azure portal URL link in the console:

Click the link to view the traces in Azure portal, and feel free to share it with your team members.

Storage

Traces in Azure ML workspace/AI project are persisted in an Azure Cosmos DB associated with the workspace/project. It will be automatically setup the first time you execute CLI command pf config set trace.destination for a workspace/project.

Set different destination

Prompt flow also supports to log traces to different workspace/project across different flows. To configure this, you need to set config to azureml via CLI command:

pf config set trace.destination=azureml

Then, you need to prepare the configuration files pointing to different workspace/project; prompt flow currently recognizes workspace configuration file config.json, you can manually create one or download from Azure portal, this JSON file contains all required informations of a workspace/project:

{
    "subscription_id": "<subscription-id>",
    "resource_group": "<resource-group-name>",
    "workspace_name": "<workspace-name>"
}

When trace.destination is set to azureml, prompt flow will search for a config.json, starts from .azureml under flow folder, then goes up to parent folder until it finds one. If no config.json is found, an error will be raised. It is recommended to place config.json under a folder named .azureml in your flow directory, which makes prompt flow to find it easily.

Below is an example folder structure:

flows
├── flow1
│   ├── .azureml
│   │   └── config.json  # workspace/project A
│   ├── flow.flex.yaml
│   ├── llm.py
│   ├── data.jsonl
│   ...
├── flow2
│   ├── .azureml
│   │   └── config.json  # workspace/project B
│   ├── flow.dag.yaml
│   ├── hello.py
│   ├── data.jsonl
└── ...

Then when execute flow1, traces will be logged to workspace/project A, while execute flow2, traces will be logged to workspace/project B.

Disable logging to cloud

When you want to disable logging traces to cloud, you can switch back to local by below CLI command:

pf config set trace.destination=local

local is the default value for pf.trace.destination, and no traces will be logged to Azure anymore with this value; note that traces will still be logged to local.

Disable tracing feature

Use below CLI command to disable prompt flow tracing feature:

pf config set trace.destination=none

Then no traces will be logged to neither local nor cloud.