{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Tracing and Observability\n", "\n", "AutoGen has [built-in support for tracing](https://microsoft.github.io/autogen/dev/user-guide/core-user-guide/framework/telemetry.html) and observability for collecting comprehensive records on the execution of your application. This feature is useful for debugging, performance analysis, and understanding the flow of your application.\n", "\n", "This capability is powered by the [OpenTelemetry](https://opentelemetry.io/) library, which means you can use any OpenTelemetry-compatible backend to collect and analyze traces.\n", "\n", "## Setup\n", "\n", "To begin, you need to install the OpenTelemetry Python package. You can do this using pip:\n", "\n", "```bash\n", "pip install opentelemetry-sdk\n", "```\n", "\n", "Once you have the SDK installed, the simplest way to set up tracing in AutoGen is to:\n", "\n", "1. Configure an OpenTelemetry tracer provider\n", "2. Set up an exporter to send traces to your backend\n", "3. Connect the tracer provider to the AutoGen runtime\n", "\n", "## Telemetry Backend\n", "\n", "To collect and view traces, you need to set up a telemetry backend. Several open-source options are available, including Jaeger, Zipkin. For this example, we will use Jaeger as our telemetry backend.\n", "\n", "For a quick start, you can run Jaeger locally using Docker:\n", "\n", "```bash\n", "docker run -d --name jaeger \\\n", " -e COLLECTOR_OTLP_ENABLED=true \\\n", " -p 16686:16686 \\\n", " -p 4317:4317 \\\n", " -p 4318:4318 \\\n", " jaegertracing/all-in-one:latest\n", "```\n", "\n", "This command starts a Jaeger instance that listens on port 16686 for the Jaeger UI and port 4317 for the OpenTelemetry collector. You can access the Jaeger UI at `http://localhost:16686`.\n", "\n", "## Instrumenting an AgentChat Team\n", "\n", "In the following section, we will review how to enable tracing with an AutoGen GroupChat team. The AutoGen runtime already supports open telemetry (automatically logging message metadata). To begin, we will create a tracing service that will be used to instrument the AutoGen runtime. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from opentelemetry import trace\n", "from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter\n", "from opentelemetry.sdk.resources import Resource\n", "from opentelemetry.sdk.trace import TracerProvider\n", "from opentelemetry.sdk.trace.export import BatchSpanProcessor\n", "\n", "otel_exporter = OTLPSpanExporter(endpoint=\"http://localhost:4317\", insecure=True)\n", "tracer_provider = TracerProvider(resource=Resource({\"service.name\": \"autogen-test-agentchat\"}))\n", "span_processor = BatchSpanProcessor(otel_exporter)\n", "tracer_provider.add_span_processor(span_processor)\n", "trace.set_tracer_provider(tracer_provider)\n", "\n", "# we will get reference this tracer later using its service name\n", "# tracer = trace.get_tracer(\"autogen-test-agentchat\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "All of the code to create a [team](./tutorial/teams.ipynb) should already be familiar to you. An important note here is that all AgentChat agents and teams are run using the AutoGen core API runtime. In turn, the runtime is already instrumented to log [runtime messaging events (metadata)] (https://github.com/microsoft/autogen/blob/main/python/packages/autogen-core/src/autogen_core/_telemetry/_tracing_config.py) including:\n", "\n", "- **create**: When a message is created\n", "- **send**: When a message is sent\n", "- **publish**: When a message is published\n", "- **receive**: When a message is received\n", "- **intercept**: When a message is intercepted\n", "- **process**: When a message is processed\n", "- **ack**: When a message is acknowledged \n", " " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from autogen_agentchat.agents import AssistantAgent\n", "from autogen_agentchat.conditions import MaxMessageTermination, TextMentionTermination\n", "from autogen_agentchat.teams import SelectorGroupChat\n", "from autogen_agentchat.ui import Console\n", "from autogen_core import SingleThreadedAgentRuntime\n", "from autogen_ext.models.openai import OpenAIChatCompletionClient\n", "\n", "\n", "def search_web_tool(query: str) -> str:\n", " if \"2006-2007\" in query:\n", " return \"\"\"Here are the total points scored by Miami Heat players in the 2006-2007 season:\n", " Udonis Haslem: 844 points\n", " Dwayne Wade: 1397 points\n", " James Posey: 550 points\n", " ...\n", " \"\"\"\n", " elif \"2007-2008\" in query:\n", " return \"The number of total rebounds for Dwayne Wade in the Miami Heat season 2007-2008 is 214.\"\n", " elif \"2008-2009\" in query:\n", " return \"The number of total rebounds for Dwayne Wade in the Miami Heat season 2008-2009 is 398.\"\n", " return \"No data found.\"\n", "\n", "\n", "def percentage_change_tool(start: float, end: float) -> float:\n", " return ((end - start) / start) * 100\n", "\n", "\n", "async def main() -> None:\n", " model_client = OpenAIChatCompletionClient(model=\"gpt-4o\")\n", "\n", " planning_agent = AssistantAgent(\n", " \"PlanningAgent\",\n", " description=\"An agent for planning tasks, this agent should be the first to engage when given a new task.\",\n", " model_client=model_client,\n", " system_message=\"\"\"\n", " You are a planning agent.\n", " Your job is to break down complex tasks into smaller, manageable subtasks.\n", " Your team members are:\n", " WebSearchAgent: Searches for information\n", " DataAnalystAgent: Performs calculations\n", "\n", " You only plan and delegate tasks - you do not execute them yourself.\n", "\n", " When assigning tasks, use this format:\n", " 1. : \n", "\n", " After all tasks are complete, summarize the findings and end with \"TERMINATE\".\n", " \"\"\",\n", " )\n", "\n", " web_search_agent = AssistantAgent(\n", " \"WebSearchAgent\",\n", " description=\"An agent for searching information on the web.\",\n", " tools=[search_web_tool],\n", " model_client=model_client,\n", " system_message=\"\"\"\n", " You are a web search agent.\n", " Your only tool is search_tool - use it to find information.\n", " You make only one search call at a time.\n", " Once you have the results, you never do calculations based on them.\n", " \"\"\",\n", " )\n", "\n", " data_analyst_agent = AssistantAgent(\n", " \"DataAnalystAgent\",\n", " description=\"An agent for performing calculations.\",\n", " model_client=model_client,\n", " tools=[percentage_change_tool],\n", " system_message=\"\"\"\n", " You are a data analyst.\n", " Given the tasks you have been assigned, you should analyze the data and provide results using the tools provided.\n", " If you have not seen the data, ask for it.\n", " \"\"\",\n", " )\n", "\n", " text_mention_termination = TextMentionTermination(\"TERMINATE\")\n", " max_messages_termination = MaxMessageTermination(max_messages=25)\n", " termination = text_mention_termination | max_messages_termination\n", "\n", " selector_prompt = \"\"\"Select an agent to perform task.\n", "\n", " {roles}\n", "\n", " Current conversation context:\n", " {history}\n", "\n", " Read the above conversation, then select an agent from {participants} to perform the next task.\n", " Make sure the planner agent has assigned tasks before other agents start working.\n", " Only select one agent.\n", " \"\"\"\n", "\n", " task = \"Who was the Miami Heat player with the highest points in the 2006-2007 season, and what was the percentage change in his total rebounds between the 2007-2008 and 2008-2009 seasons?\"\n", "\n", " tracer = trace.get_tracer(\"autogen-test-agentchat\")\n", " with tracer.start_as_current_span(\"runtime\"):\n", " team = SelectorGroupChat(\n", " [planning_agent, web_search_agent, data_analyst_agent],\n", " model_client=model_client,\n", " termination_condition=termination,\n", " selector_prompt=selector_prompt,\n", " allow_repeated_speaker=True,\n", " )\n", " await Console(team.run_stream(task=task))\n", "\n", " await model_client.close()\n", "\n", "\n", "# asyncio.run(main())" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---------- user ----------\n", "Who was the Miami Heat player with the highest points in the 2006-2007 season, and what was the percentage change in his total rebounds between the 2007-2008 and 2008-2009 seasons?\n", "---------- PlanningAgent ----------\n", "To accomplish this, we can break down the tasks as follows:\n", "\n", "1. WebSearchAgent: Search for the Miami Heat player with the highest points during the 2006-2007 NBA season.\n", "2. WebSearchAgent: Find the total rebounds for the identified player in both the 2007-2008 and 2008-2009 NBA seasons.\n", "3. DataAnalystAgent: Calculate the percentage change in total rebounds for the player between the 2007-2008 and 2008-2009 seasons.\n", "\n", "Once these tasks are complete, I will summarize the findings.\n", "---------- WebSearchAgent ----------\n", "[FunctionCall(id='call_PUhxZyR0CTlWCY4uwd5Zh3WO', arguments='{\"query\":\"Miami Heat highest points scorer 2006-2007 season\"}', name='search_web_tool')]\n", "---------- WebSearchAgent ----------\n", "[FunctionExecutionResult(content='Here are the total points scored by Miami Heat players in the 2006-2007 season:\\n Udonis Haslem: 844 points\\n Dwayne Wade: 1397 points\\n James Posey: 550 points\\n ...\\n ', name='search_web_tool', call_id='call_PUhxZyR0CTlWCY4uwd5Zh3WO', is_error=False)]\n", "---------- WebSearchAgent ----------\n", "Here are the total points scored by Miami Heat players in the 2006-2007 season:\n", " Udonis Haslem: 844 points\n", " Dwayne Wade: 1397 points\n", " James Posey: 550 points\n", " ...\n", " \n", "---------- WebSearchAgent ----------\n", "Dwyane Wade was the Miami Heat player with the highest points in the 2006-2007 season, scoring 1,397 points. Now, let's find his total rebounds for the 2007-2008 and 2008-2009 NBA seasons.\n", "---------- WebSearchAgent ----------\n", "[FunctionCall(id='call_GL7KkWKj9ejIM8FfpgXe2dPk', arguments='{\"query\": \"Dwyane Wade total rebounds 2007-2008 season\"}', name='search_web_tool'), FunctionCall(id='call_X81huZoiA30zIjSAIDgb8ebe', arguments='{\"query\": \"Dwyane Wade total rebounds 2008-2009 season\"}', name='search_web_tool')]\n", "---------- WebSearchAgent ----------\n", "[FunctionExecutionResult(content='The number of total rebounds for Dwayne Wade in the Miami Heat season 2007-2008 is 214.', name='search_web_tool', call_id='call_GL7KkWKj9ejIM8FfpgXe2dPk', is_error=False), FunctionExecutionResult(content='The number of total rebounds for Dwayne Wade in the Miami Heat season 2008-2009 is 398.', name='search_web_tool', call_id='call_X81huZoiA30zIjSAIDgb8ebe', is_error=False)]\n", "---------- WebSearchAgent ----------\n", "The number of total rebounds for Dwayne Wade in the Miami Heat season 2007-2008 is 214.\n", "The number of total rebounds for Dwayne Wade in the Miami Heat season 2008-2009 is 398.\n", "---------- DataAnalystAgent ----------\n", "[FunctionCall(id='call_kB50RkFVqHptA7FOf0lL2RS8', arguments='{\"start\":214,\"end\":398}', name='percentage_change_tool')]\n", "---------- DataAnalystAgent ----------\n", "[FunctionExecutionResult(content='85.98130841121495', name='percentage_change_tool', call_id='call_kB50RkFVqHptA7FOf0lL2RS8', is_error=False)]\n", "---------- DataAnalystAgent ----------\n", "85.98130841121495\n", "---------- PlanningAgent ----------\n", "The Miami Heat player with the highest points during the 2006-2007 NBA season was Dwayne Wade, who scored 1,397 points. The percentage increase in his total rebounds from the 2007-2008 season (214 rebounds) to the 2008-2009 season (398 rebounds) was approximately 86%.\n", "\n", "TERMINATE\n" ] } ], "source": [ "await main()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can then use the Jaeger UI to view the traces collected from the application run above. \n", "\n", "![Jaeger UI](jaeger.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Custom Traces \n", "\n", "So far, we are logging only the default events that are generated by the AutoGen runtime (message created, publish etc). However, you can also create custom spans to log specific events in your application. \n", "\n", "In the example below, we will show how to log messages from the `RoundRobinGroupChat` team as they are generated by adding custom spans around the team to log runtime events and spans to log messages generated by the team.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "-- primary_agent -- : Leaves cascade like gold, \n", "Whispering winds cool the earth.\n", "primary_agent: Leaves cascade like gold, \n", "Whispering winds cool the earth.\n", "\n", "-- critic_agent -- : Your haiku beautifully captures the essence of the fall season with vivid imagery. However, it appears to have six syllables in the second line, which should traditionally be five. Here's a revised version keeping the 5-7-5 syllable structure:\n", "\n", "Leaves cascade like gold, \n", "Whispering winds cool the air. \n", "\n", "Please adjust the second line to reflect a five-syllable count. Thank you!\n", "critic_agent: Your haiku beautifully captures the essence of the fall season with vivid imagery. However, it appears to have six syllables in the second line, which should traditionally be five. Here's a revised version keeping the 5-7-5 syllable structure:\n", "\n", "Leaves cascade like gold, \n", "Whispering winds cool the air. \n", "\n", "Please adjust the second line to reflect a five-syllable count. Thank you!\n", "\n", "-- primary_agent -- : Leaves cascade like gold, \n", "Whispering winds cool the air.\n", "primary_agent: Leaves cascade like gold, \n", "Whispering winds cool the air.\n", "\n", "-- critic_agent -- : APPROVE\n", "critic_agent: APPROVE\n" ] } ], "source": [ "from autogen_agentchat.base import TaskResult\n", "from autogen_agentchat.conditions import ExternalTermination\n", "from autogen_agentchat.teams import RoundRobinGroupChat\n", "from autogen_core import CancellationToken\n", "\n", "\n", "async def run_agents() -> None:\n", " # Create an OpenAI model client.\n", " model_client = OpenAIChatCompletionClient(model=\"gpt-4o-2024-08-06\")\n", "\n", " # Create the primary agent.\n", " primary_agent = AssistantAgent(\n", " \"primary_agent\",\n", " model_client=model_client,\n", " system_message=\"You are a helpful AI assistant.\",\n", " )\n", "\n", " # Create the critic agent.\n", " critic_agent = AssistantAgent(\n", " \"critic_agent\",\n", " model_client=model_client,\n", " system_message=\"Provide constructive feedback. Respond with 'APPROVE' to when your feedbacks are addressed.\",\n", " )\n", "\n", " # Define a termination condition that stops the task if the critic approves.\n", " text_termination = TextMentionTermination(\"APPROVE\")\n", "\n", " tracer = trace.get_tracer(\"autogen-test-agentchat\")\n", " with tracer.start_as_current_span(\"runtime_round_robin_events\"):\n", " team = RoundRobinGroupChat([primary_agent, critic_agent], termination_condition=text_termination)\n", "\n", " response_stream = team.run_stream(task=\"Write a 2 line haiku about the fall season\")\n", " async for response in response_stream:\n", " async for response in response_stream:\n", " if not isinstance(response, TaskResult):\n", " print(f\"\\n-- {response.source} -- : {response.to_text()}\")\n", " with tracer.start_as_current_span(f\"agent_message.{response.source}\") as message_span:\n", " message_span.set_attribute(\"agent.name\", response.source)\n", " message_span.set_attribute(\"message.content\", response.to_text())\n", " print(f\"{response.source}: {response.to_text()}\")\n", "\n", " await model_client.close()\n", "\n", "\n", "await run_agents()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "In the code above, we create a new span for each message sent by the agent. We set attributes on the span to include the agent's name and the message content. This allows us to trace the flow of messages through our application and understand how they are processed." ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 2 }