Cerebras

Cerebras has developed the world’s largest and fastest AI processor, the Wafer-Scale Engine-3 (WSE-3). Notably, the CS-3 system can run large language models like Llama-3.1-8B and Llama-3.1-70B at extremely fast speeds, making it an ideal platform for demanding AI workloads.

While it’s technically possible to adapt AutoGen to work with Cerebras’ API by updating the base_url, this approach may not fully account for minor differences in parameter support. Using this library will also allow for tracking of the API costs based on actual token usage.

For more information about Cerebras Cloud, visit cloud.cerebras.ai. Their API reference is available at inference-docs.cerebras.ai.

Requirements

To use Cerebras with AutoGen, install the autogen-agentchat[cerebras] package.

!pip install autogen-agentchat["cerebras"]~=0.2

Getting Started

Cerebras provides a number of models to use. See the list of models here.

See the sample OAI_CONFIG_LIST below showing how the Cerebras client class is used by specifying the api_type as cerebras.

[
    {
        "model": "llama3.1-8b",
        "api_key": "your Cerebras API Key goes here",
        "api_type": "cerebras"
    },
    {
        "model": "llama3.1-70b",
        "api_key": "your Cerebras API Key goes here",
        "api_type": "cerebras"
    }
]

Credentials

Get an API Key from cloud.cerebras.ai and add it to your environment variables:

export CEREBRAS_API_KEY="your-api-key-here"

API parameters

The following parameters can be added to your config for the Cerebras API. See this link for further information on them and their default values.

• max_tokens (null, integer >= 0)
• seed (number)
• stream (True or False)
• temperature (number 0..1.5)
• top_p (number)

Example:

[
    {
        "model": "llama3.1-70b",
        "api_key": "your Cerebras API Key goes here",
        "api_type": "cerebras"
        "max_tokens": 10000,
        "seed": 1234,
        "stream" True,
        "temperature": 0.5,
        "top_p": 0.2, # Note: It is recommended to set temperature or top_p but not both.
    }
]

Two-Agent Coding Example

In this example, we run a two-agent chat with an AssistantAgent (primarily a coding agent) to generate code to count the number of prime numbers between 1 and 10,000 and then it will be executed.

We’ll use Meta’s LLama-3.1-70B model which is suitable for coding.

import os

from autogen.oai.cerebras import CerebrasClient, calculate_cerebras_cost

config_list = [{"model": "llama3.1-70b", "api_key": os.environ.get("CEREBRAS_API_KEY"), "api_type": "cerebras"}]

Importantly, we have tweaked the system message so that the model doesn’t return the termination keyword, which we’ve changed to FINISH, with the code block.

from pathlib import Path

from autogen import AssistantAgent, UserProxyAgent
from autogen.coding import LocalCommandLineCodeExecutor

# Setting up the code executor
workdir = Path("coding")
workdir.mkdir(exist_ok=True)
code_executor = LocalCommandLineCodeExecutor(work_dir=workdir)

# Setting up the agents

# The UserProxyAgent will execute the code that the AssistantAgent provides
user_proxy_agent = UserProxyAgent(
    name="User",
    code_execution_config={"executor": code_executor},
    is_termination_msg=lambda msg: "FINISH" in msg.get("content"),
)

system_message = """You are a helpful AI assistant who writes code and the user executes it.
Solve tasks using your coding and language skills.
In the following cases, suggest python code (in a python coding block) for the user to execute.
Solve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.
When using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.
Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.
If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
When you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.
IMPORTANT: Wait for the user to execute your code and then you can reply with the word "FINISH". DO NOT OUTPUT "FINISH" after your code block."""

# The AssistantAgent, using Llama-3.1-70B on Cerebras Inference, will take the coding request and return code
assistant_agent = AssistantAgent(
    name="Cerebras Assistant",
    system_message=system_message,
    llm_config={"config_list": config_list},
)

# Start the chat, with the UserProxyAgent asking the AssistantAgent the message
chat_result = user_proxy_agent.initiate_chat(
    assistant_agent,
    message="Provide code to count the number of prime numbers from 1 to 10000.",
)

User (to Cerebras Assistant):

Provide code to count the number of prime numbers from 1 to 10000.

--------------------------------------------------------------------------------
Cerebras Assistant (to User):

To count the number of prime numbers from 1 to 10000, we will utilize a simple algorithm that checks each number in the range to see if it is prime. A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself.

Here's how we can do it using a Python script:

```python
def count_primes(n):
    primes = 0
    for possiblePrime in range(2, n + 1):
        # Assume number is prime until shown it is not. 
        isPrime = True
        for num in range(2, int(possiblePrime ** 0.5) + 1):
            if possiblePrime % num == 0:
                isPrime = False
                break
        if isPrime:
            primes += 1
    return primes

# Counting prime numbers from 1 to 10000
count = count_primes(10000)
print(count)
```

Please execute this code. I will respond with "FINISH" after you provide the result.

--------------------------------------------------------------------------------
Replying as User. Provide feedback to Cerebras Assistant. Press enter to skip and use auto-reply, or type 'exit' to end the conversation:  

>>>>>>>> NO HUMAN INPUT RECEIVED.

Tool Call Example

In this example, instead of writing code, we will show how Meta’s Llama-3.1-70B model can perform parallel tool calling, where it recommends calling more than one tool at a time.

We’ll use a simple travel agent assistant program where we have a couple of tools for weather and currency conversion.

We start by importing libraries and setting up our configuration to use Llama-3.1-70B and the cerebras client class.

import json
import os
from typing import Literal

from typing_extensions import Annotated

import autogen

config_list = [
    {
        "model": "llama3.1-70b",
        "api_key": os.environ.get("CEREBRAS_API_KEY"),
        "api_type": "cerebras",
    }
]

Create our two agents.

# Create the agent for tool calling
chatbot = autogen.AssistantAgent(
    name="chatbot",
    system_message="""
        For currency exchange and weather forecasting tasks,
        only use the functions you have been provided with.
        When you summarize, make sure you've considered ALL previous instructions.
        Output 'HAVE FUN!' when an answer has been provided.
    """,
    llm_config={"config_list": config_list},
)

# Note that we have changed the termination string to be "HAVE FUN!"
user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    is_termination_msg=lambda x: x.get("content", "") and "HAVE FUN!" in x.get("content", ""),
    human_input_mode="NEVER",
    max_consecutive_auto_reply=1,
)

Create the two functions, annotating them so that those descriptions can be passed through to the LLM.

We associate them with the agents using register_for_execution for the user_proxy so it can execute the function and register_for_llm for the chatbot (powered by the LLM) so it can pass the function definitions to the LLM.

# Currency Exchange function

CurrencySymbol = Literal["USD", "EUR"]

# Define our function that we expect to call


def exchange_rate(base_currency: CurrencySymbol, quote_currency: CurrencySymbol) -> float:
    if base_currency == quote_currency:
        return 1.0
    elif base_currency == "USD" and quote_currency == "EUR":
        return 1 / 1.1
    elif base_currency == "EUR" and quote_currency == "USD":
        return 1.1
    else:
        raise ValueError(f"Unknown currencies {base_currency}, {quote_currency}")


# Register the function with the agent


@user_proxy.register_for_execution()
@chatbot.register_for_llm(description="Currency exchange calculator.")
def currency_calculator(
    base_amount: Annotated[float, "Amount of currency in base_currency"],
    base_currency: Annotated[CurrencySymbol, "Base currency"] = "USD",
    quote_currency: Annotated[CurrencySymbol, "Quote currency"] = "EUR",
) -> str:
    quote_amount = exchange_rate(base_currency, quote_currency) * base_amount
    return f"{format(quote_amount, '.2f')} {quote_currency}"


# Weather function


# Example function to make available to model
def get_current_weather(location, unit="fahrenheit"):
    """Get the weather for some location"""
    if "chicago" in location.lower():
        return json.dumps({"location": "Chicago", "temperature": "13", "unit": unit})
    elif "san francisco" in location.lower():
        return json.dumps({"location": "San Francisco", "temperature": "55", "unit": unit})
    elif "new york" in location.lower():
        return json.dumps({"location": "New York", "temperature": "11", "unit": unit})
    else:
        return json.dumps({"location": location, "temperature": "unknown"})


# Register the function with the agent


@user_proxy.register_for_execution()
@chatbot.register_for_llm(description="Weather forecast for US cities.")
def weather_forecast(
    location: Annotated[str, "City name"],
) -> str:
    weather_details = get_current_weather(location=location)
    weather = json.loads(weather_details)
    return f"{weather['location']} will be {weather['temperature']} degrees {weather['unit']}"

We pass through our customer’s message and run the chat.

Finally, we ask the LLM to summarise the chat and print that out.

import time

start_time = time.time()

# start the conversation
res = user_proxy.initiate_chat(
    chatbot,
    message="What's the weather in New York and can you tell me how much is 123.45 EUR in USD so I can spend it on my holiday? Throw a few holiday tips in as well.",
    summary_method="reflection_with_llm",
)

end_time = time.time()

print(f"LLM SUMMARY: {res.summary['content']}\n\nDuration: {(end_time - start_time) * 1000}ms")

user_proxy (to chatbot):

What's the weather in New York and can you tell me how much is 123.45 EUR in USD so I can spend it on my holiday? Throw a few holiday tips in as well.

--------------------------------------------------------------------------------
chatbot (to user_proxy):

***** Suggested tool call (210f6ac6d): weather_forecast *****
Arguments: 
{"location": "New York"}
*************************************************************
***** Suggested tool call (3c00ac7d5): currency_calculator *****
Arguments: 
{"base_amount": 123.45, "base_currency": "EUR", "quote_currency": "USD"}
****************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION weather_forecast...

>>>>>>>> EXECUTING FUNCTION currency_calculator...
user_proxy (to chatbot):

user_proxy (to chatbot):

***** Response from calling tool (210f6ac6d) *****
New York will be 11 degrees fahrenheit
**************************************************

--------------------------------------------------------------------------------
user_proxy (to chatbot):

***** Response from calling tool (3c00ac7d5) *****
135.80 USD
**************************************************

--------------------------------------------------------------------------------
chatbot (to user_proxy):

New York will be 11 degrees fahrenheit.
123.45 EUR is equivalent to 135.80 USD.
 
For a great holiday, explore the Statue of Liberty, take a walk through Central Park, or visit one of the many world-class museums. Also, you'll find great food ranging from bagels to fine dining experiences. HAVE FUN!

--------------------------------------------------------------------------------
LLM SUMMARY: New York will be 11 degrees fahrenheit. 123.45 EUR is equivalent to 135.80 USD. Explore the Statue of Liberty, walk through Central Park, or visit one of the many world-class museums for a great holiday in New York.

Duration: 73.97937774658203ms

We can see that the Cerebras Wafer-Scale Engine-3 (WSE-3) completed the query in 74ms – faster than the blink of an eye!