Handling failures

Hide code cell source 
# %load -r :19 ../_init.py
import pathlib
import sammo
from sammo.runners import OpenAIChat
from sammo.base import Template, EvaluationScore
from sammo.components import Output, GenerateText, ForEach, Union
from sammo.extractors import ExtractRegex
from sammo.data import DataTable
import json
import requests

API_CONFIG_FILE = pathlib.Path().cwd().parent.parent / "config" / "personal.openai"
API_CONFIG = ""
if API_CONFIG_FILE.exists():
    API_CONFIG = API_CONFIG_FILE
if not API_CONFIG:
    raise ValueError('Please set API_CONFIG to {"api_key": "YOUR_KEY"}')

_ = sammo.setup_logger("WARNING")  # we're only interested in warnings for now

Handling failures#

There are two main types of failures that can happen during the execution of a metaprompt – network failures and processing failures. By default, SAMMO will retry the network request a few times before giving up.

Network request failures#

The two most common network request failures are timeouts and rejected requests (mostly due to rate limiting).

Timeout errors#

Let’s simulate a timeout error.

runner = OpenAIChat(
    model_id="gpt-3.5-turbo-16k",
    api_config=API_CONFIG,
    cache=CACHE_FILE,
    timeout=0.01,
)
Output(GenerateText("Generate a 5000 word essay about horses.")).run(
    runner, progress_callback=False
)

09:14:34,419: TimeoutError: {'messages': [{'role': 'user', 'content': 'Generate a 5000 word essay about horses.'}], 'max_tokens': None, 'temperature': 0}
09:14:34,438: TimeoutError: {'messages': [{'role': 'user', 'content': 'Generate a 5000 word essay about horses.'}], 'max_tokens': None, 'temperature': 0}
09:14:34,438: Failed to generate text: TimeoutError()

+---------+-------------------------------------------------------------+
| input   | output                                                      |
+=========+=============================================================+
| None    | TimeoutResult(value='TimeoutError()'..., parent=TextResult) |
+---------+-------------------------------------------------------------+
Constants: None

Here, we can see that SAMMO re-tried it once and then returned a TimeoutResult.

To customize how these are handled, you can specify the following parameters:

  • timeout: The timeout for the network request. Defaults to 60 seconds.

  • max_timeout_retries: The maximum number of times to retry a network request in case of a timeout. Defaults to 1.

Network Errors#

Let’s see how SAMMO behaves under network errors.

import openai

runner = OpenAIChat(
    model_id="gpt-1",
    api_config=API_CONFIG,
    cache=CACHE_FILE,
    max_retries=1,
    retry_on=(openai.error.InvalidRequestError,),
)
Output(
    GenerateText("Generate a 5000 word essay about horses.", on_error="empty_result")
).run(runner, progress_callback=False)

09:14:34,615: Failed to generate text: InvalidRequestError(message='The model `gpt-1` does not exist', param=None, code='model_not_found', http_status=404, request_id=None)

+---------+-----------------------------------------------------------+
| input   | output                                                    |
+=========+===========================================================+
| None    | EmptyResult(value="InvalidRequestError(message='The model |
|         | `gpt-1` does not exist', param=None,                      |
|         | code='model_not_found',..., parent=TextResult)            |
+---------+-----------------------------------------------------------+
Constants: None

Here, SAMMO returns an empty result after retrying once.

Component failures#

Component failures typically are errors that occur when the code in a component cannot be run correctly. This often happens when the LLM output cannot be parsed correctly or contains the wrong number of rows for a minibatch. By default, these failures are raised in order to have the user make an explicit decision on how exceptions should handled, with the exception of GenerateText which returns an empty result since those failures are very common.

Note

A little different from typical Python, SAMMO encourages developers to not let errors bubble up but catch them where they happen. This makes it easier to locate errors in the pipeline.

To manage exceptions, you can specify the following parameter when creating certain Component instances:

  • on_error: Choose between raise (default) or empty_result (other options might be available). If empty_result is chosen, the component will return an EmptyResult object instead of raising an exception.

Note

Components that are expected to always complete have no on_error option.

Example: Parsing error#

The input string here is invalid JSON, so after a failed parse attempt, we will get a EmptyResult.

from sammo.extractors import ParseJSON
from sammo.base import VerbatimText

parsed = Output(ParseJSON("{[}", parse_fragments="whole", on_error="empty_result")).run(
    runner, progress_callback=False
)
parsed

09:14:34,649: Error extracting from [TextResult(value='{[}'..., parent=NoneType)]: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

+---------+-----------------------------------------+
| input   | output                                  |
+=========+=========================================+
| None    | EmptyResult(value=None..., parent=list) |
+---------+-----------------------------------------+
Constants: None

What to do with an EmptyResult#

When we have empty results in the final output, it is up to the developer to decide how these cases should be handled.

A common case is to replace all empty results with a sensible default value, e.g., 0 when numbers are required.

parsed.outputs.normalized_values(on_empty=0)

[0]

Alternatively, we can simply filter them out.

parsed.outputs.nonempty_values()

[]
Contents