promptflow.evals.synthetic module#

class promptflow.evals.synthetic.AdversarialScenario(value)#

Bases: Enum

An enumeration.

ADVERSARIAL_CONTENT_GEN_GROUNDED = 'adv_content_gen_grounded'#
ADVERSARIAL_CONTENT_GEN_UNGROUNDED = 'adv_content_gen_ungrounded'#
ADVERSARIAL_CONVERSATION = 'adv_conversation'#
ADVERSARIAL_QA = 'adv_qa'#
ADVERSARIAL_REWRITE = 'adv_rewrite'#
ADVERSARIAL_SUMMARIZATION = 'adv_summarization'#
class promptflow.evals.synthetic.AdversarialSimulator(*, azure_ai_project: Dict[str, Any])#

Bases: object

Initializes the adversarial simulator with a project scope.

Parameters:

azure_ai_project (Dict[str, Any]) –

Dictionary defining the scope of the project. It must include the following keys:

  • ”subscription_id”: Azure subscription ID.

  • ”resource_group_name”: Name of the Azure resource group.

  • ”project_name”: Name of the Azure Machine Learning workspace.

  • ”credential”: Azure credentials object for authentication.

__call__(*, scenario: AdversarialScenario, target: Callable, max_conversation_turns: int = 1, max_simulation_results: int = 3, api_call_retry_limit: int = 3, api_call_retry_sleep_sec: int = 1, api_call_delay_sec: int = 0, concurrent_async_task: int = 3, jailbreak: bool = False)#

Executes the adversarial simulation against a specified target function asynchronously.

Parameters:

  • scenario (promptflow.evals.synthetic.adversarial_scenario.AdversarialScenario) –

    Enum value specifying the adversarial scenario used for generating inputs. example:

  • target (Callable) – The target function to simulate adversarial inputs against. This function should be asynchronous and accept a dictionary representing the adversarial input.

  • max_conversation_turns (int) – The maximum number of conversation turns to simulate. Defaults to 1.

  • max_simulation_results (int) – The maximum number of simulation results to return. Defaults to 3.

  • api_call_retry_limit (int) – The maximum number of retries for each API call within the simulation. Defaults to 3.

  • api_call_retry_sleep_sec (int) – The sleep duration (in seconds) between retries for API calls. Defaults to 1 second.

  • api_call_delay_sec (int) – The delay (in seconds) before making an API call. This can be used to avoid hitting rate limits. Defaults to 0 seconds.

  • concurrent_async_task (int) – The number of asynchronous tasks to run concurrently during the simulation. Defaults to 3.

  • jailbreak (bool) – If set to True, allows breaking out of the conversation flow defined by the scenario. Defaults to False.

Returns:

A list of dictionaries, each representing a simulated conversation. Each dictionary contains:

  • ’template_parameters’: A dictionary with parameters used in the conversation template,

    including ‘conversation_starter’.

  • ’messages’: A list of dictionaries, each representing a turn in the conversation.

    Each message dictionary includes ‘content’ (the message text) and ‘role’ (indicating whether the message is from the ‘user’ or the ‘assistant’).

  • ’$schema’: A string indicating the schema URL for the conversation format.

The ‘content’ for ‘assistant’ role messages may includes the messages that your callback returned.

Return type:

List[Dict[str, Any]]

call_sync(*, max_conversation_turns: int, max_simulation_results: int, target: Callable, api_call_retry_limit: int, api_call_retry_sleep_sec: int, api_call_delay_sec: int, concurrent_async_task: int)#

Submodules#