Module tinytroupe.validation.propositions
There are various general desireable simulation properties. These can be useful under various circumstances, for example to validate the simulation, or to monitor it during its execution.
Expand source code
"""
There are various general desireable simulation properties. These can be useful under various
circumstances, for example to validate the simulation, or to monitor it during its execution.
"""
from tinytroupe.experimentation import Proposition
#################################
# Auxiliary internal functions
#################################
def _build_precondition_function_for_action_types(action_types:list, check_for_presence:bool):
"""
Builds a precondition function that checks if the action is or is not in a list of action types.
The resulting function is meant to be used as a precondition function for propositions.
Args:
action_types (list): A list of action types to check against.
check_for_presence (bool): If True, the function checks if the action type is in the list.
If False, it checks if the action type is NOT in the list.
Returns:
function: A precondition function that takes a target, additional context, and claim variables as arguments.
"""
def precondition_function(target, additional_context, claim_variables):
action_type = claim_variables.get("action").get("type")
if check_for_presence:
# Check if the action type is in the list of valid action types
if action_type in action_types:
return True
else:
return False
else:
# Check if the action type is NOT in the list of valid action types
if action_type not in action_types:
return True
else:
return False
return precondition_function
###############################
# Agent properties
###############################
persona_adherence = \
Proposition(\
f"""
THE AGENT ADHERES TO THE PERSONA SPECIFICATION:
the agent behavior seen during the simulation is consistent with the agent's persona specification, it is
what is expected from the agent's persona specification. In particular, consider these criteria:
- The personality traits specified in the persona are respected.
- The persona style is respected.
- The persona beliefs are respected.
- The persona behaviors are respected.
- The persona skills are respected.
- Any other aspect of the persona specification is respected.
How to evaluate adherence:
- Each of the above criteria should have equal weight in the evaluation, meaning that the score is the average of the scores of each criterion.
- The adherence should be checked against all actions in the simulation trajectory. The final score should be an average of the scores of all
actions in the trajectory.
""",
include_personas=True,
double_check=True)
action_persona_adherence = \
Proposition(\
"""
THE NEXT AGENT ACTION ADHERES TO THE PERSONA SPECIFICATION:
the agent's next action is consistent with the agent's persona specification, it is
what is expected from the agent's persona specification. In particular, consider these criteria:
- The personality traits specified in the persona are respected.
- The persona style is respected.
- The persona beliefs are respected.
- The persona behaviors are respected.
- The persona skills are respected.
- Any other aspect of the persona specification is respected.
THIS IS THE NEXT ACTION: {{action}}
How to evaluate adherence:
- Each of the above criteria should have equal weight in the evaluation, meaning that the score is the average of the scores of each criterion.
- The adherence is ONLY ABOUT the next action mentioned above and the persona specification. DO NOT take into account previous actions or stimuli.
- The general situation context is irrelevant to this evaluation, you should ONLY consider the persona specification as context.
- Do not imagine what would be the next action, but instead judge the proposed next action mentioned above!
- The simulation trajectories provided in the context DO NOT contain the next action, but only the actions and stimuli
that have already happened.
""",
include_personas=True,
double_check=False,
first_n=5, last_n=10,
precondition_function=_build_precondition_function_for_action_types(["THINK", "TALK"], check_for_presence=True))
hard_persona_adherence = \
Proposition(\
f"""
THE AGENT FULLY ADHERES TO THE PERSONA SPECIFICATION:
the agent behavior seen during the simulation is completely consistent with the agent's persona specification, it is
exactly what is expected from the agent's persona specification. Nothing at all contradicts the persona specification.
How to evaluate adherence:
- For any flaw found, you **must** subtract 20% of the score, regardless of its severity. This is to be very harsh and avoid any ambiguity.
""",
include_personas=True,
double_check=True)
hard_action_persona_adherence = \
Proposition(\
"""
THE NEXT AGENT ACTION FULLY ADHERES TO THE PERSONA SPECIFICATION:
the agent's next action is completely consistent with the agent's persona specification, it is
what is exactly expected from the agent's persona specification. Nothing at all contradicts the persona specification.
THIS IS THE NEXT ACTION: {{action}}
How to evaluate adherence:
- For any flaw found, you **must** subtract 20% of the score, regardless of its severity. This is to be very harsh and avoid any ambiguity.
- The adherence is ONLY ABOUT the next action mentioned above and the persona specification. DO NOT take into account previous actions or stimuli.
- The general situation context is irrelevant to this evaluation, you should ONLY consider the persona specification as context.
- Do not imagine what would be the next action, but instead judge the proposed next action mentioned above!
- The simulation trajectories provided in the context DO NOT contain the next action, but only the actions and stimuli
that have already happened.
""",
include_personas=True,
double_check=False,
first_n=5, last_n=10,
precondition_function=_build_precondition_function_for_action_types(["THINK", "TALK"], check_for_presence=True))
self_consistency = \
Proposition(
f"""
THE AGENT IS SELF-CONSISTENT:
the agent never behaves in contradictory or inconsistent ways.
""",
include_personas=False,
double_check=True)
action_self_consistency = \
Proposition(
"""
THE NEXT AGENT ACTION IS SELF-CONSISTENT:
the agent's next action does not contradict or conflict with the agent's previous actions.
THIS IS THE NEXT ACTION: {{action}}
How to evaluate action self-consistency:
- Consider the previous actions ONLY to form your opinion about whether the next action is consistent with them
- Ignore stimuli and other previous events, the self-consistency concerns ONLY actions.
- Actions and stimuli ARE NOT part of the persona specification. Rather, they are part of the simulation trajectories.
- Ignore the agent's persona or general background, the self-consistency concerns ONLY the actions observed
in simulation trajectories.
- If there are no previous actions, the next action is self-consistent by default.
""",
include_personas=False,
first_n=5, last_n=10,
precondition_function=_build_precondition_function_for_action_types(["THINK", "TALK"], check_for_presence=True))
fluency = \
Proposition(\
"""
THE AGENT IS FLUENT. During the simulation, the agent's thinks and speaks fluently. This means that:
- The agent don't repeat the same thoughts or words over and over again.
- The agents don't use overly formulaic language.
- The agent don't use overly repetitive language.
- The agent's words sound natural and human-like.
""",
include_personas=False,
double_check=True)
action_fluency = \
Proposition(\
"""
THE NEXT AGENT ACTION IS FLUENT.
The next action's words sounds natural and human-like, avoiding excessive repetition and formulaic language.
THIS IS THE NEXT ACTION: {{action}}
How to evaluate fluency:
- Fluency here is ONLY ABOUT the next action mentioned above. Previous actions are the **context** for this evaluation,
but will not be evaluated themselves.
- Previous stimuli and events that are not actions should be completely ignored. Here we are only concerned about actions.
""",
include_personas=False,
first_n=5, last_n=10,
precondition_function=_build_precondition_function_for_action_types(["THINK", "TALK"], check_for_presence=True))
action_suitability = \
Proposition(\
"""
THE NEXT AGENT ACTION IS SUITABLE:
the next action is suitable for the situation, task and context. In particular, if the agent is pursuing some
specific goal, instructions or guidelines, the next action must be coherent and consistent with them.
More precisely, the next action is suitable if at least *one* of the following conditions is satisfied:
- the next action is a reasonable step in the right direction, even if does not need to fully solve the overall problem, task or situation.
- the next action produces relevant information for the situation, task or context, even if does not actually advances a solution.
- the next action is a reasonable response to the recent stimuli received, even if it does not actually advances a solution.
It suffices to meet ONLY ONE of these conditions to be considered **FULLY** suitable.
THIS IS THE NEXT ACTION: {{action}}
How to evaluate action suitability:
- The score of suitability is proportional to the degree to which the next action satisfies *any* of the above conditions
- If only **one** condition is **fully** met, the next action is **completely** suitable and gets **maximum** score. That is to say,
the next action **does not** need to satisfy all conditions to be suitable! A single sataisfied condition is enough!
- The suitability is ONLY ABOUT the next action mentioned above and the situation context.
- If a previous action or stimuli is inconsistent or conflicting with the situation context, you should ignore it
when evaluating the next action. Consider ONLY the situation context.
- The simulation trajectories provided in the context DO NOT contain the next action, but only the actions and stimuli
that have already happened.
""",
include_personas=True,
first_n=5, last_n=10,
precondition_function=_build_precondition_function_for_action_types(["THINK", "TALK"], check_for_presence=True))
task_completion = \
Proposition(\
"""
THE AGENT COMPLETES THE GIVEN TASK.
Given the following task: "{{task_description}}"
The agent completes the task by the end of the simulation.
This means that:
- If the task requires the agent to discuss or talk about something, the agent does so.
- If the task requires the agent to think about something, the agent does so.
- If the task requires the agent to do something via another action, the agent does so.
- If the task requires the agent to adopt some specific variations of behavior, the agent does so.
- If the task includes other specific requirements, the agent observes them.
""",
include_personas=False,
double_check=True)
quiet_recently = \
Proposition(
"""
THE AGENT HAS BEEN QUIET RECENTLY:
The agent has been executing multiple DONE actions in a row with few or no TALK, THINK or
other actions in between.
How to evaluate quietness:
- The last 2 (or more) actions of the agent are consecutive DONE actions. This means that the agent
was done with his turn before doing anything else for a couple of turns.
- There are no other actions in between the last 2 (or more) DONE actions.
""",
include_personas=False
)
##################################
# Environment properties
##################################
divergence = \
Proposition("""
AGENTS DIVERGE FROM ONE ANOTHER.
As the simulation progresses, the agents' behaviors diverge from one another,
instead of becoming more similar. This includes what they think, what they say and what they do. The topics discussed become
more varied at the end of the simulation than at the beginning. Discussions do not converge to a single topic or perspective
at the end.
""",
include_personas=False,
double_check=True)
convergence = \
Proposition("""
AGENTS CONVERGE TO ONE ANOTHER.
As the simulation progresses, the agents' behaviors converge to one another,
instead of becoming more different. This includes what they think, what they say and what they do. The topics discussed become
more similar at the end of the simulation than at the beginning. Discussions converge to a single topic or perspective
at the end.
""",
include_personas=False,
double_check=True)