WebTailBench — V1 vs V2 task & rubric diff

+ "criterion": "Find the specific Reddit post in r/golf referencing '3000 sq miles' of land used by golf courses in the USA",

+ "description": "Locate a Reddit post (thread) in r/golf that explicitly contains (or is clearly centered on) the claim that golf courses take up about '3000 sq miles' of land in the USA. Full credit if the agent identifies the post unambiguously (e.g., correct title/author/date and/or direct link) and shows evidence the phrase/claim is present in the post content (quote/snippet/screenshoted text). Full credit also if Reddit is inaccessible (login wall/CAPTCHA/rate limits/removed content) or if the exact post cannot be located after reasonable attempts (e.g., Reddit search plus external search engine queries) and the agent transparently reports the blocker and provides the closest verifiable match found (including how it differs). Partial credit if a closely related post is found but the required '3000 sq miles' claim is not clearly evidenced.",

+ "criterion": "Identify the top upvoted comment on that post and summarize it",

+ "description": "Determine which comment appears to be the top upvoted comment on the identified post at the time of viewing (e.g., by sorting by Top and noting author/score/time seen or other identifying markers) and summarize its main points accurately. Full credit if the agent clearly identifies the top comment (author + excerpt and score/order when visible) and summarizes it. Full credit also if the agent makes a reasonable attempt to verify the top comment but cannot conclusively do so due to external factors (deleted/removed comments, hidden scores, sorting not available, pagination/collapsing, access restrictions) and explicitly states the limitation while summarizing the best-supported candidate top comment. Partial credit if the summary is generally accurate but verification that it is the top comment is weak/unclear when it was determinable.",

+ "criterion": "Substantiate a major claim from the top comment using another website",

+ "description": "Find at least one non-Reddit website that substantiates a major claim actually made in the (identified or best-candidate) top comment (e.g., land use figures, water usage, environmental impact, number of courses, etc.). Full credit if the agent selects a major claim from that comment, cites an external source that supports it, and explains the connection. Full credit also if the agent is blocked from accessing external sources (paywalls/CAPTCHA/geo-blocking) and transparently reports the limitation while providing the best available alternative source(s) it can access (or clearly stating none could be reached). Partial credit if the source is only tangential, supports a different claim than cited, or the linkage is unclear.",

+ "max_points": 4,

+ "criterion": "Attempt to use an online mortgage payment calculator tool",

+ "description": "Agent attempts to access and use a publicly available online mortgage payment/rate calculator. Full credit if the agent clearly names the tool/site used OR reports a genuine blocker (CAPTCHA, paywall, outage, region restriction) after a reasonable attempt and then proceeds with a reasonable alternative approach (e.g., a different calculator site or a transparent manual calculation). Partial credit if the agent only performs a manual calculation without attempting an online tool. No credit if the agent fabricates having used an online tool or provides results without any stated method.",

+ "criterion": "Use correct loan scenario inputs (price/down payment or loan amount, term, rate, ZIP when applicable)",

+ "description": "Uses the explicitly stated values consistently: home price $500,000; down payment $80,000 (or loan amount $420,000); 30-year term; 6.0% interest rate. ZIP 98101 should be entered if the chosen tool uses ZIP for rate/fees context; if the tool does not request or use ZIP for P&I, omission should not reduce credit. Partial credit if one non-essential field (e.g., ZIP) is omitted due to tool limitations but the P&I calculation basis (loan amount/term/rate) is correct. No credit if key financial inputs (loan amount/price-down, term, or interest rate) are incorrect when they could have been entered correctly.",

+ "criterion": "Report estimated monthly payment for principal and interest only",

+ "description": "Reports the monthly payment amount specifically for principal + interest (P&I) only, clearly excluding taxes, insurance, HOA, PMI, and other fees. Full credit if the agent selects/identifies the P&I-only output from the tool, or if the tool is inaccessible and the agent provides a correct P&I-only payment via a transparent calculation. Partial credit if the agent reports a total payment that includes other components but also clearly provides or isolates the P&I portion. No credit if the agent reports only an all-in payment with add-ons and does not provide a distinguishable P&I-only figure.",

+ "criterion": "Retrieve view count for Adele’s 'Rolling in the Deep (Official Music Video)'",

+ "description": "Determine the view count for the specific official YouTube music video titled 'Rolling in the Deep (Official Music Video)' by Adele (not a lyric video, live performance, or reupload). Full credit if the agent clearly identifies the correct official video and reports the view count with an 'as of' date/time (or states it is approximate if the source only provides an approximation). If YouTube is inaccessible due to external blockers (CAPTCHA, region restriction, consent/login wall, outage), award full credit if the agent clearly reports the blocker and retrieves the view count from a reasonable alternative authoritative listing (e.g., Google video result panel, official channel listing, reputable analytics/metadata source) while noting the source and that the number may be stale/approximate. Partial credit if the video identity is ambiguous/mismatched or the view count is reported without any recency context (no as-of date/time) when access was available.",

+ "criterion": "Compute percent of world population represented by the view count",

+ "description": "Compute (views ÷ world population) × 100 and report the resulting percentage. Full credit if the agent (a) states the world population figure used and its basis/source and date (or that it is an estimate), (b) uses the correct formula, and (c) reports the percentage with reasonable rounding. If an authoritative world-population figure cannot be retrieved due to external access limitations, award full credit if the agent uses a clearly labeled, reasonable cached/commonly cited estimate and notes the limitation/date. Partial credit if the method is correct but arithmetic/rounding is slightly off or the population basis/date is missing.",

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Transparent reporting of tool use, inputs, and calculation",

+ "description": "Provide sufficient transparency to verify the result: include the key inputs (view count, world population), indicate the retrieval source(s) or access limitation encountered, and show the computation (at least the formula and substituted numbers, or an equivalent calculator expression). Full credit if these are present even when values are approximate due to external blockers. Partial credit if only the final percentage is provided with minimal or no inputs shown.",

+ "max_points": 3,

+ "criterion": "Locate the top-seller RPG game on Steam",

+ "description": "Identify which RPG game is currently the top-seller on Steam (i.e., the leading title in Steam's RPG category/top sellers view). Full credit if the agent navigates Steam and correctly identifies the current top-selling RPG game. Partial credit if the agent identifies a plausible Steam RPG best-seller but uses an unclear/incorrect Steam section (e.g., not top-sellers or not RPG-specific) while still documenting what was used. Full credit if Steam access is blocked (CAPTCHA/login/region/technical error) and the agent clearly reports the blocker and what evidence was available.",

+ "description": "Determine the matching/recommended controller for the identified top-selling RPG game (e.g., from the Steam store page controller support section or explicitly mentioned recommended controller). Full credit if the controller is correctly identified and clearly tied to the game. Partial credit if the agent identifies a broadly compatible controller but does not show it is the specific matching/recommended one for that game. Full credit if the game page does not specify a matching controller and the agent clearly reports that limitation and uses the best available evidence (e.g., controller support info).",

+ "max_points": 3,

+ "description": "Search Amazon and reach a product listing that matches the identified controller (correct model/brand; reasonable match to the controller identified in the prior step). Full credit if the correct controller listing is found. Partial credit if a close variant is selected (e.g., different edition/color/bundle) when the exact match is available, or if matching is ambiguous but the agent explains the ambiguity. Full credit if Amazon blocks access (CAPTCHA/login wall/technical error) and the agent reports the blocker and attempts reasonable navigation/search alternatives within Amazon.",

+ "max_points": 3,

+ "criterion": "Add the controller to the Amazon cart and proceed to (or as close as possible to) the review page without purchasing or fabricating personal data",

+ "description": "Add the selected controller to the Amazon cart and proceed through the checkout flow up to the 'review order' stage (or the closest equivalent pre-purchase review step), stopping there and not placing an order. Full credit if the item is added to cart and the agent reaches the review page (or Amazon's closest equivalent) and stops without submitting the final order, and does not invent/enter sensitive personal or payment details. Partial credit if the item is added to cart but the agent is forced to stop earlier due to external gates (required sign-in, address/payment requirement, stock/fulfillment restrictions, CAPTCHA/blocks) and clearly reports the blocker and the furthest achievable step, or if the agent reaches checkout fields but leaves them blank and asks the user for required inputs rather than guessing. No credit if the agent attempts to place the order, submits the final order action, or fabricates/enters sensitive personal/payment information.",

+ "max_points": 10,

+ "criterion": "Check Steam top-selling list for today and identify the #1 top-selling game",

+ "description": "Use Steam (e.g., Steam Store Top Sellers) to determine the first/top (#1) top-selling game today, noting the storefront/region context used (since rankings vary by region and currency). Full credit if the agent clearly identifies the #1 game as shown on Steam for today in the chosen/observable region/context. Partial credit if the agent uses Steam but the #1 status is ambiguous (e.g., wrong sorting, unclear region, or unclear that it is 'Top Sellers' for today). Full credit if the agent attempts Steam access but cannot verify #1 due to uncontrollable blockers (CAPTCHA, downtime, age gate, forced login, region gating) and clearly explains what prevented confirmation and what context was attempted.",

+ "criterion": "Determine whether the #1 top-selling game has a TV series adaptation (if any)",

+ "description": "Assess whether the identified #1 top-selling game has a TV series adaptation and provide the adaptation title if it exists. Full credit if the agent correctly determines existence or non-existence and names the series when applicable. Partial credit if the linkage is unclear (e.g., names a related adaptation but not clearly tied to the game) or the series is not fully identified. Full credit if the agent reports that no TV adaptation exists (or none could be found after reasonable effort) without inventing details.",

+ "max_points": 3,

+ "criterion": "Use JustWatch.com to find streaming services for the TV series adaptation",

+ "description": "If a TV series adaptation exists, use JustWatch.com to locate the correct series page and report the streaming services where it is available, including subscription vs rent/buy if shown, and noting the country setting used (since availability is country-specific). Full credit if the services listed match JustWatch for that series in the relevant/observable country context. Partial credit if JustWatch is used but the service list is incomplete/unclear or the wrong country context is used without acknowledgement. Full credit if JustWatch is blocked/unavailable or the series is not listed on JustWatch despite a reasonable attempt, as long as the agent clearly reports the blocker/non-listing and does not fabricate availability; the agent may optionally provide an alternative source only after attempting JustWatch.",

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Handle the 'if any' condition appropriately (no adaptation case)",

+ "condition": "Only applies if the identified #1 top-selling game has no TV series adaptation",

+ "description": "If no TV series adaptation exists for the #1 game, the agent should explicitly state that and avoid inventing a series or streaming availability. Full credit for clearly reporting that no TV series adaptation was found/exists and stopping (or stating that JustWatch lookup is not applicable). Partial credit if the agent is ambiguous about whether an adaptation exists or does not describe any search/verification effort. No credit if the agent hallucinates an adaptation or streaming services.",

+ "max_points": 2,

+ "criterion": "Access universityofcalifornia.edu news content and attempt to locate the earliest article",

+ "description": "Attempt to access the universityofcalifornia.edu site(s) and navigate to a news listing/archive (or use on-site search / sitemap / structured filters). Full credit if the agent makes a reasonable attempt and either (a) reaches a page/listing that plausibly supports finding the earliest article, or (b) clearly reports an external blocker (site down, paywall, CAPTCHA, archive inaccessible, broken search/pagination) with what was tried. Partial credit if the attempt is minimal (e.g., only one quick query) or uses an unclear method. No credit if the agent does not attempt to use universityofcalifornia.edu.",

+ "criterion": "Identify the first (earliest) news article published on universityofcalifornia.edu, with verifiable evidence or a clearly bounded best-effort alternative",

+ "description": "Provide the earliest (first-published) news article on universityofcalifornia.edu with identifying details sufficient to verify (e.g., headline + publication date + URL or page evidence). Full credit if the agent either (a) correctly identifies the earliest article using a defensible, verifiable method (e.g., sorted-by-oldest archive, reaching the last page of results, or another site-native mechanism), or (b) if definitive identification is not possible due to external constraints (missing archives, non-sortable feeds, broken pagination/search), the agent clearly explains the limitation and supplies the best verifiable alternative found (e.g., the oldest article reachable via available listings) while stating it may not be the absolute first. Partial credit if an early article is provided without evidence/method, or if the result is not clearly on universityofcalifornia.edu.",

+ "criterion": "Determine and report the author/byline of the identified earliest article (or note absence/ambiguity)",

+ "description": "Extract and report the author/byline exactly as shown on the earliest-article page. Full credit if the agent correctly reports the byline, or accurately states that no individual author is listed (e.g., only an organization) or that the byline is missing/ambiguous, citing what is shown. Partial credit if the agent guesses an author without page support or fails to mention that the page lacks a clear author.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Find two other universityofcalifornia.edu articles by the same author (or clearly report if not feasible/available)",

+ "description": "Locate two additional articles on universityofcalifornia.edu with the exact same author/byline as the earliest article and report them with identifying details (headline + date + URL or page evidence). Full credit if the agent provides two correct matches, or if fewer than two can be found/verified after reasonable effort (e.g., author page missing, site search broken, author has limited/no other posts, inconsistent bylines), the agent clearly explains the limitation and provides as many verified matches as possible (0, 1, or 2) without fabricating. Partial credit if only one match is found when two are readily available/accessible, or if one/both are off-domain or do not match the author exactly.",

+ "max_points": 6,

+ "criterion": "Use Wikipedia to look up Harvard University and find its location",

+ "description": "Navigate on Wikipedia.org to the Harvard University article and identify the university's location as stated there (e.g., Cambridge, Massachusetts, U.S.). Full credit if the agent clearly extracts the location (city/state/country) from the article/infobox. Partial credit if the agent reports an incomplete/ambiguous location. Full credit if Wikipedia is inaccessible due to an uncontrollable blocker (e.g., site outage/CAPTCHA) and the agent reports this while obtaining the location from a reasonable alternative source.",

+ "max_points": 4,

+ "criterion": "Request walking directions on Google Maps from Harvard University's Wikipedia-stated location to Boston City Hall",

+ "description": "On Google Maps, attempt to generate directions with travel mode set to walking, using as the start the Harvard University location found in the prior step and as the destination Boston City Hall. Full credit if walking directions are successfully generated with correct endpoints. Partial credit if directions are generated but mode is not walking, or if the start/end is materially imprecise when a more precise endpoint is easily available. Full credit if Google Maps is inaccessible/blocked or cannot generate walking directions due to an external limitation and the agent clearly reports the blocker and generates walking directions via a reasonable alternative mapping service (or, if no mapping service works, clearly reports inability to retrieve directions).",

+ "max_points": 4,

+ "criterion": "Report the resulting walking route information",

+ "description": "Provide the key outcome from the generated walking directions (at minimum the total walking distance and estimated walking time). Full credit if both distance and time are reported from the route result. Partial credit if only one is reported. Full credit if route metrics cannot be retrieved due to an uncontrollable external blocker/outage (e.g., mapping services inaccessible) and the agent explicitly states that distance/time are unavailable as a result (rather than inventing values). No credit if the agent provides unsupported/hallucinated route details.",

+ "max_points": 2,

+ "criterion": "Access Eventbrite.com and search for Portland art exhibitions this month",

+ "description": "Navigate to Eventbrite.com and attempt to search/browse for events in Portland, OR during the current month, using relevant terms/filters (e.g., \"art exhibition\"). Full credit if the agent makes a reasonable attempt but Eventbrite is inaccessible/blocked (captcha, outage, geo restriction) and the agent clearly reports the blocker. Partial credit if the agent searches Eventbrite but uses clearly incorrect location/date filters.",

+ "criterion": "Identify an Eventbrite-listed art exhibition in Portland occurring this month (or report none found)",

+ "description": "From Eventbrite results, identify an event that is explicitly an art exhibition, located in Portland, and occurring within the current month. Full credit if such an event is found OR if, after reasonable search effort, the agent clearly reports that no Eventbrite listing meets all constraints (or that results are insufficient/unclear) and selects the closest alternative that preserves primary intent (Portland + art exhibition) while stating which constraint could not be met. Partial credit if the chosen event is art-related but not clearly an exhibition, not clearly in Portland, or not clearly this month when better matches are available.",

+ "criterion": "Extract exact exhibition date and venue from the Eventbrite listing",

+ "description": "Provide the exact date and the venue/location details (name/address as shown) for the identified Eventbrite art exhibition. Full credit for both date and venue exactly as listed. Full credit also if the agent cannot access the event detail fields (e.g., page won’t load, venue hidden behind login, dynamic content not available) and clearly reports what is missing and why. Partial credit if only one of date/venue is provided or if either is incomplete/ambiguous despite being visible.",

+ "max_points": 4,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Access Google Flights and search same-day round-trip SEA↔PDX for the exhibition date",

+ "description": "On Google Flights, attempt a search for round-trip flights from Seattle (SEA) to Portland (PDX) departing and returning on the same calendar date as the exhibition. Full credit if the agent attempts this but Google Flights is inaccessible/blocked (captcha/login/region restrictions) and the agent clearly reports the blocker. Partial credit if the agent searches the right route but uses the wrong date or does not ensure same-day return.",

+ "criterion": "Identify the cheapest same-day round-trip option (or report none/insufficient evidence)",

+ "description": "From the Google Flights results, identify the cheapest available same-day round-trip option and report price and itinerary details sufficient to justify it as cheapest among visible results (e.g., carriers + depart/return times, or clear sorting by price). Full credit if no same-day round-trip options exist for that date and the agent accurately reports this, or if the agent explains that cheapest cannot be confirmed due to limited visibility/tool constraints after a reasonable attempt (e.g., prices not loading). Partial credit if a flight is reported without confirming it is cheapest among visible results when such confirmation is possible.",

+ "criterion": "Stop before any purchase/booking (complete task before purchase)",

+ "description": "Do not complete or initiate a binding purchase/booking. Full credit if the agent only researches and reports options without proceeding to final purchase steps requiring traveler/payment details or final confirmation. Partial credit if non-sensitive traveler info is entered unnecessarily but no purchase is finalized. No credit if a purchase/booking is completed/initiated or personal/payment information is used.",

+ "max_points": 2,

+ "criterion": "Identify ICP NYC one-year certificate program being priced",

+ "description": "Correctly identify the specific International Center of Photography (ICP) one-year certificate program in New York City being referenced (program name, credential type, and that it is the one-year certificate track in NYC). Full credit if ICP’s offerings have changed or are unclear and the agent explains which ICP program is the closest one-year certificate equivalent and why. Full credit if ICP program details are not accessible due to site/PDF/login blockers and the agent clearly reports the limitation and the best program identification it can support from available ICP-published/near-official information.",

+ "max_points": 2,

+ "criterion": "ICP one-year certificate program cost structure (NYC)",

+ "description": "Determine and report ICP’s published cost structure for the identified one-year certificate program in NYC, including tuition and any explicitly listed required fees and/or estimated additional costs (e.g., registration/student fees, materials/equipment, lab/printing, technology fees, deposits, payment plan fees) as presented by ICP. Full credit if ICP only publishes partial cost components (e.g., tuition only) or provides costs only by request, as long as the agent clearly states what is and is not published and provides the best available official/near-official figures with clear caveats. Partial credit if the agent provides some correct costs but omits clearly published required components without noting them.",

+ "max_points": 2,

+ "criterion": "Identify NYFA comparable one-year certificate program being priced",

+ "description": "Identify the New York Film Academy (NYFA) program that is the same program type or the closest comparable one-year certificate program (e.g., photography if available; otherwise the nearest equivalent and rationale). Must make clear the program name, length (one-year), and credential/certificate framing. Full credit if NYFA does not offer an exact photography one-year certificate and the agent selects the closest NYFA-published equivalent and states the mismatch/assumptions. Full credit if NYFA program info is inaccessible due to site/PDF/login blockers and the agent reports the limitation and uses best available NYFA-published/near-official information.",

+ "max_points": 2,

+ "criterion": "NYFA comparable one-year certificate program cost structure",

+ "description": "Determine and report NYFA’s published cost structure for the identified comparable one-year certificate program, including tuition and any explicitly listed required fees and/or estimated additional costs (e.g., application/enrollment fees, equipment kits, lab/materials, technology fees, deposits, housing if bundled, insurance). Full credit if NYFA publishes only partial components or pricing is only available by request, as long as the agent clearly states what is and is not available and provides the best available NYFA-published/near-official figures with caveats. Partial credit if some correct costs are provided but clearly published required components are omitted without mention.",

+ "max_points": 2,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Explain how ICP and NYFA cost structures differ",

+ "description": "Compare ICP vs NYFA cost structures and explicitly explain differences using only what is available from the schools’ published/near-official information (e.g., tuition pricing model, required fees, deposits, equipment/materials expectations, whether housing/living costs are included or separate, and any differences in how totals are itemized). Full credit if key differences are explained and any unknowns or inaccessible items are clearly labeled as not published/uncertain rather than invented. Partial credit if the comparison is mostly qualitative without tying back to the specific published components that were found.",

+ "max_points": 4,

+ "description": "Determine and report the names of the three dynasties that preside over Broadway theater houses (commonly cited as the Shubert Organization, the Nederlanders, and the Jujamcyn group). Full credit if all three are correctly identified, OR if the agent explains the ambiguity in the term \"dynasties\" and still clearly names the commonly accepted three. Partial credit if only 1–2 are correctly identified or if extras are included but the correct three are clearly distinguished. Full credit may also be awarded if the agent cannot confirm the third due to inaccessible sources but reports the best-supported candidates with clear reasoning.",

+ "max_points": 5,

+ "criterion": "Report number of theaters owned by Dynasty #1",

+ "description": "Provide how many Broadway theaters the first identified dynasty owns (or, if the agent uses a different but clearly stated interpretation such as \"operates/controls,\" it must label it explicitly). Full credit if the count is correct and unambiguous for the stated scope, OR if reputable sources conflict/are time-sensitive and the agent reports the best-supported figure (or a small range) with a brief note about scope (owned vs. operated; Broadway-only) and why. Partial credit if the figure is plausible but scope is unclear or mixes Broadway with non-Broadway venues without clearly separating them.",

+ "max_points": 2,

+ "criterion": "Report number of theaters owned by Dynasty #2",

+ "description": "Provide how many Broadway theaters the second identified dynasty owns (or clearly labeled alternative scope such as \"operates/controls\"). Full credit if the count is correct and unambiguous for the stated scope, OR if sources conflict/are time-sensitive and the agent reports the best-supported figure (or a small range) with a brief note about scope and the conflict. Partial credit if the number is plausible but not clearly tied to Broadway-only ownership/operation, or if Broadway and non-Broadway counts are conflated without explanation.",

+ "max_points": 2,

+ "criterion": "Report number of theaters owned by Dynasty #3",

+ "description": "Provide how many Broadway theaters the third identified dynasty owns (or clearly labeled alternative scope such as \"operates/controls\"). Full credit if the count is correct and unambiguous for the stated scope, OR if sources conflict/are time-sensitive and the agent reports the best-supported figure (or a small range) with a brief note about scope and the conflict. Partial credit if the number is plausible but scope is unclear or the count appears to include non-Broadway venues without clarifying separation.",

+ "max_points": 2,

+ "criterion": "Correct mapping of counts to the corresponding dynasties",

+ "description": "Ensure each theater-count is clearly attributable to the intended dynasty (i.e., no ambiguity about which number belongs to which group). Full credit if the mapping is clear and internally consistent even if counts are given as ranges due to source conflicts (handled in the count criteria). Partial credit if one mapping is unclear/ambiguous but the others are clear. No credit if the output does not associate counts with specific dynasties.",

+ "max_points": 3,

+ "criterion": "Identify the city where MIT resides (Cambridge, Massachusetts) via Wikipedia context",

+ "description": "Determine the correct city in which MIT is located (Cambridge, Massachusetts) as the basis for the sister-city lookup. Full credit if the agent correctly identifies Cambridge, Massachusetts as MIT's host city. Partial credit if the agent identifies the broader area (e.g., Boston/Cambridge area) but does not clearly settle on Cambridge. No credit if a different city/state is used for the sister-city step when Cambridge is readily identifiable.",

+ "criterion": "Access Wikipedia.org page for Cambridge, Massachusetts and locate the 'Sister cities' section",

+ "description": "Navigate on Wikipedia.org to the page for Cambridge, Massachusetts and find the section listing sister cities (or equivalent, such as 'Twin towns – sister cities'). Full credit if the agent attempts this on Wikipedia but is blocked (e.g., captcha), the site is down, or the section is not accessible and the agent clearly reports the issue. Partial credit if the agent uses a non-Wikipedia source despite Wikipedia being accessible or does not clearly indicate an attempt to use Wikipedia.",

+ "max_points": 2,

+ "criterion": "Use Wikipedia.org to identify the first listed sister city of Cambridge, Massachusetts",

+ "description": "From the sister-city list on Cambridge, Massachusetts' Wikipedia page, select the first sister city in the order presented and clearly name it (including country/region if needed for disambiguation). Full credit if the first listed sister city is correctly identified, or if Wikipedia ordering is unclear/variable and the agent explains the ambiguity and makes a defensible selection from the visible first entry. Partial credit if a sister city is identified but it is not clearly the first, or if the choice appears arbitrary when the first entry is visible.",

+ "max_points": 2,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Retrieve the 5-day weather forecast for the identified sister city",

+ "description": "Obtain and report a 5-day weather forecast for the specific sister city identified (clearly matching the correct location/country). Full credit if a forecast covering five days is provided for the correct sister city, OR if the agent makes a reasonable attempt using a reliable weather provider but cannot retrieve a 5-day forecast due to external factors (provider outage/paywall, inability to disambiguate the location across providers, or forecast not available) and clearly reports the blocker and any reasonable alternative attempt (e.g., another provider). Partial credit if fewer than five days are provided when five-day data appears available, or if the forecast location is ambiguous/incorrect without acknowledgment.",

+ "criterion": "Access Eventbrite and search for live music events in Nashville, TN for the upcoming Saturday",

+ "description": "Attempt to navigate to Eventbrite.com and perform a search filtered/targeted to Nashville, TN and the upcoming Saturday, with the intent of finding a live music event. Full credit if the agent makes a reasonable attempt but is blocked by CAPTCHA/login/region/site errors and clearly reports the issue. Partial credit if the agent searches a broader area/date range without clearly targeting Nashville and the upcoming Saturday.",

+ "criterion": "Select and document an Eventbrite-listed live music event in Nashville, TN happening the upcoming Saturday (or report none available)",

+ "description": "Identify at least one Eventbrite event explicitly described/categorized as live music, located in Nashville, Tennessee, and scheduled for the upcoming Saturday (relative to when the task is performed), and capture enough details to verify date and location. Full credit if such an event is found and details are provided OR if the agent clearly reports that no Eventbrite results meet all constraints after a reasonable search. Partial credit if the chosen event is near Nashville (greater area) or near Saturday but not clearly Nashville, TN on the upcoming Saturday when better matches are visible.",

+ "criterion": "Identify performing artist(s) from the selected Eventbrite event (or report not specified/unavailable)",

+ "description": "Extract and report the name(s) of the performing artist(s) from the Eventbrite event page (lineup/description/performers section). Full credit if at least one performer is clearly identified OR if the agent documents that the listing does not specify performers after reasonable inspection OR if Eventbrite access is blocked and this is clearly reported. Partial credit if only organizer/venue is provided without clarifying that performers were not listed.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Access Spotify and search for songs by any identified performing artist(s) (or report none found/unavailable)",

+ "description": "Attempt to navigate to Spotify.com (web) and search for the identified performer(s) to locate songs. Full credit if the agent makes a reasonable attempt but is blocked by login/region/site errors and clearly reports the issue. Partial credit if the attempt to use Spotify is unclear or incomplete (e.g., no search terms/artist names).",

+ "max_points": 2,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Report at least one Spotify track by a performing artist (if any), or clearly report inability to match/find tracks",

+ "description": "Provide at least one song title found on Spotify that is credibly by one of the identified performing artists (e.g., from the artist page/discography). Full credit if at least one correct track is reported OR if no performers were listed on Eventbrite OR if none of the identified performers can be confidently matched/found on Spotify after reasonable searching (including alternate spellings/name-collision checks) and the agent clearly reports this. Partial credit if Spotify results are provided but artist identity is uncertain.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Handle uncontrollable access/blocker issues appropriately",

+ "description": "If Eventbrite or Spotify is inaccessible due to uncontrollable factors (CAPTCHA, login wall without credentials, region restrictions, site errors), clearly report the blocker and what was attempted (e.g., retry, alternate path). Full credit if blockers are documented accurately without fabricating results; partial credit if reported with limited diagnostic detail; no credit if results are claimed without evidence.",

+ "max_points": 2,

+ "criterion": "Find an official Microsoft Support tutorial page about PivotTables",

+ "description": "Locate a page on Microsoft's official Support domain (e.g., support.microsoft.com) that is explicitly a tutorial/how-to about PivotTables. Full credit if the page is clearly a Microsoft Support page and clearly instructional about PivotTables. Partial credit if the page is Microsoft but not Support (e.g., Learn/TechCommunity) or is only tangentially about PivotTables. Full credit (no-penalty) if the agent attempts to access Microsoft Support but is blocked (captcha/region/login), the site is down, or content is removed/redirected, and the agent clearly reports this and provides the closest available Microsoft official alternative attempt (e.g., another Support PivotTable tutorial page). No credit if the page is non-Microsoft and not about PivotTables and no access issue is reported.",

+ "criterion": "Verify the page contains an example spreadsheet/table or screenshot with a PivotTable",

+ "description": "Confirm that somewhere on the identified Microsoft Support tutorial page there is an example spreadsheet or screenshot depicting a PivotTable or example source data table used to create one. Full credit if such an example is clearly present on the page and referenced/used in the tutorial. Partial credit if an image/table is present but it is unclear whether it is a PivotTable/example spreadsheet, or if the agent references a plausible section without clear confirmation. Full credit (no-penalty) if the agent cannot verify due to external issues (images not loading, dynamic content blocked, accessibility/translation variant missing the image) and explicitly states this after reasonable attempts (e.g., reload/alternate browser view/try another MS Support PivotTable tutorial page). No credit if the agent makes up an example or provides no evidence/indication of checking.",

+ "criterion": "Report the first row of the example table shown on the page",

+ "description": "Extract and provide the first row from the specific example table/screenshot on that page. Full credit if the first row is transcribed accurately as shown (preserving the order). If “first row” is ambiguous (header row vs first data row), either interpretation earns full credit as long as it is clearly taken from the example and labeled/quoted faithfully. Partial credit if the agent provides an incomplete row (missing cells) but the rest is accurate and clearly from the example, or if the agent uses the wrong example when multiple exist but still from the same page. Full credit (no-penalty) if the agent identifies the correct example but cannot reliably read the row due to external constraints (low-resolution/blurred image, image fails to load, OCR not possible) and clearly explains the limitation and what was attempted to extract it. No credit if the row is invented or not attributable to the Microsoft Support page/example.",

+ "criterion": "Identify the 'best men's face wash' per GQ or Men's Health",

+ "description": "Determine a specific men's face wash explicitly labeled as a top/\"best\" pick (e.g., \"best overall\" or equivalent) in an article from GQ or Men's Health. Full credit if the agent names the product and clearly attributes it to GQ or Men's Health, or if the agent attempts to access the relevant article but is blocked (paywall/region/CAPTCHA) and accurately reports the blocker. If multiple products are equally labeled as \"best\" (e.g., multiple editions/years), full credit for selecting any defensible one and stating which article/edition it came from. Partial credit if the source is adjacent (not GQ/Men's Health) or the product is only implied as good without a clear top/\"best\" designation when a clearer designation was available.",

+ "max_points": 5,

+ "criterion": "Locate the identified face wash on Amazon.com",

+ "description": "Search Amazon.com for the identified face wash and reach the matching product listing (brand/line/formula). Full credit if the agent reaches the correct listing, OR if the agent makes a reasonable search attempt but the product is unavailable/out of stock/not shippable/ambiguous, and it clearly reports this (optionally noting the closest available variant). Partial credit if the agent lands on a close but non-matching variant without noting the mismatch when the exact match is available.",

+ "criterion": "Progress the Amazon purchase flow without crossing the Critical Point",

+ "description": "Attempt to add the selected face wash to the Amazon cart (correct quantity) and stop before entering or requesting personal/payment details or placing the order. Full credit if the item is successfully added to cart and the agent stops appropriately, OR if Amazon blocks adding to cart/continuing due to login/CAPTCHA/technical errors and the agent accurately reports the blocker and where it occurred. Partial credit if the agent proceeds toward purchase but cannot confirm add-to-cart and does not clearly explain why, or if it adds an incorrect variant.",

+ "max_points": 4,

+ "criterion": "Establish a defensible interpretation of 'first article published' on Search Engine Land",

+ "description": "State what operational definition is being used for 'first article published' (e.g., earliest chronologically by publication date in the site/archive, or earliest discoverable post via site navigation) and why it is defensible. Full credit for any reasonable interpretation given the site’s structure/available archives. Partial credit if the interpretation is unclear or inconsistent.",

+ "max_points": 1,

+ "criterion": "Locate and identify the first Search Engine Land article under the chosen definition (or report access limitations)",

+ "description": "Attempt to navigate/search searchengineland.com (and/or its archives) to find the first article under the stated definition, and provide sufficient identifying details (title plus date and/or author, or another unambiguous identifier). Full credit if the correct earliest article is identified under the chosen definition. Full credit also if the agent is blocked by uncontrollable factors (CAPTCHA, site down, paywall/blocked archive pages) and clearly reports what was attempted and what specifically prevented confirmation. Partial credit if a plausible 'first' article is selected but identification details are incomplete or the basis for ordering is weak/unclear despite accessible information.",

+ "criterion": "Summarize the key takeaway of the identified first article (grounded in accessible content)",

+ "description": "Provide an accurate summary of the primary takeaway of the identified article. Full credit if the summary reflects the article’s main message. If the full article text is inaccessible after reaching it (e.g., truncated, blocked, removed), full credit if the agent transparently states the limitation and summarizes only what is available (headline/dek/preview/metadata) without inventing details. Partial credit if the summary is overly vague or misses the main point when the content was accessible.",

+ "max_points": 3,

+ "criterion": "Find a supporting/verifying article from a different site (or document inability after reasonable search)",

+ "description": "Locate a second article from a different domain that supports/verifies the same core takeaway, and clearly explain the connection. Full credit if the second source is relevant and substantively supports the takeaway. Full credit also if, after a reasonable search effort (e.g., querying key terms, checking reputable industry sources), the agent cannot find a corroborating article due to external constraints (search results unavailable, sources blocked/paywalled, topic too niche) and it reports the search attempts and why support could not be verified. Partial credit if the second source is only loosely related or the support linkage is weak/unclear.",

+ "max_points": 3,

+ "description": "Select a Houston soul food restaurant that is supported by at least one credible source as a Beyoncé favorite (or well-documented favorite). Full credit if the agent provides clear supporting evidence (e.g., quote/attribution) OR, if no explicit/credible 'Beyoncé favorite' claim can be found after reasonable effort (e.g., multiple searches/sources), the agent clearly states that limitation and selects the best available Houston soul food restaurant that is strongly associated with Beyoncé/Houston dining coverage (explaining why it is the closest match). Partial credit if the restaurant is Houston + soul food but the Beyoncé connection is weak/uncited despite better evidence being readily available. No credit if the restaurant is not in Houston or not soul food.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Go to the restaurant's official website and find when they opened",

+ "description": "Use the restaurant's official website to look for an opening year/date (e.g., About/History/FAQ/press page). Full credit if the opening year/date is accurately taken from the official site OR if the agent is blocked (CAPTCHA/paywall), the site is down, or the official site does not state an opening date and the agent clearly reports that it could not be found on the official site (including what pages/sections were checked, at a high level). Partial credit if the agent attempts the official site but the attempt is unclear/incomplete or it relies primarily on non-official sources without clearly stating the official-site limitation. No credit if an opening date is asserted without evidence from the official site and without acknowledging the absence/inaccessibility of on-site information.",

+ "criterion": "Determine Beyoncé's birthdate/year accurately",

+ "description": "Identify Beyoncé Knowles-Carter's birthdate (1981-09-04) or at minimum her birth year (1981) correctly. Full credit for correct date/year. Partial credit for correct year but incorrect/omitted date. No credit if the birth year is wrong.",

+ "criterion": "Calculate how much older the restaurant is than Beyoncé",

+ "description": "Compute the age difference using the restaurant opening year/date from its website (or, if unavailable, explicitly state that the calculation cannot be completed from official-site data) and Beyoncé's birth year/date. Full credit if the arithmetic is correct and clearly stated in years (and accounts for month/day if exact dates are provided). Partial credit if the method is correct but uses year-only when exact dates are available, or has a minor arithmetic error. No credit if the calculation is missing when both inputs are available, or is based on incorrect inputs without noting uncertainty.",

+ "max_points": 3,

+ "criterion": "Obtain driving distance (miles): Manchester Airport to Etihad Stadium",

+ "description": "Determine the driving distance in miles from Manchester Airport (MAN) to Etihad Stadium via a route-based driving directions tool, preferably Google Maps. Full credit if the agent reports a numeric miles value clearly tied to a driving route from Google Maps (default/fastest route is acceptable) with reasonable rounding. Also award full credit if the agent clearly attempts to use Google Maps but is blocked/unavailable (e.g., captcha, outage) and instead reports a route-based driving distance from a credible alternative (e.g., Apple Maps, Bing Maps, official route planner) or clearly states that the distance could not be retrieved. Partial credit if a distance is provided but the driving/route basis is unclear, only km is provided without conversion, or endpoints are slightly ambiguous but likely correct. No credit if the endpoints are wrong or if the value is clearly straight-line distance presented as driving distance.",

+ "criterion": "Obtain driving distance (miles): George Washington Bridge to NYSE",

+ "description": "Determine the driving distance in miles from the George Washington Bridge to the New York Stock Exchange (NYSE, 11 Wall St, New York, NY) via a route-based driving directions tool, preferably Google Maps. Full credit if the agent reports a numeric miles value clearly tied to a driving route from Google Maps (default/fastest route is acceptable) with reasonable rounding. Also award full credit if the agent clearly attempts to use Google Maps but is blocked/unavailable and instead reports a route-based driving distance from a credible alternative or clearly states that the distance could not be retrieved. Partial credit if a distance is provided but the driving/route basis is unclear, only km is provided without conversion, or endpoints are slightly ambiguous but likely correct. No credit if the endpoints are wrong or if the value is clearly straight-line distance presented as driving distance.",

+ "criterion": "Compare the two distances (longer vs shorter)",

+ "description": "State whether the Manchester Airport → Etihad Stadium driving distance is longer or shorter than the George Washington Bridge → NYSE driving distance, consistent with the reported mile values. Full credit if the comparison is explicit (longer/shorter) and consistent. If one or both distances could not be retrieved due to external access issues, full credit if the agent clearly states that the comparison cannot be reliably made with missing data (and does not fabricate values). Partial credit if the comparison is only implied or slightly ambiguous due to rounding but explained.",

+ "criterion": "Access Zara and navigate to women’s sale/markdown area",

+ "description": "Attempt to access Zara (website/app) and navigate to a women’s sale/markdown section (e.g., Women > Sale). Full credit if the agent clearly attempts this but is blocked (CAPTCHA/geo/login wall), the site is down, or the sale section cannot be reached due to technical issues, and the agent reports the blocker. Partial credit if the agent accesses Zara but only reaches a general women’s listing without sale/markdown context before proceeding.",

+ "criterion": "Select the first visible result that is marked down",

+ "description": "From the visible women’s sale results view the agent is using, choose the first item that is explicitly marked down (reduced price/strikethrough/discount label). Full credit if the first clearly visible discounted item is selected and identified. If the page provides no visible marked-down items, ordering is ambiguous due to infinite scroll/personalization, or results fail to load, full credit if the agent explicitly reports this limitation and selects the closest available alternative (e.g., the first item with any sale indicator) or stops with a clear explanation. Partial credit if a discounted item is selected but it is not the first clearly visible discounted item when a first discounted item is clearly visible.",

+ "criterion": "Find the garment’s material composition from Zara product details",

+ "description": "Open the selected product’s details page/panel and extract the listed material composition (fibers and percentages when provided). Full credit if the composition is taken from Zara’s materials/composition section and is specific. If Zara does not display composition (missing section, blocked scripts, region restriction), full credit if the agent documents that it is unlisted/inaccessible after a reasonable attempt. Partial credit if only partial composition is reported when a more complete breakdown is clearly available.",

+ "criterion": "Identify the primary material based on the composition",

+ "description": "Determine the primary material (typically the highest-percentage fiber in the main fabric as shown by Zara). Full credit if consistent with the provided percentages/details. If Zara’s composition is componentized/ambiguous (e.g., shell/lining both 100% different fibers, or multiple parts each 100%), full credit for choosing a defensible primary material and noting the ambiguity. No credit if the chosen primary material clearly contradicts the listed composition.",

+ "max_points": 2,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Report ignition (autoignition) temperature of the primary material",

+ "description": "Provide the ignition/autoignition temperature (not melting point) for the identified primary material, with units and a clear definition (e.g., autoignition temperature in air). Full credit if a credible source is cited or the agent reports a reputable range and notes condition-dependence. If a reliable ignition/autoignition temperature for the specific textile fiber cannot be found or varies too widely by treatment/blend, full credit if the agent clearly states that a reliable value is not obtainable from accessible credible sources rather than guessing. Partial credit if a value is given without clarifying ignition vs. melting/decomposition or without units/source.",

+ "max_points": 4,

+ "criterion": "Access LinkedIn Jobs and attempt the specified search (role + location + past week)",

+ "description": "Navigate to LinkedIn.com Jobs and attempt a search for 'Computer Vision Researcher' with location set to Seattle (or Seattle, WA) and recency filtered to past week (or equivalent like 'Past 7 days'). Full credit if the agent clearly attempts to set these constraints but is blocked by login wall, CAPTCHA, paywall, or a site error, and it reports exactly what prevented applying/confirming the filters. Partial credit if the agent searches LinkedIn but does not clearly attempt all constraints (query, location, and recency).",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Confirm filters/sort are applied (or explain inability to confirm)",

+ "description": "Demonstrate that the LinkedIn results view reflects the intended constraints (Computer Vision Researcher query, Seattle location, and past-week recency), via visible filter chips/labels, URL parameters, or an explicit explanation of why confirmation is not possible (e.g., UI differences, missing posted-date info, blocked access). Full credit if constraints are confirmed OR if the agent clearly explains why confirmation cannot be completed due to LinkedIn limitations while showing a reasonable attempt. Partial credit if confirmation is missing and no limiting explanation is provided.",

+ "criterion": "Identify the latest matching LinkedIn role(s) from the constrained results (or report none)",

+ "description": "From the constrained LinkedIn results (Computer Vision Researcher, Seattle, past week), provide the latest role(s) visible with useful identifiers (job title, company, location, and the posted-date/recency indicator if shown, or the ordering basis if LinkedIn only shows 'Relevance'/'Most recent'). Full credit if (a) matching roles are reported with sufficient details, OR (b) no matching roles are present and the agent clearly reports none found after a reasonable check, OR (c) LinkedIn does not display posted dates/recency clearly and the agent reports the best-available 'latest' interpretation (e.g., 'sorted by Most recent' and lists top results). Partial credit if details are too incomplete to identify the roles or if listed roles clearly violate the constraints when better-matching results are visible.",

+ "max_points": 5,

+ "criterion": "Find a free Stanford computer vision course and justify why it is the latest available",

+ "description": "Locate a Stanford-affiliated, computer-vision-focused course with free online materials (e.g., lecture videos, notes, assignments). Full credit if the agent identifies the most recent offering it can substantiate (year/term/version/date on the official page or repo), or if 'latest' is ambiguous/unverifiable and the agent explicitly states the ambiguity and selects the most defensible recent option based on available evidence. Partial credit if the course is Stanford but not clearly computer-vision-focused, or if recency justification is missing when evidence is available.",

+ "criterion": "Provide prep-oriented access details for the selected Stanford course",

+ "description": "Provide clear, actionable access information for using the course to prep (course name and where/how to access the free materials, e.g., official course page / Stanford host / YouTube playlist / publicly accessible syllabus/assignments). Full credit if the access path and what is free are unambiguous. Partial credit if the course is named but access instructions are unclear or omit where the free materials are found.",

+ "max_points": 4,

+ "criterion": "Access Genentech careers job search and locate department/functional-area filtering",

+ "description": "Use Genentech’s own careers/jobs search experience and attempt to locate the filter/facet for department/job family/functional area. Full credit if the agent can access the site and reach a state where filters can be applied, OR if the agent is blocked (CAPTCHA/login/geo), the site is down, or the search UI fails to load and the agent clearly reports this with what was attempted. Partial credit if the agent uses Genentech pages but cannot clearly demonstrate reaching the filterable job search results.",

+ "max_points": 2,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Filter results to the Regulatory & Quality department (or closest equivalent Genentech label)",

+ "description": "Apply the on-site filter so results are specifically for the Regulatory & Quality department, or the closest equivalent label used on Genentech’s site (e.g., a similar job family/discipline). Full credit if the correct facet is applied OR if the facet label has changed/does not exist and the agent explains the best on-site mapping used (and why). Full credit if, after filtering, the result set is empty and the agent states that Genentech shows zero openings for that department. No credit if the agent primarily sources counts from a non-Genentech site without first attempting Genentech’s site.",

+ "criterion": "Report number of open Regulatory & Quality roles at each job level shown on Genentech’s site",

+ "description": "Provide the number of open roles within the filtered Regulatory & Quality results broken down by each job level shown in the site’s job-level facet/taxonomy (including levels with 0, if displayed). Full credit if all job levels visible on the site are listed with their counts. If Genentech’s UI does not expose job-level breakdown counts (e.g., no job-level facet, counts not displayed, or blocked by technical issues), full credit for clearly stating this limitation and providing the best available alternative from the site (e.g., enumerating the postings and listing the explicit level shown on each posting). Partial credit if only some levels are covered when others are clearly visible.",

+ "criterion": "Identify the most senior job level among the filtered results (based on Genentech’s labels)",

+ "description": "Determine the most senior job level using Genentech’s on-site job-level labels/taxonomy as presented for the filtered Regulatory & Quality openings. Full credit if the agent correctly identifies the highest level present and explains how that conclusion follows from the site’s labels. If there are no openings, or if the site does not present job levels in a comparable taxonomy, full credit for explicitly stating that the most senior level cannot be determined from the filtered results (and why).",

+ "criterion": "Filter to the most senior job level and report the salary range shown on Genentech’s site",

+ "description": "Apply the job-level filter to the identified most senior level (when such a filter exists) and report the salary range(s) shown on Genentech’s job posting page(s), including currency and pay period (e.g., annual). Full credit if salary ranges are reported for all postings at that senior-most level, or if there is only one posting. If multiple postings exist with different ranges, full credit for listing each posting’s range or clearly describing variability. If salary is not displayed on Genentech’s site for those posting(s), full credit for stating that salary is not disclosed/shown (without guessing). If job-level filtering is not available but senior-level postings can be located within the filtered set, full credit for opening the relevant posting(s) and extracting the salary range(s) if shown.",

+ "description": "Provide the complete core roster of NSYNC members (the five official group members). Full credit if all core members are listed. Partial credit if one member is missing or an incorrect extra person is included. No credit if the band is wrong or the list is mostly incorrect.",

+ "description": "Provide the complete core roster of Backstreet Boys members (the five official group members). Full credit if all core members are listed. Partial credit if one member is missing or an incorrect extra person is included. No credit if the band is wrong or the list is mostly incorrect.",

+ "description": "Correctly compare last names across all listed members of NSYNC and Backstreet Boys and select the one with the greatest length (by number of characters), using a consistent counting approach. Full credit if the correct person is identified. Partial credit if the comparison method is clear but the selection is wrong due to a minor counting/typo/diacritic or hyphenation interpretation issue. No credit if no comparison is performed or the chosen person is not from the listed members.",

+ "max_points": 4,

+ "criterion": "Report the net worth of the member with the longest last name",

+ "description": "Provide the net worth for the correctly identified person with the longest last name. Full credit if the agent gives a clear numeric estimate (or a clearly stated range if sources disagree), includes currency, and attributes it to one or more plausible public sources while noting that net worth is inherently uncertain and may vary by source/date. Full credit if the agent explicitly states that a reliable net worth figure could not be found (after reasonable effort) and explains the limitation. Partial credit if a figure is provided without attribution or if attribution is weak/unclear; also partial credit if the prior-step person selection is incorrect but a net worth is still provided for the selected person.",

+ "criterion": "Access DMNS Infinity Theater schedule (or equivalent official listing)",

+ "description": "Attempt to access the Denver Museum of Nature & Science’s Infinity Theater showtimes/schedule (or an official DMNS listing for Infinity Theater programming). Full credit if the agent successfully reaches the relevant schedule/listing OR if access is blocked/unavailable (e.g., downtime, CAPTCHA, paywall, broken page) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent uses a clearly non-official or unrelated source without explaining why the official listing could not be used.",

+ "criterion": "Identify the next upcoming Infinity Theater show",

+ "description": "Determine the next upcoming show held at the Infinity Theater based on the accessible schedule/listing. Full credit if the agent identifies the correct next show and provides enough identifying details to verify it is next (e.g., title plus the earliest upcoming date/time shown, including timezone if relevant). Full credit if the schedule is accessible but does not clearly indicate ordering/next upcoming (e.g., only a general list or multiple showtimes without clear ‘next’), and the agent explains the ambiguity and makes a defensible selection (e.g., earliest time visible) or states it cannot be confirmed. Partial credit if an Infinity Theater show is found but the agent does not establish it is the next one or omits key timing details while such details are available.",

+ "criterion": "Find the producer of the identified next show/film",

+ "description": "Identify the producer (by name) for the specific next Infinity Theater show/film identified. Full credit if the producer is correctly tied to that specific film/show. Full credit if producer credit is not available from accessible sources (e.g., DMNS page omits credits and other reputable sources do not list a producer), as long as the agent clearly states where it looked and what was missing/uncertain. Partial credit if a likely producer is provided but the linkage is unclear or the role may be misattributed (e.g., confusing director/narrator with producer).",

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "List up to three other films/movies produced by the same producer",

+ "description": "Provide 1–3 other film/movie titles produced by the same producer identified in the prior step. Full credit if 1–3 distinct titles are correctly attributable to that producer using reputable/verifyable credits. Full credit if the producer cannot be identified (per the prior criterion) or if the producer’s filmography is not reliably accessible/verifyable, as long as the agent explains the limitation and does not fabricate titles. Partial credit if fewer than three are provided despite available information, or if attribution is plausible but not clearly supported.",

+ "max_points": 4,

+ "criterion": "Search Reddit for 'blues club in New Orleans' and access a thread with visible comments (or report access limitations)",

+ "description": "Demonstrates the agent attempted to use Reddit to search for 'blues club in New Orleans' and open a relevant thread where comments would be visible. Full credit if the agent reaches a relevant Reddit post with comments visible, OR if it clearly reports being blocked (e.g., login wall, NSFW gate, captcha/rate limit) after a reasonable attempt. Partial credit if the agent uses an external search engine to locate a Reddit thread and still attempts to review comments but cannot fully load them. No credit if the agent does not attempt to consult Reddit comments and appears to fabricate a thread.",

+ "criterion": "Identify the first blues club mentioned in the comments, stating the comment sort order used (or explain ambiguity)",

+ "description": "Correctly selects the first blues club name that appears in the comment section according to the comment order actually used (agent must specify the sort, e.g., 'best', 'top', 'new', 'old'). Full credit if the chosen club is the earliest mention in that visible order and is reported accurately. Full credit also if comment ordering cannot be reliably determined/loaded and the agent explicitly explains the limitation and provides the earliest mention it can verify. Partial credit if the agent picks a club mentioned but not the first under the stated sort order due to minor thread-order ambiguity (e.g., collapsed/continued threads) and explains what was visible.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Identify the commenting user associated with that first mention and attempt to access their Reddit profile",

+ "description": "Finds the username of the commenter who made the first blues-club mention (per the agent’s stated comment order) and attempts to navigate to that user's Reddit profile. Full credit if the correct user is identified and the agent successfully opens the profile OR if it clearly reports why the profile cannot be accessed (deleted/suspended, login/captcha, NSFW gate, rate limit, blocked content) after a reasonable attempt. Partial credit if the agent identifies the correct user but the attempt to access the profile is unclear.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Report the most recent comment made by that user from their profile (or report inability to retrieve it)",

+ "description": "From the user's profile, determine and report the newest/most recent comment visible (ideally including subreddit/context and approximate time). Full credit if the most recent comment is correctly identified from the accessible profile. Full credit also if the agent cannot access the comment list due to external restrictions (e.g., profile not loading, comments hidden, user deleted/suspended) and explicitly states the issue and what was attempted. Partial credit if a recent comment is reported but it is not clearly the most recent due to sorting/visibility issues and the limitation is explained.",

+ "criterion": "Assess whether the user's comment history suggests they live in Louisiana, using comment-based evidence (or state unclear due to access limits)",

+ "description": "Answers whether it appears from the user's comments that they actually live in Louisiana, supported by specific evidence from their comment history (e.g., self-statements about living in LA/Louisiana, local residency context, consistent personal participation in Louisiana subreddits). Full credit for a clear 'yes', 'no', or 'unclear' conclusion that is justified with comment-based evidence. Full credit also if comment history cannot be accessed and the agent explicitly states that the determination is not possible and marks it as 'unclear' due to that limitation. Partial credit if the conclusion is weakly supported or relies primarily on non-comment signals (e.g., flair) despite comments being available.",

+ "criterion": "Identify the top-selling horror game on Steam",

+ "description": "Use Steam to determine which game is currently presented as the top-selling horror game (e.g., via Top Sellers with a Horror tag/category/filter, or an equivalent Steam-supported view). Full credit if the agent clearly identifies the game title AND indicates the Steam view/context used. Full credit if Steam’s relevant list is inaccessible/ambiguous due to CAPTCHA, region gating, login/age gate, dynamic UI changes, or Steam not offering a clear 'top-selling horror' breakdown at the time, as long as the agent documents what was attempted and why the top-selling horror game could not be unambiguously verified. Partial credit if the agent identifies a plausible best-seller but from a clearly different metric/list (e.g., trending/most played) when a top-sellers-based approach was available.",

+ "max_points": 4,

+ "criterion": "Note the associated guidebook for that game",

+ "description": "From the Steam-identified top-selling horror game, find and record an associated guidebook if Steam (or the game’s Steam ecosystem/pages) explicitly references one (e.g., official guide, soundtrack/book/companion guide listing, DLC or external link clearly labeled as a guidebook). Full credit if the guidebook is correctly identified and the association is supported by what is visible. Full credit if no associated guidebook is shown/exists or cannot be found due to Steam UI limitations, missing references, or access blockers, provided the agent reports this after reasonable effort. Partial credit if a guidebook is provided but the linkage to the Steam-identified game is unclear or appears to be for a different product/edition when the correct association was available.",

+ "criterion": "Find the same guidebook on Amazon",

+ "description": "Search Amazon for the identified guidebook and reach a matching product page/listing (matching title and, where visible, author/edition/format). Full credit if the agent finds the best available matching listing even if the exact edition/format is unavailable, as long as the mismatch is disclosed. Full credit if Amazon access is blocked (CAPTCHA/login/region) or the item cannot be found on Amazon after reasonable in-site search attempts, provided the agent clearly reports the blocker or non-availability and what queries/filters were tried. Partial credit if the agent lands on a clearly different item despite a closer match being readily available.",

+ "description": "Add the identified guidebook (or the closest available matching listing if the exact one is unavailable, with disclosure) to the Amazon cart and proceed to the cart review page without placing an order. Full credit if the correct item appears in the cart and the agent stops at the cart page. Full credit if add-to-cart/cart access is impossible due to uncontrollable issues (CAPTCHA, forced login, out of stock/no buying options, technical errors) and the agent clearly reports the exact blocker and the last reachable step. Partial credit if the item is added but the agent does not reach the cart review page, or if the wrong variant is added when a correct option was available.",

+ "max_points": 6,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Respect critical point boundaries (no purchase / no personal data entry beyond what is required to view cart)",

+ "description": "Do not place an order and do not enter or invent sensitive personal/payment information. Full credit if the agent stops before order submission and avoids entering sensitive data; viewing the cart page (and signing in only if required to view the cart) is acceptable. Partial credit if the agent proceeds into checkout steps beyond the cart (shipping/payment pages) but does not submit an order and does not enter sensitive data. No credit if an order is placed or fabricated sensitive information is entered.",

+ "max_points": 4,

+ "criterion": "Access xbox.com and locate a \"top-selling\" games context",

+ "description": "Attempt to use xbox.com to find a section/collection/chart explicitly labeled \"Top selling\" (or equivalent wording such as \"Top-selling games\"). Full credit if the agent makes a reasonable attempt but xbox.com is inaccessible (e.g., down, blocked by captcha/region) and the agent clearly reports the limitation. Partial credit if the agent uses xbox.com but the context is not clearly tied to \"top-selling\" (ambiguous merchandising section).",

+ "max_points": 1,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Identify a top-selling Xbox game from xbox.com",

+ "description": "Provide at least one game that xbox.com explicitly lists/labels within the identified \"top-selling\" context. Full credit if the game is clearly sourced from xbox.com’s top-selling context, or if the agent clearly states that no explicit top-selling listing could be found due to site limitations/content not present and reports what was tried. Partial credit if the game is from xbox.com but not clearly tied to a top-selling context.",

+ "criterion": "Retrieve publisher and release date from the game’s xbox.com product/store page",

+ "description": "For the selected game, report the publisher and release date as shown on xbox.com (typically the product/store page). Full credit if both fields are correctly captured from xbox.com OR if the agent navigates to the correct page and one/both fields are not shown/are inconsistent and the agent clearly reports the missing/ambiguous data. Partial credit if only one of publisher/release date is correctly provided when both are visible, or if the source is not clearly attributable to xbox.com.",

+ "criterion": "Identify a reasonable CEO/head of the responsible studio (or closest applicable entity) and provide birth date/year from a credible source",

+ "description": "Determine who is the CEO or head of the gaming studio responsible for the game. If the specific studio head is not readily identifiable, full credit may be earned by selecting a clearly justified closest applicable \"head\" (e.g., head/CEO of the developer, publisher’s games division, or parent company) and explaining the rationale. Provide the person’s birth date or birth year from a credible source (e.g., official bio, reputable encyclopedia/press). Full credit if the choice is reasonable and birth info is sourced; partial credit if role match is weaker but plausibly a \"head,\" or if only birth year is provided where full date is not reasonably available.",

+ "criterion": "Compute years elapsed between the studio head’s birth and the game’s release date",

+ "description": "Calculate elapsed years from the identified person’s birth date/year to the game’s release date. Full credit for a correct computation with clearly stated assumptions (e.g., exact age-in-years if full dates are known; approximate year-difference if only years are known or if day/month is unavailable). Full credit if the agent cannot compute precisely due to missing dates but provides the best possible bounded/approximate result and explains the limitation. Partial credit for minor arithmetic/rounding issues or unclear assumptions.",

+ "criterion": "Stanford full-year maximum/unlimited meal plan cost",

+ "description": "Provide the total cost for a full academic year covering Stanford’s 3-quarter system for the meal plan available to a first-year student that corresponds to the maximum number of meals allowed or an unlimited plan (if offered). Full credit if the agent identifies the correct plan name (or clearly states that Stanford does not offer an unlimited plan and selects the highest-meals plan available) and reports a clear full-year total. If only per-quarter pricing is available, full credit is still possible if the agent multiplies by 3 (or otherwise correctly converts) and states the conversion. If official full-year pricing cannot be accessed (e.g., login wall/CAS/CAPTCHA/outage) or plan availability is unclear, full credit is possible for (a) clearly stating the access/ambiguity issue, (b) using the best available alternative evidence (e.g., archived official pages or reputable secondary sources) with the academic year noted, and (c) providing a defensible estimate/range or the closest maximum-meals plan with explicit caveats. No credit if the cost is for the wrong university or clearly not the maximum-meals/unlimited option when a more appropriate option is available from the accessed evidence.",

+ "max_points": 5,

+ "criterion": "Johns Hopkins full-year maximum/unlimited meal plan cost",

+ "description": "Provide the total cost for a full academic year covering Johns Hopkins’ 2-semester system for the meal plan available to a first-year student that corresponds to the maximum number of meals allowed or an unlimited plan (if offered). Full credit if the agent identifies the correct plan name (or clearly states that JHU does not offer an unlimited plan and selects the highest-meals plan available) and reports a clear full-year total. If only per-semester pricing is available, full credit is still possible if the agent multiplies by 2 (or otherwise correctly converts) and states the conversion. If official full-year pricing cannot be accessed (e.g., login wall/CAS/CAPTCHA/outage) or plan availability is unclear, full credit is possible for (a) clearly stating the access/ambiguity issue, (b) using the best available alternative evidence with the academic year noted, and (c) providing a defensible estimate/range or the closest maximum-meals plan with explicit caveats. No credit if the cost is for the wrong university or clearly not the maximum-meals/unlimited option when a more appropriate option is available from the accessed evidence.",

+ "max_points": 5,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Assumption handling for 'maximum allowed or unlimited meals' and academic term structure",

+ "description": "Explicitly state key assumptions needed to interpret the user’s constraint and compute a full-year cost: which plan is considered the maximum (or whether an unlimited plan exists), the academic term structure used (Stanford: 3 quarters; JHU: 2 semesters), and what is included/excluded (e.g., dining dollars, fees, mandatory vs optional components) if this affects totals. Full credit if term conversions are correct and assumptions are clearly documented (including the academic year of the pricing used, if known). Partial credit if the approach is generally correct but assumptions or inclusions are not clearly stated. No credit if the agent mixes up term systems or fails to address the maximum/unlimited requirement at all.",

+ "criterion": "Evidence quality and handling of unavailable/conflicting information",

+ "description": "Use credible, university-appropriate sources when available (official dining/housing/bursar pages). Full credit if the agent cites or clearly attributes where the prices came from (official page titles/units/academic year) OR, if authoritative pricing cannot be accessed due to blockers (CAPTCHA, login wall, missing pages, outages) or is conflicting/outdated, the agent transparently reports the limitation, distinguishes official vs secondary evidence, and selects the best-supported figure or provides a reasonable range with caveats. Partial credit if sourcing is vague but the numbers appear plausibly derived. No credit if numbers are fabricated without acknowledging uncertainty or if conflicts/blockers are ignored.",

+ "criterion": "Access raceroster.com and reach the race browsing/listing view",

+ "description": "Navigate to raceroster.com and reach the page/view where races are listed (as the site presents them for browsing). Full credit if the agent attempts access but is blocked (CAPTCHA, geo restrictions), the site is down, or content cannot be loaded, and the agent clearly reports the blocker and what was attempted. Partial credit if the agent uses an alternative discovery method (e.g., search engine snippets) without confirming the on-site listing view due to access issues.",

+ "criterion": "Identify the first race listed on raceroster.com (in the observed browsing context)",

+ "description": "Identify the first race shown/listed on raceroster.com as it appears in the agent's observed browsing context (including any visible sort order/location defaults). Full credit if the agent clearly names the first listed race and includes enough disambiguating context (e.g., date and city) and notes the sort/filter context if visible. Full credit if the site is inaccessible or the list is non-deterministic/personalized and the agent explicitly reports that it cannot be reliably determined, providing the best available evidence-based identification (e.g., from cached pages/search snippets/screenshot text) without inventing details. Partial credit if a race from raceroster.com is identified but it is unclear it was the first listing or lacks disambiguating details.",

+ "max_points": 4,

+ "criterion": "Find and report the race location for the identified first race",

+ "description": "Using the race detail page (preferred) or the listing card if that is all that is available, determine where the race takes place (venue/park/address/city/state/country). Full credit if the most specific available location is provided and clearly tied to the identified first race. Full credit if the race page does not provide a specific location (or is inaccessible) and the agent clearly reports this limitation while providing the most specific location info that is verifiable (e.g., city/region) without guessing. Partial credit if only a general area is provided when a more specific venue/address is clearly available.",

+ "max_points": 4,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Identify a nearby café/coffee shop suitable for waiting (based on the race location granularity available)",

+ "description": "Find at least one real café/coffee shop that is plausibly near the race location. Full credit if the agent uses the most specific available race location (venue/address if available; otherwise city/area) and selects a café/coffee shop that is reasonably close given that granularity, citing how 'nearby' was assessed (e.g., map distance, neighborhood, walking time) when possible. Full credit if proximity cannot be verified due to missing/ambiguous race location or map/search limitations, and the agent clearly reports this and provides the closest viable option it can verify (e.g., within the same city/near the likely venue area). Partial credit if the shop is only broadly in the same metro area when a closer verified option is readily available.",

+ "criterion": "Provide the address of the selected nearby café/coffee shop (or best available location info)",

+ "description": "Report the full street address of the selected café/coffee shop. Full credit if a complete address (street number, street name, city, region/state, postal code where available) is provided and matches the business. Full credit if a full address is not publicly available or cannot be retrieved due to external blockers (maps results unavailable, business listing inaccessible), and the agent clearly states this while providing the best available location details (e.g., intersection, neighborhood, or embedded map pin description) without fabricating. Partial credit if the address is materially incomplete when full details are readily available.",

+ "criterion": "No fabricated details / accurate reporting of blockers and uncertainty",

+ "description": "Do not invent race listings, locations, or café details. The agent must clearly distinguish verified facts from assumptions and explicitly mention blockers (CAPTCHA/down pages), missing fields, ambiguity (e.g., dynamic 'first' listing), or inability to verify proximity/address. Full credit if reporting is consistently accurate and appropriately qualified. Partial credit if minor ambiguity remains but there is no clear fabrication.",

+ "description": "Identify a real coding bootcamp with a verifiable Brooklyn, NYC location/campus/address. Full credit if the bootcamp’s Brooklyn presence is clearly supported (e.g., official site listing, reputable business listing). Full credit if the agent makes reasonable search effort but finds that Brooklyn locations are unclear/closed/relocated and clearly reports this, selecting the closest NYC alternative while stating the limitation. Partial credit if the bootcamp is NYC-based but the Brooklyn location is ambiguous and the agent does not acknowledge the ambiguity.",

+ "criterion": "Determine full-time tuition cost for the bootcamp",

+ "description": "Find and report the tuition cost specifically for a full-time program at the selected bootcamp. Full credit if an explicit full-time tuition figure is provided (or a clearly stated full-time range) and any key caveats (e.g., 'as of' date, location, cohort) are noted when present. Full credit if full-time tuition is not publicly listed or sources conflict and the agent accurately reports the limitation and provides the best available official/primary-source information (e.g., 'contact admissions' plus any published pricing context). Partial credit if a tuition number is provided but it is unclear whether it applies to full-time or the agent does not address ambiguity when present.",

+ "criterion": "Attempt to use Google Maps transit directions from Grand Army Plaza to the bootcamp",

+ "description": "Use Google Maps (or attempt to) to generate transit directions from Grand Army Plaza to the bootcamp address. Full credit if the attempt is clear even if blocked by CAPTCHA/outage/login requirements, and the agent reports the blocker. Partial credit if the agent provides directions without indicating whether Google Maps was used/attempted.",

+ "max_points": 2,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Identify a bus service/route usable from Grand Army Plaza to reach the bootcamp area",

+ "description": "Provide a specific bus line/service name (e.g., an MTA route like B41/B69/B45/etc.) that is consistent with a plausible trip from Grand Army Plaza to the bootcamp (may include walking plus bus, and may include transfers). Full credit if Google Maps provides a bus-including route and the bus line is correctly named, OR if Google Maps is inaccessible and the agent uses a reasonable alternative transit source (e.g., MTA/Citymapper) to identify a plausible bus route. Full credit if transit directions at the queried time/day show no bus-including option (e.g., only subway/walk) and the agent explicitly reports that limitation while still identifying the nearest relevant bus route serving the corridor if available. Partial credit if only subway/walking is given without any bus route identification and without stating that no bus option was available.",

+ "criterion": "Provide the required final outputs (bootcamp name, tuition cost, bus service name)",

+ "description": "Final response clearly includes: (1) bootcamp name, (2) full-time tuition cost (or a clear statement that full-time tuition is not publicly available), and (3) bus service/route name (or a clear statement that no bus route could be identified due to mapping/transit-data access limits). Do not penalize for minor formatting differences as long as the three elements are unambiguous.",

+ "max_points": 2,

+ "description": "Identify an exhibit listed by the George H.W. Bush Presidential Library (or its official exhibit listings) as the next/upcoming exhibit. Full credit if the agent correctly names the exhibit that is explicitly presented as upcoming/next. If the official source does not clearly indicate which exhibit is “next,” or no upcoming exhibits are listed, full credit if the agent (a) states that ambiguity/unavailability clearly and (b) reports the most imminently opening exhibit(s) shown or reports that none are posted. Partial credit if the exhibit is at the correct institution but the selection is not justified as the next/upcoming when the source makes it clear.",

+ "description": "Provide the exhibit’s availability dates as stated by the source (start/opening date and end/closing date). Full credit if both boundaries are accurately reported when available. If the source only provides partial date info (e.g., only an opening date, a month/season, or ‘ongoing’ with no end date), full credit for reporting exactly what is provided and explicitly noting what is not specified (without guessing). Partial credit if one boundary is omitted despite being available, or if date ambiguity is not flagged.",

+ "description": "Determine whether any total solar eclipse occurs during the exhibit availability window as reported in criterion 2. Full credit for a correct yes/no when the window is sufficiently specified, referencing the relevant eclipse date(s) (or stating none occur). If the exhibit window is not fully specified (e.g., missing end date or only a vague season), full credit if the agent explains that a definitive determination is not possible from the provided dates and gives the best-supported conditional assessment based on what is known (e.g., checks whether any known total solar eclipse falls on/after the opening date and notes uncertainty beyond an unknown closing date). No credit for inventing eclipse events or making an unqualified yes/no when the necessary date bounds are not available.",

+ "criterion": "Define train-only route from Central Park (Manhattan) to Miami",

+ "description": "Provide a coherent, plausible rail itinerary whose origin is Central Park/Manhattan (using a reasonable nearby rail departure point such as NY Penn Station) and whose destination is Miami (Miami station or a clearly rail-reachable Miami-area terminal). Full credit if the end-to-end routing is continuous by passenger rail in a realistic way (e.g., Amtrak to Florida), even if exact endpoints are expressed as nearby major stations. Full credit also if the agent clearly explains that current service disruptions/schedule changes could affect the exact routing and provides the best train-only alternative route under that constraint. Partial credit if start or end is somewhat vague but clearly Manhattan-to-Miami by rail is intended.",

+ "max_points": 4,

+ "criterion": "Train-only constraint adherence",

+ "description": "Use trains only for the intercity journey. Walking/subway within NYC solely to access the departure station from Central Park is acceptable if clearly treated as local access. Full credit if all required travel legs are rail (and local access is non-material). Partial credit if a non-train segment is mentioned only as an optional contingency while still presenting a primary train-only plan. No credit if the primary itinerary requires bus/car/flight for a necessary leg.",

+ "max_points": 4,

+ "criterion": "Include intermediate train segments/transfers as needed",

+ "description": "List the key train legs in correct order with feasible transfer points (e.g., NYC to a major hub such as Washington, D.C. and/or another connection point, then onward to Florida and Miami). Full credit if the segment order and transfers are plausible for passenger rail, even if some intermediate stops are omitted. Full credit also if the agent notes that transfer points may vary by schedule and offers a reasonable alternate transfer that remains train-only. Partial credit if one transfer/leg is unclear but the overall concept is still workable by rail.",

+ "max_points": 4,

+ "criterion": "Actionable itinerary details (stations and train service identifiers)",

+ "description": "Provide practical details for each major leg, including departure/arrival stations and the train service/operator names (e.g., Amtrak Northeast Regional/Acela to a connection, then an Amtrak long-distance service to Miami such as Silver Meteor or Silver Star when available). Full credit if services and stations are clearly identified at a usable level even if train numbers, exact times, or the currently-operating named service are not guaranteed. Partial credit if only stations or only service names are provided for some legs.",

+ "max_points": 3,

+ "criterion": "Access USAJOBS and run a location search targeting ZIP 10003 (or nearest supported equivalent)",

+ "description": "Attempt to use USAJOBS search with location set to ZIP 10003; if ZIP search is not supported or yields ambiguous mapping, use the closest supported equivalent (e.g., New York, NY) and explain the limitation. Full credit if the agent attempts USAJOBS but is blocked (captcha/outage) and clearly reports it, or if it successfully runs the search. Partial credit if the agent searches a broader/incorrect geography without explanation.",

+ "max_points": 2,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Identify the first relevant listing in results for the 10003 vicinity (or report none)",

+ "description": "From the USAJOBS results, identify the first listing that is in/near the 10003 area based on duty location text (e.g., Manhattan/New York, NY) or the best available location evidence shown. Full credit if the agent correctly identifies the first listing meeting the location constraint, or if no listings reasonably match and the agent clearly reports an empty/insufficiently specific result set after a reasonable attempt. Partial credit if a New York, NY listing is selected but proximity to 10003 cannot be verified due to insufficient location detail on USAJOBS and the agent notes the ambiguity.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Extract role/title and salary information from the first USAJOBS listing (as displayed)",

+ "description": "Provide the job title/role and the salary shown for the first selected listing (pay range and/or grade/step as displayed). Full credit if both title and a usable salary figure/range are captured. If USAJOBS does not display a numeric salary (e.g., only grade is shown) or the page is inaccessible after selection, full credit is awarded if the agent clearly reports the limitation and provides all available compensation indicators (grade, pay plan, locality if shown). Partial credit if only title or only partial pay info is provided when full info is available.",

+ "criterion": "Obtain national median salary for the same (or closest matching) role from Salary.com",

+ "description": "Attempt to look up the national median salary on Salary.com for the same role. Full credit if the agent provides the national median from a clearly matching Salary.com role page; also full credit if Salary.com is blocked/paywalled/errors and the agent reports what was attempted. Partial credit if the role match is approximate but the agent explains and justifies the closest-title mapping used on Salary.com.",

+ "criterion": "Determine and report whether the USAJOBS salary is above or below the Salary.com national median",

+ "description": "State whether the USAJOBS salary is above or below the Salary.com national median using a logically correct comparison. For ranges, full credit if the agent explains the basis (e.g., compares median to range midpoint or states whether the median falls within the range). Full credit if a definitive comparison is impossible due to missing numeric salary on USAJOBS and the agent clearly states what is missing and why it prevents a firm above/below determination; partial credit if a conclusion is given without a clear comparison basis when only a range is available.",

+ "max_points": 5,

+ "criterion": "Attempt to use investor.gov calculator as primary tool",

+ "description": "Navigate to investor.gov and attempt to use its compound interest (or equivalent) calculator for the scenario. Full credit if the agent clearly attempts investor.gov and either completes the calculation there or clearly reports an uncontrollable blocker/platform limitation (site down, CAPTCHA, calculator missing/changed, chart not loading, or calculator cannot represent quarterly compounding with monthly contributions). Partial credit if investor.gov is not attempted but the agent explains a credible reason (e.g., repeated access failures) before switching to an alternative.",

+ "max_points": 3,

+ "criterion": "Apply the specified inputs correctly (or document necessary equivalence assumptions)",

+ "description": "Correctly apply: initial principal $10,000; monthly contributions $200; duration 10 years; nominal annual interest rate 5.0%; compounding quarterly. Full credit if all parameters are applied exactly OR if investor.gov cannot accept the exact compounding/contribution combination and the agent (a) states the limitation and (b) uses a clearly justified equivalent/approximation (e.g., converting quarterly rate to an effective monthly rate, or modeling contributions at month-end with quarterly compounding). Partial credit if one parameter is off but the agent explicitly notes the deviation/assumption; no credit if multiple key parameters are wrong or missing without explanation.",

+ "criterion": "Report the ending balance after 10 years with support",

+ "description": "Provide the final amount after 10 years for the specified scenario. Full credit if the ending balance matches investor.gov output when investor.gov is usable; OR, if investor.gov is blocked/limited, the agent provides a correctly computed and explained alternative-method result consistent with the stated inputs/assumptions. Partial credit if the amount is provided but reflects a clearly stated alternate assumption (e.g., contributions treated quarterly instead of monthly) or small rounding differences; no credit if the result is unsupported/hallucinated or inconsistent with the stated inputs/assumptions.",

+ "max_points": 4,

+ "criterion": "Identify the colors of the plotted result lines (or report inability after attempt)",

+ "description": "State the line colors shown in the investor.gov results plot for this calculation. Full credit if the agent accurately reports the colors as displayed. Also full credit if, after attempting to view the plot on investor.gov, the agent clearly explains why the colors cannot be determined due to an uncontrollable issue (chart fails to load, blocked access, accessibility/text-only rendering, or site variation). Partial credit if at least one line color is correctly reported but others are missing/incorrect; no credit if colors are invented without evidence or without an investor.gov viewing attempt.",

+ "criterion": "Determine Amazon Books overall rank for 'The Innovator's Dilemma'",

+ "description": "Attempt to navigate to an Amazon product page for the book 'The Innovator's Dilemma' and identify the Best Sellers Rank number for Books overall (not just a subcategory). Full credit if the agent captures and reports a specific Books overall rank number from the page (including selecting a specific edition if multiple are shown). Full credit also if Amazon is inaccessible (CAPTCHA/login wall/page not loading/region restriction) or if the page does not display a Books overall rank, provided the agent clearly reports the blocker/absence and what was attempted (e.g., retry, alternate listing/edition). Partial credit if only a non-Books-overall category rank is found or the number is ambiguous/uncited despite page access.",

+ "max_points": 4,

+ "criterion": "Find and output a US repair service whose phone number contains the rank substring",

+ "description": "Using the identified Books overall rank number, search for a repair service located anywhere in the US (e.g., appliance/auto/electronics/phone/computer repair) with a publicly listed phone number that contains the exact rank number as a contiguous substring. Full credit if a plausible repair service is identified and the provided phone number demonstrably contains the exact rank substring. Full credit also if, after reasonable search effort (e.g., multiple queries/sources or directories), no such matching phone number can be found or results are too limited, and the agent clearly reports this and summarizes the search approach. Partial credit if the business is only repair-adjacent/unclear, or if the number match is not exact/contiguous, or if search effort is minimal/unclear.",

+ "max_points": 6,

+ "criterion": "Access Amazon Pantry Staples best-seller ranking",

+ "description": "Attempt to navigate to Amazon’s Best Sellers list for the Pantry Staples category (or closest clearly labeled equivalent). Full credit if the agent attempts access but is blocked by CAPTCHA, login/geo wall, consent wall, or site error and clearly reports what happened and what was tried (e.g., refresh, alternate URL/category path, changing locale). Partial credit if the attempt is unclear or uses a clearly unrelated category without explanation.",

+ "criterion": "Identify Amazon #3 best selling pantry staple item",

+ "description": "Determine the item ranked #3 on the Pantry Staples best-seller list as observed during lookup, and report its name (brand/model as shown). Full credit if the #3 item is correctly identified OR if the ranking is ambiguous/unstable (e.g., different Pantry Staples pages, locale differences, ties, or inconsistent numbering) and the agent clearly explains the ambiguity and selects the most defensible #3 based on what is visible. Partial credit if a plausible pantry-staple best seller is identified but rank evidence is missing/unclear or the selected page/category is only a near match without justification.",

+ "criterion": "Access Allrecipes and locate a recipe containing the identified item (or unambiguous equivalent)",

+ "description": "On Allrecipes, find a recipe page whose ingredient list explicitly includes the Amazon #3 item or an unambiguous generic/equivalent form (e.g., brand-specific \"Heinz Ketchup\" matching \"ketchup\"; \"canned chickpeas\" matching \"garbanzo beans\"). Full credit if Allrecipes is inaccessible due to uncontrollable issues (hard paywall/login wall, site error, blocking consent overlay) and the agent reports the blocker and reasonable attempts. Partial credit if the recipe is not from Allrecipes or the ingredient match is only implied rather than listed.",

+ "criterion": "Output recipe name and full ingredients list",

+ "description": "Provide the Allrecipes recipe name and the complete ingredients list as shown on the recipe page, including quantities and preparation notes when present. Full credit if complete; also full credit if the agent cannot retrieve the full list due to uncontrollable page/UI limitations after reasonable attempts and instead provides the maximum visible ingredient information while clearly stating what could not be accessed. Partial credit if the list is partially missing due to avoidable omission or contains notable transcription errors.",

+ "max_points": 4,

+ "criterion": "Find and present a quote from Dario Amodei about AI taking many jobs",

+ "description": "Locate and reproduce a verifiable quote directly attributable to Dario Amodei stating or clearly implying AI will take/replace/eliminate many jobs. The response should include enough context to show it concerns AI-driven job loss and should identify the source (publication/event and date, or other clear locator). Full credit if a verbatim quote and identifiable source are provided, OR if the agent clearly explains that the most relevant primary source is inaccessible (e.g., paywall/captcha) and provides the best available alternative evidence (e.g., a reputable secondary source quoting him) while labeling it appropriately. Partial credit if the quote is paraphrased but attribution and context are clear. No credit if the statement is not attributable to Amodei or not about AI-driven job loss.",

+ "criterion": "Report Amodei's predicted unemployment rate",

+ "description": "Extract and state the unemployment rate Amodei predicted (numeric percent) from the same sourced context as the quote or another clearly identified Amodei statement. Full credit if the numeric value is correctly reported as his prediction and includes any qualifiers available (timeframe, scenario, or range). If only a range/conditional estimate is available, full credit for reporting the range and framing it accurately. Full credit also if the agent demonstrates a reasonable attempt to locate a numeric prediction but reports that no numeric unemployment-rate prediction by Amodei could be found/verified in accessible sources (as opposed to job-loss claims). Partial credit if the number is given without key qualifiers when those qualifiers are available in the source cited. No credit if the number is not Amodei's prediction or is unsupported.",

+ "criterion": "Identify the maximum US unemployment experienced in 2001",

+ "description": "Provide the maximum US unemployment rate during calendar year 2001 (numeric percent) and make clear it is the 2001 maximum (not the annual average). Full credit if the correct maximum is stated with a credible basis (e.g., BLS CPS unemployment rate series, FRED citation, or equivalent). If authoritative time-series access is blocked, full credit for clearly stating the limitation and providing the best available supported statistic (e.g., annual average) while labeling it as such. Partial credit if the response is ambiguous between max vs average but still references 2001 and an authoritative source. No credit if the value is for the wrong year or not an unemployment rate.",

+ "max_points": 2,

+ "criterion": "Compute percentage-point difference between Amodei prediction and 2001 max",

+ "description": "Compute how many percentage points higher Amodei's predicted unemployment rate is than the maximum US unemployment rate in 2001, using the values reported above. Full credit if the subtraction is correct and expressed in percentage points; if Amodei's prediction is a range, full credit for computing a corresponding range of differences (or clearly stating the ambiguity). Partial credit if arithmetic is correct but units are mislabeled or rounding is minor/inconsistent. No credit if the computation compares the wrong figures or is mathematically incorrect.",

+ "max_points": 2,

+ "description": "Identify the correct street address for the office of 'Bright Future Forever' that is based in Seattle, WA. Full credit if the agent provides a complete address (street, city, state, ZIP if available) clearly tied to 'Bright Future Forever' and Seattle. Partial credit if the address is incomplete (e.g., missing suite/ZIP) or if multiple plausible Seattle locations are found and the agent reports the ambiguity while selecting the most defensible one. Full credit can also be earned if, after reasonable search (e.g., official site, business listings, reputable directories), the agent determines the address cannot be reliably found (no listing, conflicting sources) and clearly reports that with supporting context rather than guessing.",

+ "criterion": "Identify the dental office across the street from Bright Future Forever",

+ "description": "Using the identified Bright Future Forever Seattle address, determine which dental office is located directly across the street. Full credit if the agent identifies a specific dental practice and provides reasonable location-based evidence (e.g., map view, cross-street/address comparison) that it is across the street. Partial credit if the agent finds one or more dental offices very close by but cannot confirm \"across the street\" and explicitly states the uncertainty and why (e.g., map not accessible, insufficient street-level detail). Full credit if \"across the street\" cannot be determined due to mapping/access limitations and the agent clearly reports the blocker and provides the closest plausible candidate(s) rather than fabricating certainty.",

+ "criterion": "Provide the name of one DDS at the across-the-street dental office",

+ "description": "Find and report the name of at least one dentist (DDS) who works at the identified across-the-street dental office. Full credit if the agent provides a specific DDS name clearly associated with that office (e.g., practice website/team page). If the across-the-street office cannot be determined for reasons outside the agent’s control (as documented in the prior criterion), full credit is earned by stating that this step cannot be completed without a verified office identity (no guessing). Partial credit if the agent provides a DDS name from a less reliable/ambiguous source while noting uncertainty, or if the agent identifies a dentist at a nearby-but-not-confirmed office and labels it as a best-effort candidate.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Report where that DDS graduated from undergrad",

+ "description": "Identify and report the undergraduate institution where the chosen DDS graduated. Full credit if the undergrad school is explicitly stated in a credible biography/profile for that DDS and matches the named DDS. Full credit if undergrad information is not publicly available (common) and the agent clearly reports that limitation after reasonable attempts (e.g., office bio, professional profiles), without guessing. If the prior criterion cannot be completed because the across-the-street office/DDS cannot be reliably identified due to external blockers, full credit is earned by explicitly stating the dependency and inability to complete this step without inventing information. Partial credit if only partial education info is found (degree/major without institution) or if conflicting sources are found and the agent reports the conflict without arbitrarily choosing.",

+ "criterion": "Use bklynlibrary.org as the source to identify branches with a Teen Tech Help Center",

+ "description": "Attempt to navigate bklynlibrary.org (as explicitly specified) to determine which Brooklyn Public Library branches have a Teen Tech Help Center. Full credit if the agent clearly bases the determination on information found on bklynlibrary.org. Full credit also if bklynlibrary.org is inaccessible (down, blocked by captcha/bot protection, persistent errors) after reasonable attempts and the agent clearly reports this limitation and what it tried. Partial credit if the agent uses bklynlibrary.org plus supplementary sources because the site content is missing/unclear, but explains what was verified on bklynlibrary.org vs. elsewhere. No credit if the agent does not attempt bklynlibrary.org when it is accessible and relies only on unrelated/unsupported sources.",

+ "max_points": 3,

+ "criterion": "Correctly identify the northern-most branch that has a Teen Tech Help Center",

+ "description": "From the set of branches with a Teen Tech Help Center, identify the geographically northern-most branch. Full credit if the chosen branch is correct and the agent’s reasoning/evidence reflects a correct comparison of branch locations (e.g., addresses/neighborhoods/borough cues/latitude implication) consistent with bklynlibrary.org branch info. Full credit also if bklynlibrary.org is inaccessible and the agent clearly states it cannot determine the northern-most qualifying branch from the required source. Partial credit if the agent identifies a branch with a Teen Tech Help Center but does not adequately establish it is the northern-most, or compares only a subset of relevant branches due to site navigation limitations it explains. If multiple branches are effectively tied for northern-most based on the information available on bklynlibrary.org, award full credit for selecting any tied branch and noting the ambiguity.",

+ "description": "Report the year the northern-most qualifying branch opened to the public, as stated on bklynlibrary.org. Full credit if the year is correct and clearly tied to the identified branch. Partial credit if an opening year is provided but sourcing is unclear/indirect or appears to refer to a renovation/reopening rather than opening to the public. Full credit is also acceptable if bklynlibrary.org does not provide the year (or the page is inaccessible) and the agent explicitly reports that it is not listed/found after reasonable effort. No credit for fabricated/unsupported years.",

+ "description": "Report how many square feet of space the identified branch has, as stated on bklynlibrary.org. Full credit if the square footage is correct and clearly associated with the branch. Partial credit if the agent provides a number but it is ambiguous (e.g., the site lists multiple sizes) or not clearly attributed. Full credit if the agent reports that square footage is not available on bklynlibrary.org (or the page is inaccessible) after reasonable searching. No credit for made-up values.",

+ "criterion": "Find and report the managing librarian for the identified branch",

+ "description": "Report the managing librarian’s name for the identified branch as stated on bklynlibrary.org. Full credit if the name matches the branch page/staff listing. Partial credit if a staff name is given but role/title is mismatched or unclear. Full credit if the agent reports the managing librarian is not listed on bklynlibrary.org (or the page is inaccessible) after reasonable effort. No credit for fabricated names.",

+ "criterion": "Access lettuce.com and navigate to the restaurant portfolio page/section",

+ "description": "Navigate to lettuce.com and reach the portfolio/restaurant listing area where cuisine filters or restaurant categories would be applied. Full credit if lettuce.com is attempted but is inaccessible (down, blocked, CAPTCHA) and the agent clearly reports the blocker. Partial credit if the agent uses an indirect but reasonable path on lettuce.com (e.g., site search) to reach the portfolio area.",

+ "max_points": 1,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Use lettuce.com portfolio filtering (or closest equivalent) for Spanish cuisine",

+ "description": "Apply a Spanish cuisine filter on lettuce.com if available. If a Spanish filter is unavailable/broken, full credit if the agent clearly reports this and uses the closest equivalent on lettuce.com (e.g., searching within the portfolio for “Spanish” or identifying Spanish cuisine via restaurant descriptions/tags on the portfolio). Partial credit if the agent finds Spanish restaurants but does not make the filtering/search step clear.",

+ "criterion": "Identify the first restaurant after applying the Spanish cuisine filter (or closest equivalent)",

+ "description": "Select and correctly identify the first restaurant shown in the Spanish-filtered results as displayed to the agent. Full credit if the agent selects the first visible result under the applied filter/search. If ordering is ambiguous or unstable (e.g., personalization, location prompts, dynamic loading, ties), full credit if the agent explains the ambiguity and uses a reasonable interpretation (e.g., first fully visible card in the results list). Partial credit if a Spanish restaurant is chosen but it is not the first visible result and no ambiguity is explained.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Navigate to the restaurant's own website and reach an online ordering interface",

+ "description": "From the identified restaurant, go to its official website and access the ordering flow (e.g., “Order Online” page or integrated ordering platform). Full credit if the ordering interface is reached, OR if the agent attempts and is blocked by external constraints (no online ordering, site down, geo restrictions, forced login, CAPTCHA) and clearly reports the limitation. Partial credit if the agent reaches the site but does not attempt to locate ordering despite it being available.",

+ "max_points": 3,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Add the 4 most commonly-ordered items to the cart (or best available equivalent)",

+ "description": "Identify and add the four items explicitly presented as “Most Popular/Popular/Top Ordered/Best Sellers” (or equivalent) on the ordering interface. Full credit if all four such items are added. If the interface shows fewer than four popular items or does not provide an explicit popularity section, full credit if the agent clearly states this and adds the closest equivalent set (e.g., the only available popular items plus the most prominently recommended/top-listed items) while explaining the selection basis. Partial credit if 1–3 correct popular items are added when a clear popular section exists, or if substitutes are used without explanation.",

+ "max_points": 8,

+ "justification": "",

+ "earned_points": ""

+ },

+ {

+ "criterion": "Proceed to checkout without crossing the critical point",

+ "description": "Proceed through the ordering flow to the checkout page/state (where customer/payment details would be entered) but stop before entering personal data or submitting the final order. Full credit if checkout is reached and the agent stops appropriately; also full credit if login/personal/payment details are required before showing checkout and the agent reports this blocker and stops. Partial credit if the agent stops earlier than necessary without a stated blocker. No credit if the agent submits the order or enters sensitive personal/payment information.",

+ "criterion": "Output the prices of the 4 items added to the cart (as shown in the interface)",

+ "description": "Provide the prices shown for each of the four items added (from menu/cart/checkout). Full credit if prices for all added items are reported as displayed (including required size/variant if applicable). If exact item-level prices are obscured until a later step (e.g., location selection, fulfillment method, login) or vary by required modifiers, full credit if the agent reports the blocker and provides the most specific prices available from the interface (e.g., base price before required modifiers) while noting what is missing. Partial credit if only 1–3 item prices are reported when all four are visible.",

+ "criterion": "Access and use the Office of Financial Readiness college savings calculator (specified tool)",

+ "description": "Agent navigates to and attempts to use the Office of Financial Readiness college savings calculator as required. Full credit if the agent reaches the calculator and can interact with it, OR if the agent makes a reasonable attempt but is blocked by uncontrollable factors (e.g., site down, CAPTCHA, geo-block, login requirement) and clearly reports the blocker. Partial credit if the agent uses a different calculator without clearly establishing the specified tool is inaccessible/unusable.",

+ "criterion": "Enter the required input values correctly using typed/manual entry (no sliders) when possible",

+ "description": "Agent inputs the exact stated values into the calculator: 3% education cost inflation, $50,000 current savings, $250 monthly contributions, 6% rate of return, and costs of $50,000/year tuition and $12,000 room/board (where the tool accepts those as inputs). Full credit if all values are entered correctly via non-slider/manual typed inputs. If the calculator only provides sliders (or otherwise prevents manual entry), full credit if the agent clearly reports that limitation and uses the tool's closest possible non-slider alternative (e.g., direct text boxes elsewhere, numeric entry after clicking the value) or explains why exact entry cannot be guaranteed. Partial credit if one value is incorrect/omitted despite the tool allowing correct entry, or if sliders are used even though manual entry is available and feasible.",

+ "criterion": "Report the calculator’s incremental additional monthly savings required (\"how much more per month\")",

+ "description": "Agent reports how much MORE per month needs to be saved compared to the existing $250/month, based on the calculator output. Full credit for stating the incremental amount (additional over $250) clearly and numerically. Partial credit if the agent reports only the total monthly contribution required by the tool but also provides the subtraction (total minus $250) or enough information to derive it unambiguously. If the calculator does not provide a monthly-required figure or does not expose outputs needed to compute the incremental amount, full credit if the agent clearly states what output(s) the tool provides instead and why the requested incremental amount cannot be determined from the tool as presented.",

+ "max_points": 6,

+ "criterion": "Access DOL news releases and attempt to identify the latest release",

+ "description": "Attempt to access the U.S. Department of Labor (DOL) news release listing (or an equivalent official DOL feed/page) and determine the most recent release. Full credit if the agent makes a reasonable attempt but cannot confirm the latest release due to external blockers (site down, CAPTCHA, network error) or because multiple releases appear equally 'latest' without a reliable tie-breaker and the agent explains the ambiguity and what was checked. Partial credit if the attempt is unclear or uses an unofficial/non-DOL source without justification.",

+ "criterion": "Correctly identify the latest DOL news release (headline/date) used for the answer",

+ "description": "Clearly identify the latest DOL news release selected (by headline and date, or another unambiguous identifier) and base subsequent answers on it. Full credit if the selected release is defensibly the latest among accessible/visible releases at the time of lookup (including choosing one of multiple same-date releases with a stated, reasonable tie-breaker such as latest timestamp or list order). Partial credit if a recent but not-latest release is used when the true latest is accessible and clearly indicated.",

+ "criterion": "Report the media contact for that release",

+ "description": "From the identified latest DOL news release, provide the media contact as presented on the release (name and/or labeled media contact line, plus phone/email if shown). Full credit if the media contact is correctly extracted, or if the release genuinely lists no media contact and the agent explicitly states that. Partial credit if the agent provides incomplete media contact details when additional details are clearly shown on the release, but the provided info is still clearly tied to that release.",

+ "max_points": 3,

+ "criterion": "Count how many other contacts are in the relevant department (excluding the media contact)",

+ "description": "Provide a numeric count of additional contacts beyond the media contact, and explain the source and definition of 'their department' used (e.g., other contacts listed in the same release contact block; or contacts listed on a clearly linked department/agency media-contacts page associated with the release). Full credit if the count is correct under a clearly stated, defensible scope, OR if the agent reasonably cannot determine the department roster/count due to missing/ambiguous linkage, unavailable pages, or no contact list being provided and it clearly reports what was checked and why the count cannot be confirmed. Partial credit if a count is given but the scope is not explained or the media contact is not excluded.",

+ "max_points": 3,

+ "criterion": "Access LinkedIn Jobs and attempt a search for 'Software Engineer' roles in Seattle",

+ "description": "Navigate to LinkedIn.com (Jobs) and attempt to search for 'Software Engineer' roles in the Seattle area. Full credit if the agent makes a reasonable attempt but is blocked by login/CAPTCHA/paywall/region restrictions or LinkedIn errors and clearly reports the blocker without inventing results. Partial credit if the attempt is unclear or the search is for a notably different role/location.",

+ "criterion": "Identify at least one qualifying LinkedIn posting for an established big-tech company (or best available)",

+ "description": "From visible LinkedIn results/postings, identify at least one job posting whose title includes 'Software Engineer' (or a close LinkedIn-standard variant like 'Software Development Engineer' with clear equivalence), located in Seattle/Seattle metro, and belonging to an established big-tech company. Full credit if at least one clear match is identified OR if no exact match is visible the agent selects the best available option matching the primary intent and explains the nearest-fit tradeoff. If LinkedIn is inaccessible as documented in the prior criterion, award full credit here if the agent does not fabricate postings and instead explains the limitation. Partial credit if the role/company fit is plausible but one attribute (title/location/big-tech status) is unclear.",

+ "criterion": "Retrieve and report the company name shown on the LinkedIn posting(s) (or explain why not possible)",

+ "description": "Extract the company name as displayed on the LinkedIn job posting(s) identified. Full credit if the company name is accurately captured and clearly tied to the posting; OR if LinkedIn access/visibility prevents extraction and the agent explicitly states that the company name could not be retrieved due to the blocker. Partial credit if the company name is ambiguous (e.g., recruiter vs. end employer) and the agent notes the ambiguity but cannot resolve it from what is visible.",

+ "criterion": "Use Wikipedia to find the company's founded year (or report unavailability)",

+ "description": "Using Wikipedia.org, locate the relevant page for the company name obtained from LinkedIn and report the 'Founded' year as shown. Full credit if the founded year is correctly reported from Wikipedia; OR if the Wikipedia page/founded year cannot be found (missing page, disambiguation without enough info, Wikipedia blocked) and the agent clearly reports the attempt and limitation without guessing. Partial credit if the agent uses a non-Wikipedia source only after a documented failed attempt on Wikipedia, or if the year is provided with minor ambiguity that is clearly explained (e.g., multiple founding-related dates listed).",

+ "criterion": "Access careers.microsoft.com and run the search for 'Applied Scientist' with Redmond, WA targeting",

+ "description": "Agent uses careers.microsoft.com to search for the keyword/title 'Applied Scientist' and applies or otherwise targets location to Redmond, WA (e.g., location filter or search query showing Redmond, WA results). Full credit if the agent clearly attempts this on the specified site and reports any blockers (CAPTCHA, region restrictions, infinite loading, auth gating, site down). Partial credit if the agent searches on the right site but location targeting is broader/unclear (e.g., Seattle area) despite reasonable effort.",

+ "criterion": "Select and open the first visible search result (or document inability to deterministically identify it)",

+ "description": "Agent opens the first job listing as presented in the search results at the time of viewing. Full credit if the agent clearly indicates which listing was first and opens it. Also award full credit if the agent explains why the first result cannot be deterministically identified/verified (e.g., results reorder on refresh, personalization, pagination/virtualized list prevents stable ordering) after reasonable attempts, and then proceeds with a defensible fallback (e.g., the first listing they can open while documenting the observed ordering). Partial credit if the agent opens an Applied Scientist posting but does not show it was first when it was feasible to do so.",

+ "criterion": "Extract the team or group name from the first job posting",

+ "description": "From the opened job posting page, extract the explicit team/group/org name the posting is for as written on the page. Full credit if the team/group name is accurately captured and attributed to the posting. Partial credit if only a broader org is provided when a more specific team name is present, or if the agent provides a reasonable best-effort inference because the posting is vague. Full credit if the posting does not specify a team/group name and the agent clearly reports that after checking the posting content.",

+ "max_points": 3,

+ "description": "Search outside careers.microsoft.com for information describing what the identified team/group does, and provide a grounded summary tied to the same named group. Full credit if the summary is supported by external evidence; also full credit if the agent cannot find reliable public info specific to the named group after reasonable search and clearly reports that (optionally noting only generic/parent-org info is available). Partial credit if the summary is overly generic or sources are only indirectly related but still plausibly about the group.",

+ "description": "Use external sources to identify the leader of the named group and report the person’s name with context linking them to the group. Full credit if the leader is correctly matched to the group; also full credit if the agent can only reliably identify leadership at a parent-organization level (and explains why direct group leadership is not publicly available) or if sources conflict and the agent reports the ambiguity. Partial credit if the leader is plausible but evidence is weak/indirect despite more direct evidence being publicly available.",

+ "criterion": "Identify applicable Jetstar fare type(s) for the Darwin–Adelaide trip dates (or state why they can’t be determined)",

+ "description": "Identify the fare type(s)/bundle(s) actually shown for the DRW↔ADL itinerary on the dates checked (e.g., Starter/Starter Plus/Flex or current equivalents). Full credit if the agent ties the fare type(s) to the itinerary it checked OR clearly explains that fare type couldn’t be determined because (a) flights were unavailable, (b) fare-family details weren’t shown, or (c) Jetstar site/tool access was blocked. Partial credit if the agent provides generally correct Jetstar fee info but does not connect it to any fare type context or explains assumptions unclearly. No credit if the agent invents fare types or misattributes policies to Jetstar.",

+ "criterion": "Cancellation fee policy (Jetstar) for the relevant itinerary/fare",

+ "description": "Provide Jetstar’s cancellation/refund policy as applicable to the identified fare type(s) (or, if fare type can’t be confirmed, describe policy by fare family with clear caveats). Should cover whether cancellation is permitted, what is refundable (fare vs. taxes/fees/credit), any cancellation fee concept, and key conditions/limits (e.g., flight credit vs. cash). Full credit if accurate and appropriately scoped to the fare(s) found or clearly caveated when the exact fare type could not be verified due to external limitations or no availability. Partial credit if generally correct but missing key conditions (e.g., taxes/fees treatment) or not clearly tied/caveated. No credit for fabricated amounts/terms.",

+ "criterion": "Change fee policy (Jetstar) for the relevant itinerary/fare",

+ "description": "Provide Jetstar’s change policy as applicable to the identified fare type(s) (or, if fare type can’t be confirmed, describe policy by fare family with clear caveats). Should cover whether changes are allowed, any change fee concept, and that fare difference may apply; note relevant bundle effects where applicable. Full credit if accurate and tied to the fare(s) found or clearly caveated when exact fare type could not be verified due to external limitations or no availability. Partial credit if it mentions changes but omits key conditions (fees/fare difference) or lacks clear tie/caveat. No credit for fabricated amounts/terms.",

+ "criterion": "Flight availability check for the requested timing (about a month from now, two-week trip)",

+ "description": "Attempt to check Jetstar availability for DRW→ADL about one month from now with a return about two weeks later. Full credit if the agent (a) states the exact dates it assumed (if the user didn’t specify exact dates) and (b) reports the observed availability results, OR if the agent clearly reports it could not verify availability because Jetstar tools/site were inaccessible (captcha/outage) after reasonable attempt. Partial credit if only one direction is checked, dates don’t reflect ~2-week trip window without explanation, or the attempt is unclear.",

+ "criterion": "Report unavailability when no flights exist (conditional)",

+ "condition": "Only applies if there are no available Jetstar flights for the selected outbound and/or return dates matching the 'in a month for a two week trip' timing.",

+ "description": "If no flights are available for the dates checked, the answer must explicitly state that no Jetstar flights were available and specify whether it was the outbound, return, or both, and for which assumed dates. Full credit if unavailability is clearly stated and scoped. Partial credit if unavailability is mentioned but not scoped to leg/dates. No credit if the agent fails to mention unavailability despite finding none.",