WebTailBench Rubric Comparison

Description Successfully navigate to Steam and retrieve the list of today's top-selling games. Partial credit could be awarded for finding a Steam sales or top-sellers page but not specifically for today's list.

Description From the retrieved list, correctly determine the first game that has an existing TV series adaptation. Partial credit may be given for naming a game that is a top-seller but misidentifying its adaptation status, or noting correctly that none exist.

Description Search JustWatch.com for the identified TV series adaptation and list the streaming platforms where it is available. Partial credit for listing some but not all current providers, or for correctly stating that the series is not found or not available on streaming.

Description Access Steam and view today's Top Sellers list to establish the current ranking. Partial credit if Steam is accessed and the Top Sellers section is located but 'today' cannot be confirmed.

Description From today's Top Sellers ranking, find the highest-ranked (first) game that has a TV series adaptation. Provide the game title and confirm that the adaptation is a TV series. Full credit also awarded if none of the top sellers have a TV series adaptation today and this is clearly stated. Partial credit if a game is identified but not verified as a TV series adaptation, or if ranking/order is not respected.

Description Search JustWatch.com for the identified TV series adaptation and list the streaming services where it is available. Partial credit if JustWatch is used but the list is incomplete or availability cannot be fully determined.

Description Attempt to open Steam’s Top Sellers/Top Selling games list as of today. Full credit if the agent makes a reasonable attempt to access the correct Steam list but is blocked by CAPTCHA, region restrictions, outage, or login/age gate and clearly reports what prevented verification. Partial credit if the agent uses a nearby but not equivalent Steam list (e.g., Popular/Trending) while explaining the limitation.

Description If the Top Sellers list is accessible, correctly report the top-ranked (#1) game shown for today/time of check. Full credit if the #1 game is captured from the Steam Top Sellers page; partial credit if the agent provides a plausible top seller but the source/timeframe is unclear. Full credit if identification is impossible solely because Steam access was blocked and the agent clearly states that the #1 game could not be confirmed.

Description Assess whether the identified #1 game has a TV series adaptation and state the conclusion. Full credit if the agent correctly determines either that a TV series adaptation exists (naming it) or that none exists, with reasonable support. Full credit if the agent cannot confidently confirm due to inaccessible/conflicting sources and explicitly reports what was checked and the remaining uncertainty.

Description If a TV series adaptation exists, attempt to find that series on JustWatch.com. Full credit if the agent successfully locates the correct series entry on JustWatch, or if JustWatch is inaccessible (CAPTCHA/outage/forced login/consent flow that prevents search) and the agent clearly reports the blocker. Partial credit if the agent relies on non-JustWatch sources despite JustWatch being apparently accessible.

Description Report the streaming services where the series is available as shown on JustWatch (subscription vs rent/buy if presented; at minimum list the services). Full credit if the agent lists the complete set of services visible for the user’s JustWatch locale. Full credit if JustWatch does not show availability for that title/locale or availability cannot be loaded due to location settings/consent/technical issues and the agent clearly reports this limitation and what was/was not visible. Partial credit if only some services are listed when more are shown.

Description Successfully navigate to or open the Eventbrite.com website as specified.

Description Search within Eventbrite for events in Nashville, TN and identify at least one live music event scheduled for the upcoming Saturday. Partial credit for finding a relevant event but with missing date confirmation.

Description Extract the name(s) of the artist(s) performing at the chosen event. Partial credit if only some artists are identified or names are ambiguous.

Description Successfully navigate to or open the Spotify.com website as specified.

Description Search for and list songs by one or more of the previously identified performing artist(s) on Spotify. Partial credit for finding at least one song or partial artist match.

Description Successfully navigate to or use the Eventbrite.com website as specified in the task.

Description Find an event that is explicitly live music and located in Nashville, Tennessee on Eventbrite.com. Partial credit may be awarded if an event in Nashville is found but the 'live music' aspect is unclear, or a live music event is found but the location is not clearly Nashville. Full credit is awarded if no such event exists and the agent clearly reports that.

Description Ensure the selected event is scheduled for the upcoming Saturday relative to the current date. Partial credit may be awarded if the event is on a Saturday but not clearly the upcoming one, or the date is ambiguous. Full credit is awarded if no qualifying events exist and the agent explicitly states that.

Description Extract the performing artist names as listed on the Eventbrite event page. Partial credit may be awarded if artist information is limited or not listed, provided that this is clearly stated.

Description Navigate to or use Spotify.com as specified in the task (without requiring login or other personal information).

Description Search for the identified artist(s) on Spotify.com and find at least one song (track) by any of them. Partial credit may be awarded if the artist page is found but specific song titles are not identified. Full credit is awarded if no songs exist or cannot be found and the agent clearly states that.

Description Attempt to navigate to Eventbrite.com and perform a search for events in Nashville, TN that are described/categorized as live music. Full credit if the agent makes a reasonable attempt but is blocked by a CAPTCHA/login wall/site outage and clearly reports the blocker. Partial credit if the agent searches the wrong location or does not clearly use Eventbrite first.

Description From Eventbrite results/event pages, identify at least one event that is explicitly live music, located in Nashville, TN, and scheduled for the upcoming Saturday (relative to execution date). Full credit if an exact match is found OR if, after reasonable search/filtering, no exact match appears to exist and the agent clearly reports that (optionally providing the closest available live-music Nashville alternative and explaining the mismatch). Partial credit if an event is live music in Nashville but on a different date, or on the correct Saturday but outside Nashville, when closer matches are available.

Description Extract and report the performing artist name(s) as listed on the Eventbrite event page. Full credit if at least one performer is correctly identified OR if the event page does not list performers (or only lists a venue/DJ night without a clearly named act) and the agent explicitly states that limitation. Partial credit if the agent provides an ambiguous performer identification while noting uncertainty, or mistakes a venue/organizer for an artist when the performer is actually listed.

Description Attempt to use Spotify.com to search for at least one of the identified performers and provide at least one song by a correctly matched artist. Full credit if a correct song is found OR if Spotify is inaccessible (CAPTCHA/login wall/site error) and the agent reports the blocker OR if Spotify is accessible but the performer cannot be found/does not appear to have a Spotify catalog and the agent clearly reports that outcome after reasonable search (including disambiguation attempts such as adding location/genre). Partial credit if the agent finds an artist page but does not name any song, or returns a similarly named but unverified/likely incorrect artist without noting uncertainty.

Description Successfully find and access the Amazon product page for the book titled 'The Innovator's Dilemma'.

Description Identify and record the book’s current 'Best Sellers Rank' in the Books category as shown on its Amazon page.

Description Search for and identify a repair service located in the United States whose telephone number contains the rank number (as a continuous substring).

Description Provide the exact name of the repair service and its full phone number that matches the rank substring requirement.

Description Navigate to and view the specific Amazon listing for the book 'The Innovator's Dilemma'. Partial credit may be awarded if the agent clearly attempts to find the correct page but cannot access it or identifies a closely related listing.

Description Locate and extract the book's overall rank in the Books category from the Amazon page. Partial credit may be given if category-specific ranks are reported instead of the overall Books rank, or if the agent explicitly notes that the overall rank is not available.

Description Identify a repair service located anywhere in the United States (any type of repair service qualifies). Partial credit may be given if a repair service is found but the US location is unclear or not verified.

Description Confirm that the repair service's phone number includes the exact sequence corresponding to the extracted rank as a substring. Partial credit may be granted if there is a close match but with formatting ambiguities, or if a reasonable attempt is made but the substring match is incorrect.

Description Provide the name and phone number of the selected repair service. Partial credit may be awarded if only one of these (name or phone number) is provided.

Description Attempt to navigate to the Amazon product page for "The Innovator's Dilemma". Full credit if the agent attempts access but is blocked by CAPTCHA/login/region gating/page load failures and clearly reports the blocker. Partial credit if the agent navigates to an Amazon results page or a different edition without clarifying uncertainty.

Description From the Amazon product page (if accessible), extract the rank in Books overall (not a category/subcategory rank) and record it exactly as shown. Full credit if the agent provides the exact overall Books rank; also full credit if the rank is not visible/unstated due to page variability or blockers and the agent explicitly explains why it cannot be determined. Partial credit if the agent provides only a category rank but clearly flags it as such or explains the ambiguity. No credit if the rank is fabricated or sourced off-Amazon without attempting Amazon first.

Description Using the extracted overall Books rank digits as a contiguous substring, search for a repair service located in the US with a publicly listed phone number containing that exact substring. Full credit if a verifiable repair service is found and its phone number contains the substring; also full credit if the agent demonstrates reasonable search effort (e.g., multiple queries/sources) and reports that no matching phone number could be found. Partial credit if a repair service is found but the digit match is incorrect (non-contiguous/mismatched) or if search effort is minimal/unclear.

Description Provide the final answer with the repair service name and its phone number that contains the rank substring. Full credit if both are provided and correspond to a verified match. If criterion 3 concludes no match exists after reasonable search, full credit for explicitly stating that no qualifying repair service phone number was found (rather than inventing one). Partial credit if only the name or phone is provided, or if the pairing is inconsistent.

Description Navigate to wikipedia.org and locate the official Harvard University article. Partial credit if the agent reaches Wikipedia but not the correct page; full credit for accurately accessing the Harvard University entry.

Description Identify and report the location information of Harvard University as stated on its Wikipedia page (e.g., city, state, country). Partial credit for partially correct location (e.g., city but missing state), full credit for the complete correct location.

Description Go to Google Maps, input the Harvard University location as the origin and Boston City Hall as the destination, select walking directions, and provide the route (including major steps, total distance, and estimated walking time). Partial credit for finding the route but omitting details; full credit for complete, accurate walking directions.

Description Access Wikipedia.org and open the Harvard University article as explicitly requested. Partial credit may be awarded for reaching Wikipedia but not the correct page, or demonstrating attempts to find the correct page.

Description Identify Harvard University's location information from the Wikipedia article (e.g., Cambridge, Massachusetts; address/campus location if available). Partial credit may be awarded for correctly identifying the city/state without a more precise address. Full credit is awarded if the agent indicates that detailed location is not available on the page.

Description Use Google Maps to obtain walking directions from the Wikipedia-derived Harvard University location to Boston City Hall. Full credit includes using walking mode, correct origin and destination, and providing the resulting directions outcome (e.g., route/distance/duration or a directions link). Partial credit may be awarded if directions are provided but not in walking mode, the origin is approximate, or a different mapping service is used. Full credit should be awarded if Google Maps indicates directions are unavailable and the agent clearly states that.

Description Use Wikipedia.org to look up the 'Harvard University' article and identify its stated location (e.g., Cambridge, Massachusetts, United States). Full credit if the agent uses Wikipedia and reports the correct city/state/country (or equivalent specificity) as shown on the page. Partial credit if the agent uses Wikipedia but reports an incomplete/ambiguous location (e.g., only 'Massachusetts' or only 'Harvard University'). Full credit if Wikipedia is inaccessible (blocked/down/CAPTCHA) and the agent clearly reports the issue and uses a reasonable alternative source to determine the location, stating that it is an alternative.

Description On Google Maps, attempt to obtain directions with the origin set to the Harvard University location found in the prior step and the destination set to 'Boston City Hall', with travel mode set to walking. Full credit if the agent correctly sets origin/destination and selects walking mode, OR if Google Maps is inaccessible/fails to load directions and the agent clearly reports the blocker and provides the best available alternative method/provider for walking directions (or clearly states that walking directions could not be retrieved). Partial credit if directions are obtained but the travel mode is not walking, or if the origin is materially imprecise/mismatched to the Wikipedia-derived location when a more precise origin is available.

Description If walking directions are successfully retrieved from Google Maps (or a clearly stated alternative due to Google Maps failure), report at least total walking time and total distance (optionally include main streets). Full credit if both time and distance are reported. Partial credit if only one of time or distance is reported. If directions could not be retrieved due to external blockers and the agent clearly reported that in the prior step, do not penalize here (award full credit). No credit if the agent fabricates time/distance without evidence or fails to report available time/distance from the retrieved directions.

Description Identify an existing coding bootcamp company located in Brooklyn, New York City. Partial credit if a Brooklyn-based tech school is identified but it is unclear whether it is primarily a coding bootcamp.

Description Provide the cost of the full-time tuition for the identified Brooklyn coding bootcamp. Partial credit for an approximate range if an exact figure is not available.

Description Use Google Maps (or equivalent mapping service) to identify a bus service that runs from Grand Army Plaza to the bootcamp location. Partial credit if a route is identified but with transfers or disclaimers.

Description Present the bootcamp name, full-time tuition cost, and the bus service name in the response as specified by the task.

Description Find and name a coding bootcamp company that has a location in Brooklyn, NYC. Partial credit may be awarded if a coding bootcamp is identified but the location is ambiguous or not in Brooklyn.

Description Provide the tuition cost for the bootcamp's full-time program at the identified Brooklyn location. Full credit if the exact full-time tuition is given or if it is clearly stated that no full-time program exists (with that unavailability noted). Partial credit may be awarded for a close estimate, a range, or a tuition figure from a different program with that distinction clearly noted.

Description Identify which bus service (route name/number) can be taken from Grand Army Plaza to reach the bootcamp, using Google Maps. Full credit requires a bus service name derived via Google Maps for the specified origin and destination. Partial credit may be awarded for a plausible bus route without explicit mention of Google Maps or with incomplete details.

Description Explicitly output the bootcamp name, the full-time tuition cost, and the bus service name as requested. Partial credit may be awarded if one or two of these items are provided but not all three.

Description Identify at least one coding bootcamp company that is located in Brooklyn, NYC (address/neighborhood indicates Brooklyn). Full credit if the bootcamp is clearly in Brooklyn. Full credit also if the agent makes a reasonable attempt to verify a Brooklyn location but finds the bootcamp has moved/closed or the location cannot be verified from accessible sources, and then clearly reports this and provides a reasonable alternative bootcamp that is verifiably in Brooklyn. Partial credit if the bootcamp is in NYC but the borough is unclear or not verified. No credit if the selected bootcamp is not in Brooklyn when verifiable Brooklyn options are available.

Description Find and report the bootcamp's full-time tuition amount. Full credit if a specific numeric full-time tuition is provided and is clearly tied to the full-time program (including clearly stated mandatory fees if presented as part of tuition). Full credit also if the bootcamp does not publish full-time tuition (or it is not accessible due to paywalls/login/region gating) and the agent clearly states that the full-time tuition is not publicly available, optionally providing the best available related pricing info (e.g., range, ISA terms) with appropriate caveats. Partial credit if only a range or ambiguous/outdated figure is provided without clarifying uncertainty.

Description Using Google Maps directions (Transit), determine a bus service/route that can be taken from Grand Army Plaza to reach the selected bootcamp location. Full credit if (a) a specific MTA bus route/service name (e.g., B41, B45, B67) is provided and is plausibly part of the Google Maps transit itinerary, OR (b) Google Maps is inaccessible (captcha/outage) and the agent clearly reports the blocker and provides the best available alternative bus route information from another credible transit source while explicitly noting it is not from Google Maps, OR (c) Google Maps transit directions do not include any bus leg (or show no feasible bus option) and the agent clearly reports that outcome and provides the closest feasible transit alternative shown by Google Maps. Partial credit if only general guidance is given (e.g., 'take a bus') or if the bus route is incomplete/unclear.

Description Output includes all three explicitly requested items: (1) bootcamp name, (2) full-time tuition cost (or a clear statement that it is not publicly available), and (3) bus service name (or a clear statement that Google Maps provides no bus option / Google Maps inaccessible with noted alternative source). Full credit if all three are present and correspond to the same selected bootcamp/directions (or if a required item is unavailable but the agent clearly reports the limitation as described above). Partial credit if one of the three is missing or not clearly labeled. No credit if two or more are missing or mismatched (e.g., bus route for a different destination than the named bootcamp).

Description Successfully navigate to or access the lettuce.com website as specified in the task.

Description Apply the Spanish cuisine filter on lettuce.com to display only Spanish restaurants. Partial credit may be awarded for attempting the filter but not fully limiting results to Spanish cuisine.

Description Select or identify the first restaurant listed after applying the Spanish cuisine filter. Partial credit if a different restaurant is selected but the process of choosing the first listing is shown.

Description Click through from lettuce.com to the identified restaurant's own website. Partial credit if the link is located but fails to load, with the failure noted.

Description On the restaurant's website, determine which four menu items are most commonly ordered and add all four to the cart. Partial credit for fewer items or unclear identification of popularity.

Description List the names and prices of the four items added to the cart. Partial credit if names are correct but some prices are missing or incorrect.

Description Navigate to the checkout page without entering personal or payment information. Partial credit if the checkout button or link is located but not clicked.

Description Successfully navigate to or access the lettuce.com website as specified in the task.

Description Use the site's portfolio filter to select Spanish cuisine. Partial credit if the agent attempts the filter but it is unavailable or yields no results and this is clearly stated.

Description Correctly identify the first restaurant listed under the Spanish cuisine filter. Partial credit if a Spanish restaurant is identified but not necessarily the first, or if ordering is impossible because none are listed and this is reported.

Description Navigate from the portfolio to the identified restaurant’s website and locate the ordering interface (on-site or via a linked ordering platform). Partial credit if only the restaurant website is reached without a clear ordering path, with this limitation noted.

Description From the ordering interface, identify and add four items labeled as most popular/commonly ordered to the cart. Partial credit for fewer than four items, or if such labels are absent, for selecting a reasonable equivalent and stating that limitation.

Description Navigate to the checkout or order review page, stopping before entering any personal or payment information. Partial credit for reaching the cart page if the platform requires login or personal info to proceed further and this is stated.

Description Provide the names and prices of the selected four items as displayed on the ordering interface. Partial credit if some prices are missing due to unavailability and this is clearly noted.

Description Use lettuce.com as the starting platform and attempt to reach the portfolio/listing area where cuisine filters can be applied. Full credit if the portfolio/listing area is reached, OR if access is blocked (captcha, outage, geo restriction, access wall) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent switches to alternative sources without first attempting lettuce.com.

Description Apply the Spanish cuisine filter (or the closest available equivalent, e.g., 'Spain/Spanish') on lettuce.com's portfolio and identify the first restaurant in the filtered results as displayed. Full credit if the filter is applied and the first visible result is identified. Full credit if the filtered results are empty and the agent clearly reports that. Full credit if the site’s ordering is ambiguous/unstable (e.g., no clear sort order, infinite scroll, personalization) and the agent clearly explains how 'first' was interpreted (e.g., topmost visible result) and proceeds accordingly. Partial credit if a Spanish restaurant is selected without demonstrating that the Spanish filter was used when it was available.

Description From the restaurant identified on lettuce.com, navigate to the restaurant's official website or the official online ordering page linked from it and reach the point where menu items can be added to a cart. Full credit if the ordering interface is reached. Full credit if the restaurant has no online ordering or ordering is unavailable (closed hours, delivery disabled, location selection required, login wall) and the agent clearly reports what prevented progress and any visible alternatives (phone/in-person/third-party) without fabricating availability. Partial credit if the agent uses an unofficial/third-party ordering site without first attempting the restaurant's own official path when available.

Description Add one distinct item that is among the four most commonly-ordered/most popular/top ordered items as indicated by the ordering UI. Full credit if added. If the site does not display 'most commonly-ordered' (or only shows it after selecting location/time), full credit if the agent clearly reports the absence/constraint and uses the best available proxy exposed by the platform (e.g., 'Most Popular' section, 'Featured', or top items shown) and explains the choice. Partial credit if the correct item is identified but cannot be added due to external blockers (sold out, ordering disabled) that are clearly reported.

Description Add a second distinct commonly-ordered/popular item to the cart, completing required modifiers using defaults or reasonable selections when forced. Full credit if added or if prevented by external constraints that are clearly reported (sold out, modifier bug, ordering closed). Partial credit if added but required selections are clearly incomplete or invalid when a default/obvious choice existed.

Description Add a third distinct commonly-ordered/popular item to the cart. Full credit if added or if prevented by uncontrollable constraints (sold out, ordering closed, item unavailable at selected location) that are clearly reported and the agent selects the next-best commonly-ordered alternative if available.

Description Add a fourth distinct commonly-ordered/popular item to the cart. Full credit if added or if prevented by uncontrollable constraints that are clearly reported and the agent selects the next-best commonly-ordered alternative if available.

Description Proceed from the cart to the checkout flow (e.g., checkout page, delivery/pickup selection, order summary) but stop before entering any personal data (name, email, phone, address) or payment information, and do not place the order. Full credit if the agent reaches checkout and stops appropriately, OR if the site requires login/personal/payment details before showing checkout and the agent reports this blocker and stops. No credit if the agent places the order or enters sensitive personal/payment details not provided by the user.

Description Provide the displayed prices for each of the four added items (item name paired with its line-item/base price as shown in the cart/checkout). Full credit if all four are reported accurately as displayed, including any required-option price changes if applicable (or clearly noting when price varies by required options/location). Partial credit if fewer than four prices are provided or if prices are reported without clarifying that required modifiers/size affected the price when that ambiguity is visible.

Description Navigate to the Booking.com website as specified in the task.

Description Perform a search on Booking.com for hotels in Jakarta for a 3-night stay starting December 15, 2025 for 2 adults.

Description Apply a filter to include only hotels with a review score of 8/10 or higher.

Description Among the filtered hotels, find the room option with the lowest total price for the specified stay.

Description Extract the full address of the chosen hotel from its listing.

Description Use the hotel's address to search for and identify the nearest coffee shop.

Description Output the name and full address of the closest coffee shop found.

Description Use Booking.com to search in Jakarta with check-in on December 15, 2025 for a three-night stay (Dec 15–18, 2025) for 2 adults. Full credit if all parameters (site, city, dates, length, and guests) are correctly set. Partial credit if some parameters are correct. Full credit also if the agent reports that no results are available for the exact parameters.

Description Ensure that only hotels with a review score of at least 8/10 are considered. Full credit if an explicit filter is applied or the chosen hotel clearly meets 8/10+. Partial credit if the hotel appears highly rated but the threshold is not clearly verified.

Description From the qualifying results, identify the cheapest available hotel room for the specified dates and party size. Full credit if the selected option is the cheapest among those meeting the criteria and is available for the dates. Partial credit if a valid option is found but not confirmed as the cheapest, or if no qualifying options exist and this is clearly stated.

Description Provide the hotel's street address obtained from Booking.com (or the hotel’s official listing) to use for the proximity search. Partial credit if only a partial or approximate address is provided.

Description Using the hotel's address, search for the nearest coffee shop and output the coffee shop's name and address. Full credit requires identifying the closest option and providing both name and address. Partial credit if a nearby coffee shop is provided but not clearly the closest, or if either the name or address is missing.

Description Attempt to use Booking.com and set: destination Jakarta; check-in Dec 15, 2025; check-out Dec 18, 2025 (or 3 nights); 2 adults. Full credit if the agent clearly attempts to use Booking.com with these parameters or explicitly notes an equivalent UI flow (e.g., selecting 3 nights). Full credit if Booking.com is inaccessible (CAPTCHA, outage, login wall) and the agent clearly reports the blocker and what was attempted. Partial credit if minor parameter mismatch occurs but is clearly disclosed and corrected, or if the attempt is unclear.

Description Apply a review-score filter (8.0+) or otherwise verify from Booking.com that the chosen property is rated at least 8.0/10. Full credit if enforced via filters or verified on the property page. Full credit if, after a reasonable attempt, no 8.0+ properties appear available for the dates/guests and the agent clearly reports this. Partial credit if a score is mentioned but the Booking.com source/threshold is not clearly confirmed.

Description From the Booking.com results consistent with the parameters and 8.0+ constraint, identify the lowest-priced available room option for the entire 3-night stay, clearly stating whether the price shown is total vs per-night and whether taxes/fees are included as displayed. Full credit if the agent demonstrates a reasonable comparison among visible 8.0+ options and selects the cheapest shown. Full credit if no qualifying availability exists (or prices cannot be retrieved) and the agent clearly reports this with evidence from the Booking.com attempt; optionally, it may provide the best available alternative (e.g., closest-to-cheapest among 8.0+ or cheapest below 8.0) while explicitly noting the deviation. Partial credit if the comparison is unclear or the price basis (total vs nightly / taxes) is not stated.

Description Report the hotel's physical address as displayed on Booking.com for the selected property. Full credit for a complete address (street/area + city; postal code if shown). Full credit if the agent cannot retrieve the address due to a Booking.com blocker/limited listing details and explicitly states this while providing the best available location information shown (e.g., neighborhood, map pin area) without fabrication. Partial credit if the address is materially incomplete but still plausibly identifies the location.

Description Using the hotel address as the anchor, attempt to find the nearest coffee shop via a maps/search tool and output the coffee shop’s name and address. Full credit if the agent clearly bases the search on the hotel address and provides both name and address. Full credit if map/search tools are inaccessible or results are ambiguous (e.g., multiple equidistant options, address too imprecise) and the agent reports the blocker/ambiguity and provides the best-supported nearest option(s) with an explanation of the basis used (e.g., closest shown by the tool). Partial credit if only name or only address is provided, or if “closest” is asserted without any clear basis when a basis was available.

Description Successfully navigate to or access the bklynlibrary.org website as specified in the task.

Description Locate and list all Brooklyn Public Library branches that offer a teen tech help center. Partial credit may be given if only some are identified.

Description From the branches identified, correctly determine which one is geographically the northern-most. Partial credit for approximate selection.

Description Find and report the exact year that the northern-most branch opened to the public. Partial credit for a close or approximate year.

Description Locate and report the official square-footage of the northern-most branch. Partial credit for a close estimate.

Description Identify and report the name of the managing librarian for the northern-most branch. Partial credit for a close or partial name.

Description Successfully navigate to or access the bklynlibrary.org website as specified in the task.

Description Find library branches on bklynlibrary.org that have a Teen Tech Help Center and correctly determine which one is the northern-most. Partial credit may be awarded for locating relevant branches but failing to identify the northern-most, or for a reasonable attempt if the site lacks clear geographic cues. Full credit is awarded if the site indicates no such centers exist and the agent clearly states that.

Description Locate and report the year the identified branch opened to the public from bklynlibrary.org. Partial credit may be given for related opening information (e.g., renovation year) if the specific 'opened to the public' year is not available and this is clearly noted.

Description Locate and report the total square footage of the identified branch from bklynlibrary.org. Full credit is awarded if the site indicates this information is unavailable and the agent clearly notes that.

Description Locate and report the name of the managing librarian for the identified branch from bklynlibrary.org. Full credit is awarded if the site indicates this information is unavailable and the agent clearly notes that.

Description Attempt to use bklynlibrary.org (site search, branch listings, and/or individual branch pages) to determine which branch(es) have a Teen Tech Help Center. Full credit if the agent attempts access and clearly reports if blocked (captcha/paywall/outage) or if Teen Tech Help Center information cannot be located on the site after reasonable searching. Partial credit if the agent uses bklynlibrary.org but the attempt is superficial/unclear. No credit if the agent does not attempt bklynlibrary.org while it appears accessible.

Description Identify at least one branch explicitly indicated on bklynlibrary.org as having a Teen Tech Help Center. Full credit if the qualifying branch list is correctly drawn from bklynlibrary.org pages. Partial credit if the agent mixes in non-bklynlibrary sources but still correctly identifies qualifying branches and indicates which claims are from bklynlibrary.org. Full credit if the site is accessible but it appears bklynlibrary.org does not provide any Teen Tech Help Center-by-branch information and the agent clearly states that finding.

Description From the bklynlibrary.org-identified set of branches with a Teen Tech Help Center, select the geographically northern-most branch. Full credit if the selection is correct given the available location/address information on bklynlibrary.org. If bklynlibrary.org does not provide enough information to unambiguously rank branches by latitude (or addresses are missing/unclear), award full credit if the agent clearly explains the ambiguity, shows reasonable comparison effort (e.g., comparing addresses/neighborhoods), and provides the best defensible choice. Partial credit if the agent selects a qualifying branch but provides no comparison/justification when comparison appears feasible.

Description Report the year the selected branch opened to the public using bklynlibrary.org branch information. Full credit for the correct year when present. If bklynlibrary.org does not list an opening year (or only lists renovation/reopening dates without original opening), award full credit if the agent clearly states the information is not available/unclear on bklynlibrary.org after reasonable searching and does not invent a year.

Description Report the branch's square footage as listed on bklynlibrary.org. Full credit for the correct square footage when present. If square footage is not provided on bklynlibrary.org (or is ambiguous between building vs. lot size), award full credit if the agent clearly reports that the value is missing/ambiguous on bklynlibrary.org after reasonable searching and avoids guessing.

Description Report the managing librarian name for the selected branch as shown on bklynlibrary.org. Full credit for the correct person and role when present. If managing librarian info is not available on bklynlibrary.org (or staff roles are not listed), award full credit if the agent clearly states it cannot be found there after reasonable searching and does not substitute another staff role without noting the mismatch.

Description Do not invent Teen Tech Help Center status, opening year, square footage, or managing librarian. Full credit if all reported facts are consistent with bklynlibrary.org or the agent transparently reports blockers, missing fields, or ambiguity. Partial credit if minor ambiguity exists without clear attribution. No credit if key facts are fabricated or if the agent claims bklynlibrary.org support when it does not.

Description Navigate to Google Flights as specified in the task.

Description Enter origin (DFW), destination (MIA), departure date (Jan 20, 2026) and return date (Jan 25, 2026) correctly in Google Flights.

Description Locate and select the flight option with the lowest total round-trip price.

Description Note the exact local arrival date and time of the chosen flight into MIA.

Description Navigate to Rentalcars.com as specified in the task.

Description Search for compact cars for pickup at Miami International Airport, ensuring the pickup datetime is at least one hour after the recorded flight arrival.

Description From the search results, find the compact car option with the lowest daily rate.

Description For the first (cheapest) compact car result, output the daily price, the vehicle make and model, and the number of seats.

Description Navigate to and use Google Flights as explicitly required by the task. Partial credit may be awarded if the agent searches for flights but not via Google Flights. Full credit should still be awarded if Google Flights indicates no available flights for the specified dates and the agent reports that clearly.

Description Configure a round-trip search from Dallas (DFW) to Miami (MIA) departing Jan 20, 2026 and returning Jan 25, 2026, and identify the lowest-price option. Partial credit if the correct route and dates are set but the agent does not confirm it is the lowest price, or provides incomplete pricing. Full credit if no flights are available and the agent states that explicitly.

Description Identify and clearly note the arrival timestamp of the outbound flight into Miami (local time). Partial credit if the arrival time is provided but ambiguously labeled or without local context.

Description Navigate to Rentalcars.com, set pickup location to Miami International, set the pickup time to begin no less than one hour after the flight’s arrival time, and filter/search for compact cars. Partial credit if the site and location are correct but the timing is not at least one hour after arrival, or if compact filtering is not applied. Full credit if compact cars are unavailable and the agent reports that clearly.

Description From the configured search results, identify the cheapest compact car and initiate the booking flow up to selection (e.g., selecting the vehicle or proceeding to the next step) without entering personal information or completing checkout. Partial credit if the cheapest compact option is correctly identified but not selected.

Description Provide the price per day, make/model, and number of seats for the first result in the search results list. Partial credit if only some of the required fields are provided.

Description Attempt to use Google Flights to search a round-trip itinerary from Dallas (DFW) to Miami (MIA) departing Jan 20, 2026 and returning Jan 25, 2026. Full credit if the agent performs the search on Google Flights or clearly reports being blocked (CAPTCHA/outage/region restriction) after a reasonable attempt. Partial credit if the agent searches the wrong dates/airports or does not make clear what was searched.

Description From the available Google Flights results for the correct route/dates, identify the lowest-priced round-trip option visible at the time of search. Full credit if the cheapest visible option is selected OR if Google Flights results cannot be accessed and the agent uses a reasonable alternative source (e.g., airline site/other major flight aggregator) while preserving route/dates and explains why. Also award full credit if the agent clearly reports that no valid itineraries/prices are shown for those dates (e.g., error/no availability). Partial credit if an option is selected but it is not the cheapest when a cheaper one is clearly visible and no justification is given.

Description Provide the arrival date/time in Miami for the selected outbound (DFW→MIA) flight. Full credit if the correct timestamp is reported. If the source does not display an arrival timestamp (or is blocked), full credit if the agent states this and provides the closest available equivalent (e.g., scheduled arrival window or asks for a preferred itinerary/airline to proceed). Partial credit if time is incomplete/ambiguous but leg/city are correct.

Description Attempt to use Rentalcars.com with pickup location set to Miami International (MIA) and a rental start time no less than one hour after the reported flight arrival time. Full credit if the constraint is applied as stated, OR if Rentalcars.com cannot be accessed (blocked/outage) and the agent clearly reports the blocker after a reasonable attempt. Partial credit if the pickup location is correct but the time constraint is not met or the time is not clearly set.

Description From the results, select the cheapest option in the compact category (as shown) and output: price per day, make/model, and number of seats for the first/cheapest compact result. Full credit if all three fields are provided for the cheapest compact option visible. If no compact cars are available (or compact filtering is not possible/unclear on the platform), full credit if the agent clearly states this and selects the cheapest closest alternative category available (e.g., economy) while still providing the three required fields. Partial credit if one required field is missing or if the chosen car is not the cheapest available in the stated category when a cheaper one is clearly visible.

Description Do not complete flight or car booking/purchase steps that require personal data, account login, or payment information. Full credit if the agent stops at a pre-checkout stage and does not enter fabricated personal/payment details. Partial credit if the agent navigates close to checkout but does not submit/confirm. No credit if the agent completes a booking/purchase or enters personal/payment information.

Description Navigate xbox.com and identify which game is listed as the top-selling Xbox title.

Description Find and report the name of the publisher for the top-selling Xbox game as stated on xbox.com.

Description Find and report the official release date of the top-selling Xbox game.

Description Determine who is currently the CEO or head of the studio that developed or published the top-selling Xbox game.

Description Find and report the birth date (day, month, year) of the identified CEO or studio head.

Description Compute how many years have passed from the CEO or head's birth date to the game's release date.

Description Find and name a game that xbox.com explicitly lists as a top-selling Xbox game. Full credit requires that the selection clearly comes from xbox.com. Partial credit if a likely candidate is identified without clear confirmation from xbox.com, or if the agent reports that xbox.com does not provide a top-selling list and states that limitation.

Description State who the identified game was published by. Partial credit if the developer is provided instead of the publisher or if there is ambiguity that is acknowledged.

Description Provide the game's release date. Partial credit if only the month/year are provided or if multiple regional dates are noted with a reasonable choice or explanation.

Description Find the CEO or head of the identified publisher (the gaming studio) and provide their date of birth. Partial credit if a senior leader (e.g., president, studio head) is given when a CEO is not applicable, or if the role is correctly identified but the birth date is uncertain and the limitation is stated.

Description Calculate how many years elapsed (i.e., the leader’s age at release). Full credit for a correct calculation considering exact dates; partial credit for an approximate calculation or clearly stated assumptions when exact dates are unavailable.

Description Attempt to navigate to xbox.com (Microsoft/Xbox store pages) and locate a context that lists or labels games as “Top-selling” (or equivalent, e.g., “Top selling games”). Full credit if the agent makes a reasonable attempt and clearly reports a blocker (CAPTCHA, login wall, region lock, site error, dynamic content preventing verification). Partial credit if the attempt is unclear or uses only non-xbox.com sources without first attempting xbox.com.

Description Name a game that xbox.com explicitly labels/lists as “top-selling” in the accessed context. Full credit if the top-selling designation is clearly tied to xbox.com. If xbox.com access/verification is blocked, full credit if the agent clearly states the limitation and uses a reasonable alternative signal (e.g., cached page, reputable third-party capture, or Microsoft/Xbox official channels) while explicitly labeling it as not directly verified from xbox.com. Partial credit if a game is from xbox.com but the top-selling context is not established.

Description For the selected game, report the publisher and release date as shown on xbox.com. Full credit if both are provided with clear linkage to xbox.com. If one/both fields are not shown, are inconsistent across locales, or are inaccessible due to blockers, full credit if the agent explicitly states what was missing/unavailable on xbox.com and (optionally) provides the missing info from an alternative reputable source clearly labeled as non-xbox.com. Partial credit if only one of the two fields is provided without explaining why the other is missing, or if sourcing is unclear.

Description Identify the relevant gaming studio for the chosen game and name the CEO or studio head (or closest reasonable equivalent if there is no single clear leader), plus their birth date/year. Full credit if the choice of leader is justified when ambiguous (e.g., co-heads, division president vs. studio head) and the birth information is attributed to a reputable source. Partial credit if the leader is plausible but birth info is missing, or if birth year is given without credible attribution. Full credit if the agent explains that no verifiable birth info is publicly available after reasonable effort and proceeds with year-only or an alternative clearly labeled approach.

Description Correctly compute elapsed years between the studio head’s birth date/year and the game’s release date. Full credit if the computation is consistent with the level of date precision available (e.g., uses exact date-boundary logic when full dates are known; uses year-difference with an explicit note about uncertainty when only years are known). Partial credit if the arithmetic is roughly correct but ignores date-boundary logic despite having full dates, or if uncertainty is not acknowledged when only partial dates are available.

Description All reported facts should be internally consistent and supported by the stated sources (xbox.com where available; otherwise clearly labeled alternates). The agent should not fabricate titles, dates, publishers, or biographical details. Full credit if citations/attribution are clear enough to distinguish xbox.com-derived facts from external facts. Partial credit if attribution is somewhat unclear but facts are likely correct; no credit if key claims are invented or contradict the agent’s described evidence.

Description Navigate to careers.microsoft.com and perform a search for the job title 'Applied Scientist' with the location set to Redmond, WA. Partial credit if the correct site is accessed but location or title filter is incomplete or incorrect.

Description From the search results, clearly identify which job posting appears first. Partial credit if multiple results are listed but the first one is not singled out.

Description Open the first job posting and extract the explicit team or group name associated with the Applied Scientist role. Partial credit if a related name is provided but not exactly as stated in the posting.

Description Conduct an external search (beyond careers.microsoft.com) to determine the function, objectives, or main activities of the extracted team or group. Partial credit if general team purpose is given but key activities are missing.

Description Conduct an external search to find the name of the leader (e.g., manager, director, head) of the identified team or group. Partial credit if a related leadership role is found but the exact leader’s name is not.

Description Navigate to careers.microsoft.com and perform a job search for the role 'Applied Scientist' with the location set to Redmond, WA. Partial credit may be awarded for accessing the site or initiating a search without correctly applying the location filter. Full credit should also be awarded if the site is inaccessible or has no such search functionality and the agent clearly states this.

Description From the returned search results, correctly identify the first (top-most) result and access/open its job posting details. Partial credit may be given for listing results but not confirming/opening the first one. Full credit should be awarded if there are no results and the agent explicitly indicates that.

Description From the job posting of the first result, find and report the team or group name the position is for. Partial credit may be awarded for extracting related organizational information (e.g., division/organization) if a specific team/group name is not present. Full credit should be awarded if the posting does not provide a team/group name and the agent clearly indicates this.

Description Use external sources (outside careers.microsoft.com) to find and summarize the group's function, mission, or scope of work. Partial credit may be given for citing sources but providing limited or incomplete information. Full credit should be awarded if credible sources are checked and the information is unavailable, provided the agent clearly states that.

Description Find the leader of the group (name and role/title) from external sources. Partial credit may be awarded for identifying either the leader's name or role, or for identifying leadership at a higher organizational level if the specific group's leader is not publicly available. Full credit should be awarded if no reliable information exists and the agent clearly indicates that.

Description Attempt to use careers.microsoft.com to search for the keyword/title 'applied scientist' and constrain location to Redmond, WA (or the closest available location filter if Redmond is not offered). Full credit if the agent makes a reasonable attempt but is blocked by uncontrollable factors (CAPTCHA/login wall/outage) and clearly reports the blocker and what was attempted. Partial credit if the agent searches without a location constraint or uses an incorrect primary keyword/title when the site is accessible.

Description From the search results page produced by the attempted query, select/open the first job result shown and clearly identify it as the first listing at the time of search (e.g., by position in list). Full credit if no results appear for the exact constraints and the agent clearly reports that and then proceeds with the closest alternative that preserves primary intent (e.g., Applied Scientist in Greater Seattle/WA/nearby, or removing radius constraint), while stating the deviation. Partial credit if the agent opens a non-first result despite first being available and no justification is given.

Description Accurately extract and report the team or group name as stated in the first job posting. Full credit if the team/group name is explicitly present and is quoted or clearly attributed to the posting. Full credit (uncontrollable) if the posting does not specify a team/group name (after checking typical sections like header/summary/org/Responsibilities/Qualifications) and the agent clearly states that limitation and, if present, reports the closest higher-level org named in the posting (e.g., division). Partial credit if the agent provides only an inferred/guessed team name when the posting provides clearer org/team wording.

Description Use at least one external (non-careers.microsoft.com) source to research what the identified group/team does and provide a concise description consistent with the source(s). Full credit if reputable sources are used (e.g., Microsoft official pages/blogs, reputable news, conference talks, LinkedIn org pages). Full credit (uncontrollable) if the group is not publicly described, sources are inaccessible (paywall/blocked), or only the parent org is findable; in that case, the agent should clearly report the limitation and summarize the closest verifiable parent-org function without inventing details. Partial credit if the description is overly generic or weakly sourced while better public info is readily available.

Description Identify the group's leader (e.g., Head/Director/GM/VP) via external research and report the name with evidence/attribution (e.g., Microsoft leadership page, press release, LinkedIn profile showing the role). Full credit if a specific leader for the identified group is found and attributed. Partial credit if only leadership at a broader parent-org level is identified and the agent clearly states the scope mismatch. Full credit (uncontrollable) if no verifiable leader information is publicly available for the group (or sources are inaccessible) and the agent clearly reports that after reasonable search, without guessing.

Description Determine which show will be presented next at the Infinity Theater in the Denver Museum of Nature and Science. Partial credit if the agent finds a list of upcoming shows but not the very next one; full credit only if the correct next show is named.

Description Locate and state the name of the producer responsible for the next Infinity Theater show. Partial credit if a list of multiple people involved is given but the producer’s name is not clearly identified; full credit if the producer is correctly named.

Description Provide the titles of up to three additional films or movies produced by the same producer. Partial credit for fewer than three accurate titles; full credit for up to three correct titles.

Description Find and clearly state the title of the next scheduled show at the Infinity Theater at the Denver Museum of Nature & Science. Partial credit may be awarded if multiple upcoming shows are listed without clearly identifying the next one, or if the schedule cannot be determined but the agent explains the limitation and provides the closest available information.

Description Determine and report who the producer is for the identified show. Partial credit may be awarded if the agent provides a plausible producer (e.g., a production company or named producer) when multiple producers exist, or acknowledges when producer information cannot be confirmed from authoritative sources.

Description Provide the names of up to three other films or movies produced by the same producer identified for the show. Partial credit may be awarded for fewer than three valid titles or for explaining if such information cannot be reliably confirmed.

Description Attempt to access the Denver Museum of Nature & Science Infinity Theater schedule/listings (via the museum site or clearly identified official DMNS channels). Full credit if the agent reaches the schedule/listing OR clearly reports an uncontrollable blocker (site down, CAPTCHA, geo-blocking, broken page) and describes what was attempted. Partial credit if the agent uses an unrelated/unauthoritative source without attempting DMNS/official listings first.

Description Determine the next upcoming Infinity Theater show as presented by the accessible schedule/listings, and report the title plus the next listed date/time (or the earliest showtime shown). Full credit if the agent correctly identifies the next upcoming show with its corresponding next showtime/date when available. Also full credit if the schedule is ambiguous (e.g., multiple formats/filters, multiple films with the same earliest showtime, or only recurring daily times without a clear 'next') and the agent explains the ambiguity and selects a defensible 'next' based on the earliest time/date shown. Partial credit if the title is provided but the 'next' ordering is not established when it could have been, or if showtime/date is omitted despite being clearly shown.

Description Find and report the producer (person or production company, as credited) for the identified next Infinity Theater show, citing/grounding it in an authoritative source (DMNS listing or official film credits page). Full credit if the producer credit is correctly extracted, OR if producer credit is not available on accessible authoritative sources / sources are blocked and the agent clearly reports where they looked and that the producer could not be confirmed. Partial credit if a producer is given without clear linkage to the specific show or if the agent likely confuses producer with director/narrator when clearer credits were available.

Description Provide 1–3 other film/movie titles that the identified producer has produced, grounded in reliable filmography/credits sources. Full credit for 1–3 correctly attributed titles, OR full credit if no additional producer credits can be found due to unavailable/blocked sources or the producer appears to have no other producership credits and the agent reports this after reasonable search. Partial credit if titles are not clearly verified as producer credits (e.g., other roles only) when better verification was feasible.

Description Provide the complete roster of NSYNC members. Partial credit may be given if only some members are listed; full credit requires all official members.

Description Provide the complete roster of Backstreet Boys members. Partial credit may be given if only some members are listed; full credit requires all official members.

Description Compare the last names of all listed members and clearly identify which member has the longest last name. Partial credit if the comparison is shown but the correct member is not named.

Description Research and state the net worth of the member with the longest last name. Partial credit if an approximate range is given; full credit for a sourced figure.

Description Provide a complete and correct list of all members of the band NSYNC. Partial credit may be awarded if most members are listed but one or more are missing or incorrect.

Description Provide a complete and correct list of all members of the band Backstreet Boys. Partial credit may be awarded if most members are listed but one or more are missing or incorrect.

Description From the combined lists, correctly determine which single member has the longest last name. Partial credit may be awarded if there is a reasonable attempt (e.g., correctly handling ties or minor miscounts).

Description Provide a net worth figure for the correctly identified member with the longest last name. Partial credit may be awarded if a net worth is given but the member was misidentified or if the figure lacks precision.

Description Provide a complete list of all official members of the band NSYNC. Full credit if all members are listed (Joey Fatone, Justin Timberlake, JC Chasez, Chris Kirkpatrick, Lance Bass). Partial credit if some members are listed but at least one is missing or if a non-member is incorrectly included. No credit if the band’s members are largely incorrect or the wrong group is listed.

Description Provide a complete list of all official members of the band Backstreet Boys. Full credit if all members are listed (AJ McLean, Howie Dorough, Nick Carter, Kevin Richardson, Brian Littrell). Partial credit if some members are listed but at least one is missing or if a non-member is incorrectly included. No credit if the band’s members are largely incorrect or the wrong group is listed.

Description Determine which individual (from both bands’ member lists) has the longest last name (by number of letters). Full credit if the correct person is identified and the comparison set is clearly the members of both bands. Partial credit if a plausible candidate is chosen but the method is unclear, ties are mishandled, or the comparison appears incomplete. No credit if the identified person is not in either band or is clearly not the longest last name given the provided names.

Description Provide a net worth estimate for the identified member with the longest last name. Because net worth is externally dependent and varies by source/date, full credit if the agent (a) reports a reasonable net worth figure or a small range for the correct person and (b) indicates the estimate’s source and/or that figures differ across sources (or that the value is approximate/as of a given year). Also award full credit if the agent clearly explains it cannot reliably verify a net worth figure due to unavailable/inaccessible sources but provides the best available estimate or states that no reliable figure could be found. Partial credit if a net worth figure is provided but the person is wrong, or if the figure is ambiguous (e.g., missing currency/context) while still clearly intended as net worth. No credit if no net worth is provided and no clear attempt/limitation is communicated, or if the value is clearly unrelated (e.g., salary, revenue, or another person’s net worth).

Description Navigate to the Zara website and successfully find the section or filter for women's clothes on sale.

Description Select or note the first item in the sale results that is specifically marked down.

Description List all materials the selected item is composed of and correctly identify the primary material based on proportion.

Description Research and report the temperature at which the primary material ignites.

Description Find and access Zara's women's sale items (e.g., via Zara website or search results that show women's items on sale). Partial credit if Zara is accessed but the sale section or women's sale items are not clearly reached.

Description Identify and choose the first result that is explicitly marked down from the women's sale items. Partial credit if an on-sale women's item is selected but not the first, or if the selection is not clearly marked down. Full credit if the agent notes that no marked-down items are available and states this clearly.

Description Find and report the materials the selected item is composed of as listed on the product page (ideally including percentages if shown). Partial credit for incomplete or partially accurate composition; full credit if the agent notes that composition is unavailable on the page.

Description Determine the primary material (e.g., the highest-percentage component) from the composition and state its ignition (autoignition) temperature with units. Partial credit for providing a plausible temperature range or closely related thermal property with explanation, or for explaining unavailability of ignition data.

Description Attempt to navigate Zara (zara.com or a regional Zara site) to a women's sale/discounted/marked-down product listing. Full credit if the agent reaches the relevant listing OR if Zara is inaccessible (CAPTCHA, region block, site down, requires app/login) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent browses Zara women’s items but cannot establish any sale/markdown context and does not clearly explain why.

Description From the women's sale/markdown results that are actually visible to the agent, select the first item showing an explicit reduction (e.g., reduced price, struck-through original price, discount label). Full credit if the agent either (a) selects the first visible marked-down item, or (b) explains why the “first” ordering cannot be reliably determined (dynamic sorting/infinite scroll/personalization) and selects the earliest marked-down item they can verify. Partial credit if a marked-down item is chosen but the agent provides insufficient evidence that it was first/earliest among visible markdowns when that ordering is clearly viewable.

Description Open the selected product’s details and extract the material composition as listed by Zara, including percentages when available. Full credit if the composition is provided with fiber names and percentages OR if Zara does not display composition (hidden behind unavailable accordion, blocked scripts, geo/app gating) and the agent clearly reports the limitation and where they looked. Partial credit if fiber types are provided but percentages are omitted despite being clearly available.

Description Identify the primary material as the highest-percentage fiber from the reported composition (or, if multiple components are separately listed and no single overall percentage is determinable, choose a defensible primary component and explain). Provide the ignition temperature for that material with units and attribution to a credible reference; a reasonable range is acceptable if sources vary or if the reference reports a range. Full credit if the primary material identification is consistent with the composition and the ignition temperature is plausibly sourced/attributed; if ignition temperature cannot be determined (e.g., composition unknown due to Zara gating), full credit for clearly stating that dependency and not fabricating a value. Partial credit if the primary material is correct but the ignition temperature lacks units and/or lacks any attribution.

Description Locate Amazon's pantry staples best-selling list and correctly determine which product is ranked third. Partial credit can be given for selecting a pantry staple but with an incorrect rank or for identifying the correct item but mislabeling its rank.

Description On AllRecipes, search for and select a recipe that explicitly includes the identified pantry staple item as an ingredient. Partial credit for choosing a recipe that uses a closely related ingredient or for failing to verify that the identified item appears in the ingredient list.

Description Provide the exact recipe title and reproduce the complete list of ingredients from the selected AllRecipes recipe. Partial credit if either the recipe name or some ingredients are missing or incorrect.

Description Find and name the item that is ranked #3 in Amazon's Best Sellers for the Pantry Staples category. Full credit requires clearly identifying the #3 item from Amazon within the Pantry Staples category. Partial credit may be awarded if a top seller is identified but not the #3 item, or if the category is correct but the rank/item is ambiguous.

Description Locate a recipe on AllRecipes that lists the identified pantry staple item as one of its ingredients. Full credit requires that the recipe is from AllRecipes and explicitly includes the identified item (or an unambiguous equivalent) in its ingredients. Partial credit may be awarded if the recipe is on AllRecipes but the ingredient match is unclear or slightly off.

Description Provide the recipe name and the complete ingredients list as presented on AllRecipes. Full credit requires both the recipe name and the entire ingredients list (not a subset). Partial credit may be awarded if only the name is given or if the ingredients list appears incomplete.

Description Attempt to navigate Amazon’s Best Sellers page(s) for a pantry staples/grocery/pantry category and locate the visible ranking list. Full credit if the agent makes a reasonable attempt but is blocked (CAPTCHA/login/region restriction), the page is unavailable, or rankings cannot be viewed, and it clearly reports what was attempted and the blocker. Partial credit if the agent uses an unrelated Amazon page or provides no evidence of attempting to view a Best Sellers ranking.

Description Determine and report the product shown as rank #3 on Amazon Best Sellers within the chosen pantry staples/grocery pantry category at the time of access, with enough detail to uniquely identify it (e.g., full product name/brand/size). Full credit if #3 is clearly identified (or if Amazon rankings are inaccessible and this is already documented under the access criterion, with no further penalty here). Partial credit if a plausible best-seller is provided but rank #3 is not verified, the category is unclear, or the product details are insufficient to uniquely identify the item. If rankings appear inconsistent due to region/personalization/ties/rapid changes, full credit if the agent states this uncertainty and reports what was observed (including timestamp/context) and still provides the best-supported #3 item.

Description Attempt to use AllRecipes to find a recipe whose ingredient list includes the identified Amazon item’s underlying ingredient (recognizing that recipes typically list generic ingredients rather than brand/SKU). Full credit if the agent attempts AllRecipes but is blocked, the site is down, or ingredient lists cannot be accessed, and it clearly reports the blocker and attempts. Partial credit if the agent does not use AllRecipes and does not report an access issue.

Description Select an AllRecipes recipe where the ingredient list explicitly includes the identified Amazon item or an unmistakable equivalent ingredient name (e.g., Amazon product is 'canned chickpeas' and recipe lists 'garbanzo beans/chickpeas'). Full credit if the ingredient match is explicit on the AllRecipes page, or if no such AllRecipes recipe can be found after reasonable search attempts and the agent clearly reports that outcome (optionally providing the closest match on AllRecipes). Partial credit if the recipe is not from AllRecipes when AllRecipes is accessible, or if the ingredient match is ambiguous/unsupported when clearer matches are available.

Description Provide the exact title/name of the selected AllRecipes recipe (or clearly state that no qualifying recipe could be accessed/found on AllRecipes). Partial credit if the name is incomplete but still uniquely identifies the recipe.

Description Provide the complete ingredients list as shown on the AllRecipes recipe, including quantities and descriptors/qualifiers. Full credit if all ingredients are included; partial credit for minor omissions while most of the list is correct. If the AllRecipes page/ingredient list cannot be accessed due to external blockers already documented, full credit if the agent clearly reports that it could not retrieve the ingredients list (and does not fabricate).

Description Navigate to the universityofcalifornia.edu news section or archive, sort articles by date, and identify the earliest published article. Partial credit if the article is found but not confirmed as the earliest.

Description Accurately extract the author's name for the identified first news article. Partial credit for partially correctly identifying the author (e.g., surname only).

Description Find and list two additional news articles on universityofcalifornia.edu written by the same author. Partial credit for finding one correct article or listing two but with ambiguous authorship.

Description Ensure the search and selection for the first news article is limited to content hosted on universityofcalifornia.edu (including its subdomains). Partial credit may be awarded if the agent attempts to use the specified domain but is not fully consistent.

Description Identify the earliest (i.e., first) news article published on the specified domain. Full credit for clearly identifying the article and supporting why it is the first (e.g., via publication date or archive ordering). Partial credit if the agent provides a plausible earliest article from available archives or notes limitations preventing definitive identification.

Description Extract and state the author's name from the identified first news article, as needed to satisfy the next step. Partial credit if the author is not listed and the agent explicitly notes the absence or uses the byline appropriately.

Description List two distinct other articles authored by the same person identified from the first article. Full credit requires two correct, clearly identified articles (titles; links optional) that are different from the first article. Partial credit for only one correct article or if attribution is unclear but plausibly the same author.

Description Navigate to universityofcalifornia.edu and attempt to access a news section/landing page and an archive, listing, or search experience that surfaces news articles. Full credit if the agent clearly attempts access but is blocked (e.g., CAPTCHA, paywall/login, site down) or if the archive/listing function is inaccessible, and the agent explicitly reports the blocker. Partial credit if the agent uses an unclear/incorrect section of the domain (not news) but demonstrates reasonable effort to find a news listing.

Description Find and report the earliest (first chronologically published) news article available on universityofcalifornia.edu, providing at least title and publication date (URL optional). Full credit if the agent correctly identifies the earliest article and provides identifying details, OR if the agent explains why definitive verification is not possible due to site limitations (e.g., no oldest-sort, incomplete archive, inconsistent dates) and instead provides the best-supported earliest article they can find along with the method/evidence used (e.g., oldest reachable page, earliest search result with date). Partial credit if an early article is provided but the effort to determine/justify it as earliest (or best-supported earliest) is weak or unclear. No credit if the item is not on universityofcalifornia.edu or is not a news article.

Description Using the author of the first identified article, find two other articles by that same author, preferably on universityofcalifornia.edu, and provide at least their titles (dates/URLs optional). Full credit if both additional articles are clearly attributed to the same author, OR if author discovery is impeded by external constraints (missing/variable bylines, absent author page, site search limitations) and the agent documents reasonable attempts (e.g., searching the domain for the author name, using an author tag page if present) and reports the best available author-matched results or clearly states that fewer than two could be verified. Partial credit if only one additional verified article is found or if one of the two has unclear attribution despite reasonable effort. No credit if the additional articles are not by the same author or are off-domain without a clearly stated, justified blocker.

Description Locate and correctly identify the exhibit at the George H.W. Bush Library that has the nearest start date after the current date, using an authoritative source such as the library's official website.

Description List the exact start and end dates for the identified upcoming exhibit, ensuring accuracy and clarity.

Description Consult a reliable astronomical source (e.g., NASA eclipse data) to check whether any total solar eclipse falls within the exhibit’s availability dates, and state the finding clearly.

Description Find and correctly name the exhibit that is scheduled to open next at the specified library. Full credit if it is clearly the next upcoming exhibit relative to the current date, or if none are upcoming and the agent explicitly states that. Partial credit if an exhibit is identified at the correct institution but it is not the next upcoming or the status is unclear.

Description State the exhibit's start and end dates during which it will be available. Full credit for both precise start and end dates; partial credit for providing only one of the dates or an approximate timeframe. If dates are not available, full credit for clearly stating that and explaining the limitation.

Description Determine if a total solar eclipse takes place at any time between the exhibit's start and end dates, and clearly state yes or no (with a brief explanation if applicable). Partial credit for addressing eclipses but with incomplete timeframe consideration or uncertainty noted.

Description Determine the next upcoming (soonest not-yet-started) exhibit at the George H.W. Bush Presidential Library & Museum using authoritative sources (official library website pages, official announcements, or equivalent). Full credit if the agent correctly identifies the exhibit title and clearly ties it to the Bush Library, or if official information is unavailable/unclear (e.g., site down, CAPTCHA, conflicting listings, no “upcoming” exhibits posted) and the agent clearly reports that limitation and what it checked. Partial credit if an exhibit is identified but “next/upcoming” status is not well-justified when multiple future exhibits are listed.

Description Provide the exhibit’s availability date range (opening/start date and closing/end date) as shown by an authoritative source. Full credit for clearly stating both dates; OR, if the official source does not list an end date (or lists it as TBD/ongoing), full credit for reporting the known start date and explicitly noting that the end date is not announced/unknown. Partial credit if only one date is provided without clarifying whether the other is unavailable vs. omitted, or if dates are ambiguous but reasonably inferable.

Description Using the exhibit availability window (inclusive) and a reliable eclipse source (e.g., NASA or equivalent), determine whether any total solar eclipse occurs at any time within that interval. Full credit for a correct yes/no conclusion clearly tied to the date window; OR, if authoritative eclipse data cannot be accessed due to external issues (blocked sources, downtime), full credit for clearly reporting the access limitation and the best-effort reasoning/attempt. Partial credit if eclipse information is mentioned but overlap with the exhibit dates is not clearly evaluated.

Description List the date(s) of any total solar eclipse(s) that fall within the exhibit date range. Full credit for correct eclipse date(s). Partial credit if an eclipse date is provided but the eclipse type is wrong (not total) or the date is slightly mis-scoped while still attempting to match the exhibit interval. If eclipse-date sources are inaccessible, the agent should not be penalized provided it clearly reports the limitation after a reasonable attempt.

Description Successfully navigate to or access the Yelp.com website.

Description Find a vegetarian restaurant in San Francisco on Yelp with a rating of at least 4.5 and at least 100 reviews. Partial credit for meeting some but not all criteria (e.g., correct location and type but lower rating or fewer reviews).

Description Accurately obtain the full street address of the selected restaurant as listed on Yelp.

Description Successfully navigate to or access the Rentalcars.com website.

Description Input the restaurant's address as the pickup location on Rentalcars.com.

Description Select November 15, 2025 as the pick-up date and November 18, 2025 as the drop-off date on Rentalcars.com.

Description Search for and identify a compact car option available for the specified dates, prioritizing proximity to the entered pickup address.

Description Use Yelp.com to identify a vegetarian restaurant located in San Francisco that meets BOTH thresholds: rating ≥ 4.5 and ≥ 100 reviews. Extract the restaurant’s street address to use for the car rental step. Partial credit may be awarded if the agent finds a vegetarian restaurant but misses one of the thresholds, finds one in SF but does not provide the address, or attempts on Yelp but cannot confirm the criteria. Full credit should be awarded if no such restaurant exists and the agent clearly indicates that unavailability.

Description On Rentalcars.com, set up a search using the restaurant’s address as the pickup location (or the nearest available pickup point determined from that address) for the dates November 15 to November 18, 2025. Partial credit may be awarded for using a nearby location (e.g., city center) instead of the precise address, or for setting up the search but with incorrect dates. Full credit should be awarded if results cannot be shown for that address and the agent clearly indicates the limitation while attempting a reasonable nearest location.

Description From the search results, choose a compact car option at the nearest available pickup location to the restaurant’s address and proceed up to the booking/reservation stage (e.g., viewing deal or proceeding toward customer details) without entering personal information or completing the booking. Partial credit may be awarded for selecting a non-compact car, choosing a not-closest location, or failing to proceed to the booking stage. Full credit should be awarded if no compact cars are available at the nearest location and the agent clearly indicates that unavailability.

Description Find a vegetarian (or clearly vegetarian/vegan-focused) restaurant located in San Francisco. Full credit if the agent identifies a specific restaurant and, from a reasonable source, verifies BOTH: rating ≥4.5 and review count ≥100. Also award full credit if, after reasonable search/verification attempts, the agent clearly reports that it cannot confirm both thresholds from available sources or that no visible results meet both constraints, and then selects the best available highly rated/popular vegetarian alternative consistent with the task’s primary intent. Partial credit if the restaurant is vegetarian and in San Francisco but only one threshold is verified or the verification is unclear. No credit if the restaurant is not vegetarian/veg-focused, not in San Francisco, or clearly fails thresholds when qualifying options are readily available.

Description Obtain the restaurant’s full street address (or the most precise address available from sources). Full credit if the address is clearly captured and then used to anchor the rental search, either by entering the address directly on Rentalcars.com OR by selecting the nearest unambiguous pickup area/location derived from that address (e.g., closest downtown/rail/hotel/landmark option shown by the site) when exact address entry is not supported. Partial credit if only a partial address/neighborhood is used but the linkage to the restaurant location is clear. No credit if the address is missing or the rental search is anchored to an unrelated/incorrect location without justification.

Description Attempt the workflow on Rentalcars.com using the restaurant address (or nearest derived pickup location) as the pickup anchor. Full credit if the agent reaches search results OR if Rentalcars.com is inaccessible (CAPTCHA, outage, blocking, geo restrictions) and the agent clearly reports the blocker after attempting. Partial credit if the attempt is made but the pickup location used is broadly in San Francisco without being clearly tied to the restaurant area. No credit if Rentalcars.com is not attempted when accessible or the search is for an unrelated city.

Description From Rentalcars.com results, filter for or select the compact car category (or the closest equivalent category label available on the site, if “compact” is not offered). Full credit if compact (or nearest equivalent) is clearly selected/shown in results, OR if results do not offer a compact category and the agent clearly reports this and selects the closest available small-car alternative consistent with intent. Partial credit if nearby rental options are found but car category is not clearly compact/equivalent. No credit if a clearly non-compact category is chosen without explanation when compact/equivalent is available.

Description Enter/select pickup date Dec 15, 2025 and drop-off date Dec 18, 2025 on Rentalcars.com. Full credit if both dates are correctly applied in the search/results/booking flow, OR if the platform prevents selecting those dates (e.g., limited inventory horizon, site error) and the agent clearly reports the limitation encountered. Partial credit if only one date is correct or if the year is ambiguous but intent is clear and the agent attempts correction. No credit if dates are clearly wrong when correct entry was possible.

Description Select a specific compact (or compact-equivalent) offer at the nearest feasible pickup location and proceed to a details/pre-checkout page. Stop before entering personal details, creating an account, logging in, or providing payment information, and do not finalize a reservation. Full credit if the agent reaches a pre-checkout/details stage and stops appropriately, OR if the site requires login/personal details/payment earlier than expected and the agent stops and reports that constraint. Partial credit if an offer is selected but the agent does not meaningfully progress toward the booking details page. No credit if the agent completes a binding booking/checkout or enters fabricated personal/payment details.

Description Agent should identify and provide the link or clear reference to the exact Reddit thread in r/golf discussing that golf courses take up 3000 square miles in the USA. Partial credit if a closely related post is found but details (subreddit or figures) are incorrect.

Description Agent should extract the highest-voted comment from that post and produce an accurate, concise summary of its key points. Partial credit for capturing some but not all major points or misrepresenting the comment.

Description Agent should find another reputable website or source that supports at least one central claim made in the top comment (e.g., land area statistics, environmental impact). Partial credit if source is marginally relevant or less authoritative.

Description Successfully locate and identify a post in r/golf that discusses golf courses taking up "3000 sq miles" of land in the USA. Partial credit may be awarded if a closely related r/golf post is found that addresses similar land usage but does not explicitly mention the "3000 sq miles" figure, or if the subreddit or topic is slightly off yet relevant.

Description Correctly identify the top upvoted comment on the located post and provide a concise, accurate summary of its key points. Partial credit may be given if a comment is summarized but it is not the top upvoted one, or if the summary captures some but not all major points of the comment.

Description Find another website that credibly and directly supports at least one major claim made in the top comment, and make clear how it substantiates the claim. Partial credit may be awarded if the external source is relevant but does not directly corroborate the claim, or if the source's credibility is weak.

Description Attempt to use Reddit (and specifically r/golf) search/browsing to locate a post explicitly referencing that US golf courses take up about "3000 sq miles". Full credit if the agent demonstrates a reasonable attempt but is blocked by Reddit (captcha/login/region/app restrictions) or the post appears deleted/unavailable and the agent clearly reports this limitation. Partial credit if the agent searches but in the wrong subreddit or without targeting the key phrase/number.

Description Full credit if the agent identifies the correct r/golf post and provides sufficient evidence (title/context/quote showing the "3000 sq miles" mention). Full credit also if, after reasonable searching, the agent clearly reports that the exact post cannot be found (e.g., deleted, not indexed, search limitations) and documents what was tried. Partial credit if a similar post is found but the explicit phrase/number is missing or the post is outside r/golf.

Description Provide a summary of the single top upvoted comment for that post. Full credit if the summary captures the major points of that specific top comment. If comments/top sort are inaccessible (deleted thread, locked, blocked by Reddit UI/access limits), full credit if the agent clearly reports the limitation and why the top comment cannot be confirmed. Partial credit if the agent summarizes a non-top comment despite the top comment being visible.

Description Extract at least one major, checkable claim stated in the top upvoted comment (e.g., a statistic, land-use comparison, water-use claim, number of courses, economic claim). Full credit if the claim is clearly stated and attributable to the top comment. If the top comment is not accessible/confirmable, award full credit for explicitly stating that no attributable claim can be extracted.

Description Provide an independent (non-Reddit) website/source that directly substantiates at least one identified major claim from the top comment. Full credit if the source is relevant and supports the claim with data or credible reporting. Full credit also if the agent reports that no substantiating source can be found after reasonable effort and explains why (e.g., claim appears unsourced/unverifiable). Partial credit if the source is only loosely related or does not actually support the specific claim.

Description Ensure the reported post context, the top comment summary, and the external substantiation accurately reflect what is actually visible in the cited sources. Full credit if nothing is fabricated/misattributed. Low/no credit if the agent invents a post/comment, misstates what the external source says, or implies verification that was not possible due to access limits.

Description Successfully navigate to or access the Eventbrite.com website as specified in the task.

Description Locate an art exhibition listing on Eventbrite for Portland occurring within the current calendar month. Partial credit if exhibition is in Portland but outside this month or if date information is unclear.

Description Extract and report the precise date and the venue name/address of the identified art exhibition.

Description Navigate to or access the Google Flights website as specified in the task.

Description Perform a search for a round-trip flight departing Seattle (SEA) and returning the same day from Portland (PDX).

Description Determine and report the lowest-priced same-day round-trip flight option, including departure time, return time, and fare. Partial credit if missing one of these elements or if a slightly more expensive flight is chosen.

Description Ensure the process ends before initiating any booking or entering personal/payment information, in compliance with the instruction to complete the task before purchase.

Description Navigate to Eventbrite.com and locate at least one event that is explicitly an art exhibition in Portland, scheduled for the current calendar month. Partial credit may be awarded if an event is found but it is not clearly an 'art exhibition', not in Portland, or not within this month. Full credit should still be awarded if there are no qualifying events this month and the agent clearly states that after checking Eventbrite.

Description Provide the exact event date (day and month) and the venue name for the chosen art exhibition. Partial credit may be awarded if only one of date or venue is provided, or if the date is imprecise (e.g., only a date range without specifying a specific date when applicable).

Description Access Google Flights, set origin to Seattle (SEA) and destination to Portland (PDX), configure a same-day round-trip (depart and return on the same calendar day), and identify the cheapest available option including the price. Partial credit may be awarded if the agent sets the route correctly but does not ensure same-day round-trip, or identifies options without confirming the cheapest. Full credit should be awarded if no same-day round-trip options are available and the agent explicitly states that after checking.

Description Complete the research and identification steps without initiating a purchase or entering any personal information. The agent should clearly stop before any checkout or booking confirmation.

Description Use Eventbrite.com to perform a search for events in Portland, OR within Portland, Oregon, and apply reasonable query terms/filters to target art exhibitions occurring within the current calendar month. Full credit if the agent attempts Eventbrite but is blocked (CAPTCHA/login hard block), the site is down, or results cannot be loaded, and the agent clearly reports the blocker and inability to verify listings. Partial credit if the search is conducted but the location/month constraint is applied incorrectly or only loosely (e.g., Portland metro without clear Portland, or a wider date range without checking this month).

Description From Eventbrite search results, identify at least one event that is explicitly an art exhibition, located in Portland (or clearly described as Portland, OR), and scheduled within the current calendar month. Full credit if an eligible listing is found; OR if none are available that meet all constraints and the agent clearly states that no exact match was found after reasonable checking, optionally providing the closest alternative that preserves the primary intent (art-focused event in Portland this month) while noting which constraint(s) were not met. Partial credit if the selected event is art-related but not clearly an exhibition, or is in the broader area but not clearly Portland when better matches are visible.

Description Open the chosen Eventbrite event page and extract (1) the exact event date as stated and (2) the venue/location name. Full credit for both, unambiguous. If the page does not provide a specific single date (e.g., recurring/multi-date series) or the venue is missing/online-only/TBA, full credit if the agent accurately reports what is shown (e.g., date range/recurrence details and the listed location status) and states that an exact single date or venue name is not available on the listing. Partial credit if only one of date/venue is provided when both are clearly shown.

Description Use Google Flights to search flights from Seattle (SEA) to Portland (PDX) with round-trip selected and departure/return on the same calendar day as the chosen event date (or, if the event date is not a single specific date, choose a reasonable specific date within the event’s stated schedule and explicitly note the assumption). Full credit if the agent attempts to use Google Flights but is blocked (CAPTCHA/login hard block), the site is down, or results cannot be loaded, and the agent clearly reports the blocker. Partial credit if the correct route is searched but same-day constraint is not applied or date selection is unclear.

Description From the Google Flights results, identify the cheapest itinerary that satisfies same-day round-trip SEA↔PDX. Report the total price and enough itinerary detail to substantiate the choice (e.g., airlines and departure/return times). Full credit if the cheapest visible option is correctly identified; OR if no same-day round-trip options exist for that date and the agent clearly reports this (optionally noting the closest alternatives such as next-day return, different nearby airports, or different date within the event schedule). Full credit if prices cannot be verified due to blocking/unstable results and the agent clearly states this. Partial credit if an option is provided but is not the cheapest when a cheaper same-day option is visibly available.

Description The agent must stop after reporting the Eventbrite event date/venue and the cheapest same-day round-trip flight option, without completing any purchase/booking or submitting personal or payment information on Eventbrite, Google Flights, airline sites, or third-party booking sites. Full credit if it clearly stops before checkout/payment. Partial credit if it navigates near checkout but does not submit information. No credit if it completes a purchase/booking or submits personal/payment credentials.

Description Navigate to Google Flights and enter origin as Edinburgh (EDI), destination as Manchester (MAN), and date as December 28, 2025. Partial credit if only some parameters are correct or if a different date within one day is used.

Description From the search results, find and record the single cheapest one-way flight option. Partial credit if a flight is recorded but not the cheapest or if multiple flights are listed without identifying the least expensive.

Description Locate and record the specific aircraft model used for the least expensive flight. Partial credit if an approximate family (e.g., 'A320 series') is given instead of the exact model.

Description Research and report the standard all-economy seating capacity of the flight’s aircraft type. Partial credit if capacity ranges are given without a clear value.

Description Research and report the usual seating capacity for an all-economy Boeing 747-8. Partial credit if a credible range is provided but no single figure.

Description Compute and state how many fewer passengers the flight’s aircraft can carry compared to the 747-8 all-economy capacity. Partial credit if the calculation is shown but arithmetic error occurs.

Description Access Google Flights and perform a search for a one-way flight from Edinburgh (EDI) to Manchester (MAN) on December 28, 2025. Partial credit if Google Flights is used but one parameter (origin, destination, date, or trip type) is incorrect.

Description From the search results, determine the lowest-price one-way option for EDI→MAN on the specified date and record the flight details (price and flight identification such as airline and, if available, flight number/time). Full credit includes noting if no flights are available and stating that explicitly. Partial credit if a flight is recorded but it's not clearly the cheapest or missing key details.

Description Provide the aircraft model/type used for the specific flight identified. Partial credit if only a general aircraft family is given or if type cannot be confirmed but the limitation is clearly stated.

Description Compute how many fewer passengers the identified aircraft type can carry compared to a Boeing 747-8 (passenger variant) in an all-economy configuration, and report the numeric difference. Partial credit if capacities are provided without the difference or if there is a small calculation error. Full credit may be awarded if necessary capacity information is unavailable and the limitation is clearly explained.

Description Attempt to use Google Flights to search Edinburgh (EDI) → Manchester (MAN), one-way, on December 28, 2025. Full credit if the agent performs the correct search OR clearly reports an uncontrollable blocker (e.g., CAPTCHA, outage, results not loading, pricing unavailable). Partial credit if the agent attempts Google Flights but uses slightly incorrect parameters and corrects/acknowledges the mismatch.

Description From the results the agent can actually view, identify the least expensive one-way option for EDI→MAN on Dec 28, 2025 and record enough identifiers (at minimum: price with currency, airline/flight number or airline + departure time). Full credit if (a) the agent selects a cheapest option among the visible results, including handling ties (any tied-cheapest is acceptable), OR (b) Google Flights provides no priced options and the agent clearly reports that outcome. Partial credit if a plausible cheap option is provided but the agent does not substantiate that it is cheapest among what was visible.

Description Report the aircraft type for the selected cheapest flight. Full credit if the aircraft type is shown directly in Google Flights for that itinerary/flight. If Google Flights does not show aircraft type or it is unavailable for that date, full credit if the agent clearly states this limitation and uses a reliable alternate source tied to the specific flight number/route/date when possible (or labels it as a typical/expected aircraft for that flight/route if only that is possible). Partial credit if an aircraft type is given without clearly tying it to the specific flight option selected.

Description Compute how many fewer passengers the identified aircraft type can carry compared to a 747-8 in an all-economy configuration. Full credit if the agent: (a) states a sourced or explicitly-assumed capacity for the identified aircraft type (noting that capacity varies by configuration), (b) states a sourced or explicitly-assumed 747-8 all-economy capacity (noting that this is not a single universal number), and (c) correctly computes (747-8 capacity − identified aircraft capacity) as 'fewer passengers.' If exact capacities cannot be uniquely determined, full credit for a clearly explained, reasonable assumption with citations and correct arithmetic under those assumptions. Partial credit if arithmetic is correct but one of the capacity assumptions/sources is missing or unclear.

Description Ensure the final reported flight and analysis are within scope (one-way EDI→MAN on Dec 28, 2025) when such results are available, and do not present unverified details (price/aircraft/capacity) as certain. Full credit if the agent either stays within scope or, if blocked by platform/data unavailability, explicitly labels uncertainty and does not fabricate specifics. Partial credit for minor omissions (e.g., missing currency) that do not materially change the result interpretation.

Description Specifies the train station nearest to Central Park in Manhattan as the departure location.

Description Specifies the train station in Miami as the final arrival point of the itinerary.

Description Ensures that every segment of the journey is by train, with no use of other transportation modes.

Description Lists all train legs in sequence, naming departure and arrival stations (and connections) for each segment. Partial credit for incomplete or missing legs.

Description The itinerary explicitly starts from Central Park, Manhattan and ends in Miami, Florida. Partial credit if the plan starts in New York City generally but does not clearly anchor to Central Park.

Description All travel segments use trains only (e.g., subway, commuter rail, intercity rail). No buses, flights, cars, or rideshares are included. Partial credit if incidental walking within stations is mentioned but all travel legs are by train.

Description Provides a clear sequence of train segments from Central Park to Miami, including local rail from the starting area to a major intercity station and the onward intercity train(s), with station names and train services identified. Partial credit for a high-level plan that lacks specific lines or stations.

Description Specifies the arrival point in Miami (e.g., the relevant train station/terminal) and makes clear that the endpoint is within Miami. Partial credit if Miami is stated without a specific arrival station.

Description Itinerary should clearly begin from Central Park in Manhattan (or a nearby appropriate rail access point such as Penn Station/Grand Central/Harlem–125th) and explain a plausible train-only connection from Central Park to the first intercity departure station (e.g., NYC Subway). Full credit if the start is correct and the rail connection is plausible. Partial credit if it starts generally in Manhattan without mentioning Central Park or a reasonable nearby station connection. No credit if it starts outside Manhattan or from an unrelated city.

Description Itinerary should end in Miami proper and specify a Miami-area train arrival station (e.g., Miami Amtrak Station) and/or a train-only last-mile connection if arriving first at a nearby rail station in the Miami metro area. Full credit if it clearly reaches Miami by train. Partial credit if it ends at a nearby metro-area station (e.g., Fort Lauderdale) but includes a train-only continuation to Miami. No credit if it ends in a different city/state or requires non-train transport with no train-only continuation proposed.

Description All legs of the itinerary must use trains only (subway/commuter rail/intercity rail are allowed). Full credit if every segment is train-based. Partial credit if one segment is described using a non-train mode but the agent explicitly flags it and provides a train-only alternative for that segment. No credit if any required leg relies on non-train transport without a train-only alternative.

Description Provide a coherent sequence of train segments from Manhattan to Miami, including key intermediate stations and transfer points (NYC departure station, major transfer city/station if used, and Miami arrival station). Full credit if the route is end-to-end, internally consistent, and transfers are understandable. Partial credit if the route is mostly clear but missing one key station/transfer detail or has minor ambiguity while still being followable. No credit if the itinerary is incomplete or logically incompatible (e.g., missing the intercity portion entirely).

Description Itinerary should rely on real, operational passenger rail services for the corridor and plausible connectivity between segments (e.g., Amtrak services and appropriate local rail). Full credit if services cited are appropriate and the plan is plausible; also award full credit if the agent notes that exact schedules/through-cars can change and advises verifying current timetables, and/or provides a reasonable alternate rail-only routing in case a named service/segment is suspended. Partial credit if there are minor inaccuracies (slightly wrong station name/service name) but the overall rail concept and connectivity are correct. No credit if it invents passenger services, uses freight-only lines as passenger service, or proposes impossible rail links.

Description Retrieve the current number of views for the specified video. Partial credit may be awarded if an approximate or slightly outdated view count is provided.

Description Determine what percent of the world's population the view count represents. This should include specifying the world population figure used and correctly computing (views / population) × 100. Partial credit may be given for using an approximate population estimate or showing the correct formula with minor arithmetic errors.

Description Locate the specified official music video and accurately report its current view count. Partial credit may be awarded for finding the video but not reporting the count, giving an approximate/outdated count, or explaining inability to access the count while providing a reasonable estimate.

Description Identify a reasonable current estimate of the world's population to use as the denominator for the percentage calculation. Partial credit for using a recent estimate or clearly stating and justifying an approximate figure.

Description Compute (views ÷ world population × 100) and present the resulting percentage, explicitly using a calculator or equivalent tool. Partial credit for correct calculation without explicitly noting tool usage, or for minor arithmetic errors with a correct method.

Description Determine and report the currently visible number of views for the specific YouTube video titled "Rolling in the Deep (Official Music Video)" by Adele (not a lyric video, live performance, or reupload). Full credit if the agent clearly identifies the correct video and provides the view count as shown on YouTube at time of access (or explicitly notes rounding if it rounds). If YouTube cannot be accessed due to external blockers (CAPTCHA, region restrictions, login wall, site outage), award full credit if the agent clearly reports the blocker and provides the best available alternative view-count evidence (e.g., search-result snippet, cached page, reputable secondary reporting) while noting potential staleness/that views change over time. Partial credit if the video version is somewhat ambiguous but closely related, or if the view count is approximate without stating it is rounded/approximate.

Description Using the retrieved view count (or best-available alternative if YouTube was blocked) and a stated world population estimate from a reputable source/year (or a clearly labeled current estimate), compute and report the percentage: views  world population  100. Full credit if the calculation is correct given the stated inputs and rounding is reasonable. Do not penalize for small differences due solely to different reputable population estimates/years, as long as the population figure used is stated or clearly implied and the arithmetic matches it. Partial credit if the method is correct but there are minor arithmetic/rounding errors or the population reference is not clearly stated.

Description Determine and state the correct city (and state) in which Massachusetts Institute of Technology is located by consulting Wikipedia.

Description Navigate to the identified city’s Wikipedia page and find the section or list specifically detailing its sister cities. Partial credit if the page is found but the sister cities section is not clearly identified.

Description From the sister cities list, correctly identify and name the first city mentioned. Partial credit for identifying a sister city but not confirming it is the first in the list.

Description Obtain and present the five-day weather forecast for the identified sister city from a reputable weather source, including daily high/low temperatures and conditions. Partial credit if fewer than five days are provided or key forecast details are missing.

Description Correctly determine the city in which MIT is located, as the basis for the sister city lookup. Partial credit may be awarded if the broader metro area is identified but not the exact city.

Description On Wikipedia.org, navigate to the identified city's page and locate the Sister cities/Twin towns section. Report the first sister city listed as it appears on the Wikipedia page. Partial credit for finding the sister cities list but not selecting the first item, or for naming a sister city without confirming it is the first. Full credit if the Wikipedia page has no sister city information and this is clearly stated.

Description Provide a 5-day weather forecast for the correctly identified sister city (with correct location disambiguation, e.g., city and country). Partial credit for fewer than 5 days or for a current/short-term forecast, or if the forecast is for a similarly named but incorrect location. Full credit if a 5-day forecast is not available and the limitation is clearly explained with the closest available alternative provided.

Description Determine the correct city in which the Massachusetts Institute of Technology (MIT) is located (the city used to find sister cities). Full credit for correctly identifying the city (e.g., Cambridge, Massachusetts). Partial credit if the agent identifies a broader/adjacent area that clearly leads to the correct city page but is ambiguous. No credit if the wrong city is used when the correct one is readily available.

Description On Wikipedia.org, locate the page for the identified city and find its "Sister cities" (or equivalent) section, then select the first sister city listed. Full credit if the agent clearly identifies the first sister city as shown on Wikipedia. If Wikipedia is inaccessible (blocked/down/CAPTCHA) or the relevant section is unavailable, award full credit if the agent clearly reports the limitation and uses a reasonable alternative source (e.g., another Wikimedia mirror or an official city page) while noting it is not Wikipedia. Partial credit if the agent finds a sister city but not the first one despite Wikipedia being accessible, or if the alternative source is used without explaining why Wikipedia could not be used.

Description Provide a 5-day weather forecast for the identified first sister city, from a reputable weather provider. Full credit if five distinct days are provided and the forecast is clearly for the correct city (dates and daily conditions/temperatures, as available). If a 5-day forecast cannot be retrieved due to external limitations (weather site/API blocked/down, paywall, location ambiguity preventing a reliable match), award full credit if the agent clearly reports the limitation and provides the best available alternative (e.g., fewer days available, or a nearby-location forecast) while explicitly stating the mismatch/limitation. Partial credit if fewer than five days are provided without explaining the limitation, or if key daily details are missing despite being available.

Description Navigate to raceroster.com, identify the very first race on the listing page, and accurately extract the venue or location details (e.g., city, address, or venue name). Partial credit can be given if only the city or venue name is found without a full address.

Description Using the identified race location, find a café or coffee shop in close proximity and provide its name and full street address. Partial credit may be awarded if only the name or partial address is provided.

Description Successfully navigate to or access the raceroster.com website as specified in the task. Partial credit if the agent references the site but cannot access it due to limitations and states this clearly.

Description Find the first race listed on raceroster.com and provide its location as shown (e.g., venue, city, and/or address). Partial credit for identifying a race but not confirming it is the first, or for providing only partial location details. Full credit if the site does not clearly show a single 'first' race and the agent explains the limitation.

Description Locate a café or coffee shop near the identified race location and provide its street address. Partial credit if only the name is given without the address, or if proximity is not clearly established.

Description Navigate to raceroster.com and view a race listing page where races are ordered (e.g., default homepage listings, a directory/search results page, or a location page). Full credit if the agent reaches a page that clearly shows an ordered list of races and states what context/sort is being used (default sort, selected filters/location if any). Full credit if raceroster.com is inaccessible (CAPTCHA/down/login wall/geo-block) and the agent clearly reports the blocker and what was attempted (e.g., refresh, alternate page, different browser path). Partial credit if the agent finds Raceroster content but the ordering context for 'first' is unclear.

Description Determine which race appears first in the ordered list the agent observed and provide enough identifying detail to verify it (e.g., race name and date, and optionally the event page/link or screenshot context). Full credit if the race is clearly the first item on the viewed list. Partial credit if a race is identified but the evidence that it is first is ambiguous (e.g., list not clearly ordered, filters not stated) or if a non-first race is chosen when the first item is visible. Full credit if the site is inaccessible and this is clearly reported (as captured in the access criterion).

Description Report the race location as presented on the race listing/detail page (city/state and venue/address if available). Full credit for accurately reporting the most specific location information that is available on the page. Partial credit if only partial location is provided when more specific details are clearly available. Full credit if the race page does not list a location or only provides ambiguous/online/virtual details and the agent clearly reports this limitation.

Description Identify at least one cafe/coffee shop plausibly near the race location (near the venue if a venue/address is given; otherwise near the stated city center or a clearly stated reference point). Full credit if the agent uses reasonable evidence of proximity (e.g., map results, stated distance/walking time, or clear neighborhood/adjacent landmark). Partial credit if the cafe is only in the same city with no attempt to establish nearness when the venue/reference point is available. Full credit if the race location is too vague to anchor 'nearby' and the agent clearly explains this and provides a best-effort option near the most specific available reference (e.g., city downtown) or requests the missing detail.

Description Provide a complete street address for the selected cafe/coffee shop (street, city, state/zip if available). Full credit if the address is provided and corresponds to the chosen cafe. Partial credit if the address is incomplete when a full address is readily available. Full credit if address data cannot be obtained due to external blockers (maps/search inaccessible, business listing not available) or if the business has no published street address, as long as the agent clearly reports the limitation and provides the best available location info (e.g., cross streets, neighborhood, or map pin description).

Description Provide the city and specific venue name where NeurIPS 2025 will be held. Partial credit if only the city is specified or if the venue name is incomplete or approximate.

Description Offer at least one recommended local dish or restaurant located near the identified conference venue, including the name and why it is notable. Partial credit for generic or incomplete recommendations.

Description Accurately identify the host city and specific event venue for the 2025 NeurIPS conference. Partial credit if only the city or only the venue is provided. Full credit if the agent determines that the official location/venue is not yet announced and clearly states that, rather than guessing.

Description Provide high-quality local food recommendations in close proximity to the identified venue (e.g., within reasonable walking or short transit distance). Assess proximity and quality/relevance of the recommendations. Partial credit if options are provided but are not clearly near the venue, are generic, or lack rationale. Full credit may be awarded if the venue is not announced and the agent explicitly notes the limitation and avoids tying recommendations to an unconfirmed venue.

Description Locate and report where NeurIPS 2025 will be held. Full credit if the agent provides the correct host city and venue (or official event site/venue name if listed). Partial credit if only the city or only the venue is correctly identified but the full location context is missing/ambiguous. Full credit if the agent clearly states that the official NeurIPS 2025 location (city and/or venue) is not yet announced and supports this with a credible source or clear indication from official NeurIPS communications.

Description Using the identified location details, recommend at least one well-regarded local food option plausibly near the venue. Full credit if the agent (a) provides a clear local recommendation and explains why it is considered good and nearby, OR (b) if venue/address-level detail is unavailable or unverified, clearly states this blocker and instead recommends strong local food options in the host city (or within the venue’s district/area if known) while explaining the limitation. Partial credit if recommendations are local but proximity is not addressed at all, or if the options are generic/non-local without tying them to the locale. Full credit should not be withheld due to external volatility (e.g., closures, hours, reservation availability, shifting ratings) as long as the agent reasonably attempts to identify reputable options and notes any uncertainty.

Description Navigate to the Steam website, find the RPG category's top-sellers list, and correctly identify the number-one selling RPG title. Partial credit if Steam and RPG section are accessed but the top-seller is misidentified.

Description Determine a game controller that is compatible with and appropriate for the identified top-selling RPG. Partial credit if a generally compatible controller is chosen but not specifically recommended for the game.

Description Search Amazon for the specified controller model and locate the correct product page. Partial credit if a similar or generic version is found instead of the exact model.

Description Add the identified controller to the Amazon shopping cart without proceeding to checkout. Partial credit if the correct product page is reached but 'Add to Cart' is not executed.

Description Navigate to the Amazon order review page (showing cart contents ready for purchase) and stop before entering any personal or payment information. Partial credit if the cart is visible but the review page is not reached.

Description Successfully identify the current top-selling RPG game on Steam. Partial credit may be awarded for navigating to the Steam RPG category or Top Sellers list but not conclusively identifying the top entry, or for noting if a top-seller list is unavailable/region-restricted at the time.

Description Determine and name a game controller that matches the identified RPG game (i.e., a controller suitable for playing the game). Partial credit may be awarded for proposing a reasonable, commonly used controller even if the exact match is unclear.

Description Locate the identified controller on Amazon. Partial credit may be awarded for reaching a closely related product page or stating clearly if the exact controller is unavailable/out of stock.

Description Add the identified controller to the Amazon cart. Partial credit may be awarded for attempting to add to cart and indicating any blockage (e.g., out of stock, region restrictions). No checkout or personal information entry should be performed.

Description After adding to cart, stop at a review page (e.g., product reviews section or cart review) without entering any personal information or proceeding to checkout. Partial credit may be awarded for clearly stopping at the cart page or product page if a distinct review page is not accessible.

Description Identify the current top-selling RPG game on Steam using Steam’s Top Sellers with RPG category/tag filtering (or an equivalent Steam-supported view). Full credit if the agent demonstrates it used a top-seller-ranked view and clearly identifies the top-selling RPG at time of lookup. Full credit if Steam is inaccessible (blocked/down/region/CAPTCHA) and the agent clearly reports the blocker and uses a reasonable alternate Steam-accessible method/view (or clearly explains why verification cannot be completed). Partial credit if the agent names a plausible top-selling RPG but does not establish that it is the top seller (unclear sorting, not a top-sellers view, or missing evidence of ranking).

Description From the identified game’s Steam page, determine any explicitly indicated controller match (recommended/named controller) or, if no specific model is named, the explicit controller support shown (e.g., “Full controller support,” supported controller standards/types). Full credit if the agent correctly ties the controller information to the specific game page. Full credit if the game page does not specify a particular controller model and the agent explicitly states that no specific matching controller is named while reporting the supported controller type/standard shown on Steam. Partial credit if the agent chooses a generic controller without linking it to explicit Steam evidence when such evidence is available.

Description Search Amazon and land on a product page that matches the identified controller (correct brand/model/variant where applicable). Full credit if the product page corresponds to the identified controller. Full credit if the exact controller cannot be found due to external factors (regional catalog differences, listing removed, out-of-stock/no Buy Box, restricted item) and the agent clearly reports this and selects the closest clearly-labeled equivalent that matches the primary intent. Partial credit if the agent lands on a close but not exact match when the exact match appears available.

Description Add the selected controller to the Amazon cart and proceed toward checkout until reaching the order review stage (“Review your order” or equivalent), then stop without placing the order. Full credit if the item is added and the agent reaches the review page and stops. Full credit if the agent cannot reach the review page due to uncontrollable blockers (login/CAPTCHA/address required/region restrictions/checkout gating) but clearly reports the blocker and shows the item was added to cart (or that adding was prevented by the blocker). Partial credit if the item is added to cart but the agent stops at cart without attempting to proceed toward the review step when no blocker is present.

Description Navigate to and open an online mortgage rate calculator tool as specified in the task description.

Description Input the correct home purchase price of $500,000 into the mortgage calculator.

Description Input the specified down payment amount of $80,000 into the calculator.

Description Select or input a loan amortization period of 30 years as required.

Description Enter the annual interest rate of 6.0% into the mortgage calculator.

Description Provide the location information by entering the zip code 98101 if required by the tool.

Description Obtain and report the calculator's estimated monthly payment, including only principal and interest.

Description Successfully access and use a mortgage rate calculator tool online, as explicitly required by the task. Partial credit may be given if an online tool is identified but not fully used, or if the tool is inaccessible and this is clearly documented.

Description Input the specified parameters into the calculator: home price $500,000; down payment $80,000; loan term 30 years; interest rate 6.0%; and location/ZIP 98101 (if the tool accepts or requires a location). Partial credit may be awarded if most inputs are correct but one is missing or misentered, or if the tool does not accept ZIP and this is explicitly noted.

Description Ensure the monthly payment shown includes only principal and interest (exclude taxes, insurance, HOA, PMI, etc.). Partial credit may be awarded if extras are shown but the P&I portion is clearly isolated and reported, or if the tool cannot exclude extras and this limitation is clearly stated along with the P&I figure derived from the tool’s breakdown.

Description Provide the estimated monthly payment amount (principal and interest only) obtained from the online calculator. The result should be clearly stated. Partial credit may be awarded for an approximate figure or range if the tool provides limited precision and this is explained.

Description Agent uses or clearly attempts to use an online mortgage/mortgage payment calculator. Full credit if the agent successfully uses an online calculator, OR if it documents an uncontrollable blocker (e.g., CAPTCHA/paywall/site down) and then uses a different accessible online calculator to obtain the estimate. Partial credit if the agent provides a plausible estimate but the use of any online calculator is unclear. No credit if the agent neither uses nor attempts to use an online tool and does not explain why.

Description Inputs reflect the task: $500,000 home price and $80,000 down payment (equivalently $420,000 loan amount), 30-year term, 6.0% interest rate. ZIP/location 98101 should be entered if the calculator supports it; do not penalize if the calculator has no ZIP field or if ZIP does not affect the principal-and-interest computation and the agent notes this. Full credit if all core financial inputs are correct or entered via equivalent fields. Partial credit if one core input is slightly off but the agent otherwise demonstrates correct setup/intent, or if ZIP is omitted because the calculator does not support it. No credit if core financial inputs are materially wrong when correct entry was possible.

Description Agent reports the monthly payment amount specifically for principal + interest, excluding taxes, insurance, HOA, PMI, etc. Full credit if the P&I value is clearly labeled as such (or the calculator explicitly shows a P&I component and that is the value reported). If the calculator only provides a total payment that appears to include other costs, full credit if the agent explains the limitation and uses another calculator/setting to isolate P&I; partial credit if the agent reports the total but also provides a reasonable P&I estimate with a clear caveat. No credit if the agent reports a payment that includes other costs without clarifying or isolating P&I.

Description Navigate to and open the specified college savings calculator on the Office of Financial Readiness website.

Description Input the 3% annual inflation rate for education costs into the appropriate field (not using sliders).

Description Input the existing college savings amount of $50,000 into the calculator (not using sliders).

Description Set the monthly contribution to $250 in the calculator (not using sliders).

Description Input the expected annual rate of return of 6% into the calculator (not using sliders).

Description Input the annual tuition cost of $50,000 into the appropriate field (not using sliders).

Description Input the annual room and board cost of $12,000 into the calculator (not using sliders).

Description Ensure that all numeric inputs are entered manually or via text fields rather than using slider controls.

Description Use the calculator’s output to determine and report how much more per month must be saved to meet the projected college costs.

Description Navigate to and use the specified Office of Financial Readiness college savings calculator (not a different tool). Partial credit if the agent references it but appears to use another calculator.

Description Input the child's current age as 2 (or the equivalent setting in the tool reflecting a 2-year-old). Partial credit if the agent indicates the child’s age but does not clearly apply it in the calculator.

Description Input tuition as $50,000 per year and room/board as $12,000 per year, as specified. Partial credit if only one of the two values is entered correctly or if a correct combined total is entered when the tool requires a single annual cost input.

Description Input the following exactly: 3% education cost inflation, $50,000 current savings, $250 monthly contributions, and 6% annual rate of return. Partial credit for correctly entering some but not all values.

Description Ensure values are entered without using sliders (typed or otherwise manually entered). Partial credit if most inputs are typed but one appears adjusted by slider.

Description Retrieve and clearly state the incremental additional monthly savings required according to the tool (beyond the current $250/month). Full credit if the agent provides the exact incremental figure; partial credit if only the total required monthly amount is given without the 'more per month' delta.

Description Navigate to and attempt to use the Office of Financial Readiness college savings calculator to compute the result. Full credit if the agent uses this specific tool to produce the result, OR if the agent clearly documents being blocked by an uncontrollable issue (site down, CAPTCHA, login requirement, broken calculator, tool not loading). Partial credit if the attempt is unclear or the wrong tool is used without justification.

Description Input all required fields exactly as specified using typed/manual entry (not sliders): 3% education cost inflation, $50,000 current savings, $250 monthly contributions, 6% rate of return, tuition $50,000 per year, and room/board $12,000 (or the closest equivalent fields if labeled differently). Full credit if all values are entered correctly via manual entry. If the tool enforces sliders only or lacks one or more of these fields, full credit can still be earned by (a) attempting manual entry where possible and (b) explicitly stating which fields are unavailable/slider-locked and therefore could not be entered as requested. Partial credit if one value is entered incorrectly or the manual-entry constraint is not followed when avoidable.

Description Read the calculator output and answer: how much more per month needs to be saved beyond the stated $250/month (i.e., additional monthly amount). Full credit if the incremental amount is clearly stated and consistent with the tool output (either directly shown by the tool or correctly derived from a total monthly amount shown by the tool). If the tool output does not provide an incremental figure or the relevant output is not visible due to an uncontrollable tool issue, full credit can still be earned by clearly stating what the tool did show (e.g., total required monthly contribution) and why the incremental amount cannot be determined from the tool as presented. Partial credit if only the total required monthly contribution is reported without clearly converting to the 'more per month' amount when the conversion is possible from the displayed output.

Description Navigate to investor.gov and locate the appropriate calculator for computing future value with regular contributions.

Description Enter an initial principal of $10,000, monthly contributions of $200, a 10-year timeframe, a 5.0% interest rate, and quarterly compounding, then perform the calculation. Partial credit for applying some but not all parameters correctly or for arithmetic errors.

Description Report the colors of the lines shown in the results plot. Partial credit if some but not all line colors are correctly identified.

Description Navigate to investor.gov and locate/open the relevant compound interest or savings calculator needed to perform the computation. Partial credit if the site is accessed but the correct tool is not clearly identified.

Description Use the tool with exactly the given inputs: initial principal $10,000; monthly contributions $200; time horizon 10 years; annual interest rate 5.0%; compounding quarterly. Partial credit if most inputs are correct but one or more (e.g., compounding frequency, contribution frequency, or time) are set incorrectly or not demonstrated.

Description Obtain the calculation result from the tool and clearly report the final future value. Partial credit if an amount is computed but not clearly identified as the final total, or if the value is approximate without stating it is an estimate.

Description Identify and state the colors of the lines shown in the results plot produced by the tool. Partial credit if only one color is reported or if the agent notes that no plot is available for the given tool/context.

Description Navigate to investor.gov and attempt to use an on-site calculator/tool relevant to computing investment growth. Full credit if the agent makes a reasonable attempt but is blocked (e.g., site down, CAPTCHA, tool not loading) and clearly reports the blocker. Partial credit if the attempt is unclear or investor.gov is not attempted despite being available.

Description Compute the final account value for the specified scenario. Full credit if the agent uses investor.gov successfully OR, if investor.gov is inaccessible/unusable, uses a reasonable alternative method (e.g., explicit finance math or another reputable calculator) and explains that it is a substitute due to the blocker. Partial credit if the method is plausible but under-specified or not clearly tied to the parameters.

Description Apply the task parameters correctly: initial principal $10,000; monthly contribution $200; time horizon 10 years; interest rate 5.0%; compounding quarterly. Full credit if all parameters are correctly applied (via investor.gov inputs or equivalent math). Partial credit if one parameter is slightly wrong but the agent acknowledges/identifies the discrepancy or provides both interpretations (e.g., reconciling monthly contributions with quarterly compounding). No credit if multiple key parameters are wrong or omitted.

Description Provide the final computed account value after 10 years consistent with the stated parameters (allowing minor rounding differences). Partial credit if a near-correct value is provided but the agent appears to have used a different convention without reconciling it. No credit if the amount is missing or clearly inconsistent with the provided parameters.

Description State the colors of the lines shown in the investor.gov results plot. Full credit if all line colors are correctly identified as displayed. Full credit also if the agent cannot view the plot due to an uncontrollable issue (e.g., investor.gov/tool/plot not loading, blocked access) and explicitly reports that limitation rather than guessing. Partial credit if only some line colors are provided or if descriptions are ambiguous.

Description Successfully navigate to and identify the most recent news release on the US Department of Labor website.

Description Accurately extract the name of the media contact listed in that news release. Partial credit for incomplete or ambiguous identification.

Description Correctly count how many additional contacts are listed in the same department aside from the media contact.

Description Navigate to and open the most recent (latest) news release from the US Department of Labor. Partial credit may be awarded for accessing a DOL news release without verifying it is the latest or for identifying the correct section but failing to open the specific release. Full credit can be awarded if the site does not clearly indicate recency and the agent explains the limitation.

Description Find and report who the media contact is for that specific news release (e.g., the name listed under 'Media Contact'). Partial credit may be awarded for identifying a contact but not clearly specifying it is the media contact, or for noting that no media contact is listed if that is the case.

Description Determine how many other contacts (besides the primary media contact) are listed as being in the same department as the media contact on that release. Partial credit may be awarded for counting total contacts without filtering by department, or for explaining that department information is unclear and providing the most reasonable count available. Full credit if no other contacts exist and the agent explicitly states zero.

Description Navigate to the official US Department of Labor site (e.g., Newsroom/News Releases listing) and attempt to determine how items are sorted by recency (date/time, pagination). Full credit if the agent successfully reaches the listing and can evaluate recency ordering, OR if the agent is blocked by an uncontrollable issue (site down, CAPTCHA, access denied) and clearly reports what prevented access. Partial credit if the agent uses an unofficial mirror/source without explaining why the official site could not be used.

Description From the accessible official listing, select the most recent item that is clearly a "news release" and identify it (e.g., title and date/time). Full credit if the agent correctly identifies the latest release, or if recency is ambiguous (time zones, multiple items same date, mixed content types) and the agent selects a defensible near-latest release while explaining the ambiguity. Full credit if the agent cannot confirm the latest due to an uncontrollable blocker and clearly documents the limitation. Partial credit if the agent selects an older release when a clearly newer news release is visible.

Description From the identified latest news release page, extract and report the media contact exactly as labeled (person or office). Full credit if correctly reported, OR if the release has no media-contact field/contact block and the agent explicitly states that none is listed on the page. Partial credit if the agent provides a general DOL contact that is not labeled as the media contact when a media contact is present, or if the contact is incomplete (e.g., missing name/office when shown). Full credit if the agent cannot access the release page due to an uncontrollable blocker and clearly reports the issue.

Description Determine how many additional contacts (beyond the media contact) are listed in the same contact block/press office/department contact section on that release page, and report the count with a brief note on what was counted (e.g., additional named individuals vs. offices). Full credit if the count is correct, OR if the page provides no additional contacts and the agent reports 0, OR if the page’s contact information is not presented as a list of distinct contacts (e.g., only a generic email/phone, or no contact section) and the agent states that the number of "other contacts" cannot be determined from the release. Partial credit if the method is unclear or the agent likely counted non-contacts (e.g., phone numbers) as separate contacts without noting ambiguity. Full credit if the agent cannot access the release page due to an uncontrollable blocker and clearly reports the issue.

Description Find and name at least one soul food restaurant in Houston that Beyoncé has mentioned as a favorite. Partial credit if the restaurant is commonly cited without explicit Beyoncé reference; full credit only if there is evidence it is one of her favorites.

Description Navigate to or locate the restaurant’s official website from which opening information can be obtained. Partial credit for finding a third-party source if no official site is available; full credit for the official site.

Description Locate and report the date (year and, if possible, month/day) when the restaurant first opened. Partial credit for estimating the year if the exact date isn't listed; full credit for the precise opening date.

Description Using Beyoncé’s birth year (1981) and the restaurant’s opening date, compute the age difference in years. Partial credit for an approximate calculation; full credit for the accurate age difference.

Description Find one restaurant that meets all explicit constraints: it is in Houston, serves soul food, and is known as one of Beyoncé's favorite restaurants. Partial credit if the restaurant matches some but not all constraints (e.g., in Houston but not clearly soul food, or Beyoncé-associated but not confirmed as a favorite).

Description Navigate to and access the restaurant’s official website (not a third-party listing). Partial credit if only a third-party page is found when an official site appears unavailable, with this unavailability noted.

Description Locate the opening year/date from the restaurant’s official website. Full credit if the site clearly provides the opening date. If the official site does not provide this information, full credit may be awarded for explicitly stating that it’s unavailable on the site; partial credit for unclear or inferred dates without confirmation from the site.

Description Compute the age difference between the restaurant’s opening date and Beyoncé’s birth date. Full credit for a correct calculation accounting for years and, where possible, months/days; partial credit for an approximate year-only difference when exact dates are not available.

Description Determine a specific Houston soul food restaurant that is explicitly described by at least one credible source as one of Beyoncé's favorites (or a clearly equivalent phrasing such as she ‘loves,’ ‘frequents,’ or it’s a ‘go-to’). Full credit if the restaurant is correctly identified and the Beyoncé connection is supported with evidence/citation. Full credit is also acceptable if, after reasonable search effort, no explicit ‘favorite/go-to’ phrasing can be found; in that case the agent should clearly state this limitation and select the best-supported Houston soul food restaurant that is credibly linked to Beyoncé (e.g., reported as visited/recommended by her). Partial credit if the restaurant is a plausible Houston soul food spot but the Beyoncé connection is weak/uncited/ambiguous. No credit if the restaurant is not in Houston, not soul food, or not connected to Beyoncé.

Description Attempt to use the identified restaurant's official website to locate information stating when it opened (date or year), and clearly attribute the information to the site if found. Full credit if the opening year/date is taken directly from the restaurant's website (e.g., About/History page). Full credit if the agent attempts the official website but it is inaccessible (down/blocked/CAPTCHA/login), or if the site does not state an opening date; the agent must clearly report the blocker/absence and where they looked on-site. Partial credit if the agent provides an opening date from a third-party source after failing to obtain it from the official site, as long as the official-site attempt and failure is clearly documented. No credit if an opening date is fabricated or presented as coming from the official website when it is not.

Description Provide Beyoncé's birth date or at minimum birth year correctly (needed for age comparison). Full credit for correct value (e.g., born 1981; full date acceptable). Partial credit if only an approximate/uncertain year is provided but is close enough to enable a comparison with explicit caveats. No credit if incorrect year/date is used or invented without basis.

Description Calculate the difference between the restaurant's opening year/date and Beyoncé's birth year/date and report the result. Full credit for correct arithmetic and a clear statement (e.g., 'opened in YEAR, Beyoncé born YEAR, restaurant is N years older'). If only years (not exact dates) are available, full credit for a clearly stated year-based difference and noting it is approximate with respect to months/days. Partial credit if the calculation is slightly off due to missing month/day precision but the approach is explained. No credit if the comparison is not provided or is numerically wrong without explanation.

Description Navigate to Wikipedia and locate the page or section that clearly states which is the oldest university in the United States. Partial credit if the agent finds relevant university names but does not confirm which is oldest.

Description From the Wikipedia page, extract the city in which the identified oldest university is located. Partial credit if the agent finds the university but misidentifies the city.

Description Open Rentalcars.com, enter the identified city as the pickup location, and set pickup date to November 17 2025 and drop-off date to November 19 2025. Partial credit if the correct website is accessed but dates or location are entered incorrectly.

Description Apply the compact car filter on the search results page. Partial credit if a close vehicle category is selected instead of exactly 'compact'.

Description Among the filtered compact car results, find and report the lowest daily or total rental price. Partial credit if the agent reports a reasonable low-cost option but not the absolute lowest.

Description Access Wikipedia.org and identify the city where the oldest university in the United States is located. Full credit requires correctly naming the city and showing it was determined using Wikipedia. Partial credit may be awarded if the university is identified but the city is missing or incorrect, or if the source used is unclear.

Description Navigate to Rentalcars.com and initiate a car rental search using the city found on Wikipedia as the pickup location, with dates set to November 17–19, 2025. Full credit requires correct site access, correct location, and correct dates. Partial credit may be awarded if only some parameters (e.g., dates or location) are correctly set.

Description From the Rentalcars.com search results, find the Compact category and determine the lowest priced option for the specified dates and location. Full credit requires clearly reporting the lowest price and key details (e.g., provider and vehicle/class). Partial credit may be awarded if compact options are listed but the lowest is not clearly identified, or if compact cars are unavailable and this is explicitly stated.

Description Use Wikipedia.org to determine the oldest university (or oldest institution of higher education/university, as described on Wikipedia) in the United States and extract the city where it is located. Full credit if the agent cites/grounds the choice in Wikipedia and states an unambiguous city. Partial credit if the university is correct but the city is missing/unclear, or if the city is correct but the Wikipedia grounding is weak. If Wikipedia presents ambiguity (e.g., multiple candidates depending on definition), full credit if the agent notes the ambiguity and proceeds with a defensible Wikipedia-supported choice and city.

Description Attempt to navigate to Rentalcars.com and initiate a search using the identified city as the pickup location and the specified dates (Nov 17–19, 2025). Full credit if the agent performs the search with correct location and dates, OR if Rentalcars.com is inaccessible (e.g., CAPTCHA, outage, required login, geoblock) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent makes a minor input mistake but corrects it, or if it uses another platform only after clearly documenting that Rentalcars.com could not be used.

Description From the Rentalcars.com results for the specified city and dates, identify the lowest priced option within the 'Compact' car category. Full credit if (a) a compact-category option is available and the agent demonstrates it is the lowest-priced compact shown (e.g., by sorting, filtering, or visibly comparing compact results) and records key details (price as shown, supplier/company, and compact-category confirmation), OR (b) no compact category is available / no cars are available / prices are not shown and the agent clearly reports this limitation from the site. Partial credit if a compact option is identified but the agent does not establish it is the lowest-priced compact, or omits one key detail when the listing is otherwise clear.

Description Progress only as far as needed to determine the lowest-priced compact option and its details, without completing a booking or entering any personal/payment information. Full credit if the agent stops before final reservation/checkout steps and does not provide or fabricate user details. Partial credit if the agent navigates near the final booking stage but does not submit/confirm anything. No credit if the agent completes a booking/reservation or enters personal/payment information.

Description Find an exact quotation in which Dario Amodei states that AI will take a lot of jobs. Partial credit may be awarded for a close paraphrase or a correct citation with minor wording differences.

Description Extract the specific unemployment rate percentage that Amodei predicted in the quote. Partial credit may be awarded for approximate values or formatting errors.

Description Compute how many percentage points higher Amodei’s predicted unemployment rate is compared to the highest US unemployment rate in 2001. Partial credit may be given for correct methodology with minor arithmetic mistakes.

Description Locate and present a clear, direct quote attributed to Dario Amodei that explicitly conveys the idea that AI will take many jobs. Partial credit may be awarded for quotes that imply job displacement but are less explicit, or for paraphrases clearly attributed to Amodei.

Description Identify and report the specific unemployment rate prediction made by Amodei in connection with the quote or related context. Partial credit may be awarded for an approximate figure or range if a precise value is unavailable.

Description Determine and state the highest unemployment rate the US experienced in 2001. Partial credit may be awarded for providing a closely related figure (e.g., annual average) if the maximum is not found, but full credit requires the maximum as requested.

Description Compute how many percentage points higher Amodei’s predicted unemployment rate is compared to the 2001 maximum, and report the result. Partial credit may be awarded for an attempted calculation with minor arithmetic errors.

Description Provide at least one attributable quote from Dario Amodei that explicitly conveys that AI will take many jobs (e.g., mentions job loss, job displacement, or large-scale automation). Full credit if the quote is clearly attributed and contains the relevant claim. Partial credit if the statement is paraphrased rather than quoted, or if the quote is ambiguous about job loss. Full credit if the agent reports that no such quote could be found after reasonable search, including briefly stating what sources/queries were tried and noting blockers like paywalls/captchas.

Description State the unemployment rate Dario Amodei predicted (percent) in the cited source. Full credit if the numeric rate is correctly extracted and clearly presented (optionally including timeframe/context if present). Partial credit if the agent provides a plausible figure but the context is unclear, the figure is presented as a range when only a point estimate was asked (or vice versa), or it appears to be from a closely related but not definitively Amodei-attributed source. Full credit if, after a reasonable attempt to locate/verify the prediction in accessible sources, the agent clearly reports it cannot verify a specific numeric rate (e.g., due to paywall, conflicting reports, or inability to locate the original statement), and explains the limitation.

Description Find and state the maximum US unemployment rate experienced in calendar year 2001 (percent), indicating it is the maximum (not the annual average). Full credit if the maximum value is correctly reported and tied to a credible public source (e.g., BLS series). Partial credit if a 2001 unemployment figure is provided but it is not established to be the maximum or the source is unclear. Full credit if the agent makes a reasonable effort using alternative public sources and clearly reports inability to verify the maximum due to access limitations or source unavailability.

Description Compute and report the difference in percentage points: (Amodei predicted unemployment rate) minus (maximum US unemployment rate in 2001). Full credit if the arithmetic is correct and expressed in percentage points. Partial credit if the method is correct but there is a minor arithmetic/rounding error, or if the result is mistakenly reported as a percent change rather than percentage points. Full credit if the agent cannot compute the difference solely because one or both required numeric inputs could not be verified due to external/source-access limitations, provided the agent explicitly states what is missing and why (and computes the difference if later sufficient numbers are available).

Description Navigate to the USAJobs (usajobs.gov) site without requiring login or entering any personal information.

Description Use the site’s search functionality to filter or search for jobs specifically in ZIP code 10003.

Description Locate the first job in the search results and accurately extract the listed salary information for that position.

Description Go to salary.com, search for the same role title found in the first listing, and determine the national median salary for that role without logging in or providing personal data.

Description Compare the salary from the first listing against the national median salary and clearly state whether it is above or below that median.

Description Access the USAJOBS website and locate job postings tied to the 10003 area. Full credit if the agent searches specifically for jobs in 10003 and notes if none are available. Partial credit if searching nearby locations or indicating issues with filtering. No requirement to sign in or apply.

Description From the search results, use the first listing returned and capture the job title and salary (or salary range) shown. Partial credit if the listing is found but salary is missing and this is explicitly stated.

Description Use salary.com to look up the national median salary for the identified role. Partial credit if a closely related role is used when an exact match is unavailable, or if unavailability is clearly reported.

Description Compare the first listing’s salary to the salary.com national median and clearly state whether it is above or below. Partial credit if numeric values are provided but the conclusion is unclear, or if a reasonable method (e.g., midpoint of a range) is used when needed.

Description Attempt to use USAJOBS to search for jobs in/near ZIP code 10003 (or, if USAJOBS does not support ZIP targeting cleanly, an equivalent query such as "10003" location, "New York, NY" with radius, or a map-based filter). Full credit if the agent makes a reasonable attempt and either reaches results relevant to the 10003 area or clearly reports a blocker (CAPTCHA, outage, login wall, blocking). Partial credit if the agent searches NYC generally without explaining why 10003-specific filtering could not be applied or verified.

Description From the USAJOBS results page (under the observed default/selected sort order, which should be stated or evident), identify the first listing shown and report its job title/role and the salary (range or stated pay). Full credit if the first listing is unambiguous and salary is captured accurately (from results or the listing detail page). Full credit if the first listing is identifiable but salary is not displayed/available and the agent clearly reports that limitation after checking the detail page. Partial credit if either title/role or salary is missing/incomplete despite being available, or if the ‘first listing’ selection is ambiguous due to not indicating the ordering used.

Description Use Salary.com to locate a national median salary figure for the same (or closest clearly justified) role category matching the USAJOBS listing’s title/role. Full credit if the agent finds and reports the Salary.com national median. Full credit if Salary.com is inaccessible (paywall/blocking) or no suitable matching role exists and the agent clearly reports the limitation and what was tried. Partial credit if the match is loose without noting assumptions or mismatch.

Description Using the USAJOBS salary and the Salary.com national median, explicitly state whether the USAJOBS pay is above or below the national median. If the USAJOBS listing provides a range, full credit if the agent uses a defensible, clearly stated method (e.g., compares midpoint to median, or states whether the entire range is above/below/overlaps the median and then gives a clear above/below determination based on the chosen method). Full credit if a comparison cannot be made because either the USAJOBS salary or Salary.com median is unavailable and the agent clearly states why comparison is not possible. Partial credit if an above/below conclusion is given but the method for handling ranges is unclear.

Description Agent navigates to and identifies an official Microsoft Support web page that provides a tutorial on creating or using pivot tables.

Description Agent verifies that the page includes an example spreadsheet or a screenshot illustrating a pivot table.

Description Agent finds the specific example table or screenshot on the page and determines which row is the first data row of that table.

Description Agent accurately extracts and transcribes the content of the first row of the example table as shown on the page.

Description Find and identify an official Microsoft support page (e.g., on support.microsoft.com) that is specifically a tutorial about pivot tables. Partial credit may be awarded if the page is official but not clearly a tutorial, or if it is a Microsoft page about pivot tables but not on the support subdomain.

Description Verify that the identified page includes an example spreadsheet or a screenshot illustrating a pivot table. Partial credit may be given if the page references examples but does not clearly display a spreadsheet or screenshot.

Description Accurately determine and provide the content of the first row from the example table depicted on the page. Partial credit may be awarded if the row is reported with minor inaccuracies or omissions, or if multiple examples exist and one is chosen with a clear indication of which it is.

Description Locate a page on an official Microsoft Support domain (e.g., support.microsoft.com) that is explicitly a tutorial/how-to about PivotTables. Full credit if the agent identifies such a Support page or, if Support pages are inaccessible/unavailable (e.g., blocked, down, region-gated), clearly reports the access issue and provides the closest Microsoft-owned alternative it could access (while stating it is not a Support page). Partial credit if the page is Microsoft-owned but not on Microsoft Support or only tangentially about PivotTables.

Description Confirm that the identified Microsoft Support page itself contains an embedded example spreadsheet or screenshot depicting a PivotTable. Full credit if an actual on-page example is present and the agent points to it; also award full credit if the agent cannot verify due to external issues (images not loading, script-blocking, access restrictions) but clearly reports this and explains what was attempted/observed. Partial credit if the page has images but they do not clearly show a PivotTable, or if the only PivotTable example is only accessible via an external link/download rather than being clearly on the page.

Description Report the first row of the example spreadsheet/table shown in the PivotTable example on the page. Full credit if the first row is transcribed exactly as shown; if “first row” is ambiguous (header row vs first data row), accept either interpretation as long as the agent states which one it used. If the row text is not legible/extractable due to external factors (low-resolution image, missing alt text, image not loading), full credit if the agent clearly states that it is unreadable and provides the best-supported partial extraction (e.g., any visible cells/labels) without fabricating content. Partial credit if the correct row is targeted but transcription has minor errors or omissions.

Description Navigate to the Steam website/storefront as specified in the task.

Description Find and correctly identify the current top-selling horror game on Steam.

Description Determine and note the official or most relevant guidebook linked to the identified horror game.

Description Navigate to Amazon’s website to begin the guidebook search.

Description Use Amazon’s search functionality to find the exact guidebook title identified from Steam.

Description Successfully add the found guidebook to the shopping cart and navigate to the cart review page without proceeding to checkout.

Description Navigate Steam and determine the top-selling game within the Horror category/tag at the time of the task. Full credit if the agent clearly identifies the top-selling title or notes if Steam does not provide this information at that time. Partial credit for locating the relevant Top Sellers list but misidentifying the exact top title.

Description Identify and state the guidebook associated with the top-selling horror game (e.g., official strategy guide/art book). Full credit if the agent determines that no official/associated guidebook exists and explicitly reports that after checking. Partial credit for a likely but not clearly associated guidebook.

Description Use Amazon to search for the exact guidebook identified. Select a matching product result. Full credit if the agent reports clearly that no such listing exists after searching. Partial credit for close matches or reasonable search attempts.

Description From the correct Amazon product page, add the identified guidebook to the shopping cart. Full credit if the item cannot be added (e.g., unavailable) and the agent clearly indicates that. Partial credit for adding an incorrect edition/format that is otherwise reasonable.

Description Navigate to the Amazon cart review page to confirm the item is in the cart, and stop there. Do not sign in, proceed to checkout, or enter any personal/sensitive information. Full credit for clearly stopping at the cart page.

Description On Steam, attempt to reach a page that can credibly show top-selling titles in the horror genre (e.g., Steam Charts/Top Sellers with horror tag/category, or store tag pages with top sellers). Full credit if Steam is blocked (CAPTCHA/login/region restrictions/outage) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent uses a less direct proxy for top-selling (e.g., general popularity) without confirming a top-sellers context.

Description Determine which title is currently the top-selling game in the horror genre from the accessed Steam top-sellers/charts context. Full credit if the correct game is identified for the moment of search, or if Steam access was blocked and the agent cannot determine it (as long as the blocker/attempt is documented in the prior step). Partial credit if the agent names a plausible horror best-seller but does not clearly verify it is top-selling within a horror-specific view when such verification was available.

Description From Steam or clearly linked Steam surfaces (game page, DLC list, community hub, official announcements), identify and record the associated guidebook (title sufficient to search). Full credit if a specific guidebook is identified, OR if after reasonable checking the agent determines there is no clearly associated guidebook and reports that finding. Partial credit if only a generic guides hub is cited without a specific guidebook title when a specific one is available.

Description On Amazon, search for the same guidebook identified in the prior step and reach a product listing that matches the guidebook (or the closest clearly corresponding edition/format if the exact one is unavailable). Full credit if the matching listing is found, OR if Amazon is blocked (CAPTCHA/login/region) and the agent clearly reports the blocker and attempts. Full credit if the exact guidebook appears unavailable/out of print and the agent clearly documents that and selects the closest match that preserves primary intent (same game guidebook). Partial credit if the agent lands on a mismatched guidebook when a correct match was available.

Description Add the correct guidebook (or closest matching edition/format if exact is unavailable) to the Amazon cart and proceed to the cart page/cart review page with cart contents visible, then stop (do not proceed into entering shipping/payment details). Full credit if the item is in cart and the agent stops appropriately. Full credit if adding to cart or viewing the cart is prevented by login requirements, region restrictions, stock limitations, or other site limitations and the agent clearly reports the blocker and the furthest point reached. Partial credit if the item is added but the agent does not reach the cart page, or the wrong item is added when the correct one was available.

Description Successfully navigate to and access the Booking.com website as the starting point for the hotel search.

Description Enter the specified location (San Francisco, California), dates (November 20–24, 2025 for a four-night stay), and number of adult guests (1). Partial credit may be awarded for partially correct inputs.

Description Find and correctly identify the hotel option with the lowest total price for the specified stay. Full credit if the agent locates the cheapest available hotel; partial credit if close or if availability issues are noted.

Description Accurately record the full address of the identified cheapest hotel.

Description Using the hotel's address, locate the closest grocery store in proximity. Partial credit if a nearby store is identified but not the absolute closest.

Description Clearly state the name and full address of the nearest grocery store found.

Description Conduct the search on Booking.com as explicitly requested. Partial credit if the agent attempts to access or reference Booking.com but cannot complete due to access issues.

Description Use San Francisco, California as the destination, set the stay for 1 adult, and apply a four-night date range as stated (November 20–14, 2025). Full credit if the agent applies a four-night range using the specified dates; if the dates are inconsistent/invalid, full or partial credit may be earned by explicitly identifying the inconsistency and indicating that clarification is needed to proceed accurately.

Description From Booking.com results for the specified parameters, determine the cheapest available option that is a hotel (not another property type). Full credit if availability is confirmed and the hotel is clearly the lowest-priced option; if no hotels are available for those parameters, full credit for clearly stating that outcome.

Description Retrieve and present the hotel's address needed for the next step. Partial credit if only a partial or approximate address is provided.

Description Using the hotel's address, identify the nearest grocery store and provide both the store's name and its address. Partial credit if a nearby (but not clearly the closest) grocery store is provided, or if either the name or address is missing.

Description Attempt to access Booking.com to perform the hotel search. Full credit if Booking.com is used OR if an uncontrollable blocker occurs (e.g., CAPTCHA, login requirement, geo-block, site outage) and the agent clearly reports it and what was attempted. Partial credit if the agent switches to another site without first attempting Booking.com and without documenting a Booking.com blocker.

Description Recognize that the stated dates “Nov 20–14, 2025” are invalid/ambiguous and either (a) explicitly flag the issue and choose a reasonable interpretation consistent with a 4-night stay (e.g., Nov 20–24, 2025) while stating the assumption, or (b) report inability to proceed due to ambiguity if the agent cannot make a defensible assumption. Full credit for a clearly stated, reasonable interpretation; partial credit if the interpretation is unclear but results still reflect a 4-night stay.

Description Enter the task parameters into Booking.com: destination San Francisco, California; 1 adult; and a 4-night stay using the interpreted dates from the prior step. Full credit if these parameters are applied correctly OR if Booking.com prevents setting one of them due to site limitations and the agent clearly reports the limitation. Partial credit if one parameter is wrong but corrected later or clearly acknowledged.

Description Determine and report the cheapest available property shown on Booking.com for the specified search inputs. Full credit if the agent sorts/filters by lowest price (or otherwise provides clear evidence it is the cheapest among visible results) and reports the displayed price context (total stay or per-night as shown, and any key fee/tax notes if displayed). Full credit if Booking.com shows no availability for those dates and the agent accurately reports that. Partial credit if a low-priced option is provided but the method to ensure it is cheapest is unclear, or if price context is incomplete due to missing display elements outside the agent’s control.

Description Report the hotel’s address as shown on Booking.com (or the hotel’s official listing if Booking.com does not display it). Full credit if the address is sufficient to geolocate (street address + city/state; ZIP if available). Partial credit if only a partial but still identifying address is available due to external page limitations and the agent reports that.

Description Using the hotel address, use a reasonable mapping/directory source (e.g., Google Maps, Apple Maps, OpenStreetMap/MapQuest/Yelp) to identify the nearest grocery store and provide the store’s name and full address. Full credit if the closest grocery store is identified based on the mapping results available at the time; full credit also if mapping results are unavailable/blocked and the agent clearly reports the blocker and what was attempted. Partial credit if a plausible nearby grocery store is provided but the method for determining it is closest is not stated or the address is incomplete.

Description Search for any AI-related conferences or workshops occurring in San Francisco during the current month. Partial credit if only some events are found or if events outside the month are included.

Description For each identified event, provide the exact date(s) and venue or location in San Francisco.

Description Use Google Flights to search for a round-trip flight matching the dates of the selected summit. Partial credit if only one direction or incorrect dates are retrieved.

Description Present one or more round-trip flight options including departure and return dates and times, airlines, number of stops, and price. Partial credit for incomplete details.

Description Search for and identify at least one AI-focused conference or workshop taking place in San Francisco during the current month. Partial credit may be awarded for attempts that find related events but outside San Francisco or not in the current month, or for clearly stating that no such events are available this month.

Description Provide the date and location (venue or neighborhood) for the identified event(s). Partial credit may be awarded if only the date or only the location is provided, or if details are incomplete but a reasonable attempt is made.

Description Access and perform the flight search specifically on Google Flights as requested. Partial credit may be awarded if the agent explains inability to access Google Flights but still outlines the steps or results consistent with using Google Flights.

Description Set up a round-trip search from Toronto (YYZ) to San Francisco on the summit date. Partial credit may be awarded for correct origin/destination but misaligned date, for using San Francisco area airports (e.g., SFO/OAK/SJC), or for clearly stating the date is ambiguous and making a reasonable assumption.

Description Present a specific round-trip flight option that aligns with the summit date and is reasonable (e.g., feasible timing and typical pricing). Partial credit may be awarded for listing search results without selecting a specific itinerary, or if only one-way options appear. Full credit is awarded if the agent indicates no viable flights are available and states this clearly.

Description Demonstrate stopping before the booking step—do not enter personal information or complete checkout. Partial credit may be awarded for explicitly stating the intent to stop prior to booking even if the process is not fully shown.

Description Search for AI-related conferences or workshops occurring in San Francisco during the current month. Full credit if at least one clearly AI-relevant event is found OR if, after reasonable search effort (e.g., checking multiple sources/queries), the agent reports that no qualifying events were located. Partial credit if only AI-adjacent events are found or if the effort appears limited (e.g., a single quick query) but still reports findings.

Description For each found event (or at minimum the one used as the travel anchor), provide the event date(s) and the location in San Francisco (venue/address when available). Full credit for clearly providing both date and the most specific location available from the listing; if the listing does not provide a venue/address, city-level location plus the source context is sufficient. Partial credit if only one of date or location is provided despite availability, or if the location is needlessly vague when a venue is available.

Description Explicitly identify the chosen event’s “summit date” (or start date) and use it to set the outbound (and a reasonable return) date for the flight search. Full credit if the linkage is clear and dates are aligned to the selected event. Partial credit if an adjacent/approximate date is used with some explanation (e.g., arriving the day before). No credit if flight dates are unrelated to any found event date when an event date was available.

Description On Google Flights, search for a round-trip flight from Toronto (YYZ) to San Francisco (SFO) with the outbound aligned to the summit date (and a reasonable return date), and select a viable itinerary (e.g., reasonable duration/layovers). Full credit if a viable itinerary is selected and the agent stops before any binding booking step (no purchase/checkout; no entry of personal/payment info). Full credit also if Google Flights is inaccessible (CAPTCHA/login/region restrictions/site down) or no flights are available for the required dates/route, provided the agent demonstrates a reasonable attempt and clearly reports the blocker/unavailability, stopping without attempting to book elsewhere unless asked. Partial credit if the agent uses another platform without first attempting Google Flights when Google Flights appears accessible, or if the itinerary is only marginally viable without explanation.

Description Access the USAJobs.com website and correctly apply the filter or search parameter to show only job listings requiring a Secret security clearance. Partial credit if the site is accessed but the clearance filter is applied incorrectly.

Description From the filtered results, find and clearly identify the very first job in the list that displays an annual salary. Partial credit if a job with salary is found but it is not the first in the list or the salary detail is misread.

Description Take the annual salary amount for the identified job and use a secondary tax-calculation tool to determine the after-tax take-home pay. Partial credit if the calculation is attempted but some tax parameters are missing or the tool is used incorrectly.

Description Navigate to USAJobs.com and perform a search/filter so that the results show jobs requiring Secret security clearance. Partial credit for accessing the site or searching but not clearly filtering to Secret clearance.

Description From the Secret clearance results, scan in the displayed order and select the first posting that explicitly shows an annual salary (per year). Provide the job title and the annual salary figure. Partial credit if a Secret job is identified but not the first in the list, or if the salary is captured but its period (annual vs hourly) is unclear. Full credit if no postings list annual salary and this is clearly stated.

Description Use an external calculator/tool to estimate after-tax take-home pay based on the identified annual salary. Include any necessary assumptions (e.g., location/state, filing status) or clearly note limitations if required inputs are unavailable. Partial credit for using a tool but omitting key assumptions or providing only a federal estimate without noting limitations.

Description Attempt to use USAjobs.com to search for jobs and target roles requiring a 'Secret' security clearance (via filters or query terms). Full credit if the agent reaches USAjobs and makes a reasonable attempt but is blocked (CAPTCHA/login/region block/site down) and clearly reports what happened and what it tried. Partial credit if the attempt is unclear or uses a non-USAjobs source without first attempting USAjobs.

Description From USAjobs, produce a results list that is clearly filtered to (or strongly targeted toward) jobs requiring 'Secret' clearance. Full credit if the results view shows the Secret clearance filter applied or the listings clearly indicate Secret. Partial credit if results are only loosely related (e.g., general security jobs) or the Secret requirement is not verified due to limited page visibility, while the agent explains the limitation.

Description Using the Secret-clearance results list in the order presented by USAjobs at the time (noting the sort order if visible), select the first listed job that explicitly shows an annual salary (or an annual salary range) either on the results card or after clicking into the first few listings as needed. Record the job title and the annual salary amount/range used for later computation. Full credit if the job is the first qualifying one given the visible ordering and the salary is read correctly. If none of the visible Secret-clearance listings show an annual salary (e.g., only hourly/unclear) or the site requires extra clicks to reveal pay, full credit if the agent clearly reports this and chooses the earliest listing where annualized pay can be reasonably derived/shown (explaining the derivation) or states that no annual salary is available from the accessible information. Partial credit if the selected job is Secret-clearance but not the first qualifying one when the first is available, or if the salary is slightly mis-copied.

Description Use a tool distinct from USAjobs (e.g., a paycheck/tax calculator website or spreadsheet) to estimate after-tax take-home pay for the selected annual salary (explicitly stating whether using the min, max, or midpoint of a range). The agent must state key assumptions that materially affect taxes (at minimum: filing status and state/location, or explicitly that a default state was assumed due to missing location info). Full credit if a distinct tool is used and a take-home estimate is reported with assumptions. Full credit also if the tool is inaccessible/blocked and the agent clearly reports the blocker and uses a reasonable alternative method (another calculator or transparent manual estimation). Partial credit if assumptions are unclear or the tool used is not clearly distinct.

Description Navigate to LinkedIn.com and perform a search specifically for the job title 'Computer Vision Researcher'. Partial credit for accessing LinkedIn without performing the correct search.

Description Apply a location filter to show only job postings based in Seattle. Partial credit if Seattle is selected but other locations are still included.

Description Apply a date-posted filter so that only jobs posted within the last week are shown. Partial credit if date filter is applied incorrectly (e.g., showing older posts).

Description Provide the list of 'Computer Vision Researcher' positions in Seattle posted in the past week, including at least the job title, company name, posting date, and link. Partial credit for incomplete details or fewer postings.

Description Find and report the most recent computer vision course offered by Stanford that is available for free online, including the course name and link. Partial credit if course is identified but link or free status is missing.

Description Access and operate on LinkedIn.com as the specified site to perform the job search. Partial credit if the agent attempts to reach LinkedIn but cannot access it and clearly states the limitation.

Description Search for roles with the title 'Computer Vision Researcher' in Seattle on LinkedIn and apply the 'posted in the past week' time filter. Partial credit if some but not all filters are correctly applied (e.g., correct location but wrong time window, or correct time window but imprecise title).

Description Present the matching job postings found (titles and links are acceptable), or explicitly state that no such postings exist in the past week. Partial credit if results are mentioned but missing clarity (e.g., missing dates or links).

Description Find and name the latest computer vision course from Stanford that is available for free online, and provide an access link. Partial credit if a Stanford computer vision course is provided but its recency is unclear or it is not confirmed as free.

Description Navigate to LinkedIn.com Jobs and attempt a search for the keywords 'Computer Vision Researcher'. Full credit if the agent makes a clear attempt but is blocked by login/CAPTCHA/geo restrictions or LinkedIn is otherwise inaccessible, and it reports the blocker and what was attempted. Partial credit if the agent searches LinkedIn but in a non-jobs area or with unclear query.

Description From the LinkedIn Jobs search, apply (or attempt to apply) the location filter to Seattle and the date filter to 'Past week'. Full credit if all constraints are correctly applied OR if the agent cannot apply them due to LinkedIn restrictions (login/CAPTCHA/limited UI access) but clearly explains which filters could not be set and why. Partial credit if only one of the two filters is correctly applied when access is available, or if the intended filters are stated but not actually reflected/attempted.

Description From the filtered results (keywords + Seattle + past week), summarize the matching postings demonstrating review of recency: include job title, company, and posted time/date (e.g., 'X days ago') plus any visible location/remote details. Full credit if multiple postings are listed with recency evidence consistent with 'past week', OR if the filtered search shows zero results and the agent clearly reports that, OR if LinkedIn access is blocked and the agent states it cannot view postings despite attempting. Partial credit if only one posting is provided or if recency evidence is missing but the posting otherwise appears to match the role/location intent.

Description Identify a Stanford computer vision course with free online access (e.g., publicly available lecture videos/materials or a platform that can be accessed for free such as audit/free course materials). Provide the course name and hosting source, and justify why it is the 'latest' using the best available evidence (most recent term/year on the course site, most recent playlist upload date, or last-updated timestamp). Full credit if the selection is clearly Stanford + computer vision + free, and the 'latest' claim is supported with cited recency evidence or the agent explains that multiple Stanford CV offerings exist and picks the most recent one based on available date/version signals. Partial credit if Stanford/free/CV is met but 'latest' is weakly supported or unclear.

Description Briefly connect the course topics to role-relevant skills for Computer Vision Researcher positions (e.g., CNNs/transformers for vision, detection/segmentation, self-supervised learning, optimization/training, evaluation/metrics, datasets, research reading). Full credit if the mapping is specific and relevant; partial credit if generic but still plausibly related to CV roles.

Description Identify and provide the title or URL of the first article published on searchengineland.com as specified in the task.

Description Provide a clear and concise summary of the main point or key takeaway from the identified article. Partial credit can be awarded for capturing some but not all core ideas.

Description Locate and cite another article from a different website that supports or verifies the key takeaway. Partial credit if the article is relevant but the connection to the takeaway is weak.

Description Access searchengineland.com and identify the 'first article published' as requested. Full credit if the agent clearly identifies a specific article as the first (using a reasonable interpretation), or explains any ambiguity and chooses a defensible approach. Partial credit if the agent attempts to locate it but cannot determine it definitively and transparently states limitations.

Description Provide a clear and accurate summary of the primary takeaway from the identified article. Partial credit if the summary is generally correct but misses important nuances or is overly vague.

Description Identify another article from a different website (not searchengineland.com) that supports or verifies the key takeaway. Full credit if the source clearly corroborates the takeaway. Partial credit if the source is from a different site but only tangentially related or offers weak support.

Description Determine the earliest (first by publication date) article on searchengineland.com and report enough to disambiguate it (e.g., title + date and/or author). Full credit if the agent correctly identifies the earliest article based on verifiable on-site evidence (e.g., archive page, pagination to earliest, site search results ordered oldest-first). Full credit also if the agent makes reasonable attempts to locate an authoritative archive/oldest ordering but cannot conclusively determine the earliest article due to external limitations (e.g., archives missing, non-chronological pages, infinite scroll, blocked by CAPTCHA/paywall, site downtime) and clearly explains what was tried and why the earliest article cannot be confirmed. Partial credit if the agent identifies an early article but does not justify why it is the first/earliest or relies on a page that is not shown to be ordered by oldest publication.

Description Provide a concise summary of the central message of the identified first Search Engine Land article. Full credit if the summary matches the article’s main point and avoids adding unsupported claims. Partial credit if the summary is overly vague or contains minor inaccuracies. If the agent could not access the article content due to external blockers, full credit may be earned by accurately reporting the access limitation and summarizing only what is reliably available (e.g., snippet/abstract) while clearly labeling uncertainty.

Description Locate a source on a different domain (not searchengineland.com) that substantively supports/verifies the key takeaway. Full credit if the second source is an article (or similarly authoritative publication) from another site and clearly corroborates the same claim/recommendation. Full credit also if the agent performs a reasonable search (e.g., targeted queries, checking a few credible publications) but cannot find a clear corroborating article or is blocked by external factors (paywalls/CAPTCHA/removals), and clearly reports the search approach and limitation. Partial credit if the second source is only loosely related, not clearly independent, or not substantively corroborative.

Description Explicitly connect the second article’s content to the first article’s takeaway with specific overlap (e.g., matching factual claim, similar guidance, confirming statement/data). Full credit if the linkage is concrete and accurate. Partial credit if the explanation is high-level or only implies support without pointing to the shared point. If no corroborating source could be found due to external constraints, full credit may be earned by clearly stating that and explaining why verification could not be completed.

Description Locate an upcoming jazz event in Los Angeles that lists multiple performers and occurs soon. Partial credit for proposing an event with incomplete details; full credit for event name, date, venue, and artist lineup.

Description Determine which performer is billed as the headliner for the identified event. Partial credit for choosing a plausible artist; full credit for correctly naming the official headliner.

Description Navigate to Spotify.com and locate a track by the chosen headline artist. Partial credit for finding the artist’s Spotify page; full credit for selecting a specific song.

Description Successfully initiate playback of the chosen song on Spotify.com. Partial credit for attempting playback; full credit if the song begins playing without errors.

Description Find and identify a jazz event that is (a) in Los Angeles, (b) happening in the near future/upcoming, and (c) features multiple artists on the bill. Full credit if all constraints are clearly satisfied. Partial credit if one or more constraints are unclear or unmet (e.g., location is LA but not clearly near future, or only a single artist). If no such event can be found, partial to full credit may be awarded if the agent explicitly states that after a reasonable search.

Description Identify the headliner for the event found. Full credit if the headliner is explicitly identified from the lineup; partial credit if the agent makes a reasonable selection when the headliner is not explicitly labeled and provides rationale.

Description Navigate to Spotify.com and locate the headline artist, then identify a specific song by this artist. Full credit if a valid artist page and song are found on Spotify.com. Partial credit if the artist is found but not a specific song, or if the song is identified but not on Spotify.com.

Description Initiate playback of the identified song on Spotify.com. Full credit if playback is started or if the agent clearly indicates any platform limitations (e.g., login required) and acknowledges the constraint. Partial credit if instructions to play are provided but playback is not attempted.

Description Attempt to find an event listing/page for a near-future jazz event in Los Angeles. Full credit if the agent reaches a credible event page/listing or if reasonable attempts are made but the agent is blocked by external factors (e.g., paywall, site down, CAPTCHA) and clearly reports the blocker. Partial credit if the agent searches but the event page/listing is not clearly reached or is not credible.

Description From the listing/page, verify the event is (a) in Los Angeles, (b) jazz, (c) upcoming in the near future, and (d) features multiple artists on the bill/lineup. Full credit if all are clearly supported by the listing, OR if after reasonable effort no clearly qualifying multi-artist near-future LA jazz event can be found/verified due to limited/ambiguous information and the agent clearly reports this and selects the closest alternative that best matches the primary intent (upcoming LA jazz event). Partial credit if the event is LA and jazz but the near-future timing or multi-artist nature is unclear while better-verified options appear available.

Description Correctly identify the headline/top-billed artist from the event’s lineup. Full credit if headliner status is clearly indicated on the event page/listing and the agent selects that artist, OR if headliner billing is not clearly indicated/ambiguous despite reasonable inspection and the agent explicitly states the ambiguity and selects a reasonable presumed headliner (e.g., first-billed/featured). Partial credit if the chosen artist is plausible but the agent provides no basis for headliner determination when the listing provides clearer billing.

Description Use Spotify.com to locate the selected headline artist and attempt to start playback of one of their songs. Full credit if playback is initiated, OR if the agent reaches the correct artist/track page on Spotify.com and clearly reports an external blocker preventing playback (e.g., login requirement, CAPTCHA, region restriction, autoplay/browser limitation, site error). Partial credit if the correct artist is found on Spotify.com but playback is not attempted when it appears possible.

Description Use Google Maps to find the driving distance in miles from Manchester Airport to Etihad Stadium. Partial credit may be awarded for approximate distances; full credit for the correct value.

Description Use Google Maps to find the driving distance in miles from the George Washington Bridge to the New York Stock Exchange. Partial credit may be awarded for approximate distances; full credit for the correct value.

Description Determine and clearly state whether the Manchester Airport–Etihad Stadium distance is longer or shorter than the George Washington Bridge–NYSE distance, based on the obtained mileages.

Description Explicitly use Google Maps to obtain the driving distances. Partial credit if distances are provided but the source (Google Maps) is not stated.

Description Correctly interpret 'Manchester Airport' and 'Etihad Stadium' in Manchester, UK as the endpoints. Partial credit if endpoints are plausible but ambiguous or misidentified.

Description Provide the driving distance in miles per Google Maps for Manchester Airport to Etihad Stadium. Partial credit if a distance is given but not in miles or not clearly a driving distance.

Description Correctly interpret 'George Washington Bridge' and 'NYSE' (New York Stock Exchange at 11 Wall St, Manhattan) as the endpoints. Partial credit if endpoints are plausible but ambiguous or misidentified.

Description Provide the driving distance in miles per Google Maps for George Washington Bridge to the NYSE. Partial credit if a distance is given but not in miles or not clearly a driving distance.

Description State clearly whether the Manchester route is longer or shorter than the NYC route, consistent with the reported distances. Partial credit if a comparison is attempted but unclear or inconsistent.

Description Determine the driving distance in miles between Manchester Airport and Etihad Stadium using Google Maps directions. Full credit if the agent reports a clear miles value from Google Maps for a driving route (optionally noting the chosen route). Partial credit if the agent provides an estimate without Google Maps, provides distance in the wrong unit without converting to miles, or gives transit/walking distance instead of driving when driving is available. Full credit if Google Maps is inaccessible (e.g., blocked/CAPTCHA) and the agent clearly reports the blocker and uses a reasonable alternative mapping source to obtain driving miles.

Description Determine the driving distance in miles between the George Washington Bridge and the New York Stock Exchange (NYSE) using Google Maps directions. Full credit if the agent reports a clear miles value from Google Maps for a driving route. Partial credit if the agent uses a different start/end location than specified (e.g., wrong bridge/NYSE location), gives distance in the wrong unit without converting, or uses a non-driving mode without stating/justifying why. Full credit if Google Maps is inaccessible and the agent clearly reports the blocker and uses a reasonable alternative mapping source for driving miles.

Description Based on the two obtained driving distances, explicitly state whether the Manchester Airport  Etihad Stadium drive is longer or shorter than the George Washington Bridge  NYSE drive. Full credit if the comparison is correct and clearly stated. Partial credit if the agent provides both distances but gives an ambiguous comparison or makes an arithmetic/comparison mistake despite having the right numbers. No credit if the agent omits the comparison entirely.

Description List exactly three jazz clubs located in Chicago. Partial credit if fewer than three are listed or if non-jazz venues are included.

Description Provide the correct neighborhood for each of the three listed jazz clubs. Partial credit for correctly identifying some but not all neighborhoods.

Description Alphabetically sort the three neighborhoods and correctly identify which neighborhood comes first in the list.

Description Navigate to Booking.com (or equivalent) and enter the selected neighborhood, December 28, 2025 check-in date, one-night stay, and two adults as search criteria. Partial credit if some parameters are omitted or incorrect.

Description From the Booking.com search results, find and present the hotel with the lowest price for the specified date, neighborhood, and party size. Partial credit if the hotel is identified but price details are missing or unclear.

Description Provide the names of three jazz clubs located in Chicago. Partial credit if fewer than three clubs are identified or if one is misclassified (not a jazz club).

Description Specify the Chicago neighborhood for each of the three clubs. Partial credit if some neighborhoods are missing or incorrect.

Description Alphabetically sort the three neighborhoods and clearly identify the first neighborhood to be used for the hotel search. Partial credit if a neighborhood is selected but sorting is not demonstrated or is incorrect.

Description Access and use Booking.com (not another platform) to perform the hotel search as specified.

Description Set the search for the selected neighborhood in Chicago, with check-in on December 28, 2025 and a one-night stay (implying check-out December 29, 2025) for 2 adults. Partial credit if some parameters (date, length of stay, occupancy, or neighborhood) are set but not all.

Description From the Booking.com results for the specified criteria, identify the lowest-priced hotel option in the selected neighborhood and provide the hotel's name and price. Full credit if there are no available hotels and this is clearly stated. Partial credit if a hotel is provided without confirming it is the least expensive or if price details are missing.

Description Agent identifies exactly three distinct jazz clubs that are located in Chicago. Full credit if all three are real, operating/known venues in Chicago. Partial credit if one club is not actually a jazz club (e.g., general music venue) or is outside Chicago city limits but nearby, or if fewer/more than three are provided. Full credit is also acceptable if the agent reasonably reports that a candidate venue has closed/changed format (external change) and replaces it with another valid Chicago jazz club.

Description Agent provides the Chicago neighborhood for each of the three identified clubs. Full credit if neighborhoods are correct and clearly paired to each club. Partial credit if one neighborhood is wrong/overly broad/unclear (e.g., only 'Downtown' without a neighborhood) or if only 2 of 3 neighborhoods are provided. Full credit is also acceptable if neighborhood naming is reasonably ambiguous (e.g., commonly used sub-neighborhood vs official community area) and the agent provides a defensible rationale.

Description Agent sorts the three neighborhoods alphabetically (by the neighborhood names it provided) and correctly identifies which neighborhood is first in that sorted order, then uses that neighborhood for the hotel search. Full credit if the chosen neighborhood is demonstrably the first alphabetically among the three. Partial credit if sorting is attempted but a tie/variant naming causes ambiguity (e.g., 'Near North Side' vs 'River North') and agent explains rationale.

Description Agent makes a good-faith attempt to use Booking.com to search lodging in the selected neighborhood for a one-night stay on December 28, 2025 for 2 adults. Full credit if Booking.com is used OR if the agent is blocked by CAPTCHA, outage, region restrictions, paywall/login wall, or other access limitation and clearly reports the blocker. Partial credit if the attempt is unclear or uses a different platform without first attempting Booking.com and without reporting why Booking.com could not be used.

Description Within Booking.com (if accessible), the agent applies the correct parameters: Dec 28, 2025 check-in for 1 night (check-out Dec 29, 2025) and 2 adults, and constrains results to the selected neighborhood (or the closest available neighborhood/area filter Booking.com provides, clearly explained). Full credit if parameters are correct or if the site UI prevents exact neighborhood filtering and the agent transparently uses the closest equivalent area constraint available. Partial credit if one parameter (date/occupancy/area) is incorrect while others are correct.

Description Agent identifies the least expensive available hotel shown by Booking.com for the specified neighborhood, one-night stay on Dec 28, 2025, for 2 adults. Full credit if the agent sorts by price (or otherwise verifies lowest price) and reports the hotel name and the price as displayed (noting whether it includes taxes/fees as shown). Full credit if Booking.com shows no properties available for those parameters and the agent clearly reports no availability. Full credit if Booking.com is inaccessible/blocked and therefore the agent cannot determine the least expensive option, as long as the agent clearly reports this limitation (optionally providing a best-effort alternative from another source clearly labeled as not from Booking.com). Partial credit if the agent provides a plausible low-price option but does not clearly establish that it is the lowest among visible results when the site is accessible.

Description Research and determine which men's face wash is ranked or highlighted as the best by either GQ or Men's Health. Partial credit if a close alternative from the specified sources is provided; full credit if the exact product and source are clearly specified.

Description Search Amazon.com to find the exact product identified in the first step. Partial credit if a matching listing is found but details (size, formulation) differ; full credit if the precise product is located or noted as unavailable.

Description Demonstrate the process of adding the located product to the Amazon cart without completing checkout. Full credit for successfully adding to cart; partial credit if the item is placed in a wish list or cart but quantity or variation is incorrect.

Description Access and reference either GQ or Men's Health to locate their content specifically identifying the 'best men's face wash.' Partial credit may be awarded for accessing one of the sites but not reaching the specific 'best' recommendation. Full credit if the agent determines the source does not provide a singular 'best' and clearly states this.

Description Extract the exact product (brand, name, and variant if specified) designated by the chosen source as the 'best men's face wash.' Partial credit may be awarded if a recommended product is identified but it is not explicitly the 'best' or if the variant is unclear.

Description Locate the product listing on amazon.com that matches the identified item (brand and correct variant). Partial credit may be awarded for a close match or providing a related product link. Full credit is awarded if the exact product is unavailable on Amazon and the agent clearly communicates this.

Description Assist with the buying step up to adding the item to the Amazon cart (or provide clear steps to do so) without attempting checkout or entering any personal information. Partial credit may be awarded for reaching the product page and preparing to add to cart.

Description Determine a product recommended as the best (e.g., 'best overall' or an equivalent primary/top pick) men's face wash from either GQ or Men's Health, and clearly identify the product name and that the recommendation came from GQ/Men’s Health. Full credit if the agent cites a current/clearly relevant GQ/Men’s Health grooming list and selects the top/overall pick (or, if multiple 'best' categories exist, chooses one defensible primary pick and explains which category it came from). Full credit also if the agent attempts to access GQ/Men’s Health but is blocked by paywall/CAPTCHA/site error and clearly reports the blocker and what could/couldn’t be verified. Partial credit if the source is GQ/Men’s Health but the 'best' status is ambiguous (e.g., not clearly a top pick) or the selection rationale is unclear. No credit if the source is neither GQ nor Men’s Health or if the product chosen is not a face wash.

Description Search for and locate the same identified face wash product on Amazon.com. Full credit if the Amazon listing matches the chosen product (same brand/product line; size/pack count may vary) and is a face wash. Full credit also if the exact match is not available (out of stock, delisted, region-restricted) and the agent selects the closest clearly related variant (e.g., different size or multipack of the same face wash) and explicitly notes the mismatch/unavailability. Partial credit if the agent finds the correct product but cannot confidently verify it matches (unclear listing/variant confusion) or chooses a less-close variant while a closer match is available. No credit if the agent selects an unrelated product type (e.g., moisturizer) or an unrelated face wash when the correct one is available.

Description Add the selected face wash to the Amazon cart (or equivalent buy flow) and stop before entering any personal/shipping/payment information or placing the order. Full credit if the item is added to cart and the agent stops at the cart or sign-in/checkout boundary. Full credit also if the agent makes a reasonable attempt but is prevented from adding to cart or proceeding due to external blockers (login wall, CAPTCHA, site error, seller restrictions, or out-of-stock) and clearly reports the blocker and where the flow stopped. Partial credit if the agent reaches the correct listing but does not attempt add-to-cart despite it being possible. No credit if the agent attempts to complete checkout/purchase or enters/makes up personal details.

Description Agent must locate and provide the full street address of the Bright Future Forever office located in Seattle, WA. Partial credit if the city is correct but the street address or zip code is missing or incomplete.

Description Agent must name one dentist (DDS) who works at the dental office across the street from Bright Future Forever and specify where that dentist graduated from for their undergraduate degree. Partial credit for providing only the name or only the institution; full credit for both.

Description Provide the correct street address for the office of 'Bright Future Forever' based in Seattle, WA. Partial credit if the agent identifies the business but provides incomplete or approximate address details, or surfaces multiple possible addresses with reasoning. Full credit is acceptable if the business cannot be found or verified and the agent clearly states that.

Description Determine which dental office is directly across the street from the identified 'Bright Future Forever' address. Partial credit if the agent identifies a nearby dental office with reasonable justification but cannot confirm it is directly across the street, or provides multiple candidates with rationale. Full credit is acceptable if no such office exists and the agent clearly states that.

Description Name at least one DDS who works at the identified across-the-street dental office and state where they completed their undergraduate degree. Partial credit if only the DDS name is provided without the undergraduate school, or if the undergraduate school is provided but the DDS name or employment at the office is not clearly established. Full credit is acceptable if this information is not publicly available and the agent clearly states that.

Description Determine and report the physical street address (including suite/unit and ZIP if available) for the office of 'Bright Future Forever' located/based in Seattle, WA. Full credit if a verifiable address is clearly provided and tied to the correct organization. Partial credit if only a partial address is found or if the Seattle connection is plausible but not clearly confirmed. Full credit if, after reasonable searching across multiple sources (e.g., official site, business listings, Washington filings, reputable directories), no verifiable physical address is publicly listed or results are conflicting and the agent clearly reports this (without guessing).

Description Using the located 'Bright Future Forever' address, identify a dental office directly across the street (opposite side of the same street) and report its name and address. Full credit if the across-the-street relationship is supported by map/address evidence (e.g., corresponding address ranges, map pin positions, street-view confirmation). Partial credit if the agent identifies a nearby dental office but does not substantiate it is across the street. Full credit if the across-the-street dental office cannot be reliably determined due to insufficient address precision, map ambiguity, multiple plausible candidates, or inaccessible mapping data, as long as the agent explains the ambiguity and does not guess.

Description Report (1) the name of at least one dentist with the DDS credential who works at the identified dental office and (2) where that DDS graduated from undergrad (college/university), citing a reasonable public source (e.g., office bio, professional profile). Full credit if both the DDS name and undergraduate institution are correctly given and clearly attributable to that office. Partial credit if only the DDS name is provided, or if the education listed is not clearly undergraduate, or if the DDS credential/association to the office is unclear. Full credit if the agent makes a reasonable attempt to find the undergrad institution but it is not publicly available (or sources are inaccessible) and the agent clearly reports that it could not be found without guessing. If criterion (2) cannot be completed due to indeterminate 'across the street' identification, full credit is earned by explicitly stating the dependency and providing the best-supported nearest-candidate analysis without asserting it is across the street.

Description Successfully navigate to the official City of New York jobs website without attempting to log in or apply for a position.

Description Use the site’s search functionality or filters to locate job listings titled "Compliance Specialist." Partial credit if the search is attempted but yields no results; full credit if a relevant listing is found.

Description From the Compliance Specialist listing, find and note the salary range, then determine the highest salary figure in that range.

Description Navigate to smartasset.com and locate the appropriate take-home pay or paycheck calculator without entering any personal or sensitive information.

Description Input the maximum salary value into the calculator, set the location to New York City (and other defaults as needed), and record both the net (take-home) pay and the effective tax rate.

Description Navigate to and use the official City of New York jobs site (NYC Jobs) as specified in the task. Partial credit if the agent searches but does not clearly indicate the official NYC jobs portal.

Description Find at least one job posting titled 'Compliance Specialist' (or clearly equivalent compliance role explicitly titled as such) on the NYC Jobs site and extract the stated salary range from the posting. Include a reference (e.g., job title and link or requisition ID) to the specific listing. Full credit also awarded if no such listing exists and the agent explicitly states that none are available on NYC Jobs at this time.

Description From the identified salary range, correctly select the maximum salary figure to be used for subsequent calculations. Partial credit if the salary range is identified but the max end is misread or not clearly stated. If no listing exists, full credit awarded for explicitly noting that a maximum salary cannot be determined.

Description Use SmartAsset's calculator to compute take-home pay using the maximum salary figure. Partial credit if an alternative calculator is used or SmartAsset is consulted but the calculation is incomplete. If no listing exists (and thus no salary), full credit may be awarded for clearly stating that the calculation cannot be performed due to lack of a salary figure.

Description Provide the computed take-home pay and the effective tax rate as requested. Partial credit if only one of the two values is provided or values are reported without clear identification. If no calculation can be performed due to no available salary, full credit may be awarded for explicitly stating this limitation.

Description Locate an actual job posting titled 'Compliance Specialist' on the NYC Jobs site for the City of New York and clearly identify it (e.g., agency/department and that it is a NYC government role). Full credit if the agent finds and identifies such a posting OR, after a reasonable search (including using site search/filters and/or a web search), clearly reports that no such posting exists at the time. Partial credit if the agent finds a closely related title (e.g., 'Compliance Officer') or finds the correct title but cannot confirm it is on the NYC Jobs City of New York site due to access limitations. Full credit if the site is inaccessible (e.g., down/CAPTCHA) and the agent documents the attempt and limitation.

Description From the identified job posting, extract the posted salary range and correctly select the maximum (top) value. Full credit if the salary range and chosen maximum are stated correctly and clearly tied to the posting. Partial credit if the agent identifies compensation but it is ambiguous (e.g., hourly vs annual not clear) and the agent states the ambiguity and a defensible interpretation, or if access issues prevent viewing the full range but the agent reports the limitation. No credit if the salary figure is fabricated or not sourced/grounded in the posting when the posting is accessible.

Description Use SmartAsset.com to compute take-home pay and effective tax rate for the maximum salary from the posting, using NYC as the location (and any necessary assumptions explicitly stated, e.g., filing status). Full credit if the agent uses SmartAsset and reports both take-home pay and effective tax rate consistent with the inputs. Full credit if SmartAsset is inaccessible/blocked (CAPTCHA, outage, paywall) but the agent clearly documents the attempt and limitation; in that case, partial credit if the agent provides a clearly-labeled alternative estimate method/source (not claimed to be SmartAsset) and explains the assumptions. No credit if the agent reports numbers as 'from SmartAsset' without evidence/consistency or fabricates outputs.

Description Use Reddit's search function or browse relevant subreddits to find a post or thread about blues clubs in New Orleans.

Description Review the comments in the identified thread and record the first mention of a specific blues club and the username of the commenter who posted it.

Description Navigate to the commenter’s Reddit profile page and extract the text of their latest comment.

Description Analyze the content of the user's comment history for any indication (e.g., local references, events, or geotags) that suggests they reside in Louisiana.

Description Provide the text of the user's most recent comment and clearly state whether it appears from their comments that they live in Louisiana.

Description Access Reddit and search for 'blues club in New Orleans' to locate a post with comments discussing blues clubs in New Orleans. Partial credit if a related thread is found but is not clearly about blues clubs in New Orleans or lacks usable comments.

Description Read the comments of the identified Reddit post and determine the first blues club mentioned, along with the username of the commenter who mentioned it. Partial credit if a club is identified but the 'first' is ambiguous or the username is missing.

Description Navigate to that user's Reddit profile and report their most recent comment. Partial credit if the profile is accessed but the most recent comment cannot be determined (e.g., profile is private, no comment history, or deleted content) and this is clearly stated.

Description Review the user's comments to evaluate whether it appears they live in Louisiana, answering yes/no/uncertain with a brief rationale derived from their comment history. Partial credit if an assessment is made but with limited or weak evidence.

Description Agent attempts a Reddit search (native Reddit search or web search limited to Reddit) for “blues club in New Orleans” (or a very close variant) and opens a thread with a comments section discussing blues clubs in New Orleans. Full credit if Reddit is inaccessible due to login/CAPTCHA/outage and the agent clearly reports the blocker and what could not be accessed after reasonable attempts. Partial credit if the query is meaningfully different but still yields a clearly relevant New Orleans blues-club comments thread.

Description From the opened thread, agent identifies the first blues club mentioned based on the comment order as displayed to the agent, and names the user who mentioned it. Agent should indicate the comment sort/order used (e.g., best/top/new) or note if order is ambiguous/unstable. Full credit if the agent correctly follows the displayed order or, if the platform prevents determining a stable 'first' (e.g., collapsed comments, sort changes, login wall), the agent explains the limitation and uses the best-available interpretation from what is visible. Partial credit if a plausible club is identified but 'first' ordering is not verified or the sort/order is not stated.

Description Agent navigates to the identified user’s Reddit profile and finds the most recent comment shown (typically in the Comments tab, sorted by New). Full credit if the agent accurately reports the most recent comment content (quote or precise paraphrase) and where it appears, OR if the profile/comments are inaccessible (deleted/suspended, NSFW/login wall, CAPTCHA/outage) and the agent clearly reports the blocker and what could/couldn’t be verified. Partial credit if the agent reaches the profile but the reported comment is not demonstrably the most recent due to sorting confusion or missing evidence.

Description Using evidence from the user’s accessible comment history, provide a reasoned determination (yes/no/unclear) about whether it appears they live in Louisiana. Full credit if the agent cites specific comment evidence (explicit location statements, consistent local references, etc.) or clearly states that the history is insufficient/unavailable to infer location due to access limits. Partial credit if the agent gives a conclusion with weak/uncited support while stronger evidence is available.

Description The agent should correctly identify all three dynasty names explicitly mentioned in Broadway theater ownership context. Partial credit may be awarded if one or two names are correct; full credit only if all three are accurately listed.

Description The agent should accurately state how many Broadway theaters are owned by each of the three dynasties. Partial credit can be given for each correct count; full credit only if all three ownership numbers are correctly matched to their respective dynasties.

Description Find and name the three entities that preside over Broadway theater houses (the major Broadway theater-owning groups). Partial credit may be awarded for correctly naming one or two; full credit requires all three.

Description For each of the three named dynasties, provide how many Broadway theaters they own. Partial credit may be awarded for each correctly matched dynasty-to-count pair; full credit requires accurate counts for all three and correct mapping.

Description Correctly find and report the names of the three groups/families commonly characterized as the major Broadway theater-house “dynasties.” Full credit for listing all three correctly. Partial credit for listing only 1–2 correct dynasties, or listing 3 but with one incorrect. Full credit is still possible if the agent explains credible source conflict/ambiguity (e.g., different articles define the “three” differently, or mix in major operators) and justifies their chosen set based on reputable sources.

Description Provide how many Broadway theaters are owned by the first identified dynasty. Full credit if the count is clearly stated and tied to a reputable source with date/context (since counts can change). Full credit may also be earned if reputable sources disagree or the definition differs (e.g., owned vs operated/presented/managed): in that case the agent should report the conflicting figures (or a range), explain the reason for discrepancy, and state which definition it is using. Partial credit if a plausible count is provided but sourcing/date/definition is unclear. No credit if the count is missing or clearly for the wrong entity.

Description Provide how many Broadway theaters are owned by the second identified dynasty. Full credit if the count is clearly stated and tied to a reputable source with date/context. Full credit may also be earned if reputable sources disagree or the definition differs (e.g., owned vs operated/presented/managed): report the conflicting figures (or a range), explain discrepancy, and state the definition used. Partial credit if a plausible count is provided but sourcing/date/definition is unclear. No credit if the count is missing or clearly for the wrong entity.

Description Provide how many Broadway theaters are owned by the third identified dynasty. Full credit if the count is clearly stated and tied to a reputable source with date/context. Full credit may also be earned if reputable sources disagree or the definition differs (e.g., owned vs operated/presented/managed): report the conflicting figures (or a range), explain discrepancy, and state the definition used. Partial credit if a plausible count is provided but sourcing/date/definition is unclear. No credit if the count is missing or clearly for the wrong entity.

Description Counts should be correctly matched to the corresponding dynasty names (no swapping), and the answer should make it clear which count belongs to which dynasty. Full credit if each dynasty is paired with its corresponding reported count (or range, if the agent explains source disagreement). Partial credit if the mapping is somewhat unclear but can be reasonably inferred. No credit if counts are misattributed to the wrong dynasties.

Description Specify the first week of December (e.g., December 1–7) as the booking and ticket search window. Partial credit if the agent recognizes December but not the exact first-week span.

Description Establish that the hotel search must be in New York City, specifically the Times Square area. Partial credit if NYC is identified but Times Square is omitted.

Description Locate the lowest available nightly rate hotel in Times Square for the given dates, and provide the hotel name, rate, and source/link. Partial credit if a low-cost option is found but missing source or exact rate.

Description Provide at least one show date during the first week of December for The Lion King or MJ The Musical, along with ticket price and booking source/link. Partial credit for identifying a show but omitting pricing or date details.

Description Constrain both the hotel search and the show ticket search to the first week of December. Full credit if the agent clearly uses that week (e.g., Dec 1–7) consistently; partial credit if applied to only one of the two searches or if the timeframe is left ambiguous.

Description Identify the lowest-priced hotel option located in or immediately around Times Square for the specified week. Include the hotel name and the lowest price found with the corresponding date(s). Partial credit if Times Square hotels are listed but the cheapest is not clearly identified, prices are missing, or the location constraint is not met. Full credit if none are available and the agent states that clearly.

Description Locate at least one performance for either The Lion King or MJ the Musical in the first week of December and present ticket availability with a date/time and price (or price range). Partial credit if only showtimes are provided without ticket availability or pricing, or if the week constraint is not followed. Full credit if no tickets are available and the agent reports that.

Description Search lodging options explicitly located in Times Square (or clearly described as Times Square) for dates within the first week of December and compare prices across multiple properties/sources (e.g., at least 3 hotels or multiple listings). Full credit if the agent (a) selects specific dates in that week, (b) reports the lowest price it can verify among the options it checked (nightly and/or total), and (c) names the hotel and explains why it qualifies as Times Square. Also award full credit if the agent is blocked by CAPTCHAs/paywalls/site errors or if no Times Square hotels show availability for the chosen dates, provided it clearly reports what was attempted/checked. Partial credit if only one property is checked, dates are not specified within the first week of December, or Times Square location is only loosely “nearby” without justification.

Description Locate at least one available performance for The Lion King during the first week of December and report actionable details: performance date/time and the lowest available listed price (or lowest price tier shown). Full credit if the agent finds availability and provides these details, or if it reasonably checks official and/or major ticketing sources and accurately reports tickets are unavailable/sold out for the dates checked. Also award full credit if ticketing sites are inaccessible/blocked (e.g., CAPTCHA) and the agent clearly reports the limitation and what sources were attempted. Partial credit if only a schedule is provided without any price/availability details, or if the date is outside the first week of December.

Description Locate at least one available performance for MJ the Musical during the first week of December and report actionable details: performance date/time and the lowest available listed price (or lowest price tier shown). Full credit if the agent finds availability and provides these details, or if it reasonably checks official and/or major ticketing sources and accurately reports tickets are unavailable/sold out for the dates checked. Also award full credit if ticketing sites are inaccessible/blocked (e.g., CAPTCHA) and the agent clearly reports the limitation and what sources were attempted. Partial credit if only a schedule is provided without any price/availability details, or if the date is outside the first week of December.

Description Do not complete any hotel booking or ticket purchase. Full credit if the agent stops at information-gathering and, at most, navigates to seat-selection/checkout pages without entering personal or payment details. Partial credit if the agent initiates a reservation/purchase flow but stops before personal/payment information is required. No credit if the agent completes a booking/purchase or enters fabricated personal/payment information.

Description Provide detailed cost information for the ICP one-year certificate program in NYC, including tuition, fees, materials, or other required costs. Partial credit if only tuition or partial cost components are identified.

Description Provide detailed cost information for the NY Film Academy one-year certificate program in NYC, including tuition, fees, materials, or other required costs. Partial credit if only tuition or partial cost components are identified.

Description Analyze and articulate the differences between the two programs’ cost structures, highlighting which components differ and by how much. Partial credit if only some differences are noted.

Description Find the International Center of Photography's one-year certificate program in New York City and provide its cost structure. Full credit for a clear breakdown (e.g., tuition and required/standard fees). Partial credit if only total tuition is provided or if the agent clearly states that specific cost details are not publicly available and acknowledges that limitation.

Description Find the New York Film Academy's corresponding one-year certificate program and provide its cost structure. Full credit for a clear breakdown (e.g., tuition and required/standard fees). Partial credit if only total tuition is provided or if the agent clearly states that specific cost details are not publicly available and acknowledges that limitation.

Description Explicitly describe the differences in cost structure between the two programs (e.g., tuition amounts, required fees, categories/inclusions). Full credit for concrete, specific differences; partial credit for high-level or incomplete comparisons; full credit may also be awarded if differences cannot be determined due to lack of public data, provided the agent clearly explains this.

Description Ensure the agent focuses on the one-year certificate program (not degree or short workshops) and correctly scopes ICP as being in New York City. Partial credit if the program type is generally correct but scope details are ambiguous.

Description Find and report the cost structure for ICP’s one-year certificate program in New York City, clearly naming the specific program/track priced (as ICP labels it). Full credit if the agent reports the key published cost components (e.g., tuition/total program cost and any explicitly listed required/typical fees such as registration, lab/materials, equipment, student fees) OR, if ICP does not publicly provide a breakdown or places details behind an inquiry/login wall, the agent clearly states what is publicly available (e.g., only a headline tuition figure or only per-credit pricing) and what is not accessible, without guessing. Partial credit if the agent provides only a single headline price while a fuller breakdown is publicly visible and accessible.

Description Find and report the cost structure for NYFA’s comparable one-year certificate program in New York City, clearly naming the specific program/discipline priced (as NYFA labels it). Full credit if the agent reports the key published cost components (e.g., tuition/total program cost and any explicitly listed required/typical fees such as equipment, supplies, lab/studio fees, insurance, registration, housing/estimated living costs if NYFA presents them as part of the cost structure) OR, if NYFA does not publicly provide a breakdown or places details behind an inquiry/login wall, the agent clearly states what is publicly available and what is not accessible, without guessing. Partial credit if the agent provides only a single headline price while a fuller breakdown is publicly visible and accessible.

Description Provide an explicit comparison of how ICP’s and NYFA’s cost structures differ for the cited one-year certificate programs, grounded in the reported components (e.g., what is included in tuition vs. billed as separate fees, equipment/supplies policies, lab/studio fees, deposits, payment plan/schedule, estimated additional costs). Full credit if the comparison is as specific as the schools’ published information allows; if one or both schools do not publish comparable detail, full credit is earned by clearly stating the limitation and comparing based on the available categories (e.g., one publishes equipment fees separately while the other does not disclose them publicly). Partial credit for vague comparisons not tied to stated components when component information is available.

Description Ensure the programs compared are truly one-year certificate programs in NYC for both ICP and NYFA by stating the program names and confirming campus/location and credential/length as presented by the schools. Full credit if the agent acknowledges and resolves (or transparently reports) ambiguities such as multiple one-year certificate variants/tracks, conflicting prices across pages, outdated vs current tuition years, or inaccessible pages (captcha, broken links, inquiry/login walls), and uses reasonable official alternatives (e.g., official catalog PDFs, tuition/fees pages) without making up numbers. Partial credit if the programs may be mismatched but the agent explicitly flags the mismatch/uncertainty. No credit if the agent presents mismatched programs as equivalent or fabricates costs.

Description Provides the current cost for a full-year meal plan at Stanford, based on the maximum‐allowed or unlimited meals assumption. Partial credit if an approximate or slightly outdated figure is given; full credit if the exact and current cost is clearly stated.

Description Provides the current cost for a full-year meal plan at Johns Hopkins, based on the maximum‐allowed or unlimited meals assumption. Partial credit if an approximate or slightly outdated figure is given; full credit if the exact and current cost is clearly stated.

Description Find the cost for a full academic year at Stanford for a freshman on the maximum-allowed or unlimited meal plan, annualized over the appropriate number of terms (3 quarters). Partial credit if only per-quarter pricing is provided without annual total, or if a non-freshman plan is used but the correct top-tier plan is identified.

Description Find the cost for a full academic year at Johns Hopkins for a freshman on the maximum-allowed or unlimited meal plan, annualized over the appropriate number of terms (2 semesters). Partial credit if only per-semester pricing is provided without annual total, or if a non-freshman plan is used but the correct top-tier plan is identified.

Description Clearly select the unlimited plan; if no unlimited exists, select the highest-allowance plan, and explicitly state that choice. Ensure the full-year total reflects the proper term count (3 quarters for Stanford, 2 semesters for Johns Hopkins). Partial credit if the intent is followed but assumptions are not clearly stated.

Description Correctly identify the Stanford meal plan option that represents the maximum number of meals allowed or an unlimited plan for a freshman (as defined by Stanford’s dining/meal plan materials for the relevant academic year). Full credit if the agent clearly explains which plan is the maximum/unlimited and notes any relevant constraints (e.g., quarters vs annual contract, required freshman plan) OR clearly states that Stanford does not offer an unlimited plan (if that is what the source indicates) and instead identifies the highest-meal-count plan available. Partial credit if a near-maximum plan is identified or if freshman applicability is unclear but the plan is plausibly the maximum tier.

Description Report the total cost in USD for a full academic year (3 quarters or equivalent) for the identified maximum/unlimited (or highest available) Stanford meal plan, with clear units and what period it covers. Full credit if the agent provides an official full-year figure, or correctly sums/derives it from per-quarter/per-term pricing, clearly stating assumptions. Also full credit if official pricing cannot be accessed or is not published (e.g., page blocked, pricing listed as TBD, requires login) and the agent transparently reports this limitation and provides the best-supported estimate/alternative (e.g., last published year, range, or per-term cost with an explicit full-year conversion) without fabricating. Partial credit if only per-term pricing is given without a full-year conversion but enough information is present to infer it, or if the year/coverage is slightly ambiguous.

Description Correctly identify the Johns Hopkins meal plan option that represents the maximum number of meals allowed or an unlimited plan for a freshman (as defined by JHU dining/meal plan materials for the relevant academic year). Full credit if the agent clearly explains which plan is the maximum/unlimited and notes any relevant constraints (e.g., required freshman plan, semester vs annual). If JHU does not offer an unlimited plan per sources, full credit for identifying the highest-meal-count plan available and stating that no unlimited plan exists. Partial credit if a near-maximum plan is identified or if freshman applicability is unclear but the plan is plausibly the maximum tier.

Description Report the total cost in USD for a full academic year (2 semesters or equivalent) for the identified maximum/unlimited (or highest available) Johns Hopkins meal plan, with clear units and what period it covers. Full credit if the agent provides an official full-year figure, or correctly sums/derives it from per-semester/per-term pricing, clearly stating assumptions. Also full credit if official pricing cannot be accessed or is not published (e.g., page blocked, pricing listed as TBD, requires login) and the agent transparently reports this limitation and provides the best-supported estimate/alternative (e.g., last published year, range, or per-term cost with an explicit full-year conversion) without fabricating. Partial credit if only per-term pricing is given without a full-year conversion but enough information is present to infer it, or if the year/coverage is slightly ambiguous.

Description Navigate to Genentech’s official website and reach the careers or job listings section as specified in the task.

Description Identify and filter the job listings to show only positions in the Regulatory & Quality department.

Description Determine and list the number of open positions for each distinct job level within the Regulatory & Quality department. Partial credit may be given if some levels are omitted or counts are incorrect.

Description From the job levels identified, determine which one is the highest or most senior level.

Description Find and report the salary range associated with the identified most senior job level.

Description Successfully navigate to Genentech's official website and find the careers/jobs section specific to the Regulatory & Quality department. Partial credit may be awarded for reaching the careers page but not isolating the Regulatory & Quality department.

Description Determine and accurately report how many open roles exist in the Regulatory & Quality department broken down by each job level (e.g., Associate, Senior, Director, etc.). Partial credit may be awarded for incomplete breakdowns or counts, or for indicating that there are no open roles if applicable.

Description Correctly identify which job level is the most senior among the open roles in the Regulatory & Quality department. Partial credit may be awarded if multiple senior levels exist and the agent reasonably determines the top tier.

Description Find and report the salary range for the most senior job level as listed on Genentech's website. Full credit includes clearly stating if the salary range is not provided on the site and indicating that unavailability. Partial credit may be awarded for reporting salary details from specific listings if a unified range is not shown.

Description Agent navigates Genentech’s official careers/jobs area (Genentech-controlled domain/subdomain) and attempts to isolate roles to the 'Regulatory & Quality' department/functional area via filters/search. Full credit if the agent reaches the relevant search experience but is blocked (CAPTCHA/login/region restriction/technical error) and clearly reports the blocker and what was attempted. Partial credit if the agent uses a less direct Genentech-controlled source that still lists Genentech openings due to UI limitations on the main search page. No credit if the agent only uses unrelated third-party job boards without attempting Genentech.

Description Using Genentech’s displayed job-level taxonomy (the exact job level categories available on the site for the filtered results), report the number of open Regulatory & Quality roles in each job level. Full credit if counts are provided per displayed level and clearly derive from the filtered results. Full credit if the filter returns zero roles and the agent reports zeros (or clearly states there are no openings and therefore no counts per level are available). If the site is inaccessible or does not expose job-level breakdown/filtering in a way that allows counting, full credit if the agent clearly explains that limitation and provides the closest available breakdown shown on the site (e.g., by manually scanning listings, or noting that job level is not shown). Partial credit if one level is missing or if the mapping to job levels is unclear while the site was accessible.

Description Determine the most senior job level among the Regulatory & Quality openings based on Genentech’s job-level categories shown for those results. Full credit if correctly identified from the visible taxonomy. Full credit if there are no openings or if job levels are not visible/derivable (due to site limitations or access blockers) and the agent clearly states that the most senior level cannot be determined from what Genentech displays. Partial credit if the agent infers seniority but does not tie it to Genentech’s displayed job-level categories when those categories were available.

Description Provide the salary range (min–max) as displayed on Genentech’s site for role(s) at the most senior job level within Regulatory & Quality. Full credit if the agent reports the displayed range accurately and makes clear which posting(s) it came from when multiple exist. Full credit if Genentech does not display salary for those postings (or any postings) and the agent clearly reports that salary is not provided/visible after checking relevant job postings. If site access or posting pages are blocked, full credit if the agent clearly reports the blocker and that salary could not be verified on Genentech as a result. Partial credit if only min or max is provided despite the range being visible, or if the agent provides a range from a different level/department.

Description Navigate to LinkedIn.com and apply the 'Software Engineer' role and 'Seattle' location filters when searching for jobs. Partial credit if only navigation or partial filter application is completed.

Description From the search results, determine which of the listed 'Software Engineer' roles are offered by an established big-tech company. Partial credit for correctly identifying some but not all relevant roles.

Description Extract and list the exact name(s) of the big-tech company or companies associated with the identified top job listings on LinkedIn.

Description Navigate to Wikipedia.org and find the page for the retrieved company name. Partial credit if the page is located but the name is ambiguous or requires disambiguation.

Description Identify and record the year the company was founded from the Wikipedia page. Partial credit if the founding date is found but the year is uncertain or ranges are given.

Description Navigate to or use LinkedIn.com as the specified site for finding the roles. Partial credit may be awarded if job roles are found but not via LinkedIn.com.

Description Locate job listings on LinkedIn with the title 'Software Engineer' and the location set to Seattle. Full credit includes reasonably addressing 'top' (e.g., using LinkedIn labels/filters if available or noting if such a feature is unavailable and selecting prominent/high-visibility roles). Partial credit may be awarded if only the title or location is correctly matched, or if roles are found but 'top' is not addressed.

Description Confirm that the identified roles are at established big-tech companies (widely recognized large technology firms). Partial credit may be awarded if the company is a tech company but not clearly 'established big-tech,' or if the agent explains inability to confirm based on available listing details.

Description Extract the company name(s) corresponding to the identified role(s) from LinkedIn. Partial credit may be awarded if company names are retrieved for some but not all roles.

Description Using each retrieved company name, access Wikipedia.org to find and report the year the company was founded. Partial credit may be awarded if the founding year is found for some companies, or if a reasonable Wikipedia page is selected but founding information is unavailable and this is explicitly noted.

Description Navigate to LinkedIn.com (Jobs) and attempt a search for roles with keywords equivalent to 'Software Engineer' and location set to Seattle (or 'Seattle, WA'). Full credit if the agent makes a reasonable attempt but is blocked by login wall/CAPTCHA/rate limiting/availability issues and clearly reports the blocker with the best available evidence of attempted search. Partial credit if the agent searches but location or keywords are clearly incorrect or not shown.

Description From the LinkedIn results (if accessible), select at least one posting that is clearly a Software Engineer (or substantively equivalent) role located in Seattle and associated with an established big-tech company. Full credit if such a posting is found and the big-tech/established status is reasonably justified from the listing/company identity. If no clearly qualifying posting is available/visible, full credit if the agent states that no exact match can be confirmed from the visible results and selects the best available alternative that preserves primary intent (Seattle + software engineering + large/major tech company) or reports inability to validate due to missing information. Partial credit if the role is in Seattle and software engineering-related but the 'established big-tech' requirement is weak/unclear when better options are visible.

Description Report the company name as shown on the LinkedIn job listing for the selected role. Full credit if the company name is explicitly taken from the LinkedIn posting. If LinkedIn is inaccessible (as established in the first criterion), award full credit if the agent clearly states it cannot retrieve the company name from the listing due to the access blocker. Partial credit if the company name is inferred indirectly without clear linkage to the LinkedIn posting when LinkedIn was accessible.

Description Look up the identified company on Wikipedia.org and provide the year it was founded (from the infobox or lead). Full credit if the year is correctly taken from the relevant Wikipedia article. If Wikipedia is inaccessible or the page cannot be reached, full credit if the agent clearly reports the access issue after a reasonable attempt. Partial credit if Wikipedia is consulted but the year is ambiguous/incorrect due to disambiguation not being resolved.

Description Search United Airlines for direct flights from Chicago to São Paulo departing 11/24/2025 and returning 12/14/2025. Full credit if the correct route and dates are used and direct flights (if any) are listed; partial credit if only one date or route is correct.

Description Extract the fare for economy class on the identified direct flight(s). Full credit for an accurate price; partial credit for a price range or approximate value.

Description Extract the fare for business class on the identified direct flight(s). Full credit for an accurate price; partial credit for a price range or approximate value.

Description Compute the numeric difference between the business and economy fares in the same currency. Full credit for correct subtraction; partial credit for showing the two fares without the calculation.

Description If United Airlines offers no direct flights for the given route and dates, explicitly state that no flights are available. Full credit if correctly indicated when applicable; otherwise no points.

Description Find nonstop (direct) flights operated by United Airlines from Chicago to São Paulo between 11/24/2025 and 12/14/2025 (inclusive). Partial credit if flights are found but include connections, wrong airline, or dates slightly outside the specified window.

Description Explicitly state whether there are United Airlines direct flights available on the requested dates. Full credit should be awarded if there are no available flights and the agent clearly indicates that. Partial credit if availability is stated for only some dates or is ambiguous.

Description Obtain the economy and business class fare prices for the relevant United Airlines direct flights within the date range. Partial credit if only one cabin’s price is found, prices are incomplete, or not clearly tied to the direct United flights.

Description Compute the difference between business and economy prices and report it clearly. Partial credit if the calculation is provided for only some flights/dates or is unclear, or if the difference is incorrectly calculated.

Description Search for United Airlines nonstop/direct flights from Chicago (ORD/CHI) to São Paulo (GRU/SAO) for departure dates within 11/24/2025–12/14/2025. Full credit if the agent clearly applies all constraints (United + nonstop + correct endpoints + within date range), even if it checks a reasonable subset of dates within the range due to time/tool limits or site restrictions, as long as it does not go outside the range without justification. Partial credit if there is minor ambiguity (e.g., uses ORD and GRU explicitly) but intent and filtering are still clear. No credit if the agent searches the wrong airline, uses connecting flights while claiming nonstop, or uses dates outside the specified range without justification.

Description Obtain economy-cabin pricing for the qualifying United nonstop flight(s) on the searched dates within 11/24/2025–12/14/2025. Full credit if the agent provides economy prices tied to the correct nonstop United itinerary/date(s), OR if the agent makes a reasonable attempt but cannot retrieve prices due to uncontrollable factors (e.g., CAPTCHA, login wall, site errors, tool limitations) and clearly documents the blocker and what was attempted. Partial credit if economy pricing is obtained for only some checked dates/itineraries without explanation. No credit if prices are fabricated, not tied to United nonstop flights, or for the wrong route/dates/cabin.

Description Obtain business-cabin pricing for the qualifying United nonstop flight(s) on the searched dates within 11/24/2025–12/14/2025. Full credit if the agent provides business prices tied to the correct nonstop United itinerary/date(s), OR if the agent makes a reasonable attempt but cannot retrieve prices due to uncontrollable factors (e.g., CAPTCHA, login wall, site errors, tool limitations) and clearly documents the blocker and what was attempted. Partial credit if business pricing is obtained for only some checked dates/itineraries without explanation. No credit if prices are fabricated, not tied to United nonstop flights, or for the wrong route/dates/cabin.

Description Correctly calculate and report the business-minus-economy price difference for each itinerary/date where both cabin prices are available, with currency clear. Full credit if differences are correct for all provided pairs. Partial credit if the agent provides correct cabin prices but makes a minor arithmetic/currency clarity error. If one or both cabin prices are unavailable due to documented external blockers or no qualifying flights, award full credit if the agent explicitly states that the difference cannot be computed for that reason.

Description Full credit if the agent clearly states that there are no qualifying United nonstop flights in the requested date range, OR that it cannot confirm availability due to a specific external blocker (e.g., CAPTCHA, site outage, tool limitation) and describes the attempted checks. Partial credit if the agent implies unavailability without clearly tying it to the full set of constraints (United + nonstop + date range) or without describing what was checked. No credit if the agent incorrectly claims unavailability when qualifying flights/prices are shown, or fails to mention unavailability/confirmation failure when no results can be obtained.

Description Verify whether Ryanair operates flights from Birmingham, UK to Porto, Portugal on the specified dates (outbound 11/23/2025, return 11/18/2025).

Description Count and report the number of available seats with extra legroom on the specified outbound and return flights if they exist. Partial credit may be given for providing a correct range or noting availability for one segment only.

Description Explicitly state if no flights are available for the given dates, or if the date combination is not possible (e.g., return before outbound).

Description Confirm the specified route (Ryanair, Birmingham, UK to Porto, Portugal) and dates (outbound 11/23/2025, return 11/18/2025). Detect that the return date precedes the outbound date and explicitly state that this is not possible, as requested. If dates were valid, proceed. Full credit is awarded if the agent clearly indicates impossibility due to the dates or due to no available flights on those dates.

Description Search for Ryanair flights for both legs: BHX -> OPO on 11/23/2025 and OPO -> BHX on 11/18/2025. Award partial credit for correctly checking one leg or for clearly indicating that no Ryanair flights exist for the given date(s). This step is not required if the itinerary is deemed not possible in the previous criterion.

Description For any identified Ryanair flights, determine how many extra-legroom seats are available on each relevant segment (outbound and return) using seat selection/seat map information. Partial credit for reporting one leg, or for clearly stating that the information cannot be obtained or that there are zero available extra-legroom seats. This step is not required if the itinerary is deemed not possible in the first criterion.

Description Identify that the provided return date (11/18/2025) occurs before the outbound date (11/23/2025) and clearly state that the itinerary cannot be satisfied as written unless the user corrects the dates. Full credit for explicitly flagging this inconsistency; partial credit if the agent hints at a problem but is unclear.

Description Attempt to determine whether Ryanair has flights BHX→OPO on 11/23/2025 and OPO→BHX on 11/18/2025 using Ryanair’s booking/schedule interface or an authoritative equivalent. Full credit if the agent makes a reasonable attempt and either (a) reports availability/unavailability for each leg/date, or (b) explains a concrete blocker (e.g., CAPTCHA, site outage, geo restrictions) preventing verification. Partial credit if only one leg/date is checked or the source is non-authoritative without an attempt to validate against Ryanair. No credit if the agent assumes availability/unavailability without attempting to check and without citing the date inconsistency.

Description Provide the exact count of seats with extra legroom available on the selected outbound flight as shown in Ryanair seat selection. Full credit if the agent reaches the seat map and counts correctly. Partial credit if the agent reports only qualitative availability (e.g., 'some') or provides an unclear count. If the agent is prevented from viewing the seat map due to an external blocker or Ryanair limitations, it should not be penalized here as long as that limitation is clearly reported under the no-flight/not-possible criterion.

Description Provide the exact count of seats with extra legroom available on the selected return flight as shown in Ryanair seat selection. Full credit if the agent reaches the seat map and counts correctly. Partial credit if the agent reports only qualitative availability (e.g., 'some') or provides an unclear count. If the agent is prevented from viewing the seat map due to an external blocker or Ryanair limitations, it should not be penalized here as long as that limitation is clearly reported under the no-flight/not-possible criterion.

Description Clearly state what prevents fulfilling the request and tie it to the specific leg/date (e.g., 'no Ryanair BHX→OPO flight on 11/23/2025', 'return date precedes outbound date so the trip is impossible as written', 'Ryanair seat map not accessible without purchase/CAPTCHA'). Full credit if the agent is specific and accurate about which dependency failed. Partial credit if the statement is vague or not tied to the correct leg/date. No credit if the agent invents seat counts or claims unavailability without either checking (when feasible) or identifying the date inconsistency.

Description Determine whether WestJet offers round-trip service between Waterloo, Ontario and Calgary, Alberta for travel dates September 10 – 27, 2026. Full credit for correctly stating if flights are available or indicating that none exist for those dates.

Description Identify the standard checked baggage allowance (number of bags, weight or size limits) for the specified WestJet flights. Partial credit if a general WestJet policy is described without confirming applicability to these flights.

Description List any applicable fees for checked baggage on these flights, including first and additional bag charges if relevant. Partial credit if some but not all relevant fees are provided.

Description Determine whether WestJet has flights for Waterloo, ON to Calgary, AB on Sep 10, 2026 (outbound) and Sep 27, 2026 (return). Full credit for clearly stating availability or explicitly indicating no flights are available for those dates. Partial credit if dates/route are acknowledged but availability status is incomplete or only one leg is addressed.

Description State the checked baggage allowance applicable to the specified WestJet flights (e.g., number of checked bags included/permitted, size and weight limits). Full credit if allowance details are complete and clearly tied to WestJet. Partial credit for generic WestJet domestic policy or if some allowance details are missing.

Description List the fees for checked bags applicable to the specified WestJet flights (e.g., first and second checked bag fees, per direction). Full credit if clear, specific fees are provided. Partial credit if only partial fees are given or reasonable ranges/caveats are provided when exact fees cannot be confirmed.

Description Make clear that policies/fees are for WestJet and apply to the Waterloo–Calgary route, covering both outbound and return legs (or explicitly note that the same policy applies each way). Partial credit if airline specificity is clear but the round-trip context is not fully addressed.

Description Make a reasonable attempt to check whether WestJet (or WestJet-marketed) itineraries exist for Waterloo, ON (YKF)  Calgary, AB (YYC) departing Sep 10, 2026 and returning Sep 27, 2026. Full credit if the agent clearly describes the check performed and either (a) reports results found, or (b) explains why availability cannot be confirmed (e.g., schedules not published that far out, site blocked/captcha, tool limitations). Partial credit if the check is unclear or uses a different airport/date without explicitly calling that out.

Description Provide a clear conclusion for both the outbound (Sep 10, 2026) and return (Sep 27, 2026) on the YKFYYC route: whether WestJet itineraries are available (including whether only connecting itineraries exist) OR that none are available OR that availability cannot be verified due to external factors (e.g., schedule not released). Full credit for a correct, unambiguous statement covering both directions; partial credit if only one direction/date is addressed.

Description State WestJet checked baggage allowance rules relevant to the route, including number of checked bags included vs not included, and standard weight/size limits. Full credit if the agent correctly explains that allowance depends on fare type (and optionally status/credit card) and provides the correct allowances by fare tier (or the applicable tier if known). If itinerary/fare cannot be determined due to unavailable/unverifiable flights, full credit is still possible for accurately providing the policy ranges/tiers and clearly labeling them as fare-dependent rather than itinerary-confirmed.

Description Provide WestJet checked bag fees that would apply (e.g., first/second bag) and any key conditions (e.g., fees vary by fare, when purchased online vs airport, and/or currency/route caveats) plus mention of overweight/oversize charges if part of the standard fee table referenced. Full credit if fees are accurate for WestJet policy and clearly tied to fare tiers/conditions; if flights/fare are unavailable or unverifiable, full credit is still possible for correctly presenting the fare-dependent fee structure and noting uncertainty about which tier applies.

Description Explicitly state that there are no available WestJet flights/itineraries for the relevant direction(s) on the specified dates/route, OR clearly state that availability cannot be verified and why. The agent must not imply itinerary-specific baggage fees for a specific booking when no flights exist/are found; it may provide general WestJet baggage policy only if clearly separated from itinerary-specific claims. Full credit if the statement covers both outbound and return (or clearly identifies which direction is unavailable/unverifiable).

Description Determine whether there are direct AirAsia flights from Singapore (SIN) to Langkawi (LGK) departing Nov 24 and returning Nov 27. Full credit if both legs are correctly identified (flight numbers/times). Partial credit if only one leg is checked or if indirect options are incorrectly treated as direct.

Description Find and report the cost to select a window seat on the specified direct flights. Full credit for providing the exact fee per flight or round trip; partial credit for a correct price range or approximate fee. If no direct flights exist, note that the cost cannot be determined.

Description Confirm whether there are direct flights operated by AirAsia from Singapore to Langkawi on the dates November 24, 25, 26, and 27. Partial credit may be awarded for checking some but not all dates in the range. Full credit is also awarded if the agent correctly states that there are no available direct AirAsia flights for those dates.

Description Report the specific fee to select a window seat on a direct AirAsia flight within the specified date range. Partial credit may be given for a range or a generic seat selection fee without clarifying it applies to a window seat. Full credit requires an exact amount tied to at least one applicable flight/date in the range, or a clear statement that this cannot be provided because no flights are available on those dates.

Description Confirm that the amount provided is specifically the fee for selecting a window seat, not the ticket price or other ancillary fees. Partial credit may be awarded if the answer is unclear but likely refers to seat selection; no credit if it provides airfare or a different fee.

Description Attempt to search AirAsia (or an AirAsia booking interface) for flights that match the constraints: airline AirAsia, Singapore (SIN)  Langkawi (LGK), outbound Nov 24 and return Nov 27, and direct flights. Full credit if the agent applies all constraints OR clearly explains a platform limitation (e.g., direct-only filter unavailable, captcha/blocked, site down) while still attempting to verify the route/dates/airline. Partial credit if one constraint is missed/unclear (e.g., uses city names without airport codes, or checks adjacent dates in addition to the requested ones without clarifying). No credit if the agent primarily searches the wrong route, wrong airline, or wrong dates when correct options were reasonably accessible.

Description For any found direct AirAsia itinerary matching the requested dates, progress to the seat-selection/add-ons stage and report the explicit fee to select a window seat, clearly indicating whether it applies per segment (SINLGK and LGKSIN) and the currency shown. Full credit if the agent either (a) provides the window-seat fee(s) sourced from the seat map/add-ons for the correct segments, OR (b) clearly states that the window-seat fee is not visible/retrievable due to external constraints (e.g., seat map unavailable without booking/login/payment step, page errors, currency not displayed) after a reasonable attempt. Partial credit if the agent reports only a non-window-specific seat fee (e.g., 'standard seat') or provides fees for only one segment while indicating the limitation. No credit if the fee is guessed or not tied to the correct route/dates/airline context.

Description Full credit if the agent clearly states that no matching direct AirAsia flights are available for those specific dates/route and indicates this conclusion is based on checking search results (including noting direct-only when applicable, or explaining if direct-only could not be enforced but no direct options were shown). Partial credit if unavailability is claimed but the check is incomplete/ambiguous (e.g., only checked one direction or one of the two dates). No credit if the agent asserts unavailability without a reasonable attempt or contradicts evidence it found.

Description Correctly identify the departure airport (Dallas/Fort Worth), destination (Singapore), and travel dates (April 23 departure, May 3 return). Partial credit if some details are identified correctly but others are missing or incorrect.

Description Correctly identify Japan Airlines as the carrier and Premium Economy as the cabin class. Partial credit if only one element is correct.

Description Search for flights on the specified route and dates and correctly state whether Premium Economy seats are available. Full credit if availability is confirmed or explicitly indicates no available flights; partial credit if unclear or incomplete.

Description When flights are available, list the meal options offered in Premium Economy. Partial credit for listing some but not all options; full credit for a complete and accurate list.

Description If no flights are found for the given dates, clearly state that there are no available flights in Premium Economy on those dates. Partial credit if the answer is implied but not clearly stated.

Description Check for Japan Airlines flights from Dallas/Fort Worth (DFW) to Singapore (SIN) departing April 23 and returning May 3, specifically in Premium Economy. Full credit includes clearly confirming availability for both legs or explicitly stating that there are no available flights for those dates. Partial credit may be awarded for checking only one leg or omitting the specific dates.

Description Confirm that any flight availability and meal information provided pertains to Japan Airlines and the Premium Economy cabin, as explicitly requested. Partial credit may be awarded if the airline or cabin is correctly identified but not consistently applied throughout the response.

Description List the meal options available in Premium Economy on the specified Japan Airlines flights covering DFW–SIN on the given dates. If flights are unavailable, full credit can still be earned by clearly stating the unavailability (meal details are not required in that case). Partial credit may be awarded for providing general Japan Airlines Premium Economy meal information without flight/date specificity.

Description Check whether Japan Airlines offers bookable itineraries for Premium Economy from Dallas/Fort Worth (DFW) to Singapore (SIN) departing April 23 and returning May 3. Full credit if the agent accurately determines availability status for BOTH outbound and return on the exact dates (including: JAL does not operate the route directly, only codeshares/partners, no inventory in Premium Economy, sold out, or no results). Also award full credit if the agent attempts to check but cannot due to external access issues (captcha, site outage, paywall/login restriction) and clearly reports the limitation and what was attempted. Partial credit if only one direction is checked, or if the agent uses nearby dates without clearly flagging the mismatch.

Description If eligible JAL Premium Economy itinerary(ies) exist for April 23 DFW→SIN, report the meal options shown for Premium Economy for the relevant long-haul segment(s) (and note any differences by segment if connecting). Full credit if meal options are correctly reported OR if the agent determines that meal/menu options are not publicly available for that specific date/flight/cabin (e.g., only available after ticketing/PNR, not loaded yet, or not displayed by the data source) and clearly states this after a reasonable attempt. Partial credit if meal info is provided but is generic/not clearly tied to Premium Economy or the correct segments/date.

Description If eligible JAL Premium Economy itinerary(ies) exist for May 3 SIN→DFW, report the meal options shown for Premium Economy for the relevant long-haul segment(s) (and note any differences by segment if connecting). Full credit if meal options are correctly reported OR if the agent determines that meal/menu options are not publicly available for that specific date/flight/cabin (e.g., only available after ticketing/PNR, not loaded yet, or not displayed by the data source) and clearly states this after a reasonable attempt. Partial credit if meal info is provided but is generic/not clearly tied to Premium Economy or the correct segments/date.

Description If there are no available JAL flights matching the exact dates/route/Premium Economy requirement (outbound and/or return), explicitly state that in the final answer, clarifying which leg(s) are unavailable and the apparent reason when determinable (e.g., no JAL service on route, no search results, or no Premium Economy inventory). Full credit if the statement is clear and unambiguous; partial credit if unavailability is only implied or is missing leg-specific clarity.

Description Identify and confirm the specified airline (Cathay Pacific), route (Manila to Hong Kong), and dates (November 17 – December 12). Partial credit if some details are missing or misstated.

Description Verify whether there are any flights available on the given dates. Full credit for correctly identifying availability or noting unavailability; partial credit for an incomplete or ambiguous check.

Description Find the cost to upgrade from economy to business class on the identified flights. Partial credit for providing a price range or an approximate fee if an exact cost is not available.

Description Present a clear answer stating the upgrade cost or explicitly indicating that no flights are available for the specified dates.

Description Target Cathay Pacific flights on the Manila (MNL) to Hong Kong (HKG) route for the date window November 17 to December 12 as stated. Partial credit if only some of airline/route/dates are correctly applied.

Description Check whether Cathay Pacific has available flights for the specified dates and clearly state availability status. Full credit if the agent explicitly indicates when there are no available flights for those dates, as requested. Partial credit if the availability status is vague or only partially addressed.

Description Report how much it would cost to upgrade from economy to business class for the specified Cathay Pacific Manila–Hong Kong flight(s) within the stated dates. Partial credit if a reasonable estimate/range or relevant conditions (e.g., fare-class dependence) are provided when exact pricing cannot be confirmed, but it must be framed specifically as upgrade cost (not full fare).

Description Evaluate whether the agent attempted to check Cathay Pacific upgrade cost/eligibility for flights from Manila (MNL) to Hong Kong (HKG) departing Nov 17 and returning Dec 12 (same year implied). Full credit if the agent clearly uses Cathay Pacific-operated flights (or explicitly notes when only codeshare/partner options are shown). Partial credit if the route is correct but dates are slightly off or the carrier/operating airline is unclear. No credit if the airline or route is wrong when correct options exist.

Description Report the economy-to-business upgrade cost for the specified Cathay Pacific itinerary, including currency and whether it is per segment, per direction, or total. Full credit if the agent provides a verifiable upgrade quote OR if upgrades cannot be priced/are not offered for the selected fare/flight and the agent clearly states this limitation (e.g., no upgrade inventory, fare not upgrade-eligible, upgrade only via miles/bid, requires login, or pricing not publicly available). Partial credit if only one direction is covered, the basis (per leg vs total) is unclear, or the agent provides an approximate range while clearly labeling it as non-final due to dynamic pricing. No credit if the agent guesses/hallucinates a numeric price without support or confuses upgrade cost with general fare difference without explanation.

Description Confirm whether Cathay Pacific flights are available for MNL→HKG on Nov 17 and HKG→MNL on Dec 12, and whether an economy-to-business upgrade path appears available/eligible for the selected flights (when such information is accessible). Full credit if the agent explicitly states availability for both directions, or clearly states that no Cathay Pacific flights exist/sold out on one or both dates, or that availability cannot be confirmed due to access issues (and the agent notes the blocking/limitation). Partial credit if availability is only addressed for one date/direction or is only implied.

Description If no eligible Cathay Pacific flights are available on one or both requested dates, the agent should clearly indicate this and specify which date/direction is unavailable. Full credit if the agent identifies the specific missing leg(s) (Nov 17 outbound and/or Dec 12 return). Partial credit if the agent states 'no flights available' but does not specify which leg/date. No credit if the agent omits the unavailability note or asserts availability/pricing despite having established that no flights exist for the requested leg(s).

Description Determine whether ITA offers flights from Rome to Naples departing on February 23 and returning on March 18. Full credit for confirming availability (or unavailability) for both departure and return legs; partial credit if only one leg is checked correctly.

Description Identify the duration of the available flights (outbound and inbound). Full credit for providing durations for both legs; partial credit if only one leg duration is provided or ranges are approximate.

Description Count how many daily ITA flights operate between Rome and Naples on the specified departure date (February 23). Full credit for an exact count; partial credit for an approximate count or if only one leg is counted.

Description Base the answer specifically on ITA Airways flights for the Rome-to-Naples route, departing February 23 and returning March 18. Partial credit if one of these parameters (airline, route, or either date) is mishandled or omitted.

Description Check whether ITA operates flights on both specified dates and clearly state if there are no available flights on either date. Full credit if unavailability is explicitly reported when applicable. Partial credit if availability is suggested but not clearly stated for both dates.

Description Report the flight duration for ITA flights on the Rome–Naples route for the February 23 departure and the March 18 return. Partial credit for providing typical/approximate duration or only one leg’s duration.

Description Report the number of daily ITA flights operating on February 23 (Rome to Naples) and on March 18 (Naples to Rome). Partial credit if counts are given for only one date or if a general (non-date-specific) daily count is provided.

Description Evaluate whether the agent attempts to check ITA Airways specifically (not another carrier) for flights between Rome and Naples for the requested outbound (Feb 23) and return (Mar 18) dates. Full credit if the agent clearly uses ITA as the source of availability, OR clearly reports that ITA data cannot be verified due to an uncontrollable blocker (e.g., ITA site down/CAPTCHA/login wall/search tool failure). Partial credit if the agent mixes in other airlines but still separately identifies ITA results or clearly distinguishes that ITA could not be checked. No credit if results are for a different airline only with no ITA attempt/coverage.

Description For ITA, report the number of flights available on Feb 23 from Rome to Naples and the flight duration(s) (including specifying which Rome airport if relevant). Full credit if both values are provided for the correct route/date, OR if the agent determines there are no available ITA flights and explicitly states that, OR if the agent cannot verify due to an uncontrollable blocker and explicitly states the blocker and that availability/durations cannot be confirmed. Partial credit if only one of: duration or number of daily flights is provided, or if the route/date is slightly ambiguous but clearly intended, or if the agent provides partial ITA info but cannot complete verification due to blocker. No credit if a wrong date/route is used when correct information is available/visible.

Description For ITA, report the number of flights available on Mar 18 from Naples to Rome and the flight duration(s). Full credit if both values are provided for the correct route/date, OR if the agent determines there are no available ITA flights and explicitly states that, OR if the agent cannot verify due to an uncontrollable blocker and explicitly states the blocker and that availability/durations cannot be confirmed. Partial credit if only one of: duration or number of daily flights is provided, or if the route/date is slightly ambiguous but clearly intended, or if the agent provides partial ITA info but cannot complete verification due to blocker. No credit if a wrong date/route is used when correct information is available/visible.

Description If ITA flights are not available for one or both requested dates, the agent must explicitly indicate that and specify which leg/date is affected. If availability cannot be verified due to an uncontrollable blocker (CAPTCHA/site down/login wall/tool failure), the agent must explicitly state the blocker and specify which leg/date cannot be verified. Full credit for clear, leg-specific reporting; partial credit if unavailability/blocker is mentioned but not tied to the specific leg/date. No credit if the agent makes unsupported claims of availability/unavailability or fails to mention a blocker that prevented verification.

Description Determine whether there are any Jetstar flights from Darwin to Adelaide departing in one month for a two-week round trip. Partial credit if the itinerary dates or route are misunderstood or if availability is unclear.

Description If no flights match the specified dates and route, explicitly state that no flights are available for those dates. Partial credit if the agent notes limited availability but does not clearly say 'no flights available.'

Description Provide Jetstar's cancellation fee policy for the specified Darwin–Adelaide booking, including details on deadlines, refundable amounts, and any applicable conditions. Partial credit if only general policy is given without route/date specificity.

Description Provide Jetstar's change fee policy for the specified Darwin–Adelaide booking, including details on change deadlines, fees by fare type, and any fare difference charges. Partial credit if only generic change rules are given without specifics for the itinerary.

Description Determine and clearly state the specific departure date one month from now and the return date two weeks later for the Darwin to Adelaide trip. Partial credit if the agent attempts to interpret the timeframe but does not clearly specify exact dates.

Description Search for Jetstar flights from Darwin to Adelaide on the chosen dates (round-trip for a two-week duration) and report availability. Full credit if the agent confirms availability or explicitly indicates that there are no available flights for those dates, as requested. Partial credit if the search is attempted but the availability status is unclear or incomplete.

Description Explain the applicable Jetstar cancellation policy for the Darwin–Adelaide trip in the specified timeframe, including whether cancellations are allowed and any fees or refund/credit conditions. Partial credit if a general Jetstar cancellation policy is provided but lacks key details.

Description Explain the applicable Jetstar change policy for the Darwin–Adelaide trip in the specified timeframe, including whether changes are allowed and any fees or conditions (e.g., fare differences). Partial credit if a general Jetstar change policy is provided but lacks key details.

Description Determine which Jetstar change/cancellation rules govern a DRW–ADL return trip, referencing Jetstar’s applicable fare bundle rules (e.g., Starter vs Starter Plus vs Flex) and/or Jetstar’s general change/cancellation policy pages for Jetstar Australia. Full credit if the agent correctly explains that fees/eligibility depend on the fare type purchased and cites/uses the relevant Jetstar policy/rules source(s). Partial credit if it provides only generic Jetstar guidance without clearly tying it to fare types or sources. No credit if it uses a different airline’s policies or unrelated regions.

Description Provide Jetstar cancellation outcomes relevant to the trip, including whether cancellation is allowed, whether a refund is possible vs flight credit/voucher, and any key conditions/exclusions and typical fee concepts (e.g., cancellation fee and/or forfeiture of fare, and handling of optional extras). Full credit if the answer is accurate for the identified fare types (or clearly states the fare-type dependency and accurately summarizes each). Partial credit if cancellation is addressed but refund/credit vs fee/forfeiture is unclear or incomplete. No credit if cancellation policy is omitted or materially incorrect.

Description Provide Jetstar change rules relevant to the trip, including whether date/time changes are permitted, any change fees (or fee waivers) and that fare differences may apply, plus any key timing/conditions (e.g., before departure). Full credit if the answer is accurate for the identified fare types (or clearly states the fare-type dependency and accurately summarizes each). Partial credit if only fees or only fare-difference rules are mentioned, or conditions are unclear. No credit if change policy is omitted or materially incorrect.

Description Make a reasonable attempt to check Jetstar availability for DRW↔ADL with a departure approximately one month from now and a return about two weeks later, by selecting reasonable specific dates within that window (and stating them) or by using a flexible-date view if available. Full credit if the agent attempts the check and (a) finds availability results, or (b) is blocked by external factors (captcha, site outage, geo restrictions, login requirements) and clearly reports the limitation. Partial credit if the attempted dates are not aligned to the requested window or the attempt is unclear.

Description Based on the attempted availability check, clearly state whether flights are available for the chosen dates. Full credit if the agent accurately reports either (a) flights are available, or (b) no Jetstar flights are available/sold for those dates, OR (c) it cannot be verified due to access limitations (as long as the attempt and limitation are clearly documented). Partial credit if the conclusion is ambiguous or does not match the checked dates. No credit if availability is asserted without a reasonable check or contradicts observed results.

Description Stay within non-transactional steps: do not attempt to complete purchase/checkout and do not request/use unnecessary personal or payment data. Full credit if the agent only looks up policy/availability and stops before any checkout steps. Partial credit if it goes further into booking flow but stops before any critical point. No credit if it attempts purchase or solicits payment details.

Description Agent should successfully search for Alaska Airlines flights from Seattle (SEA) to Honolulu (HNL) on 11/29/2025 and again (if applicable) on 12/03/2025. Partial credit if at least one correct date or correct route is searched; full credit if both dates/routes are correctly searched.

Description Agent should access the seating configuration for each identified flight, locate the exit rows, and count the number of seats still available in those rows. Partial credit if exit rows are identified without accurate counts or counts are incomplete; full credit if all exit row seats are accurately counted.

Description Agent should clearly report the total number of available exit row seats across all found flights. If no flights are available on the specified dates, the agent should explicitly state that no flights are available. Partial credit if one of the two outcomes is correctly reported; full credit if the correct comprehensive outcome is provided.

Description Ensure the search is restricted to Alaska Airlines flights specifically on the route from Seattle, WA to Honolulu, HI. Partial credit if either the airline or route is correct but not both.

Description Ensure the query covers the specified date window inclusively. Partial credit if only some of the dates in the range are addressed.

Description Identify whether Alaska Airlines has any flights on the specified route during the date range. Full credit if, when no flights are available, the answer explicitly states that. Partial credit if availability is checked but the no-flights case is not clearly handled.

Description Report how many exit row seats are still available on the Alaska Airlines flights within the specified date range. Full credit for accurate counts covering all applicable flights; partial credit for counts covering only some flights/dates or for incomplete quantification.

Description Check Alaska Airlines-operated flights for the Seattle, WA (SEA) to Honolulu, HI (HNL) route covering the dates 11/29/2025 through 12/03/2025 (each date in the range, or an equivalent method that clearly covers the whole range). Full credit if the agent clearly searches/filters to Alaska-operated flights and covers the full date range. Partial credit if the agent covers only some dates or mixes in other airlines without clearly separating Alaska-operated flights. Full credit is still possible if the agent attempts the correct search but is blocked by an external issue (e.g., site outage/captcha) and clearly reports what prevented full verification.

Description For each date 11/29/2025–12/03/2025, list the Alaska Airlines SEA→HNL flight options found (e.g., flight numbers and departure times), or clearly state that none appear for that date. Full credit if the set of Alaska-operated options is reasonably captured for each date, given the platform’s visible results. Full credit if the agent attempts this but cannot retrieve results due to external blockers and reports the issue. Partial credit if some dates are missing or flight listing is ambiguous.

Description For each Alaska Airlines flight found on the specified dates, open the seat map (for the relevant segment/cabin) and count how many exit-row-designated seats are still unoccupied/available. Full credit if counts are provided per flight (and per segment/cabin if applicable) with clear linkage to the correct seat map. If seat maps/exit-row labels cannot be accessed due to external factors (e.g., seat map unavailable until booking/login, aircraft not assigned, site errors/captcha), full credit is earned by clearly documenting the attempt, where it failed, and reporting that exit-row availability could not be verified. Partial credit if exit-row availability is mentioned but not counted, or if only some flights/dates have verified counts when more were accessible.

Description Clearly state that there are no available Alaska Airlines flights across the whole date range, OR that availability across the range could not be confirmed due to external blockers affecting the entire range (e.g., site outage/empty results error). Full credit if the agent makes the date-range coverage explicit and distinguishes between true unavailability vs. inability to confirm due to platform issues. Partial credit if the agent’s coverage of the range is unclear.

Description Successfully navigate to the EasyJet website or flight search interface where flight availability and pricing can be found.

Description Input Palma de Mallorca as the origin, Newcastle as the destination, departure date December 3, and return date December 23 exactly as specified.

Description Check whether any flights are available for the given route and dates, and correctly state if no flights are found.

Description From the list of available flights, locate the flight option with the lowest base fare for the specified dates.

Description Provide the full total cost for the cheapest flight, ensuring that all airline fees, taxes, and mandatory charges are included in the reported amount.

Description Search for flights specifically from Palma de Mallorca (PMI) to Newcastle (NCL) for the specified trip dates (December 3 to December 23) and ensure the airline is EasyJet. Partial credit may be awarded if only one leg/date is correctly checked or if the route is correct but the airline constraint is missed.

Description Among the available EasyJet flight options that match the specified dates, correctly determine and select the lowest-priced option. Partial credit may be awarded if an option is provided but it is not the cheapest.

Description Provide the total price for the selected EasyJet flight(s), explicitly including all mandatory taxes and fees as shown before checkout. Partial credit may be awarded if only base fare is provided or if fees/taxes are mentioned but the total is unclear.

Description If there are no EasyJet flights available that match the specified dates and route, clearly state that no flights are available instead of providing a price. Full credit is awarded for correctly identifying unavailability.

Description Attempt to search EasyJet for Palma de Mallorca (PMI)  Newcastle (NCL) with outbound date December 3 and return date December 23 (same year context as the task). Full credit if the agent uses EasyJet (site/app or clearly identified EasyJet results) for these exact dates/route OR clearly reports an uncontrollable blocker that prevents checking (e.g., CAPTCHA, site down, infinite loading, geo restrictions). Partial credit if the agent attempts EasyJet but uses slightly wrong nearby airports or adjacent dates while clearly trying to satisfy the request. No credit if the agent does not attempt EasyJet or searches an unrelated route/dates without justification.

Description If EasyJet shows bookable flights for both legs on December 3 (outbound) and December 23 (return), identify the lowest-priced itinerary that matches those dates. Full credit if the agent compares the available EasyJet options shown (times/fare types where relevant) and selects the cheapest matching itinerary. Partial credit if the agent selects a valid itinerary for the dates but does not establish it is the cheapest when cheaper options were visible, or overlooks an obviously cheaper visible option. If EasyJet shows no bookable flights for one/both legs on the specified dates (or availability cannot be verified due to an uncontrollable blocker), do not penalize under this criterion as long as the agent clearly reports that limitation elsewhere.

Description Report the all-in total price (including fees and taxes) for the cheapest EasyJet itinerary for December 3  December 23 as shown by EasyJet in the price summary/checkout flow (before entering passenger/payment details). Full credit if the agent provides the final total and indicates it includes fees/taxes. Partial credit if the agent provides only per-leg pricing or a subtotal and clearly notes that the all-in total could not be reached due to an uncontrollable blocker (e.g., checkout blocked/CAPTCHA) or that EasyJet did not display an all-in total without advancing to a blocked step. No credit if the agent fabricates a total or provides an amount not supported by the EasyJet results it accessed.

Description Clearly state that there are no available EasyJet flights for the exact dates/route if EasyJet indicates none (e.g., No flights / Sold out / no return options), or clearly state that availability could not be confirmed due to a blocker after a reasonable attempt. Full credit if the statement is explicit for the exact route and dates. Partial credit if unavailability/uncertainty is implied but not clearly tied to the exact dates/route. No credit if the agent incorrectly claims no flights exist when flights were available, or fails to mention unavailability when none were found.

Description Check Jetstar’s site (or an authoritative source) to verify whether round-trip flights from Adelaide to Sunshine Coast exist for November 18–25. Partial credit if only one leg or unclear availability is addressed; full credit if overall availability is clearly determined.

Description Locate and list any bundle deals or fare packages Jetstar offers for the specified Adelaide–Sunshine Coast round-trip on those dates, including key details (e.g., fare types, included extras). Partial credit for general mention; full credit for specific, accurate deal information.

Description If no flights are available for the given dates, explicitly state that there are no available flights instead of listing bundle deals. Partial credit if absence is mentioned but ambiguous; full credit if clearly communicated.

Description Correctly interpret and apply the exact route and trip parameters: Jetstar flights from Adelaide (ADL) to Sunshine Coast (MCY), round trip, departing November 18 and returning November 25. Partial credit if most details are correct but one element (e.g., a date) is off.

Description Check and clearly state whether Jetstar has round-trip flight availability for ADL–MCY on Nov 18–Nov 25. Full credit if the agent explicitly indicates when no flights are available for those dates. Partial credit if availability is discussed but lacks clarity or covers only one leg.

Description Determine and state whether Jetstar offers any bundle deals or packages applicable to the specified flights (e.g., fare bundles and/or packages). Provide a clear yes/no answer tied to the Nov 18–25 round trip. If no flights are available, explicitly indicate that and that bundles would not apply. Partial credit if bundles are mentioned generally without tying to the specific dates/route.

Description Ensure the findings and statements pertain specifically to Jetstar’s offerings rather than other airlines or general travel options.

Description Attempt to use Jetstar’s official site/booking flow (or Jetstar app flow if applicable) to search flights from Adelaide (ADL) to Sunshine Coast (MCY) departing Nov 18 and returning Nov 25 (same year implied). Full credit if the agent performs the correct search OR clearly reports being blocked (e.g., captcha), site outage, or another access limitation preventing confirmation. Partial credit if the agent searches with slightly incorrect dates/airports or only searches one leg.

Description Based on the Jetstar search results (or if Jetstar is inaccessible, based on the best available evidence while stating the limitation), determine whether flights are available for ADL→MCY on Nov 18 and MCY→ADL on Nov 25. Full credit if the agent correctly concludes availability/unavailability for each leg, or explains that it cannot be confirmed due to access issues. Partial credit if the conclusion is provided for only one leg/date or is ambiguous (e.g., not clear which leg is unavailable).

Description For the ADL↔MCY Nov 18–Nov 25 itinerary as searched on Jetstar, report any bundle options shown/available (e.g., fare bundles such as Starter/Plus/Flex or similar, and any flight+hotel/package offerings if presented in the flow). Full credit if the agent ties bundle/package availability (including 'none offered') to the specific itinerary/date search results OR states it could not be verified due to Jetstar access limitations. Partial credit if the agent gives only general Jetstar bundle info without indicating whether it applies/was shown for this itinerary.

Description If the Jetstar search indicates no available flights, the final answer must clearly state that no Jetstar flights are available for the affected date(s)/leg(s). Full credit for an unambiguous statement specifying which leg/date is unavailable. Partial credit if unavailability is mentioned but is unclear about which leg/date, or conflates sold-out vs. not operated without noting uncertainty. If Jetstar cannot be accessed and availability cannot be confirmed, this criterion should not be applied.

Description Find the Singapore Airlines flight number(s) for the outbound journey on November 2 from LHR to SYD via SIN. Full credit for listing all segment flight numbers accurately; partial credit for listing some or incorrect/missing segments.

Description Find the Singapore Airlines flight number(s) for the return journey on November 28 from SYD to LHR via SIN. Full credit for listing all segment flight numbers accurately; partial credit for listing some or incorrect/missing segments.

Description If there are no available Singapore Airlines flights on the specified dates, explicitly state that no flights are available. Full credit for clear indication of unavailability; zero credit if omitted.

Description Ensure the identified flights are operated by Singapore Airlines and route from London (LHR) to Sydney (SYD) via Singapore (SIN). Partial credit may be given if the route via SIN is correct but the airline is incorrect, or vice versa.

Description Provide the flight numbers for the full outbound journey leaving LHR on November 2, including both legs (LHR→SIN and SIN→SYD). Partial credit may be awarded if only one leg is provided, the date is slightly misinterpreted (e.g., connecting leg departs on Nov 3), or minor airport mismatch occurs.

Description Provide the flight numbers for the full return journey leaving SYD on November 28, including both legs (SYD→SIN and SIN→LHR). Partial credit may be awarded if only one leg is provided, the date is slightly misinterpreted (e.g., connecting leg departs on Nov 29), or minor airport mismatch occurs.

Description Verify whether flights exist for the specified dates and either provide the flight numbers if available or explicitly state that no flights are available for those dates. Partial credit may be awarded if availability is addressed for only one of the two dates.

Description When flights are available, provide only the flight numbers without additional details (times, prices, etc). If no flights are available, clearly state that fact. Partial credit may be given if minimal extra information is included but flight numbers are still correctly provided.

Description Make a reasonable attempt to look up Singapore Airlines-operated itineraries for LHR→SIN→SYD departing July 2 (e.g., airline site, GDS/OTA, or reliable timetable source). Full credit if the agent attempts but is blocked (captcha/paywall), the site is down, or live data can’t be accessed, and it clearly states this limitation. Partial credit if the attempt is unclear or uses an inappropriate/irrelevant source.

Description Provide just the relevant Singapore Airlines flight numbers for the two legs on July 2: LHR→SIN and SIN→SYD, if such SQ-operated flights are available/operating. Full credit if the flight numbers are correct for the specified routing/date, OR if the agent determines that no matching SQ-operated itinerary is available/operating for that date (based on the attempted lookup) and clearly reports outbound unavailability. Partial credit if flight numbers are provided but the date/routing is unclear, or if non-SQ-operated flights are included.

Description Make a reasonable attempt to look up Singapore Airlines-operated itineraries for SYD→SIN→LHR departing July 28. Full credit if the agent attempts but is blocked (captcha/paywall), the site is down, or live data can’t be accessed, and it clearly states this limitation. Partial credit if the attempt is unclear or uses an inappropriate/irrelevant source.

Description Provide just the relevant Singapore Airlines flight numbers for the two legs on July 28: SYD→SIN and SIN→LHR, if such SQ-operated flights are available/operating. Full credit if the flight numbers are correct for the specified routing/date, OR if the agent determines that no matching SQ-operated itinerary is available/operating for that date (based on the attempted lookup) and clearly reports return unavailability. Partial credit if flight numbers are provided but the date/routing is unclear, or if non-SQ-operated flights are included.

Description Final response should contain only the flight numbers for outbound and return, with no extra details (times, prices, cabin, URLs), unless stating that flights are unavailable (or that lookup was blocked). Full credit if output is strictly flight numbers or clear unavailability statements; partial credit if minor extra text is included but flight numbers/unavailability are still clearly identifiable.

Description Verify if there are Spirit Airlines flights from Houston to Los Angeles between November 5 and November 20. If no flights are available, explicitly indicate this in the answer. Partial credit for correctly identifying availability status.

Description Identify and report the price for a standard economy seat on the specified route and date range. Partial credit if an approximate or single-date price is provided; full credit for accurate pricing corresponding to the full date range.

Description Identify and report the price of a Big Front Seat on the specified route and date range. Partial credit if an approximate or single-date price is provided; full credit for accurate pricing corresponding to the full date range.

Description Compute how much more expensive the Big Front Seat is compared to standard economy, expressing the difference clearly. Partial credit for any calculation attempt; full credit for a correct and clear difference.

Description Focus the search on Spirit Airlines for flights from Houston to Los Angeles, as explicitly required. Partial credit if the route is correct but the airline is wrong, or vice versa.

Description Search within the specified window beginning November 5 through November 20 (inclusive). Partial credit if only some dates within the range are checked or if the range is slightly off but effort is shown to cover the window.

Description Verify whether flights are available on those dates and explicitly state if there are no available flights for the specified dates. Partial credit if availability is checked for some dates or if unavailability is noted for subsets of the range.

Description Find the pricing for both standard economy and Big Front Seat for the relevant flights/dates. Partial credit if pricing is found for only part of the date range or for a subset of flights.

Description Calculate and present the difference between Big Front Seat and standard economy for the specified dates. Full credit if differences are clearly reported (per date or summarized) and tied to the checked flights; partial credit for incomplete or approximate differences.

Description Attempt to check pricing via Spirit Airlines’ official website/app booking flow for Houston Los Angeles within the requested window. Full credit if Spirit is used directly OR if Spirit is inaccessible (e.g., CAPTCHA, errors, geo/paywall) and the agent clearly reports the blocker and then uses a clearly identified alternate source while noting prices may differ. Partial credit if only third-party sources are used without an evident attempt on Spirit when Spirit appears accessible.

Description Search flights from Houston (use Spirit-available airports such as IAH and/or HOU if offered) to Los Angeles (LAX) covering the window beginning March 5 through March 20. Full credit if the agent evaluates availability/pricing across the window using a reasonable method (e.g., Spirit low-fare calendar, or a justified representative sampling that spans the range and notes any gaps). Partial credit if the agent checks only a few dates without justification or misses one of the endpoints. No credit if the wrong route/airports/date window are used when correct options are available.

Description For any flights found in the date window, determine the incremental cost of selecting a Big Front Seat compared with standard economy as presented in the booking flow (show the underlying values used and the computed difference, per date/flight or as a minmax range). Full credit if the calculation is clearly shown and based on retrieved prices OR if the agent makes a reasonable attempt but Big Front Seat pricing is not obtainable due to external constraints (e.g., seat map wont load, BFS not offered on that flight, site blocks access) and the agent explicitly states this without inventing numbers. Partial credit if only one of the two price points is reported (economy or BFS) when the other is available, or if the calculation is unclear.

Description Clearly state that there are no available flights for those dates. Full credit if the agent demonstrates reasonable checking across the whole window (e.g., calendar/low-fare view, or checks spanning the range) before concluding unavailability, and explicitly distinguishes true unavailability from Spirit-site errors or access blockers. Partial credit if the agent claims no availability after insufficient checking or without clarifying whether the issue might be a site/access problem.

Description Agent must search for flights specifically from Sharjah (SHJ) to Delhi (DEL) with outbound on January 13 and return on January 19.

Description Agent must check whether there are any IndiGo flights available for the specified route and dates. Partial credit if attempts are made but date range misinterpreted or only one leg checked.

Description Agent must verify whether business class seats are offered on the available flights. Partial credit if correctly reports absence/presence of other classes but misses business class detail.

Description If business class is available, agent must extract and present the fare price for the round-trip. Partial credit if price range provided instead of a specific amount.

Description If no flights exist for those dates or business class is not available, agent must clearly state that fact instead of providing incorrect or placeholder pricing.

Description Ensure the response targets IndiGo flights only, the route Sharjah (SHJ) to Delhi (DEL), round trip with outbound on January 13 and return on January 19, and focuses on business class seats. Partial credit if some parameters (e.g., route or dates) are correct but others are missed or altered.

Description Check and clearly state whether IndiGo has flights for the specified dates and whether business class is available on those flights. Partial credit if only flight availability is addressed without confirming business class availability (or vice versa).

Description If business class is available on the specified IndiGo flights and dates, provide the price for the trip. If flights for those dates or business class are not available, explicitly state that in the answer. Full credit is awarded for a clear unavailability statement when applicable. Partial credit for incomplete or unclear pricing (e.g., only one segment priced, unclear total) when availability exists.

Description Attempt to check availability for IndiGo-operated flights from Sharjah (SHJ) to Delhi (DEL) on January 13. Full credit if the agent checks the correct route/date and reports available flight option(s) OR clearly reports that no IndiGo flights are available OR reports an uncontrollable blocker (e.g., site/app down, CAPTCHA/login wall, geo restriction) that prevents verifying availability. Partial credit if the agent checks the correct route but the date is wrong/unclear.

Description Attempt to check availability for IndiGo-operated flights from Delhi (DEL) to Sharjah (SHJ) on January 19. Full credit if the agent checks the correct route/date and reports available flight option(s) OR clearly reports that no IndiGo flights are available OR reports an uncontrollable blocker (e.g., site/app down, CAPTCHA/login wall, geo restriction) that prevents verifying availability. Partial credit if the agent checks the correct route but the date is wrong/unclear.

Description For both legs (Jan 13 SHJ→DEL and Jan 19 DEL→SHJ), determine whether a true 'business class' cabin is offered/available. Full credit if the agent accurately reports, per leg, one of: (a) business class offered and available, (b) business class offered but sold out/unavailable, (c) business class not offered on that flight/route/airline (including cases where IndiGo only sells economy-style fare families), OR (d) the booking channel does not provide enough cabin/fare detail to verify business class and the agent clearly states this limitation/blocker. Partial credit if business class status is only resolved for one leg or is not leg-specific.

Description Provide the business class fare price(s) for any leg(s) where business class is available, clearly tied to the correct leg/date and including currency as shown. Full credit if prices are provided for each leg where business class is available OR if the agent demonstrates a reasonable attempt to retrieve the price but is prevented by an uncontrollable blocker (e.g., fare not displayed without login/payment step, site error/CAPTCHA) and clearly states that. Partial credit if a price is missing currency/context or only one available leg is priced without explanation.

Description The final response must clearly and leg-specifically state the relevant outcome(s): no flights, business class not offered, business class sold out, or inability to verify due to access/visibility limitations. Full credit for clear SHJ→DEL (Jan 13) and DEL→SHJ (Jan 19) statements as applicable. Partial credit if the unavailability/limitation is mentioned but not tied to the correct leg/date.

Description Navigate to or access the official Thai Airways website or authorized booking platform as specified in the task.

Description Locate the flight search or booking interface and set the departure city to Bangkok, Thailand and the arrival city to Singapore.

Description List available Thai Airways outbound flights on November 19 with flight numbers, times, and fares, or explicitly state if no flights are available.

Description List available Thai Airways return flights on December 4 with flight numbers, times, and fares, or explicitly state if no flights are available.

Description Clearly indicate if Thai Airways has no available flights on the specified outbound or return dates.

Description Offer clear next steps or instructions for initiating the booking (e.g., adding selected flights to cart or proceeding to passenger details), without entering personal information.

Description Ensure the search/selection is for a round-trip flight from Bangkok, Thailand to Singapore with outbound on November 19 and return on December 4, specifically on Thai Airways. Partial credit if some but not all of route, dates, or carrier constraints are correctly applied.

Description Find Thai Airways flight options for the specified dates and route. Provide flight details and fares if available. Full credit also awarded if there are no available flights or booking is not possible and the agent explicitly states that outcome. Partial credit for attempts that surface incomplete information or show effort but miss key details.

Description Advance the booking process appropriately (e.g., select flights/fare and proceed up to the customer details/review stage) while stopping before entering any personal/sensitive information or completing payment. Do not fabricate user information. Partial credit if flights are selected but the process does not reach the pre-checkout stage, or if the agent appropriately stops due to unavailability.

Description Attempt to plan the itinerary using Thai Airways (TG) via Thai Airways’ official booking channel. Full credit if the agent (a) selects TG-operated flights for both legs when available, OR (b) correctly determines TG-operated options are not available/bookable on the route/dates and reports that, OR (c) if only TG-marketed/codeshare options appear, the agent clearly distinguishes this and prioritizes TG-operated flights when possible. Partial credit if the agent initially shows non-TG flights but then corrects or explains why TG cannot be used due to availability or access limitations. No credit if the agent proceeds with another airline despite TG-operated options being available and accessible.

Description Configure a round-trip itinerary from Bangkok, Thailand (BKK or DMK; must be Bangkok) to Singapore (SIN) and back. Full credit if cities and round-trip are correct. Partial credit if Bangkok airport is ambiguous but still clearly Bangkok↔Singapore round-trip. No credit if wrong cities are used.

Description Search for and, if available, select a Bangkok→Singapore flight departing on November 19 on Thai Airways’ channel. Full credit if a Nov 19 option is selected OR if the agent clearly demonstrates a Nov 19 search attempt and accurately reports that no TG-bookable flights exist for that date or that the channel is blocked (e.g., CAPTCHA/site error) before results can be verified. Partial credit if the agent searches near the date without confirming Nov 19 results or fails to clearly show/describe the Nov 19 check.

Description Search for and, if available, select a Singapore→Bangkok flight departing on December 4 on Thai Airways’ channel. Full credit if a Dec 4 option is selected OR if the agent clearly demonstrates a Dec 4 search attempt and accurately reports that no TG-bookable flights exist for that date or that the channel is blocked (e.g., CAPTCHA/site error) before results can be verified. Partial credit if the agent searches near the date without confirming Dec 4 results or fails to clearly show/describe the Dec 4 check.

Description After selecting flights (if available), attempt to proceed through the Thai Airways booking flow far enough to demonstrate bookability (e.g., fare selection/itinerary summary and proceed to passenger details). Do not finalize purchase or enter any unprovided personal/payment information. Full credit if the agent reaches the passenger details/payment stage and stops, OR if a blocker prevents proceeding (CAPTCHA, site errors, login required, payment page restrictions) and the agent reports the furthest step reached and blocker encountered. Partial credit if the agent stops at search results without attempting to proceed when the site appears functional.

Description Provide a clear final status: either (a) the selected Thai Airways flights for Nov 19 and Dec 4 are identified as ready to book (include key itinerary details visible such as flight numbers/times and price if shown), OR (b) explicitly state that no flights are available for those dates and/or booking was not possible, including the reason (e.g., sold out, TG does not operate/book the route, website blocked). Full credit for accurate, specific reporting and no hallucinated details; partial credit for vague reporting without reasons.

Description Navigate to or open Vueling’s official flight booking interface or website. Partial credit if an alternative official channel (e.g., mobile app) is identified but not fully opened.

Description Specify Birmingham, UK as the departure city and Barcelona, Spain as the arrival city. Partial credit if one city is correct but the other is incorrect or omitted.

Description Set the outbound date to November 28 and the return date to December 16. Partial credit if one date is correct but the other is incorrect or missing.

Description Perform the search for flights on the specified dates and route. Award full points for successfully executing the search; partial credit if an attempt is made but fails due to a minor error (e.g., wrong date format).

Description If flights are available, present the available outbound and return flight options without completing a booking transaction. If no flights are available or booking is not possible, clearly indicate that. Partial credit if one leg’s availability is reported correctly but the other leg is missed.

Description Locate and identify flight options from Birmingham, UK to Barcelona, Spain departing November 28 and returning December 16. Partial credit may be awarded for finding only one leg or incorrect dates. Full credit is awarded if no flights are available for those dates and the agent clearly indicates that unavailability.

Description Confirm that the flights identified are operated by Vueling as explicitly requested. Partial credit may be awarded if mixed carriers are shown with at least one leg on Vueling. Full credit is awarded if the requirement cannot be met and the agent clearly states that Vueling flights are not available for the requested route/dates.

Description Assist by selecting the appropriate outbound and return flights and proceed up to a pre-checkout stage (e.g., fare selection or price summary) without entering any personal information or completing the purchase.

Description Attempt to search and book using Vueling’s official channels (website/app). Full credit if the agent demonstrably uses Vueling, OR if Vueling is inaccessible (site down, blocked, CAPTCHA, broken booking flow) and the agent clearly reports the blocker. Partial credit if the agent primarily uses a third-party flight search/OTA without first attempting Vueling, unless Vueling access is demonstrably blocked.

Description Configure the itinerary as round-trip from Birmingham (BHX) to Barcelona (BCN) on Vueling. Full credit if the correct origin/destination and round-trip are selected. Full credit also if Vueling does not offer BHXBCN (directly or via its booking engine) and the agent clearly reports that the route is not available/servable on Vueling. Partial credit if the agent uses alternate nearby airports/cities without first confirming BHXBCN is not available on Vueling.

Description Enter/select departure date November 28 and return date December 16 on Vueling. Full credit if both dates are correctly set OR if, after attempting the correct search on Vueling, the agent finds no flights/inventory for one or both dates and clearly reports the unavailability. Partial credit if only one date is correct or if the agent proposes/uses alternative dates without first establishing that the requested dates are unavailable on Vueling (or that the route itself is not served).

Description Advance the Vueling booking process as far as possible without completing a binding transaction or entering unprovided personal/payment details. Full credit if the agent reaches the stage where flights/fare are selected and the next step would require passenger details/payment (or equivalent), then stops. Full credit also if the agent cannot proceed due to external blockers (no flights available, route not served, login/verification required, technical errors) and clearly reports what prevented further progress. Partial credit if the agent stops after a basic search without attempting to select a flight when options are available.

Description Provide a clear final status consistent with observed evidence: either (a) booking is ready to be finalized (flight/fare selected, awaiting passenger/payment details), or (b) no available flights for the specified dates, (c) route not served on Vueling, or (d) booking not possible due to access/technical blockers. Partial credit if the report is vague about whether the issue was availability vs. platform/technical limitation.

Description Identify the upcoming Saturday as the departure date and set the return date exactly two weeks later. Partial credit if one of the dates is correct but not both.

Description Ensure the departure airport is Istanbul Airport (IST) and the arrival airport is John F. Kennedy International Airport (JFK) for both outbound and return segments. Partial credit if only one segment uses the correct airports.

Description Find and select round-trip flights operated by Turkish Airlines for the specified dates and airports. Partial credit if flights are found but the airline differs or only one leg uses Turkish Airlines.

Description Clearly indicate whether flights are available for the chosen dates or explicitly state that no flights are available or booking is not possible. Full credit for a clear statement of availability or unavailability.

Description Add the selected round-trip Turkish Airlines flights to the booking or reservation cart, stopping before entering any personal or payment information. Partial credit if only one leg is added.

Description Identify the upcoming Saturday as the departure date and set the return date exactly two weeks later. Partial credit if the agent recognizes the two-week duration but miscalculates the specific dates.

Description Search for and identify available round-trip flights operated by Turkish Airlines from Istanbul Airport (IST) to John F. Kennedy International Airport (JFK) on the specified dates. Ensure both legs correspond to the route and airline. Full credit is awarded if no flights are available and the agent explicitly states that. Partial credit if the agent attempts the search but uses a different airline or incorrect route.

Description Select suitable flight options and guide the user through pre-booking steps (e.g., choosing fare types, proceeding up to the checkout or customer details page) while stopping short of entering personal information or completing payment. Partial credit if only the flight selection is presented without guidance on next steps.

Description Clearly state if there are no available flights for the requested dates or if the booking cannot proceed. Partial credit for acknowledging issues but not clearly communicating their impact on booking.

Description Attempt to use Turkish Airlines’ official booking channel (website or app) to search for a round-trip itinerary with origin Istanbul Airport (IST) and destination John F. Kennedy International Airport (JFK). Full credit if the agent performs this search or clearly reports being blocked (e.g., CAPTCHA, site down, persistent errors) after a reasonable attempt. Partial credit if the agent verifies the route via a third-party site due to documented Turkish Airlines access issues. No credit if the agent searches the wrong route/airports without correction when the correct route is feasible.

Description Use a defensible interpretation of 'upcoming Saturday' (relative to task execution time) as the departure date and set a return date about 14 days later (±1 day acceptable due to date-boundary/time-zone ambiguity). Full credit if the chosen dates match this interpretation or if the agent explains the interpretation used. Full credit (not penalized) if the agent cannot confirm dates because the platform is inaccessible, as long as it states what dates it attempted/intended to use. Partial credit if the return length is off by more than 1 day but the departure Saturday is correct and the agent documents the choice.

Description If flights are available and the Turkish Airlines booking flow is accessible, proceed through selection far enough to show the itinerary can be booked (e.g., select outbound/return flights and reach review/passenger details), then stop before entering personal data (name/passport/contact) or payment. Full credit if the agent reaches this stage and stops appropriately. Also award full credit if the agent cannot reach this stage due to external blockers (no availability on the dates, site/app errors, CAPTCHA/login wall) and clearly reports the blocker and where the flow stopped. Partial credit if the agent finds availability but stops earlier than necessary without a stated reason, while the site is functioning.

Description Provide either (a) booking-relevant itinerary details observed (departure/return dates, times and/or flight numbers, and price if shown), or (b) a clear statement that flights are unavailable for the required dates or that booking is not possible due to a specific blocker (sold out, site error, CAPTCHA/login wall, cannot proceed without crossing the critical point). Full credit if the reporting matches what was actually observed/attempted and is explicit. Partial credit if availability/unavailability is stated but key details (dates attempted, what failed) are missing.

Description Navigate to or load the official AirAsia flight booking website or app as specified in the task.

Description Enter the origin as Hong Kong and the destination as Manila in the flight search form. Partial credit for one city correct; full credit for both cities correct.

Description Input the departure date of December 2 and the return date of December 8. Partial credit if one date is correct; full credit if both dates are entered accurately.

Description Execute the flight search and identify any available outbound and inbound flights for the specified dates. Partial credit if search is run but results are not clearly identified; full credit if flights are listed or results clearly shown.

Description If no flights are available on the requested dates or booking cannot proceed, explicitly state that in the response. Full credit for a clear statement of unavailability.

Description Navigate to or access AirAsia’s official booking platform (website or app) to begin the flight search and booking process as specified in the task.

Description Set up a round-trip search from Hong Kong (HKG) to Manila (MNL) departing on December 2 and returning on December 8, using AirAsia. Partial credit may be awarded if some parameters (e.g., origin/destination or one of the dates) are correctly configured but others are missing or incorrect.

Description If AirAsia flights are available on the specified dates, select suitable outbound and return AirAsia flights and proceed up to a non-binding step (e.g., fare summary or pre-checkout) without entering personal details or payment. If there are no available flights for those dates or booking is not possible, clearly state that outcome. Full credit is awarded for correctly indicating unavailability or impossibility as per the task instructions.

Description Attempt to search/book using AirAsia first (AirAsia website or app). Full credit if the agent uses AirAsia to initiate the search OR if AirAsia is inaccessible/unusable due to uncontrollable factors (site down, CAPTCHA, geo/region restriction, forced login before search, persistent errors) and the agent clearly reports the blocker. If an AirAsia blocker occurs, the agent may use another source only to inform the user, but should still be scored as full credit on this criterion if the AirAsia blocker is documented. Partial credit if the agent uses a different site without first attempting AirAsia when AirAsia appears accessible.

Description Configure the itinerary to depart from Hong Kong (HKG) and arrive in Manila (MNL) within the AirAsia search flow. Full credit if correct origin/destination are selected/verified OR if AirAsia does not support selling this route (or shows no routings) and the agent clearly reports that finding based on an AirAsia attempt. Partial credit if the agent initially selects an incorrect airport/city but corrects it. No credit if the final checked route is different while the correct route was available and accessible.

Description Select/verify the requested outbound date (December 2) and inbound date (December 8) in the AirAsia search/booking flow. Full credit if both dates are correctly selected/verified OR if AirAsia shows no available flights/fare inventory for one or both requested dates and the agent accurately reports unavailability for those dates (based on an AirAsia attempt). Partial credit if only one of the two dates is correctly selected/verified or if date verification is incomplete due to a clearly described AirAsia blocker after partial progress.

Description Advance the AirAsia booking process as far as possible without completing a binding transaction or entering personal/payment details. Full credit if the agent selects flights for both legs (or the best available AirAsia option if exact dates are unavailable and the user’s instruction allows only reporting) and reaches the passenger details and/or payment stage, then stops. If progression is prevented by uncontrollable issues (errors, inventory disappearing, forced login, payment required before reviewing itinerary, etc.), full credit if the agent clearly reports where and why it could not proceed. Partial credit if the agent finds flights but does not select them or stops significantly earlier without a stated blocker.

Description Provide a clear final status aligned with what was observed: either (a) booking is staged and ready for passenger details/payment with selected flights, or (b) no flights are available for Dec 2/Dec 8 on AirAsia, or (c) booking was not possible due to an AirAsia/platform limitation (with a specific reason such as CAPTCHA, site errors, route not sold, login wall). Partial credit if the status is vague or missing the reason. No credit if the agent claims a booking was completed/purchased or asserts unavailability without evidence of checking (or without documenting an AirAsia blocker that prevented checking).

Description Access the official British Airways booking interface or website to begin the flight reservation process.

Description Select a round-trip itinerary from Manchester Airport (MAN) to London Heathrow (LHR) departing on the upcoming Friday and returning four days later.

Description Perform the search using the specified criteria and determine whether British Airways flights are available on those dates.

Description If flights are found, present the available options (including times and prices) and outline next steps without proceeding to enter personal or payment information. If no flights are available or booking cannot be completed, explicitly state this.

Description Identify the calendar date for the upcoming Friday and set the return date exactly four days after the departure date. Partial credit if only one of the dates (outbound or return) is correctly determined or if the agent notes ambiguity and makes a reasonable assumption consistent with 'four days'.

Description Ensure the search/selection is for a round-trip with the outbound leg from Manchester Airport (MAN) to London Heathrow (LHR) and the return leg from LHR back to MAN. Partial credit if the route is correct but the trip type is not set to round-trip, or if a different London airport is mistakenly used but the intent is clear.

Description Filter/select flights operated by British Airways for both legs. Partial credit if British Airways is identified but mixed with non-BA flights, or if BA-marketed flights are chosen without confirming BA operation.

Description Search for flights matching the specified dates and constraints and present available options. Full credit is awarded if no flights are available or booking is not possible and the agent clearly indicates that outcome, as requested. Partial credit if the search is attempted but results are unclear or incomplete.

Description Select the suitable BA round-trip flights and proceed to a non-binding step (e.g., review/price summary or pre-checkout) while stopping before any personal information is entered. Partial credit if only one leg is selected or if the agent clearly explains the next non-binding step to continue the booking.

Description Make a reasonable attempt to search for the itinerary using British Airways (e.g., BA website/app). Full credit if the agent attempts BA but is blocked by an external issue (CAPTCHA, outages, mandatory login preventing search, errors) and clearly reports the blocker. Partial credit if BA is not attempted first but BA-operated options are still explicitly verified elsewhere. No credit if the agent makes no BA attempt and does not justify why BA could not be used.

Description If search results can be viewed, ensure the selected flights are British Airways operated/marketed (or clearly identified as BA flights). Full credit if BA flights are selected, or if none exist for the route/dates and the agent explicitly reports that BA has no available flights. Partial credit if the airline/operator is unclear. No credit if non-BA flights are selected while BA flights are available and visible.

Description Search/select MAN 3 LHR for both outbound and return. Full credit if the correct route is used, OR if BA search cannot be completed due to an external blocker and the agent clearly states it was attempting MAN3LHR. Partial credit if one leg uses the wrong airport, unless the agent explains the correct route had no options and is attempting a nearest-airport alternative for discovery (must still clearly label it as an alternative). No credit if the agent pursues a different route despite MAN3LHR being available/visible.

Description Set round-trip with departure on the upcoming Friday (relative to execution) and return four days later. Full credit if dates match, OR if the agent demonstrates an attempt to use those dates and clearly reports that no flights are available for the exact dates (or BA cannot display results due to external blocker). Partial credit if return date is off by 1 day without clearly stating the exact-date option was unavailable. No credit if unrelated dates are used when the correct dates are available/visible.

Description If flights are available, select an outbound and return option and proceed to the step just before entering passenger personal details/payment or finalizing purchase. Full credit if the agent reaches passenger/details or payment step without completing purchase, OR if progress is prevented by an uncontrollable blocker (e.g., cannot proceed without login, repeated site errors) that is clearly reported along with what step was reached. Partial credit if flights are found but not selected and no reason is given. No credit if the agent completes the transaction or invents personal/payment data.

Description Provide a clear final result consistent with observed evidence: either selected round-trip BA flight details for the specified dates (dates/times and price if shown), OR a clear statement that flights are unavailable for those dates/route on BA, OR that booking is not possible due to a specific blocker encountered. Full credit for accurate, non-hallucinated reporting even when unavailability/blockers prevent booking. Partial credit if key details are missing despite being visible, or the unavailability claim is ambiguous. No credit if the agent claims success or unavailability without support.

Description Navigate to the Sun Country Airlines website or booking portal to initiate the flight search.

Description Input Duluth, MN as the departure airport and Phoenix, AZ as the arrival airport; partial credit for entering one correctly.

Description Choose January 17 as the departure date and January 31 as the return date; partial credit for selecting one correctly.

Description Display available flight options (including times and fares) for the specified route and dates; if no flights are available, clearly state that no flights are available.

Description If flights are available, select the desired outbound and inbound flights and proceed to the booking/reservation confirmation step without inputting passenger or payment information.

Description Navigate to and use Sun Country Airlines’ official booking system or equivalent official channel to search for the requested itinerary. Partial credit if Sun Country schedules are checked without clearly using the booking tool.

Description Search for a round-trip itinerary on Sun Country Airlines from Duluth, MN (DLH) to Phoenix, AZ with departure on January 17 and return on January 31. PHX or other Phoenix-area airport served by Sun Country is acceptable. Partial credit if only one leg/date is correct or if cities are correct but dates are slightly mishandled.

Description If flights exist that match the constraints, identify at least one valid outbound and one valid return Sun Country option on those dates. If there are no available flights for those dates or booking is not possible, explicitly state that in the answer. Partial credit if availability is discussed but feasibility is not clearly concluded.

Description When flights are available, select appropriate outbound and return Sun Country flights and proceed toward booking (e.g., fare/itinerary selection) while stopping before entering any personal or payment details. Full credit also awarded if flights are unavailable and this step is skipped with an explicit unavailability statement. Partial credit if only listing results without selecting both legs.

Description Attempt to access and use Sun Country’s official booking path (website/app) to search the requested itinerary. Full credit if the agent makes a reasonable attempt and, if blocked (e.g., CAPTCHA, outage, technical error, mandatory login) clearly reports the blocker. Partial credit if the agent primarily uses a third-party site before attempting Sun Country. No credit if Sun Country is not attempted or a different airline is used without addressing Sun Country.

Description Use Duluth, MN (DLH) to Phoenix, AZ (PHX) departing January 17 and returning January 31 in the Sun Country search. Full credit if these exact parameters are used, or if the agent discovers Sun Country’s interface cannot accept/serve one of the airports/dates and clearly reports that limitation. Partial credit if one field is initially incorrect but is recognized and corrected. No credit if the agent searches a materially different route or dates without justification.

Description Determine whether Sun Country offers flights for DLH→PHX (round trip) on Jan 17–Jan 31 and report the outcome. Full credit if the agent (a) identifies at least one available option matching the dates/route, OR (b) accurately states that no Sun Country flights are available for those dates/route, OR (c) explains that booking cannot be completed due to an external blocker (site/access/technical issue) after a reasonable attempt. Partial credit if the check is incomplete (e.g., only one-way) but the agent is transparent about limitations. No credit if the agent fabricates availability/unavailability.

Description Select specific departing and returning flights (and any required fare class) in Sun Country’s booking flow and proceed to the point just before entering passenger personal details and/or payment. Full credit if both legs are selected and the agent stops before personal/payment entry. Partial credit if only one leg is selected or progress stops earlier despite the flow being available. No credit if the agent attempts to finalize purchase or enters personal/payment information not provided by the user.

Description Final answer must accurately reflect what was found/done: selected flights and key details if available, or clearly indicate no availability/booking not possible. Full credit if the agent avoids inventing flight numbers, prices, confirmations, or unsupported claims, and clearly distinguishes observed results from uncertainty. Partial credit if minor details are unclear but uncertainty is explicitly stated. No credit for hallucinated booking confirmations or invented availability/unavailability.

Description Successfully navigate to or access the official Thai Airways booking system or website to initiate the flight booking process.

Description Perform a flight search specifically for a round-trip itinerary from Bangkok to London, departing on November 16 and returning on November 26. Partial credit for correct route with one date wrong or vice versa.

Description Identify whether Thai Airways flights are available for the specified dates. If available, provide key details of the options (flight numbers, times, basic fares). If no flights exist or booking is not possible, clearly state that.

Description Guide the user through selecting the desired flight option and show how to proceed to the booking or cart stage without requiring entry of personal or payment information.

Description Ensure the search/attempt is for Thai Airways flights on a round-trip from Bangkok to London departing on November 16 and returning on November 26. Partial credit if only some constraints (e.g., route or one date) are correctly applied.

Description Find Thai Airways flight options matching the specified dates and route, or clearly state that no flights are available or that booking is not possible. Full credit is awarded if unavailability is explicitly indicated when applicable. Partial credit for attempting to check availability but providing incomplete or ambiguous results.

Description If flights are available, select suitable flights and progress the booking process as far as possible without entering personal/sensitive information (e.g., up to fare summary or customer details page). If no flights are available or booking is not possible, it is acceptable to skip this step and clearly state the issue. Partial credit for outlining the steps to proceed without actually entering personal data.

Description Attempt to search and select an itinerary specifically marketed and/or operated by Thai Airways (TG) for the requested route/dates (e.g., on Thai Airways’ site or a credible booking channel showing marketing/operating carrier). Full credit if Thai Airways flights are selected, OR if the agent clearly reports that no Thai Airways-marketed/operated flights are available/bookable for this route/dates (including cases where only non-Thai options appear) or that the Thai search is blocked by an external issue (CAPTCHA/site error). Partial credit if the agent uses another airline without first establishing (via reasonable checking) that Thai Airways cannot fulfill the request.

Description Configure the itinerary as a round-trip from Bangkok (preferably BKK; DMK acceptable only if explicitly noted as an alternative) to London (any major London airport such as LHR/LGW/LCY/STN/LTN if supported). Full credit if the cities are correct even if airport choice is constrained by the booking tool; the agent should note any forced airport substitution. Partial credit if airports are ambiguous but cities are correct and the agent acknowledges the ambiguity.

Description Use the requested dates: depart November 16 and return November 26. Full credit if these exact dates are used, OR if the agent shows it attempted these dates but clearly reports that no flights are available/bookable on those exact dates (or that an external blocker prevented checking). Partial credit if adjacent dates are explored only after noting/justifying that the exact dates appear unavailable or unbookable.

Description Proceed through the booking process to the last step before requiring passenger personal details and/or payment, such as reaching a price summary or passenger-details page, without completing purchase. Full credit if the agent reaches that step and stops, OR if it cannot progress due to external blockers outside the agent’s control (CAPTCHA, site outage, mandatory login, session errors) and it clearly reports the blocker and how far it got. Partial credit if the agent only performs a superficial search without attempting to select an itinerary when the site is accessible.

Description Provide a clear, accurate outcome: either (a) the itinerary is ready to be booked (selected flights, dates, and any visible fare/summary) but not purchased, OR (b) explicitly state that booking is not possible because Thai Airways flights are unavailable for the requested dates/route or because an external blocker prevented completion, with a brief explanation of what was checked/where it failed. No credit for claiming a completed booking or confirmed availability without support.

Description Ensure the flight search is for Lufthansa, departing from Frankfurt, Germany, and arriving in Tel Aviv, Israel. Partial credit if one element is missing or incorrect.

Description Use November 18 as the departure date and November 30 as the return date. Partial credit if only one date is correct or if dates are transposed.

Description Search Lufthansa's booking system for available flights on the specified route and dates. Partial credit if the agent attempts a search but does not confirm availability.

Description List any available flight options with basic details (departure/arrival times, prices) and explain how to select or reserve the flight without entering personal information. Partial credit for incomplete or missing details.

Description If no flights are available or booking is not possible for those dates, explicitly state that no flights can be booked. Full credit if clearly communicated.

Description Use Lufthansa as the carrier for a flight from Frankfurt, Germany to Tel Aviv, Israel, departing on November 18 and returning on November 30. Partial credit if only some parameters are correct (e.g., correct route but incorrect dates, or Lufthansa Group/codeshare instead of Lufthansa).

Description Search for and identify available Lufthansa flights matching the specified route and dates. Present specific options if available. Full credit also awarded if there are no available flights or booking is not possible and this is clearly stated. Partial credit if an attempt is made but results are incomplete or uncertain.

Description Select a specific itinerary and guide or proceed through booking steps up to review/cart or price confirmation, without entering personal or payment information or fabricating user details. Partial credit for clearly outlining next steps if direct progression isn't possible.

Description Attempt to search for flights marketed and/or operated by Lufthansa for Frankfurt (FRA) ↔ Tel Aviv (TLV) with departure on Nov 18 and return on Nov 30 using Lufthansa’s booking flow when accessible. Full credit if the agent makes a reasonable attempt on Lufthansa.com but is blocked by an external issue (e.g., CAPTCHA, outage, geo restrictions) and clearly reports it; in that case, using an equivalent reliable Lufthansa source (e.g., Lufthansa mobile site/app screenshots, Lufthansa group booking interface, or a reputable OTA clearly showing Lufthansa-marketed flights) also earns full credit. Partial credit if the agent searches the right cities with minor date/airport deviations that are clearly justified (e.g., nearby airport only if FRA unavailable), or if Lufthansa marketing/operation is not clearly verified. No credit if the route or dates are materially different without justification when correct inputs were feasible.

Description Determine and report whether Lufthansa flight options are available for Nov 18 (outbound) and Nov 30 (return) based on the search results. Full credit if at least one valid Lufthansa option is found and reported as available, OR if no such options appear and the agent clearly reports unavailability for those dates. If availability cannot be determined due to external blockers (site inaccessible, search results not loading, persistent errors), full credit is awarded if the agent clearly states that availability could not be confirmed and explains the blocker. Partial credit if the conclusion is ambiguous or missing key evidence/details (e.g., fails to indicate Lufthansa vs. other airlines). No credit if the agent asserts availability/unavailability without any substantiated basis.

Description If flights are available, proceed through the Lufthansa booking process (or equivalent Lufthansa booking interface) by selecting an itinerary consistent with FRA↔TLV and Nov 18–Nov 30, reaching the latest possible step without completing a binding purchase/booking. Full credit if the agent reaches the passenger-details and/or payment stage and stops, OR if an external gating issue prevents further progress (e.g., login requirement, verification step, site errors, session timeouts) and the agent documents that blocker after reasonable attempts. Partial credit if the agent identifies available options but stops significantly earlier without a clear blocker or without selecting an itinerary. No credit if the agent attempts to finalize the booking/purchase or enters fabricated personal/payment information.

Description Provide a final answer that either (a) summarizes the selected Lufthansa itinerary for Nov 18–Nov 30 (confirming route and dates and that booking was progressed up to the critical point, or explaining where it stopped and why), OR (b) explicitly states that there are no available flights for those dates or that booking is not possible (including the reason, such as unavailability or technical blockers). Full credit if the outcome is clearly and correctly stated with the required fallback when applicable; partial credit if the summary is missing essential elements (route/dates/status) but the overall outcome is still understandable. No credit if the agent claims a completed booking or otherwise contradicts critical-point boundaries.

Description Navigate to or access the official Sun Country Airlines booking platform.

Description Input 'Tampa, FL' as the departure city and 'Dallas, TX' as the arrival city. Partial credit if one is correct and the other is incorrect.

Description Set the outbound date to February 9 and the return date to February 28. Partial credit for correctly entering one of the two dates.

Description Perform the flight search and either list available outbound and return flight options for the specified dates or explicitly indicate that no flights are available.

Description Select the appropriate outbound and return flights and present a booking summary or ‘Add to Cart’ step, stopping before entering any personal or payment information.

Description Access and use Sun Country Airlines' official website or booking channel to search for the requested flights, not a different airline.

Description Configure a round-trip search from Tampa, FL to Dallas, TX with outbound date February 9 and return date February 28. Partial credit if one date or city is correct but others are incorrect.

Description Search for flights on the specified dates with Sun Country. Full credit for finding suitable flight options; full credit also if no flights are available or booking is not possible AND this is clearly stated. Partial credit if the search is attempted but inconclusive.