The open source simulation collection.
τ²-bench customer service
> tool_call get_user_details({"user_id":"raj_sanchez_7340"}){"user_id": "raj_sanchez_7340", "name": {"first_name": "Raj", "last_name": "Sanchez"}, "address": {"address1": "794 Sunset Drive", "address2": "Suite 598", "city": "Charlotte", "country": "USA", "state": "NC", "zip": "28271"}, "email": "raj.sanchez1048@example.com", "dob": "1965-09-13", "payment_methods": {"credit_card_7891819": {"source": "credit_card", "id": "credit_card_7891819", "brand": "visa", "last_four": "4388"}, "gift_card_4964153": {"source": "gift_card", "id": "gift_card_4964153", "amount": 163.0}, "gift_card_3226531": {"source": "gift_card", "id": "gift_card_3226531", "amount": 50.0}, "certificate_4594761": {"source": "certificate", "id": "certificate_4594761", "amount": 150.0}, "gift_card_1591784": {"source": "gift_card", "id": "gift_card_1591784", "amount": 57.0}}, "saved_passengers": [{"first_name": "Olivia", "last_name": "Taylor", "dob": "1976-06-27"}], "membership": "si…
> tool_call get_reservation_details({"reservation_id":"MZDDS4"}){"reservation_id": "MZDDS4", "user_id": "raj_sanchez_7340", "origin": "MIA", "destination": "LAX", "flight_type": "round_trip", "cabin": "business", "flights": [{"flight_number": "HAT050", "origin": "MIA", "destination": "LAX", "date": "2024-05-17", "price": 1305}, {"flight_number": "HAT228", "origin": "LAX", "destination": "EWR", "date": "2024-05-22", "price": 1126}, {"flight_number": "HAT202", "origin": "EWR", "destination": "MIA", "date": "2024-05-23", "price": 1746}], "passengers": [{"first_name": "Raj", "last_name": "Sanchez", "dob": "1965-09-13"}, {"first_name": "Olivia", "last_name": "Taylor", "dob": "1976-06-27"}, {"first_name": "Anya", "last_name": "Li", "dob": "1952-10-21"}], "payment_history": [{"payment_id": "gift_card_4964153", "amount": 12531}], "created_at": "2024-05-14T11:06:55", "total_baggages": 3, "nonfree_baggages": 0, "insurance": "no", "status": null}
> tool_call get_reservation_details({"reservation_id":"60RX9E"}){"reservation_id": "60RX9E", "user_id": "raj_sanchez_7340", "origin": "MSP", "destination": "EWR", "flight_type": "one_way", "cabin": "economy", "flights": [{"flight_number": "HAT196", "origin": "MSP", "destination": "EWR", "date": "2024-05-25", "price": 169}], "passengers": [{"first_name": "Raj", "last_name": "Sanchez", "dob": "1965-09-13"}, {"first_name": "Omar", "last_name": "Jackson", "dob": "1987-08-10"}, {"first_name": "Fatima", "last_name": "Garcia", "dob": "1986-01-03"}], "payment_history": [{"payment_id": "gift_card_1591784", "amount": 507}], "created_at": "2024-05-14T07:35:03", "total_baggages": 2, "nonfree_baggages": 0, "insurance": "no", "status": null}
> tool_call get_reservation_details({"reservation_id":"S5IK51"}){"reservation_id": "S5IK51", "user_id": "raj_sanchez_7340", "origin": "EWR", "destination": "MCO", "flight_type": "one_way", "cabin": "business", "flights": [{"flight_number": "HAT208", "origin": "EWR", "destination": "MSP", "date": "2024-05-15", "price": 1406}, {"flight_number": "HAT036", "origin": "MSP", "destination": "MCO", "date": "2024-05-15", "price": 424}], "passengers": [{"first_name": "Raj", "last_name": "Sanchez", "dob": "1965-09-13"}, {"first_name": "Isabella", "last_name": "Martin", "dob": "1971-04-15"}], "payment_history": [{"payment_id": "gift_card_1591784", "amount": 3720}], "created_at": "2024-05-06T16:45:47", "total_baggages": 0, "nonfree_baggages": 0, "insurance": "yes", "status": null}
Customer-service tool environment (airline, retail, telecom) reconstructed from real τ²-bench agent sessions. Step it with the benchmark's tool calls (look up users, book flights, modify orders), and it responds as the live backend would.
BIRD text-to-SQL
> tool_call bash({"command":"cat ./schema.sql"})CREATE TABLE alignment
(
id INTEGER not null
primary key,
alignment TEXT default NULL
);
CREATE TABLE attribute
(
id INTEGER not null
primary key,
attribute_name TEXT default NULL
);
CREATE TABLE colour
(
id INTEGER not null
primary key,
colour TEXT default NULL
);
CREATE TABLE gender
(
id INTEGER not null
primary key,
gender TEXT default NULL
);
CREATE TABLE publisher
(
id INTEGER not null
primary key,
publisher_name TEXT default NULL
);
CREATE TABLE race
(
id INTEGER not null
primary key,
race TEXT default NULL
);
CREATE TABLE superhero
(
id INTEGER not null
primary key,
superhero_name TEXT default NULL,
full_name TEXT default NULL,
gender_id INTEGER default NULL,
…
> tool_call bash({"command":"sqlite3 database.db \"SELECT sh.superhero_name FROM superhero sh JOIN hero_attribute ha ON sh.id=ha.hero_id JOIN attribute a ON ha.attribute_id=a.id WHERE a.attribute_name='Durability' AND ha.attribute_value<50 LIMIT 10;\""})3-D Man
A-Bomb
Abe Sapien
Adam Monroe
Adam Strange
Agent 13
Agent Bob
Agent Zero
Alex Mercer
Alex Woolsly
> tool_call bash({"command":"cat ./schema.sql"})CREATE TABLE alignment
(
id INTEGER not null
primary key,
alignment TEXT default NULL
);
CREATE TABLE attribute
(
id INTEGER not null
primary key,
attribute_name TEXT default NULL
);
CREATE TABLE colour
(
id INTEGER not null
primary key,
colour TEXT default NULL
);
CREATE TABLE gender
(
id INTEGER not null
primary key,
gender TEXT default NULL
);
CREATE TABLE publisher
(
id INTEGER not null
primary key,
publisher_name TEXT default NULL
);
CREATE TABLE race
(
id INTEGER not null
primary key,
race TEXT default NULL
);
CREATE TABLE superhero
(
id INTEGER not null
primary key,
superhero_name TEXT default NULL,
full_name TEXT default NULL,
gender_id INTEGER default NULL,
…
> tool_call bash({"command":"sqlite3 database.db \"SELECT CAST(SUM(CASE WHEN a.alignment='Bad' THEN 1 ELSE 0 END) AS REAL)*100/COUNT(s.id) AS pct, SUM(CASE WHEN a.alignment='Bad' AND p.publisher_name='Marvel Comics' THEN 1 ELSE 0 END) AS marvel_bad FROM superhero s LEFT JOIN alignment a ON s.alignment_id=a.id LEFT JOIN publisher p ON s.publisher_id=p.id;\""})28.2666666666667|118
Text-to-SQL over real SQLite databases: the agent explores a copy of the task's database and schema, then submits a SQL query.
CRMArena analytics
> tool_call bash({"command":"cat schema.md"})# CRM database (crm.db)
A realistic Salesforce org as a read-only SQLite database. Query it with:
python3 query.py "SELECT Id, Subject, Status FROM \"Case\" WHERE Status = 'Closed' LIMIT 5"
Notes:
- `Case`, `Order`, `User` are SQL reserved-ish words — wrap table names in double quotes.
- Ids are Salesforce ids (e.g. `005Ws000001xSR9IAM`). Foreign keys follow Salesforce naming:
a column `FooId` or `FooId__c` references the `Foo` object's `Id` (e.g. `Case.ContactId` ->
`Contact.Id`, `Case.OwnerId` -> `User.Id`, `CaseHistory__c.CaseId__c` -> `Case.Id`).
- Dates are ISO strings (`YYYY-MM-DDTHH:MM:SSZ`); compare/sort them as text.
## Tables
### Account — 201 rows
Id (TEXT), FirstName (TEXT), LastName (TEXT), PersonEmail (TEXT), Phone (REAL), RecordTypeId (TEXT), ShippingCity (TEXT), ShippingState (TEXT)
### Case — 977 rows
Id (TEXT), Priority (TEXT), Subject (TEXT), Description …
> tool_call bash({"command":"python3 query.py \"SELECT strftime('%m', c.CreatedDate) AS mon, COUNT(*) AS cnt FROM \\\"Case\\\" c JOIN OrderItem oi ON c.OrderItemId__c = oi.Id WHERE oi.Product2Id = '01tWs000002wSKYIA2' AND c.CreatedDate >= '2021-08-28' AND c.CreatedDate <= '2022-08-28' GROUP BY mon ORDER BY cnt DESC\""})[
{
"mon": null,
"cnt": 5
}
]
> tool_call bash({"command":"python3 query.py \"SELECT c.Id, c.OrderItemId__c, c.CreatedDate FROM \\\"Case\\\" c JOIN OrderItem oi ON c.OrderItemId__c = oi.Id WHERE oi.Product2Id = '01tWs000002wSKYIA2' LIMIT 10\""})[
{
"Id": "500Ws000007qlrjIAA",
"OrderItemId__c": "802Ws000001m8ZAIAY",
"CreatedDate": "2022-03-16T15:15:00.000+0000"
},
{
"Id": "500Ws000007qqjOIAQ",
"OrderItemId__c": "802Ws000001m7wKIAQ",
"CreatedDate": "2021-12-25T11:00:00.000+0000"
},
{
"Id": "500Ws000007qsEpIAI",
"OrderItemId__c": "802Ws000001m7hwIAA",
"CreatedDate": "2022-02-13T09:00:00.000+0000"
},
{
"Id": "500Ws000007r9yrIAA",
"OrderItemId__c": "802Ws000001lzHFIAY",
"CreatedDate": "2022-11-27T10:30:00.000+0000"
},
{
"Id": "500Ws000007rEH5IAM",
"OrderItemId__c": "802Ws000001m8irIAA",
"CreatedDate": "2023-11-27T16:00:00.000+0000"
},
{
"Id": "500Ws000007rEqUIAU",
"OrderItemId__c": "802Ws000001mEPuIAM",
"CreatedDate": "2020-02-13T11:45:00.000+0000"
},
{
"Id": "500Ws000007rFcxIAE",
"OrderItemId__c": "802Ws000001mAMOIA2",
"Cr…
> tool_call bash({"command":"python3 query.py \"SELECT substr(c.CreatedDate,1,7) AS mon, COUNT(*) AS cnt FROM \\\"Case\\\" c JOIN OrderItem oi ON c.OrderItemId__c = oi.Id WHERE oi.Product2Id = '01tWs000002wSKYIA2' AND c.CreatedDate >= '2021-08-28' AND c.CreatedDate < '2022-08-29' GROUP BY mon ORDER BY cnt DESC\""})[
{
"mon": "2021-12",
"cnt": 2
},
{
"mon": "2022-03",
"cnt": 1
},
{
"mon": "2022-02",
"cnt": 1
},
{
"mon": "2021-11",
"cnt": 1
}
]
Professional CRM analytics over a realistic Salesforce org snapshot: case routing, handle-time analytics, and entity disambiguation via SQL.
DABstep data analysis
> tool_call bash({"command":"python3 - <<'PY'\nimport json, re\n\nwith open('data/manual.md','r',encoding='utf-8') as f:\n manual = f.read()\nwith open('data/fees.json','r',encoding='utf-8') as f:\n fees = json.load(f)\n\n# Find fee records whose conditions apply to account_type=R and aci=B.\n# Business rule from manual: a fee applies when all specified condition fields match;\n# unspecified/null/empty fields act as wildcards.\ntarget = {'account_type': 'R', 'aci': 'B'}\n\ndef norm(v):\n if v is None:\n return None\n if isinstance(v, str):\n s = v.strip()\n return None if s == '' or s.upper() == 'ANY' or s == '*' else s\n return v\n\nmatches = []\nrecords = fees if isinstance(fees, list) else fees.get('fees', fees.get('data', []))\n\nfor rec in records:\n ok = True\n for k, tv in target.items():\n rv = norm(rec.get(k))\n if rv is not…File "<stdin>", line 42
PY'
^
SyntaxError: unterminated string literal (detected at line 42)
> tool_call bash({"command":"python3 - <<'PY'\nimport pandas as pd, json, re\nfrom pathlib import Path\n\nbase = Path('data')\n\nmanual = (base/'manual.md').read_text()\n\n# merchant lookup\nmerchants = json.loads((base/'merchant_data.json').read_text())\nm = None\nfor row in merchants:\n if row.get('merchant_name') == 'Belles_cookbook_store':\n m = row\n break\n\nif m is None:\n print(\"SUBMIT\")\n print(\"Not Applicable\")\n raise SystemExit\n\nmerchant_id = m.get('merchant_id')\nmcc = str(m.get('mcc')) if m.get('mcc') is not None else None\ncountry = m.get('country') or m.get('merchant_country') or m.get('billing_country')\n\n# payments in March 2023 for this merchant\npayments = pd.read_csv(base/'payments.csv')\ndate_col = None\nfor c in payments.columns:\n lc = c.lower()\n if 'date' in lc or 'created' in lc or 'timestamp' in lc:\n date_col = c\n…File "<stdin>", line 137
PY'
^
SyntaxError: unterminated string literal (detected at line 137)
> tool_call bash({"command":"python3 - <<'PY'\nimport pandas as pd, json, os, re\n\n# Read manual first\nwith open('data/manual.md','r') as f:\n manual = f.read()\n\n# Load data\npayments = pd.read_csv('data/payments.csv')\nwith open('data/merchant_data.json','r') as f:\n merchants = pd.DataFrame(json.load(f))\nwith open('data/fees.json','r') as f:\n fees = json.load(f)\n\n# Identify merchant row/columns\nm = merchants.copy()\nmerchant_col = next((c for c in m.columns if 'merchant' in c.lower() and m[c].astype(str).eq('Belles_cookbook_store').any()), None)\nif merchant_col is None:\n for c in m.columns:\n if m[c].astype(str).str.contains('Belles_cookbook_store', na=False).any():\n merchant_col = c\n break\nrow = m[m[merchant_col].astype(str)=='Belles_cookbook_store'].iloc[0]\n\n# likely merchant id column\nmerchant_id_cols = [c for c in m.columns i…File "<stdin>", line 143
PY'
^
SyntaxError: unterminated string literal (detected at line 143)
> tool_call bash({"command":"ls -R . 2>/dev/null | head -50"})data
./data:
acquirer_countries.csv
fees.json
manual.md
merchant_category_codes.csv
merchant_data.json
payments-readme.md
payments.csv
Data-analysis QA over a shared payments dataset and a business-rules manual, answered with pandas in a Python shell.
FinanceBench document QA
> tool_call bash({"command":"ls docs && grep -inE \"3M|capital expenditure|capital expenditures|purchases of property|additions to property|cash flow\" docs/*.txt && awk 'FNR==1{print \"\\n### \" FILENAME} {print}' docs/*.txt | sed -n '/3M/,$p' | grep -inA3 -B3 -E \"Purchases of property, plant and equipment|Capital expenditures|Additions to property|Property, plant and equipment\" && printf 'SUBMIT\\nAnswer: FY2018 capital expenditure for 3M was USD 1,577 million.\\n'"})financebench_id_00070_ev1.txt
financebench_id_00859_ev0.txt
financebench_id_01487_ev0.txt
financebench_id_03029_ev0.txt
financebench_id_04412_ev0.txt
docs/financebench_id_03029_ev0.txt:2:3M Company and Subsidiaries
docs/financebench_id_03029_ev0.txt:3:Consolidated Statement of Cash Flow s
docs/financebench_id_03029_ev0.txt:14:Cash Flows from Operating Activities
docs/financebench_id_03029_ev0.txt:151:Cash Flows from Investing Activities
docs/financebench_id_03029_ev0.txt:159:Purchases of property, plant and equipment (PP&E)
docs/financebench_id_03029_ev0.txt:231:Cash Flows from Financing Activities
155-
156-
157-
158:Purchases of property, plant and equipment (PP&E)
159-
160-
161-(1,577)
SUBMIT
Answer: FY2018 capital expenditure for 3M was USD 1,577 million.
> tool_call bash({"command":"ls docs && grep -inE \"3M|property|plant|equipment|PP&E|PPNE|balance sheet\" docs/*.txt && sed -n '1,220p' docs/* | grep -inE \"property|plant|equipment|net\" | head -n 40 && printf 'SUBMIT\\nAnswer: 3M FY2018 net PPNE was $8.7 billion.\\n'"})financebench_id_00799_ev0.txt
financebench_id_01912_ev2.txt
financebench_id_04481_ev0.txt
financebench_id_04672_ev0.txt
financebench_id_08286_ev0.txt
docs/financebench_id_00799_ev0.txt:2:Consolidated Balance Sheets
docs/financebench_id_00799_ev0.txt:34:Property, plant, and equipment, net
docs/financebench_id_04672_ev0.txt:2:3M Company and Subsidiaries
docs/financebench_id_04672_ev0.txt:96:Property, plant and equipment
docs/financebench_id_04672_ev0.txt:108:Property, plant and equipment net
docs/financebench_id_04672_ev0.txt:228:3M Company shareholders equity:
docs/financebench_id_04672_ev0.txt:276:Total 3M Company shareholders equity
14:Trade receivables, net of allowance for credit losses of $21 and $25, respectively
17:Inventories, net
27:Assets held for sale, net
34:Property, plant, and equipment, net
43:Other intangible assets, net
153:Net revenues of $674 million in the current yea…
> tool_call bash({"command":"awk 'BEGIN{p=0} /3M Company and Subsidiaries/{p=1} p{print NR \":\" $0} NR>170 && p{exit}' docs/financebench_id_04672_ev0.txt | sed -n '90,120p'; printf 'SUBMIT\\nAnswer: 3M FY2018 net PPNE was $8.7 billion.\\n'"})91:
92:
93:13,709
94:
95:14,277
96:Property, plant and equipment
97:
98:
99:24,873
100:
101:24,914
102:Less: Accumulated depreciation
103:
104:
105:(16,135)
106:
107:(16,048)
108:Property, plant and equipment net
109:
110:
111:8,738
112:
113:8,866
114:Goodwill
115:
116:
117:10,051
118:
119:10,513
120:Intangible assets net
121:
SUBMIT
Answer: 3M FY2018 net PPNE was $8.7 billion.
> tool_call bash({"command":"files=$(ls docs); rev=$(grep -RinE \"Net sales|Sales\" docs/*.txt | head -n 20); cfo=$(grep -RinE \"Purchases of property, plant and equipment|Capital expenditures|Additions to property, plant and equipment\" docs/*.txt | head -n 20); dep=$(grep -RinE \"Depreciation|Amortization\" docs/*.txt | head -n 20); printf '%s\\n' \"$files\" >/tmp/sib_files.txt; printf '%s\\n' \"$rev\" >/tmp/sib_rev.txt; printf '%s\\n' \"$cfo\" >/tmp/sib_cfo.txt; printf '%s\\n' \"$dep\" >/tmp/sib_dep.txt; python - <<'PY'\nimport re, glob, os\ntext=\"\"\nfor f in glob.glob(\"docs/*.txt\"):\n with open(f, 'r', encoding='utf-8', errors='ignore') as fh:\n text += \"\\nFILE:\"+f+\"\\n\"+fh.read()\n\ndef find_money_near(patterns, window=300):\n vals=[]\n for pat in patterns:\n for m in re.finditer(pat, text, flags=re.I):\n s=text[max(0,m.start()-window):m.sta…File "<stdin>", line 64
PY'
^
SyntaxError: unterminated string literal (detected at line 64)
Financial-document QA over real SEC-filing evidence excerpts: the agent greps a workspace of evidence docs plus distractors and submits a final answer.
τ²-bench telecom
> tool_call get_customer_by_phone({"phone_number":"555-123-2002"}){"customer_id": "C1001", "full_name": "John Smith", "date_of_birth": "1985-06-15", "email": "john.smith@example.com", "phone_number": "555-123-2002", "address": {"street": "123 Main St", "city": "Anytown", "state": "CA", "zip_code": "90210"}, "account_status": "Active", "payment_methods": [{"method_type": "Credit Card", "account_number_last_4": "1235", "expiration_date": "12/2026"}], "line_ids": ["L1001", "L1002", "L1003"], "bill_ids": ["B1001", "B1002", "B1003"], "created_at": "2025-01-15 10:30:00", "last_extension_date": null, "goodwill_credit_used_this_year": 25.0}
> tool_call get_details_by_id({"id":"L1001"}){"line_id": "L1001", "phone_number": "555-123-2001", "status": "Active", "plan_id": "P1001", "device_id": "D1001", "data_used_gb": 3.2, "data_refueling_gb": 0.0, "roaming_enabled": false, "contract_end_date": "2026-12-31", "last_plan_change_date": "2025-01-10", "last_sim_replacement_date": null, "suspension_start_date": null}
> tool_call get_details_by_id({"id":"L1002"}){"line_id": "L1002", "phone_number": "555-123-2002", "status": "Active", "plan_id": "P1002", "device_id": "D1002", "data_used_gb": 8.7, "data_refueling_gb": 0.0, "roaming_enabled": true, "contract_end_date": "2026-12-31", "last_plan_change_date": "2024-12-15", "last_sim_replacement_date": "2025-01-20", "suspension_start_date": null}
> tool_call get_details_by_id({"id":"L1003"}){"line_id": "L1003", "phone_number": "555-123-2003", "status": "Suspended", "plan_id": "P1001", "device_id": "D1003", "data_used_gb": 0.0, "data_refueling_gb": 0.0, "roaming_enabled": false, "contract_end_date": "2026-06-30", "last_plan_change_date": "2024-10-05", "last_sim_replacement_date": null, "suspension_start_date": "2025-02-01"}
The telecom slice of τ²-bench with an optimized environment prompt (317 optimization rollouts). Same customer-service tool surface as tau-bench, tuned for the telecom domain's device and plan workflows.
Build your own
+your traces → simulation
Turn your own traces into a steppable environment.