Hi ToolBench team — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results, whose benchmarks/ directory converts public benchmarks into validated EvalPort suites.
I read a real example from this repo, data_example/instruction/inference_query_demo.json:
{"query": "I'm planning a surprise party for my best friend, and I want to include meaningful quotes in the decorations. Can you provide me with random love, success, and motivation quotes? ...", "query_id": 82217, "api_list": [{"category_name": "Social", "tool_name": "Olato Quotes", "api_name": "Love Quote"}, {"category_name": "Social", "tool_name": "Olato Quotes", "api_name": "Success Quote"}, {"category_name": "Social", "tool_name": "Olato Quotes", "api_name": "Motivation Quote"}]}That maps onto EvalPort cleanly: query -> TestCase.input, api_list (the tools the instruction is grounded in) -> TestCase.expected_output, graded with a code grader matching invoked APIs against api_list.
Would your maintainers be open to an EvalPort-format export of a ToolBench instruction subset (G1/G2/G3) living in EvalPort's benchmarks/, with attribution and license preserved? Happy to do the conversion work and open a PR for review first. No worries if this isn't useful right now.
Hi ToolBench team — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results, whose
benchmarks/directory converts public benchmarks into validated EvalPort suites.I read a real example from this repo,
data_example/instruction/inference_query_demo.json:{"query": "I'm planning a surprise party for my best friend, and I want to include meaningful quotes in the decorations. Can you provide me with random love, success, and motivation quotes? ...", "query_id": 82217, "api_list": [{"category_name": "Social", "tool_name": "Olato Quotes", "api_name": "Love Quote"}, {"category_name": "Social", "tool_name": "Olato Quotes", "api_name": "Success Quote"}, {"category_name": "Social", "tool_name": "Olato Quotes", "api_name": "Motivation Quote"}]}That maps onto EvalPort cleanly:
query->TestCase.input,api_list(the tools the instruction is grounded in) ->TestCase.expected_output, graded with acodegrader matching invoked APIs againstapi_list.Would your maintainers be open to an EvalPort-format export of a ToolBench instruction subset (G1/G2/G3) living in EvalPort's
benchmarks/, with attribution and license preserved? Happy to do the conversion work and open a PR for review first. No worries if this isn't useful right now.