Evaluation Tool
Before You Start
Batch evaluation is open to all agent developers and is used to verify the effectiveness of experts and skills. After you select the asset to evaluate and a set of cases, the platform runs the evaluation automatically and provides an overall score, pass rate, major issues, and run logs for your iteration reference.
Batch evaluation does not perform security review or content moderation, and evaluation results are not a basis for asset publication.
Entry and Flow
Go to the Batch Evaluation menu in the left navigation of the Open Platform, which contains two pages: Evaluation Management and Dataset Management.
For first-time use, follow this order: prepare and upload a dataset → wait for validation to pass → create and start an evaluation task → view and export results.
Quota and Billing
- Currently in free trial: each developer can start 10 evaluations per day, reset the next day, with no points system involved.
- Each Start or Restart consumes 1 quota; once a task is canceled, the consumed quota is not refunded.
- After the daily quota is used up, you cannot start new tasks or restart evaluations until the quota resets the next day.
Prepare and Upload a Dataset
A dataset is the source of cases for an evaluation task. This section explains how to download the template, prepare the files, upload them, and confirm the validation result.
Download the Dataset Template
Go to Batch Evaluation / Dataset Management, click New Dataset in the upper right corner, and click Download Evaluation Template in the dialog to get the dataset template.
Prepare the Dataset Files
A dataset consists of cases, default scoring settings, and optional input files. The following uses two cases from the template to illustrate how to fill them in: the inventory-counting case carries an input file, and the copy-length-limit case demonstrates custom scoring rules for a single case. The dimensions, weights, and thresholds are all example configurations and do not represent fixed platform rules; the field descriptions cover only this example and are not a complete interface specification.
File structure
评测集根目录/
├── cases.jsonl
├── header.json
└── workspace/
└── case_001/
└── supply_catalog.csv| Member | Required | Description |
|---|---|---|
cases.jsonl | Required | Case collection: includes the question, expected answer, and per-dimension scoring rubric |
header.json | Required | Dataset-level default settings: default scoring dimensions, default weights, and the default score aggregation rule |
workspace/ | Optional | Input files for some cases (such as inventory sheets or email drafts), organized into case_NNN subdirectories |
Fill in header.json: set the dataset default rules
Fill in the dataset name first, then configure the default scoring dimensions, weights, and thresholds. The example below can serve as a starting template for header.json:
{
"dataset_name": "marketing-skill-eval",
"profile": {
"dimensions": [
{
"key": "task_completion",
"name": "任务完成度",
"weight": 0.45
},
{
"key": "instruction_follow",
"name": "指令遵循",
"weight": 0.3
},
{
"key": "factuality",
"name": "事实准确性",
"weight": 0.15
},
{
"key": "rule_compliance",
"name": "规则合规",
"weight": 0.1
}
],
"aggregation": {
"pass_threshold": 60,
"excellent_threshold": 85
}
}
}| Field | Example | How to fill in |
|---|---|---|
| dataset_name | marketing-skill-eval | The dataset name; replace it with something easy to recognize. |
| profile.dimensions | Four scoring dimensions | Each object contains key (dimension identifier), name (display name), and weight. A case's scoring rubric maps to dimensions via key. |
| profile.dimensions[].weight | 0.45 / 0.30 / 0.15 / 0.10 | The example weights sum to 1. Adjust them to your business priorities; keeping the total at 1 is recommended. |
| profile.aggregation.pass_threshold | 60 | Pass threshold; the example sets it to 60, which is not the platform-wide pass line. |
| profile.aggregation.excellent_threshold | 85 | Excellent threshold; the example sets it to 85, which is not the platform-wide excellent line. |
Fill in cases.jsonl: define each case
Each line is a complete JSON object describing one user task and its expected result. Separate JSON objects with newlines only — no commas, and do not wrap them in an array. When filling it in, keep the user question, expected result, and scoring rubric on the same judgment standard.
| Field | Content type | How to fill in |
|---|---|---|
| case_name | String | The case name, such as inventory counting or copy length limit, for easy identification in the report. |
| user_question | String | The actual user question sent to the asset under test. Describe the input, operation requirements, output format, and limits. |
| expected | String | A verifiable expected result; it does not have to be the single correct answer, but should make clear which behaviors or results count as completing the task. |
| difficulty / case_type | String | The example difficulty is easy, and the types used are negative and boundary, describing difficulty and test scenario respectively; refer to the platform template for the full set of allowed values and validation requirements. |
| checklist_by_dim | Dimension identifier → string array | The specific items to check for each dimension. Keys should match the scoring dimension identifiers used by the case. |
| anchors_by_dim | Dimension identifier → score with description | Describes the performance corresponding to different scores. The example uses string keys 0, 5.0, and 10.0 for not completed, partially completed, and well completed respectively. |
| dimension_weights | Dimension identifier → number | Case-level scoring weights. The inventory-counting case omits this and inherits the defaults; the copy case sets its own five-dimension weights. |
| aggregation | Object | Case-level threshold configuration; you can set pass_threshold and excellent_threshold. Unspecified rules inherit from header.json. |
| input_files | Object array | Declare this when there are attached input files; each object uses rel to point to a file inside the dataset package. Cases without files in the example omit this field. |
Upload and Confirm the Validation Result
In the New Dataset dialog, fill in the dataset name, upload the files, and submit. A dataset can be submitted as a directory or as a whole zip package; the zip root directory should match the dataset template — do not nest another folder inside.
After a successful upload, the system automatically generates a dataset ID. Dataset names may be duplicated; datasets are distinguished by dataset ID.
After upload, the dataset enters the Validating state, which is expected to take about 2 minutes, during which it cannot be used to start an evaluation. Once validation passes, the status changes to Available.
When validation fails, the system shows the specific line numbers and field errors; fix the files as prompted and upload again.
Create and Manage Evaluation Tasks
Create an Evaluation Task
Go to Batch Evaluation / Evaluation Management and click New Evaluation Task in the upper right corner to open the dialog.
Fill in the following three required fields, and select a dataset that has passed validation:
| Field | Required | Description |
|---|---|---|
| Task name | Required | Up to 50 characters |
| Asset to evaluate | Required | You can type keywords in the search box to filter uploaded assets |
| Dataset | Required | You can type keywords in the search box to filter uploaded datasets |
After you click Start Evaluation, the system checks your remaining evaluations for the day; if the check passes, the task is created and the evaluation starts automatically. See Quota and Billing for quota rules.
View Tasks and Restart
View task status in the Evaluation Management list. After a task completes, click View Results to open the evaluation report.
To evaluate again, use the Restart entry in the task list; see Quota and Billing for quota consumption and quota handling after cancellation.
Interpret and Export Evaluation Results
View the Report
The evaluation report contains four parts:
- Overall score and statistics: the total score of this evaluation (out of 100), plus the number of cases, passes, failures, pass rate, and total duration.
- Major issues: a list of issues sorted by severity, each linked to a specific case, making it easy to locate the corresponding run log.
- Run logs: execution records of each case, also called Trace in the interface.
View Case Details
Click a case row in the run log table to open the case detail page and view the user input and run logs. Combine this with the major issues list to locate the asset behaviors that need adjustment.
Export and Share
HTML report: click Export Report at the bottom of the report to generate a self-contained single HTML file that can be opened, forwarded, or archived directly. The file contains task metadata, points consumption, overall score and statistics, major issues, and the full run log table.
